Use numpy to flog your.flowingdata

As noted in a previous post, your.flowingdata.com (yfd) is a handy way to collect personal data. I've been collecting sleep, diaper, etc. data on my newborn son. Although yfd now allows users to calculate durations between specified events, the visualization of the information isn't quite to my liking and it's clear that errors in the data can make for some odd durations (e.g., my son slept for two days!). Numpy to the rescue!

For those of you who don't know, numpy is python's powerful array package. Rather than loop myself to death, I thought it made more sense to use of numpy's powerful slicing and masking features to clean up the data. These features make it easy to find data entry errors.

I use the Enthought python distribution for convenience sake (and because I can't resist all those libraries -- most of which I'll never use). Below you'll find some screenshots that step through my little script. Refer to the complete code here. (Well, it's just a start really). The code is probably a bit verbose for what it does, but we all start somewhere.

The first step is getting the data into an array you can manipulate. For your reference, your.flowingdata yields data that looks like this:

As you can see, it's basically just events and timestamps (I'm not really making full use of the data types yfd offers, as shown by all the empty fields).

The code below creates a structured array. Typically, numpy arrays are made up of items of the same type. It occurs to me that this example isn't so great because I ended up sticking with strings (S10 = a ten character string), but you get the general idea. If you imagine a 2D array, you can define one column as floats, another as strings, and yet another as int, etc. I'm mostly interested in how much the little guy is sleeping, so the 'sleep_mask' variable creates a boolean mask of all the 'gnight' and 'gmorning' events (since they are mixed in with diaper changes and other random events).

We can use numpy's where() method to help us index the events we want. Now that I have an array of only gnight and gmorning events, I can offset the two (since they alternate) to see if there are any duplicates that might screw things up.

The first time I called 'errors', numpy returned something like the following (basically telling me when/where there are dupes):

array([('gmorning', '', '', '2009-10-24 23:45:36'),('gmorning', '', '', '2009-09-30 18:15:04'), ('gnight', '', '', '2009-09-23 21:00:03'), ('gmorning', '', '', '2009-09-23 19:15:03')])

I won't step through all the code here since it's available above, but you get the idea. One thing to watch out for: datetimes. I spent a lot of time trying to figure out the best way to handle the timestamps included with the yfd event data. There are ways to convert strings to ordinal numbers into datetime objects and back again, but really I wanted to manipulate the datetime objects directly to take advantage of numpy's array slicing and arithmetic. Luckily, numpy allows object types (technically, they are called 'dtypes'). This allows you to subtract one timestamp array from another to get the elapsed time without any conversions (though you'll have to convert at some point if you want to generate a human-readable string). Here's an example of the array you'll get at the end (heads -> sleep duration, start time, end time):

Another unexpected pain in the butt is TIMEZONES. Although yfd's UI shows the correct local time on the web page, the tab-delimited file uses UTC (GMT) timestamps. This actually makes sense if you think about it. If you travel a lot, you'll never be sure when something happened since your timezone isn't held constant. Keeping datetime in UTC solves this problem, though you have to convert to local time yourself if necessary. Handling timezones with python's datetime library, however, sort of sucks. I recommend checking out pytz. It makes timezone management a little bit easier.

Plans for the future include visualizing this data with either python or R (ggplot2 anyone?). Too bad I don't know R...

1 comment

Oct 29, 2009

nathan said...

you know python, so you should be able to pick up R pretty easily. just like any other old programming language :)

NotesToSelf

NotesToSelf

Use numpy to flog your.flowingdata

1 comment

Leave a comment...