As noted in a previous post, your.flowingdata.com (yfd) is a handy way to collect personal data. I've been collecting sleep, diaper, etc. data on my newborn son. Although yfd now allows users to calculate durations between specified events, the visualization of the information isn't quite to my liking and it's clear that errors in the data can make for some odd durations (e.g., my son slept for two days!). Numpy to the rescue!

For those of you who don't know, numpy is python's powerful array package. Rather than loop myself to death, I thought it made more sense to use of numpy's powerful slicing and masking features to clean up the data. These features make it easy to find data entry errors.

I use the Enthought python distribution for convenience sake (and because I can't resist all those libraries -- most of which I'll never use).   Below you'll find some screenshots that step through my little script. Refer to the complete code here. (Well, it's just a start really). The code is probably a bit verbose for what it does, but we all start somewhere.

The first step is getting the data into an array you can manipulate. For your reference, your.flowingdata yields data that looks like this:

array([('gmorning', '', '', '2009-10-24 23:45:36'),('gmorning', '', '', '2009-09-30 18:15:04'), ('gnight', '', '', '2009-09-23 21:00:03'), ('gmorning', '', '', '2009-09-23 19:15:03')])

I won't step through all the code here since it's available above, but you get the idea. One thing to watch out for: datetimes. I spent a lot of time trying to figure out the best way to handle the timestamps included with the yfd event data. There are ways to convert strings to ordinal numbers into datetime objects and back again, but really I wanted to manipulate the datetime objects directly to take advantage of numpy's array slicing and arithmetic. Luckily, numpy allows object types (technically, they are called 'dtypes'). This allows you to subtract one timestamp array from another to get the elapsed time without any conversions (though you'll have to convert at some point if you want to generate a human-readable string). Here's an example of the array you'll get at the end (heads -> sleep duration, start time, end time):

Just saw this...

Some friends of mine are hosting a talk by Kerry Hilton, the founder of Freeset. From the website:

Freeset exists specifically to provide freedom for women from the sex trade, women who were forced into prostitution by trafficking or poverty. These women didn't choose their profession — it was chosen for them.

Now, they're being offered a real choice. When they choose to work at Freeset, they can start new lives, regain dignity in their communities, and begin a journey towards healing and wholeness.

All profits from Freeset in Kolkata benefit the women (salary, health insurance and retirement plan) and are used to grow the business. This means more women can be employed and experience freedom.

The great thing is, when you buy a Freeset product, you directly participate in a woman's journey to freedom.

Freeset trains these women to make custom bags and tee shirts. I'm not sure how differentiated the bags are from other bags, but the story is pretty unique.

The talk starts at 2:30pm this Sunday in Tarrytown, NY at the Reformed Church of the Tarrytowns (42N Broadway, Tarrytown, NY). Stop by if you want to learn more.

This story deserves repeating (care of Gizmodo).

UPDATE: He just made it onto the Daily Show.

Personal data capture is a meme that's gaining momentum. Products such as Nike+ and, more recently, Fitbit, target those who would like to monitor daily exercise and other activities. Websites that allow users to manually track how they use their time have also started to pop-up. For those of us that like to procrastinate, these monitoring tools can help by providing regular feedback. Watching a little line move in the right direction can be pretty motivating.

Of course, I don't use any of these services. For myself.

Nevertheless, as a new father, I've found that your.flowingdata.com is an easy and useful way to track the activities of my newborn son! The service uses tweets to capture pretty much any kind of data you'd care to record. There are electronic products (e.g., Itsbeen, basically a stopwatch on steroids) that help new parents keep track of when the baby last slept, ate, poo'ed, etc. They do not, however, capture that data for analysis. My wife and I would like to see the historical data to see if we can tease out some insights about our son (e.g., how much sleep does he need before he gets cranky?). We tried using an iPhone app called Blogger that helps parents keep track of these things, but it wasn't immediate enough. We ended-up writing down events on the nursery mirror with a dry erase pen, but I really wanted to track things via a single button press. By the time I've finished dodging multiple salvos of pee and poo, multiple diaper changes due to said peeing and pooing, spit-up, puking, and sundry other lovely activities (a testament to how much I love you, boy), I can't remember anything that's happened in the last five minutes, let alone the last hour or two. So far, your.flowingdata.com has been the answer.

your.flowingdata.com ('yfd') is a service based on Twitter. Users send direct messages to 'yfd' and can visit the site for simple visualizations. Users can also download tab-delimited files with all the data. But wait, there's more! One kind soul also created a simple yfd iPhone application that allows users to send an update (e.g. 'd yfd gnight') via a single button press. Each button can be customized as well. I have no use for Twitter, but yfd got me to open an account. We're still figuring out what we want to record, but the service's flexibility and ease-of-use makes it much more likely we'll actually use it.

yfd isn't perfect. There's no built-in way to, for example, calculate the time that has elapsed between two actions (e.g. going to sleep and waking up). One has to download the data and calculate durations manually (or create a script to do it). There are other visualizations available, though. As I mentioned, I find it's much more important to make it easy to capture data for something like this. If it's a pain to capture the data, there won't be anything to analyze on the back-end anyway.

So, if you have absolutely no interest in personal fitness, time tracking, etc., you may want to check out your.flowingdata.com...for the children.

UPDATE: yfd has been updated to allow the calculation of durations between defined actions. I'd love to be able to aggregate these durations over a given time period (i.e. daily, weekly, monthly, etc.) in the form of a bar chart or something. yfd does visualize the data, but in a slightly different way. Best if you just check it out through the "Explore" link on the yfd site.

I've fallen behind on my More Or Less podcast, produced by the BBC and hosted by Tim Harford (author of The Undercover Economist). While the radio show naturally focuses on UK issues, the questions raised could easily be asked in any country. The August 7th episode was particularly good. The first story debunks a widely watched YouTube video that argues Europe will be overrun by Islamic fertility. The second story discusses how a Home Office paper that asserts the innocent are as likely as the guilty to go on to commit crimes is misleading. A member of the family is a public defender, so I was interested to hear the analysis. Finally, as a lapsed musician, I was delighted to learn more about the math underlying the scales I practiced so often (I had no idea). You can find the episode at http://news.bbc.co.uk/2/hi/programmes/more_or_less/8189480.stm

Although More Or Less features plenty of lighter fare, it does a great job of highlighting how statistics can be used for good and evil in the political arena and why it is important for citizens to dig a little deeper before taking a number for granted. Plus, the Brits just sound, like, wicked smart...