A few days ago, Dave Winer published a firehose for feeds that is currently hooked into a near real time feed of stories from the NY Times. In attempting to figure out what to do with a service like this, I find pushing out a first draft always seems to help. And since the explanation of the first draft is longer than a comment, here’s a blog post. 🙂
The Feed River Wire can be broken down as:
- Reads feeds. In this case, the NY Times feeds with support from Dave’s long-poll RSS server.
- Displays feeds as a river. In this context, to me, a river means– if it isn’t new, then it isn’t news. Dive in and read. No worries.
- Hooks the river to the wire. As new items are pulled in from the long-poll server, they are displayed on the front page almost immediately.
The river flows by.
Now, for the technical details.
Every hour, a script runs. This script is allowed to run anywhere from 55-58 minutes before it dies.
This script looks at the last known seed (cursor) and makes a request from Dave’s server for any items that have come in during the period since the last request. It tells the server to listen to its request for 30 seconds if no data is immediately available.
If data becomes available during that time, it is served up immediately.
If data does not become available during that 30 seconds, the script pauses for 60 seconds and then runs again. This pause time fluctuates depending on the number of empty requests. We don’t want to pound a server that isn’t giving up data, that won’t help.
If for some reason we go around 20 minutes without an update, the script stops and waits for the next hourly start trigger. I figure if we’ve gone 20 minutes without news from the NY Times, it’s either a slow news day or we can take a break during the middle of the night until the next hourly script is run.
On the front end, the end user is greeted with the 20 most recent items of the river whenever the page is loaded. If the page is not closed, it will ping back to the server every 5 seconds looking for new items. If one is found, it will populate.
Cool.
So check it out. No fancy bells and whistles, but I learned a couple things and the wheels are turning on draft two. Hopefully Dave can get a few more feeds hooked into the hose. 🙂