A Feed River Wire

A few days ago, Dave Winer published a firehose for feeds that is currently hooked into a near real time feed of stories from the NY Times. In attempting to figure out what to do with a service like this, I find pushing out a first draft always seems to help. And since the explanation of the first draft is longer than a comment, here’s a blog post. 🙂

The Feed River Wire can be broken down as:

  1. Reads feeds. In this case, the NY Times feeds with support from Dave’s long-poll RSS server.
  2. Displays feeds as a river. In this context, to me, a river means– if it isn’t new, then it isn’t news. Dive in and read. No worries.
  3. Hooks the river to the wire. As new items are pulled in from the long-poll server, they are displayed on the front page almost immediately.

The river flows by.

Now, for the technical details.

Every hour, a script runs. This script is allowed to run anywhere from 55-58 minutes before it dies.

This script looks at the last known seed (cursor) and makes a request from Dave’s server for any items that have come in during the period since the last request. It tells the server to listen to its request for 30 seconds if no data is immediately available.

If data becomes available during that time, it is served up immediately.

If data does not become available during that 30 seconds, the script pauses for 60 seconds and then runs again. This pause time fluctuates depending on the number of empty requests. We don’t want to pound a server that isn’t giving up data, that won’t help.

If for some reason we go around 20 minutes without an update, the script stops and waits for the next hourly start trigger. I figure if we’ve gone 20 minutes without news from the NY Times, it’s either a slow news day or we can take a break during the middle of the night until the next hourly script is run.

On the front end, the end user is greeted with the 20 most recent items of the river whenever the page is loaded. If the page is not closed, it will ping back to the server every 5 seconds looking for new items. If one is found, it will populate.


So check it out. No fancy bells and whistles, but I learned a couple things and the wheels are turning on draft two. Hopefully Dave can get a few more feeds hooked into the hose. 🙂

If Users Could Choose How They Publish

I mentioned something briefly the other day that I’ve been rolling in my head a bit ever since. A perfect topic for #blogpostfriday. 🙂

We have a bunch of fun new tech coming out of the web right now. At least two of those things, pubsubhubbub and rssCloud, deal with adding elements to feeds that tell subscribers how to sign up for quick update notification.

The painful part of this new tech is that not everybody who wants to enable it has a choice.

Sure, we could all publish our material directly and be responsible for the structure of our feeds, but that isn’t reality. We rely on Twitter to publish our Tweets, Google to publish our Shared Items, WordPress.com to publish our Blogs– the list goes on and on.

Example. I’m aching to try rssCloud concepts out on every feed that I can get my hands on. But I can’t. The only feed I can immediately provide the cloud element in is the one generated from the application that I wrote specifically for the purpose.

In a perfect world (ha), Twitter, Reader, WordPress would say – “Hi User – here are the optional elements available in ..insert publishing spec.. that we aren’t using. Would you like to fill them in?” – Then bam, I get to test cloud support on several different feeds immediately. There would still be a little manual work in notifying the cloud server when things are updated, but everyone who subscribes to the feed would know what server they should subscribe through for instant updates.

Now, spread the concept out a bit — if I can choose where the material is published, which specification is used, as well as what’s included in it, then I as a user have a wonderful amount of freedom in being able to do what I want to with my data. That is what publishing tools should be focused on.