A Feed River Wire

A few days ago, Dave Winer published a firehose for feeds that is currently hooked into a near real time feed of stories from the NY Times. In attempting to figure out what to do with a service like this, I find pushing out a first draft always seems to help. And since the explanation of the first draft is longer than a comment, here’s a blog post. 🙂

The Feed River Wire can be broken down as:

  1. Reads feeds. In this case, the NY Times feeds with support from Dave’s long-poll RSS server.
  2. Displays feeds as a river. In this context, to me, a river means– if it isn’t new, then it isn’t news. Dive in and read. No worries.
  3. Hooks the river to the wire. As new items are pulled in from the long-poll server, they are displayed on the front page almost immediately.

The river flows by.

Now, for the technical details.

Every hour, a script runs. This script is allowed to run anywhere from 55-58 minutes before it dies.

This script looks at the last known seed (cursor) and makes a request from Dave’s server for any items that have come in during the period since the last request. It tells the server to listen to its request for 30 seconds if no data is immediately available.

If data becomes available during that time, it is served up immediately.

If data does not become available during that 30 seconds, the script pauses for 60 seconds and then runs again. This pause time fluctuates depending on the number of empty requests. We don’t want to pound a server that isn’t giving up data, that won’t help.

If for some reason we go around 20 minutes without an update, the script stops and waits for the next hourly start trigger. I figure if we’ve gone 20 minutes without news from the NY Times, it’s either a slow news day or we can take a break during the middle of the night until the next hourly script is run.

On the front end, the end user is greeted with the 20 most recent items of the river whenever the page is loaded. If the page is not closed, it will ping back to the server every 5 seconds looking for new items. If one is found, it will populate.


So check it out. No fancy bells and whistles, but I learned a couple things and the wheels are turning on draft two. Hopefully Dave can get a few more feeds hooked into the hose. 🙂

Twitter As News

Brief thoughts on an idea that’s been extensive in my head lately.

Twitter has long been talked about as a news source. If it ever decides to work as a news source, or if we ever decide to actually use it as one, there will be two parts:

  1. The raw news will be complete madness, and everybody will be allowed in.
  2. Many somebodies, from many different groups and interests, will harness the madness into usefulness.

This is easier (ha) than it sounds.

Six degrees of separation, or something similar to that, will determine that I as an editor/publisher can choose my trusted raw news sources that cover my area of interest. If I’m looking for local news in my town of 30,000, I only need a handful of trusted sources sending out raw news in order for me to edit/publish an aggregation of news for that locality. A larger city (Chicago) only requires a handful more sources.

For edit/publishing tasks, I will have a way to strip out tweets that are unrelated from that stream and a way to categorize the tweets that are related. Once a category (story) has valuable information from enough sources, it will be published. Once published, it remains alive. I can remove/add information and sources. Everyone can react, republish, comment, or do whatever they would like with that story.

More later I think. I might be working on it.

Accident at Toncontin International Airport

There was a plane crash at Toncontin International Airport in Tegucigalpa, Honduras. The airport is known for a difficult approach no matter what type of plane your flying. Luckily (if you want to call it that) only 5 people were killed and this is the first accident at this airport since 1997. Now keep those facts in mind and watch these two videos:

11 years since the last accident, and that was a US Airforce cargo plane. I’m impressed.

[1] The video on YouTube was removed by the original user, so I removed the link that was here.