Phillip Pearson - web + electronics notes

tech notes and web hackery from a new zealander who was vaguely useful on the web back in 2002 (see: python community server, the blogging ecosystem, the new zealand coffee review, the internet topic exchange).

2003-12-16

Prototype of a way to make weblog polling way more efficient

One topic that just won't die is the one about how polling sucks for RSS.

It starts with someone complaining that having aggregators polling heaps of feeds all the time is inefficient, or that you don't get news until you poll, but it's not polite to poll as often as you'd like.

Then, someone either says that it would be too hard to work out an alternative, or that you shouldn't want to read news all that much anyway, or something.

Ignoring all that, here's something I hacked up tonight. It's an incremental improvement on what Dave Winer did with Weblogs.Com -- the latter maintains a big XML file of URLs of blogs that have recently updated, so if you poll that one file often enough, you can avoid having to poll a whole bunch of blogs individually.

That's a huge improvement over polling all your blogs, but it's not as efficient in terms of bandwidth as it could be, as you end up downloading the whole XML file (which gets pretty big) every time something changes.

SO, I've set up something on the Topic Exchange server that collects pings (currently from blo.gs only, because it's easy, but I'd like to get it working with Weblogs.Com too) and sends them back out to clients. Clients connect in on port 9123, and it sends a single-line message to all clients when a ping is received.

Telnet to topicexchange.com port 9123 to see what I mean. For Windows users, hit the Start button, then click Run, and type telnet topicexchange.com 9123, then click OK. A window will open, and you'll see blog URLs appearing there as they update. To see how many users are connected, open this stats page.

It's just a prototype, so the protocol may change, and I'll almost certainly put in some sort of (voluntary) identification process so people who want to be visible will show up on the stats page ... anybody interested in using this in their aggregator product, do get in touch () and we can work out how best to make everything usable.

... more like this: [, ] ... topic exchange: []