Phillip Pearson - Second p0st

tech notes and web hackery from the guy that brought you bzero, python community server, the blogging ecosystem, the new zealand coffee review and the internet topic exchange

2003-10-14

Python Community Server has a new search engine

OK, I finally got a search engine working with Python Community Server that can search down to the individual post and give you decent context in the result pages.

It's very very simple, but quite effective. I've wasted a lot of time trying to get first ht://Dig (in C++) and then Lucene (in Java), integrated into my Python code ... but this one works.

It does a simple linear search through a table of cached posts. You might think that would be slow, except some benchmarking here suggests it should be able to walk through many thousands of posts (informal benchmarks say 20,000) per second. If you post five times a day, every day, it'll take ten years before a search of your blog will take a second... so I'm not too worried.

Like the rest of PyCS, it's designed to work with 'distributed' blogging tools like Radio. These have the annoying property that the server usually doesn't know anything about the text of your posts -- it only ever sees the rendered HTML. To get around this, I've added an XML-RPC call that the client needs to call after posting, that sends the content of the post to the community server so it can be searched.

    struct xmlStorageSystem.mirrorPosts(email, password, posts)

The first two parameters are the same as for the other xmlStorageSystem methods. The last one is a list of structs, one for each post you want to put in the index. (If you've just made a new post, there will only be one entry in the list, but if you're initialising the server with a few hundred old posts, you'll want to send them in 10 at a time). Each struct needs the following members:

  date: an XML-RPC DateTime value; this is the date of the post.

  postid: a string. This is used to determine if you have sent the same post before (in which case it will be updated).

  url: a string; the permalink of the post. This is used on the results page.

  guid: currently unused; expected to be the same as the url parameter, but if you have some other sort of GUID for your posts (e.g. what some Movable Type users put in their RSS), you could put it here.

  title: a string; the title of your post. This is shown on the results page.

  description: a string; the content of your post. This is shown on the results page.

You can search it from the /system/search.py page. For example, search the pycs-devel mailing list archive blog. It doesn't do phrases, but does allow required inclusions and exclusions. Here are some examples:

  search: georg +xml-rpc -- Georg talking about XML-RPC

  search: georg -bauer -- people talking to Georg

  search: phil -pearson -- people talking to me

Right now the only tool supporting this is my bzero. If somebody wants to hack Radio or PyDS support in, that would be greatly appreciated. I might do Radio myself if I have time...
... more like this: [, ] ... topic exchange: [, ]

NewsBruiser + Topic Exchange integration, Python bindings

Leonard Richardson just announced a cool new feature for his blogging tool, NewsBruiser: Topic Exchange integration. Niice. It apparently lets you associate local categories with topic channels, and then have then automatically ping your posts over to the ITE when you publish them.

He also separated off his ITE Python bindings for any other Python hackers that want to do something similar. Very convenient.

Now there are two blogging tools that do this: NB and PyDS.
... more like this: [, ] ... topic exchange: [, ]