Phillip Pearson - web + electronics notes

tech notes and web hackery from a new zealander who was vaguely useful on the web back in 2002 (see: python community server, the blogging ecosystem, the new zealand coffee review, the internet topic exchange).

2004-12-15

Python Community Server database update

Something I've had on my to-do list for around a year (!) now is to get PyCS to use a real database rather than MetaKit, the embedded DB it uses right now. MetaKit is a good little DB, but it's not designed for some of the huge tables that appear on a production PyCS server (e.g. pycs.net ...).

One of the responses to my question a few weeks back about pure-Python database drivers linked to bpgsql, a pure-Python PostgreSQL driver -- exactly what I needed! So one evening a week or two ago, I finally got around to writing the code to connect PyCS and Postgres, and it seemed to work OK.

For PyCS hackers, _most_ but not all of the code is in PyCS CVS (you can get there via the SourceForge project) now. There are still changes to go into pycs_settings.py before your system will actually try to connect to the Postgres server, and once I've checked that in, there'll be a big change that makes the whole hit/referrer logging and reporting system use the Postgres DB rather than the internal MetaKit one.

The results are looking good so far: one referrer listing page that was taking 10 seconds or so to generate with MetaKit is down to one second with Postgres. Once I've got this up and running on pycs.net, this should greatly cut down on the long delays that occur (for all users) when someone hits a referrer page.

I should say that it's not entirely MetaKit's fault: out of the 10 seconds, only 1-2 are spent getting data out of MK; the rest are spent sorting and (perhaps) filtering the data in Python code. The Postgres server, however, is happy to do all this heavy lifting internally, where it's very very quick to do.

... more like this: []