Phillip Pearson - web + electronics notes

Nothing here that hasn't been thought many times before by smarter people with far more experience, but here goes... :)

Is it possible to set up a highly available website/service on two servers, without an external load balancer or a STONITH device?

Perhaps a better way to put it would be: what failure modes can be avoided for a website/service on two servers on one switch, with a single uplink (from the switch)?

     Internet
        |
   +----+-----+
   |          |
Server 1   Server 2

The most common failure mode I've seen is a single server hitting 100% CPU or running out of memory. In short, it's up, but thrashing too badly to do anything. We also had a disk fail in a RAID-5 array once, causing a kernel panic. Also in our previous datacenter, our uplink cable fell out (?!). Then the power can fail, perhaps only for a short while.

So there's a few categories:

• things that nearly-but-don't-quite halt a server (and might get better by themselves).
• things that halt a server immediately but are (or might be) fixed by a power cycle.
• things that halt a server immediately and leave it dead, even after a power cycle.
• things that leave a server running but cut off from the outside world.

In this situation all of the above could happen, and I'd like to think that you could deal with everything except switch/uplink failure without needing extra hardware.

When you've got more than one server, there are two ways to organise things. Either both servers share the load and one takes it all over if the other fails (active/active), or one server handles all the load, and the other only becomes active if the first one fails (active/passive).

I'll try to sketch out what an active/active system involving Apache and MySQL looks like on two servers, without a hardware load balancer.

     Internet
        |
   +----+-----+
   |          |
Server 1   Server 2
   |          |
load bal   load bal
   |          |
   +----------+
   |          |
 Apache     Apache
   |          |
   +----------+
   |          |
 MySQL ---- MySQL

HTTP traffic is split (probably unevenly) to the two servers with round robin DNS. The load balancers distribute traffic evenly to the two instances of Apache. The Apache instances distribute database traffic evenly to the two instances of MySQL, which are configured with master-master replication.

When something (this is the fuzzy bit) determines that one of the servers has fallen over, the other takes over its IP address and tells its load balancer to only direct traffic to the local Apache, and Apache to only use the local MySQL.

An active/passive system looks similar, but there's no load balancer; the active server owns a 'cluster IP address' and handles all the traffic.

     Internet
        |
   +----+-----+
   |          |
Server 1   Server 2
   |          |
 Apache     Apache
   |          |
 MySQL ---- MySQL

If something goes wrong, server 2 grabs the IP address and continues as before, but without any reconfiguration.

So, how do you do this with existing software? I'm focussing on the MySQL server at the moment, as it's the most stateful -- you can lose your PHP sessions in /tmp and nobody will care (much), but your data needs to be up to date on the backup server. The most promising software I've seen is Alexey Kovyrin's MySQL Master-Master Replication Manager (see also this blog post from Peter Zaitsev, which has an interesting discussion with Kevin Burton in the comments); I'm giving it a go on a bunch of Ubuntu virtual machines (under VMWare) at the moment. I think it's meant to work with three servers: two databases and one monitor, but want to see if it's feasible (or sensible) to run on just two.

It looks like the important parts are a monitor script (mmmd_mon), a remote agent (mmmd_agent), and a control script (mmm_control). mmmd_mon runs on the monitor host, mmmd_agent runs on every database host, and mmm_control is what you use to talk to mmmd_mon, I think.

Here's what you need to do to get it installed on Debian or Ubuntu:

apt-get install iproute fping fake libdbi-perl libdbd-mysql-perl libalgorithm-diff-perl libproc-daemon-perl subversion
svn co http://mysql-master-master.googlecode.com/svn/trunk/ mmm
cd mmm
sudo ./install.pl

install.pl puts everything in /usr/local/mmm, with symlinks into /usr/local/sbin and /usr/local/man. Init scripts for mmmd_agent and mmmd_mon are available in /usr/local/mmm/scripts/mmm_agent and mmm_mon.

The situation here is the first of the typical use cases described in the documentation (with accompanying configuration examples).

I'm starting with three virtual machines: 192.168.2.68 (db1), .69 (db2) and .70 (mon).

This is the key blog post to read - it explains what happens when you've got your databases all set up and mmmd_mon running on your monitor host.

TO BE CONTINUED (one day ...)