High availability, the social side
Everyone loves to blog about how to achieve high availability with various schemes of redundant servers. However, there are some other things to consider. How do you assure high availability, or more importantly, that you won't lose data, when your company runs out of money (hopefully temporarily)? How about if someone hacks into your system and gets a copy of your private SSH key, or your AWS credentials? What if you have disgruntled employees?
The best solution I can think of is to have a backup box on a totally different provider, ideally located in the boss's home or someplace relatively secure, that nobody can directly connect to, but which is able to take a complete snapshot of everything. Either it'll have a very privileged SSH key, or it'll have access to pull from a "dropbox" that all your servers back themselves up to.
One backup system I've set up uses the "dropbox" approach. The backup server runs the rsync daemon, with a separate user (plus randomly generated password) for each server that needs to be backed up. Each server user is restricted to its own private area in the backup filesystem, and each server runs rsync at a random time in the day to back itself up.
Then, you can create an rsync user with read-only access to the backup data, and rsync it all down to some other box. If you want to protect against angry sysadmins, several (presumably technical) people in the company should run their own private backups.
Keeping all this data private is left as an exercise for the reader :-)