IT

Mitigating IT Disasters

People who paddle have an old saying: There are two types of people. Those who have capsized and those who will capsize.

IT is the same way. It’s not if you’ll lose data, it’s when you’ll lose data. In bigger projects I have access to large scale solutions, like mirrored drives, multiple mirrored sites, and backup media (ex. tapes). In the past, all that stuff has been costly and a pain to setup. But, these days, there’s no excuse for everyone to have their stuff backed up.

First, some things you should never, ever, trust:

  • The “cloud”.
  • Any hard drive.
  • Any computer.
  • Any array (ex. RAID).
  • Any piece of software.

Hard drive and computer failure is commonplace. We’ve all had them go bad. Big disk arrays, supposed to keep you safe, have a nasty habit of getting corrupted in the middle of a rebuild. The cloud is fine and all, but what if your provider goes belly up? How many IT companies do you know that have offered the same service for years on end? Not many. And what happens where your data aren’t deleted, but get corrupted? Either by a hardware/software failure, or a just-plain “oops!”. You need to undo. Quick.

I learned a very valuable lesson over Y2K: your two most important systems are Billing and Source, in that order.

How does an individual protect themselves? And, more importantly for me, how does a independent IT consultant work protect their livelihood, without spending a boatload of money? Remember, the key is Billing and Source. If I can’t bill, or lose my bills, I can’t eat. If I lose source, I can’t eat. Important stuff.

Source Code Control

First off, without question, have a machine setup to be your source code repository. I happen to run VisualSVN on a Windows Server I have. Then I run TortoiseSVN on my development machines — Tortoise allows me to check in/out files to the repository. Anything important gets put under source code control. This includes actual source, database schema, invoices, documentation, test files — you name it. Everything I need for a client’s project.

Create a new invoice? Check into SVN. Update an Access database? Check into SVN. Create a new build for the client? Check into SVN. Disk space is cheap. Use it.

External Hard Drives

On my local dev machine, I have a big external hard drive. That’s where I keep all my client data. I put NOTHING but the OS and Program Files on my computer’s internal hard drive. All data, databases, source, etc are on that external hard drive. If you run eSATA it’s just as fast as an internal drive. Plus, when your computer goes poof! you don’t have to pull drives or try to recover an OS drive that happens to have data. I unplug the external drive and plug it into another computer. Though that’s not even that important, since everything is is SVN, so I can pull down the files from there.

Virtualize

I run a lot of stuff on VirtualBox, the free VM Host. I do this mainly because clients are on different builds (like a bunch of different versions of Access). Plus, if a machine goes poof! who cares? I boot up the VM on another host. Not to mention the ability to snapshot the VM before doing something major — if the upgrade fails, just revert to the snapshot.

CrashPlan

I use CrashPlan, and pay the annual fee for cloud backups, because you never know when a disaster (like a fire) will strike. But I don’t trust them to stay in business forever, so I have a multi-destination backup plan in place. Here’s how it works:

My primary Development machine backs up locally, on an hourly basis, to two places: The CrashPlan Cloud and to my Local Server. If I have a fire, I can get the backup from the Cloud. Otherwise, I can go right to my local server. My Local Server also backs up hourly, to an external hard drive attached to it, and to the Cloud.

The great thing about CrashPlan is you set it and forget it. No fiddling. No goofing around. Just configure and let it do its thing.

Backups of Backups of Backups

By now you can clearly see how crazy I am about backing things up. Let’s take an example of an Access application I maintain for a client.

  • The primary file is sitting on an external hard drive attached to my development machine.
  • The file on that external hard drive gets backed up automatically to the Server and Cloud hourly, regardless of whether I use SVN.
  • I use SVN religiously, and often, so I’m committing a lot (very important for Access, since you have to commit the entire DB).
  • The server backs up  the SVN repository automatically, to its external hard drive and to the Cloud.

So, for that Access app to get completely destroyed, the following would have to happen:

  • External dev HD fails.
  • External server HD fails.
  • Internal server HD fails.

Which could certainly happen in the case of a physical disaster, in which case I’d pull the backups of SVN and the original files from the CrashPlan Cloud.

And why backup both SVN and the primary files? Because never trust anything completely. SVN could go nuts, and then where would I be?