Disaster Recovery - Avoiding Catastrophic Losses

Let's talk about Disaster Recovery (DR) for a moment. I was talking with a customer about how he's managing his business in these tight times. He said that down-time cost the business about $100,000 US per hour, and that the level of impact in a tight economy felt like it was even more than that. Do you have a cost figure for down time? Is there a defined point where you call in your insurance company because down-time has become a catastrophic loss? What an awful time. Whether it's a bad disk controller, a flood, or lightning hitting the CPU (yes - that happened) a good plan is a great thing.

Why blog about this? Well, an outage gets to be longer if there is no standby system, or if key users cannot reconnect to it quickly. We have seen outages get longer when simple human error overwrites a disk or misplaces a backup. A good set of written procedures and some well-designed recovery-scenario automation scripts sure help to keep a bad thing from going to a catastrophe.

Here's a question for you - if your Disaster Recovery plan aims to reduce the impact of outages, is there also a section that looks at staying reasonably up to date on your key software releases as a way to avoid the business impact of hitting a known fixed problem? An expensive outage that was avoidable is the unkindest disaster of all.

OpenEdge databases have a very solid reputation with our customers, but "things do happen". Why not schedule a walkthrough of  your Disaster Recovery plan and a check on the release notes for new service packs to see if there are opportunities to prevent or minimize any potential problems?

Comments

[Phillip Molly Malone]

Some great points Tom. As a Progress TSE (TechSupport Engineer) I would like to add that unless you check your backups, you are still putting yourself at risk. I had no better example of this then having two customers in two days losing there databases because they had hardware outages and didn't have good backups. They backups, but both had issues and they hadn't checked the backups in a long time.

So I agree test your Disaster Plan but also test those backups on a regular bases!

All the best for the Holidays.
Molly

[Jorge Olguin]

Can't agree more... Last night about 11:30 pm, I was just playing to get backup procedure to run somewhat faster, and by looking carefully at the procedure, I noticed that when I switched server 1 YEAR AGO!!!! I rendered useless the whole process!!! my gosh.... ONE YEAR!!!!! m...f... I remember going mall, going church, with family, confident, and the blessed backup wasn't getting done!!! I think that finding this post just 9 hours later than my recent discover is a sign. So this post is just about something: CHECK THE DAMN THING!!! as part of recovery plan, you just need somekind of backup to resort to... so make sure of that. My god... techically I was out of job all this year and neither me or my employer knew about that!