It’s probably stating the obvious to say that comprehensive disaster recovery solutions are important to any business.
Here at PolicyBee we need a pretty robust back-up procedure for two main reasons: 1) so we can keep trading when all else fails and 2) to keep our clients’ data secure.
We know we’ll have to rely heavily on our ‘plan B’ if things do go wrong. So it’s got to be right.
Which begs the obvious question: what happens when it’s not right?
Well, after two years and £thousands spent trying to get our disaster recovery software to work exactly the way we'd like it to, we've been forced to come up with an answer to that particular question.
As a company responsible for holding client data, not having a reliable contingency plan in place is simply not an option. Heres' what happened when we realised our disaster recovery solution wasn't so much of a solution after all.
Counting the cost
Let’s start at the beginning. We bought disaster recovery software (we’ll call it LookAgain) recommended to us by a longstanding supplier of ours (we’ll call them Suffolk IT) in May 2009. In simple terms, the software was designed to 'mirror' the files on the servers at PolicyBee HQ to a group of servers at our disaster recovery site.
The software itself, the licences to use it and the maintenance package for the first year cost just shy of £20,000. Turns out that was just for starters.
Following installation, and more or less right from the start, we battled to keep the software up and the costs down. For example, every Windows update meant rebooting the servers; which in turn meant expensive engineer’s time.
And when we weren’t turning the servers off and on again, we had problems with the software not ‘mirroring’ all our files – again costing us time and money to resolve.
The beginning of the end
In early 2011, we needed to retrieve some lost files from our recovery site. Unfortunately, they weren’t there. The recovery server had fallen a few months behind and approximately 1,000 files were missing. Not great.
We voiced our concerns about the reliability and escalating costs of LookAgain to Suffolk IT. That prompted them to conduct a thorough check every day for two weeks.
All seemed well so they then lined up the recovery site to carry out a full test of the whole set-up at the beginning of May. The test ran...but 4 out of 5 servers failed to come online successfully.
We figured enough was enough. So we pulled the plug and now have a much better plan B in place that we know we can rely on.
Back up, punk!
You might wonder why we're telling you this.
Well, spending a shedload of cash on software that doesn’t do what it’s supposed to do, when it’s supposed to do it, is a story in itself.
The really important bit, however, is what we’ve learned from all this. It’s clear that from now on we should be:
- backing up and backing up again
- conducting our own tests
- formulating our own procedures
Also, if you do opt for a third-party solution, don’t put all your faith in it - take your own measures too.
One more thing. This story illustrates the importance of professional indemnity insurance for anyone involved in data management or IT disaster recovery services. If the systems you recommend fail to perform, you’re open to a claim against you.
What do you think?
Have you had similar problems?
Let us know. I’m sure we’re not the only ones who’ve been caught out…
Image used under license from Shutterstock.
IT and technologyIT contractorsmanaging risk