When you run a business that is completely reliant on technology, you’d like to think you’ve got the whole thing nailed down.
Stuff like hot stand-by boxes, redundant disks, uninterruptable power supplies (UPS), cloud-based infrastructure – it’s second nature, right? For those “just in case” moments.
So there I was on Monday, looking forward to the week with a spring in my step and excitement rising over instructions, exchanges, and completions when my colleague took a call from our London office. “There’s been a power cut and now we don’t have any internet.”
“There is no panic at The Partnership”
Ever since I started the business, this has been the company mantra. Despite what dramas may unfold with clients, cases, technology, lenders, or lawyers, we keep calm and sort it out. But with 40 people sitting around at 9.30 am on a Monday morning waiting to send out their weekly updates to agents and clients, the pressure was on.
The operations team went straight into diagnostic mode – they knew what to check and how to check it. This was when we realised there was a problem. Everything looked fine. I mean, absolutely fine. Our last remaining server for user management was running, the router was blinking furiously, the switches were shining their bright green “I’m OK” lights and the firewall was up and running.
But there was no internet.
At this point, we resorted to plan B and called our IT support company. I knew things weren’t going well when two hours had passed and they had started resorting to “well, the UPS was beeping so we think that might be it”. A desperate diagnosis if ever there was one.
The legal team were getting understandably twitchy. It was time for a rethink.
Google IS your friend
As a lawyer, you have come to learn that Google is not your friend. When clients call to advise you that the flying freehold is really not a problem, it’s probably a Google search behind it, much like the Wizard of Oz is behind that green curtain, pulling the strings and turning all those wheels.
People aren’t aware that when it comes to solving technology issues, sometimes, educated hunches really do pay off. We’d decided that the problem was caused by a failure of the firewall, but as no one could connect to it, they couldn’t diagnose it. We guessed that it had lost its settings and decided for no reason, in particular, to carry out a factory reset. Google revealed the default username and password for this firewall and voila – we were connected.
Then it was just a simple task of restoring the configuration from backup and we were up and running again in under a few minutes – to rapturous applause from the legal team.
Planning to fail
Naturally, after such an event, post-mortems are critical and we learned a few things. Just because our firewalls had never failed in 12 years, and we had kept them patched and updated, this didn’t mean they were infallible. They are. Power failures cause strange behaviours – it should have never reset the firewall to factory settings. But it did.
Make sure you plug everything into the UPS that needs it. Our switches, servers, and routers were plugged in, but we’d missed the firewall. An expensive oversight.
Finally, and most importantly, I had asked our IT company to buy and install a hot standby several months earlier but didn’t check if they had done this. They hadn’t.
We’ve put everything right; rewired the UPSs and bought another couple of firewalls, but it’s a salutary lesson. Just because you think you’ve got everything covered, you must double-check. Because you probably haven’t.
As published in https://todaysconveyancer.co.uk/confessions-cyber-conveyancer