Calling for backup: even lawyers might need it

Berglund Authority Level 3

by Pat McGregor <pat@hayseed.net>

Most of us love lawyer jokes. The attorney who drafted my last (that is, most recent) will has a framed poster of the T-Rex eating the lawyer from Jurassic Park with the caption, “The first thing we do, let's kill all the lawyers.”

This note that came across a recent blog (public web journal) isn’t funny, though. It’s here more or less in its entirety because it’s so typical of how life strikes when you least expect it.

What started out as a relatively normal morning at work turned into a corporate version of The Poseidon Adventure. One of the attorneys came rushing through the office about 10:30, announcing that there was water spreading through the building. Naturally, we all stood around and made smart-ass comments, which is what one is supposed to do when an emergency arises. See, emergency preparedness training really works! However, once we'd gotten that out of our systems, we all had to go take a look, and sure enough, there was a substantial amount of water working its way across the floor. We started unplugging computers and moving stuff up off the floor, and a couple of people ran around trying to figure out what they were supposed to be doing. Within a couple of minutes, there was an inch or two of water on the floor and that funky water smell in the air.

My boss and I decided to see if we could find where the water was coming from, and, oh, I don't know, maybe shut it off? We waded through the office to the back hallway door where the water was coming from. I opened up the door to see about six inches of standing water in that hallway, and down at the end of the hall, a jet of water shooting out between the doors of the Intellectual Property department's file room. "Hmm," thought I, " that doesn't bode well.”

With water rapidly coming up to my knees, I splashed down to the end of the hallway to see that, not only was there an awful lot of water coming through the crack, the doors themselves -- which open inward -- were noticeably bowed outward. I looked back at my boss and informed her that under no circumstances was I going to be the one to open that door.

It probably took us another hour to clear everyone out of the department and figure out what was going on. A fire standpipe had burst, and was pumping prodigious amounts of water through almost a hundred years of patent, trademark, and other intellectual property files. Needless to say, we've got a bunch of very sad attorneys this morning.

When I read this, I had two immediate thoughts: what a mess if there is a patent infringement case; and I sure hope they back up their machines regularly.

Because I was drafting this article, I took a look at my most recent backup record. Ow. The last complete backup was just before Christmas. We have copies of a random assortment of stuff (like my son’s homework) on CDs, but in no organized fashion.

Well, I took care of that first, and then did some research.

Most of us are still not prepared

A 2004 study by the British Department of Trade and Industry of a PricewaterhouseCoopers consortium of approximately 1000 companies (all sizes) found that:

  • 87% reported significant dependence on electronic data (up from 76% in 2002);
  • An overwhelming 88% found it easy or very easy to justify the cost of backup and disaster recovery facilities;
  • Around two-thirds of large businesses suffered an incident in the last year where they had to restore significant data from backup (e.g. systems failure or physical theft);
  • Roughly half the businesses that had a systems failure or physical theft suffered major disruption to their business operations, some for up to a month;
  • As a result, 95% of companies had some form of backup facilities in place;
  • However, these vary considerably, and may represent a false sense of security; for example, only a third of businesses store their backups off-site, and less than 20% backup their desktops;
  • Only 8% of companies have tested their disaster recovery plans to see if they would work in practice.

(“British businesses labouring under false sense of security over data backup, survey suggests,” Original source <http://www.eubfn.com/compliance/arts/press_pwc_backup.htm> no longer available )

The smaller, more recent surveys I found show only marginal improvement, and the rapid replacement of conventional desktops with laptops makes the situation worse, as it appears that most laptops are not backed up at all. Some companies have developed management requirements for desktops that are always connected to the network, but mobile data is, by definition, not. And the common accepted rule of thumb is that 60% or more of intellectual property and critical business information resides on desktops and laptops, not in the data center.

Many datacenters do not back up data in databases if that database is being accessed during the back-up period; for global enterprises this can mean that the databases are, realistically, never backed up. In companies that allow telecommuting, staff can be working at all hours, leading to the same problem even for small firms.

Think small

Think smaller than Katrina or even someone hacking your online store. When we plan for “Disasters,” we tend to think about the big things first. Hurricanes, tornadoes, earthquakes, or even things like fires in the data center or burglaries. We don’t think about an elderly driver shifting into “drive” rather than “reverse” by accident and driving across the sidewalk through the kitchen of our family owned deli. Will our business survive if we can’t serve meals for the time it will take to repair and refit?

We need to think in terms of “Business Continuity” –keeping the business running. What happens to the state-of-the-art electronic telephone switch that runs our entire office — including our dispatchers, order-takers, and voicemail — when an older substation goes down because a raccoon accidentally immolates itself? It can take us out for the entire day. If we can’t answer the phones, we don’t even know what business we lose, or critical information we miss.

Good planning consists of walking through your processes, talking to the people who actually do the work, and have alternatives ready to bridge the gap if one or more mechanism fails. In the case of the phone switch, you could have a generator standing by, or alternately an arrangement with the phone company to forward all your calls to cell phones kept in readiness for just such a need.

Some creative planning could make even the car in the kitchen a major problem, but not a business-ending disaster. Just as some large corporations have reciprocal arrangements with other companies for emergency datacenter usage, perhaps you could find a partner in the local community who let would you at least store your perishables and frozen goods with them so you don’t lose your entire inventory. Or consider renting the kitchen at a banquet facility and ferrying at least some food to your place to feed your customers.

Happy Endings

Luckily, our flooded lawyers had a contingency plan in place. My friend wrote a day or so later, when I asked about how they were doing:

We actually have a BC/DR planner! I'm amazed at how fast they've gotten the recovery machines up and running, and it's having remarkably little impact on us getting our work done. Much of the recent material was already digitized or copied and stored in secure off-site vaults, although not all. And the document recovery service we have a contract with will take all the soggy paper, even the oldest, and restore it to nearly pristine, and, if we choose, will also scan and archive it.

It was really a pleasure to see that this firm has thought ahead about not only where to put the staff if their offices were damaged, but also how to handle the materials that the archival process hadn’t gotten to yet. A pre-established contract with a recovery service will provide the fastest start on the project, which gives the best chance for success.

It also points up the fact that we, as consultants, strategists, and planners need to not make assumptions about the kinds of companies and work that we can do. All our work aims to keep the standards of assurance, integrity, and confidentiality. Planning to keep the business running as well as being prepared for interruptions is part of that triad of standards. Globally,  we can now see clearly that all businesses, large and small, need this kind of planning.