IT disaster recovery plans should focus on power and hardware failures

The disaster in Japan has midmarket CIOs focusing on their own IT disaster recovery/business continuity (DR/BC) plans -- and it's a good thing, since earthquakes, tsunamis and other disasters are responsible for only a tiny fraction of network outages, according to a recent poll by technology solutions provider CDW LLC.

Everyday power outages -- perhaps caused by a thunderstorm or high winds -- were responsible for nearly a third (32%) of the disruptions noted by respondents to the survey, a mix of midmarket and large companies. Hardware failures accounted for almost another third (29%), and a loss of telecom services caused 21% of business technology interruptions.

To guard against such business interruptions, Sisters of Mercy Health System in Chesterfield, Mo., recently flipped the switch on a new hot site that's chock full of redundancies, according to Jeff Bell, chief operating officer. "There are two generators instead of one; two utility feeds from a power generation plant; all the chillers back up to UPSes. If you look at the mechanical cross section, it's a mirror image," Bell said. This enables the IT department to take down only half of the building if there's a problem, and fix it.

And even though the new data center in Washington, Mo., is designed to take withstand a category F2 tornado, Sisters of Mercy is keeping its original data center in Sunset Hills open as a backup to the hot site. "There's always a disaster," Bell said. "The new data center is very unlikely to take a direct hit [from a tornado], but if it did, it might not survive. That's why we keep Sunset Hills up and running."

A grim truth about IT disaster recovery plans

Unlike Mercy, just over a third of the businesses responding to CDW's survey (34%) said they are updating their DR/BC plans, by improving network connectivity (30%); extending DR/BC coverage to accommodate an outage of 72 hours or more; increasing remote access for employees (18%); and improving data center power and backup capabilities (17%). However, one out of every five businesses (20%) is not planning any immediate changes to its DR/BC plans.

Since the Business Continuity Institute has deemed this Business Continuity Awareness Week, it's worth asking whether IT departments are wearing rose-colored glasses.

Judging by the results of CDW's survey, the answer would be yes. The Vernon Hills, Ill.-based firm asked 7,000 IT managers whether their businesses had suffered an outage in the prior year, and about 1,800 said they had. CDW closed the survey when 200 respondents, from enterprise and midmarket firms, had responded fully to its query.

The sobering statistic is that 82% of these 200 respondents said they'd had faith in their IT disaster recovery plans before the disruption, but 30% said at times, they were unable to operate at all due to a network disruption, and their location shut down completely.

A staggering 97% said their business had suffered detrimental effects as a result. Indeed, CDW estimates that such outages cost U.S. businesses $1.7 billion in lost profits last year. More than half of businesses (57%) reported productivity loss as the top negative effect, and more than a third (34%) said customer communications were interrupted. Internal communications also suffered at 22% of the companies, and 14% suffered data loss. Regardless of the cause, more than half of the companies said employees had problems connecting to the network, both inside and outside the building.

Solutions for IT disaster recovery

In a report on disaster recovery last year, Boston-based research firm Aberdeen Group Inc. estimated the average cost per hour of business interruption to be $98,000. In a nutshell, the report said, successful disaster recovery and business continuity involves keeping a backup of critical data off-site, and restoring it as quickly as possible.

Remember the old saw: Don't keep all your eggs in one basket. Aberdeen analysts recommend a mix of backup solutions, at various costs. Tape stores data sequentially and is ideal for data that isn't accessed frequently.

Another solution is to store backups of critical data and backups off-site. Off-site storage services often specialize in secure practices and data restoration, enabling CIOs to set parameters for service levels such as the length of time it takes to retrieve data from a backup.

In addition, a number of best-in-class companies are backing up real-time data to remote locations or servers, or using this type of remote storage for archival purposes. These companies are also implementing all three backup scenarios two to three times faster than laggard companies.

Testing key to IT disaster recovery plans

There are several steps to ensuring that an IT disaster recovery plan will work, according to CDW. First, it's important to conduct a business impact assessment by working with leaders of each functional area to understand the cost and repercussions of losing each category of data.

CIOs should then document configuration diagrams of hardware, software and network components to be used in recovery, and include personnel requirements, travel needs and associated costs. Employees who are critical to continuity operations should be trained to work remotely, even if they don't do so regularly.

Also, consider adding uninterruptible power supplies for critical servers, network connections and select computers, and back up frequently. Look at telecom options such as redundant connections to cover spot outages, and alternatives such as wireless or satellite phones.

Finally, the key to recovery is to test the DR/BC plan at least once every 12 months to make sure people and systems are ready for the unexpected.