About a third of cloud users need to learn resiliency lessons from Ian

Beyond the human cost, natural disasters like hurricane Ian can take a high toll on business continuity, causing enterprise-infrastructure damage that takes days or weeks to fix while downtime costs in the six figures per hour. If Ian didn’t impact your operations, now is the time to prepare for a future disaster that might hit your network.

Vulnerable areas include cloud providers’ managed services that might require customers to explicitly specify they want their apps, compute, and storage housed in redundant, geographically separate availability zones. According to Uptime Institute, roughly one third of enterprises are architecting cloud apps that are vulnerable to outages in single cloud availability zones, rather than distributing their workloads across multiple zones.

“Public cloud providers call cloud services fully managed, and that creates a perception challenge, that as a customer, I don’t need to build resilience in,” Forrester Principal Analyst Naveen Chhabra via email. So IT pros need to be sure they’ve got the resiliency they need. “Different services have different resilience either built in natively or to be built by clients,” he said.

Calculate the true cost of downtime

It costs a premium, but this redundancy can reduce the chances of downtime significantly, Uptime says. Justifying the cost may be difficult and require looking at the potential consequences differently than they have been in the past.

“My experience is that [disaster-recovery] programs are heavily under-invested from all perspectives—people, process, technology, and governance,” Chhabra said. “Leaders ask for ROI justifications before allocating budgets for DR programs. While ROI is reasonable for many initiatives, unfortunately [it’s] not for DR. Instead, what they need to look for is true cost of downtime.”

That price can be high. According to a recent IDC report, downtime can cost an average of $250,000 per hour. According to Doug Matthews, senior vice president and general manager, Data Protection, Veritas Technologies, 98% of organizations say a single hour of downtime can cost them over $100,000.

Preparations for natural disasters go well beyond the cloud, and focusing on infrastructure vs applications first may be the more effective choice because a single infrastructure outage can take down multiple app services, Chhabra said. “Developing app resilience requires a lot of investment that many organizations won’t find support and investments for,” he said.

That infrastructure means having multiple communications options, says Curtis Preston, chief technical evangelist at Druva, “that do not rely on typical infrastructure.” These can include a mix of broadband, MPLS, fixed wireless, and satellite.

Backing up data is also important, and should be done in accordance with the 3-2-1-1 strategy, according to Matthews. That calls for keeping at least three copies of your data in different locations; using at least two distinct storage mediums; storing at least one copy offsite; and keeping at least one copy on immutable storage.

Storm prep by providers

Other lessons can be learned from service Florida service providers that have survived previous hurricanes as well as Ian.

Verizon has battery and generator backups to provide power if the electrical grid fails. Verizon says its macro cell sites have backup battery power and 78% have backup generators, and the provider topped off fuel for the generators in the runup to the storm. (While they’re outside what enterprises can afford, Verizon says it has more than 500 “portable assets” including “generator-powered cells sites, drones, a fixed-wing aircraft that can provide service from the sky above and mid-Earth orbit and geosynchronous assets.)

Colocation provider Flexential got ahead of the hurricane by augmenting staff and designating a mobile response go-team that is trained to keep its data centers operational under extreme conditions, according to David Kidd, senior vice president of risk, compliance and governance at Flexential. The company also confirmed supply readiness at its Florida locations, which include data centers in Fort Lauderdale, Jacksonville, and Tampa.

Flexential keeps more than 48 hours’ worth of fuel at all its facilities, and that’s backed up with emergency fuel contracts that guarantee fuel delivery in emergencies. (The provider faces a double hit from Ian; it has facilities in North Carolina where the hurricane is headed.)

Backup and recovery

Some backup and recovery lessons Preston gleaned from a hurricane that took out a bio-tech company’s facilities on an island:

• Have a place prepared to recover the network. Physical facilities need to be strengthened against storm damage.
• Keep separate local copies of software needed to start recovery of data.
• Rehearse large-scale recoveries so staff has practiced what needs to be done in a real emergency.
• Have a way to backup servers during the recovery, and use your backup software’s ability to automatically backup everything.
• Prepare a space and supplies for the recovery team to eat and sleep, again in a strengthened building.

(Contributor Neal Weinberg and Senior Editor Michael Cooney contributed to this article.)