The advent of hurricane season prompted the Southwest Florida Regional Technology Partnership (RTP) to assemble an expert panel of business leaders to share best practices about disaster preparation and business recovery.
The panel was moderated by Linda Lyding, vice president of the RTP which was attended by more than 60 people at Endeavor Innovative Workspaces in Fort Myers.
The panelists were Beth Tracy, Manager, Corporate Citizenship, at IBM; Jacob Ackerman, Chief Technology Officer at SkyLink Data Centers; Wendi Fowler, President at The Client Server, Inc.; Bill Lean, Solutions Architect at Summit Broadband; Nancy O'Hara, Director of Business Systems at Shaw Development and SWFRTP Board Member; and Andrew Cooper, Executive Director Information Technology NCH Healthcare Systems.
Summit Broadband has more than 500 miles of fiber optic cable running in southwest Florida that needs protection from the evil squirrel and from evil yellow backhoe,” said Bill Lean. As a carrier, Summit requires rapid recovery when there is an outage.
In terms of preparing for a malicious style disaster event, Lean added that companies can test their staff with fake phishing emails. Exercises like this will create a level of awareness and preparedness amongst your team.
Budgeting for Disaster Prep and Business Recovery
Jacob Ackerman from SkyLink Data Centers shared that 60 percent of hacked small businesses are out of business after six months of the data breach. This statistic justifies the investment required for effective disaster and business recovery.
A proper budget and disaster plan can be determined by determining your company’s revenue per minute or hour compared to cost of your employees twiddling their thumbs when there is no power, no internet access or the local area network is inaccessible due to a hack. Managers within larger companies can use this math to work with the chief financial officer to justify the budget expense.
Ackerman added that most companies are between $900-$17,000 revenue per minute being lost when their business is without power. From there, determine how much downtime is acceptable. The budget writes itself from there. Typically, a range of 30 minutes to 1 hour of downtime is when it becomes economical for most businesses to justify investing in a disaster plan for power or internet backup.
Wendi Fowler suggested that companies remember to budget for the cost of public relations in their disaster plan too. Businesses that are heavily relied upon by consumers might end up in the news if they are unprepared for a disaster and the cost of dealing with the negative public relations can skyrocket quickly.
Budgeting must include anticipating what equipment might be needed from a disaster. For example, at Summit Broadband the field workers are issued chainsaws for their trucks so in the event of a downed tree they are not reliant upon a third party vendor such as the power company to arrive to make the cuts.
Companies that will depend upon internet access either during normal operations or in a disaster situation when employees may be working remotely need to make sure their infrastructure is sound according to Ackerman. “People skimp on internet expense and do not have redundancy,” he said. “Have multiple ISPs.”
Most companies have multiple systems they rely upon such as accounting, payroll, project management and more. One tip shared by Andrew Cooper to help with budgeting for disaster recovery is that not one size fits all. Some systems are more critical than others so go through a prioritization process to control the costs.
Assessing Outcomes of Prior Disasters
Hurricane Irma forced most companies in Florida to make decisions about how to prepare and how to overcome operational limitations after the storm caused by physical damage, flooding, employees that were unable to return to work, extended outages of power, telephone and internet service. Nancy O’Hara from Shaw Development reported that because the team at Shaw was adequately prepared they did not have panic. Arrangements were made to get their shipments out to customers ahead of the storm before they closed the business to weather the storm.
Andrew Cooper from NCH Healthcare Systems shared that his team discovered single point of failures in their hospital after a risk assessment. This analysis prompted the hospital to adjust their technology infrastructure and operations to create more redundancy and failsafes.
A highly decentralized large organization such as IBM faces several challenges when dealing with natural disasters. Beth Tracy shared that IBMers are encouraged to share their mobile phone numbers so human resources can communicate by text to ensure remote staff are safe. A crisis management team ensures that IBM employees are safe. Regardless of disaster type, regardless of company size, Tracy urged the audience to remember that safety is paramount and to make employees and their families the first priority.
In the aftermath of a natural disaster keep non-essential staff off the roads so first responders can do their jobs. “Do not have people come in if they are not critical,” Ackerman suggested. “Worry about ahead of time who will take on backup roles if someone can’t come in.”
A decision to not allocate budget for disaster preparedness combined with a failure to plan can put your company at unacceptable risk. Wendi Fowler shared a story about a client who made a decision not to invest in preparing for a disaster, despite her recommendation. Thanks to Hurricane Irma, the client’s facility was flooded with four feet of water. A boat had to be provisioned to retrieve the computer servers and relocate them to Skylink Data Centers. The process was completed within 48 hours for the servers to be relocated to the data center and get the company back up and running. In hindsight, this downtime was an unacceptable outcome for Wendi’s client but it took a real disaster to convince them that the economic impact of having a backup plan outweighs the risk of being out of business for an indefinite period of time.
In response to concerns from audience members about the safety of incorporating cloud based solutions into a technology operations plan Ackerman suggested not to fear the cloud. “Use a qualified technology person to set it up correctly. Almost any cloud system is safe if you use two factor authentication or you use your own encryption key. Breaches are typically not cloud related,” he advised.
Bill Lead added “Cloud enables business continuity at the micro level.”
Having offline solutions is another important lesson learned from Hurricane Irma. Wendi Fowler experienced senior housing facilities whose commercial kitchen menus and meal preparation systems were all automated so kitchens were not functional without power. Also,automated wristbands that patients wear did not work so vital health data was not being transmitted or monitored. Have a backup plan to operate and serve in an offline environment.
Disaster Prep and Business Recovery Comes In All Shapes and Sizes
No business is too small to have a plan documented on how to handle a disaster. Business leaders have a responsibility to their stakeholders - employees, investors, customers, vendors, strategic partners - to think through what types of disasters might happen and document a plan on how to operate through them. These plans do not need to be overly complicated. Smart small and build them over time. But a failure to plan is guaranteed to be a plan that fails.