Sandy Strikes: Ensuring ContinuitySecurity Personnel Respond to Storm Impact, Outages
As Hurricane Sandy turned west and struck the East Coast on Oct. 29, IT and IT security professionals throughout the northeastern United States were fighting to keep vital IT networks functioning.
"We have been making storm preparations since our emergency response plan was activated on Friday," state of Delaware Chief Security Officer Elayne Starkey said midday Oct. 29. Delaware was one of the first states struck by the storm and experienced significant flooding early in the day. "Employees from the Department of Technology and Information are working 12-hour shifts at the state emergency center," Starkey said. "Everything is on schedule at this point."
Wind and rain from Sandy pummeled a large swatch of the northeastern United States, flooding the darkened streets of Manhattan and causing unprecedented damage to the city's subway system.
CNN reported that as of 5 a.m. ET Oct. 30, Sandy had caused 16 deaths in the U.S. (in all, 84 have perished because of the storm), and an estimated 7.5 million customers in 15 states and Washington D.C. were without electricity. Another 160,000 residents were without power in southeastern Canada. Utilities throughout the region with a population of 60 million people have warned of massive electrical failures. Those failures could cripple IT systems that don't have adequate emergency backup power. In New York, a spokesperson for power company Con Edison told CNN that Sandy has caused the worst damage the utility has ever seen.
As of early morning Oct. 30, Sandy had weakened, but it continued to blow with 65 mph winds, the National Weather Service reported, and the system was centered roughly 90 miles west of Philadelphia. Forecasters called for continued harsh weather throughout the northeast. They said New England and parts of the Midwest will see heavy rains, while parts of Virginia, West Virginia, Maryland, North Carolina and Tennessee can expect snow.
Federal government offices were to remain closed on Oct. 30, and thousands of flights have been cancelled going to and from East Coast airports.
Impact on Financial Services
Sandy brought much of the financial world to a standstill, as expected flooding in downtown Manhattan near Wall Street caused financial markets to close Oct. 29. The stock markets, which rarely shut down because of weather, will remain shuttered on Oct. 30, with hopes of reopening by midweek.
The Office of the Comptroller of the Currency, one of the nation's top banking regulators, on Monday issued a proclamation permitting institutions in Sandy's path to close at their discretion.
Many banking institutions, including Citi, Chase and Bank of America, closed branches up and down the eastern seaboard. Some banks that opened in the morning Oct. 29 closed by early afternoon. But banking institution representatives said they were prepared to deal with the disaster. "Each year, we rehearse and train for business continuity challenges that may be presented by floods, hurricanes, ice storms, tornadoes and other potential natural disasters or other emergency events," Capital One spokeswoman Amanda Landers said.
M&T Bancorp, based in Buffalo, N.Y., identified road closures, public transportation cancellations and weather forecasts to determine which branches and facilities to close. "Over the course of the weekend, we had planning sessions with our lines of business," said Matt Speare, who heads security at the bank. With the storm arriving, M&T continued to review its plans throughout the day Oct. 29. "We have held several situational meetings to adjust our approach as the storm results have materialized," he said.
Expecting the worst, President Obama issued emergency declarations for Connecticut, Delaware, Maryland, Massachusetts, Pennsylvania, New Jersey, New York and Rhode Island. "Because of the nature of this storm, we are certain that this is going to be a slow-moving process through a wide swath of the country, and millions of people are going to be affected," Obama said.
IT and IT security managers in the region said they were ready for Sandy, but outages persisted. In New York, NYU Langone Medical Center had to evacuate 260 patients after backup power failed, and the city's Bellevue Hospital Center reportedly battled flood waters and was relying on emergency power supplies.
In contrast to what was happening in New York, at Beth Israel Deaconess Medical Center in Boston, Chief Information Security Officer Mark Olson said Oct. 29 that the center's emergency response procedure should keep its systems operating.
"On the IT side, we have a very robust set of backup controls and systems," Olson said. "The systems within our data center are on two separate data feeds, which, in turn, go to two separate PDU [power distribution units], and backup power supplies designed to provide power until the generator has started and is operating."
Meanwhile, in Vermont, the state government - taking lessons learned from last year's disastrous flooding from the remnants of Tropical Storm Irene - is well-prepared for anything Sandy can throw at it, said state CISO Kris Rowley. Moving a key data center that was flooded last year to higher ground and employing cloud computing services are among the reasons Vermont should weather Sandy, Rowley said. "We've covered our basis in places where we didn't with Irene."
Many state IT personnel can work from home because many they have stand-alone generators that keep electricity flowing during frequent winter snow stormss, Rowley said.
But having employees work from home during a major storm isn't necessarily a good solution for most organizations, either; it can leave them vulnerable, said Regina Phelps, principal of EMS Solutions [see Hurricane Sandy: Triple-Threat Disaster]. "Working from home works only if you can work from home, meaning you've got utilities, electricity and certainly an Internet connection," Phelps said. Given projections of widespread power outages and storm damage, this strategy may backfire. "The greatest issue people have really overlooked is the idea that they have put way too much hope and possibility into this strategy of working from home."
Business Continuity Plans
One vital preparation step is to test business continuity plans. Four days before the storm was to hit, Delaware state government held a statewide cyber exercise that simulated a broad-scale power outage and critical infrastructure unavailability. "Keeping COOPs (continuity of operation plans) up to date is critical; it's not a one and done process," Starkey said. Among the takeaways from the exercise, she said: Don't assume your most critical IT resources will be available and don't wait for an incident to meet your incident response partners.
Devor Culver, CEO of HealthInfoNet, Maine's statewide health information exchange, which facilitates health data sharing among 25 of Maine's 39 hospitals, also believes his organization has taken the steps needed to keep operating during the storm. HealthInfoNet's data center has two generators ready and available if the overall power from Central Maine Power fails. The data center tests this power fail monthly.
Most of the hospitals that connect to HealthInfoNet have redundant power and connectivity options to deal with the exact type of challenge that Sandy will present, Culver said. But independent clinics generally do not have failover options for power or connection. "These users are vulnerable and face the strong probability that they will not be able to access any remote services for a period of time during Sandy depending on local power availability," Culver said.
Being properly prepared for natural disasters costs lots of money that organizations don't always have. Culver said HealthInfoNet itself must do more in coming years, including the creation of fully redundant and backup systems at other locations. "We're are considering this type of added protection for next year, but it is a very expensive insurance plan," he said.
Beth Israel Deaconess, however, already has a separate disaster recovery data center. "This has the same electrical feed characteristics as the primary center. It is not capable of handling 100 percent of our data center needs, but is capable of supporting all of the key services needed to keep the hospital running," said Olson, the CISO.
The medical center regularly tests its generator and electrical infrastructure to ensure that it is fully operational. "Our objective is to maintain a constant state of readiness for these types of events," Olson said. "When we do know one is coming, we ask that all teams check their specific areas."
The National Credit Union Administration and the Office of the Comptroller of the Currency issued advisories on Oct. 26 and 29 for the institutions they oversee. The Federal Reserve Bank has not issued recommendations specific to Sandy, but the Fed recommends institutions ensure they have adequate cash supplies, since cash is always in higher demand during natural disasters.
The Fed highlights business continuity steps all institutions should take to ensure cash services are not disrupted by network and power outages or building and/or site-specific events. "Financial institutions will be notified of a disruption based on the type of outage, as well as your local Federal Reserve office's availability and notification procedures," the Fed states in its business continuity overview. "For large-scale disruptions, customers may receive notification on FedLine Web, or on this website."
The Federal Deposit Insurance Corp. recommends institutions turn to the Federal Financial Institutions Examination Council's Lessons Learned from Hurricane Katrina best practices handbook to address everything from vendor management to back-up plans for debit and credit transactions should networks go down. The FFIEC's Business Continuity Planning booklet also is recommended.
Learning New Lessons
Past natural disasters have prepared many financial institutions to be ready for Sandy, said Anthony Demangone of the National Association of Federal Credit Unions. Nevertheless, Demangone recommends that banks and credit unions heed whatever new lessons they might learn during this hurricane because there is always for improvement when it comes to business continuity.
"While the current hurricane is a serious matter, it does present a real-world measuring stick for preparedness," Demangone said. "After the storm subsides, it would be a good opportunity for credit unions, and other businesses, to review their plans to see what they learned."
[Tom Field, Marianne Kolbasuk McGee and Tracy Kitten contributed to this story.]