December 2001
Recovery in the Wake of Disaster
by Penny Lunt
On the morning of September 11, Walter Danielsson, senior vice president of technology at Euro
Brokers Inc., was driving to work. A breakfast meeting was planned at 2 World Trade Center, the
south tower where Euro Brokers had its world headquarters on the 84th floor. Although it was "casual
day," Danielsson suddenly decided he looked too casual for the meeting. He went back home to change
clothes and arrived downtown late enough to miss the attack and subsequent collapse. Sixty Euro
Broker colleagues were killed, including eight who reported directly to Danielsson.
While he will never forget that day, Danielsson and 220 fellow surviving employees of Euro
Brokers have rallied, in part by focusing on getting the business back up to speed.
"Despite their losses, the staff has worked through some things that are hard to imagine," says
Danielsson. "The amount of concentration and dedication they have demonstrated is outstanding."
Having survived the 1993 World Trade Center bombing, Euro Brokers had extremely thorough disaster
recovery plans and systems in place. All systems that had been running prior to the attack were soon
restored and operational in a backup location. Yet for all this preparedness, the devastation
reached well beyond data, leading Danielsson and countless others to rethink their idea of "crucial
information."
"In 1993, disaster recovery was, 'get your systems back up,'" Danielsson reflects. "This
situation was different. We were very fortunate to have had our computer systems and data all backed
up, but it's hard to imagine how many things are just gone—copies of documents, signatures, things
like that."
While many companies displaced by the attack have gone back to business as usual, some have fared
better than others, thanks to content management technologies. The sight of millions of paper
documents clogging the streets of lower Manhattan in the aftermath of the attack testified to the
fact that invoices, contracts, letters, statements, memos, checks and other paperwork are still at
the heart of business processes.
 |
|
The September 11 attack has many executives rethinking disaster planning,
preparedness and the nature of "critical data and information."
|
Fulfilling Their Mission, Without Interruption
The Securities and Exchange Commission (SEC) is one World Trade Center tenant that had content
management systems in place. This isn't surprising given that the SEC is the regulatory agency that
sets record-keeping standards for many financial institutions. Although the SEC's Northeast Regional
Office was destroyed, all of the agency's employees survived, and most critical case documents were
stored electronically.
"Within two days of the attack, we had retrieved all documents stored electronically and had
commenced a review of every single investigation and case currently underway in the office,"
testified SEC Chairman Harvey Pitt at a September 26 hearing before the House Committee on Financial
Services. He added that the agency would meet "the twin aims of ensuring that we do not miss any
imminent deadlines and of developing a plan for completing our investigations and cases in a timely
fashion."
At Empire Blue Cross and Blue Shield, which had its headquarters and 1,900 employees in the North
Tower of the World Trade Center, nine employees were killed, including six from the IT department.
While the company's headquarters lay in ruin, business was uninterrupted, and the health insurer's
sense of purpose was unshaken.
"Even in our memorial services, it wasn't taken lightly that we had the IT infrastructure to
immediately appease the concerns of our membership [4.1 million customers]—it was a comfort," says
Ann Mottola, assistant vice president of customer service systems. "One of our first public
statements expressed our concern about the safety of our employees, but also stated that our
customers' records were intact. They didn't need to worry."
The only documents Empire Blue Cross and Blue Shield lost were those in the incoming mail of
September 10th and 11th, as well as some regulatory records now being restored by the associated
government agencies. All other content, including the 40,000 claims and correspondence normally
handled each day at the World Trade Center, had already been sorted, scanned, processed and stored
with Captiva claims processing software and Unisys workflow and imaging systems. Digital document
images were archived and backed up on optical discs at a secure remote location.
"Once the servers were restored, most people found their documents and were able to continue
where they left off," says Mottola.
Immediately after the disaster, staff at Empire Blue Cross and Blue Shield's Albany office called
in vendors to recreate the World Trade Center electronic mailroom. Kodak Document Imaging replaced
13 scanners and Unisys and Captiva provided software. The imaging system was back up and running
within days at a temporary site.
The Empire Blue Cross and Blue Shield Web site was unaffected by the attack, so it was used to
post press releases right away. The IT team quickly configured hundreds of laptops so that employees
could telecommute. The toughest challenge was working with the phone company to quickly establish
dial-up connections. Internal email was up and running within a few days, which helped the
now-dispersed staff collaborate and communicate. Most Empire Blue Cross and Blue Shield employees
formerly based in the World Trade Center are now working out of company offices in Melville,
Yorktown, Middletown, Albany and Harrisburg, NY. The company plans to secure new offices in New York
City for its headquarters.
Getting Back to Business At Bank of New York
The Bank of New York fared better than Empire Blue Cross and Blue Shield in terms of loss of
life, and it was quickly back to business as usual. The bank had four locations in the World Trade
Center area. Its 1 Wall Street headquarters suffered power and telecom outages and was inaccessible
for a week and a half. The bank's major data and operations center at 101 Barclay Street (across the
street from 7 World Trade Center), took damage to its atrium, and the windows on the south-facing
side were shattered. The 4,000 employees who worked there have been displaced and won't be able to
return for several more months. Three employees who were in the vicinity that day were killed.
The bank immediately set up a temporary command post at 1290 Avenue of the Americas. Hierarchical
storage management and software disaster recovery tools have helped the business get back up and
running.
According to Joe Gerbino, vice president of technology, Bank of New York had imaging systems in
place to support four distinct lines of business: stock transfer, trade services, accounts payable,
and benefits disbursement. These systems have been indispensable in the recovery process. "We
potentially would have lost all those documents," says Gerbino. "Instead, those records are in
electronic storage and can be viewed in multiple locations, so this has helped us quite a bit."
In one of those lines of business, the bank acts as a securities agent for major corporations,
handling all recordkeeping and correspondence for its registered shareholders—issuing dividend
checks, handling proxies and sending out mailings. The bank uses ActionPoint capture software and an
Eastman imaging and workflow system to capture, store and retrieve digital copies of stock
certificates, statements and correspondence. These imaging facilities were also knocked out by the
World Trade Center collapse and have been rebuilt.
While no one could anticipate such a despicable attack, the incident has led many executives to
rethink planning and preparedness. For example, at Bank of New York, Gerbino says that that document
imaging will be more of a priority. "Being a bank, we deal with a lot of paper, and we want to get
the paper out of the office. In the future, we might want to get the online documents backed up in
real time rather than batch mode."
Planning for the Future
There's no doubt that in the wake of the terrorist attacks, companies are beefing up their
content and document recovery plans along with data security and backup. "For those who had
previously invested in content management products, and had locked down and secured their content
and repositories, business recovery was infinitely easier than for those that hadn't," says Lee
Roberts, CEO of FileNet, a Costa Mesa, CA-based content management systems vendor that had several
customers in the World Trade Center, including Deutsche Bank and Guy Carpenter, a risk management
and reinsurance firm.
The use of content technology has not been limited to business recovery. FileNet, IBM, Dell,
Kodak and Verizon have provided hardware and software to the New York City Medical Examiner's office
to help collect information about missing people, including the seven-page missing person forms
families filled out.
"This is an incredibly difficult task," Roberts notes. "When you stand in front of the World
Trade Center site and recognize that it used to be 110 stories high, you realize it will take a very
long time to remove all the rubble and identify all the victims."
A Look at NYC's Collaborative Command Center
 |
|
The New York Mayor's Office of Emergency Management had Eteam, a collaborative
emergency management system, up and running within 12 hours.
|
Watching Mayor Giuliani calmly brief reporters in the aftermath of the World Trade Center attack,
you could only wonder how his key officers and crucial city agencies could be so organized. The task
of managing thousands of police officers, firefighters, emergency medical services workers and
construction crews was compounded by the challenge of working out the logistics of bringing in
equipment and emergency supplies. Besides strong leadership, the city is relying on Web-based
collaboration software.
More than 1,700 workers from 200 federal, state and local agencies in New York are using Eteam,
an emergency management system based on Lotus Domino, to communicate with each other, keep
information up to date and make decisions—first for the rescue effort and now for the cleanup and
recovery activities. A hosted version of the Web-based application was in operation within 12 hours
of the New York City Mayor's Office of Emergency Management finding and assembling a downtown
operations center (its headquarters in 7 World Trade Center were destroyed).
The collaborative software was used to create and access infrastructure reports for hospitals,
fire houses, police stations and city agencies. Personnel were assigned to constantly update
incident reports for each building affected by the disaster, keeping agencies and officials apprised
of casualties, rescue priorities and emergency resources available. The system was also used to
submit and process requests for blankets, boots, dump trucks, cranes and other resources needed for
the emergency. Workflow features helped track the costs associated with each request.
Representatives from Con Edison, the local power utility, and Verizon, the local telecommunications
provider, used Eteam to track outages.
At ground zero, rescue workers accessed the Eteam system from special ruggedized laptop computers
rigged with wireless network cards. Updates could be transmitted remotely to the command center and
to a staging area and supply warehouse. To keep bandwidth demands to a minimum, the system transmits
only the changes made rather than the full content of a file.
In the days after the terrorist attack, city managers were able to see which agencies were up and
running, which were out of service and which needed help by simply glancing at the reports available
through this collaborative application. This came in handy for Mayor Giuliani's twice-daily
briefings. Instead of having to read individual one-page reports from 68 different agencies, the OEM
director could review their status in one organized report gathered by the system.
According to John Hughes, a former deputy directory of the New York City Mayor's Office of
Emergency Management and now a vice president at Eteam, "The agencies were able to all be on the
same page and utilize the same reports, like a big whiteboard in the sky."
Safeguard Your Company with a Business Continuity Strategy
In the aftermath of the September 11 terrorist attacks in New York and Washington D.C.,
businesses are turning to records retention and data recovery companies like never before. One of
the largest of these firms is Recall, an Atlanta-based company that offers document archiving as
well as data recovery services.
Among Recall's customers is the bond trading firm Cantor Fitzgerald, one of the World Trade
Center-based companies most devastated by the attack. Despite its losses, the firm was able to get
back up and running within two days.
Recall offers this prescriptive approach to disaster preparedness:
1. Determine your data loss tolerance. Back up data as often as required to reach the level of
data protection necessary to ensure business continuity—a minimum of once per day.
2. Determine all types of information that should be backed up. Consider imaging and indexing
active hard copy original documents to an online, hosted data service to ensure that active files
and data are available.
3. Maintain your backup tapes off-site in a highly secured facility. Choose an operation with
facilities secured yet available 24 hours a day, seven days a week. Facilities should ensure
integrity of all data resources with humidity, fire and climate controls. Systems should ensure
rapid and accurate retrieval of the specific series of tapes requested, and prompt delivery of tapes
to your "hot site" or alternative office.
4. Write a disaster recovery plan. The written plan must include protocols that explicitly:
- State who is authorized to declare a disaster,
- Detail the predetermined hot site or alternative site,
- Identify who is authorized to order tapes and accept delivery.
5. Make the disaster recovery plan accessible online. Publish your plan on a password-protected
Web site so that authorized individuals in the chain of command can access the contingency plan to:
- Efficiently activate an organization's disaster recovery plan, including the first call to
service providers
- Determine the specific media and documentation to be delivered to the preauthorized hot
site
- Maintain contact with the disaster coordinators to speed the delivery of vital media to other
locations.
6. Test your plan. Conduct disaster recovery simulations to ensure end-to-end operational
continuity consistently meeting expectations.
|
