For UL, and almost all organizations, having solid planning in place ensures that you’re prepared to deal with emergencies in a controlled manner so that operations remain consistent and customer data is protected. Few companies experience testing of their business continuity capabilities by an event such as the one we encountered when the building housing our corporate offices flooded due to a burst water main that feeds our fire suppression system.
The force partially blew out the adjacent wall and released 2,500 gallons per minute for approximately 20 minutes before emergency workers were able to shut it off. This resulted in the unavailability of our corporate and Disaster Recovery (DR) data center along with essential components such as phone systems, generators, air conditioning and data center equipment.
Fortunately, our primary production (customer-facing) systems are completely housed in secure data centers at alternate sites, so we did not experience any production issues. From the time our employees reported the issue to safety personnel, our teams implemented our business continuity plan.
Although the flooding was confined to the first floor and our offices are on the fourth, shared phone equipment, internet connectivity, power and generator wiring hosted on the first floor were impacted. Damage resulting from equipment being underwater as well as sprayed water and electric surge events was evident. Due to safety issue posed by this compromised equipment, fire department personnel determined it was necessary to cut power to our building.
After power was shut down, our emergency generator immediately began powering our phones, internal systems and disaster recover data center. However, fire officials soon made the decision to shut down all generators around to protect emergency workers, and shortly thereafter our backup batteries exhausted their capacity resulting in our DR data center and internal systems experiencing a hard shutdown.
Thanks to our Sr. Director of IT working with officials during the crisis, our local fire department personnel allowed us access to our data center to unplug equipment. This enabled us to bring our systems back up in an orderly fashion after power and internet connectivity were restored.
We have been extremely pleased with the outcomes and our abilities to keep our critical systems functioning properly in the wake of such a disaster. Our team worked tirelessly throughout the weekend to ensure that all functions were operating normally by Monday morning and we are happy to report that we have been in a “business as usual” mode of operation.
UL’s software and information solutions empower your entire organization to improve workforce health, safety and overall business performance.