Nlyte, a data center service management company, has tips for preparing for inevitable data center power outages, including running power outage simulations.
The Delta power outage on Aug. 8 worried many chief information officers about the stability of their own data center power chains. Data center power outages happen frequently and many cause damage without making the news, according to Mark Gaydos, chief marketing officer for Nlyte.
Gaydos said another technology professional told him “many CIOs are actually negligent when it comes to the power chain. They just assume the power chain is going to function.”
Nlyte advises that companies and Federal agencies take a proactive approach when assessing their data center power chains. When power outages occur and companies don’t know how to handle them, they can result in destruction of the brand and loss of service, customers, and jobs, according to Gaydos.
The first step that companies and agencies must take is to map any physical information technology infrastructure to the power chain.
“This has to be in a system somewhere you can search and understand connectivity,” said Robert Neave, co-founder, chief technology officer, and president of product management at Nlyte.
Neave recommends figuring out how each piece is connected and what devices make up the power chain.
Neave also encourages transparency across all data centers, multiple building management systems, mixed vendors and hardware, and facilities. Companies can achieve this by installing real-time monitoring and alarming across all systems.
Nlyte suggests that companies hold monthly power simulations to ensure that employees know how to respond and recover in the event of an outage. Companies need to be confident enough to instantiate a planned power outage in a production center, according to Neave. By practicing these scenarios, companies could identify systems without power redundancy.
“Sometimes [companies] don’t know until they have a power outage where everything is,” Neave said.
Nlyte says companies should track all data center incidents through their IT department, such as whether outages are increasing or decreasing and when the last outage occurred.
“Are the parts working together or not?” Neave said. “You can bring that information together so you can look historically across time.”
By tracking this information, companies should use trend analysis to identify potential risks before they happen. Companies and agencies should be able to plan for change instead of reacting to change and predict capacity requirements a year from now. Companies should also determine whether the power chain is under warranty, when it will expire, when each piece of the chain was updated, and by whom.
“These are machines,” Neave said. “They will fluctuate when things are not right and they will break.”
Nlyte advises companies and agencies to determine the level of security of their power chains. Companies should ask who has access to control points, when passwords were last updated, and where are the network vulnerabilities.
“They need to be documented and modeled,” Neave said.
In cases where a company works with a data center alongside a contractor, if the contractor’s control point gets breached, the hackers can often access the main company’s portion of the data center as well, according to Neave.
Many of these problems can be avoided if companies adequately prepare for power outages by familiarizing themselves with their power chain before their systems release an alarm, indicating a problem.
“Often when an alert occurs it’s too late,” Gaydos said.
