In 2017, the Defense Advanced Research Projects Agency contracted with the research arm of the National Rural Electric Cooperative Association on a program that convened power engineers, cybersecurity personnel and first responders in simulations of cyberattacks incapacitating critical infrastructure. The program aimed to develop technologies that enable a cybersecurity black start by maintaining situational awareness, supporting network isolation and rapidly characterizing cyberattacks. The research was developed with funding from the Defense Advanced Research Projects Agency (DARPA).
For the last three years, research teams from the utility and cybersecurity industries have been journeying to Plum Island, a 3-mile long by 1-mile wide (4.8 km by 1.6 km) piece of land off the coast of Long Island, New York, U.S., in the Atlantic, to test their ability to resurrect the electric grid following a potentially devastating cyberattack that incapacitates critical infrastructure.
While utilities have plenty of experience recovering from natural disasters, the program on Plum Island examined how utilities would recover following a malicious, intentional and targeted cyberattack.
Planning Ahead
According to DARPA, "A substantial and prolonged disruption of electric power would have profound economic and human costs for the United States. From a defense perspective, a major power outage could hamper military mobilization and logistics and impair the capability to project force."
If utilities are unable to restore service quickly following a wide-scale cyberattack, they may be forced to implement a multi-utility cyber black start without power, phones or the internet. Because such a massive attack has not occurred yet, this kind of recovery has not been attempted. The Plum Island exercises intended to narrow the gap between theories and reality about what it would take for a successful cyber black start.
For the National Rural Electric Cooperative Association, the program also provided a valuable opportunity to test a new system that combines sensors and network monitoring to provide out-of-band situational awareness when the grid and associated communications systems are down.
Intense Exercises
A rapid attack detection, isolation, and characterization systems (RADICS) program notably differs from more typical cybersecurity research in two ways:
- RADICS does not focus on preventing an attack; instead, it begins with the premise a successful attack has occurred.
- Live power is another difference. Unlike simulations regularly carried out to test protocols and procedures, exercises in a RADICS program are conducted using a working power plant and two substations.
The Plum Island tests — seven exercises in all — were conducted by DARPA in coordination with the U.S. Department of Energy and Department of Homeland Security. DARPA worked with its partners to develop technologies for cybersecurity personnel, power engineers and first responders that could accelerate restoration of electrical systems affected by cyberattacks. Specifically, the research focused on technologies that could maintain situational awareness, enable network isolation and rapidly diagnose cyberattacks.
Conducted on the largely deserted island that is home to the nation’s animal disease center, the exercises consisted of a cyberattack that brings down a power plant and incapacitates two substations — albeit, outside of the facility proper and with no impact on the island’s actual power supply. Eight teams worked together to determine the cause or causes and bring the plant and substations back up. NRECA’s role was to provide situational awareness.
Following the initial attack, which occurred before the NRECA team arrived, utility personnel first had to determine what was not working, what was infected and how to bring components back on-line, knowing they could be reinfected almost immediately if connected to still-infected components. Restoration started with studying the source and type of attack, and then proceeded through a detailed and complex path of incremental improvements, each checked and verified. The work was exacting and immensely complicated because of the lack of a communications infrastructure and continued malicious attacks.
While natural disasters are indiscriminate in their destruction, cyberattacks are targeted. After locating the malware, the teams were able to contain the damage and begin rebuilding. With each subsequent exercise, the cyberattacks became more extensive and damaging.
Situational Awareness
NRECA first began developing the GridState system in 2010 for use in the Plum Island research. GridState captures utility network traffic and detects anomalies using a variety of methods, including advanced artificial intelligence and machine learning. It also detects potential breaches in a matter of minutes. In short, GridState provide near-real-time situational awareness of the power grid.
In the Plum Island exercises conducted so far, GridState presented an accurate state of the power grid regardless of the state of the supervisory control and data acquisition (SCADA) system.
GridState captures and analyzes communications within and between electrical components and the industrial control system. The system looks for anomalies that indicate a cyber event has occurred and then prepares the evidence for cyber specialists to analyze. Once the device has been scoured and its functionality restored, GridState verifies normal operation.
In one test, for example, a substation breaker received a command to operate the breaker. Using GridState’s deep packet inspection, the SCADA point indicating a breaker operation could be seen and evaluated. GridState has grid connectivity onboard and uses the data to check if voltage is present downline from the breaker. GridState determines whether the breaker is open or closed. An alert notifies the system operator of a breaker operation and gives the person an opportunity to verify with the SCADA operator whether the command to operate the breaker was in fact issued by that operator.
If the breaker operation is valid, then the system operator uses the user interface to verify the operation. If the breaker operation is not authorized, then the system operator can dispatch a technician to the substation. If instructed to do so, the technician can set the breaker to local and operate it manually to the correct setting. When in local mode, most breakers will not accept SCADA commands. This protects the device from being operated with unauthorized commands.
Rather than relying on a SCADA system for state information that may be compromised, the impacted utilities can turn to GridState, which provides detailed and immediate state information on high-performance analysis of industrial control system traffic. GridState can parse virtually any control and monitoring control protocols as fast as they come in, and it recognizes anything out of the norm.
In each event exercise conducted so far, GridState provided the situational awareness necessary for recovery.
The GridState system is an advanced version of an earlier system developed by NRECA, called Essence. NRECA partnered with BlackByte Cyber Security LLC to develop the sensor, In2Lytics to create a high-performance database and Carnegie Mellon University for advanced machine learning.
Next Steps
DARPA’s collaborative research was completed in May of 2020, with partner transitioning scheduled after completion. NRECA continues to develop the GridState system.
GridState has been deployed at three cooperatives to validate its capabilities across a broader inventory of SCADA and advanced metering infrastructure and automatic meter reading systems. NRECA will deploy the system at up to 40 utilities in testing before deciding on a strategy for large-scale production deployment.
Editor’s note: The views, opinions and/or findings expressed herein are those of the author and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.
