In the rapidly evolving landscape of substation security, the importance of clear and effective communication cannot be overstated. However, the necessity for information security and confidentiality means that miscommunication and information silos are still prevalent at these facilities. This disconnect often leads to inefficiencies, gaps in knowledge transfer and missed opportunities for innovation, especially between security auditors and designers.
The current process for auditing physical substation security typically begins with regulators or auditors notifying utilities in advance about their intent to conduct an audit. During this notification, auditors request records from the utility covering a specific timeframe — typically 30, 60 or 90 days.
The utility is responsible for compiling the requested information and sending it to the auditors for preliminary review prior to a site visit. These records are reviewed against established standards, which allows auditors to gain an initial understanding of the utility's compliance efforts within the specified time.
Then, during the on-site review, auditors will indicate a specific standard or piece of evidence and ask the utility to provide a subject matter expert who can speak about the associated documents, procedures or processes in an interview. These interviews may lead to requests for additional evidence based on initial findings. The primary goal of these audits is to identify any gaps in compliance.
At the end of the audit, auditors will state if they found any areas of non-compliance or any large gaps in supporting documents. They may recommend changes to processes or procedures to close those gaps.
It is important to know that auditors have their own interpretations of standards based on their individual perspectives, which may be different than the utility’s. There is a lot of back-and-forth discussion before any official findings are determined.
An Expert’s Perspective
Chris Ott began his career in the late 1990s, joining the United States Marine Corps as a power generation and distribution expert. Following the events of 9/11, he was tasked with supporting war efforts by identifying critical infrastructure in foreign nations such as Iraq and Afghanistan. His role involved determining how to neutralize power grids, when necessary, based on available military assets.
After leaving the Marines in 2006, Ott completed an electronics engineering degree before dedicating nearly 15 years to designing security systems, mitigations and protections for various sectors, including utility companies.
In 2013, Ott joined a corporate security team for a prominent utility company in the Pacific Northwest. In this role, he specialized in threat vulnerability assessments, subsequent mitigations, designs and protections. Now, he serves as a senior physical security expert for POWER Engineers, creating tools and processes to facilitate better protections for the grid.
“I’ve been on both sides of this coin,” says Ott. “I’ve been the designer, and I’ve worked closely with auditors before, during and after audits. I’ve also done pre-audit preparations, which is like a mock audit where I would think about what an auditor might ask, look at the documents they might pull and talk to my coworkers who they might interview. That experience puts me in a unique position to see the inefficiencies on both ends.”
Below are some of the inefficiencies Ott has noticed.
Communication Silos
According to Ott, one of the biggest problems during physical substation security audits is that the right information isn’t being communicated to the right people. The designers on these security upgrade projects aren’t always in-house teams. In fact, they are often contractors and subcontractors. Auditors interview utility members, not contractors, which means that if the subject matter expert that is being interviewed by the auditors receives any feedback, there is no guarantee that the information will be conveyed back to the designers.
On top of that, some people are told not to talk to the auditors outside of an interview, which prevents more in-depth, informal discussions on ways to improve security.
“Auditors aren’t trying to trick you,” says Ott. “They genuinely want the site to do well. If these lines of communication weren’t so formal and closed off, we’d be able to talk about our challenges and work together to mitigate vulnerabilities much more effectively.”
Misaligned Priorities
When it comes to substation security, auditors and designers often have differing priorities that can lead to misalignment and consequently stifle progress in implementing effective security upgrades.
Auditors operate under a mandate to ensure that utilities comply with established standards and best practices. Their primary focus is on adherence to regulatory requirements, maintaining a high level of security compliance and ensuring that all measures are up to code. While auditors possess some flexibility within their mandate, they strive to uphold the highest standards of safety and security for critical infrastructure.
On the other hand, designers are driven by financial constraints and the need to keep costs low. Utilities face significant pressure to maintain affordable rates for customers, which often leads designers to prioritize cost-efficiency over comprehensive security measures. This financial focus can sometimes result in compromises on security enhancements if deemed too expensive.
“But the upfront cost of a security upgrade project is nothing compared to the aftermath of an attack,” says Ott. “We’re seeing unknown grid downtime, long lead times for replacement equipment and sometimes tens of millions of dollars in repair costs across the country. And on top of that, these attacks are becoming more prevalent every year. More sophisticated, too. As technology evolves, so do bad actors. We’re at a point now where mitigation tactics that worked a few years ago are no longer sufficient against modern methods of attack.”
The divergence between auditors' emphasis on strict compliance and best practices and designers' focus on cost-efficiency creates obstacles in achieving meaningful progress in substation security upgrades. When budget concerns override security priorities, necessary improvements may be delayed or neglected. This can lead to vulnerabilities that could have been proactively addressed.
Mutually Misunderstanding
Both substation designers and security auditors can misunderstand limitations due to an incomplete understanding of each other's requirements and constraints.
Designers must often adjust their designs to accommodate specific environmental conditions. For instance, they may need to consider factors such as local weather patterns, geographical challenges or site-specific requirements. These adjustments are crucial for ensuring that the infrastructure functions effectively and safely within its environment.
Auditors operate within large regulatory territories and are responsible for ensuring compliance with established security standards. Their focus is on implementing best practices that ensure a baseline level of safety and security across all regions under their jurisdiction. However, these territories can be vast, making it challenging for auditors to have detailed knowledge of the unique environmental conditions that designers must consider.
What auditors deem as best practices may not always be feasible or optimal for the local environment. For example, while a concrete wall designed to withstand tornadoes might be a best practice in one region, it may not be suitable or necessary in another area with different environmental conditions. This lack of alignment can lead to frustration and inefficiencies in implementing security measures.
“There’s a lot of overbuilding and overspending happening across the industry because analyses aren’t catching everything and security solutions aren’t being tailored to the unique needs of each site,” says Ott. “My team and I have been experimenting with new ways to improve the quality of communication and collaboration amongst designers, decision makers, auditors and other stakeholders who are involved in substation security upgrades. There’s a lot of potential to save time, money and resources if you use the right tools and involve the right people.”
Building a Bridge
Meeting compliance standards establishes a baseline for substation security measures, but true excellence requires continuous improvement and proactive strategies. Simply adhering to basic standards is not enough; failing to do so indicates that minimum requirements for effective substation security are not being met.
To bridge the innovation rift, there needs to be more open dialogue between utilities and auditors. Both parties should feel comfortable sharing information without fear of repercussions. Transparent communication allows designers to better understand the security requirements set by auditors, while auditors gain insights into the design constraints faced by designers. This mutual understanding is crucial for developing security measures that are both compliant with regulatory standards and tailored to fit specific site conditions.
Collaboration is key. Auditors and designers must work closely together to balance high-security standards with project cost management. By aligning their priorities and working towards common goals, they can ensure that substation security upgrades are both effective and cost-efficient.
Ultimately, this collaborative approach enhances the protection of critical infrastructure while meeting financial objectives. Addressing these mutually misunderstood limitations through open dialogue ensures a more practical and comprehensive approach to substation security, fostering innovation and progress in the field.
