Shining some light on Solarwinds and ICS

Should OT professionals care about Solarwinds?

As the details of the Solarwinds events continue to emerge and the realities of this campaign are fully understood by the cybersecurity community, there have been some amazing discussions and acknowledgments of interdependency risks across complex enterprise networks and what lessons can be learned from these attacks (see the SANS Solarwinds lightning summit). Meanwhile, in other parts of the organization, there are also discussions around a rising concern within the Operations Technology environments, where practitioners working in the Industrial Control System space find themselves asking what does this mean for us? There is definitely an understanding that this is not just an “IT” problem, but there is also a bit of a question around how big of an “OT” problem is this? I believe the answer is, this is likely a big OT problem.

What SPF are you?

It is important to shift our focus away from Solarwinds specifically for a couple of reasons; first, there has been a lot of great analysis on the “how” and “what” behind this complex attack and it is certainly important to understand what is known with a particular event, but we seem to have entered into the phase of this analysis where there is a lot of finger-pointing in regards to who is the weakest link that allowed this whole access operation to be conducted successfully. The reality is much of the cybersecurity world already knows the real answer: there are multiple weak links everywhere.

This is the second reason we need to move past Solarwinds in our thinking and discussions – there are so many other “Solarwinds” scenarios waiting to be uncovered. From an adversary perspective, if the goal of these campaigns was to gain broad access to a target organization with strong security practices – what better indirect target than attacking a targeted organization's security vendors. Consider a targeted organization with strong security controls or possibly a target that is subject to regulation, they would likely have fairly well-documented policies and procedures in relation to the security controls required throughout their environment. Which could mean that the frequency of security updates performed or the need for network monitoring, logging, and alerting may be a predictable known technology anticipated and leveraged by the adversary. For example; prior to a sector being regulated if an adversary wanted to target multiple utilities in a region and demonstrate a foothold as a deterrent or if needed cause an impact, they would really need to work through the vast variations of each organization's approach to cybersecurity. Now an advanced adversary would likely look to the requirements of a regulation in effect and understand which assets fall under what level of protection, and then understand how those assets are interconnected. This would help focus access operations toward the assets with the lowest security controls in place and would help inform the adversary on what trusted communications paths are in place to allow movement within the environment.

It is important to acknowledge that strong security controls programs and regulations are essential, but it is also important to understand that they do inform adversaries as well. Understanding that a particular target has remote access requirements, encryption, malicious communications detection, endpoint antivirus, security patching, change management capabilities, monitoring, logging, alerting, and routine vulnerability assessment requirements will certainly set the security bar higher and inform adversaries of what they will encounter and what they can take advantage of. For a high criticality site with anticipated security controls, an adversary’s concept of operations may pivot and indirectly target the remote clients wherever they connect securely via remote access, target security vendors to manipulate signatures and engine updates that are necessary for the security detection platforms, target the vendors that provide a hub and spoke trusted connections to endpoints in higher criticality network segments for patch testing repositories and deployments, as well as targeting security monitoring solutions or external managed service providers. With an understanding of what actions a target is required to perform and a deeper understanding of how they perform it, an adversary can also ensure that they operate on the protocols necessary for a specific category of security solution that would be allowed in an environment. As our nation continues to struggle with finding the right balance of regulations and incentives to move critical infrastructure entities to stronger levels of cybersecurity and we consider vendor allow lists or block lists to ensure only certain technologies are deployed, we are also providing a map of controls and technology our adversaries will encounter and need to interact with to achieve their objectives.

Living on an island with plenty of shade

Long ago an ICS cybersecurity professional could talk about their facilities and use the terms “air gapped” or “islanded” with the intent of describing an operational communications path that was isolated and defined. With the level of interconnectedness, interdependence, ravenous data hunger, and the speed at which data is moved; it is hard to mention the word air gapped without being laughed at or at a minimum having everyone assume the individual doesn’t know what they don’t know. Yet even with an enlightened ICS cybersecurity community there is still a gut reaction to quickly move to certainty when events occur that we really need to learn from. Examples of lessons observed and not learned:

Ukraine 2015 – “that is a problem for that country, we are way over here and no one would do that to us” “that was an electric sector attack, we are in natural gas we are different”

Ukraine 2016 – “we don’t use that technology vendor or those devices” “we don’t use those protocols”

NotPetya – “we don’t have those kind of connections to external infrastructure”

Trisis – “our safety systems are isolated and can only be directly accessed”

Oldsmar’s water – “the alarm system would have prevented any impact even if the operator did not”

With the growing concerns of work from home, connections from anywhere, cloud connectivity, transient cyber devices moving from one asset to the next, existence of force multiplier ICS modular attack frameworks, adversary demonstrated capabilities to modify a control systems protections, and the rapid implementation of digital cyber to physical devices at all levels of entity architectures; we need to ensure we are not just asking: Do you use Solarwinds – “we don’t use that product” “IT uses it but we don’t use it in our facilities” “we have it, but we were not on the affected version” “we were on the affected version, but we did not have any indicators”. These are all a throwback response similar to the comfortable “we are air gapped” perspective that allows for a rapid comfortable level of certainty. Even if the organization has high levels of confidence that all is well, it is worth conducting an evaluation exercise with operations and technology support personnel side by side walking through an analysis of what their “Solarwinds” would be and what could be an operational effect from a successful attack.

Supply chain mis-operations

When attacks have affected the ICS environment it is likely that the adversary was living in the environment for some time to test and validate a capability. Those single test events may look like equipment failures or mis-operations due to weather or power fluctuations or some other unidentifiable cause. For critical infrastructure-related operational events, it is important to perform some level of additional analysis when possible to consider a malicious actor as a potential contributing factor. In similar ways, we need to learn from recent events and ensure appropriate analysis is being performed when a supply chain-related attack occurs that is the corollary to a supply chain mis-operation. Looking at corrupt AV signature updates, or invalid detection rules, a failed vendor patch that had to be backed out, or security products needing large exception lists to function without alarms – examining these types of events with a cybersecurity lens that is asking “could this security product be an attack vector” “can I comfortably say this was a legitimate supply chain mis-operation due to a demonstratable issue or is this an unexplainable event of possible significance that needs further analysis.

Rises in the East and sets in the West

While many critical infrastructure asset owners and operators looked to the impacts of the attacks and the initial risks considered by most to be within the Level 4 environment and various East to West segments that allowed connectivity, there is a need to also examine the North to South trusted paths down to Level 2 and possibly even Level 1.

Consider a path created by many security tools or service providers; internet to reverse proxy on corporate, or repository on corporate, or jump host on corporate, or some architected solution within a network segment designed to prevent direct access to higher criticality assets. If the access is trusted and allowed then a path is potentially provided to local site or facility-specific business networks closer to the process environment and of more concern, the final step –there may be a path provided to OT assets in Level 3 and Level 2 for patching, AV, signature updates for security technology, remote access, and numerous other needs. While there may not have traditionally been a need for trusted communications to exist from Level 3 down to Level 1, there are now a number of industrial switches, controllers, and other ICS devices deployed that support log collection, authentication models, network management and host configuration enforcement. At a single facility with operational zones and conduits or multiple network segments or more commonly a flat operational environment, it may be common to find hub and spoke security-related connections broadly from Level 3 down to Level 1 devices (Figure 1).

How do we prepare for the next SunnyBreeze attack?

Understanding that this could have been any number of organizations across numerous security-related implementations, let's consider and prepare for the next attack against a fictitious representative product “SunnyBreeze”. Do not just ask – do we use it in our company, or in a business unit, or in a facility. Rather consider all connections and assume one of your trusted connections did utilize the targeted product or service. Then consider:

• How did you or a trusted connection use the service or product – describe the trusted communications paths and data flow
• What type of controls existed on the established communications paths
• How far back can you demonstrate the absence of any confirming indications and what collection scope is being analyzed
• What operational effects and impacts could an adversary have achieved
• How would you have mitigated those effects to ensure safety, integrity, and reliability of operations
• Are you a trusted communications path to other entities
• Did you have any unexplained operational mis-operations or supply chain-related unexplained failures in the last 4 – 6 months
• Pursue participation in programs that can make a difference for your entity and the larger community:
• • Neighborhood Keeper
  • CyTRICS - DOE's Cyber Testing for Resilient Industrial Control System
  • CYOTE - Cybersecurity for the Operational Technology Environment
  • CCE - Consequence-driven Cyber-informed Engineering

Understanding the interconnectedness and interdependence of our critical infrastructure, and the communications throughout those entities is complicated, however, there are industry trends that are making the environments more predictable, easier to navigate, and anticipate. From a longer-term strategic lessons learned perspective there should be less concern around what specific path or vulnerability the adversary leveraged in this campaign to achieve an objective and more of an understanding of what actions can be taken to limit the next successful attack. As a community, we need to help each other whenever we can and ask for help more than we do. If you have any questions or if I can help in any way please let me know what we can do.

Join the SANS ICS Community Forum to share tips, ask questions and help secure ICS environments. https://ics-community.sans.org/signup