ICS Threat Hunting: “They're Shootin’ at the Lights!” - PART 2

ICS Threat Hunting: “They're Shootin’ at the Lights!” blog series review:

PART 1 – In our first ICS Threat Hunting blog in this series, ICS Threat Hunting: “They're Shootin’ at the Lights!” – PART 1, we focused on extending traditional threat hunting into OT/ICS environments. We referenced the 1988 film Die Hard and its key characters Sgt. Al Powell (a threat hunter) and Deputy Police Chief Dwayne T. Robinson (a non-threat hunter), and we walked through:

The safety aspect for ICS
ICS threat hunting benefits
Several freely available threat intelligence sources to leverage
The people, process, and technology aspects
When to deploy a threat hunting initiative in your ICS security program

In this PART 2 of the blog series we will:

Identify several critical and targeted ICS assets to protect
Identify related data sources for those assets
Focus on aspects of threat intel to use for a hunt
Build a threat hunt package template to prepare for executing the actual hunt

And in PART 3 of the series, soon to come, we will:

Select several relevant hypothesis-driven hunts
Complete a threat hunt packages applicable to all critical infrastructure sectors
Further illustrate the value of technical ICS threat hunting and mapping to attack tactics, techniques, and procedures (TTPs)

More to come!

Screen_Shot_2021-11-02_at_3.11.11_PM.png

The ICS Threat Hunt Mission

IT and ICS systems have different missions, objectives, and impacts during an incident. They also have different assets, including but not limited to embedded operating systems and engineering devices speaking non-traditional industrial protocols. Adversaries targeting ICS must use different attack tactics and techniques for access, execution, collection, and persistence in order to degrade safety, manipulate control, and damage physical engineering assets or property.

Thus, while ICS hunting shares the core attributes of traditional hunting such as hypothesis-driven efforts, it caters to the industrial environments based on ICS-specific engineering assets and different ICS adversary tradecraft.

Look to Attack Trees, Then Hunt

If you’ve just started your journey into threat hunting, there is value to focusing first on your system risk surfaces using attack trees. An attack tree is a list of potentially logical attack vectors that an attacker could string together to achieve an ICS impact. By just jotting them down in a paper-based exercise you can use attack trees to help understand where there may be a need for security controls and prioritize vulnerability management efforts.

As an ICS security program matures, attack trees, and ultimately threat hunt hypotheses, are best driven by known security gaps in the control network or sector-specific threat intelligence.

Screen_Shot_2021-11-02_at_3.53.56_PM.png

Leveraging Threat Intelligence for ICS Hunts

A hypothesis-driven hunt is one of the most popular of several threat hunting methodologies. Threat hunters can use Cyber Threat Intelligence (CTI) to generate attack hypotheses, then sift through available security data sources to stop an attack in progress or identify ways to strengthen a security program before incidents occur. ICS defenders will do well to leverage both operational and tactical CTI for hunts, including indicators of compromise and adversary TTPs. We listed several freely available threat intelligence sources in ICS Threat Hunting: “They're Shootin’ at the Lights!” - PART 1.

Operational: Indicators of Compromise

Operational threat intelligence includes indicators of compromise (IoCs) in the form of malicious IP addresses, hashes of malicious files, and other technical signatures associated with evolving and ongoing attack campaigns. IoCs are commonly used to scope how far a compromise has spread. Operational indicators may also be ingested into security tools to alert security teams when a threat on an endpoint or in the network is detected. IoCs have their limitations, however. They are useful for only a limited time because it is easy for adversaries to quickly change them. For example, in terms of malware file hashes, adversaries can slightly change, recompile, and redistribute malware executables to avoid detection, as a changed file will have a different associated digital hash. IP addresses used as part of an advisory attack infrastructure, as in the case of a C2 (Command and Control server), can change as the adversary switches its attack infrastructure, using different source/destination IP addresses associated with different Internet IP blocks as part of its campaign (a technique also known as “Fast flux”). Thus, IoCs are valuable for scoping an environment primarily for point-in-time checks, but their value can decrease in usefulness over the duration of an attack campaign.

Tactical: Tactics, Techniques, Procedures

Tactical threat intelligence and adversary tradecraft intelligence focus on the behaviors of the threat activity groups – that is, the patterns of their attacks. An example of such an attack pattern and associated TTPs occurs when an adversary focuses on compromising the IT network using malware to gain access and obtain an initial foothold inside the target organization. The adversary then attempts to harvest cached credentials in IT that may be shared between the IT and ICS networks and use them through trusted Active Directory authentication infrastructure. The adversary then tries to pivot from IT into the ICS, which could be done through a trusted ICS asset such as the Data Historian system. Further attacks might target Human Machine Interface (HMI) for direct interaction with the control system, emulating an operator (with malicious purposes), as there is no need for malware code to have an impact at that point. TTPs are more difficult for adversaries to change than IoCs, and many TTPs are generally observable in multiple ICS attack campaigns. For a defender, this means that fixing gaps found through ICS threat hunts can defend against multiple attacks. Adversaries are continuing to evolve, though, even with their tradecraft. Since threat hunting can never be fully automated in IT or ICS, it remains effective as a human-powered repeatable process to generate hypotheses and ingest, comprehend, and apply hunting techniques and protections against adversary patterns. However, threat hunting needs to leverage tools to help automate parts of the hunt, such as collecting, correlating or scoping data, or other key activities. So, by using this type of threat intelligence, defenders can employ IoC scoping and TTPs to map adversary patterns in order to verify if adversary attack patterns are effective if part of an attack against their networks and critical ICS assets.

Hunting Around Critical ICS Assets with Context

It is common for ICS threat hunts to include these control system assets when using IoC scoping and/or TTP attack mapping. These assets are critical for ICS operations, and threat intelligence indicates adversaries can commonly target them to cause safety or operational damage. Let’s walk through what these assets are and how they can be targeted or abused for control system impacts.

Data Historian – This is the database system for control system process information, trending data about the process, etc. For example, a data historian in an electricity generation facility will likely store electricity demands from industrial and residential customers, as well as the rate at which power is being generated, and include data about how to improve the process. It is an asset that may have trusted connections to both IT and ICS. From here an adversary could abuse this trusted asset to pivot from a compromised asset in IT to the control network. In addition, data stored in this database could be highly sensitive and sought-after by adversaries to learn about the industrial process and/or steal intellectual property. For example, a pharmaceutical process might store the amount of different substances that makes up a vaccine in data historian as it is being produced, or the rate at which a vaccine batch is being produced.

Controllers – This refers to Remote Terminal Units (RTUs), Programmable Logic Controllers (PLCs), and Intelligent Electronic Devices (IED). Controllers in the field interface with physical hardware in the real world and monitor and change the state of operations. PLCs or Safety Instrumented Systems (SIS), or their communications, for example, can directly impact safety and availability. Controllers are said to “run the plant floor” in control systems facilities, so communications between a digital PLC and an RTU could provide command information on how to physically open or close breakers in a power system, energizing or de-energizing power to a region or city.

Engineering Workstations – This is usually a laptop or power desktop workstation that is used with engineering software to view, manage, and program network devices, PLCs, RTUs, and other field devices at the lower levels of an entire facility operation. The code to “run the plant floor” is commonly stored on this device, which usually has full access to change the plant floor programming. From here an adversary can reprogram and update controllers that are in operation.

Human Machine Interface – The HMI is a graphical interface used to interact with, change, and control the physical process at a local or remote facility. Operators use the HMI to monitor and acknowledge system alarms, observe safety conditions, and ensure that production is operating as expected. HMIs can run on traditional operating systems on OT assets, or on embedded devices closer to the process in a facility, such as on touch screen panels. From an HMI an adversary can directly interact with the process and manipulate it.

Screen_Shot_2021-11-02_at_4.12.40_PM.png

Most Relevant ICS Data Sources

To prepare a threat hunt, we must build a threat hunt package that targets the right data sources – which may or may not be currently collected. It will come as no surprise that for effective ICS hunts, it is important to have event data from controllers in the field, engineering applications such as the HMI, the data historian, and network traffic to and from identified critical ICS assets.

Network Data Sources

Network session data and firewall access control events should be obtained. Network events can be in the form of the 5-tuple or full-network package captures. Focusing on 5-tuple collection consists of capturing the source and destination IP addresses, source and destination ports, and protocol observed in network communications. Capturing the 5-tuple data requires far less storage requirements than full-packet captures. Full-packet captures consist of the entire packet, including the 5-tuple data as well as the full-packet payload (such as the query and response data, industrial commands, function codes, and other artifacts). Even files in the packet payload may be extracted for analysis during a hunt, or even in incident response cases. Files such as malware samples can be extracted, for example. Full-packet capture does consume significantly more storage space than just capturing 5-tuple data.

The key data sources in control networks that are needed to support threat hunting and should be regularly monitoring for security events include:

HMI to controller network connections
Engineering workstation to controller network connections
Internet inbound and outbound network connection attempts
Remote access logs (RDP, VPN, ICS-DMZ jumphosts, etc.)
Firewall – Network traffic IPFIX data (north/south and east/west)
Network appliance – Network full-packet capture

Endpoint Data Sources

Endpoints in ICS are not just traditional operating systems running on OT assets. In fact, many assets in ICS are not running traditional operating systems. This can make obtaining the right data sources challenging. Where practical, capture registry key data, process execution data, command line arguments, and file attributes. Beyond traditional endpoints, advances in controllers and other Intelligent Electronic Devices in recent years allow forwarding event logs from engineering assets to a syslog server or SIEM for review by ICS security teams.

The key data sources for endpoint devices in ICS that are needed to support threat hunting and should be regularly monitored for security events include:

Windows/Linux OT host logs
Controller local device events (or forwarded to syslog)
Data historian trending data and application access logs
HMI application change and access logs

A SIEM would be ideal to help automate collection and searching in these cases.

Data Source Security Event Retention

You can use this table as a guide to help track and organize assets, data types, storage capabilities, and data source retention for current states and requirements. When we start hunting, we will need to prioritize data sources with lower data retention schedules. The example in the table is of a power system.

ICS Site	Control Center	Electric Generation Facility	Transmission Substation	Combustion Turbine site
Asset	HMI	PLCs	RTUs, Protection Control Relays	Network Appliance or Test Access Point
Data Type	Windows Host logs HMI Application Logs	Syslog	Proprietary	Standard full-packet capture
Data Storage Location	Local on device	ICS SIEM	Local to device	Local to device
Data Storage Retention	60 days	120 days	7 days	30 days

Screen_Shot_2021-11-05_at_10.11.20_AM.png

Building an ICS Threat Hunt Package

As an aid to prepare a hunt, an ICS threat hunt package template, as shown in the table below, will help us outline the purpose and scope of a hunt, including which ICS assets and types of assets to include, the teams that may be required to help pull data or enable access, types of threat intel leveraged, and the hypothesis to be tested. Having a hunt package documented like this will streamline execution and allow you to track your efforts and related outcomes.

Threat Hunt Element	Description
Purpose	The reason for the threat hunt.
Hypothesis	The generated or suggested events that could occur in your ICS – driven by an already identified gap, suggested by threat intelligence, seen observed in your sector, etc.
Scope	Assets and network segments considered part of the hunt.
Related Data Sources	Identified data sources that could be endpoint-based and/or network-based.
ICS Assets	Identified assets that could be, or have been, observed as being targeted as part of an attack in ICS.
Type - Adversary TTP and/or IoC Sweep	List of identified TTPs/IoCs to be verified or swept for.
Actions	Actions to be carried out to sweep for IoCs and/or verify security controls to detect TTPs – the security control to be used, other tools to help automate collection or event correlation.
Teams	Engineering staff, operators, ICS SOC analyst, ICS SOC lead, engineering safety lead, etc.

ICS Threat Hunting Part 3

Now that we have a basic understanding of the benefits of a threat hunt, some critical/targeting ICS assets, what types of intel to leverage, and key network and endpoint data sources, we can start building a threat hunt package for the industrial environment and execute it. In PART 3 of this blog series, we will do just that! We will lay out, select, and execute several threat hunt packages relevant to all ICS critical infrastructure sectors.

Until then, it is a good practice to check the above data sources for your ICS environment(s) to ensure that data from them are being collected, or that they are on the list of data sources from which to start collecting for regular ICS SOC monitoring and ICS hunts.

Join Dean as he teaches an upcoming ICS515: ICS Visibility, Detection, and Response course:

SANS Cyber Security Central: November 2021 Online

SANS Amsterdam November 2021

Free Course Demos

SANS CISO Primer: 4 Cyber Trends

2024 Security Awareness Report

Security Policy Templates

Join the Community

Our Mission