Top 5 ICS Incident Response Tabletops and How to Run Them

How prepared is your organization to respond to an industrial control system (ICS) cyber incident? How resilient is it against Ransomware that could impact safety and operations? Does your organization have the ability to detect advanced persistent threats that use modern attack methodologies against your critical infrastructure?

Regularly conducted incident response tabletop exercises are part of a mature ICS Security Program that can identify weak points in security efforts and enable proactive defense to address this range of threats.

ICS Incident Response Tabletops Explained

ICS incident response tabletops are much like the pre-game practice drills that sports teams, like hockey teams, run before a game. Like pre-game drills, ICS incident response scenarios are designed to test all that will be needed once the game begins. In this case, however, the game is the serious business of cybersecurity, and it requires ICS defense capabilities, safety processes, and cyber preparedness. These proactive initiatives test the effectiveness of an ICS Security Program prior to an attack. Tabletops are paper-based, and they are conducted in roundtable discussions guided by an Incident Response Plan, knowledge of the engineering processes, and an understanding of the existing ICS security defenses. Weak points are identified and assigned to be addressed immediately in order to strengthen the program.

The question is, how will your industrial organization respond once the “game” begins?

The Benefits of the ICS Incident Response Tabletop

ICS incident response tabletops provide a high return on investment in several important areas.

Validation – The tabletop exercises validate readiness by comparing the defense controls against existing controls. Areas of improvement are identified in industrial incident response plans, security, and safety playbooks. Simultaneously, tabletops help train new and established team members on the industrial process and ICS-specific security.

Situational Awareness and Team Building – Reviewing threat intelligence with the teams involved will educate them about adversary capabilities and attack techniques. Regularly performing tabletops will establish and strengthen cross-departmental relationships needed for incident response events that could span multiple industrial sites across large geographic regions.

Practical Defense Actions – Tabletop exercises can identify gaps is such critical areas as threat detection, data source collection, log correlation, network segmentation changes, access control updates, security and safety process changes, and the communication of roles and responsibilities. Effectiveness in all of these areas is key for a mature program. Tabletop actions will directly improve overall response time, reduce impacts on the engineering process, and increase safety.

Planning and Running ICS Tabletops

Planning – Planning time will vary depending on team size, the scenario, resources, etc., but it typically can take anywhere from a few days up to 30 days. Even a planning phase of just 2 to 5 days is enough to provide value in the outcome. Spend time up front properly selecting realistic scenarios for your environment and selecting the right teams. Include as many team players and observers as is practical.

Tabletop Goals – Are you testing newly deployed technology, training new team members, or using through intelligence and recent sector events to validate or update your ICS Incident Response Plan? Or are your tabletops driven by compliance requirements? Set the goal and adhere strictly to timelines and frequency. Adhering to safety requirements will also be a goal in ICS.

Frequency – Some compliance programs suggest that tabletops be run every 15 months (for example, the NERC-CIP-008-6 Table R2 – Cyber Security Incident Response Plan Implementation and Testing). It is common practice to run a tabletop annually, and the exercise can be aligned with budget cycles.

Designate a Facilitator – A facilitator will keep things on track to ensure that the scenario is completely walked through and tested against the ICS Incident Response Plan. A facilitator can also ensure that everyone involved is engaged for maximize discussion, that notes are recorded, and that actions are assigned to individuals.

ICS Teams – Include all teams that are practical to involve. Invite observers to listen to the discussions for training purposes. Start with the following:

• Safety – Include the on-site safety and emergency response team. • Physical Security – Include the on-site facility physical security team.
• Compliance – Ensure that legal and regulatory compliance requirements are met.
• Cybersecurity – Since cybersecurity drives the scenario, participants must understand the defenses and the Incident Response Plan, the technologies and the industrial operations process, protocols, critical assets, the network layout, etc.
• Engineering – Include process control and field device technicians.
• Operator – These are the persons who control the process via remote and embedded HMIs, etc.
• Management – Management and director-level stakeholders for all teams involved need to have an awareness and understanding of ICS cybersecurity risk, impacts, protections, budget, resourcing, etc.

Scenarios for ICS – Select one of several suggested scenarios outlined in this document to get you started. Scenarios should be based on closing known gaps already identified in the program and on significant industry events that have occurred in your sector. Such a threat-centric approach leverages ICS threat intelligence specific to your sector.

Run Time – Run time will vary depending on the size of the teams involved and the complexity of the scenario selected. A typical tabletop for ICS can run from 2-3 hours to 1-2 days. Longer and more involved incident response exercises such as Hybrid or Live can run for several days.

Closing Gaps – Designate a person to take notes of related action items to be assigned to specific individuals. These actions might include investigations, enabling security features, completing assigned training related to a role, using a new ICS security tool, changing a network design (for example, aligning to Purdue), implementing new processes or technologies, etc.

Mini-Project, Action Tracking – Some ICS programs run tabletops as a small project internally or with an external third party. With either approach, it is common to see project managers dedicated to ensuring that tasks are completed on time and with an appropriate budget. Tracking tasks can follow the SMART (Specific, Measurable, Achievable, Realistic, Timely) objectives.

ICS Scenarios – Include Critical and Targeted Assets

There are many critical engineering ICS assets to protect, including physical systems and digital systems, and they usually span several geographic areas. Threat intelligence indicates these assets have been targeted in observed ICS attacks. At a minimum, the following critical ICS assets should be included in your scenarios:

Data Historian – This database stores operational process records. It can be abused to pivot from a compromised asset in IT to one in the ICS network(s).

Engineering Workstation – This workstation has access to software to program and change PLCs and other field device settings/configurations. Be aware of its location and of normal and abnormal access attempts to and from it. Pay attention as well to data exfiltration connections from the engineering workstation.

Human Machine Interface – The HMI is a visual interface between the physical process and operators that is used to review and control the process. Remote access, if required, should have secure, heavily controlled, and monitored multi-factor authentication.

Programmable Logic Controllers – PLCs connect the physical hardware in the real world and run logic code to read the state or change the state of the engineered process. An example is Safety Instrumented Systems (SIS) safety controllers.

The Top 5 ICS Incident Response Tabletop Scenarios

Here are 5 tabletop scenarios based on campaigns seen across multiple ICS sectors. Each scenario is presented with suggestions in the categories of Discussion, Teams, Protection, Detection, and Response. Use these categories to guide the tabletop exercise and to consider the effectiveness of your existing ICS security program. Customize the exercise to suit your needs.

SCENARIO 1: Living off the Land: Native Industrial Control System Protocol Abuse

The engineering team troubleshooting network issues observes unusual ICS protocol communications (OPC, IEC104, Modbus/TCP, DNP3, ICCP, etc.) on the network. For example, there are unusual scanning rates of OPC to and from SCADA servers to outstations and other devices at several levels of the Purdue network architecture.

As an example, the CRASHOVERRIDE ICS attack framework can abuse the IEC-104 protocol, has built-in features to abuse other ICS protocols such as OPO, and has the potential to expand to also abuse DNP3.

DISCUSSION: Are your legitimate in-use ICS protocols in a list or baselined? Is ICS-specific (ICS protocol-aware) Network Security Monitoring (NSM) visibility deployed at Levels 0-3 of the Purdue model for IPFIX or full packet captures?

TEAMS: Engineering, Cybersecurity, Safety, ICS Network Architects.

PROTECTION: ICS-specific network monitoring – ICS deep packet inspection and ICS dissectors in use.

DETECTION: Trained ICS cybersecurity team members using ICS NSM and investigating suspected events in order to initiate incident response.

RESPONSE: ICS incident response playbook with required teams, check integrity of field devices and operations, loss of control = emergency.

SCENARIO 2: Human Machine Interface Hijack: On-screen Suspected Activity

Human Machine Interface operators notice the on-screen mouse moving and clicking on different control buttons on the HMI, which is not consistent with normal operations or a scheduled change or safety emergency.

DISCUSSION: Which accounts and individuals have access to HMIs for local or remote access?

TEAMS: Engineering, Operators, ICS Security, Network Architects.

PROTECTION: Purdue Network Architecture, process control, operators having a process for reporting cyber events.

DETECTION: Secure remote access event monitoring - External->Internal, Internal->Internal - RDP, Multi-factor authentication, use of a jump box in ICS DMZ (Purdue Level 3, etc.).

RESPONSE: Disable remote access, run ICS on plant floor via embedded HMIs, investigate NSM network traffic patterns, enable islanding from Internet, IT, etc.

SCENARIO 3: Physical Access to Cyber Access Event

The Physical Security team notices a hole cut into the physical security perimeter – the fence surrounding a remote facility. The team investigates and determines that the physical attack could be a two-part attack. Physical access was gained, then attackers pivoted to a cyber attack as containment was introduced into the control network at a remote site. Traditional break-ins have been observed to be for monetary value, such as copper theft from electric utilities. Some critical remote ICS sites could be vulnerable to a physical and a physical-cyber attack. This also presents a safety concern for workers in remote facilities such as electricity substations, switching yards, oil and gas valve stations, fuel storage facilities such as marine terminals, etc.

DISCUSSION: Physical security at remote sites could be the most vulnerable ICS facilities (substations, oil/gas storage facilities, valve stations, etc.).

TEAMS: Physical Security Teams, Engineering, Cybersecurity, Safety.

PROTECTION: Security guards stationed at site(s) or security checks on rotation.

DETECTION: Physical door alarms, surveillance cameras, rotating security guards, etc.

RESPONSE: Roll trucks to site, law enforcement.

SAFETY: A concern for adversaries in dangerous life-threatening situations and workers on-site in the event of a break-in.

SCENARIO 4: Ransomware on IT or ICS/OT Networks

ICS operator workstations in a control center are infected with Ransomware and are inoperable to view or control the industrial process. Alternatively, the IT business network is inoperable due to a ransomware infection in the enterprise – critical ICS process application such as industrial billing and shipping logistics applications are inoperable.

DISCUSSION: Does ICS rely on IT, and to what extent? Is it possible to island ICS from IT in a cyber defensive position? 

TEAMS: IT, IT Security, ICS Security, Engineering, Operators, Safety.

PROTECTION: Email security (if IT is infected with the common email phishing vector), whitelisting on ICS endpoints, IT - ICS Network Segmentation (Purdue Network Architecture).

DETECTION: ICS-specific endpoint protection, ICS NSM (lateral movement)

RESPONSE: Is it possible or feasible to run the ICS process in manual mode from embedded HMIs on the plant floor in the event the primary HMIs are inoperable due to Ransomware or another threat? It may be possible to respond by cutting or limiting network segment communication for containment while fighting through the attack.

SCENARIO 5: IT or ICS Network Pivot through Trusted Connections/OT Networks

The ICS Data Historian, a critical ICS asset, is a targeted and common pivot point from IT into ICS environments for attackers. The adversary can learn about the industrial operations by gleaning sensitive ICS data from information on the Data Historian. A set of compromised IT Active Directory credentials were used to access the Data Historian, then pivot into the industrial control environment. It is critical that ICS networks be segmented from the Internet and from the IT business network(s). Aligning with the Purdue Network Architecture to configure enforcement boundaries for protection is also affordable and effective.

DISCUSSION: Network segmentation, access control to and from Data Historian, multi-factor authentication, separate no-trust ICS Active Directory from IT Active Directory.

TEAMS: IT, Network Architects, Engineering, ICS Cybersecurity.

DETECTION: Network segmentation, access control technologies, and regular log monitoring of Data Historian and other trusted assets.

PROTECTION: Separate untrusted Active Directory for IT and ICS.

RESPONSE: Limit connectivity to Data Historian, look for signs of exfiltration from Data Historian to the Internet, C2 (Command and Control) servers, etc.

BONUS SCENARIO: Contaminated Transient Device

A contaminated transient device (for example, an infected USB device or laptop) is brought into a facility bypassing all physical security and cyber technologies. It is plugged directly into Safety Instrumented System (SIS) controllers (on the plant floor – Purdue Level 0-1) for routine maintenance such as patching/firmware updates.

DISCUSSION: Network and Device Access Control at sites – MAC filtering, device interrogation, automated malware analysis of common file types on USBs prior to being used at the site, etc.

TEAMS: Physical Security, Engineering, Cybersecurity, Safety.

PROTECTION: Network Access Control, ICS plant floor kiosks/scanners, laptops on isolated interrogation segment before being connected to field devices, internally managed (never exposed to external networks), loaded with vendor software available on-site for use only at the site.

RESPONSE: Allows/Disallows based on technical digital tokens on transient devices or a “clean” scan prior to plugging in and becoming active on the network.

ICS Incident Response Tabletop Summary

Regular incident response tabletop exercises are part of a mature ICS Security Program. They work proactively to identify weak points in ICS defense efforts, build strong relationships among several teams, and are commonly driven by proactive defense or compliance requirements.

How to Start Your ICS Incident Response Tabletops

1. Select one of the presented realistic ICS Incident Response Tabletop Scenarios for your next exercise.
2. Mature the process by creating your own scenario based on your ICS threat landscape by leveraging ICS threat intelligence, internal or external gap assessments, compliance reports, etc.
3. Involve as many teams as practical, including Safety, Process Controls Engineering, Operators, ICS Network Architects, ICS Security, Plant Management, etc.
4. Discuss, learn, take action, and repeat. “ICS Defense Is Doable!”

Dean Parsons' upcoming ICS515 course runs here:

• SANSFIRE 2021 Online | July 12 - 17 | Register Today
• SANS Dallas 2021 | October 11 - 16 | Register Today

Check out Dean’s ICS Contributions and Bio here.

Join the SANS ICS Community Forum - Tips, tricks, and Q&A to secure your ICS!

Watch the original webcast of the Top 5 Incident Response Tabletop and How to Run Them: