How to Mature ICS Security with Metrics

At SANS ICS Summit in 2019, I spoke about how to create an ICS security program. Having created several similar programs across multiple critical infrastructure organizations, I started with a simple premise: just start doing something. Most times, a simple starting point can just be counting something easy, such as how many facilities or systems are covered by your security policies. Other times, new projects can present opportunities to bake-in security metrics, like the percentage of the network with OT detection in place or easy quantification of other security controls. These starting points are just that—they won’t be perfect. Still, they can help tell a story that can unlock additional budget and resources or provide executive visibility to problems within the ICS security program that need to be solved.

In the past three years since that presentation, I have been continuing to help ICS security teams and their leaders better understand their security posture through measurement. The lessons learned over the years can now be found in ICS418: ICS Security Essentials for Managers, which I co-authored with fellow SANS ICS instructor Dean Parsons. Across several modules in section two of the course, we explore how to implement an OT security program with a qualified team of subject matter experts and capture meaningful metrics for not just guiding that team but for providing an overall strategy to the organization, including executive leadership.

There are many valid reasons why ICS security leaders should implement a metrics program. First and foremost, look at your engineering and operations counterparts. Their jobs (and in some cases, their very livelihood, when safety is involved) are deeply rooted in accurate and routine measurement. This was one of the reasons why, back in 2019, I told the audience at the SANS ICS Summit that I believe (and still do) that ICS/OT security will solve cyber security measurement before IT does. Why did I say that? Because we are surrounded by measurement. We live and breathe it daily. All we need to do is tap that energy and direct it toward cyber security concerns.

Other reasons for measurement are less about our culture and background and more about the practicality of using metrics. When working with ICS security leaders, I usually focus on a few key “whys” when it comes to creating metrics, including:

• Increase Accountability: Data collection on specific metrics can identify personnel responsible for control implementation and provide input on effectiveness.

• Improve ICS Security Effectiveness: Continued metric development and reporting will track trends for the ICS security program.

• Demonstrate Compliance: Metrics can be built as compliance measures.

• Provide Quantifiable Inputs for Resource Allocation Decisions: If built correctly, metrics can help identify weak points with quantifiable data that can support additional resources for the ICS security program.

These “whys” are adapted from the National Institute of Standards and Technology (NIST) Special Publication (SP) 800-55, which is a great source document for anyone looking to start measuring items of interest in cyber security. While it’s IT-focused, there’s no need to recreate the wheel. ICS security practitioners should be well-versed in how to adapt documents like this for your own benefit.

Basic ICS/OT Security Metric Implementation Process

If the “whys” seem like something you want (accountability for security in the plant? Additional resources to address program gaps? Who doesn’t want that?), then the first question is usually, “what does it take to build a security metrics program?”

First, let’s breakdown the lifecycle for security metrics:

Each step has specific considerations for team sizing, tools, and other resources. Let’s take a look at each step:

1. Create (or Update) Metrics: The first step is always the hardest, so it’s important to understand what makes a metric useful. State your goals. Try to figure out what data is needed to build the metric. For example, if you want to examine how well your incident response and recovery program is performing, perhaps you’d leverage something like the following:

And—yes—update those metrics as needed! If a metric is not used after two cycles of review, drop it. If a targeted level of performance is achieved, move on. Let your leadership know as you implement a new metrics program that these will change over time as your ICS security posture improves.

2. Collect Data: Now that we have created (or updated) our metrics, how do we collect the data? Who actually has the data? ICS security requires multiple stakeholders, so the information may not be in your purview. As such, be prepared to ask other stakeholders for access to data. Further, this data may not all be digital (manual physical logs, for example, might need to be used for access control) and it may not all be objective. I’ve started more than a few metrics programs based on staff interviews; it’s not pretty, but it’s still valuable information.

3. Data Store: Unfortunately, many practitioners think about this after the fact and then end up with a series of Excel data sheets stored all over the place. But what if you’re planful about it? You can leverage not only searchable databases, but you can bake-in security provisions for the data with strict access controls. After all, this information will be a treasure trove of insight about your strengths and weaknesses. Consider data science storage techniques that can also aid in collection and the required analysis.

4. Analyze and Compile Data: With everything collected and properly stored, now you can look for linkages in the data (which can be challenging, depending on the data sources). Look to compile and aggregate the data in metrics as defined in step one. Did the data support your hypothesis? Does it tell a different story? Where required, leverage equations to provide quantifiable measurements and experiment with different charts, graphs, and visual tools to support further interpretation of the information being presented.

5. Report Metrics: Answer the question about what this data is telling you regarding your security program, again linking back to your original statements in step one above. This is where I’ve seen OT security leaders do great things by getting an engineer or data scientist “on loan” from another department (labs are FILLED with data science geeks that may also be interested in security). You may be limited on certain abilities here, but if you start with the original items in step one listed as examples you can always leverage simple spreadsheet charts and graphs.

6. (Actually) Use Metrics: Don’t just admire the problem, do something about it! Take the storytelling and move forward with a plan to address gaps, allocate budget, and execute on plans. The goal of using a metric is to trend toward improvement. There is no value in showing your team stagnating on a security issue, so be prepared to couple a metric with “why” it’s important and what to do about the gaps it shows. This will enable other leaders to help your cause and potentially address larger programmatic issues.

Sizing, Budgeting, and other Considerations for ICS Security Metrics

Ok, so admittedly that looks like a lot to do. And it’s not going to be easy, but this is something you can plan for and build over time. As such, there are a few “rules of the road” for anyone looking to begin this journey:

• First (and this should be obvious), the cost of collecting and analyzing the data for a metric should not exceed the cost of implementing the security control in question.
• Metrics should drive a change in behavior or an improvement in the ICS/OT security program. Be on the look out for abuse or poor morale and adjust your strategy accordingly.
• Standardize where you can across other similar metrics programs. Is there already a workflow for engineers and operators to report metrics? Will it work for your efforts? Then adopt what you can and move forward!
• Metrics collection, analysis, and reporting all cause a certain amount of overhead. You cannot just assign them to your team members without the potential of overwhelming them with additional work. In my experience, never assign more than two to four security metrics per full-time employee (FTE) on your security team (or be prepared for some burn out). Alternatively, establish a dedicated resource or team for the metrics effort.

For many of the organizations I’ve worked with, they will enter a pilot period where they attempt to measure something, maybe anything, that provides additional insights. After working with some champions and showing initial success, those teams will usually go on to expand the program with specific goals for measurement and can then create a workflow similar to what is captured above with dedicated tools and resources.

When to Start

Now.

Seriously, just start measuring something of interest. It doesn’t need to be precise, and it can just be an experiment. What data do you have access to? Where are your strengths and weaknesses, and how would you prove them? What chart or graph would you love to see about your OT security program? Don’t let perfect be the enemy of the good; start your metrics journey today, if you can.

This is just one of many topics we cover in ICS418, designed to help ICS/OT security leaders mature their program and their teams using practical tools based on real-world experience. The course cuts through the noise across dozens of industry standards, guidelines, and best practices to provide actionable guidance. Find out more and register for class at: www.sans.org/ics418

About the Author

Over the past 20 years, Jason D. Christopher has worked across multiple industries in unique roles ranging from engineering to incident response and national security. Most notably, Jason was the federal technical lead for the NERC CIPv5 while at the Federal Energy Regulatory Commission, where he was involved in several rulemakings and policy statements. Jason was also the program lead for the U.S. Department of Energy Cybersecurity Capability Maturity Model (C2M2). He has also served as a C-level executive, security researcher, and incident responder across his career. He is currently the Director of Cyber Risk for Dragos, Inc. focusing on ICS monitoring technology, threat intelligence, and industrial cyber risk. Jason has been invited to speak before the U.S. Congress on several occasions. He teaches ICS456: Essentials for NERC Critical Infrastructure Protection and is co-author of the upcoming course ICS418: ICS Security Essentials for Managers. Learn more about Jason here.

For a deeper dive on developing and enhancing your ICS/OT security program with effective, tailored metrics, download the SANS Strategy Guide: ICS Is the Business.