Landing The Plane
I know a CEO of a production plant, let’s call him Bill, who wanted his key team members to really understand the pressures his operations team endured daily to meet production goals. Bill also wanted his managers and support staff to appreciate that there is a difference between “making it happen” and “watching it happen.” In other words, he wanted his team to recognize the difference between thinking you have the skills and knowing you have skills, and to have a firm understanding that there will be a different outcome if you have only “educational knowledge” but not “knowledge combined with experience.”
What did Bill do to instill this lesson? He bought a one-hour flying lesson for 10 of his key staff members.
What? A flying lesson? Why?
To start, as Bill explained, it is not easy to coach someone to do a task if they don’t have context or prior experience. If you are expecting a successful outcome, you’ll need to reset your expectation, purposely expose them to a relevant experience, and coach them as they work through the real-world scenario. Flying is a good example. It seems simple enough, the instructor has an exact copy of the flight controls and is sitting right beside the student telling him or her what to do. But until students have enough skills and experience to correct their own mistakes and eventually land the plane, the instructor will end up taking the flight controls and making sure they don’t run into trouble.
So Bill basically used the one-hour flight scenario to instill the lesson that even a seasoned flight instructor cannot coach a student through a complex scenario like landing a plane if the student lacks experience. From the students’ perspective, they may believe they have a sufficient knowledge of the flight controls, and they may even have practiced with a control yoke connected to a computer simulation. But they will soon find out that to successfully land a plane, knowledge must be married with relevant experience, and in many cases the experience must not be simulated. Relevant experience is… well, relevant.
So, how do you gain relevant experience?
In order to answer this question, let’s talk about what “relevant” experience is. We must first look at how factories or plants are constructed and organized to create a product at a profit. Factories or plants are made up of a collection of original equipment manufacturer (OEM) equipment that is engineered and brought together to produce a product or control a process that ultimately ends up as a consumable good. We can use examples of anything from producing toothpaste to refining gasoline. These wonders of the modern world are indeed fascinating works of art, each in its own right. I have spent my over three-decade career marveling with curiosity about how things are made.
It is only recently that we have connected the machinery together in such a manner that cybersecurity has entered our vocabulary. The control discipline has not fundamentally changed for many years, but the changes in connectivity have forced us to add cybersecurity skills to our toolkit.
In most cases, an OEM has created its own specialized type of machinery or engineered a solution predicated on having expertise or experience with creating a particular product or controlling a particular process. For instance, one OEM may fully understand how to fill a toothpaste tube or put liquid in a bottle, while another may have expertise in heating and mixing all the ingredients together to make the toothpaste. But each OEM lacks the expertise or experience to do what the other does. We must also understand that while the OEM knows the design of its own machinery, the end customer will have a different type of experience that includes learning how to operate, maintain, and troubleshoot the equipment. The OEM must design its equipment taking into account the customer’s different, but no less relevant, perspective, while at the same time develop the skills of its own team to carry out its particular task as efficiently as possible. For example, an OEM’s training regimen may teach such skills as how to code or how to size a control valve. End users’ training, on the other hand, may be more meaningful if they learn how to adjust and tune the machine or troubleshoot electrical problems.
Why is this story relevant? It goes back to the premise that getting relevant experience will lead to success. For instance, I worked for several machinery OEMs, and my relevant experience in that context was understanding how to design control systems to control certain processes. The experience I needed to gain was to understand how the control valves responded with respect to the sizing and responses of the physical valves. This involved understanding the responses of the control system and combining the right control elements together so the operators could enter their desired setpoints and the physical machinery would respond in a predictable and consistent manner.
Contrast my relevant experience with that of an operator. An operator needs to understand how to set values of control setpoints and how the machinery responds, how to enter and adjust those critical parameters, and how to respond to alarm conditions. For instance, if operators were making a baked good and it required them to understand the cooking and taste differences between cane sugar and beet sugar, they would need to know how to adjust the machinery parameters so the product tastes the same regardless of the types of sugar used. For me, as someone working with the OEM manufacturer, I barely know there are different kinds of sugar and I certainly don’t know what the differences are when you bake with either sweetener.
These respective relevant experiences are different, but both are required to achieve success. We can all certainly appreciate that not all experiences are equal or relevant for every actor. This is also true when it comes to ICS cybersecurity training. The experience relevant for your training ultimately depends on what environment you operate in and what skills you need to be successful.
ICS Cybersecurity In-Depth
So now let’s talk about the goals and learning experiences we want to achieve with the SANS ICS612: ICS Cybersecurity In-Depth course. All of the course co-authors felt that we needed to distill the types of systems a security practitioner would find within a production or operational environment and expose students to hands-on labs that give them relevant experience to operate, maintain, monitor, troubleshoot, respond, restore, and defend common elements in an industrial arena. For instance, if I were asked what common elements are found in 90% of all factories or plants I have come across, I would categorize the common elements as follows:
- Real-time embedded control systems like a Programmable Logic Controller (PLC) that will control some machine or process
- Digital and analog input/output subsystems
- Protocol-based “smart” sensors and valves (e.g., Ethernet/IP, Highway Addressable Remote Transducer [HART], DeviceNet, Profibus DP, Profibus PA, etc.)
- Process visibility element(s) like the Electronic Operator Interface (EOI), Human Machine Interfaces (HMIs), and, sometimes, a Supervisory Control and Data Acquisition (SCADA) system
- Process data storage like Historian or local process trending databases
- Network equipment that can be traditional Ethernet-based protocols and some flavors of a real-time control standard such as Ethernet/IP, Modbus, Profibus, etc.
- Security controls such as firewalls, monitoring systems, etc.
The above list is a high-level, albeit not detailed, categorization of common elements that help determine the major domains of the possible training one needs. These categories should be considered in your training because chances are that you will run into these types of technologies in an industrial environment.
When we developed ICS612, we decided that each student must gain experience by participating in hands-on labs. We felt that this would provide students with relevant experience and enable them, by the end of the course, to confidently identify issues and restore plant operations. Of course, if we wanted our students to restore plant operations we needed to create a plant. So, we created an environment where students construct a working coffee factory using PLCs, Networked I/O, EOIs, an HMI server, a Historian server, and other supporting elements such as remote connectivity technology and a working industrial demilitarized zone. This allows us to work with students on each element and eventually in a “system of systems” model that really represents a system comprised of complex systems.
Understanding how to program and operate common industrial assets is a fundamentally necessary component of ICS612, but we felt the class would not be complete without working through asset attack labs. Each asset found within an industrial production environment has different vulnerabilities, and knowing how to architect and protect these critical assets is paramount. The hands-on attack labs have been mindfully constructed to show the strengths and weakness of these assets. We believe the right mix for an industrial cybersecurity class entails learning how to get the asset working in a normal operating condition, defend the asset through lab exercises, and discover vulnerabilities through attack labs. Understanding how to defend and attack is not only fun, but is also very important.
As I mentioned earlier, the course culminates on day five with a scenario involving a broken-down coffee factory. Each student is expected to get his or her part of the plant up and running. As parts of the coffee factory are restored, we encourage students to help one another get their part of the factory restored. As we know, cybersecurity is a team sport (which makes training in a virtual environment all the more difficult). Once we actually work through the failure modes, the real troubleshooting begins. Students quickly understand that working on real equipment is much different than simulated environments. When motors don’t turn on, or a fuse blows and you smell smoke, or the lights are indicating something incorrectly because the relay contact isn’t working, the adventure and confidence you get from solving the problem builds to something beyond a computer simulation. Our day five students are much different than our day one students. They have been given relevant ICS cybersecurity experience that they can immediately use in their work environment. Our goal is to provide training that allows practitioners to immediately use their ICS612 classroom training in real-world situations. Feedback from our ICS612 alumni has consistently echoed our goal that their training has been relevant and allowed them to feel more confident and effective within an industrial operation environment.
So how did the one-hour flight experience of Bill’s 10 staff members come out? He says the experience for the participants will forever change their perspective about thinking they know something as opposed to really knowing that they know something. Ask yourself, if you had to put your simulated flying skills on the line in a real airplane, how confident are you that you could land a plane?
It all goes to show that knowledge and the right experience are the key to success in anything. As a side note, one of 10 participants in Bill’s experiment went on to become a pilot, an aircraft owner, and even has his own runway!
And oh yes, by the way, I’m also a private pilot.
In my next blog, I will dive a little deeper into the learning objectives of ICS612: ICS Cybersecurity In-Depth and more specifically why we chose to challenge our students with a “broken” coffee factory to reinforce all of the skills we have honed during our first four days of class.
Hope to see you in class!
Email: jshearer@sans.org
Find Upcoming ICS612 Training In-Person:
