Introduction to ICS Security Part 3

Remote Access Best Practices

The Importance of Remote Access Connections into ICS

Prior to the arrival of the Internet, the ICS/OT environment at most organizations was “air-gapped”, meaning it had no connections to external networks. As a result, network security measures were not major considerations for ICS. As discussed in Part One of this series, however, the persistence of this mindset into the Internet era must be overcome, as ICS environments are now more connected and relied upon for real-time operational data. Subsequently, they have become high-value targets to threat groups.

The benefits of remote access connections into ICS are so significant that many organizations now rely on these types of connections in their day-to-day operations. For example, when an organization needs to check, reprogram, or update their ICS, flying a vendor technician to the site from another location is far less preferable than having the technician remotely connect to the equipment to immediately perform the work with no travel cost. Remote access is also preferable for ongoing management of ICS located at remote field sites because it enables one technician to manage several sites, maximizing his/her efficiency.

So remote connections into ICS are here to stay. Unfortunately, they have been a critical factor in several successful, high-profile cyber-attacks in recent years, including the Dragonfly campaign of 2014, the Ukraine power grid attack in 2015, and the Oldsmar incident of 2021.

However, there are best practices to follow for remote connections into ICS that greatly reduce the likelihood of successful attacks and ensure that threat actors are spotted and stopped before gaining access to critical operational technology.

Remote Access in the Purdue Enterprise Reference Architecture

To clarify, the remote connections to which we’re referring are connections from the Internet and/or an organization’s business network into its OT environment. These connections provide access to devices residing at Levels 3 and below of the Purdue Enterprise Reference Architecture, which we covered in depth in Part Two of this series:

Ideally, remote connections into ICS should pass through the demilitarized zone (DMZ) between the IT and OT segments, and in this edition of the series we will take a closer look at secure remote access architecture. Firewalls, authentication services, jump servers, and file servers all play crucial roles in conducting these connections securely.

In environments built according to best practices, we recommend multiple DMZs consisting of servers at the junction between Levels 3 and 4, each dedicated to a specific purpose. DMZ services include hosting remote access connections, managing Cloud connectivity, serving as the IT gateways into OT, and as the gateways from OT into IT environments. These servers work in concert to accommodate every function a remote user might require, while enforcing a policy of least privilege access to the OT environment.

Best Practices for Remote Access Connectivity to ICS through DMZs

Wherever possible, technicians’ remote connections should be funneled through the DMZs connecting the IT environment (i.e., the business network) to the OT environment. This provides administrators and security practitioners maximum visibility, optimal tracking and logging of activity, and authentication and access control over all users remotely accessing the OT environment.

Remote connectivity for OT staff should require two separate steps: a VPN connection into the ICS DMZ, followed by a second connection using hardened Remote Desktop (RD) via a jump host (or jump server) that provides carefully controlled, role-based access into OT systems.

The following sections address the various individual components of configuring connections through the DMZs into the ICS/OT environment.

Authentication

First, administrators should ensure that everybody who requires remote access connects using named accounts; no shared account should ever be used for this purpose. Remote access users should only be granted access to the systems that enable the tasks they must carry out, and nothing more.

Remote OT users should authenticate with multi-factor authentication (MFA) to the VPN server using dedicated remote access accounts (ideally these are not OT domain accounts). Once connected to the VPN, remote users should only be permitted to connect to a jump host or a secure file transfer mechanism. When connecting to these services, remote users are authenticating a second time, this time using OT domain credentials. At both stages, administrators should log and monitor logon activity, lock accounts after multiple failed authentication attempts, and terminate sessions for sustained inactivity.

Having an Active Directory (AD) domain on the OT/ICS side of the DMZ (typically at Level 3 of the Purdue Model) is a key component to this sequence of authorization. This domain should be dedicated to ICS and must not connect to the corporate Active Directory in any way. It is often placed on a dedicated Level 3 subnet with an enforcement boundary to control communications to and from AD. Deploying an AD server at this Purdue level offers many advantages including:

• Managing and enforcing security policy for all Windows assets in the ICS
• Serving as a central source of authentication so that local accounts do not have to be managed on each individual device
• Managing roles (with AD groups) to facilitate precise control of user activity
• Centralizing creation, modification, and removal of accounts
• Centralizing logging of all Windows activity including authentication
• Jump server authentication and permission setting

While the use of AD in OT networks admittedly increases risk by expanding the attack surface, the additional security capabilities it enables make the tradeoff worthwhile. To mitigate this risk, Active Directory should be managed by trained staff with a strong understanding of Active Directory. Don’t be afraid to take advantage of the AD knowledge and experience of your organization’s IT administrators.

Jump Hosts

For the next step, technicians should be granted access to jump hosts using role-based access. Administrators can enforce this measure using OT domain groups that grant access only to the systems and applications that remote users require to perform their work.

Jump hosts play an important role in secure remote connections which goes beyond authentication. Enforcement boundaries should restrict communication to and from jump hosts. The jump host for each role should only be able to communicate with devices that are managed by that role. These servers should also prohibit users from running any software on, or transferring any data from, their workstations (drive and clipboard redirection should be disabled in the local security policy, for example). Instead, all software and data needed for remote connections should be pre-installed on the jump hosts themselves.

For the small percentage that must bring data with them, administrators should create secure methods to scan and clean required data as it passes into and out of OT networks. This is discussed in more detail in the File Transfers section.

File Transfers

Remote management of ICS systems, whether from the business network or the Internet, often requires the transfer of files both into and out of the OT environment. Examples of file transfers into OT include software updates and patches, antivirus updates, configuration changes, project files, and other documents. Data coming out of the OT environment may include logs and diagnostic information or system backups. Transactional data like order placements and confirmations are frequently transferred both ways.

The approach to secure file transfers over remote connections is conceptually similar to the “sheep dip” method used when technicians need to transfer files using USB drives when accessing ICS environments in person. This method requires the technician to first plug the USB drive into a server at this location that scans all the files on the USB drive and copies the “cleaned” files to another USB drive plugged into the server. The technician then takes the “cleaned” USB drive and uses it to transfer the files onto a particular ICS system.

To perform the same task over remote connections, administrators should set up a file server in the DMZ configured with a write-only folder and a read-only folder and equipped with one or more antivirus scanning tools. The remote user uploads the files to the write-only folder where the server scans them with the antivirus utilities, then copies the files to a read-only folder on a separate server located at Level 3. The remote user can then take the “cleaned” files from the read-only folder and use them in the OT network. An identical process is used to move files out of the OT environment. Ideally, administrators should set up multiple sets of Write and Read folders for each group of remote users, with permissions assigned based on roles managed within the OT domain.

Ultimately, the goal for administrators is to avoid direct remote access into the OT environment, ensure all files transferred are scanned for malware, with folder permissions that prevent the file server from becoming a single, open repository.

Direct Connection Best Practices

Remote access that circumvents the path through the DMZs and connects directly to ICS from the Internet poses a much greater risk and the SANS Institute strongly recommends administrators disallow these types of connections when possible. This is not always possible, however. For example, a third-party contractor may be running extremely expensive client software on their own system for programming field controllers. In these cases, purchasing another license to install on a client’s jump server is not feasible, requiring a direct connection into the OT environment instead. While this practice is considerably more risky and therefore ill-advised given that these contractor systems are likely connecting to the networks of many different organizations, administrators can follow some steps to make direct connections as secure as possible for cases in which there is no alternative.

First, from a management perspective, organizations should create processes that require management approval to provision direct remote access to the OT network. Considering the operational importance of the devices being accessed, management should be aware of every remote connection authorized.

From an administrative perspective, these remote users should still be required to connect to the VPN and authenticate as described above, with all communication passing through the remote access DMZ. Contracts with third parties should require that they take reasonable security precautions on their devices. Measures include using devices dedicated for this purpose (i.e., never used to access the Internet), limiting who can log in and what programs can execute, and running anti-malware software. Once connected to the VPN, the device should be assigned a static IP address so that the DMZ firewall can restrict its access to only the hosts and ports necessary for the specific tasks the device is required to perform.

Where possible, additional controls should be added to ensure that VPN logins (whether for access to a jump server, or for a direct connection) will only succeed if the login originates at a trusted IP address. For example, a list of IP addresses belonging to authorized vendors and staff, along with addresses owned by the organization, can be maintained to further limit access.

Preventing Unauthorized Remote Access

The preceding sections dealt with architecting secure remote access connections into ICS, but unfortunately these measures are sometimes bypassed by users and contractors. There is no malicious intent in these instances; ICS vendors want (and sometimes need) 24/7 access to their equipment for monitoring, and so their personnel may establish an unauthorized connection to enable that access. Internal staff technicians may also set up unauthorized connections for different reasons, leveraging a general-use Internet hotspot, for example. In a typical scenario, the on-site technician may need to fix an issue with an OT server in the control room in the middle of winter (or knows in advance that he/she will need to make a change at 2 AM). Under these circumstances, the technician might connect a hotspot to that server so that he/she can make the required changes from bedside and stay warm instead of trudging across the camp to the server room in the cold.

Regardless of their origins, the unintended consequence of these connections is that they provide short cuts around an organization’s security measures that malicious actors can exploit. To prevent this from happening, administrators should confirm with vendors that they have not set up any unauthorized connections and will not do so going forward. When on site, administrators should look out for the following types of unauthorized connections:

• Cellular modem/hotspot (device),
• Dial-up modem,
• Unauthorized ISP connection,
• Direct connection to vendor networks via Ethernet, wireless, MPLS, or VPN.

To address user or vendor impatience with security protocols, administrators must strike the balance between effective security and ease of use. If vendor personnel require their own remote access connection into ICS, ensure that they are truly authorized, and put appropriate access controls in place, adhering to least-privilege and zero-trust principles so these users can only perform the tasks required of them. If possible, disable these connections by default and enable them manually only when required. Work with vendors to figure out their requirements and build a suitable solution that includes appropriate security controls. If the vendor requires a dedicated connection for monitoring, ensure that it runs through the remote access DMZ and is restricted to only a single segment of the network, by way of a jump server if possible.

Ultimately, this task comes down to educating staff and vendors and having clear policies for them to follow. Even with clear policies and communication, however, there is no replacement for visual inspection. Look around for unauthorized connections – the more remote the site, the more likely unauthorized remote connections will be there.

Conclusion

The preferred method of connecting remotely at the DMZ admittedly involves more authentication steps than those to which technicians are accustomed, so following all of these steps may be met with a lack of enthusiasm by OT staff. Changes in routine are seldom easy, but it is imperative that remote users buy into the additional security measures.

One way to gain stakeholder acceptance is through analogy. Staff in industrial settings regularly attend safety meetings and wear personal protective equipment for the sake of workplace safety. Relating those measures to the required steps for secure remote access connections can help convince remote users of their importance. Also, asking remote users to fully adopt their new routines for three days and then re-evaluate their situation can be surprisingly effective. That interval may seem short, but we have found it to be long enough for remote users to become familiar with a new routine.

Some final words of guidance on remote access connections into ICS:

• Apply the Latest Patches – Remote Access systems are accessible from the Internet, must always be available, and are used to provide external parties access to protected networks, making them a very attractive target. Keeping them up to date with the latest patches reduces your exposure to known exploits.
• Educate Staff – Provide targeted training to users requiring remote access and be sure to convey the importance of consistently following the required steps.

Additionally, the Critical Infrastructure Security Agency (CISA) provides guidance for “Configuring and Managing Remote Access for Industrial Control Systems”[1]. Although it was published in 2010, the guidance is still very relevant today. The key best practices recommended by CISA are:

• Undertake a formal threat and risk assessment
• Eliminate all direct connections to critical operational assets
• Secure modem access beyond default means
• Use DMZs to segregate business and control architectures
• Establish user-specific authentication servers
• Create a security assurance policy for all remote access
• Use only full tunnelling cryptographic technology
• Use a password policy specific to remote access elements
• Use multifactor authentication wherever possible
• Use role-based authorization levels
• Use dedicated hardware and software to support the remote access solution.

Remote access connections into ICS are here to stay due to the long list of benefits they offer to organizations. For administrators, the steps required to follow best practices for securing remote connections may seem like an equally long list, but it is important to remember that no organization can afford to have these connections exploited by malicious actors. Following best practices represents a fine line to walk, but it is nonetheless a crucial component of effective ICS cybersecurity.

In Part Four of this series, we’ll look at secure communications across the IT/OT boundary. Businesses have more reasons than ever to permit data transmissions across this boundary but keeping the boundary secure is equally important.

Sources

[1]https://us-cert.cisa.gov/sites/default/files/recommended_practices/RP_Managing_Remote_Access_S508NC.pdf

About Stephen Mathezer

• Check out Stephen's bio, ICS contributions, and see his upcoming ICS410 course teaches here.
  

Read Parts 1 & 2 of the Introduction to ICS Security blog series

• Part 1 available here
• Part 2 available here
  

Learn More About ICS410: ICS/SCADA Security Essentials