PowerShell "Remoting" is a feature that holds a lot of promise for incident response. "Remoting" is the ability to run PowerShell commands directly on remote systems and have just the results sent back to the querying machine. From an IR standpoint, this is like a built-in agent ready and waiting to answer your investigative questions-at scale. As I'll discuss shortly, Remoting provides us the ability to query a thousand machine in just minutes!
Before I get to the details of Remoting though, let me kickoff the discussion by going through the basics of PowerShell and Windows Remote Management (WinRM), which provides the foundation for the PowerShell "Remoting" feature. After covering this background information, I'll go over the significant performance benefits of Remoting and then cover the authentication details and implications for privileged account use (fortunately there's a lot of good news on this front too).
PowerShell Basics
PowerShell is the relatively new and much improved command shell. PowerShell's roots go back more than 10 years to a technology originally called Monad. Since that time, it has changed its name, been integrated into newer operating systems, and steadily become Microsoft's preferred method for managing their core products. For example, the latest versions of Exchange, SharePoint, and even Server 2012 can be managed almost entirely via PowerShell. In fact, many of the GUI management tools Microsoft provides for these applications are simply performing PowerShell operations in the background.
The latest version of PowerShell is 3.0. Here's a quick synopsis of the version history and platform integration:
- Version 1.0 was released in 2006 as an add-on feature for Windows XP, Server 2003, Server 2008, and Vista
- Version 2.0 was built into Windows 7 and Server 2008 R2
- Version 3.0 was built into Windows 8 and Server 2012
The PowerShell syntax takes a little getting used to, but there is some logic behind it. PowerShell "cmdlets", as they call them, are in the form of Verb-Noun, which makes the commands fairly descriptive in most cases. Here are a few examples:
- Get-ChildItem: Gets the files and folders in a file system drive. Effectively "dir".
- Stop-Service: Stops a service. Other cmdlets available for starting, restarting, querying, etc.
- Remove-ADUser: Removes an Active Directory user.
Probably the best feature of PowerShell is the fact that all output is object-based. This means that the output is in the form of multiple object properties which can be output directly to console, CSV, text file, etc., or used selectively as input into other PowerShell commands via pipelining (using the "|" symbol). Here's a quick example to show what I mean. In this example, I'm using the Get-Member cmdlet to show the object properties and methods available from the output of the Get-EventLog cmdlet:
As you can see, there are a number of properties output from the Get-EventLog cmdlet, any of which might be useful as input to another PowerShell command. In the following example, I use the EventID property (which is essentially an alias for InstanceID) as the value to sort with the Sort-Object cmdlet:
Of course I could have sorted on other properties such as username, source, message, or timestamp (which is the default).
This was just a simple example to show the flexibility that PowerShell's object-oriented output offers.
WinRM Basics
Now let's talk about a different technology which PowerShell can leverage.
Windows Remote Management (WinRM) is a relatively new management platform that several Windows features currently utilize. It provides a communications channel over HTTP, by default via TCP port 5985 in version 2.0 (version 1.0 used TCP/80). Although it uses the HTTP protocol, "traffic by default is only accepted by WinRM when it is encrypted using the Negotiate or Kerberos security service provider". Furthermore, you can also configure the use of HTTPS, adding an additional layer of encryption. (Jason Fossen provides a nice script to assist with configuring SSL on the SANS Windows Security Blog.)
From a PowerShell Remoting standpoint, WinRM provides the platform that allows for running PowerShell commands directly on remote machines. It's important to note though, that there other applications and features that use the WinRM service, such as the Windows Event Forwarding service.
WinRM is included by default on Vista and higher, and can be installed on XP and Server 2003 R2. However, the WinRM service is not running by default on workstation platforms (Vista/7/8), though it is started automatically on Server 2008 and 2012. Group Policy would typically be used to enable it across the workstation fleet. Many more details on setting up the WinRM service can be found in this MSDN article on Windows Remote Management.
That said, there's actually a little more to enabling Remoting than simply starting the WinRM service. Most importantly, you must also register PowerShell as an end point with WinRM so that data coming into the WinRM service is appropriately routed to the PowerShell interpreter. The simplest method that takes care of all the requirements is to run Enable-PSRemoting from PowerShell directly on the hosts. That doesn't scale well though, so Group Policy can also be used to push out these configuration changes. More details on both of these options in this TechNet Scripting Guy Blog article.
PowerShell Remoting Performance Benefits
The nice thing about using PowerShell Remoting is that the PowerShell commands are executed on the remote computer. This not only reduces processing overhead on the querying host, it also minimizes network overhead because the data being analyzed on the remote host does not have to be brought back to the querying machine for processing.
You can perform Remoting in a couple of different ways:
- Interactively on the remote machine using the Enter-PSSession cmdlet. A session is established in which you type commands one at a time on what is essentially a console to the remote machine. The commands are run on the remote machine and the results stay on the remote machine, though you can see the results in your PowerShell command window. This is effectively equivalent to an SSH/Telnet/PsExec remote session (though closest to SSH since the data is encrypted). This is a one-to-one connection.
- Non-interactively, in a batch-style process using the Invoke-Command cmdlet. The commands you request through Invoke-Command are still run on the remote machine, but the results are sent back to your local machine where they can be output directly, or fed into other commands via pipeling for additional local processing. It's very easy to specify multiple machines, so this is typically a one-to-many connection where you query many machines simultaneously. When run against more than one machine, the communications are sent in parallel, up to 32 hosts concurrently by default.
The performance gains from using Remoting are really impressive. Jason Hofferle is a PowerShell advocate who has written a number of articles on the TechNet Scripting Guy Blog. In his article "An Introduction to PowerShell Remoting: Part One," he discussed performance tests he conducted using the Get-WinEvent command. His tests gathered the last 20 logon security events (EventID 4624) from numerous machines across his US-based network. He tested in two different ways:
- In one test, he leveraged Remoting by issuing Invoke-Command to run Get-WinEvent on the remote machines, adjusting the throttle limit to query 50 hosts concurrently.
- In the second test, he used Get-WinEvent's ability to mount an event log over the wire for processing. In this case, processing of the remote event logs occurs on the querying machine, so the remote event logs had to traverse the wire.
Let me expand a little more on the second test. With many PowerShell commands, such as Get-WinEvent, there is a more traditional way of running commands against remote machines: You can specify a "-ComputerName" parameter which uses a traditional CIFS or DCOM connection to access the necessary data on the remote system. While the "-ComputerName" parameter maybe useful for an occasional one-off connection, it does not scale well.
The test results in Jason's article showcase the efficiency of Remoting. The graphs in the article show it taking just 15 seconds to process 100 computers using Remoting, while it took more than 6 hours to perform the same query without Remoting. Furthermore, he went on to test the same Remoting command against 1,000 machines and it finished in just 131 seconds. That's awesome!
The efficiency of Remoting is brought about by two factors:
- The commands are run on the remote machine (optimizing CPU usage and minimizing large data transfers)
- The commands are run in parallel against large pools of remote hosts
These features combined have the makings for a "killer IR app" in large enterprises!
Protecting Privileged Domains Accounts: PowerShell Remoting
All the benefits of PowerShell Remoting would be for not if it exposed our privileged accounts while logging into potentially compromised hosts. Fortunately, Remoting is very strong with respect to protecting privileged domain accounts.
First and foremost, Remoting does not produce an interactive logon, even when using the "interactive" style of remoting that gives you an SSH-like shell on the remote machine. This fact is a little difficult to find in Microsoft documentation, though I did find it discussed in an MSDN article titled "Remoting and the ?Double-Hop'". Specifically, it says the following:
"By default, when a user remotely logs on to a machine the logon session created is of the "remote" type, which is in opposition of an "interactive" logon session doesn't contains the user credentials. Therefore this logon session allows operations on that particular machine but prevents opening a new remote session to another machine. This is done for security purposes as it could be dangerous to grant those credentials to a different machine which can be potentially compromised."
I've also tested this and can verify that an interactive logon was not created when using Invoke-Command or even Enter-PSSession. You can easily verify this as well by checking the event logs of the target machine to see that only a logon Type 3 (network logon) is created. More details on the dangers of interactive logons, how to avoid them, and how to test your tools for interactive logons in this article on password hashes.
Remoting also does not create a delegation token, used to support the Kerberos delegation mechanism (a.k.a., the "double-hop"). Kerberos delegation allows you to authenticate to one machine, and from that machine, authenticate to a second. This is useful for multi-tier solutions like SharePoint, where you typically authenticate to a front-end webserver, and then the front-end webserver can authenticate as you to a backend SQL server in order to access just the data you have rights to access. More on Kerberos delegation here. Although delegation is a handy feature in some circumstances like SharePoint, it can be used by an attacker to steal a privileged user's delegation token and authenticate as that user to a remote machine. In the case of PowerShell Remoting, instead of creating the standard Kerberos delegation token, it uses the CredSSP Security Service Provider to provide delegation features. By default CredSSP usage is disabled, so you have to enable it on both the source and destination hosts to use it with PowerShell. However, you should almost never do this since CredSSP actually passes the username and password to the remote machine and caches it there to be used for authenticating to other machines. Again, CredSSP usage is disabled by default, fortunately. More details on CredSSP in the MSDN article "Windows PowerShell remoting and delegating user credentials".
Finally, user authentication happens via Kerberos, which safeguards your credentials over the wire and machines are even mutually authenticated to ensure you are connecting to the machine you think you are connecting to.
So for all of these reasons, do not fear PowerShell Remoting. That's not to say it is impenetrable, but as far as managing Windows machines remotely, this is as safe-or safer-than any other option. Let me put it this way: If you allow administrators to remote desktop into Windows hosts in your environment, then you should allow, and even encourage, PowerShell Remoting as an alternative since it does a better job of protecting domain credentials by avoiding interactive logons and delegation tokens. This goes for administrators and responders alike.
Final Thoughts
PowerShell Remoting holds a ton of potential for incident response. The ability to query large numbers of hosts quickly, using a powerful and flexible scripting language, and doing so in a way that protects privileged accounts is very compelling. And all of this without the need to install agents! Though, there is some work necessary to enable the service in the enterprise.
Once Remoting is enabled, there are many "out-of-the-box" PowerShell commands that would be useful for sweeping the environment. The Get-WinEvent cmdlet that Jason Hofferle used in his tests is a great example. Being able to quickly query event logs for a particular user logon (or some other event) across hundreds or thousands of hosts in just minutes is awesome. Other "out-of-the-box" commands allow you to query the file system, the registry, running processes, services, scheduled jobs, and more.
Of course many of these commands could be fooled by rootkit techniques, but PowerShell also provides flexibility to go deeper than the Win32 API. For example, there have been PowerShell scripts created that access raw memory and raw disk, so it's certainly possible that some skilled and creative coders could use PowerShell as a platform to do truly in-depth triage of remote hosts.
I'm really excited about the potential of PowerShell Remoting. Hopefully you are too!
Mike Pilkington is a Sr. Information Security Consultant and Incident Responder for a Fortune 500 company in Houston, TX, as well as a SANS Instructor.
