Protecting Privileged Domain Accounts: Safeguarding Password Hashes

Have you ever made a connection to a potentially compromised remote machine using a privileged domain account and wondered if there was any chance that your privileged credentials could be revealed in some way to the attacker? I have. After wondering and worrying about it, the curiosity (and paranoia) finally got to me and so I set off on a journey to research attacks against domain credentials, and in particular, their implication for incident responders. I've presented on this topic a few times and now I will (finally) take the time to document my findings. This is the first article in what will be a multi-part series on this research.

I find this to be a fascinating topic and one which should be of interest to the entire IR community. That said, be forewarned that these articles will not be quick reads. If you'll stick with me though, I believe it will be worth your time because you should walk away knowing exactly what you can and cannot do safely with your privileged domain accounts.

So here we go!

The Scenario

Let's start with a scenario-a worst case, yet realistic scenario-where we have a Windows XP Service Pack 2 client machine on our domain that is likely compromised. We need to analyze it to assess damage, but it must remain online.

The reason I call this a "worst case" scenario is that XP SP2 has many vulnerabilities which make it relatively simple for an attacker to gain administrative privileges (in fact SYSTEM-level privileges) on the host, thereby allowing malware to go after your credentials with full system authority. The reason for calling it "realistic" is that most organizations still have XP systems on their network and there are almost always a handful of systems that fail to get patched properly for one reason or another.

Now in order to assess and triage this potentially compromised machine, we need the ability to do several tasks:

• Query the machine, with a tool such as WMIC:
  There are plenty of things we might want to query from the system, such as running processes, installed services, logged on users, patch levels, etc.. Luckily we have a great tool like WMIC to help us with this task. For more on WMIC, check out one of several great articles from Ed Skoudis regarding this tool.
• Run processes on the machine, with a tool like PsExec:
  PsExec is a Microsoft Sysinternals tool that provides a very effective way to run tools on a remote machine. For this reason, it's very popular in our line of work and so I want to make sure to cover it. That said, it's important for me to mention that you should always avoid PsExec with alternate credentials, in which you specify the "-u" option to the command. I'll discuss the alternate credentials risks later in this article, as well as a follow-up article dedicated to PsExec.
• Copy files to & from the machine, via the command NET USE:
  Finally, we need a way to move files on and off the target system, so why not map a drive to the remote machine with NET USE?

The Requirements

In order to do effective IR, we must have administrative access to the compromised machine. Without it, we will almost certainly miss essential signs of infection. There would be no way to effectively capture memory images, disk images, fully query the machine processes, network connections, and so on, and therefore the tools mentioned above would be of little use.

So with that said, let me say this?.in all situations I can think of, the safest bet is to utilize a local administrative account if you already know the password.

Assuming you don't know a local administrative account and therefore you need to use a privileged domain account to access the target machine, we must take precautions to ensure an attacker cannot steal those domain credentials. But how? In a Windows domain architecture, there are 4 primary areas we need to protect:

• Password Hashes: Method for storing credentials on the local system
• Access Tokens: Single sign-on "-ish" functionality within Windows
• LSA Encrypted Memory: Another single-sign functionality within Windows
• Network Authentication: Protocols for authenticating to remote systems

The rest of this article will discuss password hashes and how we can protect our domain accounts against hash-stealing threats. I will cover the other areas in follow-up articles.

Password Hashes

The process of creating, storing, and authenticating password hashes provides computers a mechanism for authenticating an account using a cryptographic representation of a password rather than the actual plain-text password itself. This has obvious security benefits in that the plain-text password for an account will not be stored on the system. However, the strength of the password hashing algorithm has a great deal to do with its effectiveness. Unfortunately, Windows has been notorious for its use of weak password hashing algorithms.

For our scenario, we are concerned with protecting domain account credentials (as opposed to local account credentials). In this case, the domain account passwords that are typed into a Windows host are stored in three forms: as an LM (LanMan) hash, as an NT hash, and as a cached-credentials hash. In each case, the hashes are derived from known algorithms and they produce a relatively unique representation of your actual password. I say relatively unique because one of the problems with both LM and NT hashes is that they are not salted. A "salt" adds a unique data value to the password before passing it through the hashing algorithm so that, for example, my password of "abc123" and your password of "abc123" do not produce the exact same hash value. This lack of a salt is a huge problem for Windows because it has allowed pre-computed Rainbow Table attacks to be very effective in cracking password hashes?particularly LM hashes due to additional weaknesses in the algorithm.

These password hashes are what Windows stores and uses to authenticate accounts. So in actuality, the password hashes need to be protected every bit as much as the actual passwords. If they are not protected, there are very effective attacks that can either lead to the user's actual password being revealed (via offline cracking techniques, such as Rainbow Table attacks), or in the case of LM and NT hashes, attacks that simply allow the attacker to use the stolen password hash itself to directly authenticate to remote network resources. The latter attack is the "pass-the-hash" attack, which can be devastating to an environment because it allows attackers to easily move around the network as an authenticated account, using only the password hash for successful authentication.

Cached-credentials hashes, on the other hand, are a modified form of the NT hash and provide us a little more protection because they include a salt (the salt is the username, as described here for pre-Vista and here for Vista and higher). They are used to authenticate domain users to a local system when a domain controller is not available. Because they are generated with a salt, they are fortunately not useful in a pass-the-hash attack. However, tools like Cain and Abel and John the Ripper can crack them in an offline attack, so we want to verify whether or not cached-credentials will be available to the attacker.

I refer any readers who would like more background on the use and risk of password hashes to the following paper from the US Department of Energy:

• CIRCTech08-002: Understanding Windows Hash Dumpers and Crackers

This article is a bit rough around the edges, but I recommend it because it is fairly short and hits most of the relevant points.

If you want to go deeper into this subject, I also recommend two excellent papers from the SANS Reading Room:

• Why Crack When You Can Pass the Hash? by Chris Hummel
• Pass-the-hash attacks: Tools and Mitigation by Bashar Ewaida

Ok, so we've laid some groundwork. For the remainder of this article, I want to zero in on exactly what responders need to know to mitigate the risk of divulging their password hashes and then show some testing that proves this out.

Requirements for an Attacker to Obtain a Domain Account's Hashes

It's important to understand exactly what makes domain account password hashes available on a system and how they can be attacked when present. Both of the following must be true in order for the attacker to obtain domain password hashes:

1. The hashes must be on the compromised system. For local user accounts, this will always be true because local account hashes are stored in the local SAM database. In a domain environment, all domain account hashes will be stored in Active Directory on the domain controllers. For member servers and workstations (in other words, non-domain controllers), a domain account's password hashes can only be stored on systems where the user has performed an interactive logon. More on interactive logons in a moment. When an interactive logon occurs, the domain account's password hash will be stored on disk in the form of a cached-credentials hash. This makes sense because cached-credentials are used to authenticate a domain account when the domain controller is unavailable and therefore must be stored on disk to survive reboots. In addition, the native LM and NT hashes for a domain account will be stored in memory while the domain account is actively logged on. Once a user logs off, they are flushed from memory.
2. The attacker must have administrative rights on the compromised system. There are a number of attack tools that will retrieve the domain password hashes, but all require administrative rights to be acquired from a running system. Some tools need to be able to elevate to local SYSTEM privileges in order to access protected Registry files, while others work by injecting code into a running process via "SeDebugPrivilege", which by default is granted only to administrators. That said, this requirement turns out to be a pretty low hurdle, particularly for XP systems. There are a number of privilege escalation attacks that allow an attacker to get local administrative access, (see Metepreter's "priv" module for example), so it's by no means a stretch for the attacker to gain administrator access even if the original victim's user account is not a local administrator on the system.

In my opinion, the requirement for an interactive logon for password hashes to be cached locally is not widely documented (or at least not widely publicized) and therefore may lead to some confusion. That said, it is documented by Microsoft here and here. The corollary is even more obscure?i.e., the fact that a network logon will not result in domain hashes being stored on the remote machine. This is a very important fact and one that I will demonstrate shortly. One great resource which does document this fact is Hernan Ochoa's paper describing his tool Windows Credentials Editor (see page 25 of his paper).

Before getting to the testing, let's talk about these two types of user-initiated logons?interactive and network:

• Network Logon: A network logon is typically generated by accessing a remote system for file and printer sharing and/or interprocess communications. As we'll see shortly, examples include mapping network drives as well as querying the machine with WMIC.
• Interactive Logon: An interactive logon usually results in a logon session that presents a desktop to the user. This can be done physically by interacting with the system at the console, or virtually over the network with a remote session using Terminal Services (aka Remote Desktop) or third-party tools such as Citrix, VNC, RAdmin, DameWare, etc. In addition, there are ways to initiate an interactive logon without getting a graphical desktop. For example, the RunAs command from Microsoft, which allows you to run a particular command or application as a different user, also results in an interactive logon. We'll see shortly that Windows categorize this as a "secondary logon", but an interactive logon nonetheless.

So What Does This Mean for Us?

This means that for a responder's domain password hashes to be stored in memory or on disk, and thus available to an attacker, the responder must perform an interactive logon to the compromised machine. On the other hand, if the responder does not logon this way, and instead uses a network logon, the responder's hashes will not be at risk.

To prove this out, let's look at a few tests?.

Test Setup

For this testing, I setup a Windows 2003 Server acting as a domain controller for the "MSAD2" domain and 2 XP machines joined to the domain.

• "USER-XP-PC" is playing the role of the compromised host, running Windows XP Service Pack 2. The user account named "MSAD2-USER1" is the primary user of this machine and is a member of the MSAD2 domain.
• "IR-XP-PC" is playing the role of the responder's machine, used by incident responder "MSAD2-RESPONDER1", which is a domain admin account in the MSAD2 domain.

Testing: When Are Hashes Exposed?

In the testing below, I'm going to show local interactive logons by our victim accounts, remote network logons by our responder account, and then the results of dumping hashes on the victim machine.

I've used Windows Event Logs to demonstrate logon types. What you need to know is that Type 2 is an "Interactive Logon" and Type 3 is a "Network Logon". The logon types are described here. There is also a nice reference card available here.

• Domain user "MSAD2-USER1" issues CTRL-ALT-DEL logon sequence on workstation "USER-XP-PC" in order to logon locally at the console. As the Event Logs show here, this results in an interactive logon to the domain (Logon Type 2) for the user:

Now let's see what happens with a RunAs command, which as previously mentioned, is an interactive logon.

In this test, I opened a CMD window on "USER-XP-PC" and opened a new CMD window as user "MSAD2-USER2" via RunAs:

• Here are Event Logs showing the RunAs process starting:

• 3 log entries later, you see the interactive logon (Type 2) process for "MSAD2-USER2":

Notice in the above screenshot that the logon process is "seclogon". This is a secondary logon, as described in this Microsoft TechNet article. This is actually a service that runs automatically, as shown here:

So now that we have 2 domain users interactively logged on to the compromised host, let's remotely use our tools with our IR account. From workstation "IR-XP-PC", our incident responder "MSAD2-RESPONDER1" now logs onto the remote machine with 3 tools: NET USE, WMIC, and PsExec:

Note that I am not using the PsExec alternate credentials option here. With this feature, you specify an alternate username "-u" switch to use to connect to the remote machine. It turns out that when using PsExec in this way, it actually performs two logons?first a network logon to initiate the PsExec service and then a second interactive logon. One reason for the second interactive logon is to provide "delegation", so that the user can have full logon credentials to enable logon to a third machine (i.e., one hop beyond the PC you're currently logging onto remotely). In addition, PsExec passes the alternate account's username & password over the network in the clear. So whatever you do, do not give PsExec alternate credentials with the "-u" feature. I'll follow up soon with an article describing PsExec in more detail. I'll also discuss delegation in much more detail with a couple of articles on access tokens.

Also, when I discussed the 3 things we'd like to be able to do against the remote machine (copy files, run queries, and run processes), I said that I'd use WMIC to run queries, such as finding running processes or logged on users. For this testing, however, I wanted to perform a task that would not be transient and thus would stay active while trying to steal hashes. Therefore, I chose to use the "process call create" feature of WMIC to run a continuous ping in the background. If it is safe to initiate a continuous process, it should be safe to run a simple query (and in fact, I've tested the simple query scenario with "wmic /node:[remote-PC] computersystem get username" and the results are the same as what I'll show below).

So what was the result of running these commands against our compromised host? Take a look at the following Windows Event Logs:

The NET USE command created the following network logon (Type 3):

WMIC produced the following events, where "MSAD2-RESPONDER1" creates a network logon (Type 3) and then the WMI Provider Service process (wmiprvse) is initiated, which kicks off ping.exe with process ID 1996:

• Finally, PsExec creates a number of logs. first a network logon and then a series of log entries due to the start of the PsExec service and its launch of the requested command (cmd.exe in this case):

So now that we have domain victims logged on interactively and a domain responder connected remotely via network logon connections, let's see about dumping the hashes on this compromised machine.

There are numerous tools available that can dump local account hashes from the SAM database, such as pwdump, samdump, and Metasploit's hashdump module. However, we are not interested in these local account hashes. 

The tools that we use must be able to access the domain accounts' LM/NT hashes in memory and the cached-credentials hashes on disk. The two I'll test are:

• Windows Credentials Editor (WCE), which can grab password hashes from memory for current interactive logon sessions. It is a very powerful and versatile tool written by Hernan Ochoa and is the successor to his tool Pass-the-Hash Toolkit. Gsecdump is another tool that obtain these hashes from memory.
• Cachedump, which will extract the cached-credentials hashes of the domain users from the Registry. There are other tools in this category, such as creddump and Cain and Abel. To reiterate, a cached credentials hash is a modified form of the NT hash that is generated with a salt. It cannot be used in pass-the-hash attacks, but can be cracked offline, particularly if the password is weak.

By the way, if you care to test any of these or other password hash-dumping tools, you can download many of them from the Openwall Project. Most tools include README files that describe what they collect and how they work. Some of these tools are hard to find, so this is a useful resource.

Here are the results of dumping hashes on our compromised machine, USER-XP-PC:

The screenshot above shows that MSAD2-RESPONDER1 remained logged on remotely with NET USE and that the remotely-initiated WMIC process continued in the background. Though I was not able to show it in the screenshot, the remote PsExec logon was also still active while dumping the hashes.

The results show the following:

• Domain users "MSAD2-USER1" & "MSAD2-USER2", who were interactively logged on, had their LM and NT hashes available in memory and their cached-credentials hashes available on disk.
• Domain user "MSAD2-USER4", who had logged on some time previously but was not currently logged on, had his cached-credentials hash available on disk. LM/NT hashes were not available because the user was not actively logged on.
• The local Administrator account for USER-XP-PC had LM and NT hashes in memory because I had a RunAs command window open for this user. A cached-credentials hash was not available for this account because it is not a domain account.
• Most significantly, domain user "MSAD2-RESPONDER1", who connected remotely via network logons, had neither LM/NT hashes in memory nor a cached-credential hash on disk.

The tests above show the password hashes that can and will be left on a Windows machine based on an interactive logon session. While it can be damaging enough to have this information available for standard user accounts, it could be totally devastating to have a privileged domain account's hashes available to the attacker.

Recommendations for Protecting Password Hashes

Network logons created by tools such as NET USE, WMIC, and PsExec (without "-u" alternate credentials) are safe to use with regard to protecting password hashes. More to the point, any tool that uses only a network logon will prevent your password hashes from being available on the remote machine. To check other tools on your own, simply run the tool of interest against a remote test machine and verify in the remote Windows Event Logs that the tool only results in a Type 3 network logon.

On the other hand, you must avoid interactive logons with privileged domain accounts to prevent their password hashes from being generated and stored on the compromised machine. Interactive logons include any logon that presents you with a desktop, such logging on at the console or remote connections with Remote Desktop, Citrix, Dameware, VNC, or the like. It also includes RunAs and even PsExec with the "-u" alternate credentials feature. Any of these interactive logons to a compromised machine with privileged credentials could be devastating to a domain environment.

Also keep in mind, whether the compromised host is down the hall or on the other side of the world, using remote network logon connections is the only way to safely access the host with domain credentials. There are a number of triage tools designed for local interaction with the machine. I'm thinking particularly of those that run off a USB thumbdrive, for instance. In these situations, take care that you do not use a highly privileged domain account to logon to the machine locally (via CTRL-ALT-DEL or RunAs), or else those privileged account hashes will be at significant risk.

Much more to come on the subject of IR and domain authentication. Please stay tuned!

Mike Pilkington is a Sr. Security Analyst and Incident Responder for a global Fortune 500 company in Houston, TX, as well as a SANS Mentor and Community Instructor.