[Editor's Note: Josh Wright has some excellent recommendations and tips in this article about integrating fuzzing tools and techniques into your penetration testing engagements and ethical hacking work. Whether you are new to fuzzing or a tried-and-true expert, Josh has some great ideas for getting the most out of your pen test projects using the wonderful Sulley tool and some custom Python scripts he has created. This article was originally included in the September 2011 issue of PenTest Magazine, which has graciously granted us permission to feature it here. There are lots of cool ideas for you to check out. -Ed.]
Protocol fuzzing has been a popular technique for bug discovery with a number of tools, books, and papers describing the benefits and drawbacks. Although typically used for bug discovery in a lab environment, there are opportunities to use fuzzing in a penetration testing role too. Not only does fuzzing in a penetration test give your customer added value in an assessment but it also services to expand your skill set and develop new opportunities for attacking and exploiting systems.
Background
Not long ago, I drew the short straw on my team when divvying up responsibilities for a penetration test. While other people on the team got to target wireless flaws, web application flaws, and the like, I got the external-facing network pen test task.
You may have heard the drill before. The scope for my test included:
- Externally accessible target systems
- No client-side exploits
- No web app attacks
- No external resource management attacks (such as domain registration manipulation)
- No account enumeration/password guessing attacks
Try as we might convince people otherwise, some organizations still request this type of testing and hamper our typical approach to a pen test. Sometimes this is because they really just want a pat on the back for doing a good job securing their externally-facing systems; in this particular case, it was an audit requirement to have this level of testing done while other people on my team got all the fun on the other tasks.
As expected, my reconnaissance and network scans didn't turn up a lot of interesting flaws, but I did have one interesting target show up in the result of an Nmap scan as shown in Figure 1. My target organization leveraged a CheckPoint server for remote VPN access, of which two ports were accessible.
The CheckPoint service running on TCP/264 piqued my interest and after a little additional reconnaissance analysis I learned that it was used as an out-of-band communication channel by the CheckPoint VPN client over a proprietary, unencrypted protocol called SecuRemote.
The SecuRemote protocol is used to exchange information with the CheckPoint client, disclosing certificate information and, following authentication from the client, network topology details. A few older information disclosure vulnerabilities have been announced with public exploits, but the version my customer was using was not vulnerable to those issues. However, at this point I had some protocol documentation, a reference source, and a fair amount of time remaining in my allotted analysis hours, with which I reasoned that I could spend it doing some interesting protocol fuzzing.
The Role of Fuzzing
In my classes with the SANS Institute, I talk about how fuzzing can be used for various analysis tasks by different groups including developers, QA engineers, penetration testers, and more. Typically, I would tell my students that protocol fuzzing is not an attack, but rather is a method we can use to identify vulnerabilities in a target system. However, I'm going to change my story a bit here.
When using protocol fuzzing, we typically target a well-behaving piece of software, either a network service, client-side software reading from a malformed file, or even, in some esoteric cases, complex hardware systems. By sending malformed data to the target, we can evaluate the methods used by the developer to validate input data, potentially identifying crash conditions that could lead to an exploitable vulnerability. In order to characterize a crash condition accurately, however, we typically need to attach a debugger to the target process to capture stack and register dump information, limiting fuzzing to a local analysis technique and not something we would typically use in a remote penetration test.
Not having a lot of options remaining for my external network analysis, I opted to talk to my customer about performing a remote fuzzing analysis against their CheckPoint VPN service, targeting the SecuRemote protocol. My arguments were four-fold:
- We could replicate local fuzzing against their remote system and still obtain crashdump analysis information through the integrated CheckPoint technical support data collection vector.
- We could perform our analysis during off-hours (between midnight and 5:00 AM for the customer) to limit the impact of any system outages.
- Since the customer had to expose this service for CheckPoint VPN client users, there was value in evaluating it for flaws as a potential exploitation vector.
- The analysis could give the customer some visibility into the threat of zero-day vulnerabilities within the scope of their imposed engagement rules.
After coordinating with their internal support and networking teams, my customer was happy to indulge us in this analysis step. Next, I had to whip up a series of test cases to evaluate the target service.
Protocol Analysis
Prior to engaging in a fuzzing test, it is important to understand the portions of the protocol you will be evaluating. Since the SecuRemote protocol is not documented by CheckPoint, and to my knowledge has not been publicly reverse-engineered, I needed to dig into some packet captures for my analysis.
A quick Google search turned up several sample packet captures posted by other organizations while searching for assistance in troubleshooting or associated with prior vulnerabilities no longer an issue for my target. Looking through these packet captures, I was able to glean some details as to the working of the SecuRemote protocol, shown in Figure 2.
After the 3-way handshake, the CheckPoint VPN client sends two small packets with 4-byte payloads. This packet content appears to be consistent for a given CheckPoint server and client, though some variations in the leading byte of the first packet were noted, possibly indicating a version indicator of some kind.
After the second packet from the client, the server returns a 4-byte packet payload, similar to the first packet from the client with one additional set bit.
The fourth packet sends the string "securemote" to the VPN server, with four leading bytes and one NULL termination byte at the end of the string. The leading bytes likely indicate a 4-byte big-endian length value, since 0x0b correlates to the length of the string with the addition of the string terminator.
After the fourth packet, the server responds with the familiar leading 4-byte length indicator, followed by a string that appears to disclose human-readable certificate information revealing the server common name (CN=) and organization name (O=) .
Additional data is sent from the server, likely the continuation of certification information, followed by several instances of "none\x00" with a leading 5-byte length indicator sent by the client. One additional frame of interest is shown in Figure 3, generated with an exploit tool published to retrieve internal network topology information from earlier versions of CheckPoint VPN servers at http://www.securiteam.com/securitynews/5HP0D2A4UC.html.
In this example we see a fair amount of string information, possibly a static hash value, a configuration verb string "topology-requesz", and a lot of unknown fields. Again here we can spot some length fields followed by corresponding data, such as the \x00\x00\x00\x04 on the first line followed by the 4-byte value ending "\xc4\x1e\x43\x52". This pattern appears to repeat at the beginning of the next line with the length value "\x00\x00\x00\x4e" followed by 78 bytes of data prior to a trailing NULL byte.
While there is no simple way we can identify the meaning behind some of the bitwise fields, we can identify several interesting fuzzing targets from this analysis:
- Four-byte length fields: anytime you identify a length indicator in a protocol, fuzz it. Any assumed limitations on user-supplied data should be tested with length fields of various quantities.
- Several static bit-wise fields consisting mostly of \x00 values, giving us the opportunity to limit our fuzzing while enumerating alternate bitwise combinations.
- Regular use of NULL terminators at the end of strings. Though the recipient parser does not have to rely on NULL terminators since field length values are also used, it's a good idea to see what happens to a parser when terminator fields are no longer present.
With this knowledge, I started to prepare my test cases to deliver to the target system.
Tool Selection
In examining the protocol and selecting interesting fields for analysis with our fuzzer, I've steered my analysis technique toward the use of an intelligent mutation fuzzer. While other fuzzing techniques are certainly an option, such as randomized fuzzing and simple mutation fuzzers such as TaoF, mutation fuzzing gives us a good balance of control over what is being evaluated, a finite analysis runtime, and rapid test case development, all essential components for an analyst using fuzzing in a penetration test environment.
My preference in an intelligent mutation fuzzer is Sulley, the Python framework written by Pedram Amini. Sadly, Pedram has not continued to maintain Sulley, but with minor fixes we can still make use of this framework using recent Python 2.X releases. An alternative option for a mutation fuzzer is the Peach framework, though I prefer the Python syntax over XML development leveraged by Peach, plus we'll take advantage of native Python code to enhance our fuzzing test cases.
To install Sulley, check out the source from the project site using the Subversion tool, as shown in Figure 4.
With Sulley, we use basic data type constructs to describe how a packet is put together, identifying the presence of string-based fields, numeric fields of various widths, delimiters, and static values. All specified fields are iteratively mutated with intelligent mutation values (with the exception of static fields, which are not mutated), allowing you to test for target flaws. Sulley also gives us some advanced functionality to describe complex protocols that involve checksums and complex data elements that are best described in the project documentation.
Figure 5 is an embellished Sulley fuzzer targeting the initial 4-byte packet payload from the client. This script uses a few Python embellishments to estimate runtime to give me a chance to kill the fuzzer before it starts with CTRL+C. In this script, I used Sulley's one-byte numerical field identifier s_byte() to target the leading byte of the 4-byte payload, followed by a static definition of the three NULL bytes. I added the "full_range=True" modifier here to tell Sulley to iterate through all 256 values instead of catching the border cases of low and high values as well as common divisors (values divisible by 3, 4, 32, etc.) Copy this script to your Sulley directory and invoke it to start the fuzzer.
#!/usr/bin/env python
# Fuzzing the initial 4-byte packet from client to CheckPoint VPN server.
import time
import sys
from sulley import *
# Time to wait between mutations
SLEEP_TIME=0.5
# Time to wait before claiming a host is unresponsive
TIMEOUT=3
# number of crashes to observe before skipping the remainder of a group
CRASH_THRESHOLD=3
# Initialize the Sulley mutation descriptor
s_initialize("SecuRemote-Initial-Packet")
s_byte("\x51",full_range=True)
s_static("\x00\x00\x00")
print "Total mutations: " + str(s_num_mutations()) + "\n"
print "Minimum time for execution: " + str(round(((s_num_mutations() * (SLEEP_TIME))/3600),2)) + " hours."
print "Press CTRL/C to cancel in ",
for i in range(5):
print str(5 - i) + " ",
sys.stdout.flush()
time.sleep(1)
# For debugging purposes, uncomment these lines to see Sulley's mutations
# in hex dump format
#print "Hex dump mutation output:"
#while s_mutate():
# print s_hex_dump(s_render())
sess = sessions.session(session_filename="SecuRemote-Initial-Packet.sess", sleep_time=SLEEP_TIME, timeout=TIMEOUT, crash_threshold=CRASH_THRESHOLD)
# Tie this session to the SecuRemote-Simple-String fuzzing cases
sess.connect(s_get("SecuRemote-Initial-Packet"))
# Change this IP address to the target system
target = sessions.target("127.0.0.1", 264)
# Add the target to the session (can be repeated for multiple targets)
sess.add_target(target)
# Kick off the fuzzer, monitoring with WebUI on localhost:26000
sess.fuzz()
Figure 5. Embellished Sulley Fuzzer Targeting Initial Client Packet
Unfortunately, this initial run didn't uncover any interesting crashes against the target, though it did periodically generate varying responses from the server. In order to more thoroughly evaluate the target system, the fuzzer needed to accommodate some target protocol state.
Fuzzing Stateful Protocols
When designing a fuzzer, if the first packet in the protocol exchange is all you are targeting then you will miss some interesting vulnerability opportunities. Many fuzzers do not accommodate the more complex subsequent packets in a protocol exchange since they require the developer to accommodate some stateful awareness in the fuzzer that can manipulate the target with well-formed packets prior to sending the malformed content.
In the case of the SecuRemote protocol, the first two packets from the client to the VPN server were fairly uninteresting. The fourth packet in the exchange from the client, however, includes an interesting string and length indicator followed by a NULL byte terminator. In order to evaluate how the VPN server reacts to malformed data in this step of the protocol exchange, we need to complete the earlier two 4-byte packets, and ensure we obtain the expected response from the VPN server in step 3.
Fortunately, Sulley gives us a simple setup option that we can extend with a little Python scripting. Prior to delivering each mutated test case, Sulley will check for a callback function defined in the session with the name "pre_send". In this callback function, Sulley passes the socket handle for the mutated session (immediately following the three-way handshake). By defining our own Python callback function, we can read and write to this socket as needed to send the initial two 4-byte packets, and ensure we get a response from the server before allowing Sulley to send the mutated packet.
The data we are manipulating in packet 4 of the exchange adds a bit of complexity in the design of our fuzzer. We want to have Sulley generate large and small strings where the "securemote" string normally is, but we need to preserve the prior length field to accurately reflect the length of our mutated string value. We could place a fixed value in the length field regardless of the length of the mutated string, but this might be rejected with basic error validation procedures on the server, preventing us from reaching code that could otherwise trigger a fault on the target system.
Sulley accommodates the ability to include length fields with the s_size() construct. Using Sulley, we define a Python block beginning with an empty "if" statement and the Sulley s_block_start() construct. All the fields that are accumulated to reflect the length of the s_size() field are included in the block, followed by s_block_end(). When defining s_block_start(), we name the block of fields, and reference it in the s_size() construct prior to or after the block definition, as needed.
In our case, we need to reference the mutated string length and the NULL terminator field in our s_size() block, as shown in 6. This example is more of a minimal Sulley script, for brevity, but can be combined with the extra embellishments of Figure 5 if desired.
#!/usr/bin/env python
from sulley import *
SLEEP_TIME=0.5
TIMEOUT=3
CRASH_THRESHOLD=3
# The function Sulley will run prior to sending each mutation. We leverage
# it to setup the target system with the initial packets and response in the
# protocol exchange prior to our target packet.
def preconn(sock):
sock.send("\x51\x00\x00\x00")
time.sleep(0.5)
sock.send("\x00\x00\x00\x21")
# Set a socket timeout on the recv so we aren't waiting indefinitely if
# the server crashed from a previous test case.
sock.settimeout(5)
response = sock.recv(4)
print "Setup response: ",
for i in response:
print "%02x" % ord(i),
print
s_initialize("SecuRemote-Simple-String")
# Create a size field, which is based on the content of the named block
# Sulley uses ">" to indicate big-endian values, "<" is little-endian
s_size("client-name-string", length=4, endian=">")
# This is the block of data used for filling in the s_size
if s_block_start("client-name-string"):
# "securemote" is the default string
s_string("securemote")
# constant null terminator
s_byte("\x00")
s_block_end()
sess = sessions.session(session_filename="Securemote-Simple-string.sess", sleep_time=SLEEP_TIME, timeout=TIMEOUT, crash_threshold=CRASH_THRESHOLD)
# Call preconn() before each mutation is sent to setup the target
sess.pre_send = preconn
sess.connect(s_get("SecuRemote-Simple-String"))
target = sessions.target("127.0.0.1", 264)
sess.add_target(target)
sess.fuzz()
Figure 6. Minimal Sulley Fuzzer Targeting the "securemote" Client Packet
Continuing our development, we can leverage a similar stateful manipulation technique to test the server's reaction to malformed network topology requests. The network topology request is a much more interesting data set from a fuzzing perspective, with several additional field manipulation opportunities. This script is available at http://www.willhackforsushi.com/code/securemote-ptm.py.
Value in Pen-Test Fuzzing
In our engagement, we ran the fuzzers against the CheckPoint VPN server over the course of several nights, monitoring the results with a combination of Sulley stdout redirection to the tee utility and the Simple Watcher Perl scipt (swatch.pl). Our analysis indicated that we did trigger a small number of minor DoS events on the VPN server which were automatically rectified by the CheckPoint server and a service automatic restart function.
When reporting the findings to our customer, they were pleased on two fronts:
- Our analysis allowed the customer's security team to reinforce their internal mantra: "nothing is perfect in security", helping them voice the continued need for defense-in-depth protections on their internal network.
- They had a simple mechanism with which to communicate the flaws to CheckPoint and to validate subsequent fixes.
The customer was happy with the analysis results, and I felt as though I was able to deliver more value to the customer than an external penetration test report that would otherwise have identified no vulnerabilities.
Conclusion
In this article, we've only touched on the functionality of Sulley, leveraging it in an atypical manner. When fuzzing a target process on a Windows box such as a web server or other socket listener, Sulley can attach to the target process to capture register and stack trace information, create packet captures for each mutation sent, and even remotely control a VMware system to revert a target process to a snapshot following a crash to reset the target environment. Combined with the ability to leverage Python to enhance my fuzzing routines, it fits my needs for everything from simple to complex tests that can be put together in short order.
As an IDS analyst, I would not expect to see a flurry of fuzzing activity targeting my externally-facing systems, unless the attacker were only pursuing a DoS attack opportunity. Fuzzing is still primarily a local analysis activity where flaws are evaluated with an eye toward exploit development. However, as penetration testers, we should not rule out the option of using fuzzing in a penetration test where it makes sense to do so.
If you find this kind of analysis useful, I want you to know that I will be teaching even more detailed fuzzing trick and techniques, along with various advanced network attack techniques in a SANS vLive offering of the SANS 660: Advanced Penetration Testing course in January and February 2012, taught webcast style live across the Internet. Steve Sims and Bryce Galbraith will also be teaching sections of this class focused on in-depth exploit writing and Windows domain attacks. I'm really excited to be teaching this course. Click here for more details.
Joshua Wright is a senior instructor with the SANS Institute. Josh's papers, tools and articles are available at www.willhackforsushi.com.
