Having recently seen a number of requests on the security and forensic list servers that I participate in requesting recommendations / procedures for copying the disk (VMDK) for a specific Virtual Machine (VM) within a VMware environment for analysis in an incident response, I put together a quick How To in effort to provide some insight in to a few of the methods that I have used.
The Game Has Clearly Changed With Virtualization
Most often the files associated with a given VM are not stored locally on the physical server running ESX or ESXi and the respective VM. It is important to understand that in order to use many of the more powerful features of VMware such as vMotion and DRS the files for the VM's must reside on shared storage that is reachable from each ESX or ESXi server that needs to interact with it. Hence, when considering the acquisition of the files associated with a given VM you most often will not have the luxury of simply bringing down the physical server running ESX or ESXi and the respective VM and imaging the local hard drive as the files in question may not reside there. Further bringing down the server that is hosting the shared storage for the environment, removing the drives and using your hardware imager to copy the disk(s) will in all likelihood not be an option as there could be hundreds of other virtual machines sharing that same storage device for their files that simply cannot be taken down and must remain in production. As an example, in Figure 1 below we have 5 VM's - SRV01, SRV02, VM03, VM04 and FW01 all using the shared storage on LABVMFS01. Taking down the shared storage LABVMFS01 for traditional drive imaging is not an option as it would also bring down the associated VM's and they need to remain in production.
Using vSphere To Copy A VMDK
I typically run VMware vSphere (eval copy available here) on a Windows 2008 x64 VM to provide a portable management tool for the various client environments that I provide services for. Once I have administrative permission from the client to connect to their virtual environment I am able to literally plug right in, create a data center and simply add the specific ESX / ESXi hosts I will need to interact with (Figure 2). Naturally in order to connect to the hosts I will need to reside on the broadcast domain for the given network and I will need the administrative credentials for the given hosts that I will need to interact with. For the example shown in Figure 2 I had created a Data-center called Portable Lab and then simply connected the host ESX01 to my Data-center which then imported all of the VM's that are associated with ESX01 in to my copy of vSphere.
With ESX01 now manageable from my copy of vSphere it is simply a matter of selecting ESX01, opening the Maps tab and clicking on the Data-store "LABVMFS01". This brings up the Data-store view where I select "Browse this Data-store" which opens a Data-store Browser (Figure 3). Once in the Data-store Browser I select the VM SRV02 and the files contained for this VM are displayed (Figure 4). Now it is just a matter of "right clicking" on the VMDK file that I want to "download" from the Data-store and giving it a path to where I want the files downloaded to on my Windows 2008 VM that is running vSphere. With the VMDK for SRV02 now residing on my Windows 2008 VM I plug in a USB drive and connect it to my VM running Windows 2008 and vSphere. I keep a copy of FTK Imager on a large USB drive for acquisitions so now it is just a matter of and running FTK Imager (Figure 4) to create a forensically sound copy (Figure 5) of the VMDK file to the format desired (DD. SMART or E01) on the USB drive.
With the image of the VMDK available on the USB drive you now can use the tool of your choice such as the SANS SIFT Workstation where you can "right click" on the image in SIFT and mount it as a read-only local drive for examination or you can of course import the image in to FTK for analysis . You also have the option of using the commercial product Mount Image Pro to mount the image as if it were a local drive and then run your IR tools such as Gargoyle against the local drive.
For the purposes of illustration I loaded the image of the VMDK in to FTK 3.1 and fully processed it showing the respective directory tree below (Figure 6).
Using Veeam FastSCP To Copy A VMDK
A great alternative to using vSphere is to download, install and use the free Windows program Veeam FastSCP to copy the VMDK of the respective VM from the ESX/ESXi server. You will need connectivity to the network that hosts the ESX/ESXi server as well as the administrative credentials for the target ESX/ESXi host. You simply add the ESX/ESXi host as a server in FastSCP, enter the administrative credentials and it connects and provides an Explorer like interface for the files on the ESX/ESXi host (Figure 7). Copying is simply a matter of right clicking on the folder or file you want to copy and then providing a destination path such as a USB hard drive connected to the Windows PC running FastSCP. Another interesting feature of FastSCP is that it can also schedule the copy if you do not wish to do it immediately such as such as later during a low network traffic time period.
A Few Caveats To The GUI Approach In Copying A VMDK
While using GUI tools are perhaps "enough" for an incident response activity current GUI tools fall short of being able to be considered a forensically sound method for use in collecting evidence for use in a forensic analysis. Simply put using the GUI approach with current tools such as vSphere and FastSCP you are in fact not validating the forensic soundness of the copy of the VMDK via an MD5 or SHA hash of the VMDK before and after copying which effectively prevents the establishment of its chain of custody. Historically using the console in ESX has allowed you to easily validate the hash of the VMDK with tools like MD5SUM before and again after copying thereby facilitating the establishment of the respective chain of custody. It has been formerly announced that ESX will be going away - see ESX End of Life and is being replaced with ESXi. That creates a problem in that with ESXi to accomplish the forensic validation of the copy you are forced to use officially "unsupported" components as for all intents and purposes the console does not exist in ESXi. That being said, however the "unsupported" console uses BusyBox Linux which in fact does include the MD5SUM command. Hopefully VMware and/or third party vendors will address this issue quickly and provide the forensics community with tools that will produce a validated copy of the VMDK for use in forensic analysis from either ESX or ESXi using "supported" components that will validate the forensic soundness of the copy thereby facilitating the establishment of its chain of custody. Until then forensicators will have no choice but to work with the unsupported console in ESXi directly or via SSH (after enabling it) to use the dd and MD5SUM commands to properly create a forensically sound image of a given VMDK copied out to a NAS device (Still issues with connecting a USD drive directly to ESXi).
If you would like more information on the VMDK file structure a great resource is the VMDK Handbook — Basics that can be found at Sanbarrow.com
The author (while biased as he teaches it for SANS) highly recommends the SANS course - SEC 577 for infosec professionals that desire insight in to the new and rapidly expanding realm of Virtualization. See course description below:
SECURITY 577
One of today's most rapidly evolving and widely deployed technologies is server virtualization. Many organizations are already realizing the cost savings from implementing virtualized servers, and systems administrators love the ease of deployment and management for virtualized systems. There are even security benefits to virtualization - easier business continuity and disaster recovery, single points of control over multiple systems, role-based access, and additional auditing and logging capabilities for large infrastructures.
With these benefits comes a dark side, however. Virtualization technology is the focus of many new potential threats and exploits and presents new vulnerabilities that must be managed. In addition, there are a vast number of configuration options that security and system administrators need to understand, with an added layer of complexity that has to be managed by operations teams. Virtualization technologies also connect to network infrastructure and storage networks and require careful planning with regard to access controls, user permissions, and traditional security controls.
Attendees will learn about virtualization security fundamentals with an in-depth treatment of today's most pressing virtualization security concerns: known attacks and threats, theoretical attack methods, and numerous real-world examples. Then we'll turn our attention to today's most popular enterprise server virtualization product, VMware vSphere. Attendees will learn about every aspect of locking down ESX and ESXi servers and the vCenter management server, as well as best practices for securing the virtual machine guests that reside on ESX and ESXi platforms. We'll also cover virtualization networking techniques in detail, laying out proven strategies for proper segmentation, virtual switching and routing considerations, network access controls and layer 2 policies, as well as how to build virtual DMZs and integrate with existing network infrastructure. The latest vSphere technologies will be covered, including Distributed Virtual Switches, vShield Zones, and Host Profiles.
Finally, attendees will learn essential strategies for securing storage interfaces to vSphere, as well as best practices for backup, recovery, and redundancy. We'll then wrap up with extensive information about compliance ramifications from virtualization, strategies to create and maintain compliance-focused controls using VMware, and operations processes and concepts to focus on, such as change and configuration management, separation of duties, and least privilege.
- Virtualization Basics and Introduction
- Virtual Networking
- Virtual Switch Security Policies
- Command-line Virtual Network Configuration and Administration
- Virtual Network Architecture Design
- vCenter Security and Administration
- Virtual Infrastructure Client Security
- ESX and ESXi Security
- ESX File System Security
- VM Guest Security
- Storage Considerations
- Backup and Recovery
- Virtualization Risk Assessment
- Virtualization Threats
- Virtualization Vulnerabilities
- Virtualization Attacks
- Virtualization Audit and Compliance
- Who Should Attend
- Security personnel who are tasked with securing VMware virtualization technologies
- Network and systems administrators who need to understand how to architect, secure, and maintain virtualization technologies.
- Technical auditors and consultants who need to gain a deeper understanding of VMware virtualization from a security and compliance perspective
Paul A. Henry Forensics and Recovery.com. Follow me on Twitter
.png "Anastasia Sentsova")
