[Editor's Note: Penetration testers and ethical hackers are increasingly being called upon to evaluate the security of cloud-based applications, services, and infrastructures. Such work is fraught with political and technical complexity. Chris Brenton, one of the smartest guys I know in the cloud and distributed computing space, has written a great article that can help penetration testers sort out the issues. He provides specific tips on testing Iaas, PaaS, and SaaS cloud deployments. The nicest thing about this article is that Chris doesn't assume the reader is a cloud expert, but, instead, he walks you through various cloud issues and focuses on what professional penetration testers really need to know. This article originally appeared in Pen Test Magazine several months ago. Chris offered it to the SANS Pen Test blog to help our readers deal with cloud computing pen test issues. He's also got a brand-new blog focused on cloud issues, which you can find at http://www.sans.org/cloud. Thanks, Chris! -Ed.]
Pen Testing in the Cloud
With the phenomenal growth of cloud computing, many of us are engaging clients where one or more aspects of their cloud deployment is considered in scope. Penetration testing a cloud deployment can make for tricky waters to navigate, due to its shared responsibility model. In this article we'll demystify the cloud, as well as provide tricks and tips for navigating those waters.
Shared Responsibility
Arguably one of the biggest disruptions that cloud computing brings to penetration testing is the concept of shared ownership. In the past, if an organization contracted you for services, they would typically own all of the components on their network. This would open up all layers of the OSI for potential testing. In a cloud environment, it is entirely possible that the contracting organization controls very few of those layers. This requires additional up front work to ensure your testing remains in scope and does not negatively impact any third parties.
Here are two terms you need to be familiar with:
- Provider - The entity that built the cloud deployment, and is offering metered service to one or more tenants.
- Tenant - The entity that is contracting the metered service from the provider.
One of the first points you need to clarify when determining scope is whether the organization is a cloud provider or tenant. Note in certain cases there may be multiple clouds where the organization acts as a provider for one, and a tenant for others.
Cloud Deployment Models
Depending on the deployment model, the provider and tenant may be part of the same organization, or they could be completely different companies. Obviously this is a point you will want to ensure is clarified before defining the scope of your testing. Potential models include:
- Private - Typically the cloud is deployed on a local LAN. The provider and tenant are part of the same organization, but not necessarily part of the same workgroup or division.
- Public - Typically the provider offers compute, storage, network, etc. as a service that is consumed by the tenant on a pay for use basis. In this deployment model the provider and tenant are almost always part of different organizations.
- Community - Possibly a private or public deployment, a community cloud is managed and consumed by multiple entities that share a similar business model. For example a government could choose to set up a community cloud that is then consumed by different government agencies.
- Hybrid - A deployment that combines aspects of two or more of the above deployment models, as well as possibly including multiple service models (covered in detail later).
Identifying the deployment model is crucial to defining scope. For example if you will be testing a private cloud deployment, it may be possible to do a full cloud stack assessment. If however it is a public deployment, you will need to understand the delineation point between provider and tenant. Note that even if it is a provider that is contracting your services, there will most likely be cloud layers that are out of scope. A public provider that is identified as breaking into the tenant's domain without permission is going to quickly lose customers. You as a penetration tester should avoid such potentially difficult political situations.
Cloud Service Models
For each of the above described deployment models, different service models can be used to deploy resources. The service model defines which resources the provider brings to the table, as well as which resources the tenant will be responsible for supplying. Per the NIST SP800-145 definition, here are the three cloud service models:
- Infrastructure as a Service (IaaS) - The provider supplies hardware and network connectivity. The tenant is responsible for the virtual machine and everything that runs within it.
- Platform as a Service (PaaS) - The tenant supplies the application they wish to deploy, and the provider supplies all the components required to run the application.
- Software as a Service (SaaS) - The provider supplies the application and all the components required to run it. SaaS is designed to be a turnkey solution for the tenant.
Just from this brief description you can see that the service model will directly impact the scope of our testing. For example if we are being contracted by a tenant, an IaaS service model would require the most testing. Little to no testing would be possible for the tenant if a SaaS based service model is in use.
The Cloud Stack
Let's look at the different layers that go into building a cloud solution. We can then map these layers against each of the above service models to identify scope of testing for each. The layers or components that go into building a cloud solution are:
- Facility - The actual physical building where the cloud solution will be located.
- Network - This can be a physical or virtual network. It is responsible for carrying communications between systems and possibly the Internet.
- Compute and Storage — The physical hardware that will supply CPU time as well as file storage.
- Hypervisor - This is an optional component. When virtualization is used to manage resources, the hypervisor is responsible for allocating resources to each virtual machine. It may also be leveraged for implementing security.
- Virtual Machine (VM) or Operating System (OS) — In a virtualized environment, the VM is the virtual container responsible for running the operating system. If no hypervisor is present, the operating system runs directly on the compute and storage hardware.
- Solution Stack - This is the programming language used to build and deploy applications. Good examples are .NET, Python, Ruby, Perl, etc.
- Application - The actual application being used by one or more tenants, or their customers.
- Application Program Interface (API) or Graphical User Interface (GUI) - The interface used by the tenant or their customers to interact with the application. The most common API is RESTful HTTP or HTTPS. The most common GUI is an HTTP or HTTPS based Web site.
When we combine all of these layers, we get a cloud solution. As penetration testers, we need to be able to identify which of these layers are in scope for our testing. As mentioned earlier, scope will be heavily driven by both deployment model and service model. Whether the organization contracting our services is a provider or tenant will also factor into the scope.
Delineation of Responsibility
Figure 1 shows the delineation of responsibility based on each of the three cloud service models. This figure gives us a nice reference to quickly identify which cloud layers could be considered in scope for our penetration testing. The red line identifies which layers fall under the responsibility of the tenant, and which fall under the provider.
For example, assume you are contracted to perform OS level penetration testing. During the initial negotiations, it is identified that your customer is a tenant in a public cloud. That customer would like to include these resources in the scope of the testing.
Given the above figure, this may only be possible if they are deployed in public IaaS. If they are using PaaS or SaaS, the OS or VM resources are controlled by another entity, and thus they would be considered out of scope. As an analogy, think of being hired to perform a network level penetration test and the customer wants to include their upstream ISP in the testing. Both of these situations could lead you into legal difficulty.
When performing penetration testing, each service model has its own idiosyncrasies. In the next few sections we'll take a deeper dive on what you can expect when testing each of these service models.
Infrastructure as a Service
As mentioned in the last section, a tenant within an IaaS deployment is responsible for the VM and everything within it. So if you are contracted by a tenant, the OS and all loaded applications are fair game provided third party penetration testing is acceptable under the provider's terms of use. Let's first talk about how servers are deployed within an IaaS cloud so we can focus in on some of the common weak points.
IaaS Cloning Issues
On a legacy network, operating systems are typically loaded manually. This means that it is not uncommon to find variations in both configuration and patch level. In IaaS deployments, we use the concept of a gold standard image. The image is a VM that has been created manually and tweaked with all required applications, patches, and configuration changes. When a new server is required, the gold standard image is cloned, and that clone is made available to the tenant. It is not uncommon for providers to offer up a catalog of servers available for cloning. For example there may be one standard image with a Web server preloaded, another with a database server, variations that use Linux or Windows for the underlying OS, and so on.
However consistency can be a double edged sword. On the plus side, if I've implemented a change that increases the security posture of my servers, I can be sure that this change will be implemented across my infrastructure. On the down side, it means that any insecure settings will also be propagated across my infrastructure. So when you are performing a penetration test, if you are lucky enough to find a point of entry, that vulnerability will most likely exist in every VM performing a similar role within the infrastructure. This provides opportunities for pivoting to other systems, just remember to always operate within your scope and rules of engagement.
IaaS Patching Problems
Nowhere is this more apparent than VMs in public space. With public IaaS clouds, most providers supply unpatched versions of the operating system. Further, remote administration is enabled so that the tenant is able to gain access to their VM. For Linux systems, this is usually SSH. For Windows systems, this is usually RDP. Needless to say an unpatched OS with remote administration enabled makes for a tempting target.
There are additional issues which can exaggerate this problem even further. To start, providers charge for network and CPU usage. This means that it costs money to patch your servers. A cost conscious tenant may choose to avoid these charges by skipping the patching process. Also, most providers do not enable automatic patching by default. This means that at the very least the tenant has to reconfigure the server to have it patch automatically. However in some environments this may not be an option. For example Microsoft requires that automatic patching be disabled on any VM role deployed within Azure. So each server will only get patched when the administrator logs in and manually initiates the process.
IaaS Traffic Control
Another possible issue is that there may be little to no firewalling limiting access to the VM. In a private IaaS deployment, the cloud will be behind the perimeter so this should be a non-issue. In a public cloud however, the VM may spin up unprotected. Even if packet level control is in use, the protection may be minimal. For example if you are using Amazon EC2 security groups, the basic service only permits security groups to be assigned at the time the VM is initiated. This can make it extremely difficult to change firewall policies while the VMs are in operation. The result is that some basic customers simply create a single security group that permits all traffic, so they do not run into complication later.
IaaS Targeting
One of the most difficult aspects of penetration testing an IaaS environment is determining target IP information. Remember that a cloud is typically shared space, so you cannot count on a tenant's IP address range as being continuous. In fact if they frequently bring servers up and down, it is not uncommon for the allocated IP addresses to change.
This is where things can get a bit tricky. If you are targeting a private IaaS deployment, determine whether the entire IP range is in scope. While IP addresses are usually only revealed during white box testing, exceptions can be made to ensure uninvolved third parties do not receive collateral probing. If the contracting organization has servers in public IaaS space, you may be able to enumerate their location via DNS by identifying IP ranges that are part of known hosting providers.
Another common trick is to tie the public server back to the local LAN via a VPN. This technique is shown in Figure 2. While this nullifies many of the benefits of public cloud such as geo-location services and load balancing, it does permit the organization to use their existing perimeter security to reduce the risks against the server in public space. The public cloud server is configured such that it will only communicate down the VPN tunnel.
Remember that the public cloud is a virtual environment. So while extending the LAN may protect the publically hosted server at the network level, the VM is still connected to a hypervisor and a VSwitch via software. So it may be possible to come at the VM from another vector besides the network. However this type of testing would require that the attack be directed against layers managed by the cloud provider, which could put them out of scope if you have been contracted by a tenant.
Canned Tools
We are just starting to see penetration testing tools that are designed to address the unique needs of a cloud environment. One such tool is Core CloudInspect for Amazon Web Services. This tool is similar to other product offerings by Core, but reconfigured to play nice in a public IaaS environment. Effectively this means that all of the networking tests have been stripped out to ensure that only the tenant layers are targeted during testing. At the time of this writing, the tool only supports Amazon AWS. It also integrates with your AWS account, so it is less useful for penetration testing or third party auditing. Still, it is a start in the right direction.
Attacking The Hypervisor
Most of the above advice on IaaS cloud assumes that you have been contracted by a tenant. What if you are being contracted by a provider? How does this change our scope?
Testing the provider obviously means that most of the VMs will be out of scope. It may be possible to have the provider launch a few specific images, if you feel they could be leveraged during your testing. For example you may wish to test for susceptibility to VM escape attacks.
If you are testing a provider however, one of the juicier targets is the hypervisor itself. Since the hypervisor ties all of the VMs together, ownership of the hypervisor could potentially lead to access on every VM. An architectural drawing, complete with a virtual appliance firewall, is shown in Figure 3.
One of the popular trends is to implement security via the hypervisor using a technique called introspection. With introspection, the hypervisor can potentially access both memory and disk within each VM. While this can help augment security, it can also create points of insecurity. One of the more interesting caveats is that introspection access to the VM's memory and disk bypasses local access controls and logging. So if I can leverage introspection to scrape data off of a VM, the VM itself will have no audit trail of the data being accessed. Whether the access is recorded at all will come down to controls within the hypervisor's host environment.
Platform as a Service
With Platform as a Service, the delineation point between provider and tenant moves up the stack. This places the application stack and everything below it in scope for provider testing only. If we have been contracted by a tenant, scope is limited to their application and any available interface. Beyond that, much of the advice above can potentially be leveraged for PaaS environments as well.
From a security perspective, public PaaS can put a tenant in a bit of a quandary. Many public PaaS providers prohibit penetration testing of client applications. This is because the testing can potentially impact other tenants as well as the provider. For example if you are able to break an application and gain high level system permissions, the system is actually owned by the provider, not the tenant, which obviously places it out of scope. So the quandary becomes, how do I deploy a secure application if the provider will not let me test security?
There are a few possible ways around this problem. One alternative is to focus on code review. This may require full white box testing, but at least it gives us something to fall back on. The other possibility is that some organizations run a private PaaS cloud for development, while leveraging a public PaaS cloud for production. If the organization contracting your services has a development environment, you may be able to leverage it for application testing.
Another interesting aspect of PaaS is that it changes the paradigm of application development. In the past, applications were developed by relatively few vendors and then released to the world en masse'. Each customer would receive a local copy of the software, and there would be thousands if not millions of copies of the software in distribution. Mass distribution enabled a large number of individuals to review the application for security. Any concerns could then be conveyed to the vendor or openly discussed through Bugtraq or other communication channels.
PaaS uses a far more limited distribution model. Most applications are being developed by the organization that plans to use them. So there are fewer copies of the code in existence, which means fewer eyes reviewing the code for security. Further, PaaS applications are typically written to live within only a few PaaS clouds. Since the cloud is owned by the provider, this decreases the opportunity for third parties to scrutinize the code.
How does this impact us as penetration testers? Because of this development process, it becomes far easier to find common mistakes. You may actually find it fruitful to try extremely common attack vectors such as SQL injection and cross-site scripting. Remember that a lot of inexperienced coders are moving into the PaaS development space. Be as complete as possible with your testing.
Software as a Service
Software as a Service is designed to be a turnkey solution for the tenant. So if it is a tenant that is contracting your services, there is very little testing that can be considered in scope. This may be limited to ensuring that interactions with the application interface are being properly secured, API keys are being managed appropriately, and so on. Tools such as Webscarab and Burp can be extremely helpful in analyzing this interaction. For the layers under the provider's control, your client may be able to audit their security posture.
If you are being contracted by a provider, most of the above advice for IaaS and PaaS will be applicable here as well. SaaS may actually reduce the number of individuals reviewing the code even further, so ensure you are comprehensive in your testing. Common coding mistakes such as the OWASP top 10 and the SANS top 25 coding errors may help to get the creative juices flowing.
While some SaaS deployments execute the application completely within the provider's cloud environment, some include a local agent that is designed to run on one of more tenant systems. The most common example is some SaaS based security solutions.
Again, there are many inexperienced coders moving into this space. Look for agents that do little to secure their own deployment. Is the agent simply a set of scripts that can be easily modified? Can the agent easily be turned against the local system? Does the solution have any way of verifying whether the agent has been compromised? You may be surprised by what you find.
Conclusion
Finally, regardless of whether the solution is IaaS, PaaS, or SaaS, remember that most cloud solutions have a backend management solution. Attacking this portion of the infrastructure would only be in scope if contracted by a provider. Many solutions function out of band. However if you can obtain access to resources within the cloud itself, it may be possible to go after the management system. Make sure you understand the underlying virtualization and cloud solution being used to help identify any potential points of insecurity.
