We used to rely on Nmap for all host enumeration, but with cloud networks there's too many potential systems to scan and effectively identify targets during a penetration test. Instead, I've adopted a set of steps that I follow when performing target discovery in a cloud provider's network. In this article we'll look at the step-by-step process of scanning a cloud provider's network for target enumeration.
First, two important notes:
- Always get permission: Check the Acceptable Use Policy for the cloud provider; you may need to get permission from the provider to perform the scan)
- Scan within the provider and region: When scanning, create the scanning system in the same cloud provider and region that you are scanning. If you are scanning multiple regions, then create multiple scanning systems. This will reduce cost and perform most optimally.
I'm going to use AWS and the us-east-1 region for my examples. Adjust as needed for your goals!
TIP: If you want to download a nicely-formatted version of this content you can grab the PDF here.
Thanks to Daehee Park! Much of this content is inspired by his excellent article How To Scan AWS's Entire IP Range to Recon SSL Certificates.
Cloud Scanning
Preparing an AWS Debian Image
I used a generic Debian image for scanning purposes. No fancy AMI, just a plain Debian install.
When the system boots up, SSH in, then prepare the system with these steps (you can cut-and-paste this into the terminal):
sudo apt update && sudo apt -y install masscan jq
wget https://github.com/prbinu/tls-scan/releases/download/1.4.8/tls-scan-1.4.8-linux-amd64.tar.gz
sudo tar xvfz tls-scan-1.4.8-linux-amd64.tar.gz -C /opt
rm tls-scan-1.4.8-linux-amd64.tar.gz
wget -O tlsscanhostnames.py https://urls.sec504.org/tlsscanhostnamesraw
chmod 755 tlsscanhostnames.py
Scan the Cloud Network
Follow these steps to scan the us-east-1 AWS region. These steps apply to other providers and regions as well; update the process below accordingly for your needs.
Download the IP Range List
Download the list of IP addresses for AWS. Use jq to retrieve only the IP prefix information for the us-east-1 range:
wget -qO- https://ip-ranges.amazonaws.com/ip-ranges.json | jq '.prefixes[] | if .region == "us-east-1" then .ip_prefix else empty end' -r | sort -u > us-east-1-range.txt
If you are scanning Google Cloud or Azure, you'll need to get their IP address list information.
Google: wget -qO- https://www.gstatic.com/ipranges/cloud.json | jq '.prefixes[] | .ipv4Prefix' -r
Azure: Download the JSON file first, then jq < ~/Downloads/ServiceTags_Public_*.json '.values | .[] | .properties.addressPrefixes | .[]' -r
The IP address list should look like this:
$ head -4 us-east-1-range.txt
100.24.0.0/13
104.255.56.11/32
104.255.56.12/32
107.20.0.0/14
Scan for TCP Port 443 with Masscan
Next, start a scan with Masscan for TCP port 443. Using a t2.micro EC2 instance I've been successful scanning at 100,000 packets per second, but you might want to increase this (especially for faster instances with more bandwidth) or lower for more reliability:
sudo masscan -iL us-east-1-range.txt -oL us-east-1-range.masscan -p 443 --rate 100000
We're only scanning for TCP port 443 here; you may optionally want to add additional ports for any other TLS-based service (IMAPS, SMTPS, PostgreSQL, MySQL, RDP, TCP/8443, etc.)
The output will look like this:
$ sudo masscan -iL us-east-1-range.txt -oL us-east-1-range.masscan -p 443 --rate 100000
Starting masscan 1.3.2 (http://bit.ly/14GZzcT) at 2023-01-04 15:37:05 GMT
Initiating SYN Stealth Scan
Scanning 787650 hosts [1 port/host]
rate: 32.97-kpps, 79.37% done, 0:00:04 remaining, found=4321
Cleanup Masscan Results
When the scan complete you will have a us-east-1-range.masscan file that records the systems listening on TCP/443:
$ head -4 us-east-1-range.masscan
#masscan
open tcp 443 107.22.83.44 1672846625
open tcp 443 107.23.215.252 1672846625
open tcp 443 100.24.81.235 1672846625
We want to convert this to just a list of IP addresses:
awk '/open/ {print $4}' us-east-1-range.masscan > us-east-1-range.tlsopenThe resulting file will look like this:
$ head -4 us-east-1-range.tlsopen
107.22.83.44
107.23.215.252
100.24.81.235
100.26.92.8
If you are scanning ports in addition to TCP/443, you'll need to specify the alt port following the IP address: awk '/open/ {print $4":"$3}' us-east-1-range.masscan > us-east-1-range.tlsopen.
Target Identity Enumeration
Next we have to figure out the identities associated with the scanned systems. Fortunately, the certificate information for each identified system will disclose some identity information, often the hostname of the server.
To retrieve the identity information from certificates we can use TLS-Scan.
This Takes A While. Running TLS-Scan is the most time-consuming part of the scanning process. Give it a few hours to complete if you are scanning millions of IP addresses in the cloud provider network. Optionally adjust the timeout from the default of 10 seconds to something smaller (I use -t 3 in the example). If you have lots of CPU, memory, and network capacity, you can maximize performance using -b to specify multiple worker objects. Additional information on optimizing TLS-Scan performance with examples is available in the project README file.
cat us-east-1-range.tlsopen | /opt/tls-scan/tls-scan --port=443 --cacert=/opt/tls-scan/ca-bundle.crt -o us-east-1-range-tlsinfo.json -t 3
The output will look like this:
$ cat us-east-1-range.tlsopen | /opt/tls-scan/tls-scan --port=443 --cacert=/opt/tls-scan/ca-bundle.crt -o us-east-1-range-tlsinfo.json -t 1
elapsed-time: 0 secs | status: 4/4 | tls-handshake: 4 | target: 100.26.92.8 host: 100.24.79.57; ip: ; error: Network; errormsg: Error encountered while reading
elapsed-time: 0 secs | status: 23/23 | tls-handshake: 22 | target: 100.24.89.139 host: 107.23.173.123; ip: ; port: 443; error: Timeout
elapsed-time: 1 secs | status: 39/39 | tls-handshake: 37 | target: 100.25.59.81 host: 107.23.171.226; ip: ; error: Network; errormsg: Error encountered while reading
The resulting file will be a JSON blob:
$ ls -l us-east-1-range-tlsinfo.json
-rw-r--r-- 1 admin admin 12743546 Jan 4 15:49 us-east-1-range-tlsinfo.json
Build Target Identity List
The JSON information discloses the TLS certificate details for each host including the certificate Common Name (CN). We can extract this data using JQ to make a CSV file:
cat us-east-1-range-tlsinfo.json | jq '[.ip, .certificateChain[].subjectCN]? | join(",")' -r > us-east-1-range-tlsinfo.csvYou may get malformed JSON in the us-east-1-range-tlsinfo.json due to a bug in TLS-Scan, producing this error:
$ cat us-east-1-range-tlsinfo.json | jq '[.ip, .certificateChain[].subjectCN]? | join(",")' -r > us-east-1-range-tlsinfo.csv
parse error: Expected another key-value pair at line 3545, column 441If this happens, remove the lines that are problematic with this grep command, producing a new JSON file:
grep -v 'ocspStapled": false, }$' us-east-1-range-tlsinfo.json > out.json
### Only do this step if you want to overwrite the original file
mv out.json us-east-1-range-tlsinfo.json
The output of the command will look like this:
$ head -4 us-east-1-range-tlsinfo.csv
107.22.83.44,*.unitedauto.cloverleafapp.com,Amazon,Amazon Root CA 1,Starfield Services Root Certificate Authority - G2
107.23.215.252,sym-ac8-dev-chat-glb-1.d.isym.io,Amazon,Amazon Root CA 1,Starfield Services Root Certificate Authority - G2
100.24.81.235,100.24.81.235,D2i WebAdmin CA
100.26.92.8,*.segurosbolivar.com,*.segurosbolivar.com
This JQ command is great for extracting name information from the Common Name field, but there are other name fields in the TLS certificate as well, such as the subject alt name. The tlsscanhostnames.py script will apply some more complex checks to extract out more host name information:
$ ./tlsscanhostnames.py us-east-1-range-tlsinfo.json | head -5
100.24.0.19:occedu.co
100.24.0.220:staging.k12.com
100.24.0.241:sew.timebyping.com
100.24.10.70:supabase.co
100.24.10.70:supabase.in
$ ./tlsscanhostnames.py us-east-1-range-tlsinfo.json > cloudtargets-attributed.txt
With the list of IP addresses and attribution information from the TLS certificate details, we can search for domain names, keywords, and other strings. For example, if I were authorized to test ring.com infrastructure, I might search for ring.com in the cloudtargets-attributed.txt file:
$ grep ring.com cloudtargets-attributed.txt
100.24.219.172:ring.com
107.20.38.8:ring.com
107.20.8.46:management.lightspeedordering.com
107.22.1.182:comms.avalonflooring.com
107.23.131.93:dev.emcscoring.com
Vulnerability Discovery
Once the cloud scanning and identity enumeration is complete, we can choose the list of targets and more on to vulnerability discovery. Much of the analysis process here is the same as what you would apply to on-premises target systems (using vulnerability scanners, full port enumeration with Nmap, etc.), but there are a few concepts and tips I want to point out.
Local DNS Manipulation
Often, the hosts identified during cloud scanning won't have public DNS records. For example, searching for Ring.com targets reveals multiple hosts (and some false-positive matches):
$ grep ring cloudtargets-attributed.txt | head -10
100.24.208.96:my.ringling.org
100.24.219.172:ring.com
100.24.250.92:acessos.flamengo.superingresso.com.br
100.24.77.132:720427789759.us-east-1.prod.rms.ring.devices.a2z.com
100.24.77.132:us-east-1.prod.rms.ring.devices.a2z.com
100.25.243.62:engineeringvillage.com
100.25.243.62:www.engineeringvillage.com
100.25.32.18:839035386434.us-east-1.release.rms.ring.devices.a2z.com
100.25.32.18:us-east-1.release.rms.ring.devices.a2z.com
107.20.171.248:984380019471.us-east-1.prod.rms.ring.devices.a2z.com
Here we learn about the host at 100.24.77.132 which has two names extracted from the certificate: 720427789759.us-east-1.prod.rms.ring.devices.a2z.com and us-east-1.prod.rms.ring.devices.a2z.com. However, neither of these names are in public DNS:
$ host 720427789759.us-east-1.prod.rms.ring.devices.a2z.com
$ host us-east-1.prod.rms.ring.devices.a2z.com
$
Adding the hostnames (revealed from the certificate details) and the corresponding IP address in the /etc/hosts file and accessing the target by the hostname may make a web server vhost accessible that would not be otherwise.
Scanning with EyeWitness
EyeWitness from Chris Truncer takes a list of host names or IP addresses and captures screenshots of the website, VNC, or RDP services on the targets. EyeWitness generates an HTML report as the output, which is easier to flip through to visualize the target websites.
$ python3 /opt/eyewitness/EyeWitness.py --web -f cloudtargets-attributed.txt --prepend-https
...
[*] Done! Report written in the /tmp/02212021_115954 folder!
Would you like to open the report now? [Y/n] Y
Summary
We can use a combination of Masscan, TLS-Scan, JQ (or tlsscanhostnames.py) to rapidly scan lots of cloud systems. Remember the rules: always get permission, and make sure you scan within the same cloud provider and region for optimal performance.
With the list of discovered targets and TLS certificate information, we can perform attribution to identify target systems. Sometimes we have to manually edit local DNS as well to get web server vhosts to work as intended, but with a little tweaking we can discover and attack cloud systems using familiar tools: Nmap, vulnerability assessment scanners, EyeWitness, and more.
Defense
There are some things you can do to defend against this type of scanning and enumeration attack.
None of these defense options are great. Don't get your hopes up.
Security Groups and NACLs
One option is to limit access to your cloud targets from unauthorized IP addresses using security groups and Network Access Control Lists. In AWS security groups have an implicit deny at the end of the list, so adding the permit rules will grant only access to designated sources for specified protocols.
Using security groups or NACLs are fine, but they don't help when you need to permit access from the internet to your server. Wherever possible, think about limiting access, such as ensure security groups or NACLs prevent access to all services, inbound and outbound, except those that are absolutely necessary.Don't Worry About Scanners
Here's my advice: don't worry about scanning.
For sites that needs to be publicly accessible, it's a risk we accept as part of doing business. Using the techniques in this article, it is really hard to even detect someone doing this scanning on a per-host basis, and we seldom have a broad view of what is happening on the cloud provider's network to detect the scanning across lots of different hosts.
Instead, think about the ways you can protect your systems from the attacks that happen after scanning. Conventional tips apply here, such as keeping systems patched and using effective threat hunting techniques to detect threats on-premises and in the cloud.
Questions?
Reach me on Twitter (@joswr1ght) or email josh@willhackforsushi.com.
Special thanks to the SANS Institute for allowing me to share this content. Check out my class SEC504: Hacker Tools, Techniques, and Incident Handling for more techniques and tools like the ones covered here. ✌️
