This is the third of a three-part series on using PowerShell for audit and compliance measurements. These blog posts supplement the material presented in the free webcast series "PowerShell for Audit, Compliance, and Security Automation and Visualization".
When I work with clients to automate their compliance efforts, I frequently find myself writing a PowerShell script to import or export data using a web API (application programming interface). In this post, I'll share some tips and techniques for working with web APIs using PowerShell.
Web APIs, REST and SOAP
The term "Web API" describes any HTTP-based interface which has been developed to allow users or programs to interact with a software system. APIs usually expose data directly in some structured format like JSON or XML, without wrapping any sort of user interface around it. This straightforward data access makes API access a perfect technique for automated tasks. Let's begin with a brief description of the two types of APIs you are likely to encounter, REST and SOAP.
The original idea behind REST services was first proposed by Roy Fielding in his doctoral dissertation. Fielding's idea has evolved (been appropriated?) so that many REST services map the database CRUD (create, retrieve, update, delete) functions to existing HTTP verbs, like POST, GET, PUT, PATCH and DELETE. This straightforward design has made REST services very popular over the last several years. Many of the security and compliance tools I use in my practice make REST APIs available for controlling settings, launching actions and importing and exporting data. Many of today's REST APIs use JSON (JavaScript Object Notation) as their data interchange format.
SOAP (simple object access protocol) is the XML-based data interchange protocol that drove much of the "service-oriented architecture" revolution in web applications a few years ago. The good news is that everything is well defined in most SOAP APIs. The bad news is that compared to handling JSON data from a REST API, there's a lot of overhead involved in creating a well-structured XML document for the request and then parsing the XML-based response. Fortunately, if you are running Windows PowerShell, I'll be able to show you an often-overlooked shortcut for handling SOAP services.
Accessing REST with Invoke-RestMethod
The first step to working with any API is to review the documentation which describes the API calls, accepted data types, authentication requirements, response formatting, etc. Many REST APIs use the OpenAPI Specification, the renamed and updated version of the older Swagger documentation standard. For this post, we will consume data from GitHub's Issue-tracking API. It is documented here. This API allows us to make both authenticated an unauthenticated requests, so we'll do a little of each.
Imagine that management has asked us to retrieve a list of the 100 most recently closed issues for a particular GitHub repository, and to calculate the mean time to resolution (MTTR) for the 100 issues. Time to resolution (TTR) for a single issue would be the number of days which elapsed between the issue being opened and closed. The MTTR will simply be the average of the TTR for all 100 issues. For this example, we'll use issues from the Microsoft PowerShell repository, since it is pretty active. Upon reviewing the API documentation, we see that we can issue a GET request with query string parameters for the number of issues (up to 100) to return per "page" or request, the state of the issues (open, closed or all) to be returned, and a page number, which tells the API which result page to return.
Armed with that information, we build the PowerShell command to make a test request. We'll start by asking for a single result, using the Invoke-RestMethod cmdlet. It will make the web request and then convert the JSON data returned by the API into a PowerShell object. We'll save the object to a variable to make it easier to analyze. Then we'll use Get-Member to view the properties of the object.
$testResult = (Invoke-RestMethod -Uri
"https://api.github.com/repos/PowerShell/PowerShell/issues?per_page=1&page=0&state=closed")
$testResult | Get-Member
The created_at and closed_at DateTime objects look like they would be useful for calculating the time to resolve an issue. We'll use a TimeSpan object to calculate the TTR for this issue:
New-TimeSpan -Start $testResult.created_at -End $testResult.closed_at
Now that we know how to calculate the TTR for a single issue, let's gather the 100 most recently closed issues and calculate their MTTR. First, we'll make a GET request and save the results into a variable, then we'll count the results to ensure we got a full 100:
$issues = (Invoke-RestMethod -Uri
"https://api.github.com/repos/PowerShell/PowerShell/issues?per_page=100&page=0&state=closed")
PS > $issues.count
100
Then, we can calculate the MTTR. We'll use a calculated property for the TTR for each issue, and then use Measure-Object to get the average:
($issues |
Select-Object @{n='TimeToResolve'; e={(New-TimeSpan -Start $_.created_at -End $_.closed_at).TotalDays} } |
Measure-Object -Property TimeToResolve -Average).Average
The answer to management's question is that the MTTR for the last 100 tickets is 3.14 days. The InvokeRestMethod cmdlet took most of the work out of the problem, and allowed us to focus on gathering the needed data!
Passing Authentication Tokens
Many REST calls will require authentication. GitHub requires you to authenticate if you want to make more than 60 requests during an hour. Let's re-work the previous example with more data. Management has asked for the MTTR for all tickets closed within the last 90 days. This will require requesting multiple pages of data, and it may put us over the API rate limit for unauthenticated requests.
We'll start by logging onto Github and issuing a new API token for accessing public repositories. If you have a GitHub account, you can create a token of your own. Be sure to copy the token from the results page. Once you close that page, you won't be able to retrieve the token in plain text again.
We'll pass our newly created token to Github with each request to authenticate that it came from us, so let's save it in a variable. We'll also calculate the date for 90 days ago and save it in the time format required by the API:
$token = Read-Host -AsSecureString "Paste in GitHub Token"
$since = (Get-Date -Date (Get-Date).addDays(-90) -Format "yyyyMMddTHHmmssZ" -Hour 0 -Minute 0 -Second 0 -Millisecond 0)
We'll need to build a PSCredential object to pass our authentication token using HTTP basic authentication. It will be passed using the Credential parameter of Invoke-RestMethod.
$githubUsername = 'Sec557-Demo'
$cred=New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $githubUsername, $tokenNext, we'll build a WHILE loop to keep retrieving pages of data until we have retrieved all the issues closed after the date we specified. We'll add all the issues to a variable so we can do the same math we did above on this larger dataset.
$issues = $null
#Counter variables
$page = -1
$count = 100
#Loop until you receive < 100 results on the page
while ( $count -eq 100)
{
<p>$page++
"Processing page: $page"
#Set the URL for the request, plugging in $page as the page number
$uri = "https://api.github.com/repos/PowerShell/PowerShell/issues?page=$page&per_page=100&state=closed&since=$since"
#Get the next page and add the contents to the $issues variable
$nextPage = Invoke-RestMethod -Credential $cred -Uri $uri
$count = $nextPage.count
$issues += $nextPage</p>After retrieving several pages, we received a total of 793 issues.
Finally, we can calculate the MTTR for this larger dataset using the same math as above:
($issues |
<p>Select-Object @{n='TimeToResolve'; e={(New-TimeSpan -Start $_.created_at -End $_.closed_at).TotalDays} } |
Measure-Object -Property TimeToResolve -Average).Average</p>
Again, the authentication and data conversion are handled transparently, allowing us to focus on the data!
Accessing SOAP the Hard Way - PowerShell Core
All the examples we've seen so far will work correctly in PowerShell Core and Windows PowerShell. If you find yourself needing to access a SOAP API, I strongly recommend that you find a Windows computer so you can use Windows PowerShell. The process in PS Core is largely manual. You would need to:
- Read the WSDL (web service definition language) file to understand the XML schema used for making requests and receiving responses.
- Build the XML-formatted SOAP request and pass it as the body using
Invoke-WebRequest. - Save the XML document returned in the response and use a .NET XMLDocument object to load it into a variable.
- Work with the XMLDocument variable to extract your results.
#Based on the WSDL, create the request to get a list of country names and ISO codes.
$body = '<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://www.oorsprong.org/websamples.countryinfo">
<soapenv:Header/>
<soapenv:Body>
<web:ListOfCountryNamesByName/>
</soapenv:Body>
</soapenv:Envelope>
'
#Make the web request and save the result to an XML file
(Invoke-WebRequest -UseBasicParsing -method Post -Body $body `
-uri 'http://webservices.oorsprong.org/websamples.countryinfo/CountryInfoService.wso' `
-ContentType "text/xml").content | Out-File -Path countries.xml -Force
#Load the file into an XMLDocument object
$xmlCountries = New-Object System.Xml.XmlDocument
$fn = Resolve-Path(".\countries.xml")
$xmlCountries.Load($fn)
#Drill down in the XML to the actual result and then query it for the 'GB' ISO code
$countryCodes = $xmlCountries.envelope.body.ListOfCountryNamesByNameResponse.ListOfCountryNamesByNameResult.tCountryCodeAndName | Select-Object sISOCode, sName
$countryCodes | Where-Object sISOCode -eq 'GB'
While this is not a complicated problem, it does require a number of steps and a fait bit of research.
Accessing SOAP the Easy Way - Windows PowerShell
The simple way to access SOAP is by using the New-WebServiceProxy cmdlet. It handles all the setup and data conversion for you. Simply point the proxy at the WSDL file and then use the auto-generated object to access the API. To demonstrate this, we'll use a free SOAP API which returns country information like mappings of ISO codes to country names.
$wsp=New-WebServiceProxy -Uri http://webservices.oorsprong.org/websamples.countryinfo/CountryInfoService.wso?wsdl
$wsp | Get-Member -Type Method
One of the methods returned by the API is called ListOfCountryNamesByName. It returns a list of countries and their ISO codes. To call this function, we'll just use the variable created above.
$countryCodes = $wsp.ListOfCountryNamesByName()
$countryCodes | Where-Object sISOCode -eq 'GB'That's it! Five lines of PowerShell yielded the same results as you achieved in the previous section, with no real analysis required! In the rare event that you have to interact with a SOAP-based API, I recommend that you use Windows PowerShell and the New-WebServiceProxy cmdlet whenever possible.
