Microsoft 365 Cloud Log Extraction

So far in our blog post series on cloud log extraction, we have looked at extracting logs from AWS, Google Cloud, Google Workspace and Azure. In the fifth and final installment of this series, we’ll be looking at how we can view and extract logs from Microsoft 365.

This first post will look at Microsoft 365. One of the benefits of Microsoft 365’s logging is that they centralize all logs into the Unified Audit Log (UAL). This single log will contain user and admin activity for Azure AD, Exchange, SharePoint, OneDrive, and more. We’ll specifically look at three ways in which we can query and export the UAL:

• Microsoft Purview Compliance Portal
• PowerShell
• Microsoft 365 Management API

All three options have various strengths and weaknesses, so it’ll all depend on your use case which of the methods you want to leverage.

Microsoft Purview Compliance Portal

Accessing the logs via the Microsoft Purview Compliance Portal is best used in situations where you want to perform a quick, targeted search. It has good built-in search functionality but leveraging the data in the portal is more difficult and the export function has restrictions that make it less than ideal for use. If you do choose this method to extract logs, you can use the following steps:

1. Log into the Purview compliance portal.
2. Select Audit from the menu on the left hand side.
3. Run a search by specifying the date range, optionally applying filters, and selecting Search.
   1. Underneath the search options, a list of previous and in progress searches will appear. When your search is complete you will see a Job Status of Completed
      

4. Click on the relevant search and on the subsequent page select Export on the top left.
   1. The export may take a moment to complete, especially if the results include a large number of records. You may need to refresh the page to check for completion. When the results are ready, a green status message with a link to the download will be at the top of the screen.

Unfortunately, another drawback of this method is that the SOF-ELK Microsoft 365 parser does not support Purview exports. For SOF-ELK compatibility, you should use the PowerShell method discussed in the next section. The most important data in the CSV export is stored in the form of a JSON blob within the “AuditData” column. As such, if you are going to review the logs in Excel or another CSV viewer, you need to transform that column. Microsoft provides step-by-step instructions on how to do this in their documentation.

PowerShell

PowerShell has a Cmdlet called Search-UnifiedAuditLog that allows you to query and export the UAL via a PowerShell interface. This method is best used when trying to export a small amount of the UAL as it limits the number of records to 5,000. If you need more than 5,000 records and want to leverage PowerShell, we recommend taking a look at the Microsoft Extractor Suite mentioned in the “Open-source UAL Collection Tools” section below.

Before attempting to export the UAL via PowerShell, you need to install the Exchange Online PowerShell module which provides the cmdlets for interacting with Microsoft 365. Furthermore, you need to ensure you have the permissions required to access the UAL via PowerShell. At minimum, you need the View-Only Audit Logs role assigned to your user via a role group in the Microsoft 365 admin center. Once these prerequisites are in place, the following steps can be performed to export the logs.

1. Connect to Microsoft 365:
   
   Connect-ExchangeOnline –UserPrincipalName <UPN> -ShowProgress $true
   
   
2. Perform a search, extract the AuditData property, and pipe the output to a UTF8-encoded JSON file.
   
   Search-UnifiedAuditLog –StartDate 2023-06-01 –EndDate 2023-07-01 -ResultSize 5000 ReturnNextPreviewPage | Select-Object –ExpandProperty AuditData | Out-File –Encoding UTF8 “20230601_ual.json”
   
   Note: Additional filters can be applied. See the Search-UnifiedAuditLog cmdlet documentation for more parameters.

From this point, you can import the JSON file into SOF-ELK (by copying it to the Microsoft 365 Logstash directory) or another parsing tool of your choosing.

Microsoft 365 Management API

The final method that can be used for exporting the UAL is the Microsoft 365 Management API. This is best for environments looking to continuously query the UAL in order to ingest the logs into an external platform such as a SIEM or large environments that exceed the export limits of the PowerShell Cmdlet. The API endpoint will only return results from the last 7 days, so it must be polled on a continuous basis via a custom-developed application or commercial solution that leverages the API. The high-level steps for using the API are as follows:

1. Register and configure an application in Azure AD
2. Generate a new key and X.509 certificate for the application
3. Authorize Microsoft 365 Management API access
4. Request access token from Azure AD
5. Start a subscription to the required log
6. Retrieve blobs of logs at regular intervals

For more in-depth guidance on leveraging the API, see Microsoft’s documentation here: https://for509.com/ualmgmtapi. If you are leveraging a commercial product for collecting these logs, they will most likely have a built-in integration and configuration guide for connecting your solution to the API.

Open-source UAL Collection Tools

As mentioned in the previous blog post in this series, Invictus IR has released a suite of scripts for extracting Microsoft logs (both 365 and Azure) using PowerShell. One of the major benefits of these scripts is that it breaks down the extraction into chunks in order to avoid the record limit imposed by the Search-UnifiedAuditLog PowerShell cmdlet. The scripts related to extracting the UAL are:

• Get all the UAL entries. Supports filtering by date and userid only.
• Get the selected group UAL entries based on pre-defined groups. These groups attempt to compensate for the fact that the Search-UnifiedAuditLog cmdlet doesn’t support filtering by workload.
• Get specific records by filtering on RecordType in addition to date and userid.
• Display the total number of logs within the UAL given a date and userid filter.

These scripts can be accessed at no cost at the following Github link: https://for509.com/invictus-suite.

Conclusion

In this blog post, we discussed the various methods of accessing and exporting the Unified Audit Log (UAL). If you need a way to perform a quick and targeted search, you can leverage the Purview Compliance Portal. If you want to retrieve a small part of the UAL and export to JSON format, PowerShell has the Search-UnifiedAuditLog cmdlet. For larger organizations or those who need to continuously poll the UAL, there is the Microsoft 365 Management API.

We hope that this series of blog posts on extracting cloud logs will better prepare you for when an incident occurs in your cloud environment! If you want to learn more about leveraging these logs for incident response, check out SANS FOR509: Enterprise Cloud Forensics and Incident Response.