FOR608: Enterprise-Class Incident Response & Threat Hunting™

Overview

The FOR608: Enterprise-Class Incident Response & Threat Hunting course begins with discussions on current cyber defense concerns, and how incident responders and threat hunters can take a more active role in detection and response. Collaboration within the team and the community are a focus, as we look to incorporate shared knowledge from sources like the MITRE ATT&CK(R) framework. Furthermore, we discuss taking an active defense approach to slow attackers and facilitate detection. Specific to active detection, the use of honeypots, honey tokens, and canaries are covered, along with ways to deploy them opportunistically. This type of tripwire in the network provides defenders and responders needed visibility to find and respond to intrusions quickly.

When a compromise does occur, which is an unfortunate but inevitable truth, we continue the discussion with a focus on the processes and techniques that allow for efficient handling of intrusions. Concepts such as leading the response, managing team members, documenting findings, and communicating with stakeholders are covered in detail. The purpose-built Aurora documentation tool is presented as a means for tracking the investigation phases, from initial detection to scoping, containment, indicator development, and remediation.

We continue the discussion by examining an alert triggered in our example company network as a pivot point into a potential attack. Triage data collected by company personnel has been processed into a timeline and the data imported into Timesketch, a tool developed by the Google IR team. We utilize Timesketch as a powerful platform for scalable and collaborative analysis of forensic data. Later in the class, we also provide techniques to view the same data set with Kibana, which offers additional capabilities, such as creating dashboards for visualizations and saved searches to aid analysis.

We finish Section 1 with an examination of key threat intelligence concepts, including developing and implementing threat intelligence internally. External projects such as the MITRE ATT&CK(R) Matrix and Sigma are also leveraged. We discuss both MISP and OpenCTI as two comprehensive threat intel platforms for ingesting, tracking, and sharing threat intelligence. A threat intel report on the adversary targeting our example company, Stark Research Labs (SRL), will be presented as we start to look at potential signs of intrusion in the company.

Exercises

Development of honey tokens for active detection

• Documenting an initial alert in Aurora
• Using Timesketch to analyze a potential breach
• Using OpenCTI to analyze threat reports of actors targeting our example company's industry

Topics

• Incident Response and Threat Hunting in the Enterprise
  • Taking an Active Defense approach to threat hunting and detection
  • Using Active Defense concepts of Deny, Disrupt, and Degrade for attacker containment
  • Using the Active Defense concept of Deception for detection
  • Pros and cons of using honeypots
  • Pros and cons of canary / honey tokens
  • Deploying canary tokens into an environment for intrusion detection
• Managing Large-Scale Response
  • Fostering key principles of successful response within the team and organization
  • Structuring teams, roles, and responsibilities
  • Leading the response
  • Managing resources
  • Combining incident response and project management disciplines
  • Effective documentation and communication for tracking and reporting incidents
  • Introduction to Aurora, an open-source incident response documentation utility
• Scalable & Collaborative Analysis with Timesketch
  • Using Timesketch to perform deep-dive analysis across multiple hosts with multiple analysts
  • Annotate, label, bookmark events of interest to create custom timeline views
  • Apply analytics and visualizers to assist analysis
  • Create stories to convey findings
• Intel-Driven Incident Response
  • Understand the importance of cyber threat intelligence in incident response
  • Review various sources of threat intelligence and integrating it with the IR process
  • Leverage cyber threat intelligence available from MITRE's ATT&CK(R) Matrix
  • Discuss developing and managing intelligence in your organization
  • Use OpenCTI to catalog, organize, and visualize threat actor TTPs

Overview

Section 2 pivots directly from Section 1 as we continue to move into response mode. We will begin collecting evidence at scale to scope a potential intrusion against our example company, Stark Research Labs. SRL has Endpoint Detection and Response (EDR) tooling in place and we leverage that data to assist scoping. However, attackers sometimes bypass or otherwise subvert EDR technology, so a discussion of common bypass techniques is presented. This provides students with both awareness of EDR limitations, as well as training to look for anomalous activities within the EDR log data.

Moving beyond the analysis of commonly logged artifacts, we introduce the open-source Velociraptor tool as a powerful platform for incident response and threat hunting at scale. Velociraptor is adept at pulling forensic artifacts from across the enterprise, as well as providing analysts with a tool to deep-dive individual hosts of interest. We will show Velociraptor to be a flexible tool useful for a number of situations, as well as for a number of operating systems and architectures.

One of many useful features of Velociraptor is its ability to push collected data into Elasticsearch. Elasticsearch is another powerful and flexible tool appropriate for any responder's toolkit. As such, we use Elasticsearch to ingest and process various data types, including data from Velociraptor, from the PowerShell IR framework "Kansa", and from the "Log2timeline" tool. We then use Elasticsearch's SQL API to run fast searches and aggregations against ingested data. This is a little-known capability that can speed up analysis considerably. As an optional lab, students are also guided through the process to create dashboards and visualizations in Kibana for a more traditional approach to searching and analysis using the Elastic Stack.

After having swept the network looking for indicators of compromise in EDR log data and with tools such as Velociraptor and Kansa, there will inevitably be a subset of hosts that warrant deeper dives. We present rapid response options for targeted data collections at scale, including multi-platform tools such as Velociraptor and CyLR. In the case of Velociraptor, it can be installed on a persistent client-server basis, but also as a standalone collector. We demonstrate how to use it in either case to collect critical artifacts for tracking the adversary's progress. Rapidly post-processing the acquired data for analysis is another important piece of the puzzle. Solutions are presented to quickly take the collected artifact files and process them for analysis in Timesketch, Elasticsearch, or individual artifact review.

Exercises

• Analyzing Sysmon telemetry and log events for incident scoping/identification
• Utilize a "precooked" Velociraptor installation to analyze data collected during the intrusion of Stark Research Labs
• Deploy a small Velociraptor client-server setup in the student VM to perform hunts for artifacts generated from threat emulation tooling
• Use the Elasticsearch SQL API to quickly find anomalous activity
• Acquire forensic triage images using Velociraptor and CyLR. Use automation techniques to rapidly process results for timeline analysis.

Topics

• EDR and EDR Bypass
  • Review the capabilities and challenges of endpoint detection and response (EDR)
  • Analyze Sysmon telemetry and log events for incident scoping/identification
  • Discuss attacker techniques for subverting and bypassing EDR tooling
• Scaling Incident Response with Velociraptor
  • Describe the various use cases for Velociraptor
  • Learn to customize Velociraptor Query Language (VQL) analyzers ("artifacts")
  • Rapidly deploying Velociraptor in a client-server configuration
  • Performing hunts and acquiring forensic evidence
  • Use Velociraptor notebooks for effective post-processing and analysis
  • Export results to Elasticsearch, Splunk, or CSV flat-files for external analysis
• Scaling Analysis with ELK
  • Utilize the ELK stack (aka Elastic Stack) to ingest and analyze logs
  • Ingest structured and freeform data types into the Elastic Stack
  • Use dashboards, histograms, graphs, and saved searches to locate attacker TTPs quickly
  • Utilize the Elasticsearch SQL API for fast data aggregation and filtering
• Rapid Response Triage
  • Execute CyLR and Velociraptor to quickly acquire forensic artifacts from Windows, Linux, and Mac.
  • Create custom acquisition packages for Velociraptor
  • Post-process results for timeline analysis using Timesketch, Elasticsearch, or CSV files

Overview

Section 3 transitions to more traditional host-based forensic artifact analysis. The section starts with a look at some of the latest techniques for attacking Windows systems, including the all-too-common ransomware attack. As part of looking for precursors to ransomware attacks, as well as other targeted attacks, we spend time focusing on attackers' use of "living-off-the-land" techniques to avoid detection. There are many clever ways attackers leverage built-in binaries and scripts (aka "LOLBAS" "Living-Off-the-Land Binaries and Scripts") to accomplish their goals without bringing custom malware onto the host. Learning to proactively detect and retroactively analyze these techniques is critical to investigating many modern-day intrusions.

To facilitate more rapid detection and response, we focus on the use of actionable cyber threat intelligence using Sigma rules. We use Sigma rules to hunt for suspicious activity in logs ingested into Elasticsearch. Another exercise focuses on leveraging Sigma rules packaged into the Hayabusa tool for searching Windows event logs directly. These are powerful techniques which help responders solve cases more quickly at scale.

Following the discussion on Windows threats, the remaining part of Section 3 focuses on Linux incident response and analysis. Many organizations, large and small, have Linux systems present in their environment. Although intrusions against Linux do not make the headlines as often, it's no secret that attackers regularly exploit vulnerable Linux systems to establish and maintain footholds in victim organizations.

FOR608 discusses common vulnerabilities in Linux systems and configurations, then covers common attacker exploits targeting these systems. Privilege escalation, persistence, and lateral movement are techniques we frequently associate with attacks against Windows environments, but they apply equally to Linux as well.

Our Linux discussion continues with coverage of DFIR fundamentals when analyzing Linux systems. Topics that are critical, but often cause confusion, include differences among Linux distributions, Linux file systems, the Logical Volume Manager, key log file locations, and more. Strategies are presented to handle both initial triage and deeper forensic analysis of Linux systems. Searching for unexpected logins, suspicious new files or altered files, and outliers in application logs are just a few of the techniques used to locate malicious behavior. We conclude the section with best practices for hardening systems, enhancing logging configurations, and adding monitoring capabilities to aid future investigations. Providing students with the ability to investigate Linux intrusions is a key goal of FOR608. Upon completion of the course, students will leave with important new skills and techniques for responding to large-scale intrusions across diverse enterprise networks.

Exercises

• Detecting LOLBAS activity via Sigma
• Rapid event log analysis with Hayabusa
• Linux web log analysis
• Triaging Linux hosts

Topics

• Modern Attacks Against Windows
  • Fileless malware in the wild
  • Common "LOLBAS" activity, including precursors to ransomware attacks
  • Hunting amongst the noise for suspicious "LOLBAS" usage
• Detect and Respond to Modern Attacks
  • Leverage the Sigma Project for threat detection against log data
  • Build new Sigma rules for maintaining and sharing threat intelligence
  • Perform Sigma rule searches against centralized log servers
  • Perform Sigma rule searches directly against Windows event logs
• Introduction to Linux
  • History of Linux
  • Ubiquitous nature of Linux
  • Challenges organizations face with managing, securing, and monitoring Linux systems
• Modern Attacks Against Linux
  • Exploiting vulnerable applications or operating system services
  • Misconfigurations or unpatched services lead to successful attacks
  • Attacker techniques for accomplishing the attack lifecycle, including privilege escalation, persistence, lateral movement, and exfiltration
• Linux DFIR Fundamentals
  • Exploiting vulnerable applications or operating system services
  • Misconfigurations or unpatched services lead to successful attacks
  • Attacker techniques for accomplishing the attack lifecycle, including privilege escalation, persistence, lateral movement, and exfiltration
• Linux DFIR Fundamentals
  • Understanding primary differences in file systems
  • EXT3, EXT4, XFS file system overviews
  • Understanding the Logical Volume Manager (LVM2)
  • Available timestamps in Linux file systems (comparing EXT3, EXT4, XFS, Btrfs, ZFS)
  • Typical Linux file system directory hierarchy
• Linux Log Analysis
  • Common logs and locations
  • IR strategy for log analysis
  • Reviewing logon activity
  • Mining application logs for suspicious events
• Linux Triage Collection and Forensic Readiness
  • Collecting key configuration files
  • Collecting artifact-rich logs
  • Scripting collection for simplicity and consistency
  • Hardening Linux configurations
  • Improving audit policies
  • Adding endpoint security tooling

Overview

By this point in the course, students have undertaken a wide-range of tasks, including collecting of host-based data, deployment of live-response tools to catch attackers "in the act", and utilizing "big-data" analysis platforms to find suspicious activity at scale. Students have also taken a deep dive into the Linux operating system and discovered important ways to respond to the inevitable attacks against these systems.

In the next module, we move on to look at key aspects of the Apple macOS operating system. These hosts have become more prevalent in many enterprise networks. Therefore, it's important that incident responders have some understanding and training for responding to such systems. Before diving into the incident response techniques, we discuss the history and current ecosystem of macOS and Apple mobile devices. We then move into important topics such as the Apple Filesystem (APFS), the file and directory structure, and important file types for Mac analysis such as the Property List (plist) configuration file.

After a discussion of the fundamentals, we turn our attention to the challenges and opportunities for responding to macOS incidents. Questions such as how best to acquire disk and triage data, how to review those acquisitions, and which logs and other artifacts are most useful in spotting suspicious activity are all covered in detail.

After establishing a solid foundation for Linux and Mac forensic analysis, we then turn our attention to the concept of containerized microservices. Containers are a popular way to deploy applications and services in a reliable and repeatable way. The most common platform for containers is Docker, which is where we focus our attention in FOR608. Discussions on the architecture and management of Docker containers help students understand where to focus their analysis. A specific triage workflow is also covered to arm analysts with a repeatable process for quick and effective response against individual containers as well as the container host.

Exercises

• Mount and analyze APFS disk images
• Review macOS artifacts and logs
• Docker administration and logs
• Docker triage and IR

Topics

• macOS Foundations
  • A history of Apple operating systems
  • Apple in the enterprise
• Apple Filesystems
  • APFS characteristics
  • macOS timestamps
  • macOS file & directory structure
  • Key file types such as Property List (.plist) files
• Mac Incident Response
  • Challenges with forensic acquisitions
  • Options for mounting disk images
  • Profiling users and system configurations
  • Review common persistence methods
  • Log analysis for macOS
• Containers in the Enterprise
  • Conceptual overview of containers
  • Containers vs. virtual machines
  • Introduction to Docker
  • Attacks against containers
  • Forensic challenges of containerized environments
• DFIR for Containers
  • Metadata collection and analysis
  • Using snapshots to save containerized files
  • Log analysis for Docker
  • Gather ephemeral data
  • Review image files and history

Overview

This day is focused on responding to incidents in the major cloud platforms from Microsoft and Amazon. Although the analysis focuses on those platforms, we cover log analysis techniques, architecture designs, and automation initiatives that can be applied to just about any cloud provider. We also cover ways attackers leverage cloud environments and the artifacts that might be left behind in such cases.

Cloud environments provide unique challenges for incident response, but some exciting opportunities too. A quick intro into these factors will start the day. Once again, we find that the MITRE ATT&CK(R) framework is useful for organizing our defenses and detections - specifically the Cloud Matrix.

Moving into Microsoft 365 (M365) and Azure, several popular SaaS offerings are discussed. These include Entra ID (formerly known as Azure AD) for the underlying identity provider service, which supports other hosted services such as Exchange, SharePoint, and Teams. Many organizations subscribe to these services, and predictably, attackers have become proficient at finding weaknesses in their implementations. We therefore review a variety of common attack scenarios against M365 and Azure. Log analysis is critical to solving these cases, so log acquisition and review is a major focus for discussion. Specifically, we look for suspicious user logon and email activity from the Unified Audit Logs (UAL) as a common method for detection. The Entra ID (Azure AD) Audit log and others are useful resources as well.

Important for any incident response is the Recovery phase, which typically includes implementing security enhancements to detect or prevent similar attacks in the future. Therefore, we cover some of the more useful security enhancements in M365 and Azure.

The second part of the day delves into the Amazon Web Services (AWS) cloud platform. Its general architecture and components are covered to provide a solid foundation for those new to AWS. We then go into detail on the many logs and services that provide critical detection and analysis data for responders. This includes CloudTrail logs, VPC flow logs, GuardDuty alerts, and more.

The section concludes with discussions on architecting for response in the cloud for faster and more effective analysis. This involves setting up security accounts for a secure enclave within AWS. Template VMs (AMIs) are also recommended for performing analysis against volume snapshots, network packet captures, and log data. Finally, we look at common IR tasks that can be automated and how to do it rather seamlessly using AWS Lamda and Step Functions. While the solutions presented in this section are AWS-centric, the concepts can (and should) be applied to almost any cloud platform with significant use by an organization.

Exercises

• M365 log analysis
• Finding attacker cloud exfil infrastructure
• AWS CloudTrail log analysis
• AWS VPC Flow log analysis

Topics

• DFIR in the Cloud
  • Cloud service models (IaaS, PaaS, SaaS)
  • Cloud forensics vs. traditional forensics
  • MITRE ATT&CK(R) Cloud Matrix
• Incident Response in Azure & M365
  • M365/O365 SaaS offerings
  • Azure IaaS and PaaS platform
  • Entra ID (formerly Azure AD) architecture
  • Common attack scenarios
  • Important log sources & log extraction
  • Investigating suspicious user logons and email activity
  • Securing M365 & Azure
• Attackers in the Cloud
  • Investigating attacks that leverage the cloud
  • Discover host-based artifacts from attacker's cloud infrastructure
• AWS Foundations
  • Organizational and account hierarchy
  • AWS Identity and Access Management (IAM)
  • Authentication and identity types
  • AWS regions and API endpoints
  • AWS computing, storage, and networking constructs
• Incident Response in AWS
  • Leveraging the AWS Incident Response Guide
  • AWS incident domains
  • Critical log sources such as CloudTrail, CloudWatch, and S3 access logs
  • Threat detection and response services such as GuardDuty and Detective
  • Network analysis with VPC flow logs and traffic mirroring
  • Architecting for analysis in the cloud
  • Acquiring logs and snapshots
  • Planning and practicing likely scenarios
• IR Automation in AWS
  • Identifying tasks for automation
  • Using AWS VM templates (AMIs) for quick response
  • Leveraging AWS Lamda and Step Functions for automation and orchestration

Overview

Section 6 will serve as a capstone for the class and a chance for students to put into practice the knowledge they've gained thus far. We will be providing an exercise that focuses on utilizing the tools and techniques discussed in the previous five sections of the course. Students will be provided with a data set from a compromised environment spanning multiple host operating systems and cloud environments. Students will utilize tools and techniques learned throughout the course to uncover the steps of the breach, end-to-end. In the live classroom setting, students will work in small teams to divide and conquer the analysis in order to solve this complex case most efficiently, similar to real-world incident response scenarios.

Exercises

• Day 6 CTF Challenge