FOR577: LINUX Incident Response and Threat Hunting™

Overview

Incident responders and threat hunters should be armed with the latest tools, techniques, and processes (TTPs) to identify, track, and contain advanced adversaries and to remediate incidents. It is important that our DFIR knowledge includes our own TTPs and those used by our adversaries. Section 1 introduces the fundamentals of incident response and then looks at the specific needs to carry out our duties in a Linux environment. The section starts by examining the reasons why we need incident response and presents SANS' six-step incident response methodology as it applies to an enterprise's response to a targeted attack.

This section will also introduce the Stark Skunkworks intrusion scenario, which sets the stage for our lab exercises and capstone challenge. This is followed by looking at how, as incident responders, we can use the Linux command line to our advantage and analyze common activity such as installing specific software packages.

We finish the section by looking at the importance of developing cyber threat intelligence to impact the adversaries' kill chain. We'll demonstrate forensic live response techniques and tactics that can be applied both to single systems and across the entire enterprise.

Exercises

• SIFT Workstation orientation
• Situational awareness in incident response: Understanding Stark Skunkworks
• Introduction to Linux commands and how to use them in Digital Forensics and Incident Response (DFIR)
• Reviewing package management evidence
• Threat intelligence and threat hunting

Topics

• Why Incident Response Is Needed
  • Who are our adversaries?
  • The current state of Linux intrusions
• The Incident Response Process
  • Preparation: Key tools, techniques, and procedures that an incident response team needs to respond properly to intrusions
  • Identification/Scoping: Proper scoping of an incident and detecting all compromised systems in the enterprise
  • Containment/Intelligence Development: Restricting access, monitoring, and learning about the adversary in order to develop threat intelligence
  • Eradication/Remediation: Determining and executing key steps that must be taken to help stop the current incident and then move to real-time remediation
  • Recovery: Recording the threat intelligence to be used in the event of a similar adversary returning to the enterprise
  • Avoiding "Whack-A-Mole" Incident Response: Going beyond immediate eradication without proper incident scoping/containment
• SRL Skunkworks
  • Introduction to the course scenario
  • Client background
• Introduction to Linux
  • Linux basics
  • DFIR challenges
  • The distro problem
  • Linux terminal basics
• Package Management
  • Distro differences
  • Package management tool differences
  • Manual analysis
• Threat Intelligence and Host-based Threat Hunting
  • Hunting vs. reactive response
  • Intelligence-driven incident response
  • Building a continuous incident response/threat hunting capability
  • Forensic analysis versus threat hunting across endpoints
  • Threat hunt team roles

Overview

Disk evidence collection and analysis skills are crucial for incident responders, forensic investigators, and threat hunters because they allow for identifying the source and scope of a security breach. Digital forensic experts need to collect and preserve data from disk storage devices such as hard drives, solid-state drives, and USB drives in order to determine how an attack occurred, what data was accessed or stolen, and who was responsible. Without this critical evidence, it is challenging to reconstruct the events leading up to the breach and determine the necessary steps to prevent similar incidents from happening in the future.

Moreover, disk analysis skills help responders and investigators identify the type of malware or malicious code used in the attack. This information is essential to determine the tactics, techniques, and procedures used by the attackers and their motivations. By analyzing the data stored on disks, responders and investigators can identify suspicious files, unusual network traffic patterns, and other indicators of compromise. They can then use this information to develop countermeasures to mitigate the risk of further attacks.

Fundamentally, the ability to collect evidence from disks is critical for DFIR because most digital evidence is stored on disk storage devices, making the devices an essential source of information for responders, investigators, and threat hunters. Even if you just need to collect log data, being able to collect it from a disk image opens up opportunities for a broad range of incident response solutions. In addition, disk storage devices often hold deleted files and other remnants of past activities, which can provide valuable clues to the sequence of events leading up to an incident.

Exercises

• Introduction to the Sleuth Kit
• Reviewing filesystem data
• Disk evidence collection
• Reviewing operating system filesystems

Topics

• The Sleuth Kit
  • Introduction and the layers model
  • Filesystem layer tools
  • Filename layer tools
  • Metadata layer tools
  • Data units layer tools
  • Application layer tools

• Linux File Systems
  • Overview
  • Basic structures - superblocks and inodes
  • Ext family
  • XFS family
  • Manually extracting data
• Disk Evidence Collection
  • Physical vs. virtual systems
  • dd
  • dcfldd
  • dc3dd
  • Ewfacquire
• Image Mounting
  • RAW/Simple files
  • E01 format evidence files
  • Complex files
• Operating System File Structures
  • File system hierarchy
  • Boot file locations
  • Binary file locations
  • Configuration file locations
  • Devices and driver file locations
  • Shared libraries
  • User profiles
  • Optionally installed files
  • Temporary file locations
  • Runtime data
• File System Artifacts
  • Hunting tips
  • Areas to investigate

Overview

Section 3 looks at how to use the data logged by the operating system to profile the device and analyze boot sequences, kernel activity, logins and user events. The section covers default log data, Auditd (although this isn't enabled by default on all Linux distros, you should definitely consider turning it on) and the Operating System Journal.

Log data is a fundamental evidence source for incident response and threat hunting. It allows investigators to understand what happened and when it happened. Using built-in capabilities, we can peel back the actions of our adversaries and, with well-configured logging, make it almost impossible for an attacker to completely hide from our investigation. Unfortunately, Linux logging can be significantly different from what we are used to -- especially if we have come from a Windows DFIR background. Significant issues faced by investigators include the different ways Linux distro's log data and a mix between UTC and local timestamps. This section will look at strategies you can implement to manage and mitigate these issues.

Exercises

• System and log profiling
• Reviewing system logs
• Analyzing authentication logs
• Reviewing webserver logs
• Reviewing database logs
• AuditD logs the Journal

Topics

• Device Profiling

  • Evidence management

  • Confirm the device
  • Check time zones
  • Validate the distro

• Linux Logs
  • Linux logging basics
  • Log analysis strategies
  • Syslog and Logrotate
  • Global system logs -- logging the kernel, boot processes, system messages and background services
  • Authentication logs -- authentication, privilege use, binary and plain text log formats
  • Application logs -- webservers, databases, filesharing and firewall logs

• Auditd

  • Introduction
  • Log file format
  • Analysis techniques

• The Operating System Journal
  • Introduction
  • How the journal works
  • What gets logged
  • Analysis techniques

Overview

Section 4 expands on the knowledge we have built so far and introduces tools and techniques to respond to intrusions in larger enterprises. The section starts by looking at how to scale your response and some of the tools that can assist with this. This topic is then developed further as we move into Endpoint Detection and Response (EDR) solutions for the Linux environment and introduce two alternatives to expensive commercial EDR tools -- OSSEC and Velociraptor. We'll cover how to configure and deploy both tools, enabling you to make sure that all your Linux devices have good quality monitoring and response capabilities.

Finally, this section looks at Linux memory structures and how to collect volatile data for analysis. Given that this can be a complex process, and that analytical tools today are still not what they should be, we also look at using live response techniques to view this data on a target system. This has the added benefit of being something we can leverage through EDR tools, reducing the time and bandwidth required to capture memory from systems where the installed RAM could be running in the hundreds of gigabytes.

Exercises

• EDR Tools
• Capturing RAM
• Live memory analysis

Topics

• Enterprise Response
  • Introduction
  • Problems and solutions
  • Tools to consider
• Endpoint Detection and Response (EDR)
  • Introduction
  • Linux EDR issues
  • Alternatives to commercial EDR
  • OSSEC deployment and use
  • Velociraptor deployment and use
• Linux Memory and DFIR
  • Why memory matters
  • Memory acquisition with AVML
  • Memory locations on the filesystem
• Live memory analysis
  • Reviewing /proc
  • Live response workflow

Overview

This course section builds on the previous sections by looking at how we can use our increased knowledge to enhance our incident response work. We start by looking at triage, which is essential for any modern incident response, especially in large enterprises. We introduce the concept of rapidly assessing systems to make quick decisions about which devices need further investigation. This approach allows us to quickly work through large environments and focus our investigative efforts where they provide maximum value. We'll also look at freely available tools that help facilitate triage and improve response times.

The section then moves to looking at timeline generation. Timelines are arguably an incident responder's superpower, allowing you to uncover some of the deepest secrets about an attack. We will look at two basic methods for building timelines and how to analyze them effectively.

Once we understand the timelines, we will look at how attackers try to defeat them, then examine the most common anti-forensic techniques and how incident responders can minimize their impact on the investigation. We close the section with a broad discussion on how to make incident response in Linux better.

Exercises

• Running triage tools
• Triage assessment
• Filesystem timelines
• Super timeline creation
• Super timeline analysis

Topics

• Triage and DFIR Tools
  • Introduction and concepts
  • Workflow
  • Collecting the data
  • Open-source triage tools
  • CyLR
  • GRR
  • Velociraptor offline collectors
  • Dissect
  • Triage with UAC
  • Build your own triage scripts
• Timelines
  • Introduction
  • Types of timelines
  • Filesystem timeline creation and analysis
  • Super-timeline creation and analysis
  • Targeted timeline creation
• Anti-Forensics
  • What to look for
  • Timestamp manipulation
  • Recovering deleted files
• Improving Incident Response
  • Workflows
  • Hardening the environment

Overview

This incredibly rich and realistic Intrusion Forensic Challenge is based on a real-world advanced persistent threat (APT) group. It brings together techniques learned throughout the course and tests your newly acquired skills in a case that simulates an attack by an advanced adversary. The challenge is based on a real intrusion into a Linux enterprise environment. You will be asked to uncover how the systems were compromised in the initial intrusion, find other systems the adversary moved to laterally, and identify intellectual property stolen via data exfiltration. This capstone exercise will enable you to leave the course with hands-on experience investigating realistic attacks, curated by a cadre of instructors with decades of experience fighting advanced threats from attackers ranging from nation-states to financial crime syndicates and hactivist groups.

Topics

• Work in incident response teams to analyze multiple systems in an enterprise network
• Learn to identify and track attacker actions across a multi-device environment finding initial exploitation, reconnaissance, persistence, privilege escalation, lateral movement, and data theft/exfiltration
• Witness and participate in a team-based approach to incident response
• Discover evidence of some of the most common and sophisticated attacks in the wild, including custom nation-state malware.
• Each team will be asked to answer key questions, just as they would during a real breach in their organizations, in critical areas outlined below:

Identification and Scoping:

• When did the APT group breach our network?
• How did the attackers get into the environment?
• What systems were compromised?
• What accounts and privileges did the attackers attain on each system?
• When and how did the attackers first laterally move to each system?

Containment and Threat Intelligence Gathering:

• Once on other systems, what did the attackers look for on each system?
• What data was exfiltrated and how? Determine what was stolen (recover any archives exfiltrated, find encoding passwords, and extract the contents to verify extracted data) and perform damage assessments.
• Collect and list all malware used in the attack.
• Develop and present security intelligence or an indicator of compromise for the APT group "beacon" malware for both host- and network-based enterprise scoping. What specific indicators exist for the use of this malware?

Remediation and Recovery:

• What accounts need password changes? Did any malicious accounts get created?
• Based on the attacker techniques and tools discovered during the incident, what are the recommended steps to remediate and recover from this incident?
  • What systems need to be rebuilt?
  • What IP addresses need to be blocked?
  • What countermeasures should we deploy to slow or stop these attackers if they come back?
  • What recommendations would you make to detect these intruders in our network again?