How You Can Start Learning Malware Analysis

This blog was originally published on August 4, 2021 and updated on January 7, 2025.

Malware analysis sits at the intersection of incident response, forensics, system and network administration, security monitoring, and software engineering. You can get into this field by building upon your existing skills in any of these disciplines. As someone who’s helped thousands of security professionals learn how to analyze malware at SANS Institute, I have a few tips for how you can get started.

What is Malware Analysis?

Malware analysis is the process of studying malicious software to understand its origins, functionality, and impact on affected systems. This process involves dissecting the malware to determine how it operates, spreads, and evades detection. Analysts use various techniques, including static analysis, which examines the malware without running it, and dynamic analysis, which observes the malware in action within a controlled environment. The goal is to develop effective defense strategies and mitigation measures. By understanding malware behavior, cybersecurity professionals can create signatures to detect similar threats, patch vulnerabilities, and enhance overall security posture against potential cyber attacks.

Benefits of Malware Analysis

Malware analysis plays a key role in helping security professionals and organizations defend against cyber threats. Here are some of the benefits it offers:

Stronger Threat Detection

By examining malware, analysts can uncover Indicators of Compromise (IOCs) like malicious file hashes or behaviors. These insights help develop detection signatures to spot similar threats in the future, enhancing overall defenses.

More Effective Incident Response

Malware analysis provides clarity on how an attack happened which helps response teams identify the root cause, contain the threat, and mitigate its impact.

Smarter Defensive Strategies

Studying malware behavior provides insight into attacker tactics which allows organizations to improve security measures like configuring endpoint detection systems or refining monitoring techniques.

Valuable Contributions to Threat Intelligence

Analysis insights can be shared with the cybersecurity community to better collective knowledge and improve defenses against similar attacks.

Skill Growth for Security Teams

Malware analysis is all about hands-on learning to help professionals refine their reverse engineering, forensics, and threat hunting skills.

In short, malware analysis equips security teams with the knowledge and tools needed to stay ahead of threats and strengthen organizational resilience.

Understand Where You Currently Fit Into the Malware Analysis Process

There are several ways to describe the skills you to analyze malicious software. I like grouping them into four categories, which I detailed in the post Mastering 4 Stages of Malware Analysis. I’m outlining them below, so you can begin thinking about your current capabilities in these areas:

The Four Stages Of Malware Analysis

The malware analysis steps flow from the bottom of the pyramid (easiest) to the top (most challenging), though people vary the order based on their skills, the analysis context, and the malware characteristics. It's also common to revisit earlier steps as new details emerge during the investigation:

1. Fully-Automated Analysis

This is the initial and often the fastest step in analyzing suspicious files. By running the file in an automated analysis environment like a "sandbox," you can generate a detailed report on its observable behavior. This includes:

• File system interactions: Creation, modification, or deletion of files.
• Network communications: Any attempts to connect to remote servers or download additional payloads.
• Process activities: Execution of child processes or injection into other processes.
• System modifications: Registry changes, persistence mechanisms, or service creation.

While this approach offers valuable initial insights, it may miss complex behaviors designed to evade sandboxes or trigger only under specific conditions.

2. Static Properties Analysis

This step involves inspecting the file's metadata and embedded details without executing it. Key activities include:

• Extracting strings: Identifying human-readable text that could hint at file paths, IP addresses, or commands.
• Examining headers: Understanding the file type, structure, and potential compilation timestamps.
• Assessing entropy: Determining whether the file is packed or obfuscated, which might suggest efforts to hide malicious code.
• Identifying imports/exports: Checking functions the file relies on, which may reveal its intended capabilities.

This step lays the groundwork for deeper investigations by spotlighting areas that warrant a closer look.

3. Interactive Behavior Analysis

In this step, you analyze the malware's behavior by running it in a controlled laboratory environment, such as a virtual machine isolated from production systems. Unlike fully-automated analysis, this step involves experimenting with configurations and inputs to understand the malware's actions. Key techniques include:

• Observing environment-dependent behavior: Introducing various inputs, such as specific files or registry keys, to trigger certain functionalities.
• Studying evasive techniques: Identifying anti-sandbox or anti-virtualization mechanisms that prevent the malware from fully executing in automated environments.
• Recording system interactions: Using tools to monitor file, registry, and network activities in real time.

This iterative, hands-on process allows you to uncover behaviors that may not surface during automated analysis.

4. Manual Code Reversing

The most advanced and challenging stage of malware analysis involves examining the malware's code. By using tools like disassemblers and debuggers, analysts can:

• Decrypt obfuscated code to understand its logic and intent.
• Pinpoint the malware's core capabilities, such as exfiltration routines or command-and-control (C2) communication.
• Analyze specific functions and algorithms, such as encryption or anti-analysis techniques.
• Identify artifacts that may assist in creating IOCs or developing mitigation strategies.

This step requires a thorough understanding of programming, assembly language, and common malware techniques. It's also where you'll fill in the gaps left by earlier steps, giving you the most granular insight into the malware's functionality.

Supplemental Forensics Techniques

In addition to the core analysis stages, memory, file system, and network forensics can provide crucial context:

• Memory forensics: Analyzing RAM snapshots to uncover malicious processes, injected code, or sensitive data in memory.
• File system forensics: Investigating disk artifacts to trace malware persistence mechanisms or uncover additional payloads.
• Network forensics: Examining captured traffic for command-and-control communications or data exfiltration attempts.

Getting Started

Ask yourself, “What skills do I have today, and where do they fit into the malware analysis process?” This will help you identify the best starting point. For instance:

• If you're comfortable with scripting, focus on static properties analysis and automation.
• If you're adept at system administration, interactive behavior analysis might be a natural starting point.
• If you have a programming or reverse engineering background, you may feel ready to dive into manual code reversing.

Start experimenting with tools and techniques in the area where you feel most comfortable and build from there. The following resources can help guide your journey.

Review and Learn From Others’ Malware Analysis and Findings

If you’re just getting into malware analysis, you can start by examining the reports published by the more experienced analysts and automated sandboxes. As you review these details, note which aspects of the analysis make sense to you, and which areas require further study. This is one way for you to generate a learning plan.

If you’re wondering where to find malware analysis reports, here’s one starting point: Look at the sources of the posts I make on the LearnREM page I maintain on Facebook. There, you’ll see website and blogs that I like to read to keep up with the industry. (You can view it even without signing into Facebook.)

When reading these reports, you’ll come across some malware sample that you’d like to examine more deeply. Make a note of the hash and other attributes of that file, then look for that file’s report in public malware analysis sandboxes.

You can access several malware analysis sandboxes for free. Search them for the malware you wish to explore; chances are good that they’ve already analyzed that file. Examine the reports, paying attention to the flagged behaviors that indicate that the file might be malicious. Look for IOCs that would help you spot that malware in the wild.

Start Experimenting With Malware in Your Lab

Set up a lab that allows you to experiment with malware in an isolated, controlled environment. I shared my recommendations for doing this in the blog post 5 Steps to Building a Malware Analysis Toolkit Using Free Tools. You can use virtualization software to set up Windows and Linux systems to facilitate your analysis.

Next, consider watching my Practical Malware Analysis Essentials talk, which will offer practical tips for analyzing Windows malware in your lab:

In addition to watching this video, consider reviewing the corresponding slides. Use these materials to repeat in your own lab the analysis I demonstrated. If you’d like a copy of the malware sample I used in this talk, reach out to me. (Be sure to exercise caution when experimenting with real-world malware like this.)

In addition to the Windows-based approach to examining malware, which the video above demonstrates, consider bringing Linux into your environment. The REMnux toolkit is designed for this very purpose, and is available as a virtual machine and a Docker container. You can see REMnux in action in my video What’s New in REMnux v7:

If you already know how REMnux works, you can jump to the 25-minute mark to see some of its tools in action. Reach out if you want a copy of the malware sample I used in that demo, so you can recreate the steps.

REMnux includes a variety of malware analysis tools. You can learn about the types of activities you may need to perform by reviewing the categorized listing of the tools on REMnux. For example, you’ll find a variety of tools for static properties analysis, including FLOSS, which helps identify risky API calls in Windows malware, as documented in a Malware Triage post by Xavier Mertens.

Dive Deeper Into Code-Level Analysis

Examining malicious software at the code level is often the most challenging and time-consuming part of the process, especially when you’re disassembling compiled programs. If you try to learn assembly on its own, it might feel too dry and discouraging. For this reason, I recommend getting to know assembly in the context of a debugger.

The following Introduction to Malware Analysis video might feel a bit dated, but is worth your attention. It offers another set of steps (and a malware sample) you can recreate in your lab. Moreover, it provides a glimpse at how you can analyze code dynamically by running the specimen in a debugger. In this demo, I used the now-outdated debugger OllyDbg, but this approach applies to the modern x64dbg alternative:

As this video shows, you can start dynamic code analysis of a Windows executable by setting breakpoints on risky API calls inside a debugger. This can bring you to the code worth examining, at which point you can begin figuring out the logic of the malicious assembly instructions. I listed some API calls worth considering in the Reverse-Engineering Malicious Code cheat sheet. The tool FLOSS, mentioned earlier, can help also help with spotting them.

Some malware you’ll encounter will be designed to evade detection and analysis. You can learn about such techniques, and how to examine them, my video Evasion Tactics in Malware from the Inside Out and review the corresponding slides. One of the tools I use in this demo is x64dbg, so the video will also help you get started with this powerful debugger:

To deepen your understanding of code-level analysis, start experimenting with reviewing malicious code statically, without running it in a debugger. You can do this using Ghidra, which includes a disassembler and decompiler. For an introduction to this powerful tool, see the Code Analysis With Ghidra video by Anuj Soni, and review his accompanying blog post:

Learn Malware Analysis by Examining Malicious Scripts and Document Files

In addition to being able to analyze compiled binaries, you should learn how to examine malicious scripts and documents. Such forms of malware are often used to deliver other malicious payloads and can directly implement the attacker’s logic.

For an overview of the steps you can take to examine malicious scripts, such as JavaScript, watch Evan Dygert’s presentation Shortcuts for Understanding Malicious Scripts and review the accompanying slides and malware samples:

For insights into examining malicious PowerShell artifacts, watch Mari DeGrazia’s talk Finding and Decoding Malicious Powershell Scripts:

To learn about analyzing malicious Microsoft Office documents, you can start with this in-depth video by Didier Stevens, titled Analyzing Malicious Office Documents:

You can learn more about analyzing malicious documents and practice in your own lab by following Ryan Chapman‘s workshop Understanding and Analyzing Carrier Files.

You’ll also find my Analyzing Malicious Documents cheat sheet useful.

Continue Reading, Experimenting, and Learning about Malware Analysis

Reading and watching the malware analysis resources mentioned above will help you learn about malware analysis approaches, but you’ll need to find time for focused, deliberate practice to learn how to apply them. That’s why the tips I mentioned offer pointers to several ways in which you can start practicing malware analysis.

Once you’re ready to go deeper into the field, consider taking the Reverse-Engineering Malware course, which I’ve co-authored and teach along with other experienced security practitioners.

There’s a plethora of articles, blog posts, and videos that can help you get into malware analysis, beyond what I’ve shared with you. You’ll find them with a few online searches, but here are a few additional pointers:

• Cheat Sheet for Analyzing Malicious Software: A reference, written by me, which summarizes the analysis approach outlined above and offers additional tips.
• Shellcode Analysis 101: A presentation by Jim Clausing on examining shellcode, which can be a part of attacks and malicious files.
• The State of Malware Analysis: Advice from the Trenches: A conversation among Jim Clausing, Evan Dygert, Anuj Soni, Jake Williams, and yours truly.
• Free Malware Sample Sources for Researchers: A set of sites that can provide free access to malware for your practice.

-- Lenny Zeltser

Lenny Zeltser is a Faculty Fellow at SANS Institute. He is active on Twitter. This post originally appeared on Lenny Zeltser's blog.