This was transcribed from Jake Williams' webcast on December 14th, 2020. View the full webcast here. You can find the presentation slides here.

Supply chain attacks are not common and the SolarWinds Supply-Chain Attack is one of the most potentially damaging attacks we’ve seen in recent memory. Of course, as it is an evolving situation, we will likely know more as the days progress, but this is what we know as of now.

On December 8 FireEye announced that it had been hacked by a nation-state and since that announcement they’ve been incredibly transparent, publishing information about the breach and what they’ve learned about it in their investigation.

SolarWinds Breach

On December 13 Chris Bing of Reuters broke the story that the US Treasury Department has been compromised by a sophisticated adversary. Shortly after, Ellen Nakashima of the Washington Post confirmed with background sources that the US Treasury breach was perpetrated by the same group that targeted FireEye, that SolarWinds was involved in both breaches, and that it was perpetrated by threat group APT29 (Cozy Bear/Russian SVR).

What Is SolarWinds?

SolarWinds is a software company that primarily deals in systems management tools used by IT professionals. The most widely deployed SolarWinds product is Orion, which is a Network Management System (NMS). Not to be confused with NSM, which in security is a network security monitor.

NMS are prime targets for attackers for a variety of reasons. First, the Network Management Systems must be able to communicate with all devices being managed and monitored so outbound ACLs are ineffective., making it a prime location. Second, many NMS are configured to both monitor for events and respond to them. This means that the Network Management System can make changes on behalf of its configuration. Any changes the NMS can make, the attacker can too. Even when NMS are “monitor only” the credentials used still offer some level of access to the attacker. An attacker who compromised an NMS can usually reshape network traffic for MitM opportunities and can often use credentials for system monitoring to laterally move to target systems.

The Orion NMS has broad capabilities for monitoring and managing systems, including servers, workstations, network devices, etc. Not every organization is going to have SolarWinds configured identically, but when they do have SolarWinds configured, it is definitely a great targeting point for attackers. One reason for this is because in order to monitor systems they have to do some type of system integration. At the very base level, it may be something just as simple as a ping command. That doesn’t really provide any information; either the ping succeeded or it didn’t. What we’re really looking for in the majority of cases is the status of the individual, a communications link.

One of the big takeaways from this event is going to be more involvement between our IT and IT security teams in trying to level set and answer the question: “What risk are we taking on these Network Management Systems?”

Who Uses SolarWinds?

The better question would be, who doesn’t use SolarWinds? They are one of, if not the, Network Management System. SolarWinds is to NMS as Kleenex™ is to tissues. SolarWinds has over 300,000 customers and many of them heavy hitters, much of the US Federal government including the Department of Defense, 425 of the US Fortune 500, and lots of customers worldwide.

How was the SolarWinds Malware Deployed?

The malware was deployed as part of an update from SolarWinds’ own servers and was digitally signed by a valid digital certificate bearing their name. This strongly points to a supply chain attack. The certificate was issued by Symantec with serial number 0fe973752022a606adf2a36e345dc0ed.

We don’t believe the certificate itself was compromised. This isn't a case where we know that an attacker compromised a certificate and is then using it to deploy software or malware through their own channels. In this case, they were actually deploying it through SolarWinds own distribution channels. While the certificate needs to be revoked at some point, revoking the certificate now is unlikely to do a whole lot. That’s what makes it difficult to investigate, but this isn’t the first time we’ve seen a state-backed APT targeting software vendors or masquerading as an update to deploy their malware payloads.

SolarWinds’ Response

SolarWinds has published limited information in which they state they believe the build environment was compromised.

They have identified that these updates were released between March and June 2020 and they believe only 18,000 of its 300,000 Orion customers are impacted by the update. But this all leaves a lot of questions that will hopefully be answered as SolarWinds publishes more data from their internal investigation.

Network IOCs

FireEye has released domains useful for hunting (Discovery COA) if you have DNS logs or full PCAP:

Notice that there’s no overlap in these domains. One of the takeaways from that is that the attacker is absolutely trying to segregate their initial stage from their ongoing stages and this is significant as we start looking at what we need to do to identify these attackers on the network. Another takeaway is that we need better retention on some of our logging. We need to think about looking at the indicator that we get in an incident like this and ask ourselves, “How can we operationalize these? Do we have the logs to operationalize these all the way back to March?”

Attackers Are Sophisticated

We hear “attackers are sophisticated” all the time and in this case, the attackers are definitely sophisticated. This includes sophistication on both the development and operational teams. The development teams deployed anti-analysis countermeasures. The operational teams appear to have used specific infrastructure for each victim, reducing the usefulness of network-based IOCs. FireEye mentioned in its report that a lot of the tooling their seeing in this incident does not share code with other known samples, one of the reasons it still falls in UNC2452 instead of APT29, until they can make that linkage.

Delayed Execution

FireEye notes that the malware checks file system timestamps to ensure the product has been deployed for 12-14 days before it does its first beacon. This effectively prevents the use of malware sandboxes or other instrumented environments to detect it. The 12-14 day waiting period is here specifically to prevent detection in high security environments where pre-deployment testing over hours or days is being done.

Anti-Sandbox Behavior

FireEye also notes that unless the machine is joined to a domain, the malware will not execute. We’ve seen this a lot in different keyed malware samples that are tied to specific environments here because this is being deployed to a lot of different environments, 18,000 by FireEye’s estimate.

DNS Resolution and IP Address Checks

FireEye noted that if the malware resolves a domain to a private IP address, it will not execute. Now, if you've ever used a malware sandbox before that most of the intercept traffic and then reroute that traffic so that they can capture all the commanding any exfiltration data and that kind of stuff. What I like here is that most of these are RFC 1918 IP addresses, they're private IPS, they're the multicast.

Kyle Hanslovan has published a list of paths for this SolarWinds Orion core business layer.dll. There are a lot of different paths this can be in.

Recommendations

If you have SolarWinds Orion, you should assume compromise until more is known. Until more is known, I would not assume that it’s just the published versions that are compromised.

If you have SolarWinds but not Orion, consider mapping your attack surface in case those were also compromised in the supply chain attack.

If you have an NMS other than SolarWinds Orion, don’t rest (yet).

Block access from NMS to the Internet and if it is explicitly needed, limit destinations (think Zero-Trust networking).

Threat hunt in your network and prioritize the Discovery COA (looking backward) over Detection COA (looking forward).

Attackers will be retooling, so don’t anticipate finding specifics for SUNBURST malware.

Monitor for intrusions and log, log, log. Alert on events and investigate as required.

DHS issued an emergency directive to mitigate the compromise outlining required actions. Obviously, if you’re in the private or civilian sector you don’t have to follow these guidelines, but you should. Download the recommendations, read through them and roll from there.

Conclusion

Supply chain compromised will continue. They are extremely difficult to protect against, highlighting the need for security to be considered as part of the vendor selection process. Supply chain compromises do extend SaaS applications. Understand that your SaaS vendor does not have any magic process that makes it easier for them to detect these issues. They are every bit as vulnerable to software supply chain attacks.

As we said before, this is an ongoing situation and we expect updates in the coming days. If you’re following this breach, follow #SolarWinds, #SolarWindsOrion, and #UNC2542. SANS Institute will update sansurl.com/solarwinds if there is any new information to share.