#Solorigate : SolarWinds SUNBRUST backdoor investigation using ShiftLeft’s Code Property Graph

Chetan Conikee
ShiftLeft Blog
Published in
8 min readDec 28, 2020

--

If you’ve arrived to this post, I’d suggest reading the Part-1 and Part-2 to gain context.

FireEye released additional details here (on December 24th, 2020) that is well worth reading.

With the increase of complexity in software and the availability of complex and customizable malware, the amount of work required by a malware analyst to properly analyze all incoming malware exceeds what is possible by a human. Additionally, malware developers continuously make new versions of their malware, adding or changing functionality or complexity.

There currently exists several methods for computer assisted malware classification, using both static and dynamic analysis techniques (as described in prior post here). Most static methods rely on signature matching, and are therefore weak at detecting new or obfuscated malware. The methods relying on dynamic techniques usually involve executing the malware in a controlled environment while observing it, trying to detect some behavioral signatures.

These methods are better at discovering new malware or new versions of old malware, but are resource intensive and slow, as they need to set up a new controlled environment for every malware sample and execute them. Additionally, it is not always trivial to get the malware to execute, due to environmental dependencies and deliberate anti-analysis techniques.

Fingerprinting malicious actors

Typically such actors conform to their own habits and patterns over time. To fingerprint the actor of an exploit, a system can model detection based on a feature set that is extracted from the malware sample

  • Coding style
    - Implementation pattern (Sys-call wrapping, Assembly inlining, crypto implementation)
    -
    Framework implementation style (structure for decision flow, variable definitions, naming conventions, ..)
    -
    Techniques (Exfiltration techniques, Lateral movement/Elevation techniques, Heap Spraying techniques)
    - … and many more

Scenarios in which automatic malware classification is necessary is when an incident team must quickly react to and assess the damages of an incident. In many cases, after detecting the malware infection, critical time is lost by analyzing the malware to determine the impact of the incident. Initial knowledge about the malware’s capabilities (based on past information), received from automated systems, can save valuable time and enable incident response personnel to quickly perform the correct actions.

Correctly classifying incoming malware enables analysts to prioritize their effort, and to assess potential damages.

The #SUNBURST supply chain attack was designed in a very professional way — with a clear focus on staying undetected for as long as possible.

After laterally moving into SolarWinds SDLC (Software development lifecycle) the adversary studied the team’s coding style (good and bad) and thereafter blended his/her implementation style by maintaining an illusion of code legitimacy to conform to that of the solar-winds team.

It was evident that the malware managed to escape peer code review, SAST and build-release gates successfully. Why?

Let’s piece this apart and study every line of code to determine if this was detectable using ShiftLeft Code Property Graph.

Disassembling the malware

I was able to grab a critical subset of disassembled malicious DLL (SolarWinds.Orion.Core.BusinessLayer.OrionImprovementBusinessLayer).

The disassembler used in this case is DNSpy

Malicious DLL Name : SolarWinds.Orion.Core.BusinessLayer.dll
Version MD5: b91ce2fa41029f6955bff20079468448

Disassembling , De-obfuscating and Unpacking in a single pass using de2dot (optional step)

The research and reverse-engineering community also use de4dot to disassemble and de-obfuscate in a single pass.

I will not recommend doing this as our goal is to analyze the code it’s obfuscated state and examine intent via decision flow. Only then, will we be able to automate this task and further-on build out a classifier to detect questionable intent.

Some of the community members disassembled and de-obfuscated SUNBURST using de2dot (archived and inactive project) and to my pleasant surprise, they were able to uncover malice in clear-text embedded in reversed code.

Obfuscated Backdoor strings in SolarWinds.Orion.Core.BusinessLayer.dll
De-Obfuscated Backdoor strings in SolarWinds.Orion.Core.BusinessLayer.dll via de4dot

This was possible with SUNBURST but not always possible with other obfuscating techniques. Often hashing techniques are designed to be non-reversible.

Hash techniques often use a combination of Bit-dependency, Avalanching and Non-linearity in order to make reversing computationally intensive or impossible.

Navigating the workflow of SUNBURST (abridged Summary)

Investigating backdoors

Investigating backdoors are non-trivial. I have written extensively on this topic exactly a year ago. An insider can masquerade/blend-in as a typical employee to gain trust and privilege over time. In this duration, he/she gains situational awareness in order to subvert systems that can trace their actions. In cases like #SUNBURST, the attacker would have checked-in code incrementally over a period of time in order to weaponize the malware.

We at ShiftLeft have been extending our policies to support backdoor detection for over a year.

Play-by-Play

Step-#1 : Triggering the backdoor at a scheduled interval

Any application often sets long running tasks to the background (in a separate thread), so it’s natural to consider this query as a false positive.

Detect using ShiftLeft NGSAST Policy

https://github.com/conikeec/sunburst-analysis#spawning-threads-as-a-background-task-could-be-both-benign-and-suspicious-at-the-same-time-especially-if-the-instance-spawned-is-dealing-with-obfuscated-hashes

Step-#2 (Execute-Or-Not check) : Process affinity check

Detect using ShiftLeft NGSAST Policy

https://github.com/conikeec/sunburst-analysis#affinity-checks-to-determine-name-of-current-running-process--is-always-suspicious

Step-#2 (Execute-Or-Not check) : Dormant state ~ 12 to 14 days after running for first time

Detect using ShiftLeft NGSAST Policy

https://github.com/conikeec/sunburst-analysis#conducting-date-interval-checks-in-a-control-loop-is-always-suspicious-especially-if-the-control-block-is-executing-obfuscated-commands

Step-#3 (Execute-Or-Not check) : Create a named pipe, supporting both synchronous and asynchronous read and write operations and conduct further checks

Detect using ShiftLeft NGSAST Policy

https://github.com/conikeec/sunburst-analysis#creating-a-namedpipeserver-stream-for-asyncsync-rw-operations-is-a-red-flag-get-all-methods-creating-a-named-pipe

Step #4 : Domain Checks

  • Get domain name mapped to host and if name is not valid, stay dormant
  • Compare domain name against predefined blacklist (includes hashed domain names, internal test domains of solarwinds).
  • Call function createUserId that conducts the following checks
    i) Get MAC address and check if interface is UP and not bound to loopback
    ii) Get GUID from Registry (HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography) and append to USER_ID
    iii) Create Hash of USER_ID and sleep for RAND interval

Step #5 : Process Checks

This is a very interesting anti-detection verification that needs further attention.

SearchAssemblies — verifying if malware analysis and reversing tools (processes) are running on host. This is compared against a list of obfuscated 137 names that includes x64-dbg, apimonitor-x64, etc

SearchServices and SearchConfigurations — This function issues a command `Select * From Win32_SystemDriver` to verify for system monitoring processes running on host that includes LogRhythm System Monitor, ProcMon, Dell SecureWorks, etc

Upon passing these checks, the workflow continues to execute

Detect using ShiftLeft NGSAST Policy

https://github.com/conikeec/sunburst-analysis#sunburst-checks-for-the-following-running-processes-on-host-to-determine-dormancy-or-execute

Step-#6 : Backdoor created and communication with C&C (Command and Control) begins

  1. IP_ADDRESS of api.solarwinds.com is resolved and verified against a set of hard coded IP_ADDRESS/SUBNETS
  2. DNS Query to avsvmcloud.com to resolve IP_ADDRESS and based on the IP_ADDRESS the trojan adapts its behavior.
    if IP_ADDR == 87.238.80.X { THEN create HTTPS worker for handling responses from C&C server } ...

Step-#4 : Response Handling from C&C server

The backdoor enables 18 set of commands (after HttpsWorker thread is spawned) which enables total and complete control of the system

Noteworthy

The attacker employed FNV-1a (Fowler Noll Vo) + XOR class of hash algorithms to obfuscate all of the hardcoded literals in the code base.

Why choose FNV-1a?

FNV-1a comes with zero memory overhead (no precomputed look-up tables), is an incremental (rolling) hash and has a good avalanche behavior. On an average it needs at about 4.5 CPU cycles per hashed byte, ~ 700 MB/second.

The attacker also choose naming conventions (functions, variables) that effectively blended amongst the rest of the code committed over time.

Connecting backdoors : The mutation of an “new” attack pattern — one backdoor opens several others

While analyzing the recent SolarWinds supply-chain attack security researchers have found a second backdoor, suggesting involvement of another hacker group, unrelated to the suspected government-backed threat actor that compromised SolarWinds.

Tracked as Supernova, the backdoor is a memory resident web-shell injected into SolarWinds Orion code that would allow threat actors to execute arbitrary code on systems running the compromised version of Orion. Supernova web shell was used to download, compile and execute a malicious PowerShell script (dubbed CosmicGale by some researchers).

How can we keep up with this mutation?

Conclusion

Reversing the binaries for SUNBURST, we’ve learned that even this wildly successful operation has its rough edges.

The building blocks of expressions to detect these patterns via ShiftLeft backdoor policy expressions are linked here

We at ShiftLeft have been studying and provisioning backdoor/insider detection policies using code property graph since mid 2019. Speak to us and we can help assess and recommend more efficient processes and procedures.

--

--

Engineer, InfoSec tinkerer, Seed Investor, Founder/CTO of ShiftLeft Inc., (Opinions, my own)