This deep dive is a followup to our executive summary blog.
A Clever Person Solves a Problem. A Wise Person Avoids It.
The latest SolarWinds hack forced organizations to revisit their security posture, only to realize they are not well equipped to defend against nation sponsored attacks. This is not unique. Major tech giants such as Microsoft, Nvidia and Intel, as well as government agencies also discovered that they were breached.
Given that this attack has been ongoing for close to 10 months (since at least March 2020), it brings into question the efficacy of detection technologies. Have all the top tier security controls failed to detect any of the multiple steps of the attack? The answer is: probably not. So why wasn’t the attack discovered sooner?
The truth is that security teams are drowning with alerts. Sometimes it does not matter how good the detection logic is if the SOC is dealing with hundreds or thousands of alerts per day. Even if a “real” alert detects abnormal login (for example), the SOC team usually doesn’t have the capacity to fully investigate whether it was legitimate, or performed with compromised credentials.
Security teams should consider a different approach to security, one that prevents attack vectors instead of simply reporting alerts. So without further adieu, let’s take the eagle eye view of this attack and discover where, and how it could have been prevented.
Chain of Events
The latest SolarWinds supply chain attack is also believed to be the greatest we have seen to date. 18,000 companies have updated their SolarWinds Orion platform to the version with the infected DLL containing the malicious backdoor, out of which roughly 50 companies were confirmed to be affected. It will probably take many months, if not years, to fully uncover the full extent of this attack.
The perpetrator demonstrated all the signs of a nation state: covert manual actions, very high operational security, and unconventional techniques to bypass multi factor authentication of their victims. Favoring light malware footprint, the adversary used legitimate credentials for persistence.
The breach was first discovered by FireEye, who believed they were breached by a nation state, and feared that their own tools could be misused by the perpetrators or leaked. FireEye then proceeded to release a repository of detections and countermeasures for their own tools.
As companies rushed to patch their systems against FireEye’s arsenal, the true magnitude of the attack began to unfold. With another post, FireEye revealed that they were victims of a supply chain attack via the SUNBURST backdoor. The post contains a detailed analysis of the backdoor, and the methodology used by the hackers. Additional reports from Volexity and Microsoft gave additional context and recommendations on how to detect and mitigate this threat.
Phases of The Attack
The basis of the attack, like we mentioned earlier, is securing the supply chain vector. In this case the adversaries were able to compromise SolarWinds, gain access to their repositories, inject their backdoor to SolarWinds’ Orion platform and also have it signed, making it appear legitimate.
The attack from a victim’s perspective (not including the SolarWinds compromised) could be broken down into roughly four phases:
- Initial Foothold: The malicious DLL was installed by customers that wanted to update their Orion Platform. After a dormant period of about two weeks, the backdoor code sprung into action, communicating with the C2 infrastructure.
- Lateral Movement & Remote Access: The backdoor allowed a remote operator to, according to Microsoft, “follow the standard playbook of privilege escalation exploration, credential theft, and lateral movement hunting for high-value accounts and assets”. FireEye also mentioned that “This actor prefers to maintain a light malware footprint, instead preferring legitimate credentials and remote access for access into a victim’s environment.”
- Keys to the Kingdom: The ultimate goal of these hackers seemed to be gaining access to desired accounts’ emails. The way they got their way is by stealing the ADFS private key. With this key they were able to forge their own SAML assertions, which bypassed the need for credentials or even MFA protections.
- Exfiltration: After the attackers gained the ability to impersonate any user (including bypassing their MFA) they accessed their cloud applications and exported mailboxes for target accounts. They sometimes configured a trust relationship between their tenant and the victims - to be used for later access.
Initial Foothold
The initial compromise occurs once the victim upgrades their SolarWinds Orion software through a poisoned library SolarWinds.Orion.Core.BusinessLayer.dll. The perpetrators went to great lengths to stay hidden during the update phase, so the malicious DLL would not be discovered:
- The DLL is signed with a valid SolarWinds certificate
- The changed code is minimal, and appears legitimate
- Backdoor waits for 2 weeks before establishing C2 communication
- Backdoor code doesn’t share any binary similarity or code reuse with known malware repositories
- C2 traffic imitates Orion protocols and domains
When running, SUNBURST also made sure that identified Antivirus or EDR product were neutralized on the compromised host. Information on the type of AV or EDR, and its status were then communicated back to the C2 infrastructure. This communication can be decoded using NETRESEC tools, which reveals that the attackers were able to bypass existing EDRs by effectively shutting them down:
Supply chain attacks are generally extremely hard to detect, let alone an attack with this level of sophistication. While it is a good practice to harden production servers, this would probably still not have helped in this breach.
Overall, from a security perspective, one should spend their limited resources on other phases of the attack. It is not realistic to establish trust with every line of code of every 3rd party software being used.
Lateral Movement & Remote Access
The preferred technique used by an attacker, once the backdoor is up and running, is playbook credential theft followed by lateral movement and privilege escalation. For example, one method used by an attacker for lateral movement is using PowerShell to create remote scheduled tasks:
Above: taken from Volexity
Once the attackers gained credentials that allowed them to use legitimate remote access (VPN, RDP server etc.) they often removed their own backdoors and installed tools to minimize their malware footprint.
The victims of this attack likely have Next Gen AV, EDR solutions and multiple network monitoring tools. Not to mention that some of the victims are lead vendors that actually build this technology! Yet, this group operated close to a year without being detected. What’s going on here? Shouldn’t this be the “bread and butter” of security companies?
Users logging on via remote access technologies, or admins running powershell commands against other endpoints are common activities. Even if detected, the SOC is often ill equipped to investigate whether this activity is malicious, or just another account sharing / VPN access anomaly etc.
To truly prevent lateral movement and malicious remote access, we recommend to:
- Block all client to client network access, beside exceptions
- Enable IT admins client to client access, only after they verify their identity via MFA
- Validate and enable only legitimate server to server traffic (for only required ports)
- Enable network access only between client groups and servers for groups that actually require that access - everything else should be blocked
- Remote access should only be enabled after user provides MFA
- Remote access should be restricted to endpoints the user owns or allowed to access
Implementing the above recommendations will create a “Least Privilege Network”; where entities only have network permissions to what they need, and nothing more. In such an environment, every move of the attacker is blocked, and will trigger a very high fidelity alert for security professionals. The following figure illustrates how this attack could have been block by utilizing the above recommendations:
We have developed a methodology to help organizations measure and implement a least privilege networking stance. Additionally, our Access Orchestrator enforces a ”Least Privilege Networking'' stance out of the box. Once configured, the Access Orchestrator will revoke all network permissions, except ones that are required for legitimate business activity. Network permissions are given to “protected” assets if users can prove their identity via MFA, and have legitimate business reasons for this access.
Keys to the Kingdom
After the attackers gained access to a privileged account, they proceeded to compromise the Active Directory Federation Services (AD FS) server, stealing (or registering a new) Token Signing Certificate. This certificate contains the private key which can then be used to sign SAML tokens, practically enabling the attacker to impersonate any account to any cloud application that is configured to trust the organization identity provider. A 2017 blog by CyberArk dubbed this attack "Golden SAML".
The active directory, and all the various services and roles connected to it are extremely sensitive, and should be guarded very carefully. Once an attacker gains privileged access to them, we say they have the “keys to the kingdom”, and it is usually considered game over. If malicious activity is detected to such assets, it is not only time for breach investigation, but also for full recovery of the environment. Microsoft supplied a good guide on best practices to monitor and recover from such a compromise.
Our recommendations to protecting the keys to the kingdom are:
- Privileged access to domain controllers and all of their respective services should be
- only allowed from a limited set of endpoints
- require strong authentication for every access
- Every change to an on prem or cloud directory should be
- audited
- approved by a second privileged account whenever possible (to protect against compromised identity and malicious insiders)
- Any access to credentials, private keys on sensitive servers should be closely examined for anomalies
Exfiltration
Just like the previous phase, if the attackers got this far the security team not only have to investigate and get to the root cause of the breach, they also need to recover all of their secrets and accounts. Once the attackers had all the secret keys, they were able to impersonate any account via the forged SAML token.
Our recommendation:
- There are several techniques to detect forged SAML tokens, we recommend implementing them in your environment.
- Even though MFA authentication was bypassed in this attack, it is still good practice to enforce MFA throughout your applications (the next breach might use a different technique)
Summary
The SolarWinds attack is humbling. It affected the best of the best, both companies and government agencies. We have seen the best minds being put together, collaborating to understand exactly how this adversary operates, and how to better defend ourselves.
This joint work unveiled a patient adversary, with the means and technical capabilities to orchestrate this sophisticated operation. At the same time, many techniques were textbook operations, and we should ask ourselves why our current approach failed to prevent them.
The answer, for the author of this blog, is clear: the security industry is too invested in clever ways to detect specific techniques (for this adversary and others). Detection’s main flaw is that it doesn’t stop the attack, and it generates too much “noise” for security teams. Our goal in the near future should be to focus on all the wise ways to prevent such attacks, before even requiring an investigation to occur.