This deep dive is a followup to our executive summary blog.
The latest SolarWinds hack forced organizations to revisit their security posture, only to realize they are not well equipped to defend against nation sponsored attacks. This is not unique. Major tech giants such as Microsoft, Nvidia and Intel, as well as government agencies also discovered that they were breached.
Given that this attack has been ongoing for close to 10 months (since at least March 2020), it brings into question the efficacy of detection technologies. Have all the top tier security controls failed to detect any of the multiple steps of the attack? The answer is: probably not. So why wasn’t the attack discovered sooner?
The truth is that security teams are drowning with alerts. Sometimes it does not matter how good the detection logic is if the SOC is dealing with hundreds or thousands of alerts per day. Even if a “real” alert detects abnormal login (for example), the SOC team usually doesn’t have the capacity to fully investigate whether it was legitimate, or performed with compromised credentials.
Security teams should consider a different approach to security, one that prevents attack vectors instead of simply reporting alerts. So without further adieu, let’s take the eagle eye view of this attack and discover where, and how it could have been prevented.
The latest SolarWinds supply chain attack is also believed to be the greatest we have seen to date. 18,000 companies have updated their SolarWinds Orion platform to the version with the infected DLL containing the malicious backdoor, out of which roughly 50 companies were confirmed to be affected. It will probably take many months, if not years, to fully uncover the full extent of this attack.
The perpetrator demonstrated all the signs of a nation state: covert manual actions, very high operational security, and unconventional techniques to bypass multi factor authentication of their victims. Favoring light malware footprint, the adversary used legitimate credentials for persistence.
The breach was first discovered by FireEye, who believed they were breached by a nation state, and feared that their own tools could be misused by the perpetrators or leaked. FireEye then proceeded to release a repository of detections and countermeasures for their own tools.
As companies rushed to patch their systems against FireEye’s arsenal, the true magnitude of the attack began to unfold. With another post, FireEye revealed that they were victims of a supply chain attack via the SUNBURST backdoor. The post contains a detailed analysis of the backdoor, and the methodology used by the hackers. Additional reports from Volexity and Microsoft gave additional context and recommendations on how to detect and mitigate this threat.
The basis of the attack, like we mentioned earlier, is securing the supply chain vector. In this case the adversaries were able to compromise SolarWinds, gain access to their repositories, inject their backdoor to SolarWinds’ Orion platform and also have it signed, making it appear legitimate.
The attack from a victim’s perspective (not including the SolarWinds compromised) could be broken down into roughly four phases:
The initial compromise occurs once the victim upgrades their SolarWinds Orion software through a poisoned library SolarWinds.Orion.Core.BusinessLayer.dll. The perpetrators went to great lengths to stay hidden during the update phase, so the malicious DLL would not be discovered:
When running, SUNBURST also made sure that identified Antivirus or EDR product were neutralized on the compromised host. Information on the type of AV or EDR, and its status were then communicated back to the C2 infrastructure. This communication can be decoded using NETRESEC tools, which reveals that the attackers were able to bypass existing EDRs by effectively shutting them down:
Supply chain attacks are generally extremely hard to detect, let alone an attack with this level of sophistication. While it is a good practice to harden production servers, this would probably still not have helped in this breach.
Overall, from a security perspective, one should spend their limited resources on other phases of the attack. It is not realistic to establish trust with every line of code of every 3rd party software being used.
The preferred technique used by an attacker, once the backdoor is up and running, is playbook credential theft followed by lateral movement and privilege escalation. For example, one method used by an attacker for lateral movement is using PowerShell to create remote scheduled tasks:
Once the attackers gained credentials that allowed them to use legitimate remote access (VPN, RDP server etc.) they often removed their own backdoors and installed tools to minimize their malware footprint.
The victims of this attack likely have Next Gen AV, EDR solutions and multiple network monitoring tools. Not to mention that some of the victims are lead vendors that actually build this technology! Yet, this group operated close to a year without being detected. What’s going on here? Shouldn’t this be the “bread and butter” of security companies?
Users logging on via remote access technologies, or admins running powershell commands against other endpoints are common activities. Even if detected, the SOC is often ill equipped to investigate whether this activity is malicious, or just another account sharing / VPN access anomaly etc.
To truly prevent lateral movement and malicious remote access, we recommend to:
Implementing the above recommendations will create a “Least Privilege Network”; where entities only have network permissions to what they need, and nothing more. In such an environment, every move of the attacker is blocked, and will trigger a very high fidelity alert for security professionals. The following figure illustrates how this attack could have been block by utilizing the above recommendations:
We have developed a methodology to help organizations measure and implement a least privilege networking stance. Additionally, our Access Orchestrator enforces a ”Least Privilege Networking'' stance out of the box. Once configured, the Access Orchestrator will revoke all network permissions, except ones that are required for legitimate business activity. Network permissions are given to “protected” assets if users can prove their identity via MFA, and have legitimate business reasons for this access.
After the attackers gained access to a privileged account, they proceeded to compromise the Active Directory Federation Services (AD FS) server, stealing (or registering a new) Token Signing Certificate. This certificate contains the private key which can then be used to sign SAML tokens, practically enabling the attacker to impersonate any account to any cloud application that is configured to trust the organization identity provider. A 2017 blog by CyberArk dubbed this attack "Golden SAML".
The active directory, and all the various services and roles connected to it are extremely sensitive, and should be guarded very carefully. Once an attacker gains privileged access to them, we say they have the “keys to the kingdom”, and it is usually considered game over. If malicious activity is detected to such assets, it is not only time for breach investigation, but also for full recovery of the environment. Microsoft supplied a good guide on best practices to monitor and recover from such a compromise.
Our recommendations to protecting the keys to the kingdom are:
Just like the previous phase, if the attackers got this far the security team not only have to investigate and get to the root cause of the breach, they also need to recover all of their secrets and accounts. Once the attackers had all the secret keys, they were able to impersonate any account via the forged SAML token.
The SolarWinds attack is humbling. It affected the best of the best, both companies and government agencies. We have seen the best minds being put together, collaborating to understand exactly how this adversary operates, and how to better defend ourselves.
This joint work unveiled a patient adversary, with the means and technical capabilities to orchestrate this sophisticated operation. At the same time, many techniques were textbook operations, and we should ask ourselves why our current approach failed to prevent them.
The answer, for the author of this blog, is clear: the security industry is too invested in clever ways to detect specific techniques (for this adversary and others). Detection’s main flaw is that it doesn’t stop the attack, and it generates too much “noise” for security teams. Our goal in the near future should be to focus on all the wise ways to prevent such attacks, before even requiring an investigation to occur.