Prevention Engineering using LDAP and RPC Firewall: DEATHCon 2024

Summary

Prevention engineering means focusing on attacks you can actually block and audit, rather than only detecting them after the fact. In this DEATHCon 2024 workshop, Zero Networks researchers Sagie Dulce and Dekel Paz show how two free, open-source tools, the LDAP Firewall and the RPC Firewall, run on domain controllers to audit and block the LDAP and RPC operations behind BloodHound enumeration, Kerberoasting, shadow credentials, DCSync, PsExec, and NTLM coercion. They walk through installing each tool, baselining in audit mode, and switching rules to block.

Key takeaways

• Prevention engineering puts effort into attacks you can block and audit, such as enumeration, coercion, and DCSync, instead of detections you can’t act on in time.
• LDAP must stay open on domain controllers, so the LDAP Firewall audits every LDAP operation and can block by operation, attribute, user, group, or source IP.
• RPC underpins most Windows services and many attacks. The RPC Firewall is more granular than Windows’ built-in RPC filters, which are still needed for protected processes.
• Both tools evaluate rules top to bottom, first match wins. Start in audit mode, baseline legitimate traffic, then turn the default rule into a block.
• Both tools are free on GitHub at github.com/zeronetworks. The LDAP Firewall requires Windows Server 2016 or later.

What does this workshop cover?

[00:00] Sagie Dulce: Hello and welcome, everybody, to our workshop, Prevention Engineering Using Our Open Source Tools: the RPC Firewall and the LDAP Firewall. My name is Sagie Dulce, and I’ve been doing security research for the past couple of decades. I’ve covered all sorts of topics, from data protection through containers and network security to reverse engineering, and in recent years I’ve been running a lot of open source tools, some of which we’ll see in this presentation.

[00:36] Dekel Paz: I’m Dekel Paz. I’ve been in the IT and cybersecurity field for the last 15 years, working on both offensive and defensive security projects throughout my career. Together with Sagie, I developed various open source tools, including the LDAP Firewall, which I’ll present today.

[00:53] Sagie Dulce: This is going to be a session with a lot of hands-on experience. But first, I want to start with a demo that shows you the why: what’s interesting about our open source tools and how many attacks they can help you prevent. Then we’ll talk about prevention engineering, what it is, and why you might want to use it in your environment. Then we’ll walk you through each tool, the LDAP Firewall and the RPC Firewall, on domain controllers, and show you how to configure them to stop many different forms of attack. Let’s get started with the demo.

Demo: Can the LDAP Firewall and RPC Firewall stop real attacks?

[01:47] Dekel Paz: Now we’ll run some attacks from our attacker machine. We’ll start with BloodHound, using the SharpHound collector, which is usually the first thing an attacker does. We’re getting no results back. Even though any user should be able to run this, we get nothing, because the LDAP Firewall is stopping all of these queries. Next, we’ll run GetUserSPNs, which is used for Kerberoasting attacks. Even though some users have SPNs, we get no results from this tool either. Lastly, we’ll run Whisker, which is used for shadow credentials attacks and works by modifying an attribute. Again, the modification is blocked.

[02:36] If we open the Event Viewer and refresh it, we can see what these queries look like behind the scenes. First, we see some legitimate searches. They weren’t blocked, and everything is working correctly. Then we see a search that was blocked: the BloodHound search, with all the attributes it queries for. Even though we’re running as a high-privilege user, the search was blocked because it’s associated with BloodHound. Next is the GetUserSPNs operation; you can see the servicePrincipalName attribute in the query. And finally there’s msDS-KeyCredentialLink, which is part of the shadow credentials attack. That modification was also blocked.

[03:23] Sagie Dulce: Now let’s look at how the RPC Firewall blocks many types of attacks. We’ll start with Mimikatz, attempting a DCSync attack. Even though I’m a domain admin, I get access denied. The second attack is executing code remotely using PsExec. Again, access denied. Last, I’ll run the Coercer tool, which uses many different RPC interfaces and tries to create an NTLM relay attack through each one. The output shows all the interfaces it tries, and for each one, it gets access denied.

What is prevention engineering?

[04:31] Sagie Dulce: Now that you’ve seen prevention engineering in action, you probably have a good idea of what it is. It’s not rocket science. Prevention engineering means deciding whether you want to stop an attack, or, more accurately, whether you can stop it. If you can’t, it may not be worth doing anything. With classic detection engineering, you may collect a lot of logs and write a lot of rules. That can be all well and good, but the question is whether you get enough value from it compared to the effort you put in.

[05:18] Often the answer is no. The effort that goes into detection engineering can be huge, but eventually you have to ask: can it actually block an attack? Would I respond? Maybe I’d only respond hours or even days later, and by then the detection may not be worth the effort. Attacks like DLL injection are very hard to detect and even harder to block. Credential dumping is the same. And C2 communication is notoriously hard to detect or block.

[05:57] But for some attacks, the answer is yes: you can block them. Some of them we just saw in the demo, like enumeration, coercion attacks, and DCSync. If you can block them, it’s worth the time and effort, so go ahead and block them. Just as important, create an audit trail that helps you pinpoint who did what and isolate the attack further. That, in a nutshell, is prevention engineering: focusing on prevention, blocking, and auditing, rather than only detecting attacks after the fact.

What is LDAP, and why do attackers target it?

[06:42] Dekel Paz: LDAP is the protocol that lets you interact with Active Directory, reading or writing information about users, computers, groups, permissions, and more. LDAP is used constantly in Windows environments. For example, say you open Outlook and search for me by typing “Dekel.” Behind the scenes, an LDAP search operation goes to the domain controller and queries for any user that matches the string “Dekel.” LDAP supports many operations: search, add, modify, delete, and so on.

[07:28] The LDAP directory is structured like a set of folders. At the top is the company domain, in this example company.com. Under that are organizational units, or OUs, which work like folders. In this example, there’s a London OU, a New York OU, and a Paris OU, representing different company branches. You can create any OUs you want. Under each OU are objects: users such as John Doe, groups, computers, and so on.

[08:05] To refer to a specific object, we use a distinguished name, or DN. DNs are long strings, and they’re usually easiest to read from right to left. At the end, we see DC=com, and before that, DC=company. Those are the domain components, so we’re talking about the company.com domain. Before that is OU=London, the London OU. And before that is the common name, John Doe. So this DN refers to the user John Doe in the London OU of the company.com domain.

[08:49] Each object has attributes, which are fields that hold data. In this example, there’s a given name, John; a last name, Doe; a telephone number; a mail address; and the object class, which specifies what kind of object this is. There are many more attributes; this is just a small sample. Windows offers several ways to view them. One is the Attribute Editor in the Active Directory Users and Computers tool. Another is PowerShell: Get-ADUser returns some of the relevant attributes, though not all of them.

[09:46] Why does LDAP matter? The main issue is that LDAP has to stay open and available on your domain controllers all the time. You can’t firewall it off. That means anybody in your organization can at least read sensitive information, and many users also have rights to modify objects or perform other operations. And in Windows, it’s very hard to audit and control this. You can try to manage it with ACLs and user permissions, which get very convoluted, and there’s no easy way to get audit logs.

[10:35] So on one side, you have a protocol that must stay open, and on the other, many very sensitive attributes. For example, nTSecurityDescriptor holds information about ACLs and permissions. Attributes like msDS-AllowedToActOnBehalfOfOtherIdentity and msDS-AllowedToDelegateTo are used by tools like BloodHound to build their attack graphs. Attackers also like to modify attributes such as servicePrincipalName and msDS-KeyCredentialLink, and they can even create or delete objects. LDAP sits behind many attacks and attack tools: BloodHound does much of its discovery over LDAP, and Kerberoasting, shadow credentials, and evil GPO attacks all use LDAP behind the scenes.

How does the LDAP Firewall work?

[11:33] Dekel Paz: This is why we created the LDAP Firewall. Because LDAP is so hard to control natively in Windows, we built an open source tool that you install on the domain controller. It does two things. First, it audits all LDAP operations, logging the operation type, attributes, filters, the user who performed the action, and network information such as the IP address and port. Second, it lets you create rules to allow or block operations. For example, you can say, “Only this group of users can perform this type of operation,” or “Only this IP can perform specific operations against my domain controller.” We’ll look at real-world configuration examples later.

[12:38] The configuration works like a classic firewall: rules are evaluated one by one until there’s a match. Each rule has fields. The operations field specifies which operations the rule applies to, for example only modify operations. There’s IP, which specifies the source; users; groups; and LDAP-specific fields like DN and attributes. So, for example, I can create a rule that prevents anyone from modifying specific attributes.

[13:24] Before that rule, I might add one that always allows operations coming from my management server or a management user. After that, no other user can modify the sensitive attribute. And after everything else, I audit all remaining requests, so I don’t break anything but still get logs for everything that happens over LDAP. Once it’s running, events appear in the Windows Event Viewer. At the top of each event is the operation type, in this example an LDAP add operation creating a new object. At the bottom is the network information, showing which IP performed the operation, and the user. Here, an administrator account tried to add a new computer, and the configuration blocked it.

[14:29] There’s a lot of information out there about LDAP and LDAP attacks, and I’ve curated some useful links if you want to read more. Our blog at zeronetworks.com also has several posts on stopping specific attacks with the LDAP Firewall.

How do you install and configure the LDAP Firewall?

[15:05] Dekel Paz: Getting started with the LDAP Firewall is easy. Go to our GitHub page at github.com/zeronetworks/ldapfw. You’ll find all the files, the releases, and plenty of information on the README page, including links to blog posts on using the LDAP Firewall, deep dives on LDAP-based attacks and how to stop them, and video tutorials, including one from a previous DEF CON. The README also covers the different ways to use the LDAP Firewall and how to prevent specific attacks.

[15:51] You can also look at the example config folder, which has configs for different scenarios. Probably the most comprehensive one is the DACL baselining and prevention config, which you can use to baseline and block DACL-based attacks. Some of its rules are in the config on our lab server, which we’ll look at shortly. To get started, go to the releases page, download the latest zip file, and copy it to a domain controller running Windows Server 2016 or later.

[16:41] Here we are on the domain controller. This is from the DEF CON workshop lab, and the files are already here with the LDAP Firewall running. First, we run the LDAP Firewall manager with /status, which shows that the LDAP Firewall is already installed. If it weren’t, all you’d need to do is run /install, and it installs everything.

[17:15] Now let’s look at the configuration on the server, in the config.json file. You can open it in any text editor. It’s built like a classic firewall, with a list of rules. Whenever an LDAP operation comes in, the LDAP Firewall checks the rules one by one, and the first rule that matches decides what happens.

[17:52] The first rules in this config allow, without auditing, any operation performed locally on the domain controller, matched by user. Many LDAP searches and other operations happen locally behind the scenes, and auditing them would create a lot of noise. And if an attacker is already on the domain controller, it’s game over anyway, so we might as well ignore those. There are similar rules matched by IP, and we also allowlist the other domain controllers so they can communicate with each other without issues.

[18:40] This config follows the prevention engineering approach of baselining first, so there are no block rules yet. They’re all audit rules. The goal is to reach a baseline where we only log attacks, or activity we suspect is an attack. Next is a rule to detect SharpHound, using the SharpHound filter. Then there’s a list of attributes BloodHound uses, such as adminCount, msDS-AllowedToDelegateTo, and pwdLastSet, with a subtree scope. When BloodHound queries the domain controller, it typically runs a subtree search requesting all of these attributes. So if any of them matches, we get a log entry, and there’s a good chance BloodHound is running.

[19:40] After that, a similar rule catches modifications to attributes used in attacks; our blog has more on each of them. If any of these attributes is modified remotely, that’s usually a reason for concern. If the change happens locally on the domain controller, it matches the earlier local rule and never reaches this one, so it won’t break anything. If you have management servers or tools that legitimately make these changes, you can allowlist them by adding a rule before this one. In this case, we’re only auditing.

[20:31] Next are rules to catch Kerberoasting and to ignore searches we know aren’t malicious. Base searches, for example, are very limited in scope, so we don’t audit them, which cuts noise. A few more rules suppress other activity we don’t need to audit; some are specific to this lab and some are generic. Finally, a catch-all rule allows and audits everything else. Any operation that doesn’t match a known query is always allowed and always audited, so we can investigate it later.

[21:22] With this config running, we can open the Event Viewer, go to Applications and Services Logs, and select LDAP Firewall to see what an event looks like. This one is an LDAP search operation. The user who performed it is a-fw, and the action was allowed. The event shows the DN the search was performed from, the one-level scope, the filter specified, the attributes requested, and the IP address that sent the query.

How do you move from audit mode to block mode?

[22:01] Dekel Paz: Now let’s say we want to move into prevention mode, starting with the BloodHound fingerprinting rule. All we need to do is add "action": "block" to that rule. From then on, any query that performs a subtree search for one of these attributes will be blocked. To apply the change, we go back to the command prompt and run the update command. Once that’s done, the configuration is updated, and those requests are blocked instead of just audited.

[23:15] If you’re using Microsoft Sentinel in the lab, we created a function called GetLDAPFWEvents. Run it to see the queries from the last 24 hours, already parsed into their individual fields. You can also look at how the function is built. It takes the parameters from the event log, adds some extra information, and parses the data based on the operation type, since each operation carries different data. Then it creates a column for each field and returns the results.

What is RPC, and why do attackers target it?

[24:13] Sagie Dulce: Now that we know how to set up the LDAP Firewall for prevention engineering, let’s do the same for the RPC Firewall. RPC stands for remote procedure call. It’s what it sounds like: a client calls a procedure on a server, just as you would call a function. In Windows, this happens through the RPC mechanism. The server can be in the same process, in a different process, or on a different computer, in which case the RPC call goes over the network.

[24:55] Here’s a practical example from Microsoft’s documentation for the Remote Registry protocol. The documentation lists the interface’s unique identifier, or UUID, which tells you the call is going to the Remote Registry interface. Each function in that interface has its own operation number, or opnum. Together, the UUID and opnum tell the server exactly which interface and which function the client is calling. These RPC calls can travel over SMB, TCP, HTTP, and other transports.

[26:02] What many people don’t realize is that RPC is used by almost every Windows service: printing, task scheduling, domain controller management, user management, directory replication, and remote management through DCOM or WMI, among many others. If all of these functions can be called remotely, that’s very attractive to attackers, which is why we see RPC used in attack after attack: Certified Pre-Owned attacks, remote scheduled tasks for lateral movement, the DCSync attack we saw earlier, PsExec, and many more. So we want a way to protect against these attacks.

How is the RPC Firewall different from Windows RPC filters?

[27:00] Sagie Dulce: That’s what the RPC Firewall is for. It wasn’t the first RPC firewall. Microsoft already has a built-in mechanism, RPC filters, that can block specific RPC calls, so someone did think it was a good idea when RPC was developed. But RPC filters have problems. They can be buggy, so you can’t always rely on them, and they aren’t as granular as the RPC Firewall. Their only advantages are that they’re built in and that they can protect protected processes.

[27:45] The RPC Firewall exposes both mechanisms, RPC filters and the RPC Firewall itself, so you can use either or both depending on your configuration. The bottom line: unless a process runs as a protected process, there’s little reason to use RPC filters. The RPC Firewall is more granular, supports wildcards, and is better at identifying the client’s source IP during the RPC call. So use the RPC Firewall, and use RPC filters only for protected processes.

[28:28] The RPC Firewall relies on a simple configuration file that tells it which RPC calls to audit and which to allow or block. For example, you can tell it to audit only specific protocols and opnums on a specific UUID, meaning a specific RPC interface. Or you can block all RPC calls from a specific SID, the security identifier of a user or group in your environment. You can build almost any configuration you need.

[29:22] Rules are evaluated from top to bottom. Each line is a rule, like a firewall rule. The RPC Firewall checks each one; if there’s a match, it makes a decision, and if not, it moves to the next rule. If you enable auditing on an RPC filter or RPC Firewall rule, it generates audit events. RPC Firewall events are more detailed and easier to read than the built-in Windows security events for RPC. They show the correct client network address and port, the operation number that was called, and authentication information in a more readable form.

[30:16] To read more about RPC and the RPC Firewall, see our GitHub page, a recent blog post we wrote about Certified Pre-Owned attacks, and a couple of videos. You can get started from there.

How do you deploy the RPC Firewall?

[30:40] Sagie Dulce: Now let’s get to prevention engineering with the RPC Firewall, much as we did with the LDAP Firewall. It’s even simpler. To get started, go to the GitHub page. I recommend reading through it, because it has a lot of useful information. We’ve covered most of it in this workshop, but some of it may be new to you.

[31:22] The page also links to videos we’ve made, including our Black Hat talk, and to our Slack community, where you can join other people using the RPC Firewall and LDAP Firewall to ask questions and share information. I highly recommend it. To download the RPC Firewall, go to the releases page.

[32:00] I’ve already downloaded and installed the RPC Firewall on one of the lab’s domain controllers. The first command I run on every RPC Firewall installation is the status command. It shows the status of RPC filters and the RPC Firewall separately. In this case, neither is installed, and there are no relevant filters or firewall configuration. Even though we haven’t installed anything, RpcMessages.dll is present. That happens when multiple users are on a server and one of them holds the DLL, which makes it hard for the RPC Firewall manager to delete. It’s not a problem: the DLL rarely changes, and it’s fine for it to be there.

[33:04] After checking the status, I recommend running the interface command. It lists every RPC interface that RPC servers are listening on remotely, usually over TCP or SMB, along with detailed information about each one. For example, process 528 listens on two bindings: TCP port 135, and a named pipe for the endpoint mapper, which runs over SMB. You can also see the interfaces this process is listening on.

[34:01] The output also shows that process 528 runs unprotected, so we can defend it with the RPC Firewall module. That isn’t true of every process. Another process in the list runs with WinTcb-Light protection, which means we can’t inject the RPC Firewall into it, so we have to use RPC filters to protect its interfaces. Further down, we see that only two processes running RPC servers are protected. We can protect every other process with the RPC Firewall.

[34:54] Normally, you can start with the built-in configuration that comes with the RPC Firewall download. This time, we’ll use a slightly more advanced configuration from one of the templates on our GitHub page. There are templates for filters only, for the RPC Firewall only, and for both. I recommend building your configuration from scratch, because it will be more tailored to your environment. But a template is a good starting point for testing, or if you want something that already works for more than 90% of use cases. Here, I’ll use the template that configures both the RPC Firewall module and the RPC filters module.

[36:03] The configuration starts with a set of interfaces we simply allow. These are benign interfaces we’ve seen in many environments, with no well-known attack vectors. Each of these lines starts with “fw,” meaning the rule applies to the RPC Firewall module. The action is allow, because we’re allowing the RPC call, and audit is false. We set audit to false because we want to focus only on suspicious events, the ones prevention engineering should block. That way, we only audit the “bad” RPC calls we’re interested in.

[37:09] Next are the domain controller allow rules, which allow RPC traffic from other domain controllers. Many sensitive RPC calls are made between domain controllers, and those are legitimate. We could write more granular rules here, but for simplicity, allowing DC-to-DC RPC communication is good enough. If an attacker has already compromised a domain controller, you have bigger problems. The configuration then goes through each RPC interface we expect to see on a domain controller, allowing it without auditing, until the end of the file.

[38:08] At the end is a “default block” rule. It’s in quotes because it doesn’t block yet: it still allows the RPC traffic, but it audits it. So wherever we forward Windows events, such as Sentinel or Splunk, we only see audits of potentially bad events. We can refine the configuration iteratively, and once we’re confident in it, turn the default block into an actual block. Then the rule audits and blocks the RPC call, so instead of just auditing bad behavior, we’re preventing RPC attacks like the ones in the demo.

[39:08] After reviewing the configuration, all that’s left is to install the RPC Firewall and start it. Then we run the status command again to confirm everything is installed and running. For the RPC Firewall, it lists all the processes being protected. For RPC filters, it lists the filter rules taken from the configuration file.

How do you tune RPC Firewall rules using Sentinel?

[39:41] Sagie Dulce: Back in Sentinel, we’ve created a function called GetRPCFirewallEvents to query RPC Firewall events. It takes the RPC Firewall event logs, extracts the different columns, and adds information such as the interface name and a short description. It also filters out local RPC activity. Some RPC calls go over the network stack but stay on the same server, and those aren’t interesting for prevention engineering, so we exclude them. The function wraps that entire query.

[40:38] When I run the function, there are no new events. That means the configuration held up for the past 48 hours: no suspicious RPC events were written. But if I expand the window to three days, some results appear. Had the default rule been set to block, those RPC operations would have been blocked, which might not be what we want. So we need to see exactly which RPC interfaces were called and whether the calls are benign or suspicious.

[41:29] To do that, we summarize the results by interface ID and operation. Here, three interfaces were called, each with a single opnum that wasn’t in the original configuration. Next, we update the RPC Firewall configuration, because these operations all appear legitimate, except perhaps the [TODO: unclear word] key, which comes up more rarely. I’ve already added opnums 76 and 77 and made the other updates to my configuration. Once the configuration is updated, run the update command, and that’s it.

[42:28] From here, keep monitoring the RPC Firewall logs for new events and check whether they’re suspicious. After a week or two, once you’re confident, switch the default rule to block. Then you’re in full prevention engineering mode, blocking all sorts of RPC attacks.

Hands-on homework

[42:57] Dekel Paz: Finally, we’ll leave you with some homework if you want more hands-on experience with our tools. First, run different attacks against the domain controllers and check whether you get the alerts or logs you expect, or use the logs to better understand how the tools work. Try BloodHound, Coercer, CornerShot, PsExec, and evil GPOs. Many tools use RPC and LDAP, and running them is a great way to see what they’re doing behind the scenes: what the RPC calls and LDAP queries actually look like.

[43:51] You can also implement some detections in Splunk. And of course, install the RPC Firewall and LDAP Firewall on your own domain controllers to get a feel for installing and configuring them yourself. Run some attacks, create block rules and allowlist rules, and please share what you find. If you discover detections we could add, configurations that block additional attacks, or activity that should be allowlisted, share it with us and the community. Good luck.

[44:40] Sagie Dulce: Thank you for taking part in this workshop. Let us know if you have questions or issues with the LDAP Firewall or RPC Firewall, or general questions about prevention engineering. Remember, you can always join our Slack channel to reach us or consult other members of the community.