A story about tampering EDRs

Introduction

The sophistication of attacks has increased in recent years and malicious hackers are finding new ways to circumvent the protection provided by endpoint security products such as antivirus (AV) or endpoint protection (EPP). I think we can all agree in the information security community that prevention at the endpoint is very important in preventing malicious activity, but at the same time it can never be 100% effective in preventing all malicious activity. However, this realization should not be interpreted negatively, as it helps us understand that simply trying to stop malicious activity is no longer enough. In order to enable organizations and defenders to understand complex attacks, verify possible attack hypotheses, generally gain more visibility at the endpoint, and overall, better understand their own corporate network or IT infrastructure, it is necessary to implement Endpoint Detection and Response (EDR) systems in addition to Endpoint Protection.

A pure EDR system is not designed to actively prevent malicious activity on the endpoint, but rather to detect more complex malicious activity or non-legitimate behavior on the endpoint using machine learning, deep learning, behavior-based detection, etc. (detection) and notify the system administrator or threat hunter of the suspicious detection (response). In my opinion, EDRs have made it much more difficult in recent years for red teamers to operate undetected or as inconspicuously as possible on the endpoint. The reason for this lies in the mechanisms used by EDRs that give them high visibility at the endpoint. As the mechanisms used by EDRs are often very similar to those used by rootkits, wicked tongues claim that the name EDR does not stand for Endpoint Detection and Response, but for Endpoint Defensive Rootkit. Joking aside, most EDRs generally use a combination of user space and kernel space components, or the kernel components often form the basis for implementing user space protection mechanisms. A good example of this is the interaction between callback routines and user space API hooking.

Only by registering the "Process Notify" routine in the designated callback array is it possible for the EDR to implement user space API hooking based on DLL injection. In short, this means that code executed on the endpoint in the context of specific Windows APIs (WIN32 APIs), e.g., VirtualAlloc, or in the context of the corresponding native API NtAllocateVirtualMemory in user space (regardless of the user's integrity level), is redirected by API hooking (detour / JMP instruction) to a separate hooking DLL of the EDR before the transition to kernel space and is scanned for malicious code. Only if the analysis by the hooking DLL of the EDR does not detect any malicious code or behavior, the transition to the Windows kernel takes place by means of a system call (see figure below).

In recent years, this has become a game of cat and mouse, with attackers finding new ways to bypass user space API hooking and EDR vendors implementing new detection mechanisms. Direct system calls are a good example. Attackers or red teamers have discovered that EDRs typically place their API hooks in kernel32.dll or NTDLL.dll in user space. This means that instead of calling the WIN32 API (kernel32.dll), then the native API (ntdll.dll) and then the system calls as part of the user space code execution, we can integrate the system call directly into our POC using assembly instructions. This way we bypass the EDR's API hooks in kernel32.dll and ntdll.dll, so the EDR has no insight into the code being executed. Of course, this did not go unnoticed by the EDR vendors for long, and new mechanisms such as stack tracing were implemented to allow an EDR to determine whether the system call was made directly (Direct System Call) or via the legitimate Windows APIs and Native APIs.

I enjoy learning about Windows EDRs, how they work, and finding new ways to get around them. In this article, however, we're going to focus on EDR tampering rather than EDR bypassing. That is, instead of trying to bypass the detection mechanisms of EDRs, we want to first understand in detail which mechanisms or components of EDRs are commonly used on Windows, and then look for ways to disable key components of EDRs through manipulation. Ultimately, we want to be able to temporarily or even permanently disable key mechanisms of an EDR, such as user space API hooking. However, since most organizations rely on a combination of prevention (AV/EPP), detection and response (EDR), we expand our goals a bit. By the end of this article, we want to be able to override key components of an EPP/EDR combination through controlled manipulation. In other words: We want to temporarily or even permanently override key components of an EPP/EDR combination in the context of prevention, detection and response. As Red Teamers, we want to prevent our activities from being prevented (AV/EPP), our activities from being actively detected and reported (Active Response), and our activities from being recorded by the EDR in the form of telemetry. Similarly, we want to ensure that the corporate defenders (Blue Team) cannot use the EDR's web console functions against us, either temporarily or permanently (until a controlled reset). For example, we want to prevent them from isolating our compromised host using Host Isolation, from accessing our compromised host using Real Time Response Shell, or from restarting the tampered EDR sensor using the Sensor Recovery function in the web console.

Important for the remainder of this article. When I talk about EDRs, I am referring to the EPP/EDR combination often found in enterprises. Also, this article is not about zero-day exploits, but about gaining a better understanding of EDRs on Windows and how we can temporarily or permanently disable important parts of them through manipulation. Again, this is purely my personal research and experience, and I make no claim to accuracy or completeness.

Possible Scenario

Even if the attacker or red team manages to escalate to a user with privileged rights on the compromised endpoint, most EDRs remain very annoying (from the red team's point of view). On the one hand, vendors are constantly improving the quality of prevention, detection and response. On the other hand, if the blue team has done its homework and tokenized the EDR rollout (assuming, of course, that the EDR supports this feature), it is not easy to uninstall the EDR in the context of a privileged user. In other words, uninstalling the EDR requires that the uninstall token is known. However, since we do not want to be dependent on the uninstall token, we are looking for alternatives to temporarily or even permanently disable important features or components of an EDR through controlled manipulation.

Important Components and Mechanisms

Before we look at how EDR systems can be tampered with, we will examine the mechanisms and components of EDR on Windows. We will look at key components in the user space and the Windows kernel. The following components and mechanisms in the Windows user space will be examined in more detail:

• EDR processes in the system session (protected processes)
• EDR user space service (protected service)
• EDR registry keys/subkeys/values

In the Windows kernel, the following components and mechanisms are examined in more detail:

• EDR callback objects/callback routine
• EDR filter driver / minifilter driver

I have designed the rest of this article to take a step-by-step look at the above components in Windows user space and in the Windows kernel. We will start in user space, look at how the component or EDR mechanism works, and consider the possibilities of disabling the component or mechanism by manipulation. We will then look at the impact on the functionality of the EDR if the component could be disabled, and the impact on the quality of prevention, detection and response of the EDR if the component is disabled. Does disabling the EDR component or mechanism in question give us a real advantage, or do other components and mechanisms of the EDR continue to ensure that the key functions of prevention (AV/EPP), detection and response (EDR) are maintained?

User Space: EDR Processes

As a first step, let's take a closer look at how EDRs handle their processes in the system session in user space, the ways in which they can be tampered with, and the effects of tampering on the EDR's capabilities. The top EDRs I know of protect themselves from tampering with their own processes by initiating them as Protected Process Light (PPL-PS_PROTECTED_ANTIMALWARE_LIGHT (0x31)). As a result, we cannot access or terminate the address space of the PPL-protected EDR process, either in the privileged user context (High Integrity) or in the system context (System Integrity). Targeted termination of the PPL process is only possible if a process that is also flagged for PPL with the same or higher PPL level than the EDR process is available or has already been successfully compromised. Other options are to use a trusted installer, or to use a vulnerable kernel driver (Bring Your Own Driver, or BYOD) that gives us write access to the Windows kernel even as an unprivileged user (medium integrity), for example by using NULL DACL access.

If you want to know a bit more about the PPL mechanism itself, I suggest you take a closer look at Windows Internals 7th Edition Part 1 and read the following blog post Do You Really Know About LSA Protection (RunAsPPL)?

In our case, we are taking a closer look at the Bring Your Own Driver technique and how we can use it in the context of manipulating EDR processes in user space. Over the past few years, we have seen malicious attackers or groups of attackers posing as Advanced Persistent Threats (APTs) exploit vulnerable kernel drivers to disable components of EDRs. This technique has been used by the following ransomware groups, among others: Trickbot, Ryuk, DoublePaymer, Dharma and Conti. But how does this technique work in detail and why can it be helpful in the context of manipulating EDR processes?

As mentioned earlier, it is not possible to access the address space of PPL-protected processes, even as a privileged user (High Integrity) or in system context (System Integrity). PPL processes can only be killed from within the Windows kernel. Since kernel drivers on Windows logically reside in the kernel, vulnerable drivers can be used to gain write access to the Windows kernel via the BYOD attack. For example, the signed driver from MSI (rtcore64.sys / Afterburner), which has a NULL DACL access vulnerability, can be used to do this. Write access to the kernel helps us because processes or executed code in the Windows kernel do not have process isolation, unlike code in user space. In other words, the BYOD attack theoretically allows access to the entire Windows kernel (see figure below).

In our case, we can exploit this to gain write access to the Windows kernel via a BYOD attack, locate the EDR PPL process array (EPROCESS STRUCTURE) and temporarily patch the PPL flag. As a result, the affected EDR process in the system session is no longer protected by PPL and can be killed in a privileged user or system context.

There are several tools that can be used to access the kernel through vulnerable device drivers and terminate PPL-protected processes. On the one hand, there are tools such as Cheeky Blinder, PPL-Killer or Mimikatz that use a third-party device driver or their own driver that has not been signed by Microsoft. On the other hand, there is Backstab, which uses the Microsoft-signed driver from the Sysinternals Process Explorer tool. Personally, I prefer the Microsoft-signed driver approach as it seems more legitimate and unobtrusive. In the end, it doesn't matter which tool you use. In several tests with different EDRs, I found that it is sometimes possible to kill the user space process(es) of the EDR in the system session, but this usually only takes a few seconds, in the worst case up to 1-2 minutes, and the previously killed process is reinitialized as a PPL process. Even if the EDR process was killed several times, the killed process was reinitialized each time. It was also interesting to observe that even during the time measured from the end of the EDR process to reinitialization, the EDR's prevention, detection and response capabilities remained fully functional. On the one hand, these observations raise the question of why a terminated PPL process is always reinitialized, or which component of the EDR is responsible for this. On the other hand, why is the overall impact on the functionality of the EDR so small, even during the time when an EDR process is terminated?

Conclusion

If you really want to get rid of an EDR in its entirety, it is far from sufficient to simply kill the EDR process(es) (PPL) in the system session. I would describe the impact on the functioning of the EDR as low.

User Space: Protected Service

In this section we will take a closer look at the User Space Service of the EDR. The figure below shows that the User Space Service is responsible for re-initializing the previously scheduled PPL process of the EDR. From this observation we can see that the User Space Service and the PPL process together form the User Space component of the EDR.

It seems logical to attempt to terminate the EDR user space service in the context of a privileged user (high integrity) or in the system context (system integrity). This would result in the user space component of the EDR no longer being initialized. In other words, if we can terminate the user space service of an EDR, we are permanently rid of the user space component of the EDR. However, this is not straightforward as EDRs normally initialize their user space service as a protected service. Initialization as a protected service is done by an EDR component in the kernel called the Early Launch Antimalware driver, or ELAM driver for short. An EDR protected service is therefore also called an ELAM service. Like the EDR PPL process, the protected service uses a kernel component that does not allow us to pause, stop or disable the EDR protected service, even as a privileged user (high integrity) or in system context (system integrity) in Windows user space. Unlike PPL-protected processes, I am not currently aware of any way to permanently or temporarily stop the protected service of an EDR.

Conclusion

But even if we cannot directly influence the Protected Service now, we have gained an important insight. Because if we could find a way to disable the initialization of the EDR's Protected User Space service, we could permanently disable the EDR User Space component (consisting of the Protected Service and the PPL process). If we were able to disable the EDR's Protected Service, I would rate the impact as medium.

User Space: Registry Keys

In the previous two sections we learned a little about PPL processes and Protected Services. Building on what we learned in the last section, we are looking for a way to permanently disable the initialisation of the Protected Service. In this section we will take a closer look at the registry keys of the user space component of the EDR. The registry keys or subkeys and values that we are interested in can usually be found in the registry editor under "Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services". Once the registry key or subkeys have been found, there are several interesting entries depending on the EDR.

In this case the entry "REG_DWORD: LaunchProtected" is interesting on the one hand and the entry "REG_DWORD: Start" on the other. After a little research I found out that the LaunchProtected entry is responsible for initializing the linked user space process of the EDR as a PPL process. However, it has no effect on the initialization of the protected service itself, as this is still done by the ELAM driver in the Windows kernel. If you want to influence the initialization of the user space EDR process, try changing the value of the Launch Protected entry so that the EDR process is initialized without the PPL flag in the future. This would have the advantage that we would no longer need the BYOD technique to terminate PPL-protected EDR processes. However, we would still have the problem that the EDR process (even without the PPL flag) would continue to be reinitialized by the protected service after successful termination.

It would make much more sense to change the "Start" entry of the EDR user space registry key so that when the compromised client is restarted, there is no further initialization of the protected service and thus the entire user space component. However, in the case of the two user space registry keys, the problem is that EDRs use tamper protection to protect their own registry keys from tampering attempts. This means that even if we manage to escalate to system rights in user space, we will not be able to modify or manipulate the EDR-protected registry keys or the values of the entries.

Conclusion

Caution: With some EDRs, attempts to manipulate registry keys will not go unnoticed, i.e., an attempt to change the value of an entry may trigger an Active Response in the product's web console. This will alert corporate defenders to possible malicious activity on the compromised host, and there is a high probability that the compromised host and user will be isolated and locked down via the EDR feature. If we were able to manipulate the startup entry of the user space component so that the user space service is no longer initialized, I would rate the impact as medium.

Interim status

At first glance, the results so far sound somewhat sobering, as we have not yet found a way to temporarily or permanently disable the EDR and thus get rid of important protection mechanisms in the areas of prevention, detection and response. In the second step, however, we have gained some interesting insights from the last three sections. First, we now know that the EDR user space component consists of the PPL process and the protected service. However, we currently have no direct way of stopping, pausing or disabling the protected service. However, we now know that we can directly influence the initialization of the EDR's protected service via the "Start" entry in the User Space component's registry key.

This means that if we change the value of the 'Start' entry from Autoload (2) to Disabled (4), the Protected Service will no longer be initialized and therefore the EDR's Protected Process will no longer be initialized. In other words, the EDR user space component would be permanently disabled. However, the current problem is that EDRs use tamper protection to protect their own registry keys from tampering attempts.

However, if we can find a way to bypass or disable the registry key tamper protection, we can permanently disable the EDR user space component and get a big step closer to our goal.

Kernel Space: Callback Routine

In the last few sections, we have learned a little about the user space component of EDRs. However, depending on the EDR, they also have a component in the Windows kernel. Originally, before the release of Windows XP x64 SP3, it was possible for EDRs or, at that time, antivirus systems to execute the hooking mechanism in the Windows kernel, e.g., in the form of the SYSENTER hooking or System Service Dispatch Table, or SSDT hooking for short. When Microsoft experienced repeated stability problems (BSOD) due to these kernel activities, Patch Guard was introduced into the Windows kernel. This is a mechanism that checks certain areas of code in the Windows kernel for manipulations such as hooks at unknown or irregular intervals and, if necessary, prevents further execution of Windows by shutting down via a Blue Screen of Death (BSOD). Rumor has it that there are EDR products that continue to implement hooks in the Windows kernel using Patch Guard Bypass, but I cannot confirm this from my observations and experience to date. At least since the introduction of Patch Guard, it is no longer officially allowed or possible to implement hooks in the Windows kernel. As a result, EDRs have been largely banned from the kernel and API hooks have been implemented in Windows user space instead.

However, Microsoft recognizes that EDRs still need a way to use the Windows kernel to implement prevention, detection and response. For this reason, Microsoft has introduced Kernel Callback Objects. This mechanism allows EDR products to register callback objects in the form of callback routines through filter drivers or minifilter drivers. This allows EDRs to register different callback routines in a callback array in the Windows kernel to perform different tasks in user space.

For example, EDRs can register the "Process Notify" routine in the callback array, which allows various prevention, detection and response tasks to be mapped to user space. On the one hand, the registration implements the user space DLL injection or API hooking mechanism. As mentioned in the introduction, API hooking redirects code executed in the context of specific or hooked Windows APIs or native APIs via a detour/JMP instruction to a separate "hooking DLL" of the EDR, where the EDR can examine the executed code for malicious content or activity. In addition, the registered "Process Notify" routine can be used by an EDR to capture telemetry associated with processes.

For example, when a process is killed or reinitialized in user space, the "Process Notify" callback is triggered, and the telemetry generated is captured. As a simple example, we start a cmd.exe on the compromised host and the initialization of the cmd.exe is captured by the EDR or "Process Notify" routine in the form of telemetry, which can then be used for active threat hunting, for example. From a red team perspective, we want to avoid both; we don't want processes with API hooks, and we don't want our activities in the process context to be captured in the form of a footprint using telemetry.

In addition, EDRs can register other callbacks in the Windows kernel that are also used to capture telemetry while performing other tasks in user space. Examples are the "Load Image Notify" and "Create Thread Notify" routines. I think that now we have understood the concept of callback routines using the "Process Notify" routine, the functionality or scope of the tasks is self-explanatory. However, for the sake of completeness it should be mentioned that the "Load Image Notify" routine is used to prevent and detect unauthorized DLL mapping in user space processes and the "Create Thread Notify" routine is used to prevent and detect unauthorized or illegitimate code or process injections.

In our case, however, there is a much more important callback routine, the "CmRegister" routine. This can be registered in the callback array by EDRs using a filter driver or minifilter driver, providing user space tamper protection for the EDR registry keys. This means that if we can find a way to patch or remove the "CmRegister" callback temporarily or permanently, we can temporarily or permanently disable the Windows user space tamper protection of the EDR registry keys. If this is successful, we can set the value of the 'Start' entry in the EDR User Space component registry key to 'disabled' (4). This would result in the protected service and PPL process not being initialized when the compromised host is rebooted. In other words, the initialization of the EDR user space component is completely disabled.

We will investigate this in detail and look for a way to patch the relevant callback routine temporarily or permanently. However, in order to manipulate the callback routines registered by the EDR using minifilters, we need to regain write access to the Windows kernel. This means that we must go back to the BYOD technique and gain write access to the Windows kernel via a vulnerable kernel driver. In my case, I use a tool called Cheeky Blinder, which can be found on GitHub. Firstly, I use Cheeky Blinder to gain write access to the Windows kernel by loading the vulnerable kernel driver rtcore64.sys from MSI (MSI Afterburner) via POC. Secondly, I use Cheeky Blinder to analyze the callback array and can list various registered callbacks. For example, if you list the registered "Process Notify" routines, you can usually see which filter drivers or mini-filter drivers have registered "Process Notify" routines. Depending on the EDR, you may also find the EDR's registered "Process Notify" routines.

In my case, I will stay with the "Process Notify" routine and look at it in more detail. I have already mentioned that the "CmRegister" routine can be registered to protect the EDR's registry keys. However, not all EDRs use this callback to protect their own registry keys. Depending on the EDR, the "Process Notify" routine is also used to implement tamper protection. So, to get rid of the tamper protection in our case, we will focus on the "Process Notify" routine.

As shown in the first demonstration, we use the Cheeky Blinder tool to list all the "Process Notify" routines registered in the array. Using the name of the filter driver, we can find the EDR's registered callback in the callback array and then patch it using Cheeky Blinder. This has the effect of at least temporarily disabling the tamper protection on the EDR registry key. This in turn means that we can now finally set the "Start" entry for the EDR User Space component to "disabled" (4). As you can see in the demo, when the host is rebooted, the protected service is no longer started and the EDR User Space component is no longer started. However, the demo also shows that disabling the user space component has very little impact on the functional quality of the prevention, detection and response capabilities. On the one hand, the problem remains that the execution of malicious code (in this case mimikatz.exe) is still detected and blocked, and on the other hand, the telemetry generated at the endpoint is still collected by the EDR. Furthermore, despite disabling the EDR user space component, it is still possible to use EDR functionality in the web console, e.g., to isolate the host or access the host via remote shell.

Conclusion

In summary, we now have a methodology that allows us to gain write access to the Windows kernel in the context of a privileged user or with a vulnerable device driver already loaded, patch the EDR's "Process Notify" callback, and thus disable tamper protection for the EDR registry key in a controlled manner. However, the problem remains that the main functions of the EDR are active in the context of prevention, detection and response. In our case, this is because the main prevention, detection and response functions of the EDR are implemented by the filter driver, which remains active even after the user space component is disabled. I would rate the impact of temporarily patching the "Process Notify" routine as high. I would rate the impact of disabling the user space component of the EDR as medium.

Kernel Space: Filter Driver

In the last section we came a little closer to our goal of disabling the EDR by manipulation, but we are still not completely satisfied. In the final section, we will take a closer look at the EDR's filter driver or minifilter driver. For your information: A minifilter driver is one that depends on a filter driver itself, e.g., WdFilter.sys. As mentioned above, EDRs can use their own filter drivers to register various callback objects in the form of callback routines in the Windows kernel, thus performing various tasks in user space. In summary, depending on the EDR, or to the extent that an EDR builds its functionality on the functionality of the filter driver, the filter driver is the central element for prevention (AV/EPP), detection and response (EDR). Regardless of whether the user space component of the EDR is enabled or disabled, if the filter driver of the EDR is active, the registration of the callback routines takes place. In other words, if the filter driver is not disabled, the EDR will continue to collect user space telemetry in the context of processes, images, threads, etc., and user space functions such as API hooking will continue to be implemented based on the registered EDR callbacks.

In our case, this means that if we, as a Red Teamer, can disable the EDR filter driver in a controlled manner through manipulation, we can achieve a permanent disablement of important functions in the context of prevention, detection and response. So, let's take another look at the EDR registry keys, and in our case, we can see that the kernel component, i.e., the filter driver, has a separate registry key that is very similar in structure to the user space component. This means that the filter driver registry key also has a 'start' entry, which determines the initialization behavior of the filter driver. In other words, we use the same concept as disabling the user space component and permanently disable the initializations of the filter driver.

Conclusion

As the second demonstration shows, permanently disabling the EDR filter driver has a much greater impact on the functionality of the EDR. Although the user space component of the EDR is active, the main prevention, detection and response functions of the EDR are no longer available. In other words, by disabling the filter driver, callback routines are no longer captured. As a result, telemetry is no longer captured by the EDR sensor, and we as the red team are invisible to the EDR at the endpoint. In addition, important features such as user space API hooking are no longer available, meaning that the EDR loses much of its visibility at the endpoint. In addition, depending on the EDR web console, features such as host isolation, response shell or sensor recovery will lose functionality. In summary, an EDR relies on the filter driver for much of its functionality, and permanently disabling the filter driver can have far-reaching consequences. I would rate the impact on the disabled kernel component filter driver or minifilter driver as critical.

Summary

As we now know, an EDR consists of several components in the Windows user space and in the Windows kernel that are closely interrelated. For example, EDRs use a filter driver in the Windows kernel to register the "CmRegister" to implement tamper protection for registry keys in user space, or to register the "Process Notify" routine, which can also be used to protect registry keys, but is also used to implement API hooking in user space. Depending on which component of the EDR can be manipulated in user space or in the Windows kernel, this has a small or large impact on the functional quality of the EDR. For example, the figure below shows that terminating the EDR user space process has a rather small impact, whereas disabling the EDR filter driver has a very large impact.

However, it is not possible to generalise about the effect of disabling a particular component on the functional quality of the EDR. This depends largely on the EDR itself and the mechanisms it uses under Windows. For example, there are or were EDRs that relied heavily on the user space API hooking component, then there are EDRs that play a lot through the filter driver. It should also be noted that this article does not cover all the components that can be used by EDRs under Windows. For example, Etw or EtwTi was not covered in this article, but may be in my next article.