Guide for Apple IT: Threat Detection and the Endpoint Security Framework

Editor's note 5/1/23: This post has been deprecated and replaced by an updated guide to endpoint detection and response (EDR).

In macOS 10.15 Catalina, Apple released new frameworks that have big implications for the future of macOS threat detection and the development of security solutions: the Endpoint Security framework and the System Extension framework. Beyond expanding the safety and security of macOS, Apple is using these frameworks as userspace alternatives to kexts (kernel extensions).

In the future, if a Kernel Extension feature has an equivalent system extension, the kext feature is considered deprecated, and the developer should work to migrate those features to a system extension. A list of deprecated kexts is available in this Apple support article.

In this guide, we’ll look at what these releases mean for Mac security and development and explain how they work.

A Primer on macOS Threat Detection

Apple has always been known for its innovative security solutions. From building security capabilities into the silicon of their device hardware to designing deep system and app security processes that offer maximum protection without interfering with usability, every new release of macOS is a step toward safer and more stable devices.

Back in macOS 10.7.3, Apple released Gatekeeper, a program that requires downloaded software to be signed by Apple before it can be run. Apple also uses XProtect (also referred to as File Quarantine or Known Malware Detection in their documentation) as a signature-based system linked to Gatekeeper. It checks the signatures of suspicious files against records of known malware to make sure only safe files are opened on your device.

macOS Catalina continues this trend, including more security features that have big implications for macOS threat detection. One key capability for effective threat detection and security tools is a process and file monitor, which watches various system and file events for anomalous or malicious activity, such as trojan applications that trick users into activating adware, malicious websites, or documents that install dangerous programs once opened.

Creating a comprehensive process and file monitor on previous releases of macOS has been difficult. The best method was to use kexts to receive data about events in real-time or attempt to block malicious behaviors before they were carried out. However, starting with macOS 10.15 Catalina, Apple is offering new frameworks to give developers more visibility into process, network, and file system events.

Enter the Endpoint Security and System Extension frameworks, two promising new releases on top of which future endpoint security programs will be built. In the rest of this guide, we’ll break down what these are and what implications they have for the future of macOS threat detection.

What’s the Apple Endpoint Security Framework?

As we mentioned earlier, developing endpoint security programs for macOS has been tricky in the past. But with the announcement of Apple’s Endpoint Security framework, all of this changes.

Apple’s Endpoint Security framework is a new API that monitors system events for malicious activity. As Apple describes in its support documentation:

“Endpoint security clients, including Endpoint Detection and Response software and antivirus software, can now leverage the new Endpoint Security API to monitor and even block system events to better conform with security policies and protect from potential malicious activity.”

In this way, the Apple Endpoint Security framework lets registered clients receive notification of events that have already occurred and authorize events that are still pending. This pertains to events such as process executions, forking processes, signal raising, mounting file systems, and more.

The framework will also let new and existing security software developers reinvent their existing macOS security agents, which an MDM solution can easily install and approve.

What Are System Extensions?

Like kexts, System Extensions extend the functionality of the operating system. However, unlike kexts, System Extensions run in userspace, outside the kernel. They are thus free from many of the restrictions developers had to face while coding in the kernel. This significant departure from kext development opens up advanced macOS threat detection capabilities.

At its Worldwide Developers Conference (WWDC) in 2019, Apple announced three types of system extensions:

1. Network extensions replace network kernel extensions, including the capability to filter network traffic, reroute that traffic, and connect to VPNs.
2. Endpoint security extensions replace kexts that monitor security events with the kauth interface. Endpoint security extensions are intended to be used for security apps such as endpoint detection and response, antivirus, or data loss prevention apps.
3. Driver extensions replace device driver kexts that use IOKit. Driver extensions let programs control USB, serial, network interface controllers (NIC), and human interface devices (HID).

You probably noticed that each of these system extensions replace kexts with similar capabilities. This is part of an effort by Apple to phase out kexts in order to create a more stable and secure environment for macOS; system extensions can help make that happen. Kexts are still needed for some operations, but system extensions should be used in place of kexts whenever possible.

In order to explain why, let’s take a look at some of the shortcomings of kexts and how System Extensions solve them.

The Problem with Kexts

Kexts have been a hallmark of Mac since the beginning. Essentially, they're bundles of code that let developers extend the capabilities of the kernel. Because the kernel is in charge of everything on the system, and because all kexts are given kernel privileges, these extensions are very powerful.

But their power is also their problem. Since actions taken within the kernel are prioritized over everything else in the system, developing and debugging apps in the kernel leaves no room for error. If a program isn’t perfect, it can halt all of the system’s processes and potentially cause it to crash. When creating kexts, developers are also limited to using a single framework and can write code using only the C or C++ languages.

The power of kexts also leads to some security concerns. Since one of the most important jobs for the kernel is to define and enforce the system’s security policy, and kexts operate within it, they’re effectively “above the law.” That means if a kext has a bug that makes it vulnerable to being compromised, then the entire system is at risk–and there’s no security rule that can restrain it.

To rectify these problems, Apple introduced system extensions. Let’s take a look at how they compare to kexts, and why they’re such a big step forward for macOS threat detection.

Kexts vs. System Extensions

One of the biggest differences between system extensions and kexts is that system extensions run in userspace rather than in the kernel. This means developers can code and debug outside of the restrictions of the kernel. That, in turn, means their apps don’t have to be practically bug-free to avoid the risk of causing kernel panics, crashing the entire machine, or making it slow and unpredictable. With system extensions, the days of worrying about dynamic memory allocation, synchronization, and latency are over.

System Extensions also give developers more freedom over their frameworks and programming languages. While kexts supported only one framework and required C or C++, developers writing system extensions can use any framework in the macOS SDK and any programming language. (One notable exception to this comparison: Because of their close connection to hardware, driver extensions must still be written in C or C++, and developers must use the DriverKit framework.)

How System Extensions Increase Security

We already mentioned that kexts are “above the law” since they function within the kernel. Because of this, vulnerabilities within a kext can leave the entire system at risk, without restraint. This underpins the most significant security and stability issue with kexts: Their power means that small errors or gaps in development leave the entire system vulnerable to being compromised and controlled.

System extensions change this. Rather than running in the kernel with complete control over the entire system, these extensions run in userspace like an ordinary app. That means system extensions are answerable to the system’s security policies rather than above them.

This doesn’t mean that system extensions can perform only the functions that apps can. System extensions are granted certain permissions to perform advanced functions, such as having direct access to associated hardware devices or using APIs to talk to the kernel.

This separation between the kernel and system extensions means that a buggy extension won’t crash the whole system, leave it vulnerable to compromise, or make it unstable. In other words, even if the extensions crashes, the rest of the system can keep running, unaffected.

This is a significant step forward in making macOS safer. Taken in conjunction with the Apple Endpoint Security Framework, developers gain advanced security capabilities to monitor and authorize system events, detect and deter malicious activity, and develop extensions in userspace that communicate with the kernel, without leaving the system vulnerable or unstable.

Apple’s new Endpoint Security Framework and system extensions have a lot of promise. Here at Kandji, we look forward to the new opportunities it opens up for device management solutions. As the only MDM with a pre-built library of security controls and the industry’s first one-click compliance agent, we take security seriously and look forward to developing modern features that make the most out of the latest evolutions in the world of macOS threat detection and security development.