By Jonathan Erb, threatER Principal Software Engineer
At threatER, we’re well known for our ability to block all known bad traffic using any amount of threat intelligence (even feeds with millions of indicators). Our solution far surpasses the amount of third party threat intelligence other blocking solutions (such as next generation firewalls) are generally able to leverage. We spend a lot of time in our blog and marketing materials talking about the benefits of this.
What we don’t often discuss is the deep, detailed R&D that we are constantly undertaking to make sure that we consistently have the most capable and easy-to-use platform for automated threat intelligence enforcement at scale.
One clear area where R&D matters in a platform like this is where the rubber meets the road: at our Enforcement layer itself. On that note, recently my CTO came to me with a complex request: “Jon, can you investigate some new alternative arrangements for a packet inspection framework that could inspect all inbound and outbound traffic on a given Linux network interface, without leveraging Data Plane Development Kit (DPDK)?” At first I thought: well that does not seem too complicated. The request came with some additional requirements that didn’t seem too noteworthy at the time but quickly presented significant challenges:
- The packet inspection had to operate in user space since it needed to interface with some existing user space policy evaluation software specific to our organization.
- Inspection needed to occur for packets in both directions (for packets entering and leaving any non-virtual interface).
- The user space application did not have to evaluate all packets; only those containing information about an establishing or closing a connection.
- The inspection had to be nearly transparent; meaning that inspected packets (that we allowed) needed to continue on their original journey through the kernel stack as if they were never inspected.
- Packets not allowed by our inspection software needed to be dropped.
- We needed to be able to intercept and inspect packets on interfaces that were attached to a software bridge.
- If possible, we didn’t want to interfere with any networking configurations (including Netfilter) nor did we want to introduce any virtual network interfaces. Other applications running on the system could not be impacted by our presence.
- The performance impact of the newly introduced components needed to be almost negligible.
- Initially we would only need to inspect IPv4 traffic with IPv6 possible in the future.
It’s Time to Learn Something New!
After some thorough investigation, I determined that the features that I would need weren’t going to be readily available with the existing Linux kernel (6.1) features.
Over the last several years my focus has been primarily working with fast packet bypass tools like DPDK and Netmap as they provide fantastic performance for fast packet processing applications. Even with my bias in favor of these tools, they simply could not be made to work here because we couldn’t significantly disrupt the network path or swap out the kernel’s default network device drivers in favor of those of DPDK or Netmap. Recall that we wanted to silently inspect traffic without disturbing the default networking configuration. I was going to have to find a new way and learn something new along the way!
The New Toolbox
There is no doubt the Linux kernel has added a plethora of nice networking features over the years. After some exhausting low-level research, I identified two subsystems that had the potential to be valuable in the eventual solution: Netfilter and Extended Berkeley Packet Filter (eBPF).
Nothing Is Ever Easy
It turned out though that the stock Netfilter and eBPF subsystems were insufficient by themselves to meet my requirements.With respect to Netfilter, it lacks an attachment point that is acceptable in both the ingress and egress path. In addition, it would likely require transfer of each packet to user space and cause an unacceptable performance impact. With regard to eBPF, there is simply no existing method to forward packets to userspace which precludes the use of it alone.
Leveraging the “Best of Both”
While keeping what I learned about Netfilter and eBPF in mind and studying how packets traverse the ingress and egress paths, I discovered that some minor customizations to the kernel might just permit the use of both Netfilter and eBPF in a way that would meet the requirements. More precisely, I’m aiming to have small eBPF programs that can be attached to the ingress and egress traffic control hooks of non-virtual interfaces. These programs will have access to BPF maps and thus be able to communicate with the user space application. Then, working with the shared data in the BPF map the eBPF program code will determine which packets should be forwarded to user space for inspection. All these capabilities are readily available in newer kernels without modification.
When a packet is encountered that requires user space inspection, the eBPF program will redirect it instead of allowing it to continue as normal. However, here we will apply a kernel customization that performs a redirection to a new (custom unpublished) Netfilter hook rather than to another interface. Fortunately, I have discovered that adding a new custom Netfilter hook is trivial. Thus, careful placement of the new Netfilter hooks and their associated reinjection functions should allow eBPF programs to forward packets to userspace via Netfilter. Packets that are allowed by user space can be reinjected so they can complete their journey through the kernel stack.
Obviously putting all this together is a little bit more complicated than the brief explanation above. However, the kernel patch file I’m currently testing is around 100 lines so the changes are not terribly significant. In fact, the eBPF code is much more complex than that of the kernel customizations. Once the kernel patch has been fully vetted and released, we’ll be publishing the kernel modification for potential upstream inclusion so that the entire Linux community can benefit. The important takeaway here is that opening a path between an eBPF program and user space should be fairly trivial if existing Netfilter features are leveraged.
With luck the eventual solution will offer the following benefits:
- Ability to turn on & off the user space inspection by attaching or detaching the eBPF program at any time.
- Will limit the number of packets that need to be examined by userspace and thus have negligible performance impact.
- Will continue to support inspection even if the interface is added to a bridge.
- Is mostly transparent as it does not use virtual interfaces or published Netfilter features.
Conclusion
The proposed kernel modifications above coupled with an eBPF program enable a userspace application to inspect packets entering and leaving a system interface. The inspection occurs at existing ingress and egress eBPF attachment points. For ingress this is very early in the packet receive path and for egress it is just before the packet is queued for transmission. The kernel modifications allow eBPF programs to redirect packets to newly introduced Netfilter hooks where existing Netfilter capabilities can then be leveraged to forward packets to a userspace application for inspection. The packets can then either be accepted (for reinjection) or dropped.
And the collection of those things allowed me to go back to my CTO and say, “I did it!” And thereby, our customers benefit from the ongoing research we are constantly undertaking to continue to provide the world’s best automated network security enforcement platform.