Source-linked AI summary
Influence of Logging Frameworks on Bind9
Max Schrötter, Hannes Signer, Bettina Schnor
TL;DR
The paper investigates how logging pipelines can become bottlenecks that let attackers hide activity in host-based IPS deployments. It evaluates BIND9-based setups, introduces FIPS IPC to reduce logging overhead, and reports higher logging capacity and faster banning than conventional frameworks. The study also identifies evaluation boundaries at current traffic-generation and application-workload settings.
Problem
Existing host-based IPS logging paths can be overwhelmed, allowing attackers to hide traces, while high-speed alternatives shift the bottleneck to logging backends.
Method
The paper evaluates the BIND9 and Fail2Ban logging pipeline and introduces FIPS IPC with kernel bypass, shared-memory ring buffers, multiple consumers, and minimal copying.
Results
FIPS introduces almost no overhead versus disabled logging, logs more requests than evaluated alternatives, bans malicious clients 2.5× faster than file logging, and sustains 2^16 attacking clients at one million requests per second.
Takeaways & Limitations
The results indicate that logging IPC deserves attention as network speeds increase, because it can materially affect IPS performance under attack.
Takeaways & Limitations
The evaluation does not exceed 3,000,000 requests per second and does not include TCP applications such as nginx.
Abstract
from arXiv · showhide
Host-based Intrusion Prevention Systems (IPS) rely on application logs to detect and block malicious activity. However, on modern high-speed networks the logging subsystem itself becomes a bottleneck: an attacker can hide traces simply by generating enough traffic to overwhelm the application's log pipeline, dropping crucial traces. In this work, we show that widely deployed setups such as Fail2Ban monitoring BIND9 can be defeated with less than 65 Mbps of DNS traffic. Further, we show that when replacing core components of the IPS architecture with their higher-performance equivalent, iptables with eBPF and regex matching with Hyperscan, the logging backends themselves become the bottleneck. Therefore, we present FIPS, a new IPC designed for high-performance logging that bypasses the kernel and reduces copying of the log messages to a minimum. FIPS uses per-thread lock free shared memory ring buffers, supporting multiple independent consumers reading the same log stream at their own pace. FIPS offers both a native API and a drop-in replacement for the syslog interface. Our evaluation with BIND 9 shows that FIPS introduces almost no overhead compared to disabled logging, logs more requests than any other evaluated framework, and enables the IPS to ban malicious clients $2.5\times$ faster than with file logging while sustaining $2^{16}$ attacking clients at one million requests per second.
1. Introduction
The introduction frames application logging as essential to host-based intrusion detection while identifying the logging path as vulnerable to overload attacks. It motivates examining the full pipeline and improving IPC performance for high-volume logs.
- Logging in host-based intrusion detection: Application logs are analyzed by collectors, forwarded to SIEMs and IDSs, and illustrated through BIND9 with syslog as the backend.Collectors parse text logs into structured fields and store them for later analysis.
- Logging in host-based intrusion detection: Widely used logging frameworks include file logging, syslog through local Unix sockets or network transports, and stdout or stderr forwarding.Local daemons may process, forward, or relay messages to other logging systems.
- Threat model: Attackers can hide traces by modifying logs after compromise or overwhelming an application or IPC component in the logging pipeline.This work focuses on the latter threat rather than post-compromise log tampering.
- Threat model: High-volume DNS logging requires IPS and logging frameworks to cope with traffic while reducing log volume through mechanisms such as rate limiting.The motivating scenario involves malware using DNS and domain generation algorithms to resolve command-and-control servers.
- Paper scope: The paper examines the full stack from log-producing applications through IPC transport to log collectors.This scope connects application behavior, message transport, and downstream log processing.
- Paper scope: The study evaluates how easily Fail2Ban can be overwhelmed and measures the performance impact of logging frameworks on BIND9.It also motivates a high-performance IPC after identifying logging backends as a bottleneck.
2. Limitations of IPS in High-Speed Networks
The experiments show that Fail2Ban can be overwhelmed by high-volume spoofed DNS traffic, while BIND9 logging substantially reduces query-processing performance and becomes a system bottleneck.
- 2.1. Ban Capabilities under Attack: Fail2Ban initially bans all 255 attacking clients, but after the third ban cycle it can no longer keep up with the log volume.The attack sends 100,000 requests per second from 255 different IP addresses.
- 2.1. Ban Capabilities under Attack: 10,000 packets per second is sufficient to trigger Fail2Ban failure when the attack uses 2047 spoofed IP addresses.Fail2Ban also takes significantly longer to ban 2047 addresses than 255 clients.
- 2.1. Ban Capabilities under Attack: Less than 65 Mbps of DNS traffic corresponds to 100,000 requests per second when each request is 648 bits long.
- 2.1. Ban Capabilities under Attack: 15.1% of total CPU time is spent in BIND9's logging function, while an off-CPU analysis finds threads unable to run for 80% of the time because of its lock.The system remains 65% idle under high load, confirming that logging is the bottleneck.
- 2.2. Impact of Logging: At 900,000 requests per second, BIND9 answers 99.94% without logging but only 11.79% with journald, 6.53% with file logging, and 4.64% with syslog.All available logging options show a significant performance drop compared with no logging.
3. Related Work
Prior work develops structured logging, binary DNS logging, shared-memory queues, and multi-consumer ring buffers, but identifies contention and transport overheads that limit existing approaches.
- Logging formats: Existing applications largely retain established logging functions, while newer formats include qlog and DNS-specific binary logging through DNSTAP.DNSTAP uses Google Protocol Buffers and forwards DNS events over ring buffers.
- Shared-memory logging: Shared-memory logging frameworks reduce reliance on conventional logging paths by enqueueing messages in memory-mapped storage for later forwarding.The prior mqlogd design dequeues messages and forwards them through named pipes to an output utility.
- Binary DNS logging: DNSTAP transports binary DNS events asynchronously over Unix domain sockets, creating a one-to-one stream and adding context-switch overhead.Multiple consumers require an external software multiplexer.
- High-performance queues: High-performance queue designs can suffer from consumer or producer contention, delayed tail updates, and cache interference.The cited approaches generally support only the classical producer-consumer model, whereas logging requires redundant readers.
- FIPS IPC: FIPS proposes multiple ring buffers, one per producer, instead of a single ring buffer with writer contention.
4. Design of the new IPC for High-Performance Logging
FIPS is a high-performance logging IPC designed to reduce producer overhead while allowing multiple consumers to process the same log stream independently. Its design combines shared-memory, lock-free ring buffers with native and syslog-compatible interfaces.
- Interfaces: Existing applications can use FIPS through a native API or a preloaded library that replaces syslog functions.The syslog-compatible variant is called FIPS Syslog.
- Architecture: Multiple consumers can independently read all available log messages at their own pace.Each consumer may use multiple reader threads, supporting parallel and asynchronous processing.
- Architecture: FIPS assigns each producer thread a dedicated lock-free ring buffer to reduce logging contention.This simplifies the design to a single-producer, multiple-consumer ring buffer.
- Ring buffer operation: The ring-buffer protocol uses producer claim-and-commit phases so applications write directly into shared memory without later copies.Consumers subsequently claim the same read-only mapped memory.
- Synchronization: Single-thread-owned write and commit pointers avoid locks and atomic operations on the producer path.Acquire/release ordering ensures the commit pointer advances only after the ring-buffer cell is written.
- Synchronization: Readers use separate read and release pointers, with a release queue handling out-of-order returns.In-order returns bypass the mutex-protected queue, reducing synchronization overhead.
5. Evaluation
The evaluation compares FIPS with established BIND9 logging frameworks and tests its effect on IPS banning under high-rate DNS attacks. FIPS maintains logging and client-response performance better than file logging in the reported workloads.
- Logging performance: FIPS IPC incurs almost no performance loss compared to no logging in BIND9.DNSTAP has similar performance, but FIPS logs significantly more messages.
- IPS impact: 2.5× faster response rates for valid clients were sustained by FIPS over more than 99% of the banning evaluation.FIPS reached 99.5% response rate at 2.8 seconds, while file logging reached it at 7.1 seconds.
- IPS impact: 29.95% was the average valid-client response rate before attackers were banned with FIPS, versus 3.2% with file logging.The comparison used simple-Fail2Ban under 3,000,000 packets per second of attack traffic and a 50,000-packet-per-second whitelist workload.
- System-scale evaluation: 2^16 attacking clients sending 1 million requests per second were handled by FIPS IPC with simple-Fail2Ban.Fail2Ban previously struggled with 1024 clients sending 100,000 requests.
- System-scale evaluation: After 2^14 clients, file logging answered less than 20% of valid requests, while FIPS kept up with the log volume.FIPS Syslog could not handle 2^16 DoS clients, indicating overhead from BIND9 serialization and string conversion.
- Bottleneck analysis: The logging subsystem became the dominant bottleneck once higher-performance packet filtering and pattern matching were used.The evaluation used simple-Fail2Ban with C, Hyperscan, and an alternative packet-filtering setup to remove the original IPS bottleneck.
6. Discussion
The discussion presents FIPS as a broadly integrable logging IPC whose benefits extend beyond the BIND9 evaluation, while noting that workload characteristics affect its performance impact.
- Scope: FIPS's performance impact may be smaller for applications with high workload per request.The authors identify workload generated by each request as a factor affecting the observed impact.
- Implications: nginx spent 50% of CPU time writing query logs while serving static files, illustrating potential relevance beyond DNS.This observation is presented as evidence of the potential of the new logging IPC.
- Related mechanisms: Compared with shared-memory approaches, io_uring still incurs syscall overhead and has faced security-related disablement in several projects.The discussion contrasts FIPS's kernel-bypassing approach with io_uring's reliance on io_uring_enter.
- Implications: FIPS can be integrated through the syslog API or directly through its own API, allowing use by other applications.The BIND9 integration required 179 lines of code.
7. Future Work
Future work must test FIPS at higher network speeds and in TCP applications, because the reported evaluation reached only 3,000,000 requests per second and did not include stateful TCP traffic.
- Higher-speed evaluation: 3,000,000 requests per second was the upper limit of the reported evaluation, below current 400 Gbps interfaces and anticipated 800 Gbps networks.The authors call for further IPS-performance investigation at higher network speeds.
- TCP evaluation: TCP applications such as nginx remain for future analysis because the load generator struggled with stateful TCP traffic when the IPS dropped packets.This limitation prevented their inclusion in the current evaluation.
8. Conclusion
The BIND9 case study shows that conventional logging paths can become the dominant host-based IPS bottleneck, allowing attackers to hide traces. FIPS IPC addresses this bottleneck with high-throughput shared-memory logging and substantially improves logging and banning performance.
- 65 Mbps of DNS traffic can defeat Fail2Ban with BIND9 by overwhelming file-, syslog-, or journald-based logging paths.
- FIPS IPC bypasses the kernel with per-thread lock-free shared-memory ring buffers, supports independent consumers, and minimizes memory copies.A syslog-compatible replacement supports adoption without source-code changes.
- Almost no overhead compared to disabled logging was observed for FIPS IPC, which consistently logged more requests than every other evaluated framework.
- 2.5× faster banning than file logging was achieved with FIPS when integrated with an IPS.
- 2^16 attacking clients at one million requests per second were sustained by FIPS, beyond the regime where conventional logging backends collapse.
- The results identify logging IPC, rather than detection logic, as requiring closer attention as link speeds approach 400 and 800 Gbps.