Source-linked AI summary
SysComb: Fine-Grained Transparent System Call Filtering for Attack Surface Reduction
Matthew Rossi, Marco Abbadini, Michele Beretta, Dario Facchinetti, Stefano Paraboschi
TL;DR
Existing syscall-filtering approaches require application or kernel modifications, limiting transparent deployment for externally maintained software. SysComb uses eBPF to enforce temporally specialized, state-aware per-thread filters with selectable enforcement strategies. Evaluations on widely deployed applications report accurate filtering, increased selectivity, and overhead comparable to seccomp, while limitations remain for green-thread and interpreted-language runtimes.
Problem
Existing syscall-filtering solutions require application or kernel modifications to activate runtime filters, making deployment intrusive and impractical for externally maintained code.
Method
SysComb uses eBPF probes and enforcement programs to track application states and apply temporally specialized per-thread filters without modifying application or kernel code.
Results
SysComb achieved accurate state-aware filtering and increased filter selectivity on nine widely deployed applications, with overhead comparable to seccomp.
Takeaways & Limitations
SysComb provides transparent, synchronous, nonbypassable kernel enforcement that reduces attack surface and mitigates attacks in real-world applications.
Takeaways & Limitations
SysComb is not directly compatible with green threads or interpreted languages; supporting them requires runtime-specific probes and knowledge of runtime internals.
Abstract
from arXiv · showhide
Restricting the system calls available to applications shrinks the kernel's attack surface and greatly mitigates the impact of compromised programs. Recent approaches showcase techniques to generate system call filters, however, all existing solutions require either kernel or application modifications to activate them at runtime. This is intrusive, error-prone, and often impractical, especially when the code is maintained by external parties. This paper presents SysComb, a novel eBPF-based solution to enforce temporally-specialized system call filters based on the application state, without requiring any modification to the application or the kernel code, and thus addressing the above limitations. Moreover, SysComb lets the developer choose between two distinct enforcement strategies: seccomp-like, ensuring no new privileges are gained after a state transition is performed, and least-privilege, which applies to each state the most restrictive filter. We evaluated SysComb using widely used software, showcasing accurate state-aware system call filtering and an overhead comparable to built-in kernel solutions, demonstrating the practicality of our approach.
1 Introduction
Restricting system calls reduces kernel attack surface and limits compromised applications, but production software commonly uses overly permissive startup filters. SysComb transparently activates state-specific filters without application or kernel modifications.
- System call restrictions reduce vulnerable kernel code exposure and limit compromised applications’ capabilities.
- Production applications commonly retain overly permissive filters because developers lack syscall knowledge, restructuring is costly, and rapid deployment is prioritized.
- Prior research generated seccomp-compatible filters and demonstrated benefits from separate initialization and serving filters.
- SysComb automatically identifies and transparently activates state-specific, per-thread syscall filters without application changes, kernel patches, or user-space delegation.
- SysComb supports seccomp-like progressive restriction and least-privilege enforcement, while validating state transitions at runtime.
2 Background
Seccomp restricts syscalls through kernel-evaluated filters that can only become tighter, while eBPF safely extends kernel functionality and supports state through shared maps. Together, these mechanisms provide the foundation for SysComb.
- Seccomp: Seccomp defines kernel-evaluated syscall allow-list filters that are inherited by descendants and can only be tightened.
- Seccomp: Installing seccomp filters generally requires application configuration or library integration, potentially demanding architectural restructuring.
- eBPF: eBPF extends kernel functionality dynamically without source modifications or new kernel modules by running verified programs at designated hooks.
- eBPF: eBPF maps provide persistent shared state across program invocations, while verification checks termination, memory safety, and resource limits.
3 Requirements and Existing Solutions
The paper targets fine-grained, synchronous temporal filtering without application changes, kernel changes, or user-space delegation. Existing approaches leave gaps involving overhead, bypass resistance, dependencies, or temporal and per-thread support.
- Requirements: The objectives require per-thread temporal filters tailored to execution states, including potentially dangerous processing phases.
- Requirements: The design must avoid application refactoring, kernel patch maintenance, and user-space security monitors that add delay or attack surface.
- Requirements: SysComb must support dynamic libraries and external binaries without over-approximating unused syscalls.
- Requirements: Synchronous enforcement must block forbidden syscalls between invocation and execution, while application states are identified through developer-provided logic.
- Existing Solutions: User-space filters face scope, overhead, or bypass limitations, while binary rewriting may miss dynamic-library syscalls.
- Existing Solutions: Single startup filters can over-approximate dynamic-library behavior, whereas prior temporal approaches support limited phases or require compiler changes.
- Existing Solutions: Kernel modifications impose costly patch maintenance across kernel updates, and many eBPF or container solutions lack temporal specialization or rely on user-space enforcement.
4 SysComb Architecture
SysComb transparently tracks per-thread application states, identifies state-specific system calls, and enforces corresponding filters through eBPF without modifying application or kernel code. Its architecture supports tracing and runtime enforcement, with seccomp-like and least-privilege strategies managed through state graphs.
- Function Overview: SysComb associates each application thread or process with an execution state that changes at developer-specified transition points detected through eBPF uprobes.The probes operate on application binaries, including external dependencies, without modifying application source code or binaries.
- Filter Strategies: SysComb supports seccomp-like filters that become more restrictive and least-privilege filters that assign each state its smallest system call set.Flow-based backpropagation supports the seccomp-like strategy by preventing new privileges after valid state transitions.
- State Management: State graphs record valid transitions, helping prevent control-flow hijacking from abusing transitions to gain privileges.The same state-management logic underlies both enforcement strategies.
- Identification of System Call Filters: During tracing, eBPF programs record system calls for each tracee state and set corresponding bits in state-indexed system call bitmaps.The tracer retrieves the current state from the tracee map, indexes the application state map, and records the invoked system call.
- Enforcement: At runtime, enforcement reads the tracee’s current state and associated bitmap, allowing permitted calls while applying the developer-selected action to denied calls.The enforcement program can return an error, log the event, or kill the target when a filter rejects a system call.
- Limitations: SysComb currently assigns states to operating-system threads, while green-thread and interpreted-language support requires probes on their runtimes and knowledge of runtime internals.Green threads can be supported through runtime probes; interpreted languages require identifying runtime attachment points and relating them to application operations.
5 Discussion
SysComb’s security depends on protecting its eBPF state and constraining state transitions, because compromised applications may otherwise tamper with filters or seek more permissive policies.
- Threats and Countermeasures: SysComb’s correct operation depends on the integrity of its eBPF programs and maps, unlike seccomp’s guaranteed filter persistence until process termination.Attackers may tamper directly through the bpf system call or indirectly by terminating the tracer process holding object references.
- Threats and Countermeasures: In enforcing mode, SysComb permits only state transitions defined in the application state graph to prevent control-flow hijacks from installing more permissive filters.The threat arises because diverted control flow could otherwise jump to arbitrary transition points.
6 Effectiveness Evaluation
SysComb accurately detects application system calls and makes filters progressively more restrictive across execution states. Specialized filtering also improves protection against kernel vulnerabilities and shellcodes.
- 6.1 Accurate System Call Detection: SysComb traced nine server applications and measured filter size as the number of permitted system calls, using test suites with 79.3% average coverage.The applications included Apache Httpd, Lighttpd, Nginx, Memcached, MongoDB, Redis, MySQL, PostgreSQL, and Bind.
- 6.1 Accurate System Call Detection: SysComb’s entry-point monitoring produced tighter filters than strace’s startup-inclusive tracing, and all applications completed Minijail test suites successfully.Strace also records loader and libc initialization calls before the monitored application begins.
- 6.2 Fine-Grained Specialized Filters: SCsec reduced allowed system calls relative to SCmono by 11.82% at Listen, 20.77% at Serve, and 76.34% at Cleanup on average.SCsec initially matches SCmono at Startup and becomes stricter after state transitions.
- 6.2 Fine-Grained Specialized Filters: SClp reduced allowed system calls relative to SCmono by 35.22% at the reported state-aware comparison.The evaluation partitioned execution into Startup, Listen, Serve, and Cleanup states.
- 6.3 Attack Surface Reduction: Specialized filters improved protection in 17 of 45 kernel-CVE cases, or 37.8% of the evaluated vulnerabilities.The comparison used single filters from seccomp or SCmono versus a specialized Serve-state filter.
- 6.3 Attack Surface Reduction: Across nine applications, specialization raised blocking of 17 security-sensitive system calls from 55.9% to 72.8% and blocked 98 more shellcodes per application on average.The shellcode reference set contained 535 payloads, and the increase was reported as +27.8%.
- 6.3 Attack Surface Reduction: SysComb can enforce filters generated by other approaches without user-space monitors or kernel modifications, while supporting more than two states and least-privilege filters.It also removes the need to inject seccomp activation calls into applications.
7 Performance Evaluation
SysComb adds low interposition overhead in microbenchmarks and small application-level latency overhead in macrobenchmarks, supporting its practical use alongside kernel-based filtering.
- Microbenchmarks: SysComb averaged 297 ns in tracing mode, 303 ns in enforcement mode with raw tracepoints, and 487 ns with kprobes.Native execution averaged 166 ns; the benchmark used one million getpid calls.
- Microbenchmarks: Kernel-space techniques including SysComb added 198–487 ns per system call, while user-space-monitor approaches required 2658–12019 ns.The higher-overhead approaches incurred one- to two-orders-of-magnitude slowdowns relative to native execution.
- Macrobenchmarks: In tracing mode, SysComb added 0.002 ms to 14.596 ms over baseline across macrobenchmarks, with the maximum equal to 4.46% of Bind’s request time.The tested software included web servers, databases, and a DNS resolver.
- Macrobenchmarks: In enforcement mode, SysComb’s overhead versus seccomp ranged from 0.000 ms with raw tracepoints to 2.233 ms with kprobes.The 2.233 ms maximum was measured for Bind and represented 0.68% of its request time.
- Macrobenchmarks: Injected errors slightly increased overhead, but maximum measured latency remained 0.521 ms for MySQL.The increase was attributed to kprobes, dedicated safety checks, and additional system-call-number lookup steps.
8 Conclusions
SysComb transparently tracks application behavior and synchronously enforces per-thread specialized system call filters entirely within the kernel. Evaluation on nine widely deployed applications found accurate detection, stronger filter selectivity, improved protection, and performance comparable to seccomp.
- 8 Conclusions: SysComb requires no application or kernel-code modifications and introduces no user-space monitors.Its eBPF implementation operates entirely within the kernel.
- 8 Conclusions: SysComb provides fully transparent, synchronous, nonbypassable enforcement of per-thread system call filters for any application state.The system automatically tracks application behavior and enables specialized filters.
- 8 Conclusions: Evaluation on nine widely deployed applications showed accurate system call detection, increased filter selectivity, improved protection against kernel vulnerabilities and exploit payloads, and performance comparable to seccomp.The evaluated software included web servers, relational and NoSQL databases, and a DNS resolver.
- 8 Conclusions: SysComb offers a low-overhead solution that reduces the kernel attack surface and mitigates attacks in real-world applications.
A Backpropagation Algorithms
The section presents algorithms for propagating system call profiles through application-state graphs. It combines strongly connected component analysis, depth-first search, graph transposition, and reverse-order backpropagation.
- Algorithm overview: The high-level algorithm set consists of flow-based backpropagation, strongly connected component identification, DFS traversal, and graph transposition.These are presented as Algorithms 1 through 4.
- Flow-based backpropagation: Algorithm 1 backpropagates system call profiles across states after computing strongly connected components.It unions system calls within each component, assigns the union to component states, and propagates calls to predecessor states.
- Strongly connected components: Kosaraju-Sharir’s algorithm identifies strongly connected components using depth-first search, graph transposition, and reverse stack order.The procedure performs an initial DFS, transposes the graph, then runs DFS on unvisited states popped from the stack.
- Depth-First Search: The DFS procedure marks visited states, recursively explores successors, and pushes each completed state onto a stack.This stack supports the subsequent strongly connected component computation.
- Graph transposition: Graph transposition reverses each state-to-successor edge by adding the original state to the successor’s adjacency list.The transposed graph is returned for the second DFS phase.
B Kernel CVEs Blocked
The table reports how many application-specific system call filters SysComb identified that successfully mitigate Kernel CVEs using either a single filter or specialized filters.
- Kernel CVEs Blocked: Table 5 counts application-specific system call filters that successfully mitigate Kernel CVEs using a single or specialized filter.Further details are provided in Section 6.3.