Source-linked AI summary
Operating System Support for Accelerators as First-Class Compute Elements in Heterogeneous Systems
Mohammad Shahidzadeh, Lesley Shannon
TL;DR
Accelerators are often developed and evaluated with limited consideration of the applications they serve or their system-level environment. This work studies operating-system overheads in heterogeneous systems and introduces co-designed mechanisms, reporting up to 260× lower communication overhead and 2–6× better application wall-clock time with a better scheduler.
Problem
Accelerators are often developed and evaluated in isolation, with limited consideration of their applications or the system-level environment in which they operate.
Source: §I · p. 1Method
The paper introduces hardware–software co-designed mechanisms for accelerator communication and scheduling that treats CPUs and accelerators as a unified pool of computational resources.
Source: §I · p. 2Results
Up to 260× lower communication overhead and 2–6× better application wall-clock time with a better scheduler.
Source: §VI · p. 11Takeaways & Limitations
Application performance depends not only on accelerator speed but also on operating-system overheads, resource availability, and scheduling decisions.
Source: §I · p. 2
Abstract
from arXiv · showhide
Heterogeneous systems-on-chip (SoCs) increasingly rely on domain-specific hardware accelerators (HAs) to deliver performance and energy efficiency gains beyond what general-purpose processors can provide. However, operating systems (O/S), such as Linux, assume that processors are the Compute Elements (CEs) and HAs are peripheral devices rather than specialized CEs. As a result, HA execution incurs substantial software (SW) overheads that reduce the effective benefits of hardware (HW) acceleration. This work demonstrates that significant inefficiencies arise from ``O/S peripheral device taxes'' (O/S taxes), including SW stack-accelerator communication, invocation, interrupt handling, and scheduling. We built MEERKAT, a Field-Programmable Gate Array (FPGA)-based framework for configurable multicore systems that include HAs; it allows the user to migrate Linux O/S support from SW to HW and identify, study, and quantify the impact. Here we study four major sources of O/S tax and introduce techniques to mitigate them. These include a virtual-memory-aware (VMA) communication method that enables efficient HA access to application data, a lightweight HA invocation interface, Interrupt Policy Adaptation (IPA) for dynamic polling and interrupt selection, and an HA-aware scheduler that dynamically distributes work across Central Processing Units (CPUs) and multiple HA instances. Experimental results demonstrate up to $260\times$ faster communication and $2$--$6\times$ performance improvements in wall-clock time over conventional schedulers. Furthermore, our study reveals that overall system performance depends not only on HA speed, but also on O/S overheads, resource availability, and scheduling decisions. These findings highlight the importance of treating HAs as first-class computational resources and motivate a more accelerator-centric O/S design for future heterogeneous systems.
I. INTRODUCTION
The paper argues that CPU-centric operating systems treat accelerators as peripherals, leaving system-level overheads that can reduce the benefits of hardware acceleration. It introduces MEERKAT and co-designed mechanisms to study and mitigate these taxes across communication, invocation, interrupts, and scheduling.
I. INTRODUCTION
- Accelerators remain outside most O/S abstractions as peripheral devices, while relatively little attention has examined how O/S abstractions, resource policies, and software interfaces affect their utilization and application performance.The paper calls the resulting overheads “O/S peripheral device taxes” and identifies communication, invocation, interrupt handling, and scheduling as four major sources.Source: §I · p. 1
- The VMA communication method combines CIVAT, SGPT, and HMAC so accelerators can access virtual-memory-managed application data without software-mediated translation, copying, or coherence management.The mechanisms address address translation, non-contiguous physical pages, and processor-cache consistency, with up to 260× lower communication overhead and near-zero hardware cost.Source: §I · pp. 1–2
- A lightweight function-call interface and IPA reduce invocation and completion overhead by selecting polling or interrupt-driven completion based on predicted accelerator execution time.The techniques target software-stack overhead, unnecessary context switches, and completion handling.Source: §I · p. 2
- The accelerator-aware scheduler treats CPUs and multiple accelerator instances as a unified resource pool, dynamically selecting resources based on workload and availability.The approach reports up to 2–6× wall-clock performance improvement over conventional accelerator-selection approaches.Source: §I · p. 2
- MEERKAT is an FPGA-based Linux research framework for studying O/S taxes and CPU–accelerator interactions in heterogeneous SoCs.The framework is designed to support detailed analysis of cross-layer performance bottlenecks.Source: §I · p. 2
- The study finds that application performance depends on O/S overheads, resource availability, and scheduling decisions, not only accelerator speed.It argues for system-level optimization of accelerator-rich systems.Source: §I · p. 2
II. RELATED WORK AND MOTIVATION
Prior work addresses heterogeneous-system overheads, but often studies individual issues in isolation. The paper positions MEERKAT as a configurable Linux-capable FPGA platform and situates its communication approach among existing shared-virtual-memory methods.
II. RELATED WORK AND MOTIVATION
- Unlike Mozart, which focuses on runtime overheads for disaggregated accelerators, this work takes a broader O/S perspective on end-to-end overheads for shared-memory accelerators.The authors note that invocation, communication, coherence, interrupt handling, and scheduling have commonly been studied separately.Source: §II · p. 2
A. Evaluation Framework
- MEERKAT uses the configurable CVA5 soft processor to study O/S overheads while retaining resources for large accelerators and enabling modification of processor and O/S interactions.The paper contrasts this objective with FPGA platforms that use larger processor cores, which can limit frequency, resource efficiency, and flexibility.Source: A. Evaluation Framework · pp. 2–3
B. Communication Methods
- Shared virtual memory lets accelerators directly access virtual-memory-managed application data, but existing FPGA-oriented implementations still rely on dedicated address-translation hardware and incur translation overheads.The cited approaches report improvements over copy-based communication, while their hardware complexity and resource overhead can challenge resource-constrained FPGA systems.Source: B. Communication Methods · p. 3
C. Interrupt/Polling Trade-offs
The section identifies gaps in accelerator interrupt handling and scheduling, while MEERKAT provides a Linux-capable FPGA framework for studying heterogeneous-system O/S taxes and communication models.
C. Interrupt/Polling Trade-offs
- Prior interrupt-polling schemes typically use device load, traffic, or fixed thresholds; the paper instead uses predicted operation completion time to select polling or interrupts.The authors identify execution-time prediction for accelerator completion handling as an unexplored approach.Source: C. Interrupt/Polling Trade-offs · p. 3
D. Scheduling
- Scheduling overheads remain underexamined despite prior heterogeneous scheduling work, and worker-based frameworks can add software overhead, especially for fine-grained invocations.The paper says it quantifies these overheads and demonstrates techniques to reduce their impact.Source: D. Scheduling · pp. 3–4
III. PROPOSED MEERKAT
- MEERKAT supports simulation and FPGA emulation, running the same software stack on both without modification.The framework is built on CVA5 and targets Linux-capable heterogeneous SoCs.Source: §III · p. 4
A. Communication Tax
- The framework supports copy-based, tightly coupled, and shared-virtual-memory communication, each confronting translation, coherence, and non-contiguous-page challenges.Copy-based communication requires data movement; tightly coupled access suits small working sets because DDR-to-BRAM transfers limit scalability.Source: A. Communication Tax · pp. 4–5
B. VMA Communication Method
The VMA communication method lets accelerators access application data through virtual-memory-aware mechanisms. CIVAT uses the processor’s existing translation infrastructure and reduces translation overhead.
B. VMA Communication Method
- The VMA communication model lets accelerators access application data structures using virtual addresses without intermediate copies.It requires virtual-to-physical translation, coherence management, and access to physically non-contiguous pages.Source: B. VMA Communication Method · p. 5
C. Custom Instruction Virtual Address Translation (CIVAT)
- Linux software translation can require thousands to tens of thousands of cycles for one address, including a kernel page-table walk when the page is present.An unallocated target page first requires page-fault handling and page-table creation.Source: C. Custom Instruction Virtual Address… · p. 5
- CIVAT sends a virtual address through the processor’s load-store unit for normal TLB/MMU translation, then returns the physical address to a custom execution unit.The instruction suppresses memory requests and writes the translated address to the destination register.Source: C. Custom Instruction Virtual Address… · p. 5
- CIVAT preserves O/S protections and invokes normal page-fault handling for inaccessible or unallocated pages before returning a physical address.It exposes only physical addresses accessible to the requesting process.Source: C. Custom Instruction Virtual Address… · p. 5
- 71× lower translation overhead is achieved by CIVAT when the page is already present; the page-fault case improves 1.5× over conventional software translation.In the page-fault case, O/S page allocation dominates execution time.Source: C. Custom Instruction Virtual Address… · p. 5
D. Scatter-Gather Page Transfer (SGPT)
SGPT converts virtual-memory regions into scatter-gather DMA descriptors so accelerators can use application data across non-contiguous pages. HMAC tracks accelerator DMA bursts and generates cache-line invalidations for coherence.
D. Scatter-Gather Page Transfer (SGPT)
- The driver describes each matrix row in page-bounded DMA chunks, allowing two-dimensional transfers from original layouts without software repacking or extra copies.It uses CIVAT to obtain physical addresses and limits each descriptor to the contiguous bytes remaining in a 4 KB page.Source: D. Scatter-Gather Page Transfer (SGPT) · p. 6
- SGPT enables accelerators to access application data regardless of physical-page layout, without requiring contiguous allocations or copies into accelerator memory.The driver constructs DMA descriptor chains from virtual-memory regions and programs their heads into the SGDMA engine.Source: D. Scatter-Gather Page Transfer (SGPT) · pp. 5–6
E. HW-Managed Accelerator Coherence (HMAC)
- HMAC monitors accelerator AXI bursts and issues cache-line invalidations across each burst’s address range to maintain coherence with CPU accesses.The module records each burst’s starting address and transfer size.Source: E. HW-Managed Accelerator Coherence… · p. 6
- HMAC merges consecutive AXI bursts when a new burst starts immediately after the previously tracked address range, extending the stored transfer size.Source: E. HW-Managed Accelerator Coherence… · p. 6
F. Invocation and Scheduling Tax
The paper reduces accelerator invocation and scheduling overhead through a user-space function-call interface and scheduling integrated into the application thread. The integrated approach removes a separate scheduler thread and its idle execution.
F. Invocation and Scheduling Tax
- Function-call invocation through a user-space library eliminates kernel transitions and IOCTL overhead associated with conventional accelerator frameworks.The interface replaces the traditional device-driver route for application accelerator calls.Source: F. Invocation and Scheduling Tax · pp. 6–7
- Dedicated scheduler threads can consume resources through polling, context switches, or synchronization even when infrequent scheduling decisions leave them waiting for work.Source: F. Invocation and Scheduling Tax · p. 7
- Integrating scheduling into the application thread removes the dedicated scheduler process, reducing context-switch overhead and idle scheduler execution.The application thread invokes the scheduler immediately before accelerator execution.Source: F. Invocation and Scheduling Tax · p. 7
- Shared scheduler and accelerator locks serialize access to availability and performance state while allowing competing processes to sleep rather than spin.This avoids dedicated scheduler-thread overhead and busy-wait CPU utilization penalties.Source: F. Invocation and Scheduling Tax · p. 7
G. Interrupt Tax
Interrupt Policy Adaptation (IPA) chooses polling for short accelerator tasks and interrupt-driven completion for long ones. The supplied framework results also report low system-operation and memory costs and efficient multicore scaling.
G. Interrupt Tax
- For short accelerator executions, interrupt handling and context switches can rival or exceed accelerator time, so IPA selects polling or interrupts according to estimated execution time.The policy compares the estimate with a threshold; polling disables the interrupt source, while interrupt-driven execution enables it.Source: G. Interrupt Tax · p. 7
IV. EXPERIMENTAL SETUP
- Experiments used a Xilinx VCU118 FPGA at 100 MHz with LiteX BIOS, OpenSBI v0.8, and Linux kernel v5.18.0-rc7.The baseline CVA5 configuration was modified for Linux; the framework infrastructure uses 615 CLBs without BRAMs or DSPs.Source: §IV · p. 8, A. Framework · p. 8
A. Framework
- At 100 MHz, a null system call takes 297 cycles (2.97 µs), while memory writes require 0.37 cycles/byte and reads 1.50 cycles/byte.These are framework overhead and memory-throughput results.Source: A. Framework · p. 8
- Four cores maintain 94.7% scaling efficiency, and per-core performance remains 2.59 CoreMark/MHz, matching the reported baremetal baseline.Doubling the workload from one to two cores increases total execution time by 0.18 s (1.1%).Source: Table I · p. 8, A. Framework · p. 8
B. Applications
The benchmark suite contains two communication-bound and two compute-bound applications, supporting evaluation of communication, invocation, interrupt policy, and scheduling. Their implementations range from a data-movement-limited FFT to configurable synthetic accelerator work.
Benchmark suite
- The evaluation uses four accelerators and four representative applications; Table IV defines per-execution characteristics including calls, transfer size, CPU time, accelerator time, and speedup.The suite comprises two communication-bound and two compute-bound applications.Source: B. Applications · p. 8, Table IV · p. 9
Communication-bound applications
- Pulse Doppler transfers approximately 1 KB per invocation and achieves a 15× speedup over CPU execution.Its fully pipelined FFT core makes performance primarily limited by data movement rather than computation.Source: B. Applications · p. 8
- Memory Copy achieves a 9× speedup over SW memcpy for random-sized transfers averaging 125 KB, and a 26× speedup for 1 MB transfers.The accelerator uses a FIFO connected directly to the DMA engine to isolate communication overhead.Source: B. Applications · p. 8
Compute-bound applications
- For 512 × 512 matrix multiplication, tile sizes 64 and 128 achieve 238× and 770× speedups, respectively, over the CPU implementation.The tiled GEMM accelerator varies tile size to evaluate different computation-to-communication ratios.Source: B. Applications · p. 8
- The synthetic accelerator uses random programmable latencies to emulate different execution times and speedups, enabling controlled scheduler and IPA evaluation.Its requests can favor either polling or interrupt-based completion.Source: B. Applications · p. 8
C. Scheduling Policies
The evaluation compares five scheduling policies for heterogeneous CPU-and-accelerator resources. The proposed Earliest Finish Time policy estimates completion across available resources and assigns each task to the predicted fastest-finishing resource.
C. Scheduling Policies
- CPU Only executes every task on available CPUs, regardless of accelerator availability or system load.Source: C. Scheduling Policies · p. 9
- Accelerator Only sends every task to one designated accelerator, making incoming tasks wait when it is busy.Source: C. Scheduling Policies · p. 9
- Busy-Aware (Fuse) dispatches to an idle accelerator and otherwise immediately executes the task on the CPU.The policy is adapted from Fuse.Source: C. Scheduling Policies · p. 9
- Random selects a CPU or accelerator without considering execution time, resource utilization, or queue occupancy.Source: C. Scheduling Policies · p. 9
- Earliest Finish Time estimates each available resource's task completion time, accounting for execution time and current workload, then assigns the task to the predicted earliest finisher.This is the proposed policy for dynamic load balancing across heterogeneous processors and accelerators.Source: C. Scheduling Policies · p. 9
A. Communication tax results
The VMA communication method reduces communication overhead by up to 260× compared with copy-based communication, while scaling beyond dedicated accelerator memory. The scheduling results also show improvements over a worker-based scheduler, with the gains depending on application concurrency and CPU-bound work.
A. Communication tax results
- Copy-based communication requires CPU copies between application virtual memory and accelerator memory, so its overhead grows linearly with transfer size.Its maximum transfer size is limited by the reserved accelerator memory in DDR.Source: A. Communication tax results · p. 9
- VMA combines direct memory sharing’s low software overhead with off-chip scalability, requires no explicit copies, and outperforms both previous communication methods for large transfers without a dedicated-memory size constraint.Source: A. Communication tax results · p. 9
- Virtual-address translation accounts for approximately 33% of communication time, while DMA transfers account for 66%.The passage identifies translation as a potential target for further optimization.Source: A. Communication tax results · p. 9
- Up to 260× lower communication overhead is reported for VMA versus the conventional copy-based approach.Table V compares copy-based communication, the prior IOMMU-based SVM implementation, and VMA; the prior-work comparison is from its published paper, not a reimplementation.Source: A. Communication tax results · pp. 9–10
B. Schedluing Tax
- Across PD configurations, the proposed scheduler improves application wall-clock time by 1.5×–5.8× over CEDR's worker-based scheduler.The largest reported gain, 5.8×, occurs with one CPU core and one application; with 100 concurrent PD applications, the gain is 1.5×–2×.Source: B. Schedluing Tax · p. 10
- PD is primarily CPU-bound because the fully pipelined FFT spends relatively little time computing compared with surrounding software processing, so more CPU cores improve throughput for both schedulers.Source: B. Schedluing Tax · p. 10
C. Communication Bound Applicaions
For compute-bound workloads, scheduling across CPUs and accelerators can improve throughput, while communication-bound workloads see no gain from advanced scheduling under constrained bus bandwidth.
C. Communication Bound Applicaions
- Communication-bound PD and Memcpy throughput remains virtually identical to AO because accelerator transfers contend for constrained bus bandwidth.The evaluated systems used two FFT accelerators for PD and two FIFO accelerators for Memcpy.Source: C. Communication Bound Applicaions · p. 10
D. Computation Bound Applicaions
- Across compute-bound configurations, EFT achieves peak performance by dynamically scheduling across all available computing resources.With an accelerator 4× faster than the CPU, fully configured EFT outperforms AO and EFT(2ACC).Source: D. Computation Bound Applicaions · p. 10
- Fuse can approach EFT at moderate accelerator speedups by running tasks on the CPU while the accelerator is busy, but this becomes suboptimal as accelerator speedup grows.At higher speedups, a CPU task may take longer than waiting for the accelerator.Source: D. Computation Bound Applicaions · pp. 10–11
- Nearly 2× throughput is achievable by scheduling across CPUs and accelerators when their execution times are similar, compared with accelerator-only dispatch.The passage identifies using all available resources as beneficial in this regime.Source: D. Computation Bound Applicaions · p. 11
- With concurrent applications, distributing requests across two identical GEMM64 accelerators improves overall throughput by approximately 21% versus one accelerator.The comparison uses tile size 64 and the proposed EFT scheduler.Source: D. Computation Bound Applicaions · p. 11
E. Interrupt+Polling
The study finds that polling suits short-running accelerator tasks, while interrupts suit compute-bound tasks; IPA selects between them using a 5000 ns threshold. The paper also reports overall gains from reducing O/S taxes and outlines directions for future work.
E. Interrupt+Polling
- Polling outperforms interrupts for short-running FFT and memory-copy tasks, whereas interrupt-driven execution is preferable for compute-bound applications.For short tasks, context-switch overhead is comparable to accelerator execution time; for compute-bound tasks, the processor can do useful work while waiting.Source: E. Interrupt+Polling · p. 11
- A 5000 ns transition threshold provides the best IPA performance in the benchmark, closely matching measured interrupt-handling context-switch overhead.The threshold marks when expected accelerator execution time becomes comparable to the O/S cost of servicing an interrupt.Source: E. Interrupt+Polling · p. 11
VI. CONCLUSION AND FUTURE WORKS
- Up to 260× lower communication overhead and 2–6× better application wall-clock time are reported with the paper’s mechanisms and better scheduling.The conclusion attributes the wall-clock improvement to a better scheduler.Source: §VI · p. 11
- Future work includes hardware IOMMU translation, streaming DMA, hardware-assisted scheduling, and support for dynamically reconfigurable heterogeneous systems.These directions aim to increase accelerator autonomy.Source: §VI · p. 11