PaperlayerView exact revision on arXiv

Source-linked AI summary

EdgePace: Deadline- and Freshness-Aware LLM Scheduling on Unified-Memory Platforms

Teng Mei, Chengxuan Pei, Weijun Wang, Jie Wu, Marco Canini, Yunming Xiao

arXiv:2610.37307v1cs.PF

TL;DR

On unified-memory edge devices, LLM memory traffic can disrupt foreground deadlines, while classical schedulers trade deadline performance against fresh LLM feedback. EdgePace combines safe inference suspension, foreground monitoring, and scheduling based on contention, deadlines, and freshness. Across four edge platforms and two workloads, it reduces deadline misses while retaining LLM throughput, and all measured outputs meet freshness windows.

  • Problem

    On unified-memory edge devices, LLM memory traffic can disrupt camera deadlines, while classical schedulers trade foreground misses against LLM freshness without accounting for application SLOs.

    Source: §1 · p. 2, §2.4 · p. 4

  • Method

    EdgePace combines safe suspension points across LLM inference, foreground memory-pressure monitoring, and scheduling based on contention risk, deadline slack, and output freshness.

    Source: §1 · p. 2

  • Results

    Across four edge platforms and two workloads, EdgePace reduces foreground deadline misses by 24.1–96.1% relative to unmanaged inference while retaining 64.9–98.5% of its throughput; all measured outputs meet freshness windows.

    Source: §1 · p. 2, §6 · p. 13

  • Takeaways & Limitations

    EdgePace preserves timely sensor processing and useful LLM inference across distinct environments and workflows, with corresponding improvements in task quality.

    Source: §1 · p. 2, §6 · p. 13

Abstract

from arXiv · show

Large language models (LLMs) increasingly run on edge devices alongside tasks that continuously process sensor data. In many applications, these foreground tasks prepare observations as inputs to the LLM, whose outputs guide subsequent processing or actions. The observations must be processed by foreground deadlines, and LLM feedback must arrive while its source observations remain useful. Yet the CPU and accelerators share a unified-memory system, so LLM inference can delay foreground processing through memory-bandwidth contention even when compute resources remain available. We present EdgePace, a bandwidth-aware runtime that protects foreground deadlines while sustaining timely LLM feedback. EdgePace creates safe suspension points across prefill and decode, monitors foreground memory demand and stalls at low cost, and schedules inference according to contention risk, deadline slack, and output freshness. We evaluate EdgePace on a OnePlus 13 and three NVIDIA Jetson platforms using multi-camera processing and robotic target tracking. Relative to unmanaged inference, it reduces foreground deadline misses by 24.1%--96.1% while retaining 64.9%--98.5% of LLM throughput. All measured LLM outputs arrive within their task-defined freshness windows. On OnePlus 13, for example, camera deadline misses fall by 91.3% while 90.4% of LLM throughput is retained.

1 Introduction

EdgePace addresses memory-bandwidth contention that can delay deadline-bound sensor processing while LLM inference runs on unified-memory edge devices. Its runtime coordinates safe inference suspension, foreground monitoring, and deadline- and freshness-aware scheduling; evaluations report fewer deadline misses while retaining LLM throughput.

1 Introduction

  • On Jetson Nano Orion Super, sustained memory traffic raises four-camera p95 latency by 1.52× and deadline misses to 72.78%, while compute-intensive GPU interference is negligible.A co-running Qwen2.5-1.5B model raises the four-camera deadline miss rate to 18.33%.Source: Figure 1 · p. 1, §1 · p. 2
  • Local LLM inference supports responses without a cloud round-trip and keeps sensitive sensor data on-device, but its resource use must be balanced against foreground timing needs.The runtime must find safe pause points in both prefill and decode, gather informative pressure signals at low cost, and keep feedback timely.Source: §1 · pp. 1–2
  • Foreground sensor tasks and LLM inference share limited memory bandwidth, so LLM traffic can delay foreground requests and cause deadline misses even when compute resources are not saturated.Missed deadlines can mean dropped frames or decisions based on stale observations.Source: §1 · pp. 1–2
  • EdgePace partitions prefill and decode at safe sublayer boundaries, monitors foreground memory demand and stalls, and schedules inference using contention risk, deadline slack, and input freshness.The execution plan allows inference to pause and resume without repeating completed work.Source: §1 · p. 2
  • Across four edge platforms and two workflows, EdgePace reduces foreground deadline misses by 24.1%–96.1% versus unmanaged inference while retaining 64.9%–98.5% of LLM throughput.All measured LLM outputs arrive within their task-defined freshness windows.Source: §1 · p. 2

2 Background and Motivation

Unified memory lets edge-device processors share data without separate device-memory transfers, but their traffic competes for finite DRAM bandwidth. The workflows therefore impose both foreground input deadlines and LLM output-freshness requirements, motivating scheduling that addresses gaps in suspension control, pressure sensing, and timing policy.

2.1 Edge Platforms with Unified Memory

  • CPU, GPU, and optional NPU traffic shares physical DRAM bandwidth, so LLM memory requests can inflate CPU-task latency even when compute resources remain available.Task buffers, model weights, activations, and the KV cache occupy the same physical memory.Source: §2.1 · p. 2

2.2 Edge LLM Workflows and System Metrics

  • The observation–processing–inference–feedback workflow requires foreground jobs to meet release-based input deadlines and LLM outputs to arrive before their source observations expire.The evaluation measures deadline misses and response time, result age and freshness-window delivery, LLM throughput and completion latency, and task-specific quality.Source: §2.2 · pp. 2–3

2.3 Existing Solutions and Remaining Gaps

  • Prior approaches leave gaps for these workflows: Sarathi-Serve lacks comparable decode interruption points, while SERENO’s speculative-decoding yields address a different workflow in which foreground tasks operate independently of the LLM.Holding inference can protect input deadlines but also make feedback stale; aggregate bandwidth misses foreground stalls, while detailed tracing can interfere with protected tasks.Source: §2.3 · p. 3

2.4 Goals and Challenges

  • Safe, frequent suspension across prefill and decode must preserve completed work and unchanged outputs without extensive framework changes, while balancing response speed against control overhead.More frequent suspension can shorten reaction delay but increase scheduling and execution overhead.Source: §2.4 · pp. 3–4
  • A 28 GB/s bandwidth threshold detects only 45.26% of camera-workload misses, and 93.84% of its warnings are false positives, motivating more accurate foreground-pressure signals.Averaged bandwidth can smooth short demand bursts that delay foreground execution.Source: §2.4 · p. 4
  • In the four-camera Jetson Orin Nano Super experiment, FCFS limits misses to 3.56% but delivers only 1.60 token/s against a 2.5 token/s freshness requirement, whereas SJF sustains 5.61 token/s but misses 15.56% of foreground deadlines.Scheduling must combine pressure feedback with deadline slack and remaining freshness, while reserving time for outstanding work to drain.Source: §2.4 · p. 4

3 System Design

EdgePace coordinates foreground jobs and LLM inference by controlling submission at safe boundaries, using memory-pressure signals and task timing to balance deadline protection with output freshness.

3 System Design

  • EdgePace coordinates foreground latency-critical tasks and LLM inference through safe execution quanta, memory monitoring, and Bypass, Admit, or Hold submission modes.Bypass preserves ordinary submission, Admit permits the next quantum, and Hold stops new submissions while checks repeat periodically.Source: §3 · p. 4

3.1 Adaptive Fine-Grained Execution Plan

  • Safe sublayer boundaries create more frequent suspension opportunities across prefill and decode, while profiled quantum runtimes help the scheduler account for work before the next pause.Profiles remain specific to inference phase and decode progress; suspension retains completed computation and does not interrupt executing kernels.Source: §3.1 · pp. 4–5

3.2 Runtime Monitoring of Memory Pressure

  • Four PMU counters combine LLC-miss demand with memory-stall fraction into independently smoothed indicators; these signals measure foreground pressure, not attribution to the LLM.Demand uses both absolute miss intensity and misses per instruction, while stalls remain separate because severe blocking can coexist with low demand.Source: §3.2 · pp. 5–6

3.3 Risk- and Freshness-Aware Scheduling

  • At legal boundaries, the scheduler combines foreground slack, freshness slack, and contention risk: Bypass at low risk, Admit at medium or urgent freshness, and Hold at high risk with periodic rechecks.Freshness urgency overrides risk; while held, the scheduler repeats checks at a configurable interval, set to 1 ms by default.Source: §3.3 · pp. 6–7

4 Evaluation

EdgePace is evaluated on four unified-memory platforms using microbenchmarks and multi-camera and robotic-tracking workflows. Across both case studies, it lowers foreground deadline misses while retaining LLM throughput, and the quality metrics show improved timely-state consumption.

4.1 Experimental Setup

  • The evaluation uses four testbeds: a OnePlus 13 and three Jetsons, with models matched to platform capacity and identical platform–model–foreground settings for runtime comparisons.The phone runs Case Study 1, while the Jetsons run both case studies; only successfully completed runtime combinations are reported.Source: §4.1 · p. 8, Table 1 · p. 8

4.2 Microbenchmarks

  • On Orin Nano Super, EdgePace detects 84.0% and 86.84% of misses in the two cases, with false-alarm fractions 83.80 and 75.51 percentage points below the baseline’s lowest reported values.The comparison spans memory-bandwidth warning thresholds and reports miss detection and false alarms among warnings.Source: §4.2 · p. 8, Figure 6 · p. 8
  • On Orin Nano Super, local decision checks average 1.856 μs; P99 drain latency is 3.202–4.176 ms, and graph partitioning raises execution time by 0.25–4.50%.Hold stops submissions at a legal boundary with 0 ms delay, but outstanding GPU work drains; P99 resumption latency is 1.221–2.689 ms from Admit.Source: §4.2 · pp. 8–9, Table 3 · p. 9, Table 4 · p. 9

4.3 Case Study 1: Multi-Camera Semantic Feedback

  • The camera workload publishes aligned-frame activity and motion snapshots each epoch, letting valid, fresh asynchronous-LLM outputs select cameras and processing modes; concurrent model outputs leave robotic tracking actions unchanged.The camera deadline is one frame (33.33 ms); robotic tracking runs every 50 ms with a 40 ms sensing-to-command deadline.Source: §4.3 · p. 9, §4.4 · p. 11
  • Across seven multi-camera platform–model pairs, EdgePace has the lowest measured miss rate, reducing misses by 24.1–91.3% versus Unmanaged while retaining 64.9–93.2% of its throughput.In the four-camera Nano run, misses fall from 18.33% to 7.667% (58.2% fewer), with throughput decreasing from 8.175 to 5.848 token/s (71.5% retained).Source: §4.3 · p. 10, Figure 7 · p. 10
  • EdgePace has the lowest mean focus regret in all seven multi-camera configurations, with reductions versus Unmanaged ranging from 40.7% to 90.7%.The four-camera Nano reduction is 70.0% (0.013091 to 0.003921); the metric averages regional selection loss over epochs, including zero-loss epochs.Source: Figure 8 · p. 10, §4.3 · p. 11

4.4 Case Study 2: Robotic Target Tracking

  • Across five robotic-tracking pairs, EdgePace reduces misses by 49.6–96.1% versus Unmanaged while retaining 72.7–98.5% of its throughput.Thor’s VLM retains 98.5% of throughput; remaining misses, particularly in that result, show that scheduling mitigates rather than eliminates deadline violations.Source: §4.4 · p. 11, Figure 9 · p. 12
  • EdgePace yields the lowest excess staleness loss in every robotic-tracking configuration, reducing it by 49.5–70.5% versus Unmanaged.No-LLM runs already incur roughly 809 mm · s despite zero sensor misses, so total-loss reductions are 5.3–17.1%, compared with 49.5–70.5% for excess loss.Source: §4.4 · p. 11, Figure 10 · p. 12

5 Related Work

Prior work addresses on-device DNN scheduling, inference preemption, and execution monitoring through different control boundaries and signals. EdgePace combines safe suspension across prefill and decode with foreground memory-demand and stall monitoring, scheduling for deadlines and output freshness.

Model scheduling on edge devices

  • DNN scheduling systems target hardware-aware execution or resource–accuracy adaptation, while MCSched suspends and resumes background LLM inference using memory-controller pressure and foreground timing feedback.EdgePace adds safe sublayer suspension across prefill and decode and considers both foreground deadlines and LLM output freshness.Source: §5 · p. 12

Inference preemption and accelerator scheduling

  • Inference control ranges from token or chunk boundaries to layer, operator, and accelerator command-queue scheduling; SERENO’s yield points depend on speculative decoding.EdgePace instead provides suspension points across prefill and decode for workflows with freshness deadlines on feedback derived from foreground observations.Source: §5 · pp. 12–13

Execution monitoring

  • Profilers and tracing tools provide detailed execution views, whereas EdgePace periodically samples foreground counters to derive memory-demand and stall signals for online scheduling.The cited tools include instruction-granularity GPU probes and CPU/GPU activity correlation; the passage characterizes their continuous collection and delivery as costly on the scheduling path.Source: §5 · p. 13

6 Conclusion

EdgePace co-schedules LLM inference and latency-critical tasks on unified-memory edge devices, reducing foreground deadline misses while retaining LLM throughput.

6 Conclusion

  • Across four edge platforms and two workloads, EdgePace reduces foreground deadline misses by 24.1–96.1% while retaining 64.9–98.5% of LLM throughput.The paper reports corresponding improvements in task quality.Source: §6 · p. 13
  • EdgePace combines safe sublayer suspension with lightweight foreground monitoring to schedule inference by contention risk, deadline slack, and output freshness.Source: §6 · p. 13
  • The runtime co-runs LLM inference with latency-critical tasks on unified-memory edge devices.Source: §6 · p. 13

A Complete Pause/Resume Measurements

The appendix reports pause and resume timing measurements, distinguishing submission delays from GPU drain time and subsequent queueing.

A Complete Pause/Resume Measurements

B Case Study 1: Sensitivity

The sensitivity results show fewer misses under EdgePace across foreground loads and for both the Q4 and default models, with throughput trade-offs.

B Case Study 1: Sensitivity

  • Figure 11(a) shows three foreground loads, while Figure 11(b) compares Q4 and the default model with four camera inputs.The heavy-load and default-model points use four-camera results from Figure 7; light/medium-load and Q4 points retain earlier measurements.Source: Figure 11 · p. 17
  • Misses fall from 1.36%, 5.17%, and 18.33% to 0.75%, 2.94%, and 7.667% across light, medium, and heavy loads, respectively.Throughput retention is 78.7%, 75.2%, and 71.5%, respectively; the updated four-camera run reports a 58.2% reduction in misses.Source: B Case Study 1: Sensitivity · p. 17, Figure 11 · p. 17
  • For Q4, misses drop from 8.67% to 4.24% (51.1% fewer) with 78.0% throughput retained; for the default model, misses fall from 18.33% to 7.667% (58.2% fewer) with 71.5% retained.Throughput changes from 10.08 to 7.86 token/s for Q4 and from 8.175 to 5.848 token/s for the default model.Source: B Case Study 1: Sensitivity · p. 17, Figure 11 · p. 17

C Scheduling Decision Distributions

The recorded decision distributions vary by workload and model, and Hold-check frequency does not directly indicate elapsed waiting time.

C Scheduling Decision Distributions

  • The distributions cover 12 configurations, count scheduler checks including repeated checks during Hold, and report cumulative Hold time.Figure 12 shows decision percentages by platform–model configuration; Table 8 gives percentages and cumulative hold times.Source: C Scheduling Decision Distributions · p. 18, Figure 12 · p. 18, Table 8 · p. 18
  • The log's normal category means Bypass, and Admit includes freshness-urgency decisions as well as medium-contention decisions.Source: C Scheduling Decision Distributions · p. 18
  • Hold spans 1.13–60.52% of Case 1 checks; in Case 2, AGX Orin selects Hold for 58.09% of Qwen3-8B checks but Admit for 66.43% of Qwen3-VL-8B checks.Thor selects Bypass for 72.93% and 86.17% of Case 1 checks with its text and vision-language models, respectively.Source: Figure 12 · p. 18, C Scheduling Decision Distributions · p. 18
  • Thor's Qwen3-8B run has 84.82% Hold checks and 6.288 s cumulative waiting in Case 2, versus 8.15% and 19.553 s in Case 1.Repeated checks during Hold mean its check share does not imply the same share of elapsed time; run durations are not recorded.Source: C Scheduling Decision Distributions · p. 18

D Graph Partitioning: Phase Breakdown

Graph partitioning changes execution time differently across phases and configurations: Orin Nano Super sees a larger prefill increase than decode, while the OnePlus 13 VLM’s total increase is smaller.

D Graph Partitioning: Phase Breakdown

Related papers

Loading 2610.37307v1…