PaperlayerView exact revision on arXiv

Source-linked AI summary

DRLM: Deep Reinforcement Learning-Based LLM Query Orchestration in Edge Environments

Reza Farahani, Zoha Azimi Ourimi, Mario Colosi, Lauri Loven, Christian Timmerer, Schahram Dustdar

arXiv:2609.00442v1cs.DC

TL;DR

Heterogeneous edge devices and model configurations make per-query LLM orchestration a trade-off among latency, quality, and resources. DRLM combines quality and latency predictors with a state-aware PPO agent, reducing inference latency by up to 51% and queuing delay by up to 67% on a 64-node edge testbed.

  • Problem

    Edge LLM orchestration requires per-query choices among model-device-quantization configurations with distinct latency, quality, and resource trade-offs.

    Source: §I · p. 1

  • Method

    DRLM combines query profiling and lightweight quality and latency predictors with a factorized PPO agent that uses runtime state for per-query orchestration.

    Source: 3) Quantization impact · p. 2

  • Results

    51% lower inference latency and 67% lower queuing delay were reported for DRLM on a 64-node edge testbed.

    Source: §VI · p. 6

  • Takeaways & Limitations

    Under increasing workloads, DRLM mitigates congestion through state-aware distribution across model-device configurations while maintaining stable response quality.

    Source: 3) Comparison with baselines · p. 6

Abstract

from arXiv · show

Large language model (LLM) services increasingly process heterogeneous queries with diverse latency, accuracy, and resource requirements. While edge deployment reduces response time, the heterogeneity of devices and the diversity of model families, parameter scales, and quantization levels make efficient LLM query orchestration challenging. This paper introduces DRLM, a Deep Reinforcement Learning-based LLM query orchestration framework in edge environments. DRLM integrates two lightweight predictors: (i) a class-conditioned quality estimator that maps queries to semantic categories and infers model performance, and (ii) a feature-driven latency predictor that estimates inference time across model-device configurations. These predictions, combined with system state, feed a factorized Proximal Policy Optimization (PPO) agent that performs state-aware orchestration decisions. To enable data-driven orchestration, we construct a large-scale benchmarking dataset with 223 835 measurements spanning 1258 queries, 6 query classes, 8 model families (32 deployed instances), 5 quantization levels, and heterogeneous edge devices. Evaluation on a 64-node edge cluster and comparison with three baselines and two state-of-the-art methods show that DRLM reduces inference latency by up to 51% and queuing delay by up to 67 %, while incurring at most 8% accuracy loss. It improves latency under increasing workloads up to 61.4%, demonstrating robust and stable orchestration.

I. INTRODUCTION … 3) Quantization impact:

DRLM frames edge LLM orchestration as a per-query choice across model, device, and quantization configurations, and combines quality and latency predictions with runtime state to make those decisions. On a 64-node edge cluster, it reports lower latency and queuing delay with at most 8% accuracy loss compared with three baselines and two state-of-the-art methods.

I. INTRODUCTION

  • Edge orchestration is a per-query choice among model-device-quantization configurations because heterogeneous instances have distinct latency-quality-resource trade-offs.The paper evaluates representative configurations on a Raspberry Pi 4 and an NVIDIA Jetson ORIN AGX.Source: §I · p. 1

1) Model-family impact:

  • Qwen3(0.6, 4) and Qwen3(0.6, 16) outperform other configurations on TruthfulQA, indicating that architecture, not just size, shapes task-specific performance.Source: 1) Model-family impact · p. 1

2) Scaling impact:

  • Gemma3(1, 4) improves Mathematics accuracy over (0.27, 4) on RP but degrades TruthfulQA performance, while Gemma3(12, 4) improves Mathematics on JOA without consistently dominating across tasks.Scaling benefits are workload-dependent rather than monotonic.Source: 2) Scaling impact · p. 1

3) Quantization impact:

  • On RP, Gemma3(0.27, 16) improves TruthfulQA accuracy over (0.27, 4) but increases latency; on JOA, Qwen3(0.6, 16) has higher latency than (0.6, 4) with marginal accuracy gains.Quantization effects depend on configuration and device, so selection should be joint with model and system characteristics.Source: 3) Quantization impact · p. 1
  • Existing solutions do not jointly model query semantics, model configuration, and dynamic system state, limiting fine-grained orchestration in heterogeneous edge environments.The paper argues that effective orchestration requires state-aware modeling of query semantics, configurations, and runtime dynamics.Source: 3) Quantization impact · pp. 1–2
  • DRLM reduces inference latency by up to 51% and queuing delay by up to 67%, with at most 8% accuracy loss; under increasing workload, latency falls by up to 61.4%.These comparisons use three baselines and two state-of-the-art methods.Source: 3) Quantization impact · p. 2
  • DRLM uses a query profiler, class-conditioned quality and configuration-dependent latency predictors, and a factorized PPO agent that combines predictions with runtime state for per-query orchestration.Runtime state includes resource utilization and queue dynamics.Source: 3) Quantization impact · p. 2
  • Its benchmark contains 223 835 measurements across 1258 queries and 6 classes, capturing query-model-device interactions.Evaluation uses a Kubernetes-based cluster with 64 heterogeneous nodes and 32 LLM instances spanning 8 families, 11 parameter scales, and 5 quantization levels.Source: 3) Quantization impact · p. 2

II. PROBLEM FORMULATION

The paper models edge-cluster LLM query orchestration as an MDP in which an agent assigns each incoming query to a feasible model-device configuration. Lightweight predictors estimate inference latency and response quality to support candidate evaluation without full inference.

II. PROBLEM FORMULATION

  • Orchestration is formulated as an MDP with state space S, action space A, policy π, and reward function r.At each decision step, the agent observes system state and assigns an incoming query to a model-device configuration.Source: §II · p. 2
  • The setting comprises N heterogeneous edge devices hosting K distinct LLM instances, each defined by its model family, parameter scale, and quantization level.Queries arrive online stochastically and must be assigned to feasible model-device pairs subject to deployment and resource constraints.Source: 1) Environment · p. 2
  • Lightweight predictors estimate inference latency T(x_i, m, e) and response quality A(x_i, m), enabling fast candidate evaluation without full inference.Source: 1) Environment · p. 2

1) Environment: … 4) Reward function:

DRLM formulates query orchestration as a state-aware assignment problem, representing query and system conditions and balancing predicted quality against end-to-end latency. A PPO policy selects feasible model-device assignments to maximize cumulative reward.

2) State space:

  • At each query arrival, the state incorporates its extracted representation alongside device resource utilization and queue conditions.The queue state captures queue length and remaining processing time of active requests, enabling joint reasoning over query characteristics, model behavior, and system dynamics.Source: 2) State space · p. 2

3) Action space:

  • The action space selects a model configuration and execution device, subject to deployment placement and device compatibility.Each feasible assignment is a model-device pair.Source: 3) Action space · p. 2

4) Reward function:

  • The reward trades normalized predicted response quality against normalized end-to-end latency, with α and β controlling their relative weights.End-to-end latency combines predicted inference time with queue-induced waiting time; a normalized exponential penalty makes high-latency assignments costlier than a linear penalty.Source: 4) Reward function · p. 2
  • The policy samples model-device assignments and is optimized with PPO to maximize expected cumulative reward.At each step, the agent receives the reward associated with its sampled action and current state.Source: 5) Policy learning · p. 2

5) Policy learning: · III. DRLM DESIGN

DRLM uses predictor-generated trajectories to train an actor-critic policy for per-query model-device selection, within an architecture of five cooperating modules for heterogeneous edge clusters.

5) Policy learning:

  • The policy objective uses an advantage estimate, value-function loss, and entropy term, with coefficients c1 and c2 controlling the value-loss and entropy contributions.The passage defines ρi(θ) using current and old policy probabilities and identifies ˆAi as the advantage estimate.Source: 5) Policy learning · pp. 2–3

III. DRLM DESIGN

  • The architecture selects a model-device configuration per query, with models specified by family f, parameter scale p, and quantization level z, and executed on edge device e.Source: Figure 2 · p. 3, §III · p. 3

5) Policy learning:

  • DRLM trains its policy from predictor-generated trajectories, avoiding real LLM inference during learning and supporting training under dynamic system conditions.The trajectories are generated by predictors T and A.Source: 5) Policy learning · p. 3
  • An actor-critic architecture jointly trains policy πθ as the actor and value function Vψ(s) as the critic.Source: 5) Policy learning · p. 3
  • The value-loss and entropy-regularization coefficients separately control their contributions to policy learning.Source: 5) Policy learning · p. 3

III. DRLM DESIGN

  • The DRLM design comprises five modules that jointly support fine-grained LLM query orchestration in heterogeneous edge clusters.Source: §III · p. 3

1) Query profiler: · 2) Query classifier: · 3) Performance predictor:

The query profiler extracts lightweight structural features and a compact-encoder embedding, while a classifier abstracts query intent into semantic classes. Quality and latency predictors estimate candidate model behavior alongside runtime state and instance availability to support orchestration.

1) Query profiler:

  • The profiler represents each query with lightweight structural features, such as token length and syntactic complexity, plus a dense embedding from a compact sentence encoder.These features capture both structural and semantic properties.Source: 1) Query profiler · p. 3

2) Query classifier:

  • The lightweight classifier maps queries to semantic class distributions, falling back to a robust default class for low-confidence or out-of-distribution inputs.The class set can be extended by retraining; this abstraction reduces prediction variance while preserving query-dependent performance patterns.Source: 2) Query classifier · p. 3

3) Performance predictor:

  • The performance predictor evaluates candidate model-device assignments without inference, while the runtime monitor tracks dynamic system state and the instance manager tracks model deployment and availability.Together, these functions provide behavior estimates and system information for orchestration.Source: 3) Performance predictor · p. 3, 4) Runtime monitor · p. 3
  • The quality predictor infers expected model performance from query classes rather than noisy per-query accuracy, capturing stable patterns across semantic categories.Class-conditioned estimation enables stable quality estimates.Source: 3) Performance predictor · p. 3
  • The latency predictor estimates inference time from query features, model configuration, and device characteristics under varying system conditions.It captures execution behavior across devices and configurations.Source: 3) Performance predictor · p. 3

4) Runtime monitor: · 5) Orchestration agent:

The runtime monitor tracks device conditions and active requests, while the orchestration agent combines predictive signals with runtime state to choose model-device assignments. Its factorized policy structures those choices across model configuration and device selection.

4) Runtime monitor:

  • The edge cluster state manager continuously tracks device resource utilization, execution state, current load, and remaining processing time of active requests.These measurements form the system state (u, q) used by the orchestration agent.Source: 4) Runtime monitor · p. 3

5) Orchestration agent:

  • The orchestration agent combines predictive signals with system state to make query-conditioned decisions.The state encoder combines query representation, predicted performance, and runtime state into a compact latent decision state.Source: 5) Orchestration agent · p. 3
  • The policy autoregressively factorizes model-device assignment decisions to handle the combinatorial action space.The factorization orders choices over model configuration and device selection.Source: 5) Orchestration agent · p. 3
  • The policy factors selection as πθ(a|si) = πf(f, p|si) · πz(z|si, f, p) · πe(e|si, f, p, z).Here, (f, p, z) specifies a model instance by family, parameter scale, and quantization level.Source: 5) Orchestration agent · p. 3
  • Separate policy output heads choose model family and scale, quantization, and device, conditioning on state and preceding selections.The selected model instance is defined by family, parameter scale, and quantization level.Source: 5) Orchestration agent · p. 3
  • The policy dynamically masks invalid choices under deployment and compatibility constraints, reducing the effective action space and improving sample efficiency through structured credit assignment.The output heads are conditioned on state and preceding selections.Source: 5) Orchestration agent · p. 3

IV. EVALUATION SETUP · 1) Edge testbed: · 2) LLM models:

DRLM was evaluated on a heterogeneous 64-node edge cluster with diverse model configurations and benchmark queries. Its setup captures system telemetry, semantic query categories, and varying arrival rates.

1) Edge testbed:

  • DRLM ran containerized inference on 64 heterogeneous nodes across five device classes: Raspberry Pi 3/4, Jetson Nano, Jetson Orin Nano, Jetson Orin AGX, and KVM-based VMs.The cluster used Kubernetes with containerd.Source: 2) LLM models · p. 3
  • Prometheus, cAdvisor, and tegrastats telemetry was aggregated into system state covering utilization, active load, and processing backlog, while REST endpoints enabled dispatch to valid model-device pairs.Source: 2) LLM models · p. 3

2) LLM models:

  • The deployment comprised 32 instances from eight model families, spanning 0.27 B to 20 B parameters, with lightweight quantized models on constrained devices and larger, higher-precision models on capable nodes.This placement creates diverse latency-quality trade-offs and non-uniform actions.Source: 2) LLM models · pp. 3–4, Table I · p. 4
  • The evaluation used 1258 queries from MMLU [20], GSM8K [7], CommonsenseQA [21], and TruthfulQA, covering diverse domains and query types.Source: 3) Query workload · p. 4
  • Queries were assigned to six semantic categories using embedding-based classification rather than dataset identity, mitigating dataset-specific bias in class-conditioned modeling.Source: 3) Query workload · p. 4
  • GSM8K, CommonsenseQA, and TruthfulQA were each subsampled to 200 queries for balanced workloads, and ground-truth answers enabled response-quality evaluation.Source: 3) Query workload · p. 4
  • Query arrivals followed a Poisson process with rates λ ∈{0.5, 1, 2} to model varying workload intensities.Source: 3) Query workload · p. 4

3) Query workload: … 6) Quality estimator:

The benchmarking dataset supports supervised query classification and prediction with 223 835 query-model-device measurements. The classifier maps queries to six semantic classes, and a class-conditioned lookup estimates candidate response quality at runtime.

4) Benchmarking dataset:

  • 223 835 measurements cover query-model-device tuples and include inference latency, CPU/GPU/memory utilization, and response quality for supervised training.Each record corresponds to one query-model-device tuple.Source: 4) Benchmarking dataset · p. 4
  • The dataset shows multimodal variability in query complexity and length, bimodal accuracy from binary and softscored tasks, and heavy-tailed latency across models and devices.Latency is plotted on a log scale.Source: 4) Benchmarking dataset · p. 4

6) Quality estimator:

  • Quality labels are binarized at a 0.5 threshold, while the runtime estimator uses per-class empirical model accuracy in a precomputed lookup table.Class-conditioned estimates provide expected accuracy and are queried in constant time for candidate assignments.Source: 4) Benchmarking dataset · p. 4, 6) Quality estimator · p. 4

5) Query classifier:

  • Query embeddings use all-MiniLM-L6-v2, a 6-layer Transformer that produces 384-dimensional representations.Source: 5) Query classifier · p. 4
  • A 200-tree Random Forest with maximum depth 20 maps embeddings to six semantic classes, abstracting query semantics.Source: 5) Query classifier · p. 4
  • For low-confidence or out-of-distribution queries, classification falls back to a predefined class associated with GPT-OSS 20B.The paper says GPT-OSS 20B performs well across query categories (Fig. 4a), supporting reliable behavior under uncertainty.Source: 5) Query classifier · p. 4

7) Latency predictor: · 8) DRL training: · 9) Comparison methods:

DRLM predicts inference latency from query, model, and device features, trains a factorized PPO policy using predictor-generated trajectories, and evaluates it against three baselines and two adapted state-of-the-art methods.

7) Latency predictor:

  • LightGBM predicts inference latency from query structure, model configuration, and edge-device features, using 600 trees, 63 leaves, and a 0.05 learning rate.Features include query length and lexical diversity; model family, scale, quantization, layers, and attention heads; and hardware type and CPU/GPU execution.Source: 7) Latency predictor · p. 4

8) DRL training:

  • PPO trains the orchestration policy on predictor-generated trajectories, avoiding costly LLM execution while preserving system dynamics.The query set is divided into disjoint 80% training and 20% evaluation subsets; the actor-critic shares a two-layer, 128-unit MLP encoder, with three factorized actor heads and a scalar-value critic.Source: 8) DRL training · p. 4
  • Training runs for 1,000 episodes of 100 simulated requests, using exponential inter-arrivals at rate 1 and reward weights α = 0.8 and β = 0.3.Optimization uses Adam at 3 × 10^-4, discount 0.95, entropy coefficient 0.02, clipping 0.2, and six update epochs per rollout; reward weights were empirically calibrated to balance quality and latency under normalized rewards.Source: 8) DRL training · p. 4

9) Comparison methods:

  • Under the same testbed, deployment, and workload, Random samples feasible model-device pairs, High-Acc maximizes class-conditioned accuracy, and Fastest minimizes predicted latency.Random ignores query and system state; High-Acc ignores latency and load; Fastest ignores quality.Source: 9) Comparison methods · p. 4
  • RouteLLM-style [13] routes to GPT-OSS 20 B when its predicted advantage exceeds 0.5, otherwise selecting the best weak-pool model by class-conditioned accuracy.The weak pool comprises all other models.Source: 9) Comparison methods · pp. 4–5
  • OptLLM-style replaces cost with latency, restricts choices to deployed model-device pairs, and selects from the quality-latency Pareto frontier.Source: 9) Comparison methods · p. 5

10) Evaluation metrics: … 2) DRL convergence and orchestration time:

The evaluation measures response quality, inference latency, waiting time, and orchestration overhead, and examines predictive modeling, PPO convergence, and decision cost on an edge testbed.

10) Evaluation metrics:

  • Evaluation tracks average correctness against ground truth, model execution time, pre-execution queue delay, and per-query policy decision time.These correspond to response quality, inference latency, waiting time, and orchestration overhead, respectively.Source: 10) Evaluation metrics · p. 5

V. EXPERIMENTAL RESULTS

  • DRLM is evaluated against three baselines and two state-of-the-art methods on the same edge testbed, with averages and standard deviations reported over 10 independent runs.Source: §V · p. 5

1) Predictive modeling analysis:

  • Accuracy varies across query categories with no single model dominating; the RF classifier maps queries to semantic categories consistently, with limited cross-class leakage.This supports class-conditioned quality estimation rather than global model ranking.Source: 1) Predictive modeling analysis · p. 5
  • The latency predictor aligns predicted and measured inference times (R2 = 0.75) with 0.78 ms per-instance overhead, sufficient for preserving candidate-configuration ordering.The policy needs relative ordering rather than minimal absolute prediction error.Source: 1) Predictive modeling analysis · p. 5
  • Prompt-level features dominate latency prediction, led by input length and structural complexity, followed by model configuration; device features have lower impact.Devices are partially entangled with model deployment, so model configuration captures much of the execution-cost variance associated with devices.Source: 1) Predictive modeling analysis · p. 5

2) DRL convergence and orchestration time:

  • At approximately 35 ms/query, DRLM reduces decision latency by about 60% versus OptLLM-style and RouteLLM-style, and about 30% versus Fastest.High-Acc incurs similar overhead to optimization-based methods because it evaluates all candidates.Source: 2) DRL convergence and orchestration… · p. 5
  • DRLM’s reward rises rapidly within the first 200 episodes and stabilizes at a high plateau, while critic loss approaches zero, indicating accurate value estimation and low training variance.The factorized policy decomposes decisions into sub-actions, reducing the effective action space and avoiding exhaustive evaluation.Source: 2) DRL convergence and orchestration… · p. 5

3) Comparison with baselines: · VI. CONCLUSION AND FUTURE WORK

DRLM uses state-aware model-device orchestration to reduce latency in heterogeneous edge clusters while maintaining stable response quality under increasing workloads; future work will add cost and energy objectives.

3) Comparison with baselines:

  • DRLM assigns simple queries to quantized models on constrained devices and reserves larger models on powerful nodes for complex queries.High-Acc concentrates assignments on large models, Fastest favors aggressively quantized configurations, Random shows no structure, and OptLLM- and RouteLLM-style methods are less adaptive to device heterogeneity.Source: 3) Comparison with baselines · pp. 5–6
  • At 100 requests and λ = 2, DRLM reduces inference time by 19.2% versus OptLLM and 38% versus RouteLLM, and waiting time by 52% and 62%, respectively, with at most 8% accuracy loss.Latency rises sharply for accuracy-oriented baselines as load increases; DRLM mitigates congestion by distributing queries across model-device configurations.Source: Figure 7 · p. 6, 3) Comparison with baselines · p. 6
  • DRLM has lower median and tail waiting latency, while High-Acc, OptLLM, and RouteLLM show heavy-tailed delays, with many queries waiting above 100 s.Fastest achieves low waiting time but degrades response quality.Source: 3) Comparison with baselines · p. 6
  • DRLM maintains response quality around 0.50 to 0.52 across workloads, below High-Acc and RouteLLM at 0.54 to 0.56 but above Fastest and Random.DRLM shows only minor quality variation as workload increases (nreq ∈{25, 50, 100}, λ ∈{0.5, 1, 2}).Source: 3) Comparison with baselines · p. 6

VI. CONCLUSION AND FUTURE WORK

  • DRLM shortens inference latency by up to 51% and queuing delay by up to 67% on a 64-node edge testbed.The framework combines query semantics, class-conditioned quality and feature-driven latency prediction, and dynamic resource and queue state; future work will incorporate cost and energy.Source: §VI · p. 6
  • DRLM combines embedding-based query classification, class-conditioned quality and feature-driven latency prediction, and dynamic resource and queue state for state-aware model-device selection.Its benchmark dataset contains 223 835 measurements capturing query-dependent latency-quality trade-offs across models and edge devices.Source: §VI · p. 6
  • Future work will extend DRLM toward multi-objective optimization that incorporates cost and energy.Source: §VI · p. 6

Related papers

Loading 2609.00442v1…