PaperlayerView exact revision on arXiv

Source-linked AI summary

DRLM: Deep Reinforcement Learning-Based LLM Query Orchestration in Edge Environments

Reza Farahani, Zoha Azimi Ourimi, Mario Colosi, Lauri Loven, Christian Timmerer, Schahram Dustdar

arXiv:2609.00442v1cs.DC

TL;DR

Heterogeneous queries, devices, and LLM configurations make edge orchestration a joint latency-quality-resource decision problem. DRLM combines query-aware quality and latency predictors with a factorized PPO agent using runtime state, and evaluates the approach on a large benchmark and 64-node cluster. It reduces inference latency and queuing delay substantially while limiting accuracy loss.

  • Problem

    Heterogeneous model families, scales, quantization levels, and edge devices create query-dependent latency-quality-resource trade-offs that existing simplified or static orchestration approaches do not jointly capture.

  • Method

    DRLM combines semantic query profiling, class-conditioned quality and configuration-dependent latency predictors, and a factorized PPO agent that uses resource utilization and queue state for per-query decisions.

  • Results

    DRLM reduces inference latency by up to 51 % and queuing delay by up to 67 %, while incurring at most 8 % accuracy loss.

  • Takeaways & Limitations

    The evaluation supports state-aware, fine-grained model-device orchestration for heterogeneous edge clusters across query-dependent latency-quality trade-offs.

Abstract

from arXiv · show

Large language model (LLM) services increasingly process heterogeneous queries with diverse latency, accuracy, and resource requirements. While edge deployment reduces response time, the heterogeneity of devices and the diversity of model families, parameter scales, and quantization levels make efficient LLM query orchestration challenging. This paper introduces DRLM, a Deep Reinforcement Learning-based LLM query orchestration framework in edge environments. DRLM integrates two lightweight predictors: (i) a class-conditioned quality estimator that maps queries to semantic categories and infers model performance, and (ii) a feature-driven latency predictor that estimates inference time across model-device configurations. These predictions, combined with system state, feed a factorized Proximal Policy Optimization (PPO) agent that performs state-aware orchestration decisions. To enable data-driven orchestration, we construct a large-scale benchmarking dataset with 223 835 measurements spanning 1258 queries, 6 query classes, 8 model families (32 deployed instances), 5 quantization levels, and heterogeneous edge devices. Evaluation on a 64-node edge cluster and comparison with three baselines and two state-of-the-art methods show that DRLM reduces inference latency by up to 51% and queuing delay by up to 67 %, while incurring at most 8% accuracy loss. It improves latency under increasing workloads up to 61.4%, demonstrating robust and stable orchestration.

I. INTRODUCTION

Heterogeneous edge devices and diverse model configurations create a fine-grained, query-specific orchestration problem because latency, quality, and resource trade-offs vary across workloads. DRLM addresses this with predictive, state-aware orchestration and reports lower latency and queuing delay with limited accuracy loss.

  • I. INTRODUCTION: Edge orchestration must select jointly among model family, parameter scale, quantization, and device for each query.Configuration choice is a per-query decision because heterogeneous deployments exhibit distinct latency-quality-resource trade-offs.
  • 1) Model-family impact:: Qwen3(0.6, 4) and (0.6, 16) outperform others on TruthfulQA, indicating model architecture affects task-specific performance beyond size.The cited comparison is specific to TruthfulQA queries.
  • 1) Model-family impact:: Scaling benefits are workload-dependent: Gemma3 configurations improve Mathematics accuracy but do not consistently dominate across TruthfulQA.On RP, Gemma3(1, 4) improves Mathematics over (0.27, 4) but degrades TruthfulQA; on JOA, Gemma3(12, 4) similarly does not dominate across tasks.
  • 3) Quantization impact:: Quantization effects are configuration- and device-dependent rather than uniformly improving latency or accuracy.On RP, Gemma3(0.27, 16) improves TruthfulQA accuracy over (0.27, 4) but increases latency; on JOA, Qwen3(0.6, 16) has higher latency than (0.6, 4) with marginal accuracy gains.
  • I. INTRODUCTION: DRLM combines query profiling, class-conditioned quality prediction, configuration-dependent latency prediction, and runtime state in a factorized PPO agent.The framework was evaluated on 64 heterogeneous nodes with 32 LLM instances spanning 8 model families, 11 parameter scales, and 5 quantization levels.
  • I. INTRODUCTION: DRLM reduces inference latency by up to 51 % and queuing delay by up to 67 %, while incurring at most 8 % accuracy loss.These results come from evaluation on a 64-node edge testbed against three baselines and two state-of-the-art methods.

II. PROBLEM FORMULATION

The orchestration problem is formulated as a Markov Decision Process in which an agent assigns each incoming query to a feasible model-device configuration. The environment supplies predictive latency and quality feedback to support fast candidate evaluation under deployment and resource constraints.

  • II. PROBLEM FORMULATION: LLM query orchestration is formulated as a Markov Decision Process with state space S, action space A, policy π, and reward function r.At each decision step, the agent observes the system state and assigns an incoming query to a model-device configuration.
  • II. PROBLEM FORMULATION: Each deployed instance is a distinct configuration defined by model family, parameter scale, and quantization level on a heterogeneous device.Queries arrive online and must be assigned to feasible model-device pairs subject to deployment and resource constraints.
  • II. PROBLEM FORMULATION: Lightweight predictors estimate inference latency and response quality for candidate query-model-device assignments without performing full inference.The latency predictor uses T(x_i, m, e), while the quality predictor uses A(x_i, m).

1) Environment:

DRLM models each arriving query as a constrained assignment to a feasible model-device pair. Its state and reward jointly represent query characteristics, device conditions, queueing, quality, and end-to-end latency.

  • Environment: The state combines the query representation with device utilization and queue information for each edge device.Queue state includes queue length and remaining processing time of active requests.
  • Environment: Each action selects a model configuration and execution device subject to deployment and compatibility constraints.
  • Environment: The reward trades normalized predicted response quality against penalized end-to-end latency, including inference and queue waiting time.An exponential penalty makes high-latency assignments costlier than a linear formulation.
  • Environment: PPO samples a feasible model-device action for each query and optimizes the policy using the resulting reward.

5) Policy learning:

DRLM trains an actor-critic PPO policy using predictor-generated trajectories rather than real LLM inference. Its policy objective combines clipped policy optimization, value learning, and entropy regularization.

  • Policy learning: The PPO objective uses the probability ratio, advantage estimate, value-function loss, and entropy bonus to optimize the policy.The entropy term encourages exploration, while coefficients control value-loss and entropy contributions.
  • Policy learning: DRLM trains an actor and critic jointly, with the policy acting as the actor and a value function serving as the critic.
  • Policy learning: Predictor-generated trajectories avoid executing real LLM inference during training while retaining dynamic system conditions.
  • Policy learning: The architecture comprises five modules that jointly support per-query model-device selection in heterogeneous edge clusters.

1) Query profiler:

The query profiler converts incoming queries into structural and semantic representations, while lightweight predictors estimate quality and latency for candidate assignments without inference execution.

  • Query profiler: The profiler extracts lightweight structural features and a dense sentence-encoder embedding from each incoming query.Examples include token length and syntactic complexity.
  • Query profiler: A lightweight classifier maps each query to a semantic class distribution that abstracts query intent.Low-confidence or out-of-distribution queries use a default class associated with robust configurations.
  • Query profiler: The predictors estimate query-dependent model behavior, enabling rapid evaluation of candidate model-device assignments without inference execution.
  • Query profiler: The quality estimator infers expected model performance from query classes rather than noisy per-query accuracy predictions.
  • Query profiler: The latency predictor uses query features, model configuration, and device characteristics to estimate inference time under varying conditions.

4) Runtime monitor:

The runtime monitor supplies deployment and dynamic cluster information to DRLM’s decision process. The agent combines these runtime signals with query and prediction features through a factorized policy.

  • Runtime monitor: The cluster state manager tracks resource utilization, execution state, current load, and remaining processing time of active requests.These measurements form the system state vectors for orchestration.
  • Runtime monitor: The model instance manager tracks deployment and availability across devices, complementing the cluster state manager’s runtime measurements.
  • Runtime monitor: The agent combines query representation, predicted performance, and runtime state into a compact decision representation.
  • Runtime monitor: The factorized policy autoregressively selects model family and scale, quantization, and device to manage the combinatorial action space.
  • Runtime monitor: Separate policy heads condition each selection on the state and preceding choices, while masking removes invalid deployment or compatibility choices.This factorization reduces the effective action space and improves sample efficiency.

IV. EVALUATION SETUP

DRLM is evaluated on a heterogeneous Kubernetes edge cluster using diverse deployed LLM configurations and semantically grouped benchmark queries.

  • Evaluation setup: The evaluation setup covers the testbed, model deployment, workload, and evaluation methodology.Table I summarizes the deployed models on the edge devices.
  • Testbed: The testbed contains 64 heterogeneous nodes across five device classes, including Raspberry Pis, Jetson devices, and KVM-based virtual machines.The cluster runs Kubernetes with containerd and includes varied CPU, GPU, memory, and virtualization resources.
  • Model deployment: The deployment comprises 32 LLM instances from eight model families, spanning 0.27 B to 20 B parameters and diverse quantization levels.Lightweight quantized models target constrained devices, while larger and higher-precision models run on capable nodes.
  • Workload: The workload contains 1258 queries from MMLU, GSM8K, CommonsenseQA, and TruthfulQA, grouped into six semantic categories.Categories represent semantic query types rather than dataset identities, reducing dataset-specific bias.

3) Query workload:

The benchmarking workload combines broad query diversity with measurements that support class-conditioned quality estimation and latency prediction.

  • Benchmarking dataset: The benchmarking dataset contains 223 835 query-model-device measurements covering model configurations, quantization levels, and edge devices.Each record includes inference latency, resource utilization, and response quality for supervised predictor training.
  • Distributions: Query complexity and length are multi-modal, while accuracy is bimodal and latency is heavy-tailed across the benchmark.Accuracy binarization uses a 0.5 threshold for consistent supervision.
  • Query representation: Queries are encoded with the all-MiniLM-L6-v2 sentence encoder, producing 384-dimensional embeddings for semantic classification.The encoder is a 6-layer Transformer.
  • Semantic classification: A 200-tree Random Forest maps query embeddings to six semantic classes and falls back to a predefined GPT-OSS 20B-associated class for uncertain queries.The fallback is intended to provide reliable behavior for low-confidence or out-of-distribution inputs.
  • Quality estimation: Class-conditioned empirical statistics provide constant-time runtime estimates of expected model accuracy for candidate assignments.The estimates are stored in a precomputed class-model lookup table.

7) Latency predictor:

DRLM predicts latency from query, model, and device features, then trains a factorized PPO policy using predictor-generated trajectories and comparisons against routing baselines.

  • Latency predictor: A LightGBM regressor predicts inference latency from query structure, model configuration, and edge-device characteristics.Features include input counts and lexical diversity, model family and scale, quantization, architecture, and CPU/GPU execution.
  • Policy training: PPO trains the orchestration policy on simulated trajectories generated by the predictors, avoiding costly LLM execution while preserving system dynamics.The actor is factorized into three heads and the critic outputs a scalar value estimate.
  • Comparison methods: DRLM is compared with Random, High-Acc, Fastest, RouteLLM-style, and OptLLM-style methods under the same testbed, deployment, and workload.The comparison spans three baselines and two state-of-the-art methods.
  • Predictive modeling: The predictive-modeling figure reports model accuracy variability across query categories and the consistency of the semantic classifier.The supplied caption identifies Figure 4 as predictive modeling performance in DRLM.
  • Training and orchestration: The training-dynamics and orchestration figures cover PPO learning behavior, orchestration overhead, and waiting-time comparisons across methods.Figures 5 and 6 respectively address training dynamics and orchestration or waiting time.

10) Evaluation metrics:

Evaluation measures quality, execution latency, queueing delay, and orchestration overhead, using repeated runs and analyses of predictive behavior and decision efficiency.

  • Evaluation metrics: The evaluation reports response quality, inference latency, waiting time, and orchestration overhead as complementary metrics.These metrics respectively capture correctness, execution efficiency, queueing congestion, and policy decision time.
  • Evaluation protocol: Results are averaged with standard deviations over 10 independent runs for comparisons across methods.All methods use the same edge testbed, deployment, and workload.
  • Predictive modeling analysis: No single model dominates across query categories, supporting query-dependent rather than globally ranked quality estimation.The classifier also shows high consistency in mapping queries to semantic categories.
  • Latency predictor analysis: Input length and structural complexity contribute most to latency prediction, followed by model configuration, while device features have lower impact.Model configuration captures variance associated with execution cost because devices are partially entangled with deployment.
  • Orchestration overhead: Approximately 35 ms/query orchestration overhead is about 60 % lower than OptLLM-style and RouteLLM-style and about 30 % lower than Fastest.The factorized policy reduces the effective action space and avoids exhaustive candidate evaluation.

3) Comparison with baselines:

DRLM uses load-aware, state-aware orchestration to distribute queries across heterogeneous model-device configurations. Compared with baselines, it maintains stable quality while reducing latency and queuing delays under increasing workloads.

  • Model selection: DRLM distributes simple queries to quantized models on constrained devices while reserving larger models on powerful nodes for complex queries.This load-aware policy spans lightweight and high-capacity configurations rather than concentrating assignments on one model type.
  • Model selection: High-Acc creates hotspots by concentrating assignments on large models, whereas Fastest collapses to aggressively quantized configurations.OptLLM- and RouteLLM-style methods partially balance the trade-off but remain less adaptive to device-level heterogeneity.
  • Response quality: DRLM maintains response quality around 0.50 to 0.52 across workloads, outperforming Fastest and Random while varying less than accuracy-oriented baselines.High-Acc and RouteLLM achieve the highest accuracy, from 0.54 to 0.56, but their latency rises sharply as load increases.
  • End-to-end latency: 52% and 62%: DRLM reduces waiting time versus OptLLM and RouteLLM, respectively, under 100 requests with λ = 2.Under the same highest-load condition, inference time decreases by 19.2% and 38% versus OptLLM and RouteLLM, with at most 8% accuracy loss.
  • Overall comparison: Up to 51% lower inference latency and 67% lower queuing delay: DRLM improves latency on a 64-node edge testbed.The evaluation uses a large-scale benchmark and compares DRLM with multiple baseline and state-of-the-art methods.
Loading 2609.00442v1…