PaperlayerView exact revision on arXiv

Source-linked AI summary

Goodput Maximization for Large Language Model Edge Inference: A Two-Phase Maskable PPO Approach

Xiaojing Chen, Qi Zhang, Wei Ni, Shunqing Zhang, Yanzan Sun

arXiv:2608.25543v1eess.SYcs.AI

TL;DR

Wireless edge LLM inference must address throughput and latency while meeting strict SLOs, but prior studies overlooked throughput as a critical service metric. The paper proposes TP-MPPO, combining masked offloading optimization with analytical and greedy bandwidth allocation, and reports higher reward and goodput than benchmarks.

  • Problem

    Wireless edge LLM inference requires higher throughput and lower latency under strict SLOs, while prior studies overlooked throughput as a critical metric.

  • Method

    TP-MPPO combines maskable MPPO for task offloading with closed-form uplink and greedy downlink bandwidth allocation in a joint goodput optimization framework.

  • Results

    TP-MPPO improves system reward by 33.3%–87.5% over benchmarks and achieves the highest goodput across diverse network configurations.

  • Takeaways & Limitations

    Goodput-oriented joint offloading and bandwidth allocation can improve SLO-compliant service performance in wireless edge LLM inference.

Abstract

from arXiv · show

This paper presents a novel two-phase maskable proximal policy optimization (TP-MPPO) algorithm, which maximizes the system goodput counting request throughput with strict service level objective (SLO) compliance for large language model (LLM) inference services in wireless edge networks. In the first phase of TP-MPPO, we optimize the task offloading decisions by MPPO with action masking mechanism, effectively avoiding exploring invalid actions and reducing the action space. In the second phase, closed-form solutions are derived for uplink bandwidth allocation; a greedy algorithm is designed for downlink bandwidth allocation to provide immediate rewards for the MPPO in the next round. The two stages alternate till convergence. Simulation results demonstrate that TP-MPPO can improve the system reward by 33.3%--87.5% compared to its benchmarks and achieve the highest goodput.

I. INTRODUCTION

The paper motivates goodput-oriented resource management for wireless edge LLM inference and formulates joint offloading and bandwidth allocation under service and memory constraints. It proposes TP-MPPO to address this optimization problem with a two-phase approach.

  • Edge computing supplements cloud LLM inference by providing distributed resources and reducing propagation latency through nearby edge nodes.
  • Existing studies overlooked throughput or the differing latency requirements captured by TTFT, TPOT, and E2E SLOs.Goodput measures request throughput under SLO attainment, reflecting both service quality and cost.
  • The paper models full E2E latency and GPU memory constraints for joint task offloading and bandwidth allocation.
  • The resulting MINLP maximizes goodput while enforcing SLO and memory capacity constraints.
  • TP-MPPO uses MPPO for offloading and analytical bandwidth allocation, with closed-form uplink and greedy downlink solutions.The two phases provide immediate rewards for policy optimization and alternate until convergence.

A. Communication Model

The communication model represents uplink and downlink transmissions between users and wireless edge nodes using bandwidth allocation, transmit power, channel gains, and fading assumptions.

  • The system uses OFDM with non-overlapping frequency bands at different edge nodes to avoid interference.
  • Bandwidth fractions α^up,t_nj and α^down,t_nj are assigned to users for uplink and downlink transmissions.
  • The wireless channel combines large-scale path loss with small-scale Rayleigh fading.The effective channel gain is treated as a slot-level average under quasi-static fading.
  • Uplink and downlink rates depend on allocated bandwidth, transmit power, background noise, and channel gain.

B. Latency and Memory Analysis

The latency model decomposes each request’s end-to-end delay across scheduling, transmission, batching, inference, and output feedback, while memory accounts for model weights and KV-cache storage. Batch size, sequence lengths, model dimensions, and GPU capacity shape inference latency and memory usage.

  • Transmission latency: Uplink latency depends on each request’s prompt data size, while downlink latency depends on its output data size.Each transmission is assumed to complete within one scheduling slot under quasi-static channel conditions.
  • Latency components: End-to-end latency includes AP scheduling wait, uplink transmission, batching wait, batch inference, and output feedback.Users may wait for other assigned requests to finish prompt transmission before batch inference begins.
  • Batch inference: Requests assigned to one edge node are processed together, with batch completion latency determined by prefill and decoding latency.The inference model follows an analytical Transformer latency formulation.
  • Inference latency: Larger batch sizes, longer sequences, and larger models increase computation latency, while greater edge-node FLOP capacity reduces it.The model uses padded input length, batch-level output length, Transformer layers, hidden dimensions, and node computational capacity.
  • Memory footprint: An edge node’s batch-inference memory footprint comprises model-weight memory and KV-cache memory.KV-cache requirements vary with the batch’s requests and their generated outputs.

III. PROBLEM FORMULATION

The formulation maximizes the number of SLO-compliant completed requests by jointly selecting task offloading and bandwidth allocations under latency and memory constraints. The resulting MINLP is difficult because binary decisions are non-convex and batch assignments couple users’ latency and memory consumption.

  • Completion criterion: A request is successfully completed only when it is assigned, avoids OOM failure, and returns its output to the user.The completion indicator is tied to assignment and successful execution conditions.
  • Objective: The objective counts completed requests that satisfy the end-to-end latency SLO as system goodput.Rejected requests, OOM failures, and completed requests that violate the SLO are excluded.
  • Decision variables and constraints: The optimization jointly chooses offloading decisions and uplink and downlink bandwidth allocations under SLO and memory-capacity constraints.The objective accounts for dynamic KV-cache footprints and heterogeneous SLO requirements.
  • Metric: Goodput measures request throughput while enforcing service quality through end-to-end latency SLO attainment.This formulation bridges raw throughput and SLO compliance.
  • Optimization challenge: The problem is an NP-hard mixed-integer nonlinear program with non-convex, discontinuous binary decisions.Batch latency and memory depend on the specific combination of tasks assigned to each node.

IV. PROPOSED TP-MPPO SCHEME

TP-MPPO solves the formulation through alternating offloading optimization and bandwidth allocation. It uses closed-form uplink allocation and greedy downlink allocation to support the offloading policy.

  • Two-phase design: TP-MPPO first determines task offloading decisions with MPPO, then solves uplink and downlink bandwidth allocation.The bandwidth stage is decomposed into two subproblems.
  • Bandwidth allocation: The uplink bandwidth subproblem has a closed-form solution, whereas the downlink bandwidth subproblem uses a greedy-based algorithm.These solutions provide the next policy-optimization round with immediate rewards.

A. PPO Framework with Invalid Action Masking

The offloading phase extends PPO with state-dependent invalid-action masking. Invalid edge-node selections receive zero probability, reducing exploration to executable actions.

  • Maskable PPO: MPPO applies invalid-action masking to avoid infeasible offloading actions under state-dependent constraints.The approach extends PPO for complex action spaces.
  • Dynamic masking: A binary mask is dynamically generated for each user according to current node availability and environment state.The mask indicates whether an edge node can be selected at the current slot.
  • Action-space reduction: Action masking dynamically prunes the action space while ensuring the resulting offloading action remains executable.The masked action space contains only feasible offloading choices.
  • Action execution: During actor-network propagation, invalid-action logits are set to −∞, so Softmax assigns them zero selection probability.This prevents the policy from selecting unavailable nodes.

3) Reward:

The reward combines normalized goodput with a penalty for memory-overflow failures, while maskable MPPO selects only executable offloading actions and updates its policy from environment transitions.

  • Reward formulation: The immediate reward is a normalized utility that maximizes system goodput while penalizing memory-overflow failures.The penalty uses the number of overflow-failed requests relative to total generated requests, weighted by ν_good and ν_oom.
  • Masked action selection: MPPO dynamically prunes invalid offloading choices by assigning them zero selection probability after Softmax.The action mask ensures the selected offloading action remains executable under state-dependent constraints.
  • Two-phase interaction: For each selected offloading action, uplink and downlink subproblems determine bandwidth allocations before the environment transitions to the next state.The complete action set updates the environment, including edge-node state variables and the next state.
  • Policy optimization: The behavior actor generates masked-policy actions, the critic estimates state values, and replay-buffer trajectories update the target actor and critic.The stored trajectories contain states, masks, actions, rewards, and next states.

B. Closed-Form Solutions to Uplink Bandwidth Allocation

Batch inference makes each edge node’s uplink completion time depend on its slowest assigned user, so uplink allocation minimizes the maximum latency through min-max fairness.

  • Uplink latency: Batch inference requires all prompts to be uploaded before padding, making prompt transmission and waiting times depend on the slowest assigned user.For users offloaded to edge node j, the batch’s uplink latency is governed by the maximum user latency.
  • Optimization objective: The uplink bandwidth subproblem minimizes the maximum uplink latency among users assigned to each edge node.This formulation directly targets the bottleneck created by batch synchronization.
  • Closed-form principle: The optimal strategy equalizes uplink latencies across all users in the batch according to the min-max fairness principle.Equalization provides the closed-form allocation structure for the uplink subproblem.

C. Optimal Solution to Downlink Bandwidth Allocation

Downlink allocation maximizes the number of requests satisfying their latency SLOs by selecting requests with feasible minimum bandwidth requirements.

  • Optimization objective: The downlink problem maximizes the number of requests meeting the SLO, aligning the allocation objective with system goodput.Requests satisfying the downlink latency constraint form the set contributing to goodput.
  • Bandwidth requirements: For each request, the method computes the minimum required downlink bandwidth fraction by setting its downlink latency constraint.This converts latency feasibility into a bandwidth requirement used by the allocation algorithm.
  • Subset selection: The resulting problem seeks the largest request subset satisfying the total downlink bandwidth constraint.Each selected request contributes one unit to goodput under the equal-value structure.
  • Greedy algorithm: Sorting requests by minimum bandwidth requirement and prioritizing the smallest requirements yields an optimal solution with zero optimality gap.Algorithm 1 has complexity O(N log N), dominated by sorting.

V. NUMERICAL RESULTS

Experiments compare TP-MPPO with four offloading baselines across changing deployment, workload, model-size, and GPU-provisioning conditions. TP-MPPO achieves the highest reward and goodput-oriented performance across these configurations.

  • Experimental setup: The evaluation uses randomly deployed edge nodes and users, discrete input and output lengths, L2 GPU specifications, and stated communication and inference parameters.The baseline comparison includes MPPO, TP-PPO, Rewardless, and Heuristic.
  • Reward convergence: TP-MPPO achieves the highest reward and converges within 200 episodes.Action masking accelerates convergence by eliminating invalid-action exploration.
  • Reward comparison: 33.3%, 87.5% and 71.4% higher reward than MPPO, TP-PPO and Rewardless, respectively, is achieved by TP-MPPO.TP-PPO exhibits early cold-start difficulty, while Rewardless converges to a low reward associated with insufficient avoidance of OOM states.
  • Model-size effects: Lighter models provide lower latency and higher throughput and goodput with a smaller throughput–goodput gap.Llama-7B has a larger gap under higher memory and computational demands.
  • Deployment scaling: With 6 nodes, TP-MPPO’s goodput ratio exceeds MPPO, TP-PPO, Heuristic and Rewardless by 5.4%, 9.5%, 7.1% and 10.2%, respectively.With 3 nodes and 30 users, the corresponding margins are 3%, 7.3%, 6.5% and 7%.
  • GPU provisioning: With increasing GPU resources for Llama-30B, TP-MPPO improves the goodput ratio from 42.0% to 60.7%.The reported improvement is associated with fewer OOM failures and lower inference latency.

VI. CONCLUSIONS

TP-MPPO combines learned task offloading with closed-form uplink and greedy downlink bandwidth allocation for edge LLM inference. It improves reward by 33.3%–87.5% and achieves the best goodput across varied network configurations.

  • TP-MPPO learns task offloading decisions with MPPO, while allocating uplink bandwidth in closed form and downlink bandwidth greedily.
  • 33.3%–87.5% reward improvement over benchmarks was achieved by TP-MPPO.
  • TP-MPPO significantly outperforms benchmarks in goodput across various network configurations.
Loading 2608.25543v1…