PaperlayerView exact revision on arXiv

Source-linked AI summary

Network-in-the-Loop at Scale: GPU-Batched 5G Simulation for Massively Parallel Robot Learning

Zifan Zhang, Mingzhe Han, Kannan Athreya, Yuchen Liu

arXiv:2610.02370v1cs.NIcs.AIcs.DCcs.RO

TL;DR

GPU-parallel robot learning needs network-in-the-loop training that models shared 5G traffic without the per-scenario pace of packet-level simulation. Isaac-Net batches a 5G NR uplink across environments in lockstep with Isaac Lab; its engine reproduces 5G-LENA delay and closed-loop AoI behavior while supporting about one million robots on one GPU.

  • Problem

    GPU robot simulators use independent message delays, while packet-level simulators process one scenario per CPU process and cannot keep pace with thousands of parallel environments.

    Source: §I · p. 1

  • Method

    Isaac-Net batches a 5G NR uplink across environments and advances it slot by slot in lockstep with Isaac Lab physics.

    Source: §I · p. 2, A. Problem Setting · p. 3

  • Results

    With its scheduler and buffer-report grant procedure modeled, Isaac-Net reproduces 5G-LENA median delay from light load to saturation without fitted parameters; at scale, 1,048,576 robots run at 1.24 control steps per second with the network versus 1.50 without it.

    Source: §I · pp. 2–3, §VI · p. 13

  • Takeaways & Limitations

    The engine reproduces closed-loop AoI observed over ns-3, whereas independent per-message delays leave the AoI tail about three times too light.

    Source: §I · p. 3, §VI · p. 13

Abstract

from arXiv · show

Massively parallel GPU simulators train multi-robot policies in thousands of environments, and many fleets use private Fifth-Generation (5G) networks, where each robot's delay depends on its teammates' traffic. Network-in-the-loop training places a simulated 5G network inside this loop. However, GPU robot simulators reduce the network to an independent delay per message, while packet-level simulators run one scenario per CPU process and cannot keep pace with thousands of parallel environments. To bridge this gap, we present Isaac-Net, a GPU-batched 5G New Radio (NR) module that advances the uplink of thousands of environments in lockstep with Isaac Lab physics. Isaac-Net simulates every slot, the 0.5~ms interval in which the base station decides which robots transmit, for all environments at once. Extensive experiments confirm that its NR engine reproduces the median delay of ns-3 5G-LENA across loads, with a median delay 5--10\% low on an unseen carrier and 9\% high at 32 robots per environment in closed loop. The engine also reproduces the Age of Information (AoI), the age of each robot's newest delivered report, while an independent delay per message leaves the AoI tail about three times too light. In a configuration validated against 5G-LENA, Isaac-Net keeps the network in the loop for about one million robots on one GPU at 83\% of the Isaac Lab rate without the network, measured under a random policy. Isaac-Net is open source at https://github.com/ZzZTripleZzZ/isaac-net

I. INTRODUCTION

Massively parallel robot learning needs a network model that captures shared 5G contention while keeping pace with GPU simulation. Isaac-Net provides a GPU-batched NR module for this setting, reproducing key network behavior at large scale.

I. INTRODUCTION

  • A network-in-the-loop is needed because each robot’s delay depends on teammates’ traffic, while independent per-message delays miss contention and retransmissions that persist across messages.The network must also keep pace with thousands of GPU-stepped environments and support asynchronous partial resets.Source: §I · pp. 1–2
  • Isaac-Net advances all environments’ shared-cell uplinks slot by slot in lockstep with Isaac Lab, using fixed-shape tensors, per-environment clocks, and partial resets.Its configurable NR engine models scheduling, decoding errors, retransmissions, and uplink grants; reference, graph, and Triton backends are checked for output and state equivalence.Source: §I · p. 2
  • The package includes an Isaac Lab fleet task, tests, documentation, and co-simulation bridges to ns-3 5G-LENA and the OpenAirInterface 5G stack.The package is released as isaac-net.Source: §I · p. 2
  • Isaac-Net’s NR engine reproduces 5G-LENA median delay from light load to saturation without fitted parameters, and in closed loop an independent-delay model leaves the AoI tail about three times too light.At scale, 1,048,576 robots run with the network at 1.24 control steps per second on one GPU, versus 1.50 without it, under a random policy.Source: §I · pp. 2–3

II. DESIGN OF ISAAC-NET

Isaac-Net serves parallel environments in which robots share a cell’s radio resources, while Isaac Lab retains control of physics, policy, and reward. The design must advance each environment slot by slot despite asynchronous episode resets and differing simulation clocks.

A. Problem Setting

  • Each of E environments contains R robots attached to one gNB, whose radio resources they share; Isaac Lab owns physics, policy, and reward, while Isaac-Net models each message’s path to the edge server.A control step spans K uplink slots, and the networks should evolve as if each were simulated alone slot by slot.Source: A. Problem Setting · p. 3
  • MAC contention couples robots within each cell, so they cannot be simulated independently, and the network must advance across slots rather than only once per control step.Radio slots, physics substeps, and control steps run on different clocks; asynchronous episode endings also rule out restarting the whole simulation.Source: A. Problem Setting · p. 3

B. Overview of Isaac-Net

Isaac-Net accepts message classes and robot poses, advances the NR MAC over uplink slots for all environments together, and returns per-robot network outcomes. Its slot loop uses fixed-shape tensor operations to model scheduling, decoding, and retransmissions.

B. Overview of Isaac-Net

  • The policy submits each robot’s message class, Isaac-Net queues messages with their start-of-step poses, then converts start- and end-of-step poses to SNR before running the NR MAC.The returned per-robot outputs include delivered and dropped messages, delay, AoI, queue length, and SNR for observations and rewards.Source: B. Overview of Isaac-Net · p. 3

C. Time-Stepped Slot Loop

  • Slot-boundary updates let Isaac-Net replace an event queue with a fixed sequence of tensor operations over all environments and robots.The loop spans K uplink slots per control step; messages enter at control-step boundaries, and SNR is averaged over interpolated poses while fast fading still changes each slot.Source: C. Time-Stepped Slot Loop · p. 3

D. Fixed-Shape Network State

  • Fixed-shape state stores each robot’s channel, HARQ processes, and FIFO message buffer in tensors indexed by environment and robot; one-hot queue masks avoid GPU-to-CPU waits.The slot loop updates fading and access, schedules retransmissions before remaining resources using Proportional Fair priority, selects MCS, decodes transport blocks, and serves queued bytes in order. Partial resets rewrite only selected environments, while keyed random draws keep other environments unaffected.Source: D. Fixed-Shape Network State · pp. 3–4
  • At each control step’s end, the engine drops messages older than the timeout and computes delivered masks, delays, AoI, and queue lengths.These outputs accompany the updated network state.Source: D. Fixed-Shape Network State · p. 4

E. Backends and the Equivalence Guarantee

Isaac-Net offers graph and Triton backends to accelerate network simulation while checking their outputs against a PyTorch reference. The graph backend must match exactly, and Triton must match to floating-point rounding.

E. Backends and the Equivalence Guarantee

  • The graph backend replays a CUDA graph without Python overhead, while Triton fuses all control-step slots into one kernel with one GPU program per environment.The Triton kernel covers the NR slot from fading through the RLC layer and runs downlink as a second kernel; its scale configuration substitutes a fixed access delay for buffer-report grants.Source: E. Backends and the Equivalence… · pp. 4–5
  • Equivalence tests compare per-step outputs, per-message fields, and MAC state using identical decode draws and fading innovations.The graph backend must be bitwise identical to the reference; Triton must agree to floating-point rounding, with rare discrete-decision flips accepted if aggregate statistics remain unchanged.Source: E. Backends and the Equivalence… · p. 5
  • Tests use both full-run and per-step modes, with per-step tests restarting the fast backend from the recorded reference state to isolate disagreements.A torch.compile prototype-engine backend failed equivalence testing and is excluded from training runs.Source: E. Backends and the Equivalence… · p. 5

F. Fidelity Levels

Isaac-Net provides network models at multiple fidelity levels behind a shared API. They range from independent per-message delays to a configurable NR engine based on a 3GPP model.

F. Fidelity Levels

  • The i.i.d. delay model samples delay and loss independently per message, while the fluid model shares PRBs equally among active robots as their FIFO queues drain.The i.i.d. model can use log-normal delays or resample from a supplied delay sample.Source: F. Fidelity Levels · p. 5
  • The prototype engine uses a logistic BLER curve, one HARQ process, fixed noise power, and a minimal grant after two uplink slots.It is a frozen prototype that ignores MAC and PHY configuration fields.Source: F. Fidelity Levels · p. 5
  • The configurable NR engine replaces the prototype's simplifications with a 3GPP-based model, and both detailed engines support all three backends.The section compares both engines with 5G-LENA.Source: F. Fidelity Levels · p. 5

G. Configurable NR

The configurable NR engine exposes its settings through NRConfig and bundles common configurations as presets. It models uplink scheduling, decoding, retransmissions, access procedures, and downlink behavior.

G. Configurable NR

  • NRConfig configures every NR-engine module, and Table I summarizes the selectable network features.The frame structure follows 3GPP specifications, and transport-block sizes follow TS 38.214.Source: G. Configurable NR · p. 5, Table I · p. 5
  • The NR engine maps RBG SINRs to effective SINR with EESM, looks up BLER in Sionna or locally generated 5G-LENA, tables, and models HARQ, RLC, OLLA, and SR/BSR access.It also implements downlink queues, channel-quality reports, and HARQ feedback, which have not yet been compared with another simulator.Source: G. Configurable NR · p. 5
  • Four optional, unfitted settings reproduce loaded-cell 5G-LENA behavior through per-RBG PF updates, slot-wide TDMA retransmissions, prior-allocation-based MCS selection, and buffer-report grants.Buffer-report grants replace fixed access delay with a small first grant, then use late, coarse BSRs to size later grants and model RLC status-timer waiting.Source: G. Configurable NR · p. 5
  • The v1 validation preset reproduces the ns-3 5G-LENA scenario with one fitted access delay, while v2 adds the four load options without fitted values.The srsran_like and oai_like presets use uplink parameters fitted to public measurements; oai_rfsim uses measured OAI timing values and an access delay fitted to OAI traces.Source: G. Configurable NR · pp. 5–6

H. Multiple Cells

Isaac-Net supports multi-cell uplink simulation and couples network steps, resets, and observations to Isaac Lab. Its implementation includes package integrations and validation, with per-environment robot-count limits for Triton kernels.

H. Multiple Cells

  • Each environment can contain up to C cells, each with its own PF scheduler sharing a carrier; inter-cell uplink interference uses same-slot, same-subband transmissions.Link adaptation uses the previous slot's interference and decoding uses the current slot's; fractional power control is on by default for multiple cells, and handover follows the 3GPP A3 event.Source: H. Multiple Cells · p. 6

I. Coupling with Isaac Lab

  • Isaac-Net aligns K uplink slots with each Isaac Lab control step and uses hooks for setup, stepping, reset, and per-robot observations.Messages are captured at the start-of-step pose; after physics, the network runs before finished environments reset, and observations include AoI, SNR, queued messages, and delivery status.Source: §I · p. 6

III. IMPLEMENTATION

  • The implementation packages the engines behind a shared factory and API, offers PyTorch, graph, and Triton backends, and integrates with Isaac Lab and MuJoCo Playground.The package is pip-installable and BSD-3-Clause licensed; its MuJoCo Playground outputs match direct PyTorch replay bitwise.Source: §III · p. 6
  • An Isaac Lab fleet-task example trains with the network on Triton at 1024 × 16 for 30 iterations as an end-to-end pipeline check.The task uses velocity-controlled spheres in a 150 m square arena and a randomly appearing hazard reported through a large uplink message.Source: §III · p. 6
  • Validation includes MAC-timeline and transport-block-size tests, GPU backend equivalence checks, partial-reset replay checks, and bridges to 5G-LENA and OAI.The installed package passes 690 CPU tests; the last full GPU run passed 75 GPU tests.Source: §III · p. 6
  • Triton processes one environment per GPU program; beyond about 128 NR-engine or 256 prototype-engine robots per environment, scheduler search must be split into blocks.The NR Triton kernel spills registers at 64–100 robots, and porting the buffer-report grant procedure's nine per-robot arrays remains future work.Source: §III · pp. 6–7

IV. EVALUATION RESULTS

Across the 5G-LENA sweep, the NR engine’s load-aware v2 configuration closely matches delay behavior, while fast-backend checks show the GPU implementations track the reference. The evaluation also tests a held-out carrier and distinguishes the validated scale configuration from the more detailed v2 model.

A. Experimental Setup

B. Equivalence of the Fast Backends

  • The graph backend is bitwise identical to the reference, while triton disagrees in only 3 of 218,442 and 35 of 1,910,974 traffic robot-steps in two tested configurations.Different reduction order can occasionally flip a threshold decision; full-run aggregate message and delay outcomes remained close in the reported 64 × 16 test.Source: B. Equivalence of the Fast Backends · p. 7

C. Fidelity Against ns-3 5G-LENA

  • At moderate and saturated loads, v1’s median delay is about 27% and 34% low, despite fitting held-out-seed p50 and p95 errors of −0.2% and −6.8%.Its fitted 20 ms access delay absorbs waits that the buffer-report grant procedure models explicitly.Source: C. Fidelity Against ns-3 5G-LENA · p. 8
  • 2.5%, 4.8%, and 4.4% are v2’s median absolute errors for per-run median delay at light, moderate, and saturated loads, respectively.Across all 153 runs, median KS distance falls from 0.28 to 0.17 and median absolute p95 error from 8.6% to 3.5%.Source: C. Fidelity Against ns-3 5G-LENA · p. 8, Table II · p. 8
  • The scale configuration’s median absolute p50 error is 4.5–6.6%, but omitting buffer-report grants moves tail and light-load PRB use toward v1.The default NRConfig() median delay is 30%, 44%, and 76% low across the three loads.Source: C. Fidelity Against ns-3 5G-LENA · pp. 8–9
  • On the held-out 10 MHz carrier, v2’s median delay is 5–10% low at every load, while saturated-load drops are 4.1 percentage points too few.The carrier was not used to check the four v2 options or 8-byte TB header overhead.Source: C. Fidelity Against ns-3 5G-LENA · p. 9

D. Calibration Against Real 5G Stacks

Public OAI and srsRAN measurements anchor the prototype engine’s access procedure, while a running OAI stack provides a further calibration target. The resulting preset improves single-UE delay matching, but does not reproduce OAI saturation or multi-UE tails.

D. Calibration Against Real 5G Stacks

  • Fitting the prototype access procedure reduces W1 distance from 2.8 to 1.25 ms for srsRAN and from 3.3 to 2.5 ms for OAI.The fit reaches 2.0–3.1 ms on 36 held-out configurations and was not repeated on the NR engine.Source: D. Calibration Against Real 5G Stacks · p. 9
  • The uncalibrated prototype engine overestimates capacity by about 25–35% on ColO-RAN, while its shadowing changes with distance two to four times faster than measured.Its 6 dB shadowing lies within the 4–8 dB of two outdoor drive tests on the POWDER testbed,.Source: D. Calibration Against Real 5G Stacks · p. 9
  • The oai_rfsim preset matches single-UE small-message delays to a median W1 distance of 2.5 ms, versus 13.3 ms for oai_like and 13.2 ms for the 5G-LENA preset.Its median W1 distance on held-out runs is 5.5 ms, but no preset reproduces OAI saturation or multi-UE tails.Source: D. Calibration Against Real 5G Stacks · p. 9

E. Speed of the Network Engines

The triton backend enables the NR uplink to advance hundreds of thousands of robots within a control step. In the Isaac Lab fleet-task evaluation, the scale-configuration network is compared with the network disabled.

E. Speed of the Network Engines

  • 67.8 ms is the triton NR uplink time for 409,600 robots per control step, compared with 2,310 ms for the reference backend and 2,228 ms for graph.The triton kernel launches once per control step and keeps each environment’s state in registers; the other backends are limited by memory traffic from many small kernels at this scale.Source: E. Speed of the Network Engines · pp. 9–10
  • The Isaac Lab fleet-task comparison measures 128 robots per environment with the network off and with the NR engine in the scale configuration.Table IV reports means over three processes and ranges below those means.Source: Table IV · p. 10
  • On the triton backend, scale configuration costs 17–39% more than default: 1.97 versus 1.68 ms and 94.4 versus 67.8 ms.Source: E. Speed of the Network Engines · p. 10

F. Scale Inside Isaac Lab

Isaac-Net runs the network inside Isaac Lab at large fleet sizes, while packet-level ns-3 co-simulation is far more costly at scale. The multi-cell tests also show handovers occurring at the configured interruption and first-slot timing.

F. Scale Inside Isaac Lab

  • 1,048,576 robots run at 1.24 control steps per second with the NR engine, versus 1.50 without it, while the tested scale configuration supports training-speed operation up to 524,288 robots.At one million robots, the environment step runs at 83% of the rate without the network; the random-policy measurement includes physics, network, and observations.Source: F. Scale Inside Isaac Lab · p. 10, Figure 4 · p. 10

G. Cost of Packet-Level Co-Simulation

  • At 4096×128, ns-3 would require 1,071 CPU-seconds per control step, while the Isaac Lab step with the NR engine takes 0.41 s, about 83× faster.The ns-3 estimate assumes 32 threads at perfect parallel efficiency; packet-level simulators suit validation at small scale, while physics dominates the GPU environment step.Source: G. Cost of Packet-Level Co-Simulation · pp. 10–11

H. Multi-Cell Behavior

  • Two boundary crossings trigger exactly two handovers in 30 of 32 environments, with service resuming in the first uplink slot after the 40 ms interruption.The multi-cell timing behaved as specified in this test.Source: H. Multi-Cell Behavior · p. 11

I. Closed-Loop Behavior of the Fidelity Levels

Closed-loop tests compare network models across three traffic settings using delay, Age of Information (AoI), and messages sent. The NR engine generally tracks ns-3 more faithfully under load, while independent per-message delays miss temporal dependence and the AoI tail.

Results under load

  • At 32 robots per environment, the NR engine’s median delay is 9% above ns-3’s, at 525 against 483 ms, while its 95th-percentile AoI is 2.00 against 1.94 s.The NR engine sends 11,235 messages against ns-3’s 11,353; the log-normal i.i.d. model has a 33% lower AoI p95 and sends 4.5% more messages.Source: Table V · p. 11, §I · p. 12, Figure 5 · p. 12
  • In the 0.2 s setting, the NR engine’s delay KS distance from ns-3 is 0.12, at the 0.12–0.13 seed-noise level, while the i.i.d. delay model’s distance is 0.22.The i.i.d. model’s p95 delay is 404 ms against ns-3’s 500 ms, and its 95th-percentile AoI is 29% lower.Source: Table V · p. 11, §I · p. 12, Figure 5 · p. 12

Why independent delays miss the tail

  • The empirical i.i.d. model’s AoI tail is about three times too light: only 1.3–1.4% of robot-steps exceed ns-3’s AoI p95, versus 4.2–4.4% under the NR engine and ns-3.Successive delays for one robot correlate at 0.86–0.94, so long delays can persist for that robot and raise its AoI across messages.Source: §I · p. 12

Interpretation of task return

  • Task return is 81.8–82.9 for every model in every setting, so this controller’s return does not distinguish the network models in the tested conditions.The evaluation therefore compares delay, AoI, and messages sent.Source: §I · p. 12

V. RELATED WORK

Prior robot-network systems either keep the network out of training or couple one scene to one network instance, while GPU and packet-level tools do not jointly provide the required lockstep, closed-loop scale. The conclusion presents Isaac-Net as a GPU-batched per-slot model that meets this setting and reports validation and scale results.

V. RELATED WORK

  • Existing robot-network systems generally couple one robot scene to one network instance, while Isaac-Net runs a per-slot model for thousands of environments and about one million robots on one GPU.The cited work spans middleware co-simulation, network digital twins, and deployment over private 5G without network-in-the-loop training.Source: §V · p. 13
  • GPU network simulators target wired networks or detailed deployments, while 5G-LENA runs one scenario per CPU process; Isaac-Net adds per-robot queues, HARQ timing, and lockstep coupling for learning.Isaac-Net reuses Sionna BLER tables and validates against 5G-LENA and OAI.Source: §V · p. 13

VI. CONCLUSION

  • The conclusion reports that the NR engine reproduces 5G-LENA median delay across loads and controller-observed AoI, while independent per-message delay leaves the AoI tail about three times too light.In the validated scale configuration, 1,048,576 robots run at 1.24 control steps per second with the network, versus 1.50 without it.Source: §VI · p. 13

Related papers

Loading 2610.02370v1…