PaperlayerView exact revision on arXiv

Source-linked AI summary

CoAdapt: An LLM-based Framework for Adaptive Collaborative Perception in IIoT Robotic Swarms

Houssam Hajj Hassan, Antonia Maria Masucci, Lynda Zitoune, Salah-Eddine Elayoubi

arXiv:2609.16852v1cs.AIcs.RO

TL;DR

Dynamic IIoT swarms require collaborative perception to adapt as robot positions, occlusions, and network conditions change, but existing approaches rely on static participation or fusion assumptions. CoAdapt uses an LLM as a runtime controller that jointly selects participating robots and fusion algorithms from structured LiDAR-based scene descriptions. On OPV2V, it reduces communication cost by 38% while maintaining comparable detection precision, though real-time deployment is constrained by synthetic bandwidth evaluation and substantial LLM latency.

  • Problem

    Existing collaborative perception methods do not jointly adapt participant selection and fusion strategy to changing swarm dynamics and network conditions.

  • Method

    CoAdapt uses a Scene Abstraction Module and an LLM-based runtime controller to select participating robots and fusion algorithms from structured LiDAR scene descriptions and network state.

  • Results

    38% lower communication cost is achieved while maintaining detection precision comparable to static baseline approaches.

  • Takeaways & Limitations

    LLM-driven fusion policy selection provides a configurable tradeoff between communication cost and detection precision for adaptive collaborative perception.

  • Takeaways & Limitations

    Evaluation uses a synthetic bandwidth model, and mean LLM response times range from 21 to 44 seconds, constraining real-time deployment.

Abstract

from arXiv · show

Industrial IoT environments increasingly deploy autonomous mobile robots for tasks such as material handling, product assembly, or infrastructure inspection. In such deployments, collaborative perception enables robots to share LiDAR observations and collectively construct a richer model of their environment than an individual agent could produce alone. However, industrial environments are dynamic spaces where robot positions shift continuously, network bandwidth fluctuates, and the marginal contribution of robots to perception quality varies at runtime. Existing collaborative perception approaches are designed for static participation assumptions and cannot adapt to these dynamics without sacrificing either detection precision or communication efficiency. This paper presents CoAdapt, an adaptive collaborative perception framework for IIoT robotic swarms in which a Large Language Model (LLM) serves as a runtime fusion controller, jointly deciding which robots participate in the fusion process and which fusion algorithm to apply based on the current spatial configuration and network state. The LLM reasons over structured natural language descriptions of the scene derived from raw LiDAR point clouds, requiring no taskspecific training and generalizing to unseen swarm topologies. Evaluated on the OPV2V benchmark across 25 scenarios, our approach achieves a 38% reduction in communication cost while maintaining detection precision comparable to static baseline approaches.

I. INTRODUCTION

CoAdapt addresses the need for collaborative perception to adapt jointly to changing robot configurations, network conditions, and precision requirements. It uses an LLM to select participating robots and fusion algorithms at runtime without task-specific training, reducing communication cost while preserving comparable detection precision.

  • Existing approaches typically fix the fusion paradigm, adapt transmission within one architecture, or select participants under static topology assumptions.
  • CoAdapt jointly optimizes participant selection and fusion strategy at runtime to balance detection precision and communication cost in dynamic IIoT swarms.
  • The framework uses a Scene Abstraction Module and LLM controller to reason over LiDAR-derived scene descriptions, swarm geometry, and network state.
  • CoAdapt is evaluated on OPV2V and reports reduced communication cost while maintaining comparable detection precision.

II. RELATED WORK

Prior work improves collaborative perception, communication efficiency, or robotic self-adaptation separately, but does not jointly adapt fusion strategy and participant selection at runtime. CoAdapt’s stated novelty is applying LLM reasoning to LiDAR-derived scene data for these combined decisions.

  • Collaborative perception and data fusion: Collaborative perception combines multi-robot observations to extend sensing range and mitigate occlusions through early, late, or intermediate fusion.
  • Communication-efficient collaborative perception: Communication-efficient methods reduce overhead within fixed fusion paradigms, including confidence-based feature transmission and heterogeneous multi-agent attention.
  • Self-adaptive robotic systems: Self-adaptive robotic systems address runtime architecture, resource allocation, convergence, or replanning concerns beyond perception-specific fusion.
  • No cited prior approach jointly selects the fusion algorithm and participant subset using LLM reasoning over LiDAR point-cloud data.

III. THE ADAPTIVE DATA FUSION PROBLEM

The motivating scenario shows why a static fusion policy fails as robot geometry, occlusions, and network utilization change. The adaptive policy must jointly choose participants and fusion strategy while balancing precision against communication constraints.

  • Motivating Scenario: When robots are co-located, excluding redundant observations can preserve coverage while avoiding network load without improving detection precision.
  • Motivating Scenario: Transient occlusions change which robots provide unique coverage, so the optimal participant set can differ between timesteps.
  • Motivating Scenario: At 85% network utilization, late fusion and a reduced participant set can lower data volume while satisfying detection precision constraints.
  • Motivating Scenario: The optimal fusion policy varies continuously with environment state, detection precision, and network conditions during runtime operation.

B. Problem Formulation

CoAdapt formulates dynamic collaborative perception as selecting both participating robots and a fusion algorithm to balance detection precision, communication cost, and QoS under changing swarm and network states.

  • State Representation: CoAdapt models each timestep using robot positions and headings, available bandwidth, and a deployment-time minimum detection precision requirement.The environment state captures distances, field-of-view overlap, and occlusion geometry; application precision is measured as AP@IoU.
  • Policy Definition: The policy chooses a nonempty robot subset and one candidate fusion algorithm, then re-evaluates that decision as the system state evolves.Candidate algorithms include early, intermediate, and late fusion.
  • Communication Model: Communication cost is determined by per-robot transmission size, which varies across early, intermediate, and late fusion according to transmitted data representation.Early fusion transmits point clouds, intermediate fusion feature maps, and late fusion detection outputs.
  • Optimization Objective: The objective maximizes expected detection precision while penalizing communication cost through a positive tradeoff parameter λ, subject to application QoS constraints.Communication cost depends on both participant identities and the selected fusion algorithm.
  • Runtime Complexity: Real-time optimization is difficult because precision requires running the full fusion pipeline and the decision space grows across robot subsets and algorithms.CoAdapt therefore uses an LLM to approximate the state-to-policy mapping through structured natural-language reasoning instead of exhaustive online evaluation.

IV. SYSTEM ARCHITECTURE

CoAdapt integrates scene abstraction, LLM reasoning, and network control into an autonomic architecture that jointly adapts robot participation and fusion strategy to dynamic swarm conditions.

  • Architecture Overview: CoAdapt addresses shifting topology, occlusions, and fluctuating networks by using an LLM as the Plan component of a MAPE-K feedback loop.The LLM reasons over a compact JSON scene abstraction without exhaustive search or retraining for new environments.
  • Physical Layer: The physical layer supplies LiDAR observations and lightweight robot position and heading beacons to an ego robot that executes fusion decisions.The ego robot selects participants and produces the final detection output.
  • Scene Abstraction: The Scene Abstraction Module converts raw LiDAR observations into structured descriptions of object positions, sizes, and densities for LLM policy reasoning.It bridges raw point clouds and the structured representation required by the reasoning layer.
  • Perception Layer: The perception layer combines participating robots’ scene descriptions and poses with measured network metrics into a unified context-aware representation.This representation feeds the runtime fusion-control process.
  • Reasoning Layer: The reasoning layer prompts a local LLM to select participating robots and a fusion algorithm at each control cycle, returning a structured decision with a human-readable rationale.The rationale provides operators with a diagnostic signal for detecting reasoning failures.
  • Network Control Layer: The network control layer can adjust traffic priority or reallocate bandwidth when congestion prevents the reasoning layer from meeting application demands.The reasoning layer adapts within available bandwidth, while network control changes resource allocation itself.

V. EXPERIMENTAL EVALUATION

The evaluation examines CoAdapt’s runtime participant selection, bandwidth-dependent fusion adaptation, and communication cost relative to static approaches.

  • Evaluation Scope: The experiments evaluate whether CoAdapt selects participating robots under dynamic scene and network conditions, adapts fusion strategy to bandwidth, and reduces communication cost.The evaluation section introduces setup, participant selection, bandwidth adaptation, and communication-cost comparison.

A. Experimental Setup

CoAdapt is evaluated on multi-vehicle LiDAR scenarios using static fusion baselines, locally deployed LLMs, and simulated time-varying bandwidth conditions.

  • Dataset: The study uses OPV2V, which contains 73 CARLA scenarios across 6 road types and 9 cities, while evaluation retains scenarios with at least 3 vehicles.OPV2V supports up to six simultaneous agents per scenario, enabling swarm-level participant-selection evaluation.
  • Fusion Models: CoAdapt is compared with static early-, intermediate-, and late-fusion baselines implemented in OpenCOOD.The selected models are Cooper, attentive VoxelNet fusion, and PointPillars late fusion, respectively.
  • Scene Abstraction Module: The Scene Abstraction Module removes ground points with RANSAC, clusters remaining points using DBSCAN, and summarizes each cluster geometrically.Summaries include relative centroid, distance, and axis-aligned bounding-box dimensions.
  • LLM Configuration: Four open-source LLM configurations are deployed locally on an NVIDIA H100, with decisions made every 50 frames, corresponding to 5 seconds.Prompts include current scene descriptions, robot positions, and network state.
  • Bandwidth Configuration: Bandwidth is simulated with tiers of 55 Mbps, 20 Mbps, and 5 Mbps for morning, afternoon, and evening periods, plus a 15% sinusoidal fluctuation.The tiers represent high-, medium-, and low-bandwidth conditions, respectively.
  • Participant Selection: Figure 4 compares the number of selected CAVs across LLMs with simulated available bandwidth against a baseline that always includes all in-range CAVs.The baseline ignores both network conditions and scene geometry.

B. Evaluating Participant Selection

CoAdapt reduces swarm participation by selecting fewer robots according to bandwidth and spatial contribution rather than always including every in-range robot.

  • B. Evaluating Participant Selection: 26% average participant-count reduction shows CoAdapt prunes redundant CAVs relative to full participation.Reductions were 11% for Gemma 4 31B, 7% for GPT-OSS 20B, 6% for GPT-OSS 120B, and 40% for Llama3.3.
  • B. Evaluating Participant Selection: CoAdapt selects fewer participants during low-bandwidth periods and more when bandwidth is abundant.Bandwidth-response correlation is strongest for Gemma 4 and Llama 3.3.
  • B. Evaluating Participant Selection: Selected CAVs generally provide complementary geometric coverage, while excluded robots often offer redundant observations near selected participants.This pattern was consistent for Gemma 4, GPT-OSS 120B, and Llama 3.3 in a representative 45m communication range.
  • B. Evaluating Participant Selection: GPT-OSS 20B sometimes retains spatially redundant CAVs, indicating a limitation of purely LLM-driven selection.The paper motivates a deterministic fallback component for such cases.

C. Precision Evaluation under Dynamic Bandwidth

CoAdapt adapts fusion strategies to bandwidth while balancing communication cost and precision, with model-specific policies producing different tradeoffs.

  • C. Precision Evaluation under Dynamic Bandwidth: CoAdapt switches fusion paradigms across control cycles in response to bandwidth, with intermediate fusion favored at high bandwidth and late fusion under constraints.Gemma 4 and Llama3.3 switch across all three paradigms, whereas GPT-OSS models select almost exclusively late fusion and never early fusion.
  • C. Precision Evaluation under Dynamic Bandwidth: Gemma 4 reaches AP@0.7 of 0.792 at 152.2 KB/frame, an 11% communication reduction with a 2.3% precision drop versus rule-based approaches.The paper characterizes this as a favorable communication cost–precision tradeoff.
  • C. Precision Evaluation under Dynamic Bandwidth: GPT-OSS models keep average communication below 10 KB/frame but obtain AP@0.7 values of 0.766 and 0.774.Their low communication cost reflects almost exclusive late-fusion selection.
  • C. Precision Evaluation under Dynamic Bandwidth: 38% lower communication cost than default approaches is achieved by Llama 3.3 at AP@0.7 of 0.862.This is the best precision reported among the compared CoAdapt LLM configurations.
  • C. Precision Evaluation under Dynamic Bandwidth: Participant pruning reduces communication cost while maintaining comparable AP@0.7 precision relative to default configurations, including under challenging scenarios.Precision drops in S12 and S25 also occur for the default approach and are attributed to scene difficulty rather than CoAdapt selection.

D. Limitations

CoAdapt’s evaluation has important scope and deployment limitations. The study uses synthetic bandwidth conditions, scenarios with up to six agents, and LLM inference latency that is impractical for real-time control loops.

  • Synthetic bandwidth conditions omit bursty interference, spatial variation, and protocol-level dynamics found in real IIoT wireless environments.The model derives bandwidth tiers from OPV2V scenario timestamps and sinusoidal fluctuations; validation with real traces or a testbed remains future work.
  • OPV2V scenarios include up to six simultaneous agents, limiting evidence for industrial swarms with tens of concurrent robots.The participant decision space grows exponentially as the number of agents increases.
  • 21–44 seconds: mean LLM response times remain a practical constraint for real-time control-loop deployment.The reported range spans Gemma 4 31B at 21 seconds to Llama 3.3 70B at 44 seconds.

VI. CONCLUSION AND FUTURE WORK

CoAdapt uses LLMs as runtime fusion controllers that jointly select participating robots and fusion algorithms from spatial and network conditions. Future work targets network control, output validation and fallback mechanisms, and evaluation on physical platforms with real network traces.

  • VI. CONCLUSION AND FUTURE WORK: CoAdapt jointly selects the participant subset and fusion algorithm at each control cycle using spatial configuration and network state.A Scene Abstraction Module converts heterogeneous LiDAR observations into structured natural-language descriptions for LLM reasoning.
  • VI. CONCLUSION AND FUTURE WORK: Future work will add network control to increase robot participation when fusion-policy adjustment cannot satisfy application precision requirements.
  • VI. CONCLUSION AND FUTURE WORK: Future work will validate LLM outputs against physical and operational constraints and activate rule-based fallback policies for invalid or low-confidence decisions.
  • VI. CONCLUSION AND FUTURE WORK: Future evaluations will use physical robotic platforms and real-world network traces to assess CoAdapt’s sim-to-real transferability.
Loading 2609.16852v1…