PaperlayerView exact revision on arXiv

Source-linked AI summary

AIMS: An Agentic AI Framework for Sim-to-Real Multi-Modal ISAC

Yijie Bian, Kai Zhang, Wei Guo, Zixin Wang, Shenghui Song, Jun Zhang, Khaled B. Letaief

arXiv:2609.39964v1cs.AIcs.MA

TL;DR

Deployment-specific multi-modal ISAC requires aligned sensing and wireless data, yet configuring synthetic pipelines for a target site remains challenging. AIMS coordinates deployment-specific simulation and learning through two agents, structured planning, and validation-driven revision. Experiments report improved detection and beam prediction over considered baselines and stronger orchestration performance across deployment requests.

  • Problem

    Multi-modal ISAC models need annotated, physically aligned sensing and synchronized wireless data, while adapting synthetic pipelines requires consistent coupled scene, sensing, wireless, and learning configurations.

    Source: §I · p. 1

  • Method

    AIMS grounds a natural-language task, deployment conditions, and real-data budget in shared state, then coordinates aligned scene construction and task learning with structured planning and validation-driven revision.

    Source: B. Contributions · p. 2, §III · p. 5

  • Results

    AIMS improves vehicle detection and beam prediction over considered simulation and fusion baselines, while its orchestration benchmark reports higher plan correctness and dependency handling across both request tiers.

    Source: B. Performance Comparison for Beam… · p. 10, C. Performance Comparison for Object… · p. 11, D. Performance Comparison for Agentic… · p. 11

  • Takeaways & Limitations

    The reported experiments support AIMS for deployment-specific sim-to-real learning and orchestration across the evaluated tasks and deployment requests.

    Source: §VI · pp. 12–13

Abstract

from arXiv · show

Multi-modal integrated sensing and communication (ISAC) enables environmental perception and reliable connectivity for intelligent wireless networks. Data-driven multi-modal ISAC models depend heavily on annotated real-world data to learn relationships across sensing and wireless observations, thereby constraining scalable deployment. Although synthetic data generation reduces the burden, adapting existing simulation pipelines to a target deployment requires consistent scene, sensing, wireless, and learning configurations, while mismatches among these coupled components impair sim-to-real transferability. To address the challenge, we propose an agentic artificial intelligence (AI) framework for sim-to-real multi-modal ISAC, named AIMS. Given a natural-language deployment request specifying the target task, deployment conditions, and real-data budget, AIMS derives a deployment-specific sim-to-real configuration and coordinates its execution to produce a deployment-specific task model. A two-agent architecture coordinates scene construction with task learning. A scene construction agent generates geographically grounded, synchronized sensing and wireless records from shared physical states, while a scene understanding agent configures task-relevant modalities and mixture-of-experts (MoE) learning for zero-shot inference or few-shot adaptation. Structured domain knowledge guides dependency-aware planning, while validation evidence supports feedback-driven revision of affected decisions. Experiments on the real-world DeepSense~6G dataset demonstrate improved vehicle detection and beam prediction over the considered simulation and fusion baselines. A separate orchestration benchmark evaluates task interpretation, dependency reasoning, and feedback-driven replanning across diverse deployment requests, showing improved plan correctness with structured domain knowledge and validation feedback.

I. INTRODUCTION

AIMS addresses the coupled configuration and data-alignment challenges of deployment-specific sim-to-real multi-modal ISAC. It coordinates scene construction and task learning through structured planning and validation-driven revision, and is evaluated on DeepSense 6G and deployment-request orchestration.

I. INTRODUCTION

  • Site-specific learning requires physically aligned sensing records, synchronized wireless measurements, and task supervision, making expanded deployment coverage costly to collect.Synthetic data reduce measurement burden, but coupled scene, sensing, wireless, and learning mismatches cannot be fixed by increasing synthetic sample size alone.Source: §I · p. 1

A. Related Works

  • Prior work provides multimodal data, digital twins, and agentic wireless tools, but adapting pipelines to new sites or tasks still requires expert coordination and dependency-aware cross-domain alignment.Deployment changes can require selective reuse or regeneration of intermediate products while preserving consistency across scene construction, sensing, wireless data, and learning.Source: A. Related Works · p. 2

B. Contributions

  • AIMS grounds a deployment request in shared experiment state, uses task, capability, and dependency knowledge to plan reusable products, and revises affected decisions using validation evidence.Its domain capabilities generate sensing, wireless, and task-learning products under shared physical states, with references and provenance propagated to preserve correspondence.Source: B. Contributions · p. 2
  • AIMS is evaluated on DeepSense 6G for vehicle detection and beam prediction, and its orchestration benchmark reports higher plan correctness and dependency-handling performance across normal and challenging requests.The benchmark covers 140 natural-language deployment requests.Source: B. Contributions · p. 2

II. SYSTEM MODEL AND PROBLEM FORMULATION

The formulation represents a deployment through its task, conditions, and real-data budget, then connects shared scene states to sensing and communication observations. A roadside vehicular network provides the representative system setting.

A. System Model

  • The request specifies the target task, deployment conditions, and real-data budget; labeled real support is available when real-domain adaptation is requested.The construction specification defines the synthetic counterpart under the resulting deployment conditions.Source: A. System Model · p. 3
  • Scene geometry, environmental and surface attributes, and time-varying object identities, poses, and motion form a shared physical reference for sensing and wireless propagation.System configurations determine how these conditions are observed and used for task supervision.Source: A. System Model · p. 3
  • The representative roadside setting pairs an RSU downlink with a target vehicle as receiving terminal while sensors observe the environment.This instantiates the sensing and communication tasks evaluated in the paper.Source: A. System Model · p. 3

1) Communication Model:

  • The communication model represents the selected link using propagation paths whose coefficients and departure directions depend on scene geometry and object state.Vehicle motion and surrounding objects can therefore alter both propagation paths and the preferred transmit beam.Source: 1) Communication Model · p. 3
  • The RSU uses a unit-norm transmit beam from a codebook, and the vehicle receives the beamformed signal with receiver noise.The transmitted symbol has unit expected power; the model includes RSU transmit power and complex Gaussian noise.Source: 1) Communication Model · p. 3
  • The available sensing modalities are those present in the target deployment and represented by its simulated counterpart.Each modality has its own observation process and sensor pose and acquisition characteristics.Source: 2) Multi-modal Sensing Model · p. 3

2) Multi-modal Sensing Model:

The task formulation links deployment-specific sensing observations to task outputs and supervision. Detection and beam prediction use distinct output structures and learning objectives.

2) Multi-modal Sensing Model:

  • A modality-specific observation process, sensor pose, acquisition characteristics, and measurement imperfections shape sensing records, while shared time indices and spatial calibration relate samples to common physical states.Changes in scene geometry or object state can affect several sensing observations.Source: 2) Multi-modal Sensing Model · p. 3

B. Problem Formulation

1) Task-Specific Multi-modal Inference:

  • The task model maps selected observations and task/deployment context to task-specific outputs; detection and beam prediction use separate output structures and learning objectives.The context vector contains task and deployment attributes available to the model.Source: 1) Task-Specific Multi-modal Inference · p. 3
  • Vehicle detection predicts object locations, categories, and confidence scores against annotated vehicle boxes and categories, using classification and localization loss terms.The reference output comprises vehicles satisfying the annotation policy in the full image.Source: 1) Task-Specific Multi-modal Inference · p. 3
  • Beam prediction uses the configured codebook to define beam supervision, predicts a probability over beams, and selects sensing observations at inference.The task learns the relationship between observable scene information and the preferred RSU transmit beam under that codebook.Source: 1) Task-Specific Multi-modal Inference · pp. 3–4

2) Sim-to-Real Training and Adaptation:

AIMS instantiates a deployment-specific sim-to-real configuration from the request and experiment state. Training uses synthetic data, followed by zero-shot inference or restricted real-domain adaptation, and evaluates risk on the fixed target deployment.

2) Sim-to-Real Training and Adaptation:

  • The configuration ητ specifies synthetic construction conditions, selected task modalities, and parameter groups available for real-domain adaptation.AIMS instantiates it from the request, shared experiment state, and structured domain knowledge.Source: 2) Sim-to-Real Training and Adaptation · p. 4
  • For zero-shot inference, the synthetic-domain model is used directly; with real support, adaptation updates only the designated parameter groups while other parameters retain synthetic-domain values.The considered settings are zero-shot inference and real-domain adaptation.Source: 2) Sim-to-Real Training and Adaptation · p. 4
  • The target-domain risk is evaluated over task records from the fixed target deployment, using selected observations, task context, and reference outputs.Reference outputs are annotated vehicle sets for detection or measured beam indices for beam prediction.Source: 2) Sim-to-Real Training and Adaptation · p. 4

3) Deployment-Constrained Learning Objective:

The deployment-level objective evaluates target-task prediction under physical, task, and execution constraints. A knowledge-guided, two-agent process plans and executes the configuration, then uses validation evidence to revise affected decisions.

3) Deployment-Constrained Learning Objective:

  • The objective targets predictive performance after synthetic training and permitted adaptation, subject to physical consistency, task compatibility, and execution feasibility.Held-out target data provide the final performance evaluation.Source: 3) Deployment-Constrained Learning… · pp. 4–5
  • Physical consistency aligns task products to shared physical references; task compatibility requires appropriate modalities and supervision; execution feasibility checks valid capability dependencies.Beam prediction pairs sensing with wireless supervision, while detection pairs images with annotations.Source: 3) Deployment-Constrained Learning… · pp. 4–5

III. AGENTIC ORCHESTRATION FOR SIM-TO-REAL MULTI-MODAL ISAC

  • Structured domain knowledge grounds requests and guides planning, while execution and validation evidence update the shared state for local revision of affected decisions.The planner instantiates an experiment specification and execution plan from the grounded state.Source: §III · p. 5, A. Agentic Formulation and Two-Agent… · p. 5

A. Agentic Formulation and Two-Agent Roles

  • The scene construction agent coordinates physical conditions and shared-reference data generation, while the scene understanding agent selects task modalities and configures learning and transfer.The agents exchange requirements, products, configuration references, and validation evidence through the shared state.Source: A. Agentic Formulation and Two-Agent… · p. 5

B. Perception

Perception structures deployment requests and existing experiment state; planning selects required capabilities and reusable products. Action executes the plan, and reflection uses validation evidence to revise affected decisions.

B. Perception

  • Perception grounds the requested task, deployment conditions, and real-data budget in a structured profile, while recording available products, configuration references, and real-support status.Task knowledge, capability conditions, and product dependencies guide how the profile is interpreted.Source: B. Perception · p. 5

C. Planning

  • Planning selects the endpoint, capabilities, and compatible products, then configures modalities and zero-shot inference or adaptation according to task needs and available real support.Dependency knowledge can preserve compatible channel realizations after a codebook change but invalidates sensing and wireless products after a trajectory change.Source: C. Planning · pp. 5–6

D. Action

  • Action executes configured physical and numerical capabilities, returning outputs and validation evidence to the shared state for downstream operations.Learning uses synthetic pretraining and task-protocol-restricted real-domain adaptation.Source: D. Action · p. 6

E. Reflection

  • Reflection identifies conflicts or invalid dependencies and revises only affected downstream products, retaining products whose defining conditions remain compatible.Changes in sensing availability revise inputs; changes in real support revise the transfer operation under the task protocol.Source: E. Reflection · p. 6

IV. SCENE CONSTRUCTION AND UNDERSTANDING AGENTS

The framework turns deployment requirements into shared task conditions and task-aligned synthetic records. Scene construction grounds a static environment, activates synchronized dynamics and sensing, and supports subsequent task learning and transfer.

IV. SCENE CONSTRUCTION AND UNDERSTANDING AGENTS

  • Domain capability interfaces connect agentic decisions to physical and learning operations through specified inputs, outputs, and validation conditions.The section organizes task conditions before scene construction and synchronized data generation.Source: §IV · p. 6

A. Task and Deployment Instantiation

  • Task conditions specify required observations, annotations or wireless supervision, deployment settings, sample coverage, real support, and learning endpoints shared by both agents.Beam prediction may proceed to zero-shot inference or few-shot adaptation; vehicle detection can end at task learning.Source: A. Task and Deployment Instantiation · p. 6

B. Scene Construction Agent

  • Scene construction grounds a static environment, activates dynamic objects and synchronized sensing, projects shared states into wireless quantities when needed, and aligns products with supervision.The three stages are real-to-digital grounding, digital-to-kinetic activation, and kinetic-to-wireless projection.Source: B. Scene Construction Agent · p. 6

1) Real-to-Digital Grounding:

  • Real-to-digital grounding reuses a validated scene when geographic coverage, infrastructure, and semantic content match the deployment; otherwise, it initiates new grounding.Geometric and semantic checks correct or reject conflicts and record unresolved source ambiguities as validation gaps.Source: 1) Real-to-Digital Grounding · p. 7

2) Digital-to-Kinetic Activation:

  • Digital-to-kinetic activation reuses compatible trajectories and sensing records or regenerates affected products when mobility, environment, or sensing conditions change.Observations and annotations share actor and sample references; the validated dynamic records also support wireless projection.Source: 2) Digital-to-Kinetic Activation · pp. 7–8

3) Kinetic-to-Wireless Projection:

Wireless projection ties task-required radio products to validated physical samples. The scene understanding agent configures task-specific modalities and mixture-of-experts learning, then uses available real support to select inference or adaptation.

3) Kinetic-to-Wireless Projection:

  • Kinetic-to-wireless projection reuses compatible results or maps validated samples to task-required radio products under the deployment’s scene and radio configuration.For beam prediction, a configured codebook maps each channel response to beam-domain powers and a beam label.Source: 3) Kinetic-to-Wireless Projection · p. 8

C. Scene Understanding Agent

  • The scene understanding agent organizes validated records, admissible sensing inputs, learning configuration, and transfer strategy according to the target task and available real support.It first performs task-adaptive learning on synthetic records, then selects direct use or target-domain adaptation.Source: C. Scene Understanding Agent · p. 8

1) Task-Adaptive Multi-modal Learning:

  • The task-adaptive MoE learns contributions from selected deployment-supported modalities, fusing their representations for the task-specific head.Vehicle detection preserves spatial structure for localization, while beam prediction uses fused link-level features to predict a beam distribution.Source: 1) Task-Adaptive Multi-modal Learning · p. 8

2) Deployment-Specific Sim-to-Real Transfer:

AIMS selects zero-shot inference or real-domain adaptation according to the available real support, with the adaptation scope set by the task protocol. The evaluation uses DeepSense 6G scenarios for beam prediction and vehicle detection.

2) Deployment-Specific Sim-to-Real Transfer:

  • With no real support, the synthetic-domain model is used directly; with real support, adaptation uses the available samples and task-defined parameter groups.The adaptation scope can include modality-specific representations, the fusion module, and the task head.Source: 2) Deployment-Specific Sim-to-Real… · p. 8
  • A change in real-data budget can update the transfer operation without rebuilding a validated synthetic domain when the task and modality set remain unchanged.This preserves separation between physical data construction and deployment-specific model adaptation.Source: 2) Deployment-Specific Sim-to-Real… · pp. 8–9

V. EXPERIMENTAL RESULTS

  • The evaluation covers vehicle detection and beam prediction on the real-world DeepSense 6G multi-modal ISAC dataset.It also evaluates agentic orchestration performance.Source: §V · p. 9

A. Experimental Setups

  • Synthetic environments are reconstructed for Scenarios 3, 4, and 9; Scenarios 3 and 4 support beam prediction, while Scenarios 3 and 9 support vehicle detection.The reconstructions generate synthetic sensing and wireless data for sim-to-real learning.Source: A. Experimental Setups · p. 9

1) Sensing Settings:

The experiments pair sensing and wireless records from reconstructed deployment scenes, then use task-specific multimodal models and an agentic orchestration benchmark. The settings include distinct simulation and fusion baselines.

1) Sensing Settings:

  • RGB and GPS support the quantitative tasks in all three scenarios, with 2D LiDAR additionally available in Scenario 9.The synthetic environments use map data and CARLA to generate vehicle motion and sensor observations under deployment and illumination conditions.Source: 1) Sensing Settings · p. 9

2) Wireless Settings:

  • The wireless setup uses a 60 GHz downlink, a 16-element RSU array, and a 64-beam codebook, with synthetic beam labels aligned to the target deployment’s beam identities.Ray tracing generates channels from reconstructed geometry and dynamic states; the real records contain measured per-beam received powers.Source: 2) Wireless Settings · p. 9

3) Learning Settings:

  • Task-specific MoE models use modality-specific encoders and context-conditioned softmax gating for beam prediction and vehicle detection.Models train on synthetic data; zero-shot evaluation applies them directly, while few-shot adaptation updates task-selected parameters using labeled real samples. Baselines include scenario-agnostic, map-derived, and codebook-agnostic simulation, feature concatenation, a multimodal transformer, and full-real-data training.Source: 3) Learning Settings · p. 9

4) Agent Settings:

  • The orchestration benchmark contains 60 canonical cases expressed as 140 requests, including dependency-and-reuse, recovery, and challenging constraint cases.A direct-LLM baseline has no tool calls, validation evidence, or structured domain knowledge; tool-only AIMS retains validation and feedback but omits structured domain knowledge.Source: 4) Agent Settings · pp. 9–10

B. Performance Comparison for Beam Prediction

AIMS outperforms the considered simulation and fusion baselines for beam prediction and vehicle detection in the evaluated scenarios. Few-shot adaptation improves both tasks, while the full-real-data reference remains higher for the reported comparisons.

B. Performance Comparison for Beam Prediction

  • Both proposed beam-prediction configurations outperform all five baselines in Scenarios 3 and 4; zero-shot Top-3 gains over the stronger fusion baseline are approximately 7 and 16 percentage points, respectively.The largest gaps are against scenario-agnostic and codebook-agnostic simulation; the map-derived simulation baseline also remains below AIMS.Source: B. Performance Comparison for Beam… · p. 10, Figure 5 · p. 11
  • Using 20 real training samples improves all four plotted beam Top-k results, although the full-real-data training reference remains higher.The gap is smaller for Top-5 than Top-3.Source: B. Performance Comparison for Beam… · p. 11, Figure 5 · p. 11

C. Performance Comparison for Object Detection

  • At the final epoch, zero-shot vehicle detection exceeds scenario-agnostic simulation and both fusion baselines on AP and AP50 in both scenarios.Its AP advantage over the stronger fusion baseline ranges from approximately 6 to 13 percentage points across the two scenarios; the map-derived baseline also remains below the proposed models.Source: C. Performance Comparison for Object… · p. 11
  • For vehicle detection, 20-sample few-shot learning improves AP and AP50, but a gap to full-real-data training remains, particularly in AP.The few-shot gains persist through later training epochs and extend to the more demanding localization criteria included in AP.Source: C. Performance Comparison for Object… · p. 11

D. Performance Comparison for Agentic Orchestration

On the 70-request normal and challenging tiers, AIMS achieves higher final-plan success than Tool-only AIMS and Direct LLM. The conclusion describes AIMS as coordinating deployment-specific sim-to-real configuration and learning, with validation evidence supporting revisions as conditions change.

D. Performance Comparison for Agentic Orchestration

  • Plan-TSR covers task interpretation, target endpoint, capability selection, artifact handling, and recovery, while CSR measures structural plan validity.Artifact-dependency F1 and Capability-F1 assess artifact handling and agreement with required capabilities, respectively.Source: D. Performance Comparison for Agentic… · pp. 11–12
  • Structured knowledge raises Dependency-F1 from 87.0% to 100% and Capability-F1 from 77.4% to 100%; AIMS also has 78.6% first-try success versus 52.9%.These normal-tier values compare AIMS with Tool-only AIMS.Source: D. Performance Comparison for Agentic… · p. 12
  • On challenging requests, AIMS achieves 92.4% Dependency-F1 and 90.3% Capability-F1, versus 72.5% and 65.3% for Tool-only AIMS.AIMS first-try success falls from 78.6% on the normal tier to 68.6%; average attempts rise from 1.229 to 1.257.Source: D. Performance Comparison for Agentic… · p. 12

VI. CONCLUSION

  • AIMS grounds requests in shared experiment state, coordinates scene construction with task learning, and uses validation evidence for dependency-aware revision as conditions change.The conclusion reports DeepSense 6G experiments for vehicle detection and beam prediction, alongside an orchestration benchmark of task interpretation, dependency handling, and replanning.Source: §VI · pp. 12–13

D. Performance Comparison for Agentic Orchestration

Loading 2609.39964v1…