PaperlayerView exact revision on arXiv

Source-linked AI summary

Not All Uncertainty Matters: Simulation-in-the-Loop Fast-Slow Reasoning for Decision-Critical Autonomous Driving System

Jiayi Chen, Shuai Wang, Guangxu Zhu, Derrick Wing Kwan Ng, Chengzhong Xu, Kaibin Huang

arXiv:2610.09520v1cs.ROcs.AI

TL;DR

Cloud vision-language models offer open-world reasoning, but their cost and latency make continuous use impractical, leaving when they should influence time-critical driving decisions unresolved. SIGMA evaluates plausible semantic and geometric scene alternatives through planning and uses expected planning gain to guide cloud collaboration; in CARLA, it improves outcomes over fixed-period collaboration while reducing unnecessary interactions.

  • Problem

    Cloud vision-language models offer open-world reasoning, but their cost and latency make continuous use impractical, leaving when cloud reasoning should affect time-critical driving decisions unresolved.

    Source: §I · p. 1

  • Method

    SIGMA propagates plausible scenes under semantic and geometric uncertainty through planning, then uses expected planning gain to guide cloud invocation, guidance integration, and request prioritization.

    Source: §I · p. 2

  • Results

    50% reduction in unnecessary cloud interactions, more than 6% improvement in navigation success rate, and up to 26.2% reduction in finish time in dynamic scenarios versus fixed-period collaboration.

    Source: §I · p. 3

  • Takeaways & Limitations

    EPG provides a shared criterion for cloud invocation, guidance integration, and resource allocation, targeting assistance to cases that are decision-relevant and timely.

    Source: §VIII · p. 16

Abstract

from arXiv · show

Large vision-language models (VLMs) provide powerful open-world perception and reasoning for autonomous driving, but their high computational cost and inference latency make continuous cloud-side use impractical. This motivates fast--slow collaboration, where efficient onboard modules handle real-time perception and control while cloud models provide high-level reasoning only when needed. The key challenge is deciding when cloud reasoning should influence time-critical driving decisions. Existing methods often rely on perception uncertainty, heuristic triggers, or resource-driven policies, without assessing whether resolving an uncertainty will improve planning. We propose \textbf{SIGMA}, a simulation-in-the-loop framework for task-oriented fast--slow collaboration. SIGMA embeds the planner into uncertainty assessment and evaluates how plausible scene realizations under semantic and geometric uncertainty affect feasible trajectories and planning cost. Based on these outcomes, it estimates the expected reduction in planning cost from resolving uncertainty. We further introduce expected planning gain (EPG), a decision-level metric for cloud invocation, cloud-guidance integration, and request prioritization under deadline and resource constraints. Experiments in CARLA show that SIGMA reduces unnecessary cloud interactions while improving planning, efficiency, and navigation success in static and dynamic obstacle scenarios. Compared with fixed-period collaboration, SIGMA reduces unnecessary cloud interactions by 50\%, improves navigation success by more than 6\%, and cuts finish time by up to 26.2\% in dynamic scenarios.

I. INTRODUCTION

Because cloud vision-language reasoning is costly and slow, the introduction asks when it should influence real-time driving. It argues that uncertainty matters insofar as plausible scene alternatives change downstream planning, rather than because uncertainty is high in perception alone.

I. INTRODUCTION

  • SIGMA models semantic and geometric ambiguity, propagates plausible scenes through MPC, and uses expected planning gain (EPG) to govern cloud invocation, guidance integration, and request prioritization.EPG estimates expected normalized MPC-cost reduction over plausible scene realizations using budget-aware Monte Carlo rollouts.Source: §I · p. 2
  • Against fixed-period collaboration, SIGMA reduces unnecessary cloud interactions by 50%, improves navigation success rate by more than 6%, and cuts finish time by up to 26.2% in dynamic scenarios.The evaluation covers static and dynamic unknown-obstacle scenarios.Source: §I · p. 3

A. Uncertainty Estimation in Autonomous Driving Perception

  • Prior perception-uncertainty methods improve robustness and calibration, but do not explicitly test whether resolving uncertainty changes downstream planning.The introduction notes that uncertainty about distant or irrelevant objects may not affect the planned trajectory.Source: §I · p. 1, A. Uncertainty Estimation in… · p. 3

B. Motion Planning Under Perception Uncertainty

C. Fast–Slow Collaborative Driving

SIGMA frames fast–slow collaboration around whether resolving uncertainty improves downstream planning, rather than relying on proxy objectives or invoking cloud reasoning by default. Its system model uses expected planning gain to guide cloud invocation and integration while keeping real-time control on the edge.

C. Fast–Slow Collaborative Driving

  • Existing collaborative systems often use heuristic triggers or capability-driven designs, while SIGMA governs cloud invocation and guidance integration by expected planning gain.The criterion evaluates whether resolving uncertainty measurably improves downstream planning, rather than optimizing proxy objectives such as perception quality, resource usage, latency, or service cost.Source: C. Fast–Slow Collaborative Driving · p. 4, D. Positioning of Our Work · p. 4

III. SYSTEM MODEL AND PROBLEM FORMULATION

  • The edge handles real-time perception, planning, and control; the cloud provides slower reasoning and returns refined scene information or high-level trajectory guidance.The cloud output is incorporated into the onboard MPC planner rather than used as a low-level control command; real-time control authority and safety enforcement remain on the edge.Source: §III · p. 4
  • Uncertain or ambiguous observations can lead the onboard planner to choose conservative, unsafe, or detouring suboptimal trajectories.Cloud invocation also incurs communication delay and computation cost, and a response arriving too late may not affect the current decision.Source: §III · p. 4
  • Expected Planning Gain (EPG) is the expected normalized reduction in MPC planning cost from resolving current planning-relevant uncertainty.The oracle benchmark is not an online policy because the cloud-assisted planning cost is unavailable before cloud execution.Source: §III · p. 5
  • Cloud invocation requires estimated planning benefit to outweigh invocation and delay costs, and the response must satisfy the decision deadline.The benefit-side condition is that the benefit factor exceeds its threshold; the complete online rule also checks deadline feasibility.Source: §III · p. 5

IV. DECISION-ORIENTED UNCERTAINTY

SIGMA treats uncertainty as decision-oriented: its practical importance depends on whether resolving it changes downstream planning. It separates semantic ambiguity from geometric uncertainty because they play different roles in the planning pipeline.

IV. DECISION-ORIENTED UNCERTAINTY

  • The edge-side system architecture combines uncertainty-aware perception with onboard planning.Source: Figure 2 · p. 5, §IV · p. 5
  • Uncertainty is task-dependent: its importance is defined by its impact on planning, not merely as a generic property of perception.Some uncertainties affect trajectory optimization directly, while others indicate possible perception failures with limited influence on the optimal motion.Source: §IV · p. 5
  • Semantic uncertainty captures ambiguity in object identity and screens for additional information, while geometric uncertainty captures localization and extent ambiguity that affects collision risk and trajectory optimization.The framework models semantic uncertainty as a confidence-based screening signal and geometric uncertainty as a probabilistic 3D object-detection output.Source: §IV · p. 5

A. Semantic Uncertainty for Unknown Object Detection

SIGMA uses low detector confidence to screen for semantic ambiguity and represents geometric uncertainty probabilistically. The two components serve distinct roles: semantic uncertainty flags candidate cases for additional reasoning, while geometric uncertainty is propagated into planning.

A. Semantic Uncertainty for Unknown Object Detection

  • Semantic uncertainty represents ambiguity in identifying rare, unseen, or out-of-distribution objects and signals that local perception may be unreliable.Source: A. Semantic Uncertainty for Unknown… · p. 5
  • Low-confidence detections are marked semantically uncertain using a confidence threshold selected with object detection confidence thresholding.The thresholding objective balances recall of out-of-distribution and in-distribution detections while suppressing excessive false positives.Source: A. Semantic Uncertainty for Unknown… · pp. 5–6

B. Geometric Uncertainty for Bbox Estimation

  • Geometric uncertainty is represented as a Gaussian distribution over 3D bounding-box parameters and propagated into planning by inflating obstacles according to planar position uncertainty.Larger uncertainty yields a larger safety envelope; the resulting probabilistic obstacle representation supports trajectory optimization and task-oriented value estimation.Source: B. Geometric Uncertainty for Bbox… · p. 6

V. EXPECTED PLANNING GAIN UNDER PERCEPTION UNCERTAINTY

SIGMA estimates the planning value of resolving perception uncertainty by evaluating plausible scene realizations in the planner. Its Expected Planning Gain (EPG) is an expected normalized MPC cost reduction, estimated through budgeted Monte Carlo simulation.

V. EXPECTED PLANNING GAIN UNDER PERCEPTION UNCERTAINTY

  • EPG estimates the expected normalized reduction in MPC planning cost over plausible scene-uncertainty realizations induced by the current observation.It provides a task-specific approximation of additional information's value without maintaining a full belief-space planner.Source: §V · p. 7
  • The framework converts geometric uncertainty into probabilistic obstacles, incorporates them into dynamics-aware MPC, and samples plausible scenes to evaluate planning costs.Source: §V · p. 7
  • A budgeted Monte Carlo procedure turns the sampled planning costs into a scalar EPG estimate of uncertainty's expected impact on decisions.Source: §V · p. 7

A. Safety-Constrained Probabilistic Obstacle Modeling

SIGMA converts geometric uncertainty into safety-constrained obstacle representations and evaluates how scene configurations affect MPC planning. Comparing planning outcomes across configurations identifies uncertainty that can change the planner’s decision.

A. Safety-Constrained Probabilistic Obstacle Modeling

  • Geometric uncertainty is modeled as random translations of detected obstacle regions, linking perception uncertainty to planning constraints.The resulting safety-constrained inflated obstacle sets reshape the feasible trajectory set.Source: A. Safety-Constrained Probabilistic… · p. 7
  • A chance-constrained collision-free requirement is approximated by a deterministic inflated-obstacle constraint using a safety-level quantile of worst-case planar perturbation.Direct probabilistic collision checking over random obstacle geometry is computationally challenging in real time.Source: A. Safety-Constrained Probabilistic… · p. 7

B. Planning-Gain Evaluation via MPC

  • For each deterministic scene configuration, MPC minimizes tracking error, control effort, and control smoothness subject to dynamics, state and control limits, and collision avoidance.The formulation covers static and dynamic obstacles.Source: B. Planning-Gain Evaluation via MPC · p. 8
  • Normalized planning gain compares MPC costs across two scene configurations; negligible gain indicates uncertainty resolution does not materially change the planner.Large planning gain indicates decision-relevant uncertainty.Source: B. Planning-Gain Evaluation via MPC · p. 8

C. Budget-Aware Monte Carlo Estimation

SIGMA estimates expected planning gain by rolling plausible uncertainty-resolved scenes through the planner. The estimate guides selective cloud invocation.

C. Budget-Aware Monte Carlo Estimation

  • SIGMA evaluates plausible uncertainty realizations through the MPC planner and measures their impact on planning cost rather than relying on perception-level uncertainty alone.The module estimates whether resolving uncertainty would improve downstream planning.Source: C. Budget-Aware Monte Carlo Estimation · p. 8
  • Because expected planning gain is generally intractable to compute directly, SIGMA estimates it with importance-sampling-based Monte Carlo rollouts.The method assumes conditional independence among object-level perturbations.Source: C. Budget-Aware Monte Carlo Estimation · p. 8
  • The estimated planning gain filters out uncertainties with negligible expected benefit while preserving decision-relevant cases for cloud-assisted reasoning..Source: C. Budget-Aware Monte Carlo Estimation · pp. 8–9

VI. FAST–SLOW COLLABORATIVE DECISION AND PLANNING-GAIN-AWARE CLOUD RESOURCE ALLOCATION

EPG guides cloud activation, decision refinement, and resource allocation according to estimated planning impact. When invoked, cloud perception and semantic reasoning generate maneuver guidance for the onboard MPC, which retains real-time control authority.

VI. FAST–SLOW COLLABORATIVE DECISION AND PLANNING-GAIN-AWARE CLOUD RESOURCE ALLOCATION

  • EPG is the decision-level criterion for cloud activation, decision refinement, and allocating limited cloud computation across requests.This makes collaboration depend on estimated planning impact rather than perception uncertainty or heuristic triggers.Source: §VI · p. 9
  • Cloud assistance is activated when estimated EPG is sufficiently large; otherwise, the scenario stays onboard, avoiding unnecessary latency and computation.Cloud guidance is incorporated into local planning, and competing requests are prioritized using the same planning-gain principle.Source: §VI · p. 9

A. LVM-Assisted Semantic Decision Generation

  • Cloud perception refines object detections into a structured semantic scene representation, with segmentation-based verification refining geometry and extracting drivable regions and lane geometry.The representation integrates object categories, spatial relationships, lane structure, and drivable-region information.Source: A. LVM-Assisted Semantic Decision… · p. 9
  • The large vision–language model reasons over obstacles, lane structures, and drivable regions to generate maneuver guidance that becomes a reference trajectory for local MPC.The cloud supplies semantically informed, long-horizon guidance while onboard MPC retains real-time control authority and enforces dynamic feasibility and safety.Source: A. LVM-Assisted Semantic Decision… · p. 9

B. EPG- and Deadline-Aware Cloud-Guided MPC

Cloud requests are screened and prioritized by expected planning benefit and whether guidance can arrive before a decision-critical obstacle. Valid returned guidance updates the MPC reference; onboard MPC continues to enforce safety and feasibility.

B. EPG- and Deadline-Aware Cloud-Guided MPC

  • The lightweight EPG- and deadline-aware allocation rule is an initial mechanism, not a full cloud-scheduling optimization.The estimated planning gain represents normalized MPC planning-cost reduction from resolving uncertainty.Source: B. EPG- and Deadline-Aware… · p. 10
  • Cloud assistance is useful only when its predicted response arrives within the remaining time before the first decision-critical obstacle.That slack is measured against predicted response time, including communication and cloud-side inference.Source: B. EPG- and Deadline-Aware… · p. 10
  • Requests must meet a benefit-factor threshold and a deadline-feasibility condition to be eligible for cloud service.When computation is limited, eligible requests are ranked by a benefit-factor-efficiency score.Source: B. EPG- and Deadline-Aware… · p. 10
  • Only valid cloud guidance returned before the deadline updates the MPC reference path; otherwise, the planner falls back to the local reference.The cloud updates high-level guidance rather than directly controlling the vehicle, while onboard MPC enforces dynamic feasibility and collision safety in real time.Source: B. EPG- and Deadline-Aware… · p. 10

VII. EXPERIMENTS

The experiments evaluate perception and uncertainty modeling, including semantic screening for unknown objects. The semantic signal identifies candidates for cloud assistance, while planning determines whether that assistance is useful.

VII. EXPERIMENTS

  • The evaluation tests semantic screening and geometric localization uncertainty, using CARLA Town06 data and system experiments implemented in Python and ROS.The local detector runs at up to 100 FPS, the MPC outputs controls at 20 Hz, and the simulated sensors update at 10 Hz.Source: §VII · p. 10, A. Experiment 1: Perception and… · p. 11

1) Semantic Uncertainty for Unknown-Object Awareness:

  • Validation and test sets select confidence thresholds of 0.8 and 0.75, respectively, balancing retained unknown-object candidates against false alarms and missed ambiguities.The threshold is a screening operating point, not a claim that the local model fully recognizes unknown objects.Source: 1) Semantic Uncertainty for… · p. 11, Figure 8 · p. 11
  • The optimized local model recalls known/unknown objects at 0.83/0.71 on validation and 0.89/0.61 on test; the cloud model recalls 0.89 of known validation objects.The validation and test unknown classes do not overlap, and test unknown classes were unseen during local training and threshold selection.Source: 1) Semantic Uncertainty for… · p. 11
  • Low-confidence detections flag potentially unknown objects for cloud recognition and lane/object segmentation, while downstream planning-gain assessment determines whether cloud assistance is useful.The cloud output recovers object–lane topology for downstream decisions.Source: 1) Semantic Uncertainty for… · p. 12

2) Geometric Uncertainty for 3D Object Localization:

The experiments examine whether geometric uncertainty tracks localization difficulty and whether simulation-in-the-loop assessment helps collaboration. SIGMA reduces cloud interactions while improving static-scenario planning outcomes, and the rollouts distinguish decision-relevant from inconsequential uncertainty.

2) Geometric Uncertainty for 3D Object Localization:

  • As predicted uncertainty rises, average localization error increases from 0.129 to 0.488, with Pearson correlation 0.405 and Spearman correlation 0.536.The relationship is statistical rather than deterministic for individual objects; distance, viewpoint, occlusion, size, and LiDAR sparsity also affect localization error.Source: 2) Geometric Uncertainty for 3D Object… · p. 12
  • The uncertainty-aware detector maintains competitive 3D detection performance, and its learned geometric uncertainty is used as a representation for obstacle inflation and planning.Representative challenging cases include distant objects, sparse observations, and ambiguous object extents.Source: 2) Geometric Uncertainty for 3D Object… · pp. 12–13

1) System-Level Performance Comparison:

2) Simulation-in-the-Loop Forward Simulation Analysis:

  • Monte Carlo scene samples are propagated through MPC rollouts to assess how obstacle uncertainty changes feasible trajectories and planning outcomes.Different samples can produce nearly identical trajectories or divergent paths near obstacle clusters.Source: 2) Simulation-in-the-Loop Forward… · p. 13
  • Only uncertainty that leads to divergent planning outcomes is treated as decision-relevant, guiding cloud interaction toward situations with expected planning gain.The analysis presents this planning-space assessment as the mechanism behind improved navigation and reduced cloud use versus fixed intervals.Source: 2) Simulation-in-the-Loop Forward… · p. 14

C. Experiment 3: More Quantitative Evaluations

Across scenarios with unknown obstacles, SIGMA’s performance advantage is strongest when collaboration timing is decision-critical. In the dynamic-obstacle case, it reduces finish time relative to PCS; the conclusion identifies interaction-aware uncertainty as future work.

C. Experiment 3: More Quantitative Evaluations

  • SIGMA maintains success rates close to 100% across tested scenarios and improves success rate by over 6% versus the second-best scheme.With two unknown objects, PCS has safety issues and LOS has the lowest success rates; SIGMA also achieves the shortest trajectory length in all tested settings and the shortest finish time at U = 2.Source: C. Experiment 3: More Quantitative… · p. 14, Figure 16 · p. 15

D. Experiment 4: Scenarios with Dynamic Obstacles

  • Compared with PCS, SIGMA reduces finish time by more than 26.2% in the dynamic-obstacle experiment.SIGMA accepts a slightly longer maneuver when it supports a more time-efficient decision.Source: Table VI · p. 15, D. Experiment 4: Scenarios with… · p. 15
  • In the dynamic case, EPG identifies a car-following versus overtaking ambiguity, and cloud-generated overtaking waypoints enable the ego vehicle to pass the cybertruck.LOS fails near the turning point, so the experiment compares PCS and SIGMA.Source: D. Experiment 4: Scenarios with… · p. 15

VIII. CONCLUSION

  • The conclusion presents EPG as a unified criterion for cloud invocation, guidance integration, and resource allocation, with CARLA experiments showing reduced cloud use and improved performance.Future work targets interaction-aware uncertainty for multi-modal agent intentions and trajectory prediction ambiguity.Source: §VIII · p. 16
Loading 2610.09520v1…