PaperlayerView exact revision on arXiv

Source-linked AI summary

From Local Whole-Body VLA Behaviors to Scene-Scale Aerial Manipulation

Weixiang Guo, Rui Jin, Haotian Jin, Xinhang Xu, Ruiyang Liu, Haoran Zhao, Yi Wang, Weiqi Gai, Kun Cao, Lihua Xie

arXiv:2609.39670v1cs.RO

TL;DR

Scene-scale aerial manipulation with local whole-body VLA skills is challenged by costly demonstrations, asynchronous action-state mismatch, and cross-site composition. The framework synthesizes training supervision, aligns execution with measured progress, and composes behaviors through Scene Graph grounding and topology-guided transfer; its simulated system achieved 9/15 mission success.

  • Problem

    Scene-scale aerial manipulation requires costly, risky whole-body demonstrations, alignment of asynchronous action chunks with actual execution progress, and feasible cross-site behavior composition.

    Source: §I · p. 2

  • Method

    The framework synthesizes feasible whole-body VLA supervision, aligns returned chunks with measured progress, and composes local skills through Scene Graph grounding and topology-guided transfer.

    Source: §I · p. 2

  • Results

    The system achieves interface success in 14/15 episodes and mission success in 9/15; MPAR reduces episode-level P50 phase error by an average of 0.212 s across 60 paired C2–C3 episodes.

    Source: C. Measured-Progress Whole-Body… · p. 7, D. Causal Ablation of the Scene-Graph… · p. 8

  • Takeaways & Limitations

    The work unifies synthetic whole-body VLA training, measured-progress-aligned realization, and scene-graph-guided composition, with simulation and physical validation.

    Source: §VIII · p. 8

    The system relies on a preconstructed Scene Graph registered to a metric map and remote VLA inference, while mission-level performance remains limited by individual local skills.

    Source: §VIII · p. 8

Abstract

from arXiv · show

Vision-language-action (VLA) models enable task-conditioned interaction, but extending them to scene-scale aerial manipulation remains challenging due to costly whole-body demonstrations, latency-induced action-state misalignment, and cross-site behavior composition. We present a unified framework for synthetic policy training and scene-scale execution on articulated uncrewed aerial manipulators (UAMs). A scene-reconfigurable pipeline synthesizes task-conditioned, kinodynamically feasible trajectories and synchronized multiview observations for VLA training without physical-platform demonstrations. Measured-progress-aligned realization (MPAR) aligns asynchronously returned action chunks with measured execution progress and realizes them as continuous, dynamically feasible trajectories. A relational Scene Graph grounds language goals to object instances and feasible interaction regions, while topology-guided transfer connects local behaviors across sites. Local VLA skills achieve 39/60 successes (65.0%) in simulation under oracle target and feasible-handoff conditions. Under 500-ms added latency, with and without a transient command-update stall, MPAR reduces median takeover phase error by 0.212 s over nominal-time alignment. The complete system completes 21/50 simulated multi-site missions (42.0%) and is further validated on a physical articulated UAM.

I. INTRODUCTION

Extending local whole-body VLA behaviors to scene-scale aerial missions requires addressing costly physical demonstrations, asynchronous action-state mismatch, and feasible cross-site composition. The paper proposes a unified framework for synthetic training, measured-progress-aligned realization, and scene-graph-guided transfer.

I. INTRODUCTION

  • The framework retains learned local coordination while explicitly handling physical realization and cross-site transfer to compose scene-scale aerial missions.The authors evaluate it in simulation and representative physical trials.Source: §I · p. 2
  • The scene-reconfigurable pipeline generates task-conditioned whole-body expert supervision and synchronized multiview observations without demonstrations collected on the target UAM.This addresses costly and risky physical demonstration collection under coupled base–arm–gripper dynamics and actuation limits.Source: §I · p. 2
  • MPAR aligns asynchronous action chunks with measured execution progress and realizes continuous takeovers under whole-body motion limits.A new chunk may arrive when elapsed time no longer reflects how far the UAM has actually progressed.Source: §I · p. 2
  • The scene-graph composition interface resolves relational targets to feasible whole-body handoff states and uses polyhedral topology to guide cross-site transfer.The handoff state must let the local behavior begin, while transfer must be collision-free and kinodynamically feasible.Source: §I · p. 2

A. Vision-Language-Action Models for Aerial Manipulation

  • Prior aerial-manipulation supervision approaches leave a gap in automatically generating task-conditioned, kinodynamically feasible whole-body training data without platform demonstrations.Execution-latency methods and model-based feasible-motion planners also address the two issues largely separately.Source: A. Vision-Language-Action Models for… · p. 2

B. Scene Graphs for Scene-Scale Aerial Manipulation

Scene-scale aerial manipulation requires grounding relational goals in feasible whole-body interaction regions and transferring between sites. The system combines this composition with offline synthesized VLA supervision and online execution using measured state and feedback.

B. Scene Graphs for Scene-Scale Aerial Manipulation

  • Scene representations and planning systems resolve semantic instances or generate feasible motion, but the stated gap is connecting relational grounding to executable interaction-region selection and whole-body-feasible transfer.For articulated UAMs, the interaction region must admit a feasible whole-body state for starting the local skill.Source: B. Scene Graphs for Scene-Scale Aerial… · p. 2

III. SYSTEM OVERVIEW

  • At runtime, the mission layer resolves targets and local instructions in a Scene Graph, topology-guided transfer reaches a feasible handoff region, and the VLA predicts local base–arm–gripper chunks.MPAR aligns returned chunks using measured state; feedback closes local execution and skill outcomes advance the mission.Source: §III · pp. 2–3, Figure 2 · p. 3

IV. SCENE-RECONFIGURABLE WHOLE-BODY VLA SUPERVISION

  • The synthesis pipeline samples UAM states and object configurations, derives object-relative goals, and retains whole-body planner rollouts that satisfy motion, joint, continuity, and collision constraints.It independently places Gaussian-represented assets in a common metric frame and renders synchronized base- and wrist-camera observations along expert states.Source: §IV · p. 3
  • Figure 3 contrasts elapsed-time phase selection with MPAR projection of the estimated handoff state, followed by a C2 bridge to a feasible join and MINCO realization of the remaining path.The resulting suffix is realized as a continuous trajectory.Source: Figure 3 · p. 4

V. MEASURED-PROGRESS-ALIGNED WHOLE-BODY REALIZATION

The VLA chunk is decoded as synchronized whole-body waypoints, whose continuous components form a path. A shared phase preserves coordination while realization determines timing, and gripper commands follow retimed progress.

A. Whole-Body VLA Path Representation

  • Each predicted chunk decodes into synchronized waypoints for base position, yaw, arm configuration, gripper command, and nominal time offset.Piecewise interpolation of the continuous components defines the geometric whole-body path.Source: A. Whole-Body VLA Path Representation · p. 4
  • A shared phase indexes base, yaw, and arm together, preserving their coordination while path progress is treated separately from the nominal execution clock.Physical realization determines the timing of that progress.Source: A. Whole-Body VLA Path Representation · p. 4
  • Discrete gripper commands retain their source-waypoint associations and trigger with retimed geometric progress rather than the original chunk timestamps.This maps the commands onto the managed path after execution starts.Source: A. Whole-Body VLA Path Representation · p. 4

B. Measured-Progress Alignment and Continuous Takeover

MPAR aligns an incoming VLA chunk to estimated physical progress, then joins it to the active command through a continuous, feasible takeover. If no admissible transition is found, the chunk is rejected while the committed trajectory remains active.

B. Measured-Progress Alignment and Continuous Takeover

  • The handoff is scheduled after inference with planning lead time, while the committed trajectory stays active and the handoff remains within its valid horizon.The measured-command state residual is propagated to estimate the handoff state.Source: B. Measured-Progress Alignment and… · p. 4
  • MPAR projects the estimated handoff state onto the new VLA path near nominal phase, accepting the correction only when it improves agreement within recovery bounds.Otherwise, it falls back to nominal-time alignment.Source: B. Measured-Progress Alignment and… · p. 4
  • The selected online phase distinguishes nominal-time alignment from MPAR, and post-hoc phase error measures skipped or repeated path portions in seconds.Episode-level path-length measures sum arc lengths over valid handoffs.Source: B. Measured-Progress Alignment and… · p. 4
  • A feasible join phase and bridge duration connect the active command state to the remaining VLA-path suffix while matching endpoint derivatives.Candidate transitions violating base, yaw, or joint motion limits are rejected; if no admissible takeover exists, the chunk is rejected and replanning proceeds from a fresh observation.Source: B. Measured-Progress Alignment and… · p. 4

C. Progress-Parameterized MINCO Realization

Progress-parameterized MINCO realization turns the bridge and remaining VLA path into a continuous, constrained trajectory. Jointly choosing path phases and interval durations allows execution timing to adapt while preserving the behavior's geometric ordering.

C. Progress-Parameterized MINCO Realization

  • The bridge and remaining VLA-path suffix define the reference for constrained trajectory realization.The trajectory parameterization uses MINCO for aerial-base position and T2 MINCO for yaw and arm joints, with the initial boundary fixed to the active command state.Source: C. Progress-Parameterized MINCO… · pp. 4–5
  • Optimized interval durations and vertex phases determine execution rate, allowing the UAM to speed up or slow down without changing VLA-path ordering.Monotonic phase parameterization keeps the join and subsequent phases ordered along the path.Source: C. Progress-Parameterized MINCO… · p. 5
  • The realization jointly optimizes trajectory waypoints, interval durations, and path phases under smoothness, tracking, progress, and feasibility penalties.Tracking weights cross-track deviation more strongly than along-path error; feasibility covers motion limits and whole-body collisions, followed by a dense feasibility check.Source: C. Progress-Parameterized MINCO… · p. 5

A. Scene-Graph-Conditioned Mission Composition

A Scene Graph and available local VLA skills let an LLM compose language missions into ordered, graph-grounded task plans. A deterministic executor instantiates each plan as a state machine whose transitions follow navigation and VLA outcomes.

A. Scene-Graph-Conditioned Mission Composition

  • The Scene Graph organizes object instances, spatial relations, and free-space connectivity in an Area–Polyhedron hierarchy for language-grounded mission composition.An externally hosted LLM uses graph context and available local skills to resolve references and implicit dependencies, then composes an ordered plan.Source: A. Scene-Graph-Conditioned Mission… · p. 5
  • Each mission step names a graph-described task target instead of supplying metric coordinates or control commands.Source: A. Scene-Graph-Conditioned Mission… · p. 5
  • A deterministic executor instantiates the composed sequence as a state machine and advances transitions according to navigation and VLA outcomes.Source: A. Scene-Graph-Conditioned Mission… · p. 5

B. Instance Grounding and Interaction-Region Construction

Relational queries identify a unique object instance, and skill-conditioned interaction regions yield feasible whole-body handoff states. Topology-guided routing connects those handoff sites and verifies the interaction conditions before local VLA control begins.

B. Instance Grounding and Interaction-Region Construction

  • A deterministic resolver selects an object instance only when its category and specified relations identify a unique match.A skill-conditioned interaction region is intersected with clearance-eroded reachable Polyhedra, with whole-body feasibility used to form candidate handoff states.Source: B. Instance Grounding and… · p. 5

C. Topology-Guided Whole-Body Transfer and VLA Handoff

  • Graph search selects a connected Polyhedron route and feasible terminal handoff; the route defines a corridor between interaction sites.Local VLA control begins only when measured pose and motion satisfy the interaction-region tolerances; otherwise, handoff is rejected.Source: C. Topology-Guided Whole-Body Transfer… · p. 5

VII. EXPERIMENTS

The experiments assess compositional cross-site missions and constrained whole-body realization under asynchronous execution, with simulation and physical-UAM evaluation. MPAR achieves higher aggregate task success than nominal-time alignment and RTC-Direct, while the compositional system completes simulated missions that the monolithic baseline does not.

B. Cross-Site Mission Capability

  • 21/50 simulated cross-site missions succeed with the proposed system, while the monolithic whole-task baseline completes none of its 50 trials.The monolithic baseline has 19 collisions and 31 runtime-planning or action-feasibility terminations; 28 of the proposed system's 29 failures occur during local VLA interaction.Source: Table I · p. 6, B. Cross-Site Mission Capability · p. 6
  • 39/60 local VLA interactions succeed under oracle-target and feasible-handoff conditions, with object acquisition identified as the dominant local-behavior bottleneck.Success comprises 14/30 object-acquisition and 25/30 postacquisition trials.Source: B. Cross-Site Mission Capability · p. 6

C. Measured-Progress Whole-Body Realization

  • 78/120 trials (65.0%) succeed with MPAR, versus 71/120 (59.2%) for nominal-time alignment and 54/120 (45.0%) for RTC-Direct.The task-level effect is not universal: MPAR improves Watering and Storage but does not outperform nominal-time alignment on Grasp Rect.Source: Table II · p. 6, C. Measured-Progress Whole-Body… · p. 7
  • Across 60 paired C2–C3 episodes, MPAR lowers episode-level P50 and P95 phase errors by averages of 0.212 and 0.315 s, respectively.It also reduces skipped and repeated path lengths by 7.44 and 1.89 cm, and candidate-trajectory rejection from 17.42% to 12.81%.Source: Table III · p. 7, C. Measured-Progress Whole-Body… · p. 7
  • Collision-bearing runs decrease from 22/120 with RTC-Direct and 4/120 with nominal-time alignment to 1/120 with MPAR, with no unsafe publications for either constrained variant.Evaluations cover grasping, watering, and storage under four asynchronous-execution conditions, including 500-ms added delay with or without a transient command-update stall.Source: C. Measured-Progress Whole-Body… · pp. 6–7

D. Causal Ablation of the Scene-Graph Physical Interface

The Scene Graph physical interface is evaluated through component-removal ablations across three cross-site mission types. Relational grounding, topology-guided routing, and feasible handoff are tested alongside interface and mission success measures.

D. Causal Ablation of the Scene-Graph Physical Interface

  • The ablation compares removal of relational grounding, topology-guided routing, or feasible-region handoff across three cross-site mission types, holding instructions, scenes, local policies, realization, and task sequences fixed.The variants use five paired episodes per task.Source: D. Causal Ablation of the Scene-Graph… · p. 7, Table IV · p. 8
  • 14/15 episodes achieve interface success and 9/15 achieve mission success with the complete system; without relational grounding, no episode resolves all targets.Removing routing increases navigation collisions, while nominal-point handoffs reduce handoff success.Source: Table IV · p. 8, D. Causal Ablation of the Scene-Graph… · p. 8
  • Interface success, mission success, and navigation collision are episode-level measures; mission success additionally requires every prescribed local event.Local failures are event-level and may include multiple failures per episode; other stage metrics are conditioned on entry.Source: D. Causal Ablation of the Scene-Graph… · p. 8

E. Physical Full-Stack Validation

Physical validation shows local skills and a cross-site mission executing on an articulated UAM. The conclusion notes reliance on a preconstructed, metric-map-registered Scene Graph and remote VLA inference.

E. Physical Full-Stack Validation

  • 2/5 bottle-grasping, 1/5 rectangular-block-grasping, 3/5 watering, and 3/5 storage-placement trials succeed under oracle-target and feasible-handoff conditions.The authors report physical executability without demonstrations collected on the target UAM.Source: E. Physical Full-Stack Validation · p. 8
  • 2/5 physical cross-site Water Plant trials succeed with the full system.Source: E. Physical Full-Stack Validation · p. 8

VIII. CONCLUSION AND FUTURE WORK

  • The system relies on a preconstructed Scene Graph registered to a metric map and remote VLA inference; mission-level performance remains limited by individual local skills.Source: §VIII · p. 8

Related papers

Loading 2609.39670v1…