PaperlayerView exact revision on arXiv

Source-linked AI summary

From Spatial to Actions: Grounding Vision-Language-Action Model in Spatial Foundation Priors

Zhengshen Zhang, Hao Li, Yalun Dai, Zhengbang Zhu, Lei Zhou, Chenchen Liu, Dong Wang, Francis E. H. Tay, Sijin Chen, Ziwei Liu, Yuxiao Liu, Xinghang Li, Pan Zhou

arXiv:2510.17439v3cs.ROcs.AIcs.CVcs.LG

TL;DR

VLAs built on 2D encoders lack reliable 3D spatial understanding, while existing 3D integration approaches face modality-transfer or alignment limitations. FALCON routes rich spatial tokens from foundation models into an action head and optionally incorporates depth or pose. It consistently surpasses baselines across simulation and real-world evaluations, with state-of-the-art performance and robustness on spatially demanding tasks.

  • Problem

    VLAs built on 2D encoders lack reliable 3D spatial understanding, while existing approaches can provide weak geometric cues or disrupt vision-language alignment.

    Source: §1 · p. 2

  • Method

    FALCON uses rich spatial tokens from foundation models in a Spatial-Enhanced Action Head, with an Embodied Spatial Model that can optionally incorporate depth or camera poses.

    Source: §1 · pp. 2–3, §5 · p. 11

  • Results

    Across simulation and real-world tasks, FALCON consistently surpasses existing VLA methods, achieving state-of-the-art performance and robustness on spatially demanding tasks.

    Source: §1 · p. 3, §5 · p. 11

  • Takeaways & Limitations

    FALCON demonstrates robust performance across sensory conditions and on tasks involving spatial prompts, height awareness, and object-scale variation.

    Source: §1 · p. 3, §5 · p. 11

    Depth maps and camera poses can further enhance spatial grounding when available, but are not required.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Existing vision-language-action (VLA) models act in 3D real-world but are typically built on 2D encoders, leaving a spatial reasoning gap that limits generalization and adaptability. Recent 3D integration techniques for VLAs either require specialized sensors and transfer poorly across modalities, or inject weak cues that lack geometry and degrade vision-language alignment. In this work, we introduce FALCON (From Spatial to Action), a novel paradigm that injects rich 3D spatial tokens into the action head. FALCON leverages spatial foundation models to deliver strong geometric priors from RGB alone, and includes an Embodied Spatial Model that can optionally fuse depth, or pose for higher fidelity when available, without retraining or architectural changes. To preserve language reasoning, spatial tokens are consumed by a Spatial-Enhanced Action Head rather than being concatenated into the vision-language backbone. These designs enable FALCON to address limitations in spatial representation, modality transferability, and alignment. In comprehensive evaluations across three simulation benchmarks and eleven real-world tasks, our proposed FALCON achieves state-of-the-art performance, consistently surpasses competitive baselines, and remains robust under clutter, spatial-prompt conditioning, and variations in object scale and height.

1 Introduction

VLAs built on 2D encoders lack reliable 3D spatial understanding, challenging generalization and adaptability. FALCON integrates spatial foundation-model tokens into action decisions, and outperforms existing baselines across simulation and real-world evaluations.

1 Introduction

  • VLAs built on 2D VLMs lack reliable 3D spatial understanding, challenging generalization to novel scenes and adaptability to height or object-scale changes.The introduction identifies geometry, depth, and spatial relations as settings where explicit 3D awareness is needed.Source: §1 · pp. 1–2
  • Methods requiring explicit 3D inputs depend on specialized sensors and aligned annotations, limiting modality transfer and use when those inputs are unavailable.The passage notes that many large manipulation datasets lack aligned 3D annotations.Source: §1 · p. 2
  • Weak 3D cues may lack robust geometry, fail to exploit higher-quality inputs, and disrupt vision-language alignment when spatial tokens are concatenated with text.The cited risks include embedding drift and degraded zero-shot generalization on spatial-prompt tasks.Source: §1 · p. 2
  • FALCON integrates richer spatial foundation-model tokens that provide comprehensive geometric information from RGB signals alone.This is intended to improve robustness when explicit 3D inputs are absent.Source: §1 · pp. 2–3
  • Its Embodied Spatial Model optionally adds depth or camera-pose information when available, without requiring architectural changes.The model remains effective with RGB-only input.Source: §1 · p. 3
  • A Spatial-Enhanced Action Head uses spatial tokens for action decisions rather than forcing them into the VLM's text-token space.The design separates high-level semantic reasoning in the VLM from fine-grained motor control in the action head.Source: §1 · p. 3
  • Across three simulation benchmarks and 11 real-world tasks, FALCON consistently outperforms existing baselines with robustness and generalization.Evaluations include cluttered-scene manipulation, few-shot adaptation, spatial prompts, height-aware manipulation, and object-scale variation.Source: §1 · p. 3

2 Related Work

Prior 3D-enhanced VLAs either consume explicit 3D inputs or embed spatial features in the VLM, with trade-offs in modality transfer or alignment. Spatial foundation models advance image-based 3D reconstruction, but adapting their representations to robot action policies remains challenging.

2.1 3D-Enhanced Vision-Language-Action Models

  • Explicit point-cloud methods such as PointVLA [17] and GeoVLA [37] enhance geometric awareness but depend on particular 3D inputs, while VLM-embedded methods can disrupt pretrained alignment.The passage notes that recovering performance for the latter may require costly embodied instruction tuning.Source: §2.1 · p. 3

2.2 Spatial Foundation Models

  • DUSt3R [42], MASt3R [16], CUT3R [41], and VGGT [40] develop image-based reconstruction approaches, yet integrating their outputs into generalist robot policies remains challenging.The passage attributes the challenge to complex feature alignment or information loss during adaptation to action spaces.Source: §2.2 · pp. 3–4

3 Methodology

FALCON frames language-guided tabletop manipulation as action-sequence prediction from camera observations and language, combining VLM semantics with ESM geometry in a dedicated action head. Its training uses a pose-and-gripper loss, and its spatial model can incorporate optional depth and camera-pose inputs.

3.1 Problem Definition

  • The task is to map camera observations and a natural-language instruction to a sequence of robot actions for tabletop manipulation.Each action encodes a 6-DoF gripper pose and a binary open/close state; depth maps and camera poses are optional inputs.Source: §3.1 · p. 4

3.2 Overall Architecture

  • FALCON combines a 2D VLM, an Embodied Spatial Model (ESM), and a Spatial-Enhanced Action Head to produce robot actions from semantic and spatial representations.The VLM supplies a semantic action representation while the ESM extracts spatial tokens from a third-view image and optional geometric inputs.Source: §3.2 · p. 4

3.3 Training Objective

  • Training minimizes a composite action-horizon loss combining mean squared error for pose dimensions with binary cross-entropy for the gripper dimension.A weighting factor λ balances the two loss components.Source: §3.3 · p. 5
  • FALCON uses two post-training stages: first it aligns spatial tokens with a lightweight adapter while pretrained components are frozen, then it jointly refines the VLM and adapter.Other model components remain frozen during the second stage.Source: §3.3 · p. 5

3.4 Embodied Spatial Model

  • The ESM encodes images into spatial tokens and can incorporate depth and camera-pose conditions to build more accurate spatial representations.Training randomly injects depth and/or pose so the model can retain image-only spatial reasoning when those inputs are absent.Source: §3.4 · p. 6

3.5 Spatial-Enhanced Action Head

  • The action head fuses pooled spatial tokens with the VLM semantic action token by projection and element-wise addition, preserving the VLM’s pre-trained representation space.The fused features are passed to an action predictor.Source: §3.5 · pp. 6–7
  • For long-horizon tasks, an LSTM predictor processes a history of fused features before an MLP predicts an action chunk.The predictor alternatives also include an MLP-based architecture.Source: §3.5 · p. 7

4 Experiments

Experiments evaluate FALCON in simulation and on real robots, then test spatial-token injection, fusion, and modality transfer. FALCON outperforms baselines across the reported settings, with strong results on cluttered and unseen tasks.

4.1 Simulation Experiments

  • In CALVIN, FALCON achieves state-of-the-art results in both ABC→D and ABCD→D, improving zero-shot ABC→D Avg. Len. by 4.13 over 3DDP and 1.05 over 3D Diffuser Actor.The comparisons are against methods relying on ground-truth point clouds.Source: §4.1 · pp. 7–8, Table 1 · p. 7
  • On SimplerEnv, FALCON outperforms all baselines on WidowX tasks and achieves the best overall performance on Google Robot tasks, with a 62.9% overall success rate.Reported challenging-task results include Put Spoon on Towel and Put Carrot on Plate; on Open Top Drawer and Place Apple, FALCON reaches 41.7% versus RT-2-X at 3.7%.Source: §4.1 · p. 8, Table 2 · p. 8, Table 3 · p. 8

4.2 Real-World Experiments

  • Across nine cluttered-scene Base Task suites, FALCON has the highest average success rate at 70.0%, exceeding SpatialVLA’s 44.4%.The suites cover language grounding with random distractors and semantic understanding of unseen object poses; evaluation totals 90 rollouts.Source: §4.2 · p. 8, Figure 3 · p. 9
  • In few-shot adaptation, FALCON leads the second-best model by 27.5% in Simple and 27% in Unseen Average, and reaches 80% on an Unseen Object variation.The variation is the task open drawer and place bread.Source: §4.2 · p. 9, Figure 4 · p. 9
  • Across four spatial-understanding tasks, FALCON achieves the highest success rates in both object-scale scenarios, while baselines encounter collisions or release smaller blocks prematurely.The evaluation varies spatial complexity and probes robustness to scale changes.Source: §4.2 · p. 9, Figure 5 · p. 9

4.3 In-Depth Analysis

  • Directly injecting ESM spatial tokens into the VLM degrades CALVIN performance, whereas FALCON’s action-head injection preserves the VLM and performs better.Zero-shot ABC→D Avg. Len. falls from 3.91 to 3.79 for the VLM-token variant; element-wise addition is the best-performing fusion strategy and adds no parameters.Source: §4.3 · p. 10, Table 4 · p. 10
  • FALCON benefits from additional depth at test time, while remaining robust when depth is omitted after RGB-D training.Avg. Len. rises from 4.08 to 4.09 in ABCD→D with test-time depth, and from 3.91 to 3.95 in ABC→D for RGB-only FALCON receiving test-time depth.Source: §4.3 · pp. 10–11
  • Real-world depth and camera-pose inputs raise task success from 60% to 80% in varying-height scenarios; ESM performs comparably to VGGT with RGB and improves with added depth or pose.The reported real-world comparison includes lifting a yellow pepper and placing a white cup on a pink cloth at changed cup height.Source: §4.3 · p. 11, Figure 7 · p. 11

5 Conclusion

FALCON adds 3D spatial understanding to generalist robot policies and is reported to achieve state-of-the-art performance across simulation and real-world tasks. Its post-training paradigm integrates spatial cues in stages while preserving pre-trained capabilities.

5 Conclusion

  • FALCON integrates foundation-model spatial tokens, optional depth or camera-pose inputs, and a Spatial-Enhanced Action Head while avoiding disruption to VLM alignment.Experiments across simulation and real-world tasks are reported to show state-of-the-art performance and robustness on spatially demanding tasks.Source: §5 · p. 11

Ethics Statement

  • The work focuses on robotic manipulation, with training on standard open-source datasets and testing in controlled settings; it notes possible bias and unexpected behavior.The stated intent is academic exploration.Source: Ethics Statement · p. 15

Reproducibility Statement

A.1 FALCON Training Paradigm

  • Post-training adds 3D geometric cues without incorporating them during pre-training, preserving pre-training efficiency while avoiding added computational cost and optimization complexity.The strategy is described as maintaining pre-trained knowledge, using zero-initialization for Stage 1 stability, and gradually unfreezing components in Stage 2.Source: §A.1 · pp. 15–16
  • Stage 1 freezes the pre-trained components and trains only a zero-initialized adapter, initially limiting spatial-token contributions to preserve the pre-trained feature space.This stage aligns spatial tokens with the semantic feature space without disrupting pre-trained representations.Source: §A.1 · p. 15
  • Stage 2 unfreezes the VLM and adapter while keeping the action predictor and ESM frozen, allowing feature adaptation to incorporate spatial information while maintaining linguistic understanding.The phased approach is intended to stabilize convergence and prevent spatial features from overwhelming semantic representations early in training.Source: §A.1 · pp. 15–16

A.2 Embodied Spatial Model Training Paradigm

The ESM training setup samples frames from randomly selected scenes and trains with AdamW using different learning rates for its backbone and prediction heads.

A.2 Embodied Spatial Model Training Paradigm

  • ESM training follows the dataset and preprocessing procedures of VGGT [40], sampling 1–12 frames from a randomly selected scene and using 24 images per batch.Source: §A.2 · p. 16
  • AdamW training uses a 1e-6 learning rate for the unified transformer backbone and 1e-5 for the depth, camera, and point heads.Source: §A.2 · p. 16
  • ESM training uses Bernoulli probability p = 66% and requires 16 A100 GPUs for approximately two days.Source: §A.2 · p. 16

B Hyper-Parameters and Training Details

The hyper-parameter details cover ESM pre-training and two-stage VLA post-training, as well as real-world and simulation training protocols. The evaluation protocol fixes training duration, with checkpoint selection varying by benchmark.

B Hyper-Parameters and Training Details

  • ESM pre-training uses a 2e-5 learning rate and global batch size 128; Stage 1 post-training uses 1e-4, batch size 128, and no warmup.Stage 2 hyperparameters are referenced in Table 7.Source: B Hyper-Parameters and Training Details · p. 16
  • Real-world FALCON Stage 1 uses learning rate 1e-4 and global batch size 512; baseline settings and iteration counts vary by model and task setting.The real-world models use side-camera input only.Source: B Hyper-Parameters and Training Details · p. 16
  • Across experiments, FALCON uses AdamW with a constant learning-rate scheduler, and images including depth maps are resized to 224×224; loss weighting is λ = 0.01.Source: B Hyper-Parameters and Training Details · p. 16
  • Because offline metrics may not correlate fully with long-horizon rollout performance, models train for fixed durations; CALVIN evaluates the final checkpoint after five epochs.SimplerEnv reports the best-performing checkpoint, while real-world experiments evaluate only the final checkpoint.Source: B Hyper-Parameters and Training Details · pp. 16–17

C Implementation Details

Implementation details specify benchmark-specific observation histories, action predictors, action-chunk lengths, and execution strategies. Table 7 is identified as the source of FALCON hyper-parameters for different experiments.

C Implementation Details

  • On CALVIN, the VLM uses side and wrist cameras with 16-frame history, while the ESM processes third-view images with the same history; an LSTM predicts chunks of 10 actions.Source: C Implementation Details · p. 17
  • For SimplerEnv and real-world evaluation, the VLM and ESM use a single-step third-view image, and an MLP predicts action chunks of length 5.Source: C Implementation Details · p. 17
  • CALVIN and SimplerEnv execute ensemble actions before predicting the next chunk, whereas real-world evaluation executes the full chunk; FALCON runs at about 57 Hz using approximately 12.8 GB GPU memory on an RTX 4090.Source: C Implementation Details · p. 17
  • Table 7 presents FALCON hyper-parameter setups for different experiments and defines Ep as epochs and Iters as iterations.Source: Table 7 · p. 17

D Benchmark Details

The paper evaluates FALCON across simulation and real-world settings designed to test long-horizon manipulation, spatial perception, and generalization. In a CALVIN loss-weighting ablation, increasing λ from 0.01 to 0.05 slightly reduced average task length and increased training-loss oscillation.

D Benchmark Details

  • CALVIN evaluates long-horizon language-conditioned manipulation across four scene splits, with ABC→D and ABCD→D protocols testing unseen environments and sequential task completion.Evaluation uses the D split, 1,000 rollouts, and five consecutive subtasks; metrics are success rate and average achieved task length.Source: D Benchmark Details · p. 17
  • SimplerEnv evaluations use WidowX and Google Robot platforms; the WidowX tasks include object placement, block stacking at two distances, and transferring an eggplant into a basket.The benchmark is intended to test transfer and generalization under varied lighting, textures, colors, and camera viewpoints.Source: D Benchmark Details · pp. 17–18
  • Google Robot experiments include can picking, moving one object near another, opening or closing a specified drawer, and opening a drawer before placing an apple inside.The described setups use 75, 60, 54, and 27 trials, respectively.Source: D Benchmark Details · p. 18
  • The Real World Benchmark comprises 1,030 teleoperated expert trajectories spanning five robot-learning scenarios and 11 tasks, from simple object interactions to long-horizon sequences.The physical experiments use an xArm 6 with a Robotiq gripper and a third-person RealSense depth camera.Source: D Benchmark Details · pp. 18–19
  • Base Tasks cover nine task suites across dining-table, bedroom, and kitchen scenarios, including cluttered-scene language grounding and semantic understanding of unseen object poses.Each task uses 10 layouts with 10 trials; the training set contains 850 trajectories.Source: D Benchmark Details · pp. 18–19
  • Few-shot Adaptation tests four spatially demanding tasks with 20 demonstrations each, including unseen-object, unseen-background, and unseen-task-description variants.Spatial Understanding evaluations add two fine-tuned spatial-prompt tasks and two zero-shot tasks involving height and object-size variation.Source: D Benchmark Details · p. 19

E.1 Loss Weighting Factors

  • Avg. Len. dropped from 3.91 to 3.87 as λ increased from 0.01 to 0.05; larger λ also increased training oscillation, while convergence remained similar.The paper uses λ = 0.01 for its experiments and reports that performance is not highly sensitive to small λ variations.Source: §E.1 · p. 20, Table 8 · p. 20, Figure 10 · p. 20

E.2 LSTM Head Parameters

The appendix examines how history length and action chunk size affect a Kosmos-VLA LSTM action predictor on CALVIN ABC→D. Longer history gives the strongest reported performance and improves early training stability.

E.2 LSTM Head Parameters

  • The parameter study uses Kosmos-VLA with RGB-only input and an LSTM action predictor under the CALVIN ABC→D setting.The comparison varies history length H and chunk size C.Source: §E.2 · p. 20, Table 9 · p. 20
  • Avg. Len. was 4.23 with H=16, C=10, versus 3.90 with H=8, C=10; reducing either history length or chunk size degraded performance.The compared settings were H=16, C=10; H=16, C=5; and H=8, C=10.Source: §E.2 · p. 20, Table 9 · p. 20
  • Shortening history to H=8, C=10 slowed early loss descent and increased oscillation, while reducing chunk size maintained stable convergence.The stability difference is reported for the first 0–15k training steps; all configurations eventually converged.Source: §E.2 · p. 20, Figure 11 · p. 20

F Potential Future Works

The paper reports a small performance gain from adding wrist-camera images to FALCON’s Embodied Spatial Model and suggests multi-view inputs may provide complementary geometric cues. It also presents qualitative rollout examples for CALVIN and SimplerEnv.

F Potential Future Works

  • In CALVIN ABCD→D, adding wrist images increased Avg. Len. from 4.08 to 4.10.The authors suggest multi-view inputs may provide complementary geometric cues and identify more consistent multi-view systems as a future direction.Source: F Potential Future Works · p. 21, Table 10 · p. 21

G Rollout Examples in CALVIN

  • The appendix presents rollout examples on the ABC→D split of CALVIN.The figure caption identifies the split but does not report a comparative outcome.Source: Figure 12 · p. 22

H Rollout Examples in SimplerEnv

  • Qualitative rollout results are shown for SimplerEnv tasks, including putting a carrot on a plate and an eggplant into a yellow basket.The task names appear with the qualitative-results figure.Source: Figure 13 · p. 23

I.1 Base Tasks Rollouts

The appendix provides qualitative rollout examples for Base Tasks and spatial-understanding evaluations, alongside few-shot performance comparisons across the simple setting and several unseen variants.

I.1 Base Tasks Rollouts

  • Figure 14 presents qualitative rollout results for Base Tasks.The caption does not specify a comparative outcome.Source: Figure 14 · p. 24

I.2 Few-Shot Adaptation Rollouts and Detailed Results

  • Few-shot results are reported for the Simple setting and unseen-object, unseen-background, and unseen-task-description variants.Figure 16 compares methods across these settings, while Tables 11–15 report final and subtask success rates.Source: Figure 16 · p. 25, Table 11 · p. 26, Table 12 · p. 26

I.3 Spatial Understanding Capability Evaluation Rollouts

  • Figure 17 presents qualitative rollout results for Spatial Understanding Capability Evaluations.The caption does not state a comparative outcome.Source: Figure 17 · p. 27
Loading 2510.17439v3…