PaperlayerView exact revision on arXiv

Source-linked AI summary

Sensorimotor Alignment: A Platonic Proxy for Embodied Policy Success

Yunfei Xie, Howard H. Qian, Shiyi Lan, Kaiyu Hang, Chen Wei

arXiv:2610.31670v1cs.CVcs.LGcs.RO

TL;DR

Closed-loop rollouts make evaluating robotic foundation models costly, while perception-side proxies do not establish whether representations are organized for action. We introduce S², a training-free offline measure of sensory–motion alignment, and find that it correlates with task success across embodied model families and supports representation-level explanations of observed behavior.

  • Problem

    Closed-loop rollouts are costly, and perception-side proxies measure passive perception or generation rather than whether representations are organized for action.

    Source: §1 · p. 2

  • Method

    S² is a training-free offline metric that compares model image-text features with recorded end-effector trajectories through their neighborhood overlap.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Across embodied model families and simulated and real-world settings, S² correlates positively with task success and more strongly than VQA scores and video-quality metrics.

    Source: §1 · p. 3, §6 · p. 11

  • Takeaways & Limitations

    S² offers a low-cost proxy for prioritizing models and VLM backbones before closed-loop evaluation, and a representation-level account of VLA and WAM phenomena.

    Source: §1 · p. 3, §6 · p. 11

    The relationship with task success is correlational, and the study has not shown that using S² during training improves success rates.

    Source: §6 · p. 11

Abstract

from arXiv · show

Robotic foundation models are typically evaluated through costly closed-loop rollouts, which motivates cheap offline proxies for model capability. The Platonic Representation Hypothesis (PRH) suggests that as representations improve, different modalities such as vision and language converge toward a shared statistical model of reality. We ask whether motion trajectories form another such modality: as embodied models become more capable, should their sensory representations converge toward the structure of physical motion? To examine this, we propose Sensorimotor Alignment ($S^2$), a training-free offline metric that measures how well a model's sensory representation aligns with the geometry of physical motion. We find that $S^2$ correlates positively with downstream robot task success across embodied model families, making it a low-cost offline proxy for prioritizing models before closed-loop evaluation. $S^2$ can also identify promising vision-language model (VLM) backbones even before costly fine-tuning into vision-language-action models (VLAs). Beyond this correlation, $S^2$ offers representation-level explanations for phenomena observed in VLAs and world action models (WAMs). With $S^2$, we extend PRH to embodied models and make sensorimotor alignment empirically measurable, offering insight into why an embodied model succeeds beyond its closed-loop success rate.

1. Introduction

S² connects sensory representations to the structure of robot motion and correlates with task success across embodied models. It also provides a representation-level lens on behaviors previously observed through closed-loop results.

1. Introduction

  • Closed-loop evaluation in simulation or the real world remains the most reliable test, but is slow, noisy, and benchmark-dependent.This motivates low-cost offline proxies for robotic capability.Source: §1 · p. 2
  • Existing perception-side metrics assess passive perception or generation quality, not whether representations are organized for action.Visually similar scenes may require different trajectories, while visually different scenes can call for similar motions.Source: §1 · p. 2
  • Across model families and simulated and real-world settings, S² correlates positively with downstream robot task success.It also correlates more strongly with success than VQA scores for VLMs and video-quality metrics for WAMs.Source: §1 · p. 3
  • Fine-tuned VLA task success correlates with the S² of vision features, not text features, consistent with models acting mainly on visual input.This offers a representation-level explanation for observed instruction neglect.Source: §1 · p. 3
  • S² is a training-free offline metric that compares sensory and motor representations, providing a low-cost proxy for prioritizing models before closed-loop evaluation.The sensory representation uses image and text features; the motor representation is the recorded end-effector trajectory.Source: §1 · pp. 2–3

2. Measuring Sensorimotor Alignment

S² measures whether nearby sensory representations correspond to nearby physical trajectories. It compares episode neighborhoods built from image-text features and subsequent end-effector motion.

2. Measuring Sensorimotor Alignment

  • The reference datasets pair sensory inputs with physical motions, and the method samples episodes from DROID and BridgeDataV2.For each episode, one image frame is paired with its task instruction as the sensory input.Source: §2 · p. 3
  • The sensory representation encodes each sampled image and task instruction with the model’s vision and text encoders, then average-pools token features into one vector.The pooled vector is d-dimensional.Source: §2 · p. 3
  • The motor representation records end-effector positions, orientations, and gripper states over the next h steps after the sampled frame.Position and orientation distances reflect translation and rotation geometry.Source: §2 · p. 4
  • S² compares sensory and motor k-nearest-neighbor sets; greater overlap indicates stronger alignment between sensory representations and trajectories.Figure 2 depicts shared neighbors and sensory-only neighbors for a query episode.Source: §2 · p. 4, Figure 2 · p. 4
  • Sensory neighbors are computed with cosine distance, while motor trajectories use Dynamic Time Warping to compare motion at different speeds.Reported S² uses a centered, normalized CKNNA variant with k=10 unless otherwise stated.Source: §2 · p. 4

3. Sensorimotor Alignment Correlates with Robot Task Success Rate

The paper tests whether sensorimotor alignment (S2) tracks robot task success and can serve as an offline proxy. Across model families, alternative metrics, and evaluation settings, S2 is positively associated with success.

3.1. S2 Correlates with Task Success Across Model Families

  • S2 correlates positively with success for fine-tuned VLAs (r=+0.717), raw VLM backbones (r=+0.681), WAMs (r=+0.718), and real-robot policies (r=+0.831).The raw-VLM correlation is with the success of VLAs fine-tuned from those backbones.Source: Figure 3 · p. 5, §3.1 · pp. 5–6
  • S2 also correlates with success for frozen vision backbones under low-data behavior cloning and visual-reward learning.This validation is reported in Appendix B.6.Source: §3.1 · p. 6

3.2. S2 Correlates with Task Success More Closely Than Alternative Metrics

  • S2 correlates with VLA success at r=+0.681, whereas aggregate VQA scores show virtually no correlation (r=-0.154).For WAMs, S2 on ALOHA correlates with success at r=+0.999, more strongly than video-quality metrics SSIM (r=+0.558), PSNR (r=+0.468), and FID (r=+0.680).Source: §3.2 · p. 6, Figure 4 · p. 6, Figure 5 · p. 7

3.3. S2 Is Robust to Evaluation Choices

  • The positive S2-success correlation holds across eight computation settings, including reference datasets, trajectory horizons, implementation choices, and DTW warping.The reported checks include different reference datasets and horizons, with correlations above r=0.5 across tested horizons on both datasets.Source: §3.3 · p. 7
  • For three WAMs, using a mismatched single-arm reference dataset reduces the S2-success correlation from r=+0.999 to r=+0.246.The negative control indicates that S2 is sensitive to embodiment mismatch.Source: §3.3 · p. 7

4. Representation-Level Explanations with S2

The paper uses S2 to examine representation-level patterns in VLAs and WAMs. The analyses connect success correlations to modality use, encoder training, denoising steps, and model depth.

4.1. What Carries the Correlation: Vision, Text, or Both?

  • In fine-tuned VLAs, vision-feature S2 correlates with success at r=+0.694, nearly matching joint-feature S2 at r=+0.699, while text-feature S2 is not significant at r=+0.237.Raw VLMs show a different pattern: joint features correlate most strongly (r=+0.681), followed by text (r=+0.501) and vision (r=+0.425).Source: Figure 8 · p. 8, §4.1 · p. 9

4.2. Freezing Vision Encoders Limits S2 Gain

  • Freezing the vision encoder reduces S2 by 41% and SimplerEnv success rate by 40% in the compared VLA models.The authors suggest end-to-end fine-tuning improves performance by bringing sensory representations closer to motor representations.Source: §4.2 · p. 9, Figure 9 · p. 9

4.3. Features at Early Denoising Steps Already Carry Strong Sensorimotor Alignment

  • Across three WAMs, S2 is already high at the earliest tested denoising step and varies by at most about 6% from its initial value.The authors infer that early-step features are aligned sufficiently with trajectories for action generation, explaining why skipping iterative video denoising barely lowers success.Source: §4.3 · pp. 9–10, Figure 10 · p. 10

4.4. Mid-Depth Blocks Carry the Strongest Sensorimotor Alignment

  • Block 14 has the highest S2 in all three tested WAMs, placing peak alignment in the shared backbone’s mid-depth range.This overlaps with the mid-depth range previously identified as important for semantic reasoning over objects and interactions.Source: §4.4 · p. 10

5. Related Work

The related-work discussion positions S2 as a broader representation-based account of robotic success and extends the Platonic Representation Hypothesis to robotics.

5. Related Work

  • Earlier robotics representation studies link selected visual properties or encoders to performance, but do not account for executed trajectories or span the same range of embodied models.S2 compares nearest-neighbor sets in sensory and motor spaces and is reported to correlate with success across VLAs, raw VLMs, WAMs, and evaluated vision encoders.Source: §5 · p. 10
  • Prior PRH applications include image and video generation, multimodal understanding, and cross-modal alignment; the paper extends this perspective to robotics by treating trajectories as another projection of the world.The authors report that more capable embodied models align sensory representations more closely with trajectories.Source: §5 · pp. 10–11

6. Conclusion

The conclusion presents S2 as a low-cost offline metric whose positive association with task success supports a PRH-inspired view of embodiment. It also notes that the relationship is correlational and that evaluation covers limited model families, benchmarks, training recipes, and real-robot settings.

6. Conclusion

  • S2 is a training-free offline metric aligning sensory representations with end-effector trajectories, intended for model selection and diagnosis before closed-loop evaluation.The authors say this correlation motivates optimizing embodied models directly for sensorimotor alignment.Source: §6 · p. 11
  • S2 correlates positively with task success across VLAs, raw VLMs, WAMs, and real-robot policies, and more strongly than VQA scores and video-quality metrics.The authors also report representation-level explanations for several VLA and WAM phenomena.Source: §6 · p. 11
  • S2 offers representation-level accounts of modality dominance, vision-encoder freezing effects, skipping iterative denoising, and mid-depth blocks’ semantic reasoning role.These are phenomena previously observed through closed-loop behavior.Source: §6 · p. 11
  • The S2–success relationship is correlational, similar S2 scores can accompany different success rates, and evaluation covers limited settings and model comparisons.The authors have not demonstrated that using S2 during training improves task success; closed-loop testing remains the final model-selection step.Source: §6 · p. 11

A. Sensorimotor Alignment Metric

The section details how S2 is computed from paired sensory inputs and robot trajectories, including reference data, trajectory distances, and a local nearest-neighbor alignment estimator.

A. Sensorimotor Alignment Metric

  • The appendix specifies the full S2 computation, covering reference datasets and sampling, trajectory distance, and alignment estimation.S2 scores alignment between sensory and motor neighbor sets.Source: A. Sensorimotor Alignment Metric · p. 17

A.1. Reference Datasets and Sampling

  • DROID and BridgeDataV2 provide single-arm demonstrations, while ALOHA supplies bimanual trajectories for WAMs; sampled horizons reuse the same episodes, anchors, and sensory inputs.BridgeDataV2 retains labeled episodes, while the ALOHA reference uses sampled trajectory windows.Source: §A.1 · p. 17

A.2. Trajectory Distance

  • Single-arm motor distances combine position, orientation, and gripper components, then use symmetric2 DTW without a warping window or path-length normalization.The component weights default to one; ALOHA uses Euclidean distance between 14-dimensional joint-position vectors as its local cost.Source: §A.2 · pp. 17–18

A.3. Alignment Estimator

  • The estimator uses cosine sensory similarities and DTW-selected motor neighbors, then applies centered kernel nearest-neighbor alignment to focus on local shared-neighbor structure rather than global CKA.The directed centered score retains sensory weights and is not guaranteed to lie in [0, 1].Source: §A.3 · pp. 18–19
  • The alignment computation uses directed neighbor matrices and an unbiased HSIC estimator that corrects finite-sample bias under i.i.d. sampling.Separate masks are used for cross and self terms in the masking-before-HSIC computation.Source: §A.3 · p. 19

B. Experiments on S2 and Task Success

The experiments examine S2 and task success across model families, including VLA confound controls and frozen-vision-encoder tests. The VLA comparison covers 12 models spanning multiple action-head types and VLM training regimes.

B. Experiments on S2 and Task Success

B.1. Vision-Language-Action Models

  • The VLA cohort contains 12 models spanning four action-head types and three VLM training regimes, from frozen to end-to-end.The set includes StarVLA variants, CogACT sizes, GR00T models, Pi0-Bridge, and SpatialVLA-Bridge.Source: §B.1 · p. 20
  • The VLA comparison uses reported SimplerEnv success rates from model papers and public evaluation repositories.Source: §B.1 · p. 20

B.2. Confound Controls for the VLA Correlation

The section tests potential confounds in the VLA correlation and extends S2 evaluation to raw VLM backbones, WAMs, real-robot policies, and frozen vision encoders.

B.2. Confound Controls for the VLA Correlation

  • The fixed VLM4VLA recipe retained an S2–success correlation of r=+0.681, while three CogACT models showed r=+0.826.The three-model family check has little statistical power.Source: §B.2 · p. 20
  • Removing each architecture group kept the correlation positive, from r=+0.526 to r=+0.785, and significant in four of five cases.Removing the five StarVLA variants left seven models and a one-tailed p=0.113.Source: §B.2 · p. 20, Table 1 · p. 20

B.3. Raw Vision-Language Models

  • Eight raw VLM backbones were scored on DROID before robot fine-tuning, then compared with SimplerEnv success of their BridgeDataV2-finetuned VLAs.The tested backbones and scoring settings are specified in the section.Source: §B.3 · p. 20

B.4. World Action Models

  • Six WAMs were scored using identical bimanual ALOHA trajectories and fixed feature position, horizon, neighbor count, and DTW settings.The reported comparison uses video-backbone Block 7 features at h=15 and k=10.Source: §B.4 · p. 21

B.5. Real-Robot Policies

  • The real-robot evaluation covered 120 trials per policy across three Franka DROID platforms, four environments, and ten pick-and-place tasks per environment.At the prespecified h=7, the reported correlation was r=+0.429 with one-tailed p=0.236; h=110 was exploratory.Source: §B.5 · p. 21

B.6. Frozen Vision Encoders Under Downstream Policy Learning

  • r=+0.774 for VRF and r=+0.677 for low-data BC link frozen-backbone S2 with downstream success across 21 tasks.The association weakens as the BC demonstration budget grows, and no positive correlation appears for the reported RL seed sets or Robosuite.Source: §B.6 · pp. 21–22, Figure 11 · p. 21

C. Robustness of S2

The robustness analyses examine S2 across trajectory horizons and summarize how its association with success changes by feature type.

C. Robustness of S2

  • The robustness section tests stability across trajectory horizon and implementation choices, and whether S2 responds to reference-data mismatch.The analyses include changes to DTW warping and a mismatch in arm configuration.Source: C. Robustness of S2 · p. 22

C.1. Trajectory Horizon

  • The horizon analysis scores 12 fine-tuned VLAs on DROID using 6,000 episodes and 10 neighbors, comparing S2 with SimplerEnv-WidowX success.The tables report Pearson and Spearman correlations and their one-tailed p-values.Source: §C.1 · p. 22, Table 2 · p. 23
  • Across horizons, joint-feature r=+0.610 to +0.717 and vision-feature r=+0.554 to +0.703 remain significant; text-feature r=+0.227 to +0.274 is never significant.The reported tables compare the three feature types against SimplerEnv-WidowX success across 12 fine-tuned VLAs.Source: §C.1 · p. 22, Table 2 · p. 23, Table 3 · p. 23

C.2. Implementation Choices

The section checks whether the S2–success association persists under alternative implementation choices and DTW settings, and whether it responds to embodiment mismatch.

C.2. Implementation Choices

  • Across tested settings for neighbor count, normalization, component weights, rotation distance, and pooling, the S2–success correlation remains positive and significant, with r=+0.634 to +0.752.These checks vary one choice at a time while retaining the main-paper defaults otherwise.Source: §C.2 · p. 24, Table 5 · p. 24
  • Removing rotation or gripper-state weight lowers r most, while masked min pooling slightly exceeds masked mean and masked max is lowest.Anchor-relative trajectory normalization gives slightly higher r than absolute poses; the paper retains absolute poses for simplicity.Source: §C.2 · p. 24

C.3. DTW Alignment

  • DTW raises S2 for all 12 VLAs from h=7 onward, but the S2–success correlation remains nearly unchanged across horizons.At h=220, DTW raises S2 by 34 ± 4% on average; correlations range from +0.617 to +0.713 without DTW and +0.610 to +0.717 with DTW.Source: §C.3 · pp. 24–25, Table 6 · p. 25

C.4. Sensitivity to Embodiment Mismatch

  • With matched bimanual ALOHA reference data, S2 recovers the three WAMs’ success ordering (r=+0.999); with single-arm DROID data, r=+0.246 (p=0.421).The results indicate that reference data should match the target’s arm configuration, though it need not come from the exact target robot.Source: §C.4 · p. 25, Figure 12 · p. 25

D.1. S2 Across Feature Positions

The section compares S2 across feature positions and model groups, then examines encoder freezing and WAM denoising features.

D.1. S2 Across Feature Positions

  • Visual-feature S2–success correlation grows from r=+0.567 at vision-encoder output to r=+0.717 at LLM output, while text-only features show no significant correlation.The comparison covers 12 fine-tuned VLAs on DROID at h=7.Source: §D.1 · p. 26, Table 7 · p. 26
  • VLA mean S2 exceeds raw-VLM mean S2 at every compared feature position, with a larger relative difference at LLM output (+29.9% to +45.0%) than at vision-encoder output or LLM input (+12.7% to +18.1%).The group comparison also includes three standalone vision encoders.Source: §D.1 · p. 26, Table 8 · p. 26

D.2. Frozen Versus Unfrozen Vision Encoder

  • The frozen-encoder model has lower S2 than its unfrozen counterpart at every tested horizon, with relative reductions from 38.1% at h=1 to 49.4% at h=220.The comparison uses Qwen2.5-VL-3B models trained with the same recipe.Source: §D.2 · p. 27, Table 10 · p. 27

D.3. WAM Features Along the Denoising Schedule

  • Block 14 has the highest WAM S2 at 13 of 15 model and denoising-schedule positions; reducing LingBot-VA denoising steps from 25 to 5 lowers both success (56% to 48%) and S2 (0.288 to 0.241).The schedule-step result is measured on RoboTwin’s turn_switch task over 100 episodes.Source: §D.3 · p. 28, Table 11 · p. 28

Related papers

Loading 2610.31670v1…