Source-linked AI summary
VLA-Trace: Diagnosing Vision-Language-Action Models through Representation and Behavior Tracing
Haoyuan Shi, Xiancong Ren, Yingji Zhang, Qinfan Zhang, Jiayu Hu, Haozhe Shan, Han Dong, Jinpeng Lu, Yinda Chen, Yi Zhang, Yong Dai, Xiaozhu Ju
TL;DR
How VLA models transform multimodal knowledge into embodied control remains poorly understood. VLA-Trace connects representation analysis, attention interventions, and behavioral probes to examine that process. Tests of π0.5 and OpenVLA reveal distinct adaptation and routing patterns, alongside limits in fine-grained semantic following.
Problem
How finetuning preserves and aligns multimodal knowledge, and which modalities govern action decoding, remains unclear, limiting diagnosis of model failures and architecture design.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The findings point toward preserving visual, language, and cross-modal semantic pathways and integrating them more reliably into action decoding for compositional control.
Source: §5 · p. 9The complete evidence chain focuses on π0.5 and OpenVLA, so broader claims across VLA designs await completion of additional evaluation settings.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Understanding how Vision-Language-Action (VLA) models transform multimodal knowledge into embodied control remains an open challenge. We present VLA-Trace, a progressive diagnostic framework that analyzes VLA models through a unified evidence chain from representation dynamics to causal control attribution and behavioral manifestation. It specifically combines cross-modal and checkpoint-drift centered kernel alignment (CKA) to trace representation evolution, attention knockout interventions to identify modality-specific control pathways, and rollout-level behavioral probes to examine grounding, shortcut dependence, and semantic following. Experiments on $\pi_{0.5}$ and OpenVLA reveal three key findings. First, the two models exhibit distinct modality-specific adaptation dynamics during VLA finetuning. Second, they rely on different multimodal routing strategies and layer-wise dependencies during action decoding. Third, although VLA policies excel at visually grounded trajectory generation, they remain limited in fine-grained semantic following. These findings highlight future directions for representation-preserving adaptation, causal VLA circuits, and compositional semantic control.
1 Introduction
VLA-Trace addresses the limited understanding of how VLA finetuning reshapes multimodal representations and how visual and linguistic signals govern action decoding. Its three-stage analysis links representation dynamics, causal pathways, and behavior, revealing distinct adaptation and routing strategies in π0.5 and OpenVLA alongside limited fine-grained semantic following.
1 Introduction
- VLA-Trace links representation evolution, causal control attribution, and behavioral probes in a progressive diagnostic framework.It combines cross-modal and checkpoint-drift CKA, attention knockouts, and attention localization with input editing.Source: §1 · p. 1, Figure 1 · p. 2
- π0.5 and OpenVLA exhibit distinct adaptation dynamics: π0.5 fluctuates more in cross-modal fusion and reorganizes textual representations, while OpenVLA mainly restructures visual and cross-modal representations.OpenVLA preserves text-pooled representations and has smoother but often weaker intermediate-layer image–text alignment.Source: §1 · pp. 1–2
- π0.5 relies on a concentrated visual-to-action pathway that limits language’s functional role, whereas OpenVLA distributes control-relevant information across visual and textual access.Source: §1 · p. 2
- VLA policies localize to manipulation-relevant regions but often ignore fine-grained semantic modifications, indicating strong visually grounded trajectory imitation and weak compositional language control.Source: §1 · p. 2
- The findings point to future directions in representation preservation, causal VLA circuits, and compositional semantic control for more interpretable and robust models.Source: §1 · p. 2
2 Related Work
Related work situates VLA-Trace amid progress in embodied policies and analyses of their representations, mechanisms, and input-level behavior. It identifies a gap between these analysis strands: existing work does not connect encoded information, causal use during action decoding, and behavioral failures.
2.1 Vision-Language-Action Models
- VLA models adapt VLM priors to map multimodal representations to robot actions, and recent systems report generalization across tasks and embodiments.Source: §2.1 · pp. 2–3
- Prior work reports that direct VLM-to-VLA finetuning can degrade visual and language representations, while leaving unclear whether pretrained knowledge is preserved and functionally used for action generation.Source: §2.1 · p. 3
2.2 Mechanistic Analysis of VLA Models
- Mechanistic analyses span representation probing, causal interventions, and input-level tests of grounding and robustness, but these strands remain fragmented.Representation studies identify encoded information, while input interventions expose behavioral failures without clarifying the internal path from representation shifts to those failures.Source: §2.2 · p. 3
- VLA-Trace integrates representation geometry, causal analysis, and input intervention into a progressive pipeline for embodied-control behavior.The framework overview organizes probes from representation geometry through causal attribution to behavioral manifestation, with probe targets and benchmark and model tags.Source: §2.2 · p. 3, Table 1 · p. 3
3 Representation Analysis Framework
VLA-Trace follows VLA adaptation from representation changes through action-decoding pathways to rollout behavior. Across π0.5 and OpenVLA, it finds distinct modality dependencies and strong visual grounding alongside limited fine-grained semantic following.
3 Representation Analysis Framework
- The framework combines staged CKA analysis, attention knockouts, and rollout probes of attention localization, visual masking, and instruction editing.The probes examine grounding, visual-context dependence, and whether edited language redirects behavior.Source: §3 · p. 3, §3.3 · pp. 6–7
3.1.1 CKA Alignment Analysis
- Cross-modal CKA finds different layer-wise alignment patterns, while checkpoint-drift CKA shows π0.5 reorganizing text representations and OpenVLA primarily restructuring visual and joint spaces.π0.5’s cross-modal alignment varies substantially across checkpoints and datasets; OpenVLA’s is often smoother but weaker at intermediate layers.Source: §3.1.1 · p. 4
3.2 Causal Pathways for Action Decoding
- Attention knockouts intervene during prefill and generation to test whether cross-modal integration and modality access are required for action decoding.The framework blocks vision-language attention during prefill and blocks action-token attention to vision or language during generation.Source: §3.2 · pp. 4–5
3.2.1 Attention Knockout
- π0.5 primarily depends on visual-to-action pathways, whereas OpenVLA relies on both visual and language modalities during action generation.OpenVLA’s Gen:no text intervention reduces success to 0.00% on every LIBERO suite; π0.5’s Gen:no image intervention drives success nearly to zero across datasets.Source: §3.2.1 · p. 6
- Gen:no image drops π0.5 to 4.00% at its most vulnerable layer, while OpenVLA’s Gen:no text, prefill, and combined intervention reach 0.00% across multiple layers, consistent with broader vulnerable regions in OpenVLA.Source: §3.2.1 · p. 6
3.3.1 Attention IoU and Patterns
- Both models attend to broader robot-object interaction regions, and their attention shifts with task progression from one subgoal to the next.Robot+Object IoU90 exceeds Object-only IoU90, and the example task shifts focus from the stove to the moka pot across phases.Source: §3.3.1 · p. 7
3.3.2 Visual Patch Masking
- Visual perturbations reveal model-specific shortcut dependencies: π0.5 and OpenVLA-OFT are most affected by target-object replacement, while OpenVLA is more affected by full-robot replacement.Background masking also reduces average success across models; π0.5 shows greater reliance on gripper-centered interaction geometry, while OpenVLA depends more on full-robot visibility.Source: §3.3.2 · pp. 7–8
3.3.3 Input Editing
- Edited target-object instructions do not consistently redirect behavior; policies often remain biased toward the original visual-task configuration.Some edits cause large performance drops, such as moka pot→frypan changing success from 75% to 30%.Source: §3.3.3 · p. 8
4 Discussion and Outlook
VLA-Trace reveals model-specific adaptation and action-routing patterns, alongside a shared difficulty incorporating fine-grained semantic edits into low-level behavior. The findings motivate more carefully balanced adaptation, reliable multimodal routing, and training for compositional control.
Representation preservation and embodied modality adaptation
- π0.5 and OpenVLA show distinct modality-specific adaptation: π0.5 has more fluctuating cross-modal fusion, while OpenVLA mainly restructures visual and fusion representations.π0.5 primarily reorganizes textual representations into task-conditioned control features; OpenVLA restructures visual and cross-modal fusion representations.Source: §4 · p. 8
- Future training should balance preserving vision-language priors with robotic-control adaptation, since insufficient adaptation may limit visuomotor learning and excessive adaptation may degrade grounding or increase visual-shortcut reliance.The discussion suggests semantic-edit objectives, counterfactual instruction-image pairs, object-swap interventions, and contrastive behavior supervision.Source: §4 · pp. 8–9
Designing causal visual-language circuits for action decoding
- π0.5 primarily relies on visual-to-action pathways, whereas OpenVLA uses both visual grounding and prompt-region access across a broader range of layers.The discussion proposes routing modules, including mid-to-late fusion gates and action-conditioned cross-attention, as possible architectural directions.Source: §4 · p. 8
From visual grounding to compositional semantic control
- VLA models can capture high-level task semantics but struggle to incorporate subtle semantic modifications into low-level control, especially when edits conflict with visual or task-specific priors.Edited instructions may therefore fail to produce consistent compositional behavioral changes.Source: §4 · pp. 8–9
5 Conclusion
VLA-Trace connects representation dynamics, causal action dependencies, and rollout behavior in analyses of π0.5 and OpenVLA. The results show divergent adaptation strategies and partial, architecture-dependent semantic control despite strong benchmark performance.
5 Conclusion
- VLA-Trace combines CKA, attention knockouts, attention localization, visual patch masking, and input editing to connect representation dynamics with causal action dependencies and rollout behavior.The framework analyzes representation and control in VLA models.Source: §5 · p. 9
- Experiments on π0.5 and OpenVLA reveal divergent adaptation strategies and a shared limitation: semantic control remains partial and architecture-dependent despite strong benchmark performance.The conclusion calls for better preservation of visual, language, and cross-modal semantic pathways and their integration into action decoding.Source: §5 · p. 9
- The findings point toward more reliable integration of visual, language, and cross-modal semantic pathways into action decoding for robust compositional control.The conclusion frames this as a future direction for VLA models.Source: §5 · p. 9
Limitations
The study identifies limits in interpreting its intervention, representation, localization, and behavioral probes, and confines its complete evidence chain to two VLA models.
Limitations
- Attention knockout tests pathway necessity under a specific intervention design, so induced failures should not be equated with natural out-of-distribution failures.Activation patching and representation restoration, which test sufficiency and mediation, are left to future work.Source: Limitations · p. 9
- CKA reflects sample-level representation geometry rather than an absolute grounding, perception, or language-understanding score, and depends on dataset, pooling strategy, and checkpoint comparison.The study emphasizes relative layer profiles and checkpoint drift; it also lacks random-feature or permutation CKA baselines.Source: Limitations · p. 9
- Attention IoU depends on simulator masks and patch resolution, making it a conservative localization measure rather than a complete account of visual grounding.VLM-style grounding probes can also be confounded by whether a checkpoint supports natural-language localization outputs.Source: Limitations · p. 9
- Because π0.5 C2 is one shared LIBERO-finetuned checkpoint across suites, suite-level differences reflect evaluation distributions rather than separate per-suite weight changes.The study's complete evidence chain covers π0.5 and OpenVLA; broader settings remain for future work.Source: Limitations · p. 9
Ethics Statement
The study uses public models and evaluation resources without collecting personal data or involving human-subject studies. It cautions that deployment still requires evaluation, oversight, and safety constraints.
Ethics Statement
- The study analyzes publicly available models and evaluation resources without collecting personal data or involving human-subject studies.Source: Ethics Statement · p. 9
- The analyzed models may inherit biases, safety risks, and failure modes from their training data and design.Source: Ethics Statement · p. 9
- Real-world deployment requires careful evaluation, human oversight, and appropriate safety constraints.Source: Ethics Statement · p. 9
A Implementation Protocol Details
The protocol tracks several VLA families but focuses on π0.5 and OpenVLA because they use distinct action-generation interfaces.
A.1 Model Interfaces and Input Templates
- π0.5 uses a PaliGemma-style VLM with flow-matching action generation, while OpenVLA predicts actions autoregressively over action tokens.The analyzed π0.5 transformer prompt includes image tokens, BOS, the task instruction, and a newline; proprioceptive state is not tokenized into that stream.Source: §A.1 · p. 12
- OpenVLA uses image patches followed by an instruction prompt, so semantic tokens must be separated from structural tokens in rollout and knockout experiments.Its prompt template is “In: What action should the robot take to {instruction}? Out:”.Source: §A.1 · p. 12
- OpenVLA-OFT replaces autoregressive action-token generation with continuous action regression during fine-tuning.The protocol uses it as a supplementary setting for comparing how decoding paradigms influence representation dynamics and causal pathways.Source: §A.1 · p. 12
A.2 CKA Protocol
The CKA protocol compares visual and textual representations across layers and training checkpoints using pooled token-span representations and neutral probe prompts.
A.2 CKA Protocol
- Neutral image-description templates keep open-domain and robot-domain representation probes comparable and distinct from action-generation prompts.COCO uses “A photo of <caption>”; LIBERO uses “A photo of a robot <instruction>”.Source: §A.2 · p. 12
- At each layer, hidden states are pooled over image, text, and joint spans to create three representation views.For OpenVLA, the image span is image-patch tokens and the text span is subsequent prompt tokens; for π0.5, the text span skips BOS after the image tokens.Source: §A.2 · p. 12
- Linear CKA measures layer-wise image–text alignment within a checkpoint and compares matched layers across checkpoints for each pooled view.Checkpoint-drift CKA is averaged over layers; high values indicate preservation of a representation subspace, while low values indicate stronger reorganization.Source: §A.2 · p. 13
A.3 Attention Knockout Protocol
The attention-knockout protocol blocks selected attention pathways during prefill or action generation to test modality dependencies and localize layer-wise bottlenecks.
A.3 Attention Knockout Protocol
- Knockout adds a large negative mask to selected attention logits before softmax while keeping weights, observations, and action heads unchanged.A zero mask preserves an attention edge, whereas −∞ blocks it.Source: §A.3 · p. 13
- All-layer, single-layer, and centered-window knockouts test global pathway necessity, layer localization, and dependencies that may bypass a single blocked layer.Window sizes of 3, 5, and 7 block progressively wider neighborhoods around a target layer; a large drop in a narrow window marks a candidate bottleneck.Source: §A.3 · p. 13
- Masks distinguish visual, semantic-language, structural-prompt, and action token sets, with partitions adapted to each model’s input format.For π0.5, structural tokens include BOS and newline; OpenVLA and OpenVLA-OFT also include system-prompt and special tokens such as “In:” and “Out:”.Source: §A.3 · p. 13
- Prefill interventions test direct vision–language exchange for π0.5 and image access during context construction for OpenVLA-style models.Additional layer-wise and windowed sweeps examine pathway distribution and localized bottlenecks across model families and evaluation suites.Source: §A.3 · pp. 13–14
- Generation-stage knockout blocks action positions from attending to selected visual, semantic, or structural source tokens during decoding.For autoregressive OpenVLA-style models, masks are applied on top of the causal mask and remove only source positions otherwise visible to the current action token.Source: §A.3 · pp. 13–14
A.4 Attention Localization Metrics and Visualizations
Attention-localization analyses measure where action-conditioned attention falls on simulator-aligned task regions and visualize how attention changes during rollouts.
A.4 Attention Localization Metrics and Visualizations
- Action-conditioned heatmaps average attention from action positions to image patches and are evaluated with attention mass, thresholded IoU, and peak-hit rate.Mass measures attention allocated to the target; IoU90 measures overlap with regions above the 90th-percentile threshold; Hit tests whether the maximum-attention patch is task-relevant.Source: §A.4 · p. 14
- LIBERO-10 rollouts are split into their first and second halves as a proxy for early and late subgoals, with localization metrics computed separately for each phase and mask.Source: §A.4 · pp. 14–15
- The IoU schematic projects action-conditioned attention onto the image grid, thresholds it at the 90th percentile, and compares high-attention patches with simulator-derived target masks.Overlap forms the numerator, while the union of high-attention and target patches forms the denominator.Source: §A.4 · p. 15
- Qualitative visualizations compare action-to-image attention across rollout stages and checkpoints, and separately show text-to-image attention by instruction token.The latter distinguishes token-level language-to-image grounding from action-conditioned visual grounding.Source: §A.4 · p. 15
A.5 Visual Perturbation and Editing Protocol
The visual-editing protocol masks selected image regions and varies replacement styles to probe which visual cues VLA policies use. Supplementary materials extend the analysis across model families, benchmark suites, and attention or knockout visualizations.
A.5 Visual Perturbation and Editing Protocol
- Image regions are replaced while model weights, instructions, and rollout environment remain unchanged, isolating the effect of visual edits.The protocol tests target objects, grippers, robot body or full robot, and background, applying the corresponding edit across camera views when multiple views are used.Source: §A.5 · pp. 15–16
- Background-color, black, and mosaic masking remove different visual cues, distinguishing dependence on identity, geometry, spatial relations, and scene layout.Background-color replacement is the most aggressive; black masking can preserve a coarse silhouette, while mosaic masking retains partial occupancy and coarse geometry.Source: §A.5 · p. 15
- Task-level masking success rates are reported for π0.5, OpenVLA, OpenVLA-OFT, and X-VLA across LIBERO suites, with an additional X-VLA check on Simpler.Tables 7–10 cover LIBERO-10, Object, Spatial, and Goal; Table 11 covers Simpler.Source: §A.5 · p. 16, Table 7 · p. 29, Table 8 · p. 30
B Instruction
- Benchmark instructions cover complementary LIBERO manipulation capabilities, five selected bimanual RoboTwin2.0 tasks, Simpler real-to-sim transfer, and CALVIN sequential multi-skill tasks.The evaluated RoboTwin2.0 subset contains 5 of 50 tasks.Source: B Instruction · p. 16, Table 15 · p. 35, Table 16 · p. 36
- Knockout plots report layer-wise success rates for π0.5 on LIBERO-10, Goal, Object, and Spatial with 3-, 5-, and 7-layer windows, and for OpenVLA with 3- and 7-layer windows.OpenVLA-OFT is shown on the same four LIBERO suites with a 3-layer window.Source: Figure 6 · p. 17, Figure 7 · p. 17, Figure 8 · p. 17
- Attention-IoU uses the top 10% of action-attended image patches against a simulator-derived target mask; the mask also supports attention-mass and peak-hit measures.OpenVLA results show object IoU shifting between two instruction subgoals and attention mass allocated over robot and object regions.Source: Figure 16 · p. 23, Figure 22 · p. 28
- Qualitative action-to-image visualizations indicate that finetuning shifts attention toward task-relevant robot-object interactions; token-wise plots compare text-to-image attention across checkpoints.The token-wise comparisons cover π0.5 and OpenVLA across execution steps.Source: Figure 17 · p. 24, Figure 18 · p. 25, Figure 19 · p. 26
- Additional visualizations show action-to-text attention across timesteps and layer-wise modality attention at steps 30 and 150, comparing pretrained and fine-tuned models.The supplementary tables provide additional π0.5 layer-wise attention-routing statistics alongside IoU and mass results.Source: Figure 20 · p. 27, Figure 21 · p. 27, Table 14 · p. 34