PaperlayerView exact revision on arXiv

Source-linked AI summary

IMPACT-VLA: Interaction-aware Multimodal Propagation Attribution via Counterfactual Trajectories for Vision-Language-Action Policies

Jinwoong Kim, Sangjin Park

arXiv:2609.15005v1cs.ROcs.AI

TL;DR

VLA attribution methods leave unclear when modalities contribute to task success and how interventions propagate through closed-loop execution. IMPACT-VLA uses phase-modality blocks and counterfactual re-execution to analyze contributions and cross-phase dependencies. Experiments show phase-dependent contributions, conditional interactions, and functional recovery without behavioral recovery.

  • Problem

    It remains unclear when each modality contributes to task success and how interventions affect subsequent states, observations, and actions during closed-loop execution.

    Source: §I · p. 1

  • Method

    IMPACT-VLA defines phase-modality blocks from action-transition phases in a successful reference rollout and re-executes the policy in closed loop to analyze attribution, interactions, propagation, and recovery.

    Source: §I · pp. 1–2

  • Results

    Experiments found phase-dependent modality contributions and conditional cross-phase dependencies; negatively interacting pairs had a 3.3× later-block marginal-gain ratio under early-block replacement.

    Source: §I · p. 1, E. Cross-Phase Interaction and Recovery · p. 7

  • Takeaways & Limitations

    Later-phase contributions depend on earlier information availability, and functional compensation can occur without trajectory re-convergence.

    Source: §V · p. 8

    Evaluation was limited to OpenVLA-OFT in LIBERO and phases defined from one successful reference execution per task, so findings are relative to those references.

    Source: §V · p. 8

Abstract

from arXiv · show

Vision-Language-Action (VLA) policies perform robot manipulation tasks using multimodal inputs such as visual observations, proprioceptive states, and language instructions. However, it remains unclear at which execution stages each modality contributes to final task success and how input interventions propagate through subsequent states, observations, and actions. Existing attribution approaches primarily measure local sensitivity or temporally aggregated importance, limiting their ability to capture phase-dependent contributions and cross-phase dependencies. We propose Interaction-aware Multimodal Propagation Attribution via Counterfactual Trajectories for Vision-Language-Action Policies (IMPACT-VLA). IMPACT-VLA constructs behavioral phases from action transitions in a successful reference rollout, aligns them with policy query boundaries, and defines phase-modality blocks as attribution units. It then performs closed-loop counterfactual re-execution to quantify each block's contribution to final task success. We further analyze cross-phase non-additive interactions and trajectory propagation while distinguishing behavioral from functional recovery. Across 30 LIBERO robot manipulation tasks using OpenVLA-OFT, dominant-modality transitions occurred in 25 tasks (83.3%), and closed-loop attribution identified task-critical information more faithfully than Static Action Perturbation. Later-block marginal gains for negatively interacting pairs increased by approximately 3.3x under early-phase input replacement, while functional recovery could occur without behavioral recovery. These results reveal when multimodal inputs support task success and how their contributions become conditionally coupled during closed-loop execution.

I. INTRODUCTION

IMPACT-VLA addresses how modality contributions vary across execution phases and propagate through closed-loop VLA behavior. It uses phase-modality blocks and counterfactual re-execution to measure contributions to task success and analyze cross-phase dependencies.

I. INTRODUCTION

  • Existing approaches leave unclear when each modality contributes to task success and how interventions affect subsequent states, observations, and actions.Modality-level analysis can obscure stage-wise variation, while timestep-level units may not align with chunk-based policy queries.Source: §I · p. 1
  • Cross-phase analysis is needed because later inputs may contribute differently when earlier information is unavailable, and task recovery may differ from trajectory re-convergence.Earlier and later information can interact complementarily or redundantly.Source: §I · p. 1
  • Experiments show phase-dependent modality transitions, greater closed-loop faithfulness than immediate action sensitivity, and conditional cross-phase dependencies.Early-phase input replacement increased later-block marginal gains for negatively interacting pairs by approximately 3.3×, while functional recovery could occur without behavioral recovery.Source: §I · p. 1
  • IMPACT-VLA defines phase-modality blocks aligned with action transitions and action-chunk structure as temporal attribution units.It re-executes the policy in closed loop from intervention-altered environment states to quantify each block’s contribution to final task success across coalition contexts.Source: §I · pp. 1–2
  • The method also analyzes cross-phase non-additive interactions and trajectory propagation, distinguishing behavioral recovery from functional recovery.These measures characterize conditional dependencies and compensation following early information loss.Source: §I · p. 2

II. RELATED WORK

Prior work studies embodied-policy interpretability, input attribution, and behavior segmentation, but does not fully address phase-specific modality contributions and their closed-loop effects. This section contrasts those approaches with requirements for IMPACT-VLA.

A. Interpretability of Embodied Policies

  • Existing embodied-policy and VLA interpretability methods study salient observations, states, failures, or latent representations, but do not directly quantify when input modalities are needed or how their contributions propagate to task success.The paper motivates interventions on actual multimodal inputs and analysis of their closed-loop effects.Source: A. Interpretability of Embodied Policies · p. 2

B. Input Attribution and Evaluation

  • Input attribution methods cover individual components, coalition-based marginal contributions, temporal inputs, and interactions, while IMPACT-VLA evaluates phase–modality blocks through policy–environment re-execution.This captures intervention-induced changes in subsequent states and observations and supports cross-phase interaction, propagation, and recovery analysis.Source: B. Input Attribution and Evaluation · p. 2

C. Robot Behavior Segmentation

  • Skill-segmentation methods target reusable structure across demonstrations, whereas IMPACT-VLA requires post hoc segmentation of one execution from a trained policy.For chunk-based policies, changepoints inside an action chunk cannot define independently perturbed inputs, so temporal units must align with query and chunk boundaries.Source: C. Robot Behavior Segmentation · p. 2

III. METHODOLOGY

IMPACT-VLA segments a successful reference execution into action-transition phases and forms phase-modality blocks for counterfactual analysis. Policy and environment re-execution evaluates the blocks’ effects on final task success and supports analysis of interactions, propagation, and recovery.

A. Problem Formulation

  • The policy receives multimodal observations at query boundaries and generates an action chunk of length K for sequential execution.The policy is denoted π, and the input modalities are collected in M.Source: A. Problem Formulation · p. 2

III. METHODOLOGY

  • The IMPACT-VLA procedure segments a successful reference rollout by executed-action transitions, forms phase-modality blocks, and re-executes the policy and environment after selected input replacements.The same counterfactual re-executions support analysis of cross-phase interactions, trajectory propagation, and recovery.Source: §III · p. 2, Figure 1 · p. 3

A. Problem Formulation

  • Interventions are evaluated by re-executing the policy–environment interaction because they can change subsequent states, observations, and actions.The analysis uses a successful reference rollout with all modalities provided without intervention.Source: A. Problem Formulation · p. 3

B. Phase Construction

IMPACT-VLA segments a successful reference rollout into execution-aligned behavioral phases and treats each phase–modality combination as a distinct attribution block. It estimates block contributions by closed-loop counterfactual re-execution, using current environment observations and final task success.

B. Phase Construction

  • Behavioral phases follow executed-action transitions aligned to policy-query boundaries; each phase–modality pair is a distinct block.Channel transitions within one action chunk are grouped, and phases shorter than a chunk are merged with an adjacent phase.Source: B. Phase Construction · p. 3

C. Closed-Loop Attribution

  • Counterfactual queries acquire observations from the intervention-altered current state, and coalition value is expected final task success.Stored reference observations are not reused, allowing intervention effects to propagate through subsequent states, observations, and actions.Source: C. Closed-Loop Attribution · pp. 3–4
  • Each permutation's prefix coalitions share a rollout seed, so marginal contributions are paired within that permutation.Attribution therefore reflects contribution to final task success rather than immediate action sensitivity.Source: C. Closed-Loop Attribution · p. 4
  • Block contributions are average marginal gains across coalition contexts, estimated with uniformly sampled block permutations.LIBERO uses 128 permutations, and the attribution map indexes contributions by behavioral phase and input modality.Source: C. Closed-Loop Attribution · p. 4

D. Cross-Phase Interaction

Cross-phase interaction measures how the joint effect of an earlier and later block departs from their separate effects in the same coalition context. Its context-averaged value indicates complementarity or redundancy/suppression.

D. Cross-Phase Interaction

  • The interaction captures deviation of two cross-phase blocks' joint effect from the sum of their individual effects in the same coalition context.The earlier block belongs to phase pi and the later block to phase pj, with pi < pj.Source: D. Cross-Phase Interaction · p. 4
  • The interaction is a context-averaged finite-difference effect, not an approximation of the Shapley interaction index.Contexts are sampled so different coalition sizes are represented evenly and independently of Shapley permutations.Source: D. Cross-Phase Interaction · p. 4
  • Positive interaction is consistent with complementarity, while negative interaction is consistent with redundancy or suppression.This supports analysis of how earlier- and later-phase information interact with respect to final task success.Source: D. Cross-Phase Interaction · p. 4

E. Propagation and Recovery

The propagation and recovery analysis tracks trajectory deviation after early-block replacement and distinguishes later-block effects on trajectory stabilization from effects on final task success. Recovery is evaluated only when the early intervention produces measurable pre-damage.

E. Propagation and Recovery

  • Trajectory comparison uses matching absolute control timesteps, so intervention-induced delays are not time-warped.A rollout ending before the reference holds its last observed position through the reference horizon; extra execution beyond that horizon is excluded from trajectory comparison.Source: E. Propagation and Recovery · p. 4
  • Excess deviation is measured against full-input execution immediately before the later block activates, over that block's phase timesteps.Recovery analysis uses 32 paired rollouts and is restricted to ordered pairs with finite positive pre-damage.Source: E. Propagation and Recovery · p. 4
  • Behavioral recovery requires pre-damage and a later-block-attributable decrease in excess trajectory deviation after that block activates.The difference-in-differences criterion separates recovery attributable to the later block from natural trajectory convergence.Source: E. Propagation and Recovery · p. 4
  • Functional recovery tests whether a later block compensates for an earlier block's loss of final task success, using a full-coalition criterion.It requires the pre-damage gate, increased final success after replacing the earlier block, and a stronger success-preserving effect than under full input; the two recovery types may occur jointly or independently.Source: E. Propagation and Recovery · pp. 4–5

IV. EXPERIMENTS

The experiments evaluate IMPACT-VLA in controlled settings and on LIBERO, across attribution, phase-dependent contributions, closed-loop faithfulness, interactions, propagation, and recovery.

IV. EXPERIMENTS

  • Evaluation covers attribution accuracy, phase-dependent modality contributions, closed-loop faithfulness, cross-phase interactions, propagation, and recovery in controlled settings and on LIBERO.The supplied passage specifies evaluation scope but reports no experimental outcome.Source: §IV · p. 5

A. Experimental Setup

The setup fixes reference-rollout phase boundaries and evaluates interventions through repeated closed-loop re-executions, with task-level paired comparisons.

A. Experimental Setup

  • Each task uses OpenVLA-OFT with a fixed initial state and reference seed; one successful execution defines phases, whose boundaries stay fixed across counterfactual rollouts.Analyses use repeated closed-loop counterfactual re-executions.Source: A. Experimental Setup · p. 5
  • Mean intervention is primary, with gray, blur, and noise replacements used for robustness checks across vision, proprioception, and language inputs.Vision replacements use suite-specific means, gray, Gaussian blur, or fixed-seed noise; proprioception uses a mean state vector and language an empty string.Source: A. Experimental Setup · p. 5
  • Baselines are LOO, Static Action Perturbation, and Random; structural ablations restrict Shapley players by phase or modality, or use uniform query-aligned phases.Faithfulness comparisons use the mean intervention.Source: A. Experimental Setup · p. 5
  • Tasks are the statistical units; results use task-macro means and task-paired method differences, with 95% percentile confidence intervals from 10,000 task-cluster bootstrap resamples.The setup uses one successful reference rollout per task and repeated counterfactual rollouts.Source: A. Experimental Setup · p. 5

B. Controlled Validation of Attribution Estimation

Controlled tests assess sampled Shapley attribution against exact values, while LIBERO analyses show that modality contributions shift across phases and remain consistently ranked across intervention kernels.

B. Controlled Validation of Attribution Estimation

  • The controlled validation enumerates coalitions in Additive, Delayed Credit, and Genuine Interaction games, varying permutation budgets from 2 to 512 and repeating each condition 64 times.Accuracy is measured by MAE against exact-enumeration Shapley values and Spearman correlation with the ground-truth ranking.Source: B. Controlled Validation of… · p. 5
  • At 128 permutations, MAE stayed below 0.03 and Spearman correlation above 0.89 across all three scenarios; larger budgets lowered error and raised ranking agreement.The sampled estimator requires L(N + 1) coalition evaluations, versus 2N unique coalition values for exact subset-based computation.Source: Figure 2 · p. 5, B. Controlled Validation of… · pp. 5–6

C. Phase-Dependent Modality Attribution

  • Spatial rollouts show prominent early wrist vision and later proprioception; Object shifts from early wrist vision toward third-person vision, while Goal shifts from early language toward wrist vision and proprioception.The phase-level contributions are projected over normalized rollout progress, rather than shown as timestep-wise attribution.Source: C. Phase-Dependent Modality Attribution · p. 6
  • 25/30 tasks (83.3%) had adjacent-phase dominant-modality changes, and 37/92 changes (40.2%) had no overlap between dominant sets.Identical attribution values count as a joint dominant set.Source: C. Phase-Dependent Modality Attribution · p. 6
  • Across intervention kernels, task-level Spearman correlations range from 0.734 to 0.816 and Top-3 overlap from 0.857 to 0.933, while attribution magnitudes depend on replacement choice.The authors emphasize ranking structure because replacement choice may introduce distribution shift.Source: C. Phase-Dependent Modality Attribution · p. 6, Table II · p. 6

D. Closed-Loop Faithfulness

Closed-loop attribution more faithfully identifies task-critical information than immediate action sensitivity, while phase-aligned analysis reveals conditional cross-phase interactions and distinct forms of recovery.

D. Closed-Loop Faithfulness

  • IMPACT-VLA achieved the highest pAUC for deletion and insertion, with paired improvements over Static Action Perturbation whose confidence intervals excluded zero.The comparison evaluates whether attribution rankings reflect actual task outcomes through closed-loop deletion and insertion.Source: Table III · p. 6, D. Closed-Loop Faithfulness · p. 6
  • Compared with LOO, IMPACT-VLA showed no clear deletion difference but higher insertion performance; differences from Phase-restricted Shapley were not statistically clear.Absolute insertion pAUC remained low across methods, so it serves as a relative ranking test under limited information.Source: D. Closed-Loop Faithfulness · p. 6
  • Action-transition-aligned phases improved top-modality removal effects over uniform query-aligned phases, which outperformed Modality-only Shapley.Both paired phase-resolution differences had confidence intervals excluding zero.Source: Table IV · p. 7, D. Closed-Loop Faithfulness · p. 7

E. Cross-Phase Interaction and Recovery

  • For 1,046 negatively screened cross-phase pairs, later-block marginal gain rose from 0.025 [0.017, 0.033] to 0.082 [0.072, 0.093] after early-block replacement, a 3.3× ratio.The paired change was +0.057 [0.052, 0.063]; wrist-vision-to-proprioception had the strongest positive interaction, and approximately 31% of cross-phase pairs were negative.Source: E. Cross-Phase Interaction and Recovery · p. 7, Figure 4 · p. 7
  • Held-out validation found positive complementary interactions with high sign consistency, while negative interactions persisted in aggregate but had weaker edge-level consistency.Negative controls remained near zero.Source: Table V · p. 7, E. Cross-Phase Interaction and Recovery · p. 7
  • Mean trajectory deviation for redundant interactions increased from 2.4 cm to 10.5 cm, while complementary interactions showed substantial post-intervention deviation followed by partial reduction near the end.The authors suggest the reduction may reflect corrective behavior using subsequent observations.Source: Figure 5 · p. 7, E. Cross-Phase Interaction and Recovery · p. 7
  • After the pre-damage gate, Functional Recovery without Behavioral Recovery was 62.5% in the redundant group, showing task success can return without restoring the reference trajectory.The task-macro Functional Recovery rate was 81.3% in that group; the authors interpret it as context-specific compensation, not an independent interaction finding.Source: E. Cross-Phase Interaction and Recovery · pp. 7–8, Table VI · p. 8

V. CONCLUSION

On LIBERO, IMPACT-VLA showed phase-dependent modality contributions and greater closed-loop faithfulness than Static Action Perturbation, while later-phase contributions depended on earlier information. The evaluation is limited to OpenVLA-OFT and phases defined from one successful reference execution per task.

V. CONCLUSION

  • On LIBERO, IMPACT-VLA showed phase-dependent modality contributions and greater closed-loop faithfulness than Static Action Perturbation, while differences from Phase-restricted Shapley were not statistically clear.Later-phase contributions depended on earlier information availability, and functional compensation could occur without trajectory re-convergence.Source: §V · p. 8
  • Functional compensation could occur without trajectory re-convergence, motivating temporally structured closed-loop analysis rather than independent local sensitivity.Later-phase contributions also depended on earlier information availability.Source: §V · p. 8
  • The evaluation used OpenVLA-OFT in LIBERO and defined phase structure from one successful reference execution per task, so results should be interpreted relative to those references.The authors identify alternative references, VLA architectures, failure trajectories, and real robots as future evaluation settings.Source: §V · p. 8

Related papers

Loading 2609.15005v1…