PaperlayerView exact revision on arXiv

Source-linked AI summary

Event-Aligned Visual Action Reasoning for World Action Models

Xiaomeng Yang, Yushu Wu, Yi Gao, Yuhao Lei, Xuan Zhang, Pu Zhao, Yanzhi Wang

arXiv:2610.09427v1cs.CVcs.RO

TL;DR

World-Action Models use future visual prediction to guide action generation, but fixed temporal horizons do not reflect the different roles of task-critical interactions and connecting transitions. This paper aligns imagined visual-action context with interaction events and adds a head to identify valid actions for execution. It reports a 10.26% gain in DOMINO success rate over LingBot-VA, competitive RoboTwin 2.0 performance, and transfer from DOMINO Level 1 to Levels 2 and 3 without target-level adaptation.

  • Problem

    Fixed temporal horizons can split brief task-critical interactions across prediction chunks, despite interactions and connecting transitions serving different roles in action reasoning.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    Event-aligned visual-action supervision organizes imagined states around interactions, and an execution validity head predicts which actions in each chunk should be executed.

    Source: §1 · p. 2

  • Results

    A 10.26% gain in DOMINO success rate over LingBot-VA accompanies competitive RoboTwin 2.0 performance and transfer from DOMINO Level 1 to Levels 2 and 3 without target-level adaptation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework adapts visual reasoning granularity to interaction dynamics, with detailed state evolution around interactions and coarser progression through transitions.

    Source: §5 · p. 10

    Event and evidence boundaries come from source-demonstration annotations and are fixed offline.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

World-Action Models (WAMs) utilize future visual prediction as an intermediate reasoning process to guide action generation. However, existing WAMs typically structure visual imagination according to predefined temporal intervals, without explicitly accounting for the different roles of task-critical interactions and connecting transitions. We argue that effective visual foresight should align directly with task-relevant interactions and their corresponding reasoning demands. To this end, we introduce an event-aligned visual action reasoning framework that organizes visual-action prediction around interaction events. Through event-aligned visual-action supervision, WAM learns to generate event-aligned visual context in each imagined rollout, placing greater emphasis on critical state changes that inform action generation. This shapes the visual reasoning granularity according to the underlying interaction dynamics, with detailed reasoning around task-critical events and coarser progression through connecting transitions. Furthermore, we introduce an execution validity head that identifies the valid portion of each predicted action sequence, avoiding redundant actions during chunked inference. Experiments demonstrate a 10.26 percentage point improvement in DOMINO success rate over baseline and competitive performance on RoboTwin 2.0. It also transfers from DOMINO Level 1 to Levels 2 and 3 without target-level adaptation.

1 INTRODUCTION

World-Action Models use predicted future visuals to guide action generation, but fixed temporal horizons can fragment brief task-critical interactions. This work aligns visual reasoning with interactions and adds execution-validity prediction; experiments report improved DOMINO performance and competitive RoboTwin 2.0 results.

1 INTRODUCTION

  • Fixed-interval visual imagination may split fine-grained interactions across prediction horizons, even though those brief events provide direct evidence of task-relevant state changes.Connecting motions can occupy much of a trajectory, while grasp contact confirms whether the intended robot–object transition occurred.Source: §1 · p. 1
  • Prior WAMs focus on integrating visual predictions with actions but structure visual-action horizons uniformly across the demonstration timeline.The introduction contrasts this uniform organization with adapting imagination to different interaction events.Source: §1 · p. 2
  • Event-aligned supervision organizes visual prediction around task-critical interactions, preserving fine-grained evolution there while representing transitions with fewer intermediate states.The framework aims to generate visual context that captures critical interactions under actions and supports action generation.Source: §1 · p. 2
  • An execution validity head identifies which portion of each predicted action sequence to execute, avoiding redundant actions associated with padding.Event segments can have different valid lengths when mapped to fixed-size action chunks.Source: §1 · p. 2
  • A 10.26% gain in DOMINO success rate over LingBot-VA accompanies competitive performance on RoboTwin 2.0 and transfer from DOMINO Level 1 to Levels 2 and 3 without target-level adaptation.The introduction also reports superior performance, especially on the more challenging dynamic manipulation dataset DOMINO.Source: §1 · pp. 2–3

2 RELATED WORKS

Related work covers vision-language-action policies that generate actions from current observations and instructions, and World-Action Models that add future visual prediction. These lines of work also explore reducing dense prediction and representing task-relevant temporal information more selectively.

Vision-Language-Action Models

  • VLA models primarily map current visual observations and language instructions directly to actions, with recent approaches also incorporating temporal memory or future prediction.Examples include action-token prediction, generative action models, and adaptations of pretrained representations.Source: §2 · p. 3

World-Action Models

  • WAMs couple future visual dynamics with action generation, while related approaches reduce dense video prediction or represent temporal structure with events, keyframes, or subgoals.The passage describes these methods as exploring task-relevant temporal information rather than uniformly modeling all future states.Source: §2 · p. 3

3 INTERACTION-STRUCTURED VISUAL REASONING FOR WAM

The framework organizes visual prediction around task-relevant interactions and their action-supporting visual context, and includes execution-validity prediction.

3 INTERACTION-STRUCTURED VISUAL REASONING FOR WAM

  • The framework organizes visual prediction around task-relevant interactions and the visual context needed to support corresponding actions.Source: §3 · p. 3

3.1 WORLD-ACTION MODELS

  • WAMs autoregressively interleave visual and action chunks, but trajectory-fixed chunks can split interactions or combine scattered events.Source: §3.1 · pp. 3–4

3.2 EVENT-ALIGNED VISUAL REASONING

  • Interaction events mark task-relevant relationship changes; chunks stay within event boundaries, preserve evidence at native resolution, and compress support while retaining endpoints.Source: §3.2 · pp. 4–5

3.3 EXECUTION VALIDITY PREDICTION

  • A jointly predicted validity head identifies executable action groups, skipping repeated padding while adapting executed sequence length to each interaction.Source: §3.3 · p. 6

3.4 TRAINING OBJECTIVE

  • Training combines visual flow-matching on all frames, action flow-matching on valid positions, and validity prediction weighted by 0.1.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments evaluate event-aligned visual reasoning on DOMINO and RoboTwin 2.0, test transfer and design choices, and examine execution behavior. The results favor event-aligned sampling, interaction evidence, and validity prediction.

4.2 BENCHMARK PERFORMANCE

  • 10.26 points higher DOMINO SR than LingBot-VA: the method achieves 42.83% SR and 55.92 MS, versus 32.57% SR and 45.55 MS.Table 1 reports the cross-benchmark evaluation.Source: §4.2 · p. 7, Table 1 · p. 7
  • On RoboTwin 2.0, the method obtains 93.02% SR in clean scenes and 91.58% in randomized scenes, averaging 92.30%.It outperforms the corresponding LingBot-VA backbone and performs competitively against ImageWAM and Fast-WAM.Source: §4.2 · p. 7, Table 1 · p. 7
  • Training on DOMINO Level 1 transfers to Level 2/3 without target-level adaptation, with the best SR and MS performance on the ten-task subset.Reported examples include 45.42 MS versus Fast-WAM's 31.56 on Level 2, and 24.23 MS versus PUMA's 20.16 on Level 3.Source: Table 2 · p. 7, §4.2 · p. 8

4.3 ANALYSIS OF EVENT-ALIGNED VISUAL ACTION REASONING

  • With the same action-slot budget, Event Uniform reaches 39.60% SR versus Episode Uniform's 29.17%, supporting event-aligned granularity and budget allocation.The experiment compares uniform sampling across an episode with uniform sampling within event boundaries.Source: §4.3 · p. 8
  • Preserving interaction evidence raises SR from Event Uniform's 39.60% to Event-aligned's 42.83%; Reduced Evidence reaches only 36.17% despite 26.9% more action slots.Reduced Evidence shifts sampling toward transitions and support phases, away from interaction evidence.Source: §4.3 · p. 8
  • Reduced Context reaches 37.83% SR when surrounding non-evidence context is reduced, below Event-aligned sampling with the same protected interaction evidence.The comparison indicates benefits from both interaction evidence and surrounding support.Source: §4.3 · p. 8
  • The execution validity head raises SR from 39.11% to 42.83% under Event-aligned sampling by distinguishing valid actions from the complete predicted sequence.Without the head, the model directly executes the entire predicted action sequence.Source: §4.3 · p. 9
  • On a Scan Object episode, event-focused predictions track ordered interactions and complete the task at observation 140, while LingBot-VA remains unsuccessful at observation 496.Across 738 shared successful episodes, average issued commands fall from 167.18 to 106.46 and visual-rollout requests from 6.22 to 4.49.Source: Table 3 · p. 7, §4.3 · p. 9, Figure 3 · p. 9

5 CONCLUSION

The conclusion frames interaction structure as an organizing principle for visual imagination, with detailed prediction around interactions and coarser progression through transitions. The construction appendix specifies event categories and chunking rules that preserve evidence and event boundaries.

5 CONCLUSION

  • The framework organizes visual imagination around task-relevant interactions, varying reasoning granularity with interaction dynamics and using an execution validity head to avoid redundant commands.The conclusion reports state-of-the-art success rate on DOMINO, competitive RoboTwin performance, and fewer commands and imagination requests on shared successful episodes.Source: §5 · p. 10

A DETAILS OF EVENT-ALIGNED CHUNK CONSTRUCTION

  • Demonstrations are partitioned into interaction and transition events; interaction events contain evidence frames showing a change and support motion leading into or out of it.An interaction changes the robot's relationship with an object or the environment; a transition covers motion between interactions.Source: A DETAILS OF EVENT-ALIGNED CHUNK… · p. 14, Figure A1 · p. 14
  • Resampling selects uniformly spaced frames, anchored resampling retains segment endpoints, and SplitK cuts sequences into fixed-length chunks with repetition when needed.Repeated frames carry no new observation and are marked invalid by the execution validity head.Source: A DETAILS OF EVENT-ALIGNED CHUNK… · p. 14
  • Transition events become single chunks, while interaction chunks keep every evidence frame and allocate remaining slots to leading and trailing support.When evidence fills its chunks exactly, each non-empty support becomes a separate chunk on its corresponding side of the evidence.Source: A DETAILS OF EVENT-ALIGNED CHUNK… · p. 14
  • For K = 4, the worked grasp example shows support frames filling free slots around evidence, while no chunk combines frames from the preceding reach and grasp.The example preserves the first and last grasp frames in the chunks.Source: A DETAILS OF EVENT-ALIGNED CHUNK… · p. 14

B.1 BENCHMARKS AND TRAINING DATA

The benchmark appendix reports evaluation settings and training data for DOMINO and RoboTwin 2.0. Event-based preparation retains demonstrations with successful replay and annotations, while excluding replay failures.

B.1 BENCHMARKS AND TRAINING DATA

  • DOMINO evaluation uses clean Level 1 scenes, while RoboTwin 2.0 evaluation covers clean and randomized environments.Table B1 summarizes benchmark settings and training schedules.Source: §B.1 · p. 15, Table B1 · p. 15
  • DOMINO provides 1,750 source demonstrations, of which 1,693 are retained after excluding 57 simulator replay failures.The retained DOMINO set has successful event annotations.Source: §B.1 · p. 15
  • RoboTwin 2.0 provides 27,500 source demonstrations, with 24,844 retained after replay failures are excluded.The retained set consists of 2,465 clean and 22,379 randomized demonstrations with successful event annotations.Source: §B.1 · p. 15

B.2 MODEL IMPLEMENTATION

The model implementation uses a pretrained LingBot-VA base, with shared action-token features supporting an execution head. Training varies temporal grouping, while inference uses a fixed grouping and the stated solver settings.

B.2 MODEL IMPLEMENTATION

  • The model starts from pretrained LingBot-VA and uses 10,000 DOMINO updates or 50,000 RoboTwin updates at learning rate 10^-5.The DOMINO run uses effective batch size 32; RoboTwin uses effective batch size 64.Source: §B.2 · p. 15
  • The execution head applies a shared-parameter linear projection to final action-token features, adding 3,073 parameters and no Transformer blocks.For d = 3072, the head produces one execution logit per action slot.Source: §B.2 · pp. 15–16
  • Training randomizes temporal groups over C ∈ {1, 2, 3, 4}, while inference fixes C = 2; each future group corresponds to 16 action slots.The first inference request has nominal capacity 16 future action slots and subsequent requests have 32.Source: §B.2 · p. 16
  • Evaluation uses three video-token solver steps to s = 0.6 and ten action-token steps to s = 1.0, with video and action guidance scales of 5.0 and 1.0.Training applies noise augmentation with probability p = 0.5.Source: §B.2 · p. 16

B.3 BASELINES TRAINING CONFIGURATIONS

The appendix specifies training configurations for ImageWAM and Fast-WAM baselines and identifies additional results, cost breakdowns, and qualitative examples.

B.3 BASELINES TRAINING CONFIGURATIONS

  • ImageWAM and Fast-WAM share a 10−4 learning rate, cosine schedule, 0.01 weight decay, 1.0 gradient-norm cap, bf16 precision, and seed 42.Their effective batch sizes are 384 for ImageWAM and 128 for Fast-WAM.Source: §B.3 · p. 16
  • ImageWAM uses FLUX.2-klein-base-4B, 16-action windows, endpoint-frame supervision, and three camera views packed into 288 × 256 inputs.Its per-worker batch size is 12 with four accumulation steps; visual and action loss weights are 0.5 and 1.0.Source: §B.3 · p. 16
  • Fast-WAM uses Wan2.2-TI2V-5B, 32-action windows, a 4:1 action–video sampling ratio, and three camera views packed into 384 × 320 inputs.Its per-worker batch size is 16 without gradient accumulation.Source: §B.3 · p. 16

C ADDITIONAL RESULTS

  • The additional-results appendix covers real-world experiments, task-level results, cost breakdowns, and qualitative examples.These topics are identified as appendix contents, not reported here as findings.Source: C ADDITIONAL RESULTS · p. 16

C.1 REAL WORLD EXPERIMENTS

The real-world evaluation uses a Unitree G1 humanoid with Dex1-1 grippers on three bimanual manipulation tasks. It reports success rate across those tasks.

C.1 REAL WORLD EXPERIMENTS

  • The evaluation covers tennis-ball insertion, sequential placement of blue and green cups, and drawer opening, cube placement, and closing.Experiments use a Unitree G1 humanoid equipped with Unitree Dex1-1 grippers.Source: §C.1 · p. 16
  • The policy uses three RGB camera views and predicts 16-dimensional actions comprising 14 arm-joint targets and two gripper commands.Table C1 reports success rate for the three real-world tasks.Source: §C.1 · p. 17, Table C1 · p. 17
  • Each task has a separately fine-tuned policy, trained for 4k steps from 45, 44, and 90 teleoperated demonstrations, respectively.Each method is evaluated over 25 independent real-robot rollouts per task.Source: §C.1 · p. 17

C.2 COMPLETE TASK-LEVEL RESULTS

On DOMINO Level 1, the method outperforms LingBot-VA on aggregate success rate and manipulation score. It also transfers to Levels 2 and 3 without target-level adaptation, and reports gains without robot pretraining.

C.2 COMPLETE TASK-LEVEL RESULTS

  • 42.83% SR and 55.92 MS on DOMINO Level 1 exceed LingBot-VA’s 32.57% SR and 45.55 MS.Table C2 provides the task-level breakdown of this comparison.Source: §C.2 · p. 17, Table C2 · p. 18
  • Without target-level adaptation, Level 2 SR rises from 16.9% to 29.4% and Level 3 SR from 6.8% to 9.7%.The policies are trained on Level 1; Level 3 completion remains challenging at the reported absolute SR.Source: §C.2 · p. 17

C.3 WITHOUT ROBOT PRETRAINING

  • Without robot pretraining, the method reaches 37.43% SR and 52.07 MS, versus LingBot-VA’s 20.83% SR and 36.57 MS.The comparison initializes both methods directly from Wan2.2-TI2V-5B weights.Source: §C.3 · pp. 17–18

C.4 TRAINING-SEQUENCE COMPOSITION

These appendix sections document sampling and execution analyses, qualitative visualizations, and an offline annotation pipeline that converts task specifications and demonstration evidence into event-aligned training targets.

C.4 TRAINING-SEQUENCE COMPOSITION

  • Sampling slots count every sampled trajectory position, including repeats, and classify positions by whether they fall inside an event evidence phase.Percentages use each sampling-configuration row’s own total.Source: §C.4 · p. 18, Table C5 · p. 20

C.5 EXECUTION COST BREAKDOWN

  • The execution comparison uses 738 episodes where both methods succeed, counting issued action commands and completed visual-rollout requests.Per-task mean action-command counts are compared with percentage reductions relative to the baseline.Source: §C.5 · p. 19, Table C6 · p. 20

C.6 ADDITIONAL VISUALIZATIONS OF EVENT-ALIGNED VISUAL REASONING

  • The visualizations show predicted states and real observations; the reported chunks focus imagination on one transition or interaction event.The predicted states describe anticipated scene evolution, while observations show the realized interaction.Source: §C.6 · pp. 19–20, Figure C2 · p. 21, Figure C3 · p. 22

D EVENT ANNOTATION FOR EVENT-ALIGNED VISUAL REASONING

  • The annotation pipeline applies reviewed task specifications to demonstrations, localizes interactions, and passes event and phase annotations to chunk construction.Specifications cover 35 DOMINO and 50 RoboTwin 2.0 tasks; the annotation inputs include simulator states, contact reports, task predicates, and expert routines.Source: D EVENT ANNOTATION FOR EVENT-ALIGNED… · p. 24, §D.1 · p. 24

D.2 EVENT LOCALIZATION AND TEMPORAL ANNOTATION

  • Selected detectors localize acquisitions, releases, functional contacts, task-specific state changes, and recorded task motions.Their evidence windows follow saved contact, stability, motion, or expert-execution records rather than a fixed-radius neighborhood; event records include onset, effect, and confirmation anchors.Source: §D.2 · pp. 24–25

D.3 FROM TEMPORAL ANNOTATIONS TO CHUNK CONSTRUCTION

  • Event boundaries determine chunk membership, while evidence and support phases organize visual sampling within each interaction.Actions remain paired with selected visual frames, and repeated positions supervise the execution validity head.Source: §D.3 · pp. 25–26

D.4 ANNOTATION CHECKS AND DATASET SELECTION

  • Replay alignment and annotation consistency are checked; 1,693 of 1,750 DOMINO demonstrations are retained, while 57 with replay failures are excluded.Recorded discrepancies and insufficient evidence are retained in the annotation manifest.Source: §D.4 · p. 26

D.5 PROMPT FOR TASK-LEVEL EVENT SPECIFICATIONS

  • The task-specification prompt directs analysis of scene construction, expert procedures, and success checks, while requiring faithful predicate logic and forbidding invented timestamps.Optional specification fields are included only when supported by the source and detector contract.Source: §D.5 · pp. 26–27

D.6 TASK-SPECIFIC DETECTOR INVENTORY

The inventory maps DOMINO and RoboTwin 2.0 tasks to detector subtypes, using shared prefixes to identify detector categories.

D.6 TASK-SPECIFIC DETECTOR INVENTORY

  • The inventory maps DOMINO tasks and RoboTwin 2.0 tasks to detector subtypes across three tables.Table D3 covers DOMINO; Tables D4 and D5 cover RoboTwin 2.0.Source: §D.6 · p. 27, Table D3 · p. 28, Table D4 · p. 29
  • S, F, and G denote state transition, functional contact, and grasp detector categories, respectively.A state-transition label can mark a task-relevant interaction.Source: §D.6 · p. 27, Table D3 · p. 28, Table D4 · p. 29
  • Generic object-grasp, release, and gripper-change labels are shared across tasks and omitted from individual rows.Additional grasp entries specify bimanual, handle, or handover-receive detections; receives pair with release or handover release.Source: §D.6 · p. 27

Related papers

Loading 2610.09427v1…