Source-linked AI summary
IMPACT: Attention Is the Interaction Map for Scalable Interaction-Aware World Model Training
Rongze Tang, Jianjie Fang, Zhaolu Wang, Ziyou Wang, Xvyuan Liu, Haisheng Su, Xin Zhang, Wei Wu, Chen Gao, Yong Li, Zhibo Chen
TL;DR
World models still struggle with physically plausible interactions, and existing dense external representations are costly to obtain. IMPACT reallocates denoising supervision using calibrated manipulated-object cross-attention, and consistently outperforms MSE-trained baselines across manipulation settings.
Problem
World models still struggle with physically plausible interactions, while dense external representations used to constrain generation require costly auxiliary estimators or manual annotations.
Method
IMPACT calibrates manipulated-object cross-attention with detached local prediction errors to form an interaction map that reweights denoising supervision.
Results
IMPACT consistently outperforms uniformly supervised MSE training and strong baselines across robot-arm and human-hand manipulation, control signals, and DiT backbones.
Takeaways & Limitations
The framework improves interaction fidelity, physical plausibility, and visual quality without external spatial signals or inference-time changes.
Abstract
from arXiv · showhide
World models have made remarkable progress in action-conditioned future prediction for embodied agents, yet still struggle to model physically plausible interactions. Existing approaches address this limitation by constraining the generation process with external representations encoding motion, geometry, or semantics. Obtaining these spatiotemporally dense representations typically requires auxiliary estimators or manual annotations, limiting training scalability. We instead revisit the training objective and identify a supervision-allocation mismatch under the globally averaged mean squared error (MSE) denoising objective: prevalent static content dominates the optimization signal, leaving sparse dynamic-object regions critical to interaction generation disproportionately under-supervised. Motivated by this observation, we introduce IMPACT, a scalable Interaction-aware Model training framework with Prior-guided Attention Calibration and Targeting. IMPACT uses cross-attention associated with manipulated-object tokens as an internal spatiotemporal prior for action-conditioned changes. It samples candidate regions from this prior, calibrates them with detached local prediction errors to construct an interaction map, and uses the map to reweight denoising supervision, requiring neither external representations nor inference-time modifications. Extensive experiments on robot-arm and human-hand manipulation, spanning diverse control modalities and DiT backbones, show that IMPACT consistently outperforms the corresponding MSE-trained baselines, improving interaction fidelity, physical plausibility, and visual quality.
Introduction
World models still struggle with physically plausible interactions, while conventional remedies rely on costly dense external representations. IMPACT instead targets sparse interaction regions by recalibrating internal attention and reallocating denoising supervision.
- World models can generate action-conditioned futures but still exhibit object deformation, discontinuous motion, weak action coupling, and inconsistent contact.
- Dense motion and geometry representations constrain generation but require costly auxiliary estimators or manual annotations, limiting scalability.
- Globally averaged MSE weights regions by spatial extent, allowing prevalent static content to dominate supervision over sparse dynamic interaction regions.
- IMPACT uses manipulated-object cross-attention as an internal prior, calibrating sampled proposals with detached local prediction errors into an interaction map.
- IMPACT reweights denoising supervision toward interaction regions without external dense representations or inference-time changes, using ADS and IWS within training.
- Across robotic-arm and human-hand manipulation, IMPACT surpasses uniformly supervised MSE training and improves interaction fidelity and physical plausibility.
Related Work
Recent world models inherit strong visual and temporal representations from pretrained video backbones, but these capabilities alone do not suffice for embodied simulation. Related approaches therefore add dense motion or geometric priors to constrain interaction generation.
- Pretrained video-generation backbones provide rich visual-appearance and temporal-dynamics representations for high-fidelity prediction and coherent motion modeling.
- External-prior methods use dense optical flow, point trajectories, depth maps, or reconstructed 3D structure to emphasize dynamic regions and constrain generation.
Method
IMPACT reallocates denoising supervision toward sparse interaction regions using manipulated-object cross-attention as an internal prior, without external spatial representations. It samples and error-calibrates candidate regions into an interaction map, then applies spatially weighted training with gradient-decoupled optimization.
- Preliminaries: IMPACT builds on latent video diffusion transformers conditioned on language, reference observations, and control signals such as hand poses or robot trajectories.
- Preliminaries: Uniformly averaged flow-matching error weights spatiotemporal positions equally, causing small interaction regions to receive no extra emphasis despite rapid motion and action-conditioned changes.
- Object-Token Grounding: IMPACT grounds the manipulated object in the instruction and aggregates its cross-attention across selected DiT layers and heads into an object-conditioned proposal map without external visual models or annotations.
- Forward: Attention Distribution Sampling: ADS samples K candidate regions around the detached, tempered attention distribution using smooth coarse-grid perturbations, then converts each into a fixed-area binary mask for comparison.Coarse-grid noise produces spatially coherent variations, while fixed-area masks prevent region size from dominating candidate scores.
- Forward: Attention Distribution Sampling: Detached mean local prediction errors determine candidate weights, combining attention-based localization with current regional prediction difficulty to form the interaction map.The ADS pipeline uses no gradient tracking, preventing direct optimization of the proposal distribution or candidate weights through the regional objective.
- Backward: Interaction-Weighted Supervision: IWS assigns positions weights from 1 to γ according to the interaction map, preserves supervision over the full field, and normalizes by total weight to maintain loss scale.The main experiments use γ = 5.
- Backward: Gradient-Decoupled Optimization: Gradient routing trains cross-attention parameters with the global objective and remaining DiT parameters with interaction-weighted supervision, preventing the regional objective from optimizing its own localization signal.
Experiments
IMPACT is evaluated across robot-arm and human-hand manipulation using multiple backbones, control settings, benchmarks, and baselines. It consistently improves interaction, visual, and physical-quality metrics over corresponding MSE-trained models.
- Evaluation setup: Experiments cover WorldArena robot-arm and EgoDex human-hand manipulation using Wan 2.2 and Cosmos-Predict 2.5 backbones.Both settings use 720p training and identical optimization procedures, with IMPACT hyperparameters fixed across settings.
- Robot-arm manipulation: 62.53 EWMScore is achieved by IMPACT on Cosmos-Predict 2.5 (action), up from 55.91, the best overall result.The model also obtains the best Controllability score of 71.19 and second-best Motion Quality of 68.22.
- Robot-arm manipulation: 62.46 EWMScore is achieved on Wan 2.2-AC, improving over 58.65 by 3.81 points and 6.5%.IMPACT attains the best Physics Adherence of 55.87 and 3D Accuracy of 92.56, plus second-best Visual Quality of 60.60.
- Human-hand manipulation: On EgoDex, IMPACT cuts FVD from 366.12 to 110.94 and FID from 44.71 to 5.79 over Wan 2.2-AC on the same backbone.It also improves CLIP-Hand from 0.921 to 0.952 and Hand IoU from 0.693 to 0.772.
- Qualitative analysis: IMPACT produces sharper contact regions and more coherent object dynamics than baselines in both robot-arm and human-hand generations.Baselines more often blur or distort contact regions and drift from the target while IMPACT keeps surrounding scenes stable.
WoW
The visualizations show how IMPACT concentrates supervision on interaction regions and how its components contribute complementary gains over the AC backbone.
- ADS calibration: In the ADS visualization, darker colors represent increasing candidate errors and greater contributions to the aggregate.The map is formed by weighted aggregation of candidate regions sampled from the attention prior.
- ADS calibration: ADS converts diffuse object-conditioned attention into a concentrated interaction map around the interacting arm, gripper, and manipulated object.It samples K=8 candidate regions and weights candidates using detached local prediction errors; contact-region candidates receive higher weights than static-background candidates.
- Component ablation: IWS raises WorldArena EWMScore from 58.65 to 61.54, while adding ADS increases it by a further 0.92 to 62.46.The ablation attributes the larger direct gain to IWS and the additional calibration gain to ADS.
- Component ablation: The integrated IWS and ADS design improves EWMScore by 3.81 points overall within a single training step.The components jointly strengthen interaction-region generation.
Conclusion
IMPACT addresses supervision-allocation mismatch by converting object-conditioned cross-attention into targeted denoising supervision, without external spatial signals or inference-time changes. Experiments show consistent gains over uniform MSE training and strong baselines in robot-arm and human-hand manipulation.
- IMPACT converts object-conditioned cross-attention into targeted denoising supervision to address supervision-allocation mismatch.ADS calibrates attention proposals with detached local prediction errors, while IWS reweights training using the resulting interaction map.
- IMPACT requires neither external spatial signals nor inference-time changes.
- IMPACT shows consistent gains over uniform MSE training and strong baselines in robot-arm and human-hand manipulation.The reported gains support the framework’s effectiveness and scalability for interaction-aware world model training.
A.1 RoboTwin Data
The RoboTwin training corpus contains synchronized simulated bimanual manipulation rollouts with visual, calibration, robot-state, gripper, language, and action data. It combines clean and domain-randomized scenes across 50 diverse tasks, using fixed-length normalized training windows.
- The robot-arm corpus comprises synchronized simulated bimanual manipulation rollouts with RGB, calibration, robot-state, gripper, language, and action data.The data use a head-camera stream at 1280 × 704 resolution and 10 fps, sampled every third frame.
- Domain randomization varies distractors, textures, illumination, tabletop height, camera position, and equivalent-language instructions.The corpus includes both clean and randomized rollouts; clean backgrounds and extreme illumination each occur in 2% of randomized rollouts.
- The corpus covers 50 tasks spanning placement, stacking, ranking, tool use, switching, articulated-object manipulation, handover, and coordinated dual-arm manipulation.Training windows per task range from 99 to 28,117, producing a non-uniform task distribution.
- Each training example contains 17 consecutive frames with stride one and 17 corresponding action vectors.Boundary examples are temporally resampled or padded, and action dimension j is normalized using the first and ninety-ninth dataset percentiles.
- RGB frames and actions share temporal indices, while spatial bucket sampling and matched crop-and-resize augmentation produce 720p inputs.
A.2 EgoDex Data
The EgoDex corpus provides egocentric human-hand manipulation clips paired with camera, hand, and language information across 118 tasks. Processing preserves RGB–pose alignment and grounds manipulated objects from instructions for object-token attention.
- EgoDex contains approximately 256K egocentric clips spanning 118 tasks, paired with camera intrinsics, camera poses, articulated hand transforms, and language instructions.Each RGB clip also has a rendered hand-pose video from capture-time 3D hand tracks.
- The tasks span setup, cleanup, pick-and-place, stacking, assembly, folding, cooking, washing, insertion, writing, drawing, and tool use.
- Each source clip becomes an 81-frame training clip with shared indices for RGB and hand-pose videos.Long clips use uniformly spaced indices including endpoints, short clips repeat the final index, and contiguous source clips span 2.7 seconds.
- A language model extracts compact manipulated-object noun phrases from task instructions while excluding agents, body parts, cameras, surfaces, backgrounds, and non-interacting elements.The extraction preserves discriminative attributes such as color, material, shape, label, or container type and uses no visual or pose input.
- The annotation prompt instructs extraction from the task instruction alone, returning manipulated objects as compact noun phrases in JSON.It requires explicit objects in appearance order, removes duplicates, and avoids inferring unstated objects.
- Object phrases are aligned to tokenizer outputs, and cross-attention over their subword tokens is aggregated across heads and layers into the ADS attention map.
- EgoDex hand pose uses 21 three-dimensional joints per hand with presence masks and per-frame camera calibration.Rendered pose videos preserve RGB resolution and use an anti-aliased skeleton with specified joint and edge styling.
- RGB and pose videos retain frame-level and spatial alignment through shared indices and identical crop-and-resize transformations.
B. Implementation Details
The implementation evaluates action-conditioned diffusion-transformer backbones for robot-arm and human-hand manipulation. Action trajectories or hand-pose videos provide conditioning, while IMPACT builds candidate interaction masks from object-token cross-attention.
- B.1 Robot-Arm Models: Robot-arm experiments use Wan 2.2 TI2V 5B and Cosmos-Predict 2.5 action-conditioned diffusion transformers.Wan 2.2 has 30 blocks, hidden width d = 3072, 24 heads, and receives 17-frame RGB sequences with synchronized 14-DoF actions.
- B.1 Robot-Arm Models: The action trajectory a ∈R17×14 is flattened into 238 scalars and processed by two independent multilayer perceptrons.
- B.1 Robot-Arm Models: One action encoder output is added to the timestep embedding, while the other produces 6d = 18,432 values for adaptive normalization modulation.Action information is injected globally through timestep and adaptive-normalization pathways.
- B.2 IMPACT Configuration: IMPACT samples K = 8 candidate masks from native object-token cross-attention using Gaussian perturbations on a 5 × 8 × 12 coarse grid.The maps are upsampled, transformed with τ = 1 and κ = 0.65, and thresholded to retain the top ρ = 10% of positions.
- B.3 Human-Hand Models: The human-hand model uses the same 5B transformer and conditions on an 81-frame hand-pose video encoded by the shared Wan VAE.The VAE produces 48-channel latents with temporal and spatial compression factors of 4, 16, and 16.
- B.3 Human-Hand Models: Pose replacement is applied only to conditioned samples, with RGB and pose inputs sharing temporal indices and spatial transformations.
C.1 Evaluation Protocols and Baseline Versions
WorldArena evaluation uses 500 held-out episodes from 50 RoboTwin 2.0 tasks, while EgoDex models follow release-specific inference settings. Outputs and metric computation are standardized by resolution, frame rate, trajectory length, temporal indexing, and official sampling configurations.
- WorldArena: WorldArena evaluates 500 held-out episodes spanning 50 RoboTwin 2.0 tasks.Decoded submissions use at least 640 × 480 resolution and 24 fps.
- EgoDex: EgoDex models are evaluated using their release-specific inference settings detailed in Table 5.
- Submission Protocol: Action-conditioned submissions follow benchmark action sequences and match the corresponding reference trajectory lengths.Text-conditioned submissions contain 121 frames.
- Reference Materials: Table 4 provides representative instruction-only object-grounding examples, while Table 5 lists detailed EgoDex inference configurations.
- Metric Computation: Wan 2.2-AC and IMPACT use identical temporal indices for their 81 output frames and pose conditions.Each model follows its official sampling schedule and guidance configuration.
C.2 WorldArena Metric Definitions
WorldArena summarizes generation and task performance using 16 normalized metrics across six aggregate dimensions and the overall EWMScore. Qualitative comparisons cover diverse robot-arm and human-hand manipulation tasks under matched initial conditions and instructions.
- Overall Score: EWMScore is 100 times the arithmetic mean of the 16 normalized metrics, with all displayed entries higher-is-better.Raw metrics are normalized to [0, 1] using empirically selected boundaries.
- Metric Dimensions: WorldArena reports Visual Quality, Motion Quality, Content Consistency, Physics Adherence, 3D Accuracy, and Controllability across 16 normalized metrics.
- Results Organization: Tables 7 and 8 separate generation-quality metrics from task-oriented metrics in the full WorldArena results.Table 7 covers EWMScore and generation-quality measures; Table 8 covers physics adherence, 3D accuracy, and controllability.
- Robot-Arm Qualitative Results: Robot-arm qualitative comparisons include object placement, lifting, stacking, pressing, and food-toy placement under matched initial observations and action trajectories.
- Robot-Arm Qualitative Results: Additional robot-arm comparisons cover shoe placement, bowl stacking, service-bell pressing, and food-toy placement under matched initial observations and action trajectories.
- Human-Hand Qualitative Results: The human-hand figures compare HunyuanVideo-1.5, VACE, MimicMotion, Wan 2.2-AC, and IMPACT using matched initial observations and task instructions.
- Human-Hand Qualitative Results: Human-hand qualitative comparisons cover container and cylindrical-object lifting, drawer stacking, block adjustment, book pickup, and bowl stacking.