PaperlayerView exact revision on arXiv

Source-linked AI summary

ActWorld: From Explorable to Interactive World Model via Action-Aware Memory

Zhexiao Xiong, Yizhi Song, Hao Kang, Qing Yan, Liming Jiang, Jenson Yang, Zhoujie Fu, Stathi Fotiadis, Angtian Wang, Zichuan Liu, Bo Liu, Yiding Yang, Xin Lu, Nathan Jacobs

arXiv:2606.17730v1cs.CV

TL;DR

Interactive world models support exploration but remain limited in real-time object interaction. ActWorld combines interaction-dense data with action-aware memory to enable navigation and object manipulation, substantially improving interaction fidelity while preserving viewpoint control.

  • Problem

    Existing interactive world models predominantly support locomotion and viewpoint control, while largely neglecting real-time object-level interaction.

  • Method

    ActWorld combines a 100K-video interaction dataset with hierarchical action-aware and persistent memory that preserves interaction-critical frames, event updates, and object identities.

  • Results

    ActWorld substantially improves interaction fidelity over navigation-centric baselines while preserving locomotion and viewpoint controllability.

  • Takeaways & Limitations

    ActWorld supports flexible navigation and rich object interaction within a single real-time framework, advancing interactive world models toward actionable environments.

  • Takeaways & Limitations

    The added memory modules are designed to preserve the pretrained baseline forward pass exactly when disabled.

Abstract

from arXiv · show

Interactive world models aim to simulate environment dynamics under real-time user actions. However, their action vocabulary is largely confined to navigation: most actions correspond to motion (e.g., walk, turn, look around), while interaction with objects in the scene (e.g., pick up plates, open doors, or trigger physical responses) is either absent, restricted to game domains, or relegated to prompt-to-full-video scenarios. The resulting worlds are visually explorable but not truly actionable. In this work, we present ActWorld, an interactive world model that extends prior navigation-centric generators to support mid-rollout object interaction within a chunk-autoregressive framework. We argue that the navigation-interaction gap stems from two bottlenecks. First, a data bottleneck: the lack of human-object interaction data with accurate, dense labels. Second, a memory bottleneck: recency-biased history compression in existing world models discards the event-transition frames that causally determine subsequent object states, leading to an action-forgetting pathology. On the data side, we construct a 100K interaction video dataset, each annotated with per-chunk captions via chain-of-thought reasoning. On the model side, we introduce a hierarchical action-aware memory design that routes history compression by interaction importance, complemented by a persistent memory bank that maintains event-update and object-identity tokens across long rollouts. Experiments show that ActWorld supports both flexible navigation and rich object interaction within a single model, substantially improving interaction fidelity over navigation-only baselines without sacrificing viewpoint control. Project page is available at https://interactwm.github.io/ActWorld.

1 Introduction

Existing interactive world models mainly support locomotion and viewpoint control, producing visually explorable but not truly actionable environments. ActWorld addresses this navigation–interaction gap with interaction-dense data, action-aware hierarchical memory, and a real-time model supporting navigation and object interaction together.

  • Problem: Existing systems predominantly condition on movement and camera inputs, neglecting object-level interaction such as picking up items or opening doors.These models allow agents to walk, turn, and navigate, but largely do not support object manipulation.
  • Problem: ActWorld identifies two bottlenecks: navigation-centric data provides little object-dynamics supervision, while temporal history compression causes action-forgetting.The proposed framework is chunk-autoregressive and designed to unify object interaction with viewpoint control.
  • Method: ActWorld’s hierarchical action-aware memory routes interaction-critical frames locally and preserves event-update and object-identity tokens persistently beyond the sliding window.The local bank amplifies important frames, while the persistent bank retains compact tokens after window eviction.
  • Data: 100K videos spanning 40 action categories are annotated with dense per-chunk captions and interaction-phase labels.The dataset and annotation pipeline target the missing object-interaction coverage and fine-grained temporal supervision in existing world-model datasets.
  • Results: ActWorld jointly supports flexible navigation and rich object interaction in real time, substantially improving interaction fidelity while preserving locomotion and viewpoint controllability.The model combines action-aware memory and the interaction data pipeline with a dual-branch camera conditioning module and is evaluated on I-Bench.

2 Related Works

Related work addresses long-horizon video generation, real-time sampling efficiency, and action-conditioned world modeling. Despite progress, existing world models still result in a navigation–interaction gap.

  • Long Video Generation: Bidirectional video diffusion models preclude streaming generation and real-time interaction, while causal autoregressive variants suffer from exposure bias as errors compound.Recent work narrows this train–test gap through asymmetric distillation, video-level supervision, and rollout-aware training.
  • Real-Time Video Generation: Multi-step sampling remains the main throughput bottleneck in causal video generation.Distribution matching and consistency distillation reduce teacher sampling steps, while INT8 attention, sequence parallelism, and lightweight VAE decoder distillation further reduce per-frame latency.
  • World Models: World models learn environment dynamics from data and simulate them in response to actions, advancing in autonomous driving and robotics.Existing approaches are broadly 3D-based, synthesizing explorable scene geometry, or video-based, generating future observations directly as video.
  • World Models: Early video-based world models demonstrated feasibility in constrained, single-domain settings but suffered from short horizons and resulted in the navigation–interaction gap.The supplied passage identifies the gap as the resulting limitation of these earlier systems.

3 Method

ActWorld uses chunk-autoregressive generation conditioned on observations, actions, and camera controls, combining localized captions, decoupled camera conditioning, and action-aware memory. Its memory design preserves interaction-relevant events and object anchors beyond recency-based history limits.

  • Per-chunk semantic captions: Per-chunk captions replace coarse video-level captions to specify each interaction phase and reduce temporally ambiguous, repetitive conditioning.Videos are divided into non-overlapping 33-frame segments, annotated sequentially with prior-chunk context, and encoded by frozen UMT5.
  • Per-chunk semantic captions: The annotator detects observable object changes, classifies interaction across six phases, and emits evidence-grounded descriptions with structured labels for downstream memory.The six phases are approaching, reaching, contact, manipulating, completing, and post-action.
  • Decoupled camera control: Camera control combines per-pixel Plücker-ray FiLM with symbolic text templates for 81 discrete keyboard-mouse commands, while applying independent branch dropout pcam-txt=0.1.The shared PlückerFiLM weights add approximately 1.2% overhead to the 14B DiT and affect only current-chunk hidden states.
  • Hierarchical action-aware memory: Event-aware frame re-assignment and action-conditioned history amplification reshape recency-based memory so sparse contact or manipulation frames receive greater influence.Importance ranking combines a phase prior favoring contact and manipulation with recency, allowing distant contact frames into fine-grained memory while recent navigation frames may become coarse.
  • Hierarchical action-aware memory: A persistent action-aware memory bank stores transition, phase, action, and object-identity information across latent-buffer eviction, with at most Ktot=16 tokens by default.Visual summaries of triggering chunks distinguish object tokens, and zero-initialized writers preserve the pretrained forward pass before learning.

4 Experiments

ActWorld is evaluated on I-Bench, a long-horizon benchmark combining object actions with camera control, using visual, semantic, and geometric metrics. It outperforms representative baselines, with persistent event memory improving long-rollout interaction fidelity and user-study performance.

  • I-Bench: I-Bench evaluates long-horizon scripts that chain three actions with 2–3 natural-language camera primitives across 10 chunks, in both first- and third-person views.The benchmark contains 300 prompts organized into 30 sequences, using a 40-verb action vocabulary.
  • Evaluation: Evaluation spans VBench for visual quality, VLM-AJ for semantic instruction following, and KMF for geometric camera controllability.VLM-AJ reports mean score, Level-3 success, and Level ≥2 rate; KMF measures recovered SE(3) trajectory alignment with commanded controls.
  • Quantitative results: 57.8% Level-3 success more than doubles every baseline, giving ActWorld its largest margin on semantic instruction following while retaining visual quality and viewpoint control.Across Tables 1–3, ActWorld achieves the best or near-best score on each evaluation axis.
  • Qualitative results: ActWorld preserves manipulated objects and commanded action ordering across full clips, whereas navigation-centric baselines lose object tracking, fail fine-grained steps, or drift off-scene.Qualitative sequences include multi-action navigation and interaction such as Insert → Pickup, Carry → Place → Wipe, and Pour → Stir.
  • Ablation: Event Memory produces the largest ablation gain, raising subject consistency from 0.844 → 0.871 and Level-3 success from 54.0% → 57.8%.The result is attributed to persistent event and object slots that improve object-identity tracking across long rollouts.
  • User study: ActWorld ranks first against all six baselines on action following, key/mouse following, and overall quality in the user study.The widest margin is reported for action following.

5 Conclusion · Appendix

ActWorld extends navigation-centric video generators with mid-rollout object interaction in a single real-time framework. Its conclusion attributes the navigation–interaction gap primarily to recency-based memory compression and presents memory and conditioning changes that preserve viewpoint control while enabling object interaction.

  • 5 Conclusion: ActWorld extends prior navigation-centric video generators to support mid-rollout object interaction within a single real-time framework.
  • 5 Conclusion: The framework combines object interaction and viewpoint control within a real-time model.
  • 5 Conclusion: The conclusion identifies memory design, rather than architecture, as the primary source of the navigation–interaction gap.
  • 5 Conclusion: Recency-based history compression discards frames that drive object-level dynamics.
  • 5 Conclusion: ActWorld’s memory and conditioning innovations address the mismatch between compressed history and object-level dynamics.
  • 5 Conclusion: The resulting model performs object interaction while preserving keyboard-mouse viewpoint control.

A Data Generation Pipeline · B Label Handling and Zero-Initialisation Invariants

The pipeline preprocesses each video into cached, chunk-level diffusion targets, camera conditions, structured interaction annotations, and object-anchor features. Missing annotations are handled with zero or recency-only fallbacks that preserve baseline behavior exactly.

  • A Data Generation Pipeline: Offline preprocessing runs once per video and caches outputs, eliminating VLM, VAE, and DINOv3 forward passes during training.Per-step training cost is dominated by the diffusion target alone.
  • A Data Generation Pipeline: Each raw 24 fps video is split into non-overlapping 33-frame chunks (≈1.4 s), encoded by a frozen VAE, and invalid chunks are discarded.The resulting latent serves as the diffusion target during training and autoregressive output during rollout.
  • A Data Generation Pipeline: VIPE-recovered camera poses and intrinsics become translation-normalized Plücker-ray maps at the latent grid for PlückerFiLM conditioning.The poses are normalized to each chunk’s first frame.
  • A Data Generation Pipeline: A frozen GPT-5.4 vision-language model annotates five evenly spaced keyframes using action labels and prior-chunk descriptions for temporal continuity.The chain-of-thought procedure assesses motion, physical contact, action consistency, interaction phase, and evidence-grounded description.
  • A Data Generation Pipeline: The fixed annotation schema records HAS_INTERACTION, ACTION_MISMATCH, PHASE, and DESCRIPTION alongside the video-level action class for memory-module inputs.The structured labels form the per-chunk triple (yint_k, yph_k, a).
  • A Data Generation Pipeline: DINOv3-B/16 extracts candidate object anchors from five VAE-decoded keyframes, retaining the 32 patches with the largest L2 feature norms per keyframe.Using VAE-decoded frames aligns training inputs with inference, where the bank writer sees generated content.
  • B Label Handling and Zero-Initialisation Invariants: Unannotated clips emit None for missing labels; EAFR uses recency-only importance, ACHA uses a zero action embedding, and the memory-bank writer gate remains inactive.Each fallback is bit-for-bit equivalent to the baseline, enabling mixed annotated and unannotated batches.

B.1 Zero-initialisation Invariants

ActWorld’s added modules are designed to preserve the pretrained backbone’s forward pass at step 0. Zero-initialised projections, unchanged temporal embeddings, and normalized pose ensure later metric changes reflect learning rather than initialization shocks.

  • Overall invariant: At step 0, toggling any subset of new modules on a pretrained checkpoint matches the baseline loss curve to machine precision.Subsequent metric changes are thus attributable to learning rather than re-initialization shocks.
  • EAFR: EAFR preserves original RoPE time indices after bucket reassignment, changing only the compression kernel applied to each frame.Self-attention therefore receives the same temporal embedding as the time-only baseline.
  • ACHA: ACHA zero-initialises its final bottleneck layer, so α(ea)=softplus(s) at step 0 and recovers the unconditioned amplifier.The action-conditioned delta is therefore initially zero.
  • Plücker FiLM: Plücker FiLM zero-initialises its scale and shift heads and uses frame-0 pose normalization to preserve eX(ℓ)cur at step 0.This prevents attaching the branch to a trained checkpoint from disturbing the existing forward pass.
  • Action-aware memory bank: The action-aware memory bank zero-initialises its segment embedding, event-token out-projection, and object-token visual projection, making prepended bank tokens contribute exactly 0 initially.These projections remain inert until trained.

C Keyboard/Mouse Control Conditioning: implementation details

Keyboard/mouse control conditioning combines geometrically aligned Plücker-ray features with cached symbolic command embeddings. A shared FiLM branch preserves pretrained behavior at initialization while applying spatially varying control to current-chunk states.

  • Geometric conditioning: Each clip’s camera poses and intrinsics produce normalized, latent-grid-aligned 6-D Plücker rays, mapping 33 video frames to 9 latent frames.The resulting tensor is ρ∈R^B×6×T_l×H_p×W_p, with one ray representation per latent-grid pixel.
  • Geometric conditioning: Patch packing rearranges each patch’s Plücker rays into one feature vector, which W_patch projects to the transformer width.The packed representation yields one conditioning block per latent patch token.
  • Geometric conditioning: A shared PlückerFiLM module modulates only current-chunk hidden states per token, leaving cached history untouched and capturing spatially anisotropic camera motion.Zero-initialized scale and shift heads plus frame-0 pose normalization preserve the pretrained forward pass exactly at step 0; the branch adds approximately 1.2% of the 14B DiT.
  • Symbolic conditioning: Symbolic commands cover 81 keyboard/mouse combinations, pairing nine keyboard categories with nine mouse categories and mapping each pair to a short natural-language template.Examples distinguish motion and camera orientation, such as forward movement with upward-right camera tilt.
  • Symbolic conditioning: Cached frozen-UMT5 command embeddings are concatenated with per-chunk captions for DiT cross-attention, while independent 0.1 dropout discourages symbolic shortcutting.The dropout leaves caption embeddings and Plücker features intact, biasing learning toward geometric conditioning and moving FiLM heads off the identity manifold.

D Action-aware Memory Bank: implementation details

The action-aware memory bank combines typed event/object tokens, importance-based retention, and DINOv3-derived object anchors within a fixed-capacity history mechanism. Training pre-bakes visual features, while inference selectively computes anchors for interaction-relevant chunks to limit overhead.

  • Bank insertion and attention path: Bank tokens are prepended to current-chunk noise tokens, use segment embeddings with an event/object type bit, and enter DiT self-attention without changing cross-attention.The cross-attention path continues to carry the per-chunk semantic embedding.
  • Capacity and retention: The bank uses Ktot=16 slots, evicting the oldest unpinned token when full while pinning contact, manipulating, and completing tokens.Approaching, reaching, and post-action tokens recycle normally; the event writer fires at most once per chunk per transition tag.
  • Visual anchor extraction: DINOv3-B/16 ranks patch features by L2 norm, and the writer retains Kpc=3 object-anchor tokens from Kkf=5 keyframes per qualifying chunk.Features use 768-dimensional outputs from bilinearly resized 384×640 keyframes, with top patches pooled across keyframes.
  • Offline pre-baking: Training caches top-32 DINOv3 patch features and saliency scores per chunk, moving the DINOv3 forward entirely off the training critical path.Pre-baking the approximately 100K-clip set takes approximately 2.5 hours on 8×H100, with approximately 50 KB per chunk in fp16.
  • Online inference: At inference, the pipeline decodes generated chunks, samples 5 keyframes, runs one frozen DINOv3 forward, and admits top patches only under the writer gate.Approximately 30–40% of chunks trigger the writer, average overhead is below 15% of the chunk-generation budget, and pure-navigation chunks skip this pass.

E Distillation Details · F More Visualizations

The distillation pipeline uses multi-resolution flow matching and a 3-step DMD-style generator trained with teacher forcing, hierarchical backward simulation, and adversarial objectives. Additional visualizations are provided in Fig. 6.

  • E Distillation Details: The conditioning variable y bundles the chunk caption, text-camera embedding, action label, and Plücker tensor.Inference allocates (N1, N2, N3) steps across levels, bridging transitions with nearest-neighbour upsampling followed by re-noising.
  • E Distillation Details: Coarser levels process fewer tokens, substantially reducing per-chunk computational cost.This efficiency comes from the multi-resolution allocation used during inference.
  • E Distillation Details: Stage 3 distills the 50-step model into a 3-step generator Gθ with a DMD-style objective tailored to autoregressive chunks.Training uses pure teacher forcing with real history latents and generates one chunk per step rather than rolling out long sequences.
  • E Distillation Details: Backward simulation uses the same K levels as flow matching, iterating along each flow path and passing xk to level k+1.An I2V ODE warm-up initializes Gθ before Stage 3 and is the only deviation from [45].
  • E Distillation Details: Stage-3 distribution matching combines a CFG’d real score preal with a fake score pfake and multi-granularity discriminator heads D.The discriminators use a non-saturated GAN loss and approximate-R1 regulariser, while Gθ follows TTUR scheduling after several pfake updates.
  • F More Visualizations: Further visualizations are shown in Fig. 6.The figure provides additional qualitative visualization beyond the technical distillation details.

G Training Details

ActWorld is initialized from Helios with zero-initialized architectural additions, trained in an image-to-video regime, and optimized through a three-stage schedule totaling 19,000 steps.

  • Initialization: Initialization uses the publicly released Helios 14B chunk-autoregressive checkpoint, adapted from Wan2.1-14B.Plücker FiLM, ACHA, memory-bank embedding tables, and object-token MLP additions have zero-initialized final layers, preserving bit-for-bit equivalence at step 0.
  • Task: Stage-1 training fixes each clip’s first ground-truth chunk as a clean conditioning image and denoises subsequent chunks using captions, camera signals, and action-aware memory.The same image-to-video interface is retained at inference, conditioning the deployed model on a user-provided starting frame.

H High-Level Action Vocabulary

The training corpus defines 40 high-level action commands spanning human-centric manipulation and dynamic locomotion, organized into eight semantic groups. It combines first-person and third-person clips across human and non-human subjects, totaling 90,000 annotated clips.

  • Action taxonomy: 40 high-level action commands span manipulation and locomotion across eight semantic groups.The groups cover pick-and-place; opening, closing, and locking; attachment and detachment; force application and locomotion; surface and tool manipulation; discrete contact; handover; and orientation and material actions.
  • Action taxonomy: Pick-and-place includes 8 actions, while open/close and lock includes 4 actions.The listed actions are pickup, putdown, place, lift, drop, hold, grab, carry; and open, close, lock, unlock.
  • Action taxonomy: Attach/detach, force application and locomotion, and surface/tool manipulation contain 6 actions each.These groups include attach, detach, insert, remove, plug, unplug; push, pull, drag, throw, kick, drive; and wipe, swipe, stir, pour, peel, cut.
  • Dataset composition: The corpus includes 50K first-person clips, 30K third-person human-manipulation clips, and 10K third-person non-human clips, totaling 90,000 annotated clips.The first-person subset spans all 40 categories; the human-manipulation subset covers 32 categories; and the non-human subset is restricted to 8 dynamic-motion categories.
Loading 2606.17730v1…