PaperlayerView exact revision on arXiv

Source-linked AI summary

AcrossVAM1.0: Particle World Modeling for Text-Assisted Robot Video Prediction

Yafei Zhang, Nan Wu

arXiv:2608.28491v1cs.AIcs.RO

TL;DR

Robot video prediction must model action-relevant motion without sacrificing high-frequency appearance. AcrossVAM1.0 factorizes the task into text-assisted semantic particle dynamics and causal, context-only appearance synthesis. Its evaluations show improved trajectories and motion-region prediction, while perceptual delivery, language dependence, and cross-robot transfer remain unresolved.

  • Problem

    Robot videos combine mostly static pixels with small action-determining parts and instruction-dependent trajectories, making motion reasoning and appearance preservation difficult to assess with global metrics alone.

  • Method

    AcrossVAM1.0 uses frozen semantic particles for robot geometry, a compact text-modulated Transformer for future states, and a causal decoder combining rendered motion with last-frame appearance.

  • Results

    AcrossVAM1.0 improves trajectories and motion-region prediction, while residual delivery recovers much of the static visual fidelity.

  • Takeaways & Limitations

    Explicit particle dynamics provide a low-dimensional, interpretable interface for robot video prediction without hiding motion failures behind aggregate image metrics.

  • Takeaways & Limitations

    The model remains limited by perceptual delivery, weak language dependence, inconsistent cross-robot transfer, five-frame 1282-resolution prediction, and part-mask requirements during codec construction.

Abstract

from arXiv · show

Predicting robot videos requires both precise motion reasoning and preservation of high-frequency appearance, yet monolithic pixel models entangle these objectives and often conceal their progress behind a strong last-frame baseline. We present AcrossVAM1.0, a lightweight, text-assisted video action model that factorizes future prediction into object-centric motion and dense appearance. A frozen SAM3-DLP codec decomposes four context frames into semantic particles for the robot, arm, and gripper, together with a background latent. A 0.28M-parameter spatio-temporal Transformer aligns particle identities, rolls their states forward, and is modulated by a frozen OpenCLIP instruction embedding through FiLM. A causal dual-stream decoder combines particle-rendered motion with appearance encoded exclusively from the last observed frame; a residual refiner and learned delivery mask produce five future frames without access to future appearance. On our VRS benchmark constructed from diverse real-robot trajectories, particle dynamics reduce trajectory error by 21.0\% over persistence. Across three delivery-mask seeds, AcrossVAM1.0 improves future-frame PSNR/SSIM from 19.97/0.796 to 20.573/0.8004, while raw particle generation improves motion-region PSNR from 11.89 to 13.23. The delivered model does not yet beat persistence in LPIPS, and correct-versus- shuffled language changes trajectory error by only 2.8--3.1%. We report these limitations alongside oracle, negative-control, multi-seed, and per-robot analyses. The results show that explicit particle dynamics are a promising low-dimensional interface for robot video prediction, while robust language grounding and appearance delivery remain the principal open challenges.

1 INTRODUCTION

AcrossVAM1.0 separates robot-video prediction into inspectable particle dynamics and context-only appearance synthesis. This targets the mismatch between static pixels, action-relevant articulated parts, and language-conditioned trajectory variation.

  • Robot video prediction supports planning, representation learning, and data generation by preserving geometry and appearance for downstream visual policies.
  • Static pixels dominate real robot videos, while small articulated parts determine the action and instructions can admit visually distinct trajectories.
  • Dense-latent world models couple correspondence, dynamics, and texture synthesis, whereas object-centric states make motion easier to inspect but cannot reconstruct fine texture alone.
  • AcrossVAM1.0 assigns robot geometry to semantic particles, future-state prediction to a compact Transformer, and detail restoration to a separate causal appearance stream.
  • The dense branch uses only the last context frame, preventing future motion from leaking through the appearance path.
  • The model evaluates persistence, shuffled-language, random, oracle-particle, multi-seed, and per-robot controls to measure motion benefits and language grounding.

2 RELATED WORK

Prior work spans dense video prediction, compact latent dynamics, object-centric state modeling, and frozen foundation encoders. AcrossVAM1.0 positions itself by assigning these complementary goals to distinct modules for robot video prediction.

  • Video prediction preserves object geometry, contact events, and scene appearance while learning physical regularities relevant to manipulation.
  • Dense and diffusion world models reduce pixel-prediction costs or represent uncertainty through latent variables, recurrent state, and multimodal generation.
  • Object-centric methods decompose scenes into entities, propagate slots through video, and predict future slot states separately from decoding.
  • AcrossVAM1.0 specializes object-centric dynamics to articulated robots using VRS-anchored particles, Hungarian matching, and a dense residual code for appearance.
  • Frozen segmentation and CLIP-family encoders provide semantic structure and transferable text representations, while FiLM injects conditioning into prediction networks.
  • AcrossVAM1.0 combines frozen semantic perception, a 0.28M-parameter motion Transformer, and context-only residual appearance delivery.

3 METHOD

The method factorizes future video prediction into object-centric state dynamics and causal appearance rendering. Frozen semantic particles provide motion structure, while context-only appearance and residual delivery restore visual detail without future-appearance access.

  • 3.1 PROBLEM FORMULATION: Given four context frames x1:C and instruction c, the model predicts future frames x̂C+1:T by factorizing dynamics over object-centric state z and decoding from state plus context appearance.
  • 3.2 FROZEN SEMANTIC PARTICLE CODEC: A frozen SAM3-DLP encoder extracts three semantic particles for the whole robot, arm, and gripper, plus a spatial background latent.
  • 3.2 FROZEN SEMANTIC PARTICLE CODEC: Particle geometry is the prediction target, while appearance and background support rendering; freezing the codec prevents drift toward pixel copying.
  • 3.2 FROZEN SEMANTIC PARTICLE CODEC: Unordered particle proposals require temporal correspondence, handled with Hungarian matching; fixed VRS part identifiers bypass matching in the offline cache.
  • 3.3 TEXT-ASSISTED PARTICLE DYNAMICS: The causal spatio-temporal Transformer uses geometry, velocities, depth, presence, and slot/time embeddings to autoregressively predict residual state updates.
  • 3.3 TEXT-ASSISTED PARTICLE DYNAMICS: Frozen OpenCLIP instruction embeddings modulate Transformer blocks through FiLM, initialized at zero so training begins with unconditioned dynamics.
  • 3.4 CAUSAL MOTION–APPEARANCE DECODING: A frozen renderer produces structural RGB and alpha proposals, while a last-frame-only dense encoder, residual refiner, and confidence mask deliver future appearance.
  • 3.4 CAUSAL MOTION–APPEARANCE DECODING: At inference, every decoder input depends on context, instruction, and predicted future states, so the dense branch cannot leak future appearance.

4 EXPERIMENTS

Experiments evaluate AcrossVAM1.0 on the VRS benchmark using fixed context and prediction horizons, multiple controls, delivery seeds, robot splits, and qualitative comparisons. Results show stronger motion and global fidelity than persistence in several settings, while language effects, LPIPS, and cross-robot performance remain limited.

  • Data and protocol: The VRS benchmark contains 2,746 labeled clips, with 2,526/106/105 assigned to train/validation/test; evaluation uses four context frames to predict five future frames.Frames are resized to 128 × 128, sampled in nine-frame windows at stride three, and the future-video test contains 92 clips and 460 predicted frames.
  • Metrics and selection: AcrossVAM1.0 uses PSNR, SSIM, LPIPS, and motion-region PSNR, with persistence, particle, and pixel oracles defining comparison and upper-bound references.Checkpoints and blending thresholds are selected on validation data, while test evaluation is performed once.
  • Future video prediction: Raw particle-conditioned generation improves motion PSNR by 1.34 dB over persistence, while residual delivery improves global PSNR by 0.61 dB and SSIM by 0.0044.Delivery-mask seed dispersion is 0.009 dB PSNR and 0.0002 SSIM; LPIPS remains 0.0082 worse than persistence, and the pixel-selection oracle leaves a 1.55 dB gap.
  • Baseline comparison: The evaluation separates identical-protocol measured results from a reproduction queue because public baselines do not report the fixed VRS split, four-frame context, five-frame horizon, and motion-region metric.The reproduction protocol tracks comparability and execution status rather than quantitative results.
  • Qualitative behavior: Across the held-out examples, raw generation changes articulated regions but perturbs static texture, while blend-mask errors persist near thin links, occlusion boundaries, and newly revealed pixels.These qualitative failures are consistent with the gap to the pixel-selection oracle.
  • Robustness and ablations: Particle dynamics remain more stable than appearance refinement: diagnostics raise trajectory-error reduction to 21.0%, while DensePrior and DiffPrior delivery results retain gains but rely on shared upstream checkpoints.The stronger DiffPrior delivery is stable across three mask seeds, but its motion-region PSNR is 0.98 dB above persistence and 0.36 dB below raw generation.

5 LIMITATIONS

The model remains constrained in scope, evaluation comparability, perceptual quality, language sensitivity, and cross-embodiment transfer. Delivery calibration is identified as a major bottleneck.

  • Scope: AcrossVAM1.0 predicts only five future frames at 128^2 resolution and uses part masks during codec construction.Its lightweight characterization applies to the learned predictive core, not total deployment memory, because the frozen OpenCLIP tower is large.
  • Evaluation: Global gains over persistence are modest, while public-model baselines are not retrained on the identical VRS split.This limits direct comparison with those baselines.
  • Open limitations: LPIPS remains worse than persistence, language sensitivity is limited, and cross-embodiment generalization is inconsistent.The pixel oracle also identifies delivery calibration as a major bottleneck.

6 CONCLUSION

AcrossVAM1.0 separates semantic particle dynamics from causal appearance synthesis to make robot video prediction more interpretable. Its controls and ablations show gains in motion modeling alongside unresolved perceptual delivery and language dependence.

  • 6 CONCLUSION: AcrossVAM1.0 treats robot video prediction as semantic particle dynamics plus causal appearance synthesis.The separation is intended to support interpretable video action models without hiding failures behind aggregate image metrics.
  • 6 CONCLUSION: The language-control gap is defined as the relative trajectory-error increase under shuffled rather than correct text, averaged over three seeds.The conclusion uses this control to assess language dependence.
  • 6 CONCLUSION: Component and delivery ablations show that particle dynamics improve trajectories while residual delivery recovers static visual fidelity.The table summarizes trajectory-error reduction over persistence and delivery alternatives on the same future-video test.

A REPRODUCIBILITY DETAILS

The reproducibility protocol fixes data splits, stages training and inference-safe rollout, and audits future-information leakage. Each main table reports the effective sample count for its required records.

  • Data protocol: Model selection uses 2,526 training and 106 validation clips, while 105 testmanifest clips remain excluded from checkpoint and threshold selection.Stages with complete RGB, mask, particle-cache, and text records use fewer clips.
  • Optimization stages: Training proceeds from a frozen particle codec through cached particles, rollout dynamics, dense prior, residual refiner, and inference-safe blend mask.Future prediction is autoregressive with a 64-dimensional, two-layer, four-head dynamics configuration using four context steps and three slots.
  • Leakage audit: The appearance encoder sees only xC at inference, while future RGB, masks, oracle particles, and detail codes serve only as targets or upper bounds.Text shuffling is performed within the evaluation split.

B ADDITIONAL PER-ROBOT RESULTS

Table 8 reports per-robot trajectory diagnostics, using positive values to indicate performance better than persistence.

  • B ADDITIONAL PER-ROBOT RESULTS: Per-robot trajectory diagnostics use positive values to denote improvement over persistence.The table provides the robot-specific breakdown of trajectory performance.

C BROADER IMPACT

AcrossVAM1.0 is a research prototype, not a safety controller: inaccurate rollouts may encourage unsafe actions when treated as physical guarantees.

  • Inaccurate rollouts can encourage unsafe actions if treated as physical guarantees, so real-world validation and safety constraints remain necessary.
Loading 2608.28491v1…