PaperlayerView exact revision on arXiv

Source-linked AI summary

R2M-Bench: Evaluating Revisit Memory via Relative Consistency in Interactive Video World Models

Qiwen Gu, Bingjie Gao, Rui Chen, Geng Li, Jifan Li, Qishuai Wen, Li Niu, Jing Tang, Xiangxiang Chu, Junqiao Zhao

arXiv:2608.27328v1cs.CV

TL;DR

Absolute first-visit/return similarity can reflect limited motion or rendering stability rather than remembered scene identity. R2M-Bench introduces same-rollout relative calibration with gap-matched and short-range controls, and its Overall NMR aligns with human judgments while reducing motion dependence.

  • Problem

    Absolute revisit similarity does not cleanly distinguish revisit-specific consistency from generic temporal stability, because rendering, motion, and unchanged views can affect the score.

  • Method

    R2M-Bench compares each commanded-return pair with same-video gap-matched and short-range controls, reporting MemoryGain and normalized NMR across five consistency dimensions.

  • Results

    Overall NMR correlates with human consistency judgments at Spearman’s ρ = 0.547, and DreamX-World-Memo achieves the highest Overall NMR among evaluated video models.

  • Takeaways & Limitations

    Same-rollout relative calibration supports distinguishing revisit-selective consistency from generic temporal stability in interactive video world models.

  • Takeaways & Limitations

    The navigation-only benchmark excludes object interaction and deliberately evolving state, while execution errors can remain entangled with generated-world persistence.

Abstract

from arXiv · show

High similarity between first-visit and return frames does not necessarily show that a video world model remembered the scene; the intervening rollout may simply have changed very little. This ambiguity makes absolute revisit scores sensitive to rendering stability, repetitive content, and failed motion. We introduce \emph{R2M-Bench} (\textbf{R}elative \textbf{R}evisit \textbf{M}emory Benchmark), a benchmark of observable revisit-selective consistency. For every detected return, R2M-Bench compares the revisit pair with two controls from the same rollout: a gap-matched non-revisit pair that measures generic temporal stability and a short-range pair that estimates short-horizon consistency. These comparisons produce \emph{MemoryGain} (MG), the revisit advantage over the temporal baseline, and the \emph{Normalized Memory Ratio} (NMR), which normalizes this advantage by the short-to-baseline dynamic range. R2M-Bench combines 100 reference scenes with three leave-and-return trajectories to form 300 instances and evaluates appearance fidelity, scene and object identity, local geometry, and persistent state. Across seven action-conditioned video world models, Overall NMR correlates with human consistency judgments at Spearman's $ρ=0.547$ (95\% CI $[0.45,0.63]$). Its within-model correlation magnitude with generated motion is $0.072$, compared with $0.207$ for raw revisit similarity, indicating that relative calibration substantially reduces the slow-motion shortcut. DreamX-World-Memo achieves the highest Overall NMR among the evaluated video models. Together, these results support same-rollout relative calibration as a practical way to distinguish revisit-specific consistency from generic temporal stability.

1 Introduction

R2M-Bench addresses the ambiguity of absolute revisit similarity by measuring revisit-selective consistency relative to temporal controls within the same rollout. It combines a multi-axis benchmark with human-validated evaluation across action-conditioned video world models.

  • Motivation: Absolute revisit similarity can be inflated by low motion, rendering stability, or repetitive views, so it does not isolate memory.A benchmark must distinguish revisit-specific recovery from generic temporal stability under comparable rollout conditions.
  • Motivation: R2M-Bench defines revisit memory as recovering a previously visited scene and state more consistently than comparable non-revisit observations from the same video.This operational definition concerns observable behavior rather than any particular internal memory mechanism.
  • Benchmark Design: Each detected return is compared with a gap-matched baseline and a short-range reference, producing MemoryGain and normalized NMR scores.The gap-matched pair estimates ordinary temporal self-similarity, while the short-range pair estimates nearby-frame consistency.
  • Evaluation: The study evaluates seven action-conditioned video world-model variants and includes a separate 3D rendering reference.The evaluation also incorporates human validation, motion analysis, and trajectory-wise diagnostics.
  • Evaluation: Overall NMR correlates with human consistency judgments at Spearman’s ρ = 0.547, with a 95% bootstrap interval of [0.45, 0.63].Relative scoring is designed to assess consistency beyond generic temporal stability.
  • Benchmark Design: The benchmark combines 100 scenes with three return trajectories, yielding 300 instances across five dimensions of visual and state consistency.The dimensions include appearance fidelity, scene and object identity, local geometry, and persistent state.

2 Related Work

Related work spans video generation, action-driven interactive world models, long-horizon memory mechanisms, and increasingly multi-dimensional evaluation. R2M-Bench builds on these directions with explicit revisit and loop-closure testing.

  • Video generation and long-horizon rollouts: Video-generation research progressed from adversarial and autoregressive spatio-temporal models toward diffusion-based and transformer-based video synthesis.These developments target visual fidelity, temporal generation, and controllable long-video production.
  • Action-driven interactive world models: Action-driven world models generate future observations in response to camera controls, navigation commands, and other interactive inputs across robotics, driving, games, and environments.This line of work connects video synthesis with planning, control, and interactive simulation.
  • Memory mechanisms: Long interactive rollouts use context replay or retrieval, compressed memory, and persistent latent state to retain information across chunks.These mechanisms differ in how they select, compress, or maintain past information.
  • Evaluation: Existing benchmarks assess visual quality, temporal stability, motion, alignment, physics, controllability, and long-video coherence, while newer work targets revisit memory and state retention.The evaluation landscape has expanded beyond generic quality toward interactive and persistent-world behavior.
  • R2M-Bench’s position: R2M-Bench contributes explicit constructed revisit and loop-closure cases alongside relative within-video scoring and multi-axis consistency evaluation.Its three controlled return patterns include out-and-back, translation–rotation, and closed-loop trajectories.

3 R2M-Bench Data Composition

R2M-Bench constructs 300 navigation-based instances from 100 varied reference scenes and three model-agnostic leave-and-return trajectories. Prompts preserve salient scene cues, and commanded poses support revisit-pair mining.

  • Dataset composition: The benchmark pairs 100 reference scenes with three trajectory templates, producing 300 instances containing an image, prompt, navigation script, and commanded trajectory.The scene partition includes indoor, urban, natural, and abstract environments with varied object density and structure.
  • Trajectory construction: Trajectory construction uses model-agnostic translations and yaw rotations, converted into discrete actions or camera deltas according to each model interface.The resulting commanded pose sequence records the intended trajectory and supports later revisit-pair mining.
  • Trajectory construction: The three templates test near same-view, viewpoint-changing, and loop-closure revisits through out-and-back, translation–rotation, and closed-loop motion.Together they expose effects associated with viewpoint change, control error, autoregressive drift, and context attenuation.
  • Reference selection: Initial references are navigable scenes with recognizable layouts, stable visual anchors, and sufficient free space for plausible camera motion.The selection spans indoor, outdoor, urban, natural, and object-rich environments while excluding close-ups, blank backgrounds, overlays, severe blur, and ambiguous scale.
  • Prompt annotation: Prompts are drafted from each reference image and manually normalized to preserve persistent scene cues while removing hallucinated objects, fragile counts, transient lighting, and style-heavy wording.This prompt design retains scene category, spatial layout, major objects, materials, and distinctive landmarks.

4 Evaluation Method

R2M-Bench evaluates revisit memory through same-video relative calibration rather than absolute revisit similarity. It compares commanded returns with gap-matched and short-range controls, producing MemoryGain and Normalized Memory Ratio across complementary consistency dimensions.

  • Motivation: Absolute first-visit/revisit similarity is insufficient because rendering stability, low motion, and limited viewpoint change can produce high scores without demonstrating memory.The benchmark therefore calibrates revisit consistency against temporal references from the same generated rollout.
  • Frame-Pair Construction: For each revisit pair, a gap-matched non-revisit pair estimates ordinary long-gap self-similarity, while a short-range pair measures local temporal consistency.Baseline candidates come from the same video with comparable temporal gaps and exclude neighborhoods around revisit endpoints.
  • Frame-Pair Construction: Revisit pairs are temporally separated frames whose commanded positions and yaw angles return to the same region and viewing direction.Pairs must also satisfy a minimum temporal separation, with adaptive pose thresholds and closest-pose matching for duplicate candidates.
  • Relative Metrics: MemoryGain measures revisit advantage over the same-rollout long-gap baseline, partially canceling globally shared effects such as style, sharpness, conservative rendering, and low motion.Positive MemoryGain indicates calibrated revisit advantage beyond generic temporal self-similarity, not necessarily a causal memory effect.
  • Relative Metrics: Normalized Memory Ratio divides revisit advantage by the short-to-baseline dynamic range, with zero indicating no baseline excess and one reaching short-range consistency.Because small dynamic ranges can amplify NMR, the paper interprets NMR jointly with MemoryGain and dynamic range.
  • Consistency Dimensions: The evaluation covers appearance fidelity, scene identity, object identity, local geometry, and persistent state through complementary pairwise tests.Appearance uses PSNR, SSIM, and LPIPS; scene identity uses DINOv2, BoQ, and MutualVPR, while object identity re-detects earlier object concepts in the paired frame.

5 Experiments

Experiments evaluate seven action-conditioned video world models across calibrated revisit metrics, trajectories, consistency families, and motion controls. DreamX-World-Memo leads Overall NMR, while relative scores align with human judgments and reduce—but do not eliminate—motion-related shortcut effects.

  • Evaluation protocol: Seven action-conditioned video world model variants are evaluated with chunked autoregressive inference across three trajectory templates.Rollouts contain approximately 481 frames after removing static leading and trailing frames.
  • Benchmark results: 0.706 Overall NMR makes DreamX-World-Memo the highest-ranked evaluated video model, followed by HY-WorldPlay at 0.485, Matrix-Game 3.0 at 0.403, and Lyra-2 at 0.310.DreamX-World-Memo also has the largest MG on 8 of 11 displayed metrics and the largest NMR on 9 of 11.
  • Limitations: The retrieval-based ranking is associative because model scale, training data, generation procedure, and inference also differ.This scope boundary prevents isolating retrieval as the cause of the observed ranking.
  • Benchmark results: DreamX-World-Memo leads broadly, but its profile contracts sharply on closed-loop trajectories, while other models retain narrower strengths across object, scene, appearance, or geometry axes.No single appearance, semantic, or geometric metric adequately represents revisit consistency.
  • Motion dependence: Relative calibration reduces motion sensitivity: family-level correlations are −0.207 for Raw and −0.271 for the temporal baseline, versus 0.111 for MG and 0.072 for NMR.MG and NMR remain less motion-sensitive rather than motion-invariant, so nearly static or action-invalid outputs require separate inspection.

6 Conclusion

R2M-Bench measures revisit-selective consistency by comparing commanded revisits with calibrated same-rollout controls. Video-level human judgments align with the benchmark, while motion diagnostics show reduced dependence after relative calibration.

  • R2M-Bench measures whether models are selectively more consistent at commanded revisits than at gap-matched non-target moments in the same rollout.
  • Overall NMR correlates with video-level human consistency judgments, and optical-flow diagnostics show substantially reduced motion dependence after relative calibration.

7 Limitations

The benchmark evaluates observable revisit-selective consistency rather than internal memory mechanisms, and its findings have important causal and scope limitations. Execution errors, metric biases, and navigation-only evaluation constrain interpretation.

  • R2M-Bench evaluates observable revisit-selective consistency rather than identifying an internal memory mechanism.
  • Retrieval-related findings are associative rather than causal, because model scale, training data, generation procedure, and inference also differ.
  • Revisit mining relies on commanded poses and model-specific action interfaces, leaving execution errors entangled with generated-world persistence.
  • The navigation-only scope excludes object interaction and deliberately evolving state, motivating realized-trajectory evaluation and broader interactive tasks.

A Implementation and Evaluation Details

The appendix documents metric implementations, the dynamic-range normalization used by NMR, and the analyses supporting the main validation tables. It also describes controlled diagnostics and family-wise rank comparisons.

  • The appendix specifies metric implementations and the vision-language rubric before explaining NMR’s dynamic-range normalization.
  • It provides the complete human and motion analyses underlying compact main-text validation tables.
  • Controlled diagnostics and family-wise rank comparisons follow the validation analyses.

A.1 Metric Implementation Details

The evaluation averages pairwise appearance, identity, geometry, and persistent-state metrics across calibrated pair sets before computing MemoryGain and NMR. NMR enables cross-metric aggregation but can become unstable when its dynamic-range denominator is small.

  • All metrics are computed on individual frame pairs, averaged over revisit, baseline, and short-range sets, then used to compute MemoryGain and NMR.
  • Appearance Fidelity: PSNR and SSIM are higher-is-better, whereas LPIPS is lower-is-better and direction-flipped for MemoryGain and NMR.
  • Scene Identity Preservation: Scene identity uses DINOv2, BoQ, and MutualVPR descriptors compared by cosine similarity, with all three scores higher-is-better.
  • Object Identity: Object identity uses pairwise re-detection, masked DINOv2 crop similarity, GroundingDINO confidence, and CLIP-based semantic matching.
  • Local Geometric Correspondence: Local geometry combines SuperPoint and LightGlue correspondences with robust fundamental-matrix estimation using OpenCV RANSAC.
  • Persistent State: The persistent-state rubric judges shared local scene content while discounting viewpoint, occlusion, lighting, exposure, and rendering-noise differences.
  • Aggregation: NMR normalizes MemoryGain by the short-range dynamic range, enabling cross-scale aggregation but becoming unstable when that denominator is small.

C.1 Protocol

The protocol validates benchmark scores against human judgments while accounting for motion, camera control, trajectory, and clustering uncertainty. It uses correlations and bootstrap intervals to assess whether revisit metrics track perceived consistency.

  • Human evaluation: 210 videos were evaluated, spanning seven models, three trajectory templates, and ten rollouts per model-template combination.Six annotators rated revisit consistency, generated motion, and camera control on three-point ordinal scales.
  • Human evaluation: Videos were action-valid when mean camera-control and motion ratings were each at least 1.5.Inter-rater reliability was modest for consistency and motion, but higher for camera control.
  • Correlation analysis: Correlations were aggregated by Fisher’s z transformation after equal weighting across the five metric families.This prevents families with more metrics from dominating the aggregate correlation.
  • Correlation analysis: Partial correlations rank-residualized scores and human consistency against camera control, motion, and trajectory identity.The procedure does not remove unmeasured differences in realized viewpoint or action execution.
  • Uncertainty: Confidence intervals used 1,000-replicate percentile cluster bootstraps keyed by trajectory and reference scene.A conservative sensitivity analysis additionally resampled the seven model identities as clusters.

C.3 Results and Interpretation

Relative scores remain positively aligned with human consistency after controlling for motion-related variables, and this alignment spans all five metric families. Raw similarity has the strongest unadjusted correlation but also stronger motion dependence.

  • Overall validation: Raw has the largest unadjusted correlation at 0.594, but its motion correlation is −0.336 versus −0.053 for MG, −0.119 for NMR, and −0.151 for Overall NMR.This indicates that raw similarity benefits more from slowly moving videos.
  • Overall validation: After controlling for camera control, motion, and trajectory, NMR correlates with consistency at 0.149 and Overall NMR at 0.106.Their 95% confidence intervals are [0.05, 0.25] and [0.02, 0.26], respectively.
  • Overall validation: On the action-valid subset, NMR and Overall NMR remain positively correlated with consistency at 0.369 and 0.409.These results support perceptual alignment without implying that relative scores must exceed Raw in unadjusted correlation.
  • Metric-family results: NMR correlations are positive for Scene Identity, Appearance Fidelity, Local Geometry, Object Identity, and Persistent State.The strongest reported NMR correlations are 0.618 for Scene Identity and 0.499 for Appearance Fidelity.
  • Uncertainty: Seven-model clustering produces wider intervals, but every lower bound remains above zero.The authors treat this as a conservative sensitivity check rather than the primary uncertainty estimate.

D.1 Full Motion-Correlation Results

Metric-level analyses show that relative calibration substantially reduces motion dependence while preserving revisit-selective differences. Controlled perturbations further indicate that MG and NMR are more stable than absolute similarity under shared quality changes.

  • Metric examples: BoQ changes from ρ = −0.342 for Raw and −0.417 for the temporal baseline to 0.062 for NMR.This example illustrates attenuation of motion dependence for a metric-level result.
  • Motion dependence: Median motion-correlation magnitude falls from 0.242 for Raw and 0.293 for the temporal baseline to 0.118 for MG and 0.078 for NMR.Only 2/11 NMR correlations have confidence intervals excluding zero, compared with 9/11 for Raw and 10/11 for the temporal baseline.
  • Motion dependence: NMR is not motion-invariant because MutualVPR and RANSAC inliers retain significant positive residual correlations.Relative calibration removes much of the generic advantage of slow or conservative videos rather than eliminating all motion effects.
  • Controlled confounds: Across natural motion groups, SSIM falls from 0.507 to 0.452, while MG remains positive for every displayed metric and motion group.The MG trend need not be monotonic because the temporal baseline also changes with motion.
  • Controlled confounds: Under blur, downsample–upsample, and compression, SSIM MG remains within 0.045–0.053 and NMR within 0.33–0.37.LPIPS MG remains within 0.089–0.096, while DINO MG and NMR remain within 0.071–0.073 and 0.22.
  • Ranking analysis: Family-wise ranks average model ranks within each evaluation family, with Raw, MG, and NMR reported in that order.LPIPS Raw-score ranks are oriented so that higher is better before ranking.
Loading 2608.27328v1…