PaperlayerView exact revision on arXiv

Source-linked AI summary

Tether the Subject, Release the Scene: Query-Aware Memory Routing for Long-Horizon Autoregressive Video Generation

Chen Li, Peng Zhang, Hanyu Zhou, Jialong Zuo, Fei Wang, Daiguo Zhou, Nong Sang, Changxin Gao

arXiv:2608.26902v1cs.CV

TL;DR

Long-horizon autoregressive video generation can preserve subjects while anchoring backgrounds, viewpoints, and scene structure to past states. TetherMem routes historical memory differently for subject and scene queries, achieving the highest human-preference scores for overall quality and scene progression across eight baselines.

  • Problem

    Similar historical-access policies can preserve subject identity while causing backgrounds, viewpoints, and scene structure to remain confined to previously generated states.

  • Method

    TetherMem is a training-free router that uses region- and age-conditioned priors to route subject and scene queries differently in frozen video generators.

  • Results

    Across eight streaming long-video baselines, TetherMem leads estimated human-preference scores for overall quality and scene progression.

  • Takeaways & Limitations

    Normalized query-aware routing provides the strongest reported balance between scene progression and subject identity.

  • Takeaways & Limitations

    Evaluation uses ten prompts, three seeds, and one primary Wan2.1-T2V-1.3B/LongLive-RAG stack; results beyond approximately 30 seconds are qualitative examples.

Abstract

from arXiv · show

Streaming autoregressive video models generate long videos chunk by chunk, using historical memory to maintain consistency. Existing methods typically expose subject and scene queries to history through similar policies. This stabilizes the subject, but can also lock backgrounds, viewpoints, and scene structure to previously generated states even when local motion continues. We call this failure memory-anchored scene under-progression; consistency and motion metrics alone can miss it. We introduce TetherMem, a training-free, query-aware spatiotemporal memory router for frozen video generators. TetherMem separates subject and scene queries and modulates historical access with region- and age-conditioned priors: subject queries retain identity-bearing history, while scene queries reduce reliance on subject history and stale backgrounds. Across 2,400 blinded pairwise judgments from 10 annotators, TetherMem achieves the highest estimated expected preference among eight streaming long-video baselines for overall quality (0.780) and scene progression (0.769). On complete 30-second videos, it sustains changes in background, viewpoint, and scene state while preserving subject recognizability and temporal continuity.

1 Introduction

Streaming autoregressive video generation uses historical chunks to enable extendable long-horizon synthesis, but similar memory access can preserve subjects while anchoring backgrounds, viewpoints, and scene structure. TetherMem addresses this trade-off with training-free query-, region-, and age-conditioned routing for frozen video generators.

  • Problem: TetherMem identifies memory-anchored scene under-progression, where subjects remain stable and local motion persists while backgrounds, viewpoints, and scene structures fail to advance.Consistency and motion alone can therefore miss insufficient long-horizon scene progression.
  • Motivation: TetherMem targets the subject–scene trade-off by routing memory selectively instead of weakening or evolving historical states globally, as in prior approaches [1] [3] [22].Global relaxation can improve dynamics but weaken subject identity and structure because subject and scene content receive approximately the same access policy.
  • Method: Its normalized attention-logit routing redistributes historical access, avoiding post-attention Value Reweighting that can weaken subject identity and structure.Spatial routing alone is insufficient because it does not account for query differences or memory age.
  • Method: TetherMem distinguishes subject and scene queries, using region- and memory-age-conditioned priors to preserve identity-related history while reducing scene reliance on subject content and stale backgrounds.The priors act on attention logits and directly apply to a frozen video generator given frame-wise subject priors.
  • Results: Across eight streaming long-video baselines, TetherMem leads estimated overall-quality and scene-progression expected preference in blinded human comparisons.This evaluates sustained scene progress separately from subject stability and local motion.

2 Related Work

Prior long-video generation methods use historical states to limit error accumulation, including persistent memory carriers. Related cache-selection and attention-manipulation methods motivate TetherMem’s query-specific historical access for long-video generation.

  • Historical memory in long-video generation: Long-horizon autoregressive video generation commonly relies on historical states to limit error accumulation.The related work organizes existing approaches around strengthening historical-memory capability.
  • Historical memory in long-video generation: Persistent memory carriers include LongLive’s frame-level attention sinks [16] and Rolling Forcing’s initial-frame key–value states [11].These methods preserve long-range context or provide global context anchors.
  • Selective caching and memory routing: H2O and SnapKV select KV-cache tokens using attention-derived importance or query-dependent prompt features [10] [21].Their central operation is cache selection for computational efficiency.
  • Selective caching and memory routing: Diffusion editing methods manipulate cross- or self-attention to preserve spatial layout and appearance [2], whereas TetherMem targets query-specific historical access in long-video generation.This distinction separates TetherMem’s memory-routing focus from prior cache-selection and attention-manipulation approaches.

3 Problem Formulation

Autoregressive video generation conditions each chunk on prompts, noise, and historical key–value memory, whose repeated use preserves continuity but can anchor scenes to stale content. TetherMem formulates query-aware routing so subject and scene queries access history differently, jointly targeting identity preservation and scene progression.

  • 3 Problem Formulation: Each autoregressive chunk is conditioned on a text prompt, random noise, and historical memory containing cached or retrieved key–value states.Queries access historical tokens alongside local-context or persistent attention-sink tokens.
  • 3 Problem Formulation: Historical tokens support subject identity, local structure, and scene content, but repeated access can anchor backgrounds, viewpoints, and scene layouts to previously generated content.This creates memory-anchored scene under-progression despite cross-chunk continuity.
  • 3 Problem Formulation: TetherMem introduces a positive query–token routing prior that adjusts historical attention weights while leaving non-historical local context and sink tokens unchanged.Setting the prior to one everywhere recovers base attention, distinguishing TetherMem from controls over cache length, retrieval scope, frame weights, or overall memory strength.
  • 3 Problem Formulation: Subject queries need identity- and structure-bearing history, whereas scene queries should reduce dependence on stale backgrounds to evolve viewpoints, spatial relations, and environmental content.A shared historical policy therefore trades off subject stability against scene progression.
  • 3 Problem Formulation: The formulation asks whether role-specific historical access can jointly preserve subject identity and support continued scene progression.The overview routes identity memory to subject queries and favors recent backgrounds for scene queries within retrieved history.

4 TetherMem

TetherMem routes historical memory according to query role, key region, and memory age, preserving subject evidence while reducing stale background anchoring. It places routing priors inside attention normalization so memory selection changes without rescaling retrieved Values.

  • Normalized routing: TetherMem uses normalized routing to alter which historical locations queries read from while leaving historical Values unchanged.This avoids the query-dependent output-magnitude perturbations and accumulation that can arise when shared spatial weights reweight Values after attention selection.
  • Regional routing: Regional routing preserves same-region subject and background access while downweighting cross-region connections according to the current subject fraction.The fixed release budget is α = 0.25, and local and sink tokens retain unit priors.
  • Routing scaffold: The routing scaffold derives query and historical-key regions from a SAM 2 [13] subject track extracted from a Full-Memory reference video and downsampled to latent-token resolution.The mask is time-aligned: queries use their current-frame mask, while historical keys use their source-frame mask.
  • Regional routing: Subject queries retain historical subject evidence, whereas background queries favor historical backgrounds and reduce dependence on subject history to prevent layout anchoring.This creates complementary memory paths: identity support for subjects and scene-state progression for backgrounds.
  • Recency routing: Recency routing favors recent background states while retaining a small contribution from distant states, helping continuity as the scene changes.Older backgrounds may represent departed viewpoints, layouts, or completed scene states that cause stalling, returns, or repetition.

5 Experiments

Across complete 30-second videos, TetherMem achieves the strongest overall preference and scene progression while preserving subject identity and visual integrity. Ablations and attention analyses link these gains to regional and age-conditioned memory routing that reduces stale scene access without losing subject-associated history.

  • Evaluation: The evaluation uses 2,400 blinded pairwise judgments from 10 annotators on complete 30-second videos, prioritizing overall preference and scene progression.Ten prompts span stable scenes, subject-centered progression, and subject-free evolution, with three seeds each; longer 42-, 52-, and 120-second rollouts illustrate long-horizon behavior.
  • Main comparison: 0.780 overall preference and 0.769 scene progression make TetherMem the top method, exceeding MemRoPE’s 0.640 and LongLive-RAG’s 0.608 progression estimate.TetherMem also has the second-highest identity estimate (0.600) and highest visual-integrity point estimate (0.708).
  • Statistical reliability: The overall margin is 0.140 (95% CI [0.004, 0.273]) and progression margin is 0.161 (95% CI [0.076, 0.240]), with both crossed-cluster intervals excluding zero.Under stricter prompt-level clustering, only progression remains separated from zero, while identity and visual integrity remain comparable to leading baselines.
  • Ablations: Combining regional and age routing leads every human dimension and nT, while normalized routing provides the strongest progression–identity balance.Value Reweighting raises nT but lowers human preference and tail-subject consistency; regional routing protects subject-associated access, while recency routing prevents stale background recurrence.
  • Mechanism and qualitative analysis: TetherMem reduces subject-to-background attention from 67.2% to 35.4% and background-to-subject attention from 9.7% to 2.3%, while recency shifts background access toward recent memory.These realized attention changes connect the designed regional and age priors to the generator’s behavior; qualitative trajectories preserve subjects while scenes develop.

A Query-conditioned regional routing B Background-memory age profile

Query-conditioned regional routing limits cross-region historical access, while age profiling shifts background-memory attention toward recent history. Across complete trajectories and a transferred host, TetherMem improves scene progression while preserving subject consistency and tolerating coarse spatial priors.

  • A Query-conditioned regional routing: Regional routing reduces cross-region historical access, while TetherMem shifts background-memory attention toward the recent half relative to Routing w/o Age.The reported profiles are means with video-level bootstrap 95% CIs over seven videos.
  • A Query-conditioned regional routing: TetherMem maintains the central subject while more of the scene enters complete LongLive-RAG trajectories, and transfers to a Deep Forcing host within Wan2.1.Matched trajectories span 2–28 seconds for LongLive-RAG and 0–29 seconds for Deep Forcing transfer.
  • B Background-memory age profile: 0.640 progression EP, up from 0.360, with Overall, ID, and L1 also increasing under Deep Forcing transfer.The progression difference is 0.281 with crossed-cluster 95% CI [0.033, 0.522].
  • A Query-conditioned regional routing: Reference-mask and controlled-output re-extraction agree at IoU 0.431 over the rollout and 0.275 in the late window, while late human comparison reaches median coverage 0.574 and raw IoU 0.202.The human-mask comparison covers 21 subject-present frames at 28 seconds.
  • A Query-conditioned regional routing: A coarse 8 × 13 subject prior stays close to the original on nT and tail-subject consistency, whereas bounding boxes increase artifact vetoes and reduce tail-subject consistency.The comparison indicates tolerance to boundary coarsening but sensitivity to loose background inclusion.

6 Conclusion … C Qualitative Results

TetherMem addresses memory-anchored scene under-progression by routing subject and scene queries using regional and recency priors. Across approximately 30-second evaluations, it improves scene progression and overall preference while preserving subject identity, with qualitative and protocol details provided in the appendix.

  • 6 Conclusion: Uniform historical access causes memory-anchored scene under-progression by preserving local stability while suppressing prompted scene evolution.The conclusion identifies this failure as the motivation for query-aware routing.
  • 6 Conclusion: TetherMem improves scene progression and overall preference while preserving subject identity by retaining long-range subject evidence and routing background queries toward recent scene states.Realized-attention measurements link these gains to the regional and recency priors.
  • A Additional Results and Details: The appendix reports additional qualitative results, human-evaluation protocols, prompt and baseline configurations, metric definitions, implementation details, and ablations.Unless stated otherwise, experiments use the Wan2.1-T2V-1.3B/LongLive-RAG stack, ten prompts, three seeds, and approximately 30-second videos; longer rollouts appear qualitatively.
  • B Evaluation and Generation Setup: All long-video methods use matched prompts and seeds, 832×480 output, 16 fps, and approximately 30-second duration, while TetherMem and Full-Memory differ only in historical-memory routing.They share the generator, checkpoints, denoising schedule, context, attention sinks, and retrieval budget.
  • B Evaluation and Generation Setup: Subject and scene roles come from SAM 2 frame-aligned masks mapped to a 30 × 52 latent-token grid, while subject-free P05 uses a fixed center partition.The pipeline comprises reference rollout, mask extraction, and controlled generation.
  • C Qualitative Results: Figures 7 and 8 provide additional trajectories and routing-grid subject masks, while Figure 6B shows the router on a Deep Forcing host; quantitative comparison uses ten prompts and three seeds.Timestamps appear in frames, and prompt–seed metadata appears below each group.

C.1 Cross-Stack Human Evaluation … D.2 Tie-Aware Common Preference Scale

The cross-stack evaluation compares Deep Forcing with and without TetherMem using blinded pairwise judgments, while the rubric separates technical quality, identity continuity, spatial development, and prompt-directed progress. A tie-aware Davidson–Bradley–Terry model converts comparisons into common expected-preference scores, with TetherMem leading progression.

  • C.1 Cross-Stack Human Evaluation: The evaluation used 30 matched pairs from ten prompts and three seeds, judged by three blinded annotators for 90 total judgments.Six hidden repeats were excluded from method estimates, and 2,000 crossed-cluster bootstrap refits were performed over annotators and prompt–seed units.
  • C.1 Cross-Stack Human Evaluation: 0.640 versus 0.360 progression EP favored the routed host over Deep Forcing, with a difference of 0.281 and 95% CI [0.033, 0.522].Overall, ID, and L1 point estimates also favored the routed host, but their difference intervals included zero.
  • D.1 Annotation Rubric and Procedure: Progression combines spatial and scene development with progress toward the prompted action, event, spatial relation, or viewpoint change.Freezing, rollback, repetition, stationary jitter, and texture-only motion do not count as progress.
  • D.1 Annotation Rubric and Procedure: The rubric separately evaluates technical defects and collapse, subject identity and continuity, progression, prompt-directed progress, and overall complete-video preference.The identity criterion is marked N/A for the subject-free prompt.
  • D.1 Annotation Rubric and Procedure: Annotators compared anonymous side-by-side videos with randomized left–right order using better, approximately equal, worse, or unsure responses.Hidden repeats were interleaved with ordinary pairs and excluded from model fitting; the identity criterion additionally allowed N/A.
  • D.2 Tie-Aware Common Preference Scale: The Davidson tie extension of Bradley–Terry fits TetherMem against eight baselines using one tie parameter per criterion and fixes TetherMem’s strength to zero.The main comparison contains eight TetherMem–baseline edges and no direct baseline–baseline comparisons.
  • D.2 Tie-Aware Common Preference Scale: Expected-preference scores assign win, tie, and loss values of 1, 0.5, and 0 against a uniformly sampled opponent, with baseline–baseline probabilities model-predicted.This gives every row the same opponent distribution and produces a common interpretable scale.

D.3 Data and Uncertainty · D.4 Direct Observations and Sensitivity · E Detailed Evaluation Suite and Executed Configurations

TetherMem’s evaluation combines direct pairwise outcomes with uncertainty analyses over 2,400 judgments from 10 annotators. Progression advantages are robust to annotator exclusion and prompt-level resampling, while the Overall advantage is positive but less certain.

  • D.3 Data and Uncertainty: The analysis includes 2,400 judgments from 10 annotators; ties are retained, unsure and N/A responses excluded, and hidden repeats used only for quality control.One partially completed assignment is excluded at the annotator level.
  • D.3 Data and Uncertainty: Uncertainty estimation independently resamples annotators and 30 prompt–seed units across 2,000 crossed cluster bootstrap refits.L3 and L4 are fit independently, with Prog. computed as their average within each resample; Table 2(a) compares the strongest baseline for each criterion.
  • D.4 Direct Observations and Sensitivity: 0.687 for L3 and 0.654 for L4 are TetherMem’s half-tie scores against LongLive-RAG in the backbone-controlled comparison.Table 6 reports direct progression outcomes from TetherMem’s perspective, while common-scale EP scores are fitted separately.
  • D.4 Direct Observations and Sensitivity: TetherMem’s leave-one-annotator-out scores range from 0.761–0.808 for Overall and 0.748–0.800 for Prog.The corresponding ranges are 0.570–0.618 for ID and 0.683–0.727 for L1.
  • D.4 Direct Observations and Sensitivity: Hidden-repeat agreement is 0.750 for Overall and 0.722 across the five dimensions, based on 90/120 and 433/600 agreements.These repeats are used for quality control rather than as primary judgments.
  • D.4 Direct Observations and Sensitivity: 1,995 of 2,000 prompt-level bootstrap refits remain finite; the strongest-baseline differences are 0.140 for Overall and 0.161 for Prog.The 95% interval excludes zero for Prog. ([0.074, 0.258]) but not Overall ([−0.017, 0.290]).

E.1 Prompt Suite and Prompt-Level Robustness … F.1 Metric Definitions

The evaluation spans stable-scene, subject-centered, and subject-free progression prompts under matched long-video generation settings, with TetherMem consistently outperforming LongLive-RAG in direct Overall and progression scores. Metrics separately quantify image quality, net directional motion, subject consistency, and artifacts, with explicit aggregation and diagnostic conditions.

  • E.1 Prompt Suite and Prompt-Level Robustness: The prompt suite contains stable-scene controls P01–P03, subject-centered progression prompts P04 and P06–P10, and subject-free progression prompt P05, evaluated with three seeds.The seeds are 639693304, 1841301323, and 547794298.
  • E.1 Prompt Suite and Prompt-Level Robustness: Overall and Prog. exceed 0.5 on all ten prompts in the TetherMem–LongLive-RAG comparison.Direct scores use (W + 0.5T)/(W + T + L) from TetherMem’s perspective, while Prog. averages L3 and L4.
  • E.1 Prompt Suite and Prompt-Level Robustness: Pooled Overall/Prog. scores are 0.756/0.767 on stable controls, 0.867/0.822 on the subject-free prompt, and 0.612/0.597 on subject-centered progression prompts.The stable controls also yield direct Overall, L1, and ID scores of 0.756, 0.644, and 0.644.
  • E.1 Prompt Suite and Prompt-Level Robustness: TetherMem remains rank one for Overall and Prog. across all ten leave-one-prompt-out deletions.Pooled direct Overall and Prog. ranges are [0.660, 0.694] and [0.652, 0.681], while Davidson-model Overall EP and Prog. EP ranges are [0.766, 0.793] and [0.760, 0.780].
  • E.2 Generation Protocol: All methods use matched prompts, seeds, 832×480 output, 16 fps, and approximately 30-second durations, while retaining native samplers and cache or memory settings.The evaluation includes 270 videos lasting 29.625–30.000 seconds; implementation differences prevent identical noise tensors across codebases.
  • E.2 Generation Protocol: TetherMem modifies historical-memory routing while retaining the generator and rollout configuration, including local attention, retrieved-memory, sink, exclusion, and top-k retrieval settings.The TetherMem/LongLive-RAG configuration generates 120 latent frames, or 474 decoded frames, in three-frame blocks with denoising timesteps [1000, 750, 500, 250].
  • F.1 Metric Definitions: Img5 measures first-five-second VBench imaging quality, nT measures net directional motion over complete videos, and Subj5 and Tail subj. measure subject consistency in the first and final five seconds.nT averages dense flow vectors over time before taking the norm, so opposing directions cancel; it is reported in pixels per sampled transition after resizing to width 200 and does not encode identity or prompt fulfillment.
  • F.1 Metric Definitions: Art. is a unit-interval artifact-veto diagnostic in which lower is better, used only in the subject-prior sensitivity study.All diagnostics are computed per video and then averaged over the relevant prompt–seed cells.

F.2 VBench Diagnostics on the Human-Evaluation Pool … G.2 Offline Subject Priors

On the 270-video human-evaluation pool, standard VBench diagnostics correlate weakly or inconsistently with human-rated progression, while TetherMem achieves the highest progression score without leading consistency metrics. Its routing uses fixed spatial and age priors implemented through query-separated attention, with offline subject masks providing the subject prior.

  • F.2 VBench Diagnostics on the Human-Evaluation Pool: Table 1 evaluates subject and background consistency on native videos, while motion smoothness and dynamic degree use uniform 8-second trims.The diagnostics cover 270 videos from the main evaluation.
  • F.2 VBench Diagnostics on the Human-Evaluation Pool: TetherMem has the highest human progression EP, 0.440 for Causal Forcing and 0.429 for Rolling Forcing, despite not leading VBench consistency metrics.Across nine methods, correlations with human progression EP are −0.217 for subject consistency, −0.100 for background consistency, −0.171 for motion smoothness, and 0.366 for dynamic degree.
  • G.1 Routing Configuration: All priors become additive logit biases before softmax, and the implementation splits subject and background queries into two attention calls over the same complete context.Routing is shared across heads and applied to all denoising steps and 30 causal self-attention modules, with outputs scattered back without an additional gate.
  • G.1 Routing Configuration: The routing configuration sets a subject anchor prior of one, target average spatial prior α=0.25, and minimum ratio ρ_min=0.05 for all evaluation runs.The area-weighted prior reaches α whenever feasible; when the current subject-token fraction r_n≥α, γ_n saturates at the numerical floor.
  • G.1 Routing Configuration: Age routing indexes source frames by absolute recency, while the age prior applies only when background queries read background memory keys.The two cross-region directions share γ_n.
  • G.2 Offline Subject Priors: Offline subject priors track one subject with SAM 2 Hiera-Large using a box prompt, then resize and dilate masks onto the 30 × 52 latent-token grid.Current queries use the mask at the current time, historical keys use source-time masks, and dilation is reduced if the mask exceeds 0.25 of the grid.
  • G.2 Offline Subject Priors: Missing masks restore original attention, empty frames use the default spatial partition, and lost tracks reuse the last valid mask without frame-wise manual correction.P05 also uses the default partition because its prompt has no designated subject; mask alignment is assessed against the controlled rollout over time.

G.3 Reference-Prior Drift … I Limitations and Released Artifacts

The analyses show that reference-prior alignment deteriorates late, while TetherMem’s routing improves scene-aware behavior at added computational cost and remains sensitive to prior quality. Ablations favor combining regional and recency-aware routing, while the evaluation is limited in scope and accompanied by anonymized records and demonstrations.

  • G.3 Reference-Prior Drift: 0.431 proxy agreement over the full rollout falls to 0.275 after 20 seconds, while subject coverage declines from 0.583 to 0.424, confirming late reference-mask drift.The reference mask is well aligned early but becomes less precise late as trajectories diverge.
  • G.3 Reference-Prior Drift: The controlled-minus-reference difference is +0.161 for raw semantic IoU and +0.389 for route-envelope IoU, while false-positive area is 0.011 versus 0.131.Controlled-output masks align more closely with one annotator’s human masks; the comparison used one annotator and 21 target-present frames at 28 seconds.
  • G.4 Runtime Accounting: 2.81 ± 0.27× is the controlled rollout cost relative to Full-Memory, and the complete reference–segmentation–controlled pipeline costs 4.15±0.30×.Measurements profile seven videos on one NVIDIA H200 after warm-up, excluding model loading, encoding, and disk I/O; online mask updates are excluded.
  • G.5 Sensitivity to Subject-Prior Quality: The coarse prior remains close to the original in nT, 1.463 versus 1.490, and tail-subject consistency, 0.932 versus 0.934; the box prior trades higher nT for worse consistency and more vetoes.Coarse and box masks achieve mean IoU ranges of 0.57–0.76 and 0.67–0.84 against the original mask, respectively.
  • H.1 Post-Attention Design Baseline: Value Reweighting approaches TetherMem in nT but has lower human EP and tail-subject consistency because it lacks query splitting and age-aware routing.Value Reweighting applies weights after attention using tiled current query masks, whereas TetherMem uses normalized regional and recency-aware routing.
  • H Mechanism-Focused Ablations: TetherMem has the highest point estimate on every human dimension and nT, although intervals against the strongest single-factor variant include zero, making factor ordering directional.The ablation compares removal of regional separation or age priors against Full Memory across seven prompts, two seeds, and three judgments per pair.
  • I Limitations and Released Artifacts: The study uses ten prompts, three seeds, and one primary Wan2.1-T2V-1.3B/LongLive-RAG stack; it tracks at most one subject and requires offline reference rollout and SAM 2 extraction.Deep Forcing transfer remains within the Wan2.1 family, results beyond approximately 30 seconds are qualitative, and the reference mask degrades late.
  • I Limitations and Released Artifacts: The Code and Data Supplement releases anonymized evaluation records, figure data, and 15 compressed video demonstrations, with Figure 12 documenting the blind-evaluation interface.The interface shows prompts, video pairs, L1–L4 and Overall questions, and optional diagnostics for one Table 1 task.
Loading 2608.26902v1…