PaperlayerView exact revision on arXiv

Source-linked AI summary

Memorizon: Training World Models Beyond Their Context Window

Tingting Liao, Xuezhi Liang, Hao Li, Guangyi Liu

arXiv:2610.00544v1cs.CV

TL;DR

Long-span supervision can relate repeated visits, but attending over the full span is costly. MEMORIZON samples long spans while limiting attention to each scored chunk’s retrieved history and local context. Retrieval improves revisit consistency, and extending the span to cover first visits adds further gains, with some image-quality cost.

  • Problem

    Training clips often omit both visits to a returning place, while lengthening attention to capture them is costly and only 3.8% of 30 s windows contain a genuine return in this corpus.

    Source: §1 · p. 1

  • Method

    MEMORIZON varies the span covered by a sample, scores only its tail, and supplies history through a shared bank formed from each scored chunk’s top-K retrieved latents.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Retrieval raises revisit consistency over a sliding window on every split; MEMORIZON leads five of eight memory columns, while CaR leads the other three.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The training span is a coverage setting: reaching first visits improves revisit consistency, but extending it beyond that does not help further and can cost image quality.

    Source: §4.3 · p. 8, §5 · p. 10

    The evaluation mainly focuses on static scenes, where a return should look the same; changing scenes require judging whether differences are appropriate.

    Source: §5 · p. 10

Abstract

from arXiv · show

Streaming world models should render a place consistently across repeated visits. Directly supervising such revisits requires training samples that capture both visits, often spanning minutes. Yet dense attention over the full span incurs quadratic costs, making long-span supervision expensive. Memorizon breaks this coupling: long spans are needed for supervision, but not for attention, since the two visits can share a forward pass without including every intervening frame. A training sample covers a span of any length but is scored only on its last $k$ chunks. Instead of tokenizing the history before them, each scored chunk retrieves its own top-$K$ latents by camera co-visibility, and the union of these requests forms a shared bank. The bank is bounded by $kK$, so the sequence stays bounded however long the span; at the shortest span the recipe is exactly conventional training. Adding the bank raises the cost of a step once; beyond that, a longer span costs little, and going from 100 to 400 s adds 12% to the step time. Against a sliding-window baseline, retrieval raises revisit consistency on every split, and a span long enough to reach the first visit of each return adds a further 24% to 30%, at some cost in image quality; beyond that span, more length no longer helps. Filling the bank from another episode lowers revisit correlation by 83%, so the model uses what it retrieves. Project page: https://tingtingliao.github.io/memorizon

1 INTRODUCTION

Long training spans can capture both visits needed to supervise a return, but attending densely over those spans is costly. MEMORIZON separates span length from attention length by scoring only a sample’s tail and supplying retrieved history through a shared bank.

1 INTRODUCTION

  • Because attention cost often limits a training clip to about tens of seconds while deployment lasts minutes, returns longer than the clip yield no sample containing both visits or loss term relating them.The shortest genuine return in the corpus spans 13.25 s; stricter definitions push it past 28 s (App. B.2).Source: §1 · p. 1
  • Longer windows scale poorly because attention cost is quadratic, while a per-chunk retrieval bank lets the span grow without tokenizing the intervening history.The shared bank is formed from each scored chunk’s own retrieval requests rather than giving every chunk the same frames.Source: §1 · p. 1
  • Each chunk reads a sliding window plus its own retrieved entries, keeping the number of attended positions bounded as the training span grows.MEMORIZON is a training recipe for long video at bounded cost, not a method for extending attention over more tokens or lengthening rollouts.Source: §1 · p. 2
  • Retrieval raises revisit consistency on every split, and spans up to 200 s raise it further, while longer spans cost some image quality.The bank levels off: at K = 6 it holds 28 entries at 100 s and 31.5 at 400 s, despite a tenfold increase in candidate pool.Source: §1 · p. 2
  • MEMORIZON samples a span that can contain both visits, scores only its last k chunks, and forms a shared bank from each scored chunk’s top-K retrieved latents.The history before the scored chunks is not tokenized; the span can vary while the sequence remains bounded.Source: §1 · pp. 1–2, Figure 1 · p. 2

2 RELATED WORK

Prior work addresses long video generation and memory by compressing history or selecting past frames, but these approaches differ in what they retain and how they train. MEMORIZON instead selects history explicitly by camera co-visibility while keeping the sequence bounded.

2 RELATED WORK

3 METHOD

MEMORIZON separates the span covered by a training sample from the sequence the transformer attends over. Each scored chunk retrieves history into a shared bank, while bounded local attention and distillation address training and inference conditions.

3.1 OVERVIEW

  • Each sample spans m chunks but scores only its last k; history grows with m without being tokenized, and the bank is bounded by k · K entries.At m = k, there is no recent block or history and the sample reduces to ordinary training.Source: §3.1 · p. 3

3.2 ATTENTION AND MEMORY BANK

  • Each scored chunk attends to the first frame, a sliding window of r+1 chunks, and its own top-K retrieved latents, keeping attended positions bounded independently of span.The bank is the union of the scored chunks’ history requests, formed before the forward pass and masked so each chunk reads only its own entries.Source: §3.2 · p. 4

3.3 RETRIEVAL CRITERION

  • 57.2% of achievable gain is reached by the mixed retrieval score at λ = 0.2, level with the best value of 57.3% at λ = 0.1.Overlap alone reaches 47.4%, pose distance 55.9%, and camera-centre ranking alone 19.8%; neither cue accounts for occlusion or motion along the viewing axis.Source: Figure 3 · p. 5, §3.3 · p. 5
  • Candidates are ranked by frustum overlap penalized by relative pose, using camera poses to estimate whether a candidate saw the place the query chunk will draw.The translation term depends on corpus scene scale, and the relative-pose penalty is used rather than elapsed time.Source: §3.3 · p. 5

3.4 DISTILLATION

  • Self Forcing and distribution-matching distillation train a four-step generator on its own rollouts, placing generated chunks back into the packed training layout for teacher and critic guidance.The method targets the exposure bias between clean training context and model-generated context at inference.Source: §3.4 · p. 5

4 EXPERIMENTS

The experiments evaluate revisit consistency across scene splits, compare MEMORIZON with open-world models, and test retrieval and span choices. Per-chunk retrieval improves revisits, while longer spans trade off image quality and stop helping once they cover first visits.

4.1 EXPERIMENTAL SETUP

  • Evaluation averages five seeds across 20 training-scene clips, 16 held-out-scene clips, and 20 web photographs; Revisit correlates return frames and Gain subtracts same-gap controls.The tables report the first 60 seconds; rollouts on held-out scenes and web photographs last 400 seconds.Source: §4.1 · p. 6

4.2 COMPARISON WITH SOTA

  • MEMORIZON leads five of eight memory columns; its Revisit is 0.365 at the starting pose and 0.447 mid-path, versus CaR's 0.350 and 0.450.CaR leads the other three memory columns; Matrix-Game 3.0 is the strongest of the other baselines.Source: §4.2 · p. 7
  • MEMORIZON's Revisit Gain is 0.230 at the starting pose and 0.329 mid-path, while the other five baselines reach at most 0.089.Gain subtracts agreement between control pairs at the same time gap; Figure 4 shows one return with the input photograph as reference.Source: §4.2 · p. 7, Figure 4 · p. 7
  • MEMORIZON and CaR alone show clear memory; HY-WorldPlay 1.5 scores higher on subject and background consistency, and four baselines score higher on imaging quality.HY-WorldPlay's scores are 0.839 and 0.908 versus MEMORIZON's 0.778 and 0.857; imaging quality reaches 0.761 versus 0.705 for MEMORIZON.Source: §4.2 · p. 7

4.3 ABLATION STUDY

  • Per-chunk top-K raises Revisit on every split after 10 s training, while widening attention without retrieval lowers web-photo Revisit on 16 of 20 photographs in all five seeds.The retrieval model accesses the whole generated history at inference; the widened window attends to earlier generated chunks.Source: §4.3 · pp. 7–8, Figure 5 · p. 9
  • One retrieval for an entire scored block lowers Revisit and Gain on every split; random, empty, or other-episode bank contents also reduce mid-path Revisit.The shared-index design lets each chunk retrieve for its own pose; temporal ordering is not a clear win and lowers imaging quality on every split.Source: §4.3 · pp. 8–9
  • At 100, 200, and 400 s, spans reach the first visit of 64%, 84%, and 93% of returning latents; Revisit peaks at 200 s and image quality falls through that span.The bank barely grows past 200 s; at 400 s Revisit falls back and image quality recovers.Source: §4.3 · p. 8
  • The 200 s model beats 10 s top-K in every return-interval bin, while the 400 s model does not extend this; returns beyond four minutes number only 30 per seed.The 100 s model is ahead in four of five bins and level at two to four minutes.Source: Figure 6 · p. 9, §4.3 · p. 9

5 DISCUSSION

Memorizon separates training-span coverage from attended context, improving revisit consistency when the span reaches earlier visits while keeping attention bounded. Its scope is static scenes, and its image-quality cost is traced to conditioning on generated frames.

5 DISCUSSION

  • Memorizon decouples training span from attended sequence: each scored chunk retrieves top-K history latents, whose union forms a shared bank.The appendix identifies separate algorithms for building and scoring a training sample and generating video chunk by chunk.Source: §5 · p. 9, §A.1 · p. 14
  • A span reaching first visits raises revisit consistency over a sliding window, but gains stop beyond 200 s and come with an image-quality cost.The discussion reports that MEMORIZON leads five of eight memory columns, with CaR leading the other three.Source: §5 · p. 10
  • The study focuses on static scenes; its Revisit measure does not judge whether a changed place should look different after a return.The discussion says the image-quality gap arises because training uses clean ground-truth context while inference uses the model’s own output.Source: §5 · p. 10

A.1 TRAINING AND INFERENCE PROCEDURES

  • Rollout attention uses the same fixed number of slots regardless of prior history, although bank selection evaluates O(n) scores for chunk n.The rollout reads the first frame, retrieved bank, recent generated chunks, and current chunk.Source: §A.1 · p. 14

A.2 THE SHARED BANK IS LOSSLESS AND BOUNDED

The shared bank preserves each scored chunk’s top-K history retrieval while keeping its size and attention bounded independently of history length. The appendix also defines the camera-frustum overlap cue used to rank candidates.

A.2 THE SHARED BANK IS LOSSLESS AND BOUNDED

  • The shared bank is lossless: each scored chunk reads exactly the entries it would select by top-K retrieval from the whole history.The bank is the union of each scored chunk’s history top-K set.Source: §A.2 · p. 14
  • The bank contains at most kK latents, so each scored chunk attends to at most 1 + K + rc + c positions at any span.The total sequence length is bounded by 1 + kK + rc + kc slots.Source: §A.2 · p. 15
  • Packing scored chunks together costs 1.82 slots per scored latent, versus 3.75 when each chunk runs as its own sequence.The reported measurements use the bank size sampled on 20 episodes, with 30 draws per span.Source: §A.2 · p. 15

A.3 FRUSTUM OVERLAP

  • Frustum overlap credits a candidate for query-view points that fall inside its frustum and within the chosen depth range.The appendix samples an 8 × 8 image grid at four depths; the depth range sets how far behind the camera a candidate may be and still count as having seen the place.Source: §A.3 · p. 15

A.4 RETRIEVAL CUES

The retrieval-cue appendix tests cue behavior and scale sensitivity, then examines retrieval width and bank capacity. It recommends K = 6 as a compromise that retains most achievable gain while giving each chunk several frames to read.

A.4 RETRIEVAL CUES

  • Retrieval rules are graded on 960 scored chunks from 20 episodes, with each rule’s six picks compared against the six most similar candidates.Source: §A.4 · p. 15
  • Frustum overlap cannot distinguish candidates along the viewing axis: straight-ahead motion leaves overlap at 1.000 after 1.80 units of travel.Source: §A.4 · p. 15
  • Across a hundredfold scale error, pose distance varies by 10.2 points and overlap by 13.5 when its depth range is held fixed.The corpus shares metric units scaled by one factor, which removes most of this sensitivity.Source: §A.4 · p. 16, Table 3 · p. 16

A.5 RETRIEVAL WIDTH AND BANK CAPACITY

  • 60.0% at K = 2, 55.4% at K = 6, and 45.9% at K = 24 are achievable-gain percentages against each width’s own ceiling.A bank cap of 64 entries comes within a point of an unbounded bank; moving from 10 to 64 gains six points.Source: §A.5 · p. 16, Table 4 · p. 16

A.6 SETTINGS

  • Each scored chunk draws its own noise level, and the flow-matching loss is computed only on scored chunks.The first frame enters as a single-frame latent, as a photograph does at inference.Source: §A.6 · p. 16

B.1 TRAINING DATA

The corpus is built from scripted Unreal Engine capture walks divided into episodes, with ground-truth camera poses and fixed latent encoding. Training, seen-scene, and unseen-scene data are separated by episode or scene.

B.1 TRAINING DATA

  • The corpus uses scripted Unreal Engine 5.8 coverage walks with frozen trajectories, no physics, fixed exposure, and no motion blur, depth of field, or overlays.Walks run at 1280 × 720 and 24 fps through a pinhole camera with a 90° horizontal field of view.Source: §B.1 · p. 16
  • Walks are split into non-overlapping 15-minute episodes, with any shorter remainder dropped.Source: §B.1 · p. 17
  • Camera poses are ground-truth camera-to-world transforms recentered on the first frame and divided by four, making one unit equal 4 m.Each latent takes the pose of the central frame of its four-frame interval.Source: §B.1 · p. 17
  • Training uses 380 episodes from 51 walks across 50 scenes; final episodes are held out for seen scenes, and four scenes are held out for the unseen split.The training set totals 95.0 hours; the seen holdout has 44 episodes and 11.0 hours.Source: §B.1 · p. 17

B.2 REVISIT STATISTICS

The corpus’s returns usually require spans far longer than 10 seconds to capture both visits. Their measured intervals are lower bounds because the corpus count omits heading.

B.2 REVISIT STATISTICS

  • A return requires nearby camera centres and evidence that the camera moved away between visits; its interval uses the latest qualifying earlier latent.The corpus count covers all latents from 380 training episodes, with one unit equal to 4 m.Source: §B.2 · pp. 17–18
  • Because corpus counts omit heading, reported intervals are lower bounds; at ε = 3.0, including heading shifts the shortest interval from 13.25 to 28.25 s.The corpus definition asks whether a training window could contain both visits, unlike the evaluation definition.Source: §B.2 · p. 18, Table 6 · p. 18
  • At τ = 0.5 and ε = 1.5, median return interval is 77.5 s; 64%, 84%, and 93% of returns fit within 100, 200, and 400 s, respectively.Figure 7 summarizes return intervals, window coverage, and scene-level medians.Source: §B.2 · p. 18, Figure 7 · p. 18
  • At ε = 3.0, shortest and median return intervals rise to 28.25 and 136 s; scene medians range from 56 to 206 s.The stricter definition lengthens both the shortest and median intervals.Source: §B.2 · p. 18, Figure 7 · p. 18
  • Only 3.8% of 30 s windows and 22% of 60 s windows contain a return, versus 47% of 100 s windows and nearly every 400 s window.Figure 7b reports the share of training windows of each length containing a return.Source: §B.2 · p. 18, Figure 7 · p. 18

C.1 EVALUATION METRICS

This section specifies how returns and their controls are selected, and how image similarity is measured. It also explains which evaluation measures are sensitive to pose alignment.

C.1 EVALUATION METRICS

  • A return requires the camera to leave by more than 1.5 units or 90°; consecutive pairs from one pass are thinned to one return.Control pairs at the same time gap differ by more than 1 unit or 30°.Source: §C.1 · p. 18
  • Pixel correlation compares flattened, mean-centered RGB frames, so spatial misalignment penalizes even a correctly drawn scene at the wrong pose.Path-following measurements are used to distinguish pose error from drawing the wrong scene.Source: §C.1 · pp. 18–19
  • DINO similarity compares normalized final-layer CLS features, which forgive small pose shifts that pixel correlation penalizes.The return similarity is reported; DINO gain is return similarity minus control similarity.Source: §C.1 · p. 19
  • After a few seconds of open-loop rollout, PSNR and LPIPS against rendered ground truth measure rollout closeness, not reconstruction.Source: §C.1 · p. 19
  • On web walks, 72% of 173 returns are fold-backs, and 87% leave the place by turning away more than 90°.At a 1-unit tolerance, 98% are fold-backs.Source: §C.1 · p. 19

C.2 ABLATION ON SEEN SCENES

The supplementary ablations examine seen-scene performance, where a 10 s retrieval-trained model can use frames much older than its training span. Long-span training adds most at returns beyond that span.

C.2 ABLATION ON SEEN SCENES

  • Table 7 evaluates the same ablation on held-out episodes from training scenes with new trajectories; Table 8 reports spreads for Table 1.The seen-scene clips come from held-out episodes of the training scenes.Source: §C.2 · p. 19, Table 7 · p. 19, Table 8 · p. 19

C.3 WHERE THE 10 S MODEL’S MEMORY COMES FROM

  • The 10 s top-K run reaches 63% to 67% of the 100 s run’s DINO gain across seen scenes, unseen scenes, and web photographs.Its DINO gains are 0.144, 0.161, and 0.166, versus 0.227, 0.240, and 0.265 for the 100 s run; its pixel Gain is 71% to 85% of that run’s.Source: §C.3 · pp. 19–20
  • The 10 s model retrieves frames from the whole generated history at inference, including frames up to 59 s old on seen clips.Retrieval is by pose, and all bank entries share one temporal index, allowing learning from younger frames to transfer to older retrieved frames.Source: §C.3 · p. 20

C.4 GAIN BY RETURN INTERVAL AND TYPE

  • At web-photo returns 10–20 s and 20–60 s apart, the 100 s run leads by about 0.09; at 8–10 s, the 10 s run is slightly ahead on every split.Starting-pose returns are less diagnostic because the first frame is always attended; mid-path returns are the cleaner memory test.Source: §C.4 · p. 20, Table 9 · p. 20

C.5 TRAINING COST

The added bank raises training cost, but increasing span from 100 to 400 seconds adds relatively little step time. The section also explains which baseline comparisons are possible and how camera following is assessed.

C.5 TRAINING COST

  • Adding the bank raises step time and peak memory by factors of 2.3 and 1.8 over the sliding window; extending span from 100 to 400 s adds 12% to step time.Table 10 reports training costs; the step-time figures include loading, retrieval, mask construction, and checkpointing.Source: Table 10 · p. 20, §C.5 · pp. 20–21
  • At 400 s, the sequence remains at its 105-slot bound and peak memory stays at 28.45 GB; direct attention over a 400 s window would hold 1,601 latents.Sharing one bank across the scored block is cheaper, but loses most of the memory.Source: §C.5 · p. 21

C.6 BASELINES NOT IN TABLE 1

  • WorldMem cannot take photographs and continuous camera paths, while Context-as-Memory cannot be run because its code and weights are unavailable.The paper compares Context-as-Memory’s design using a row that retrieves once for the scored block.Source: §C.6 · p. 21

C.7 CAMERA FOLLOWING

  • Camera following is assessed by comparing prescribed yaw changes with image shifts, using Pearson correlation and the fraction of steps turning in the right direction.The implied field of view reflects image shift per degree of prescribed yaw.Source: §C.7 · p. 21
  • MEMORIZON’s mean yaw correlation is 0.961 and implied field of view is 90.6° versus 89° for rendered frames; open-world models correlate at 0.70 to 0.88.HY-WorldPlay 1.5 follows least closely, so some of its low Revisit may reflect path following rather than memory.Source: Table 11 · p. 21, §C.7 · p. 21

C.8 BANK CONTENTS

Bank-content controls show that revisit performance depends on the retrieved frames, while image-quality loss with span is tied mainly to conditioning on generated frames.

C.8 BANK CONTENTS

  • Mid-path Revisit is 0.440 with the retrieved bank versus 0.207 with same-history random frames, 0.105 with an empty bank, and 0.073 with other-episode frames; Gain is 0.319, 0.109, 0.019, and 0.016, respectively.The test holds checkpoint, trajectory, and seed fixed while varying only the bank contents at inference.Source: §C.8 · p. 22, Table 12 · p. 22

C.10 ADDITIONAL VISUAL COMPARISONS

  • Figure 9 shows returns from MEMORIZON on six web photographs, marking the first frame and return visits in blue, orange, and green.Orange and green show each return at its first visit and on coming back; the examples meet the Revisit rule or twice its tolerance where a path has only one return.Source: Figure 9 · p. 23, §C.10 · p. 23

C.9 WHY IMAGE QUALITY FALLS WITH SPAN

  • Generated-frame conditioning accounts for image-quality loss; every model pays 2–3 times more on unseen scenes, and only the 200 s run loses clearly more.On unseen scenes, a ground-truth bank alone recovers three quarters to four fifths of the gap.Source: §C.9 · p. 23

Related papers

Loading 2610.00544v1…