PaperlayerView exact revision on arXiv

Source-linked AI summary

ChronoReplay: One Answer Interface Is Not Enough for Streaming Spatial Intelligence

Yihao Hu, Yitong Lin, Pan Wang, Zhihao Wen, Hang Wang, Yiwen Zhao, Deyuan Liu, Yutong Li, Mingda Wan, Yuchen Mao, Tao Lin

arXiv:2610.34283v1cs.CV

TL;DR

Streaming spatial questions require different answer forms and histories that preserve both fine-grained relations and their origins. ChronoReplay trains on native-format targets organized as chronological episodes; under matched observation access, it improves three benchmarks together, including a 13.41-point OST gain.

  • Problem

    Streaming spatial reasoning must support distinct answer interfaces while preserving fine-grained relations and the origin of each history.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    ChronoReplay balances heterogeneous spatial supervision in native formats and replays targets as chronologically ordered episodes attached to their causal prefixes.

    Source: §1 · p. 2

  • Results

    Under matched observation access, ChronoReplay improves OVO-S, OST, and UCS together, with a 13.41-point OST gain over the base and the highest three-benchmark mean among compared settings.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Interface lock-in is an observed pattern of uneven cross-readout transfer, not an isolated internal mechanism.

    Source: §6 · p. 9

    The authors have not audited high-stakes surveillance or unsafe embodied uses and do not claim suitability for surveillance or autonomous decisions.

    Source: Ethics statement · p. 10

Abstract

from arXiv · show

Training a spatial VLM to answer which side an object lies on does not require it to preserve the metric displacement that determines that answer, while supervision for a distance estimate may leave categorical relations largely unconstrained. When post-training emphasizes one readout, improvements can therefore concentrate on that interface while transfer to others remains weak or even degrades; we call this observed specialization interface lock-in. ChronoReplay addresses it by replaying heterogeneous spatial supervision as chronologically ordered episodes, retaining source-native targets, and balancing supervision across families. Under the same observation access, a single ChronoReplay checkpoint improves OVO-S-Bench, OST-Bench, and UCS-Bench together, including a 13.41-point gain on OST-Bench, and achieves the highest three-benchmark mean among the supervision settings compared here. Same-data controls further separate the ingredients of this transfer: answer serialization redistributes gains across OST and UCS, family-balanced learning improves over ordinary joint SFT and natural turn weighting, and chronological replay exceeds a shifted-order control by about 2.41 OST points across two paired seeds. Adaptive frame selection then changes evidence access while keeping the trained model fixed, providing an additional inference-time gain at lower visual cost. Streaming spatial performance depends on what distinctions supervision preserves, how heterogeneous histories are organized during learning, and what causal evidence inference makes available at test time.

1 Introduction

Streaming spatial questions require categorical, metric, and temporal answers from the same causal history, but existing methods typically supervise one interface at a time. ChronoReplay combines heterogeneous native-format supervision in chronological episodes, addressing uneven transfer across readouts.

1 Introduction

  • Single-interface recipes leave untested whether one model can answer categorical, metric, and temporal queries over the same causal history.The paper also notes that prior mixed qualitative and metric recipes use single images.Source: §1 · p. 1
  • Single-family training transfers unevenly across readouts, while targets detached from their valid chronological prefixes can become ambiguous.Categorical-only training reaches 58.03 on OVO-S but lowers OST from 50.52 to 47.48; the paper calls this empirical pattern interface lock-in.Source: §1 · p. 2
  • +13.41 OST-Bench, +7.44 UCS-Bench, and +4.91 OVO-S-Bench points are reported for ChronoReplay under identical observation access.The paper reports a matched-access cross-benchmark gain and describes the OVO-S result as numerically comparable to a published 235B reference.Source: §1 · p. 2

2 Related Work

Prior work studies causal streaming, spatial benchmarks, and reusable representations through multiple architectures and data approaches. The paper frames joint categorical, metric, and temporal supervision over one causal stream as an open multi-task question.

Streaming spatial intelligence

Learning reusable spatial representations

  • Spatial MLLMs use geometry-aware features, reconstruction inputs, and spatial instruction or reinforcement data [4] [14] [28] [36] [38] [60], alongside persistent-state representations [17] [19] [18] [25].Some prior work mixes qualitative and metric questions on single images, but the paper identifies joint categorical, metric, and temporal training over one causal stream as untested.Source: §2 · p. 3
  • The paper examines how episode chronology, target format, and weighting affect native-benchmark outcomes for shared spatial representations.It frames these as multi-task questions about competing objectives and how fine-tuning on one distribution affects another [42] [22].Source: §2 · p. 3

3 Problem Formulation

The problem is to learn one representation that supports multiple native spatial readouts while retaining the history needed to interpret them. Interfaces refine one another, and identical recent observations can require different answers when their histories have different origins.

3 Problem Formulation

  • A representation is reusable over query–interface pairs when every native target in the family can be read from the same state.Reusability is a functional requirement; histories leading to different answers must remain distinguishable.Source: §3 · p. 3
  • Categorical targets partition spatial regions, whereas metric targets resolve variation within a region, requiring the representation to retain finer distinctions.A state retaining only the coarse partition cannot support the finer readout.Source: §3 · p. 3
  • Different history origins can yield different answers despite identical recent suffixes and local displacement, so the state must preserve history origin.The model receives a causal observation history and a query with a native categorical, metric, or timestamp interface.Source: §3 · p. 3

4 ChronoReplay

ChronoReplay post-trains a VLM to support native spatial readouts while preserving the histories they depend on. It constructs chronologically ordered replay episodes and balances normalized preservation and trajectory risks.

4.1 Overview

  • ChronoReplay has two stages: construct replay data, then combine normalized preservation and trajectory risks to control their relative contributions.The overview describes retaining native targets and ordering episodes chronologically.Source: §4.1 · p. 3

4.2 Causal Replay Construction

  • Causal Replay construction pairs single-turn preservation examples with ordered trajectory supervision, marking observations beyond a query’s causal frontier as out of scope.The replay pool contains both preservation examples and chronologically ordered trajectory supervision.Source: Figure 2 · p. 4

4.2.1 Native-Format Targets

  • Native-format targets remain in their source value spaces, avoiding additional answer-space transformation when combining categorical, metric, and temporal answers.The paper presents this as retaining source-task semantics, not as evidence that native encoding is uniformly superior.Source: §4.2.1 · p. 4

4.2.2 Chronologically Aligned State Trajectories

  • Chronological replay attaches each target to its legal causal prefix, producing nested supervised states that share an episode origin and grow in history length.Training therefore exposes accumulation across turns rather than isolated snapshots; the trajectory remains implicit, with no added map, memory module, or auxiliary state loss.Source: §4.2.2 · pp. 4–5

4.3 Family-Balanced Learning

  • Family-balanced learning normalizes risks within preservation and trajectory families, then sets their relative contribution explicitly.In the main experiments, trajectory receives one-third of total mass and preservation two-thirds; answer-length normalization prevents long responses from dominating solely by token count.Source: §4.3 · p. 5

5 Experiments

Experiments test whether ChronoReplay transfers across three spatial benchmarks under fixed observation access, then examine training components and inference-time access. ChronoReplay improves all three benchmarks, while diagnostics distinguish the effects of balancing, answer format, chronology, and adaptive access.

5.1 Experimental Setup

  • All checkpoints start from Qwen3-VL-8B-Instruct, use canonical scene-disjoint test splits, and hold causal observation access fixed for model-only comparisons.The training setup combines preservation and trajectory data; adaptive access is applied separately to the same checkpoint.Source: §5.1 · p. 6

5.2 Cross-Interface Transfer (RQ1)

  • ChronoReplay raises OVO-S-Bench from 48.98 to 53.89, OST-Bench from 50.52 to 63.93 (+13.41 points), and UCS-Bench from 45.33 to 52.77 (+7.44 points) under matched observation access.The OST-Bench gain is positive across all fifteen subtypes, and UCS-Bench transfer is positive in all eight subcategories.Source: §5.2 · pp. 6–7, Table 1 · p. 6, Table 2 · p. 7

5.3 Diagnostic Analysis (RQ2, RQ3)

  • ChronoReplay achieves the highest three-benchmark mean, 56.86, versus 50.74 and 50.76 for categorical-only and trajectory-only training.The source and task coverage differ between the single-family variants, so this diagnoses specialization rather than isolating answer format as a causal factor.Source: §5.3 · p. 7
  • With joint data and training conditions held fixed, ChronoReplay exceeds ordinary token SFT and natural turn weighting in mean score and across each benchmark in three paired seeds.Mean paired gains over natural turn weighting are 0.92 ± 0.27 on OVO-S, 1.43 ± 0.19 on OST, and 1.17 ± 0.25 on UCS, in points.Source: §5.3 · p. 8
  • Answer-format results cross: NATIVE is higher on UCS by 2.92 points, while CANONICAL is higher on OST by 1.48 points.The two-benchmark mean is 58.75 for NATIVE and 58.03 for CANONICAL; the control is a lossless serialization intervention.Source: §5.3 · p. 8
  • Chronological replay exceeds SHIFTED by a mean 2.41 OST points across two seed pairs, while SHIFTED preserves episode content but changes origin and target context.The comparison supports the temporal-origin alignment construction as a whole.Source: §5.3 · p. 8

5.4 Observation Access as an Orthogonal Factor (RQ4)

  • Adaptive access adds 4.09 OVO-S points beyond the 4.91-point training gain, using 80.50 frames on average; L4 remains the hardest level.Adaptive access adds most on L1 (+8.66) and also improves L4 (+2.13).Source: §5.4 · p. 8, Table 5 · p. 9

6 Conclusion

ChronoReplay combines chronological replay, native targets, and family-balanced learning to improve three benchmarks under matched observation access. The paper also describes its AI use, ethical boundaries, and reproducibility resources.

6 Conclusion

  • ChronoReplay improves OVO-S, OST, and UCS together under matched observation access, while the full recipe's gains are not attributed to one component or every spatial subcategory.The conclusion reports a 13.41-point OST gain and a 2.41-point mean OST advantage for chronological replay over shifted replay across two seeds.Source: §6 · p. 9

AI use statement

  • AI assistants supported research and implementation tasks, while multimodal teachers handled bounded upstream tasks without defining spatial ground truth.Targets were tied to source annotations, deterministic geometry, simulator state, or audited public labels; frozen benchmark scorers produced reported results.Source: AI use statement · p. 10

Ethics statement

  • The work uses existing datasets and public benchmarks, collects no new video or indoor scans, and does not claim suitability for surveillance or safety-critical deployment.The authors report no audit of high-stakes uses and note potential inherited biases and risks involving retention of people, objects, and locations.Source: Ethics statement · p. 10

Reproducibility statement

  • The release includes code, configurations, training-selection manifests, scoring scripts, model weights, and converted annotations, but not raw media or saved predictions.The released data preserve sample order, native targets, training weights, and supervision masks.Source: Reproducibility statement · p. 10

A Theoretical Analysis

The appendix formalizes how native readouts constrain representations, how cyclic shifts can alter target context, and what family balancing controls. It also separates training effects from observation-access effects.

A Theoretical Analysis

  • The analysis is population-level with unrestricted readouts; it describes what the objective asks of representations, not whether finite-data token optimization attains it.The paper says empirical controls examine the relevance of these properties.Source: A Theoretical Analysis · p. 17

A.1 Native Readouts as Sufficiency Constraints

  • For each native interface, optimal population risk equals target uncertainty given the representation and query, including information about the causal history missing from that representation.Zero conditional entropy for a query implies perfect prediction on that query's conditional support; lossless re-encoding preserves the target-sufficiency requirement.Source: §A.1 · p. 17

A.2 Cyclic Shifts Change the Conditioning Context

  • Cyclic shifts can raise the population log-loss floor when shifted contexts leave multiple chronological histories compatible with different targets, but the ambiguity condition need not hold for every shift.The measured 2.41-point OST advantage supports preserving source order and legal causal prefixes together, without isolating a latent origin representation.Source: §A.2 · pp. 17–18

A.3 What Family Balance Does and Does Not Control

  • Family balance makes relative family influence explicit through α, but cannot restore information absent from a supervision family or control trajectory-state coverage as episode length grows.Changing family sequence counts while retaining within-family distributions leaves the balanced objective's gradient unchanged.Source: §A.3 · p. 18

B.1 Attribution Framework

  • The attribution framework separates changing model parameters under fixed observation access from changing access while holding model parameters fixed.The complete system uses the trained model with adaptive access; matched interventions examine design choices while holding other settings fixed.Source: §B.1 · p. 18

B.2 Benchmark Protocols

The evaluation uses causal observation protocols across OVO-S, OST, and UCS, with matched comparisons separating training from observation access. External systems serve as references under their reported protocols.

B.2 Benchmark Protocols

  • OVO-S measures categorical reasoning under Uniform-128 access for training comparisons and adaptive access for the complete system; pooled accuracy is 55.69 for Uniform-128 and 60.88 for Adaptive.The reported family means are distinct: 53.89 for Uniform-128 and 57.98 for Adaptive.Source: §B.2 · p. 19, Table 6 · p. 19
  • OST replays conversations in original turn order, providing each turn's available observations and interaction history; its fixed disjoint-scene split contains 8,351 turns, with 7,814 scored.The evaluation reports the official taxonomy and its fifteen question types.Source: §B.2 · p. 19

B.3 Scope of Comparison

  • UCS evaluates all 8,114 questions using observations available at each question's timestamp, and compared Qwen3-VL-8B checkpoints share observation and scoring protocols.Attribution claims rely on matched Qwen3-VL-8B comparisons; training, adaptive access, and method components are evaluated through separate controlled contrasts.Source: §B.2 · p. 19, §B.3 · p. 19

C.1 Composition

ChronoReplay combines preservation and trajectory supervision, retaining native targets and chronological contexts while balancing family contributions. Matched controls report gains from the full objective, intermediate family allocation, and chronological replay, alongside crossed effects of answer serialization.

C.1 Composition

  • The preservation pool contains 30,635 categorical and 2,545 numerical examples, while the trajectory pool contains ordered episodes; native targets retain candidate semantics, units, and timestamps.The composition table describes the pool and relative loss mass at α = 0.5.Source: §C.1 · p. 20, Table 7 · p. 20

C.2 Episode Construction

  • Chronological replay gives each target its causal prefix, keeping later observations and responses outside that prefix and preserving episode-level spatial context.The episodes comprise 45,006 supervised states and 2,090 observation turns that provide context.Source: §C.2 · p. 20

C.3 Scene Separation

  • Training and evaluation are separated by physical scene identity, with overlapping ScanNet scenes withheld from OST evaluation while official question types remain included.Evaluation questions and answers do not guide replay construction, weighting, or target selection.Source: §C.3 · p. 21

D.1 Training Settings

  • The main controls start independently from the same Qwen3-VL-8B-Instruct parameters, train the language model and visual merger, and keep the visual encoder fixed.Headline scores use one random seed; the natural-turn comparison uses three paired seeds and the chronology comparison uses two.Source: §D.1 · p. 21, Table 8 · p. 22

D.2 Loss Normalization

  • Family-balanced learning further improves all three benchmarks over ordinary token SFT and natural turn weighting on the same joint replay data.ChronoReplay sets the relative family mass explicitly through α = 0.5.Source: §D.2 · pp. 22–23, Table 9 · p. 22
  • Across three paired seeds, the full objective improves all three benchmarks over natural turn weighting in every pair.The table reports sample standard deviations of paired differences; the comparison does not separately isolate every normalization level.Source: §D.2 · p. 23, Table 10 · p. 23

D.3 Relative Weight of Trajectory Supervision

  • Among tested trajectory-supervision weights, α = 0.50 scores highest on both OVO-S and OST, supporting an intermediate allocation between the two supervision families.This is a separate matched run set, and OVO-S uses adaptive observation access.Source: §D.3 · p. 23, Table 11 · p. 23

D.4 Target Interface

  • Canonical targets score 64.72 on OST and 51.34 on UCS, while native targets score 63.24 and 54.26, respectively; the formats redistribute transfer across tasks.The native variant has the higher two-benchmark mean, while canonical targets improve OST by 1.48 points and native targets improve UCS by 2.92 points.Source: §D.4 · pp. 23–24

D.6 Training-Topology Replication

The topology rerun preserves positive transfer across all three endpoints, while single-family controls show a lower three-benchmark mean than ChronoReplay. Adaptive access selects temporal coverage from question scope and uses a fixed policy with the trained model unchanged.

D.6 Training-Topology Replication

  • The topology rerun preserves positive transfer over the official base on all three endpoints and improves the model-only mean by 0.34 points.Relative to the headline checkpoint, OVO-S Uniform-128 increases by 0.21 points, UCS by 1.49, and OST decreases by 0.69; this is robustness evidence, not an independent-seed variance estimate.Source: Table 13 · p. 24, §D.6 · p. 24

D.7 Single-Family Supervision Control

  • ChronoReplay achieves the highest three-benchmark mean at 56.86, versus 50.74 for categorical-only and 50.76 for trajectory-only training.The categorical-only arm leads OVO-S but falls below the base on OST, while trajectory-only training gives little OST Overall gain; both single-family arms improve UCS over base.Source: Table 14 · p. 25, §D.7 · p. 25

E Adaptive Access: Temporal Observation Selection

  • The fixed temporal-coverage policy improves OVO-S for both the official model and ChronoReplay while holding model parameters fixed and varying inference observations.OST and UCS retain their original observation protocols.Source: E Adaptive Access: Temporal… · p. 25

E.1 Matching Observations to Query Scope

  • The selector assigns global coverage to named historical endpoints or persistent relative pose, multiscale coverage to motion or order, and local coverage to explicit current-state questions.Static layout and ambiguous temporal scope default to global coverage; question text alone determines selection, excluding choices, answers, benchmark categories, predictions, and scores.Source: §E.1 · p. 25, Table 15 · p. 25

E.2 Sampling the Causal History

  • Global sampling covers 128 positions across the available history, local sampling uses fixed temporal spacing, and multiscale sampling allocates 60%, 30%, and the remaining budget across three history ranges.The sampling procedure includes boundary and duplicate-removal conventions, with local and global sampling including the query boundary and multiscale intervals excluding their right endpoint.Source: §E.2 · p. 26

E.3 Contribution of Observation Selection

Adaptive temporal coverage improves OVO-S for both the official model and ChronoReplay, and its advantage over uniform sampling persists at a similar frame budget.

E.3 Contribution of Observation Selection

  • ChronoReplay improves from 53.89 to 57.98 on OVO-S under adaptive temporal coverage, while the official model improves from 48.98 to 54.08.Each comparison holds model parameters fixed and varies the observations presented at inference.Source: §E.3 · p. 26
  • 5.58 points separate adaptive selection at 57.98 from uniform sampling at 52.40, using 80.50 frames on average for adaptive selection.The comparison uses a similar frame budget and supports matching temporal coverage to query scope at approximately the same visual cost.Source: §E.3 · p. 26

F Qualitative Spatial Analysis

The qualitative cases illustrate historical retrieval, reference-frame reasoning, displacement, and native categorical–numerical complementarity, while also documenting a fixed-access layout regression. The appended diagnostics report cross-type relation-readout gains without corresponding native three-endpoint transfer.

F Qualitative Spatial Analysis

  • The qualitative cases examine historical state retrieval, reference-frame transformation, and accumulated displacement using identical ordered observations and questions for chronological and origin-shifted checkpoints.The figures are selected examples and do not estimate error prevalence or identify observation-selection failure.Source: F Qualitative Spatial Analysis · p. 26

F.1 Historical State Retrieval

  • The historical-state cases distinguish retrieving an earlier observer-relative state from evaluating a later location, and distinguish a first appearance from a visually similar fixture.The early white fixture in the first-appearance example is a bidet, not the toilet.Source: §F.1 · pp. 26, 28, Figure 4 · p. 27, Figure 5 · p. 29

F.2 Reference-Frame Transformation

  • The rotation case anchors a later direction judgment to an earlier facing, so its sign error differs from returning the wrong historical round.Both queries nevertheless depend on earlier observations.Source: §F.2 · p. 28, Figure 6 · p. 30

F.3 Accumulated Displacement and Historical Proximity

  • The displacement and proximity cases require tracking location change relative to a reference facing and comparing historical distances using source annotations.The displacement question concerns location change, not turning direction.Source: §F.3 · p. 28, Figure 7 · p. 31

F.4 Native Numerical Estimation and Supervision Complementarity

  • In the selected mirror episode, Joint replay retains the correct categorical answer and approaches Trajectory’s numerical accuracy, but Trajectory has the smallest absolute distance error.The annotated distance is 0.95 m; Trajectory and Joint have absolute errors of 0.15 m and 0.16 m, respectively, and tie at official score 0.70.Source: §F.4 · p. 32, Figure 8 · p. 33

F.5 A Global Layout Regression under Fixed Observations

  • Under identical Uniform-128 observations, Base selects the annotated layout option B while ChronoReplay selects D, a documented global-layout failure on this item.The disagreement is not an adaptive-versus-uniform comparison, and the case does not isolate the model’s internal cause.Source: §F.5 · p. 32, Figure 9 · p. 34

G.1 OVO-S-Bench; G.2 OST-Bench; G.3 UCS-Bench

  • The benchmark appendices identify the full OVO-S decomposition, OST aggregate and component gains, and UCS category and subcategory reporting, but these supplied passages do not state their score values.The reported OST gain spans every component, including Estimation and Temporal-localization.Source: §G.1 · p. 35, §G.2 · p. 35, §G.3 · p. 36

H Relation-Supervision Diagnostics

  • Cross-type relation supervision improves the withheld relation readout over the matched same-type control, but its native three-endpoint mean is lower.The state-path firewall raises UCS while reducing OVO-S and OST, and the reported formulation controls move the local readout without improving native transfer.Source: H Relation-Supervision Diagnostics · p. 37, Table 19 · p. 37

I Reproducibility Details

The reproducibility details specify pinned model and inference configurations, audited data and media, and contracts linking results to their inputs and scoring protocols.

I.1 Executable Configuration

  • Training starts from the pinned Qwen3-VL-8B-Instruct revision and follows the stated configuration without adapters, continuation, fusion, curriculum, reinforcement learning, or offload.Inference is deterministic, with task-specific decoding settings and no voting, prompt ensemble, or benchmark-dependent parameter choice.Source: §I.1 · p. 37, Table 20 · p. 37

I.2 Data and Media Audits

  • Training examples are bound to canonical scene identities and source annotations, with media, row, and scene manifests retained and assets audited before training.Items exceeding token or visual-pixel limits are rejected, and isolation audits bind canonical physical scenes.Source: §I.2 · p. 38

I.3 Scorer and Manifest Contracts

  • Each reported result binds model, observations, predictions, annotations, and scoring protocol, with evaluation following fixed benchmark protocols and manifests.A frozen reader extracts typed response values without inspecting task identity, reference answers, labels, or scores; the official OST scorer evaluates its designated 7,814 turns.Source: §I.3 · p. 38

Related papers

Loading 2610.34283v1…