Supervising the Visible State: Separating Representation from View Selection in Streaming Video

• Streaming prediction uses an observed prefix under bounded context and computation, unlike offline settings that assume the complete recording.What the model can answer depends on what it retained about history as well as what it currently sees.Source: §1 · p. 1
• Answer-level targets can leave distinctions between histories unconstrained when they share an answer, a problem termed state under-specification.The unconstrained distinctions include event count, order, and relations.Source: §1 · pp. 1–2
• The study uses one Qwen3-VL-8B backbone and measures supervision and view selection as separable levers.The recent-four-frame interface supplies the controlled comparison.Source: §1 · p. 2
• ViSS trains on 7,677 source-grounded probes of event identity, order, accumulation, relation change, and sufficiency, while TVS changes the view with the ViSS checkpoint frozen.The interventions target representation and evidence access separately; TVS adds gains at a third to a half of the frames used by exhaustive three-view selection.Source: §1 · p. 2
• Together, ViSS and TVS reach 86.33 on StreamingBench and 69.77 on OVO-Bench, +5.81 and +10.80 points over the same backbone.Strict replay attributes most of the remaining gap to a three-view oracle to a correct answer that was held and then replaced.Source: §1 · p. 2

• This work targets unconstrained temporal distinctions with source-grounded probes under a fixed recent-4 view, without adding architectural state.The model can ground only what its view exposes.Source: §2 · p. 2
• Streaming methods manage context through token compression, persistent memories, event-based retrieval, frame selection, or systems that decide what to inspect.The related-work discussion also covers streaming post-training approaches, including causal attention, multitask instruction tuning, and evolving-stream reasoning.Source: §2 · p. 2
• Figure 1 contrasts recent-frame and history-centric views with visible-state supervision and on-demand causal views.Its counting example shows that the recent view misses earlier inserts and historical sampling skips the first, while retrieving it restores the count; the example does not separate the two levers.Source: Figure 1 · p. 3
• SimpleStream [38] provides a strong recent-frame reference and motivates accessing history on demand.TVS uses a fixed family of three causal views with calibrated acquisition and commitment on the frozen ViSS checkpoint, rather than learned retrieval.Source: §2 · p. 3

3.1 Causal Streaming Prediction

• A causal streaming query arrives at an exogenous interruption time, and the relevant evidence may be recent, distributed across earlier observations, or insufficient.A causal view is a budgeted rendering of the observed prefix.Source: §3.1 · p. 3
• The prediction pipeline renders the observed prefix into a view, computes a visual-temporal representation, then produces an answer from that representation and the question.The view constructor may select, resize, and organize observations while preserving temporal order.Source: §3.1 · p. 3

3.2 State Representation and View Selection

• Visible state is the temporal content of a causal view that supports many questions about the same observed process.It is evaluated behaviorally through temporal predictions supported by the shared readout.Source: §3.2 · p. 3
• View selection determines which causal observations enter the representation, with recent-4 denoted as the fixed policy π0.An adaptive policy can acquire legal views sequentially and select among their answer proposals.Source: §3.2 · p. 3
• Adaptive views are bounded by unique source-frame and model-call budgets, and every acquired view draws only from the observed prefix.The passage specifies SKe ≤ Bf unique source frames and Ke + 1 ≤ Bc model calls.Source: §3.2 · p. 3
• The first controlled contrast measures post-training at fixed evaluation evidence, while the second changes only the view policy with learned parameters fixed.Their sum is the staged gain from public initialization to the complete system; three-view replay localizes availability and selection failures within its proposal set.Source: §3.2 · p. 4
• The contrasts are staged rather than independent because a view’s value depends on the checkpoint reading it.The study uses one 8B backbone and two benchmarks, with TVS thresholds fixed once on a source-disjoint holdout.Source: §3.2 · p. 4

4 Method

• ViSS trains shared parameters on temporally structured targets, and TVS then freezes them and varies the inference-time view.The recent-4 evaluation before TVS enables the two contrasts in the formulation.Source: §4 · p. 4

4.1 Visible-State Supervision

• Visible-State Supervision uses probes of event identity, order, accumulation, relation change, and answer sufficiency to target distinctions left unconstrained by answer-level targets.The paper argues that when targets are determined by the supplied view, separating probes require a loss-minimizing representation to preserve the distinction.Source: §4.1 · p. 4
• ViSS probes are built from source annotations or source pixels and timestamps, with checks for causal grounding and label stability.The resulting 7,677-example mixture includes 116 sufficiency examples whose answer is that the evidence is insufficient.Source: §4.1 · p. 4
• ViSS optimizes assistant-token-only masked likelihood using direct answers, excluding source metadata, teacher confidence, rationales, and hidden environment state from targets.A fresh run from the public initialization produces the trained checkpoint.Source: §4.1 · pp. 4–5

4.2 Test-Time Causal View Search

• Figure 2 depicts ViSS training a shared representation from recent-4 input, followed by frozen-checkpoint TVS acquisition and proposal selection.Its event view is acquired when the recent margin is low, and its history view when recent and event proposals disagree.Source: Figure 2 · p. 5
• TVS keeps the checkpoint fixed and expands beyond recent-4 only when more evidence is warranted.The recent view has four full-detail frames; the event view adds up to 16 earlier frames ranked by visual change, and compact history broadens the prefix at reduced resolution while retaining high-resolution recent evidence.Source: §4.2 · p. 5
• TVS activates the event view when the recent-view margin is below a threshold, then compact history when recent and event proposals disagree.A disagreeing proposal replaces the retained one only when its margin difference exceeds a calibrated threshold; thresholds are fixed using a source-video-disjoint holdout.Source: §4.2 · p. 5

5.1 Experimental Setup

• The evaluation uses full splits, official scorers, and a shared recent-4 starting view; representation comparisons vary the checkpoint, while access comparisons freeze ViSS and vary the view policy.TVS makes at most three calls and exposes at most 32 causal frames per call; no policy uses benchmark task labels or task-specific routing.Source: §5.1 · p. 6

5.2 Comparison Experiments

• On StreamingBench, ViSS adds +4.25 and TVS +1.56 points, together reaching 86.33, the highest compared score; on OVO-Bench, TVS adds +4.56 points.The OVO-Bench comparison reports TVS gains in event recall and forward reasoning, alongside declines in hallucination detection and the real-time average.Source: §5.2 · p. 6, Table 2 · p. 7, Table 1 · p. 7

5.3 Diagnostic Analysis

• With training factors strictly matched, ViSS exceeds generic temporal post-training by +1.73 OVO-Bench points while changing StreamingBench by −0.20; the OVO gap is positive across three paired seeds.The paired-seed mean gap is +1.72, with a 95% bootstrap CI of [+0.80, +2.60].Source: §5.3 · pp. 7–8
• Removing 116 sufficiency examples reduces OVO-Bench by 1.69 points and hallucination detection by 4.30, but also changes training mass, so the effect is not isolated.The comparison records 124 paired regressions against 67 repairs (McNemar p = 4.5×10−5).Source: §5.3 · p. 8
• At fixed ViSS representation, TVS nearly matches the StreamingBench history call with 43% fewer frames and exceeds OVO-Bench history and exhaustive max-confidence selection using 30.33 versus 54.33 frames.Removing the acquisition gate raises processed frames to 27.26 and 33.31 at essentially unchanged accuracy.Source: §5.3 · p. 8, Table 3 · p. 8

5.4 Why Correct Alternatives Can Still Be Lost

• Among oracle-gap cases with a correct proposal, replacement errors account for 66.9% on StreamingBench and 71.9% on OVO-Bench; this diagnosis is conditional on proposal coverage.The three-view oracle is 5.09 and 8.75 points above TVS, respectively; acquisition misses account for most of the remaining oracle-gap cases, while outright rejection is rare.Source: §5.4 · pp. 8–9, Figure 3 · p. 8

6 Conclusion

• Supervision particularly benefits hallucination detection and causal reasoning, while adaptive access particularly benefits event recall and forward reasoning; replacing a retained correct answer dominates the remaining oracle-gap selection errors.The controlled study uses one 8B backbone and two benchmarks, leaving cross-scale and cross-family transfer open.Source: §6 · p. 9
• The controlled study uses one 8B backbone and two benchmarks; cross-scale and cross-family transfer remain open.Source: §6 · p. 9

AI use statement

• A multimodal teacher proposed candidate question–answer pairs from source material, while deterministic checks filtered candidates and the authors verified code, citations, claims, and reported numbers.Source: AI use statement · p. 10

Ethics statement

• The work uses existing datasets without new video or participants, but leaves surveillance-related and demographic biases unaudited and does not claim high-stakes suitability.Source: Ethics statement · p. 10

Reproducibility statement

• The resource collection provides model weights, processors, source code, and configurations, but excludes training annotations, raw media, and internal experiment records.Source: Reproducibility statement · p. 10

A.1 Complementary Causal Views

• TVS uses query-independent views from the same causal prefix: recent frames preserve local detail, event views add sparse transitions, and compact history trades spatial resolution for temporal coverage.These views are complementary, not strictly nested; historical candidates are ranked using appearance novelty, visual-evolution curvature, and recency.Source: §A.1 · p. 17, A Causal Views and Adaptive View… · p. 17

A.2 Sequential Evidence Selection

• TVS starts with recent evidence, acquires event evidence when the recent proposal is uncertain, and consults compact history when recent and event answers differ.A frozen checkpoint proposes answers; acquisition and commitment rules determine which evidence to consult and whether to replace the retained answer.Source: §A.2 · pp. 17–18

A.3 Fixed Protocol and Validity

• The fixed protocol uses jointly calibrated thresholds and reports compiler probes, processed frames, and unique exposed source IDs as distinct costs rather than latency or token measures.Across all three views, the stated maxima are 56 processed frames and 52 unique source IDs.Source: §A.3 · p. 18
• Timestamp and strict-replay checks establish query-boundary compliance and reproduction of deployed answers, but do not establish that a selected view contains all needed evidence.Source: §A.3 · pp. 18–19

B.1 Supervision Units and Families

• Training records pair causal visual inputs with questions and direct targets, and the 7,677-example mixture includes core temporal families, sufficiency anchors, and capability probes.The capability probes include backward memory, action sequence, and repetition-counting cases; examples illustrate order and accumulation readouts without implying every question needs multiple frames.Source: §B.1 · p. 19, Table 6 · p. 19, Table 7 · p. 19

B.2 Source-Grounded Candidate Verification

• Candidates are sourced from annotations or teacher proposals behind a construction firewall, then checked for source binding, causal inputs, overlap, and option consistency.Source: §B.2 · pp. 19–20
• These checks do not establish that an individual question requires multiple frames or that its target is unambiguous to humans.The retained capability set contains 783 probes: 452 backward-state, 236 action-sequence, and 95 repetition-counting probes.Source: §B.2 · p. 20

C Controlled Experiments

• The controlled comparisons vary the checkpoint under fixed recent-4 access or vary the view policy with the ViSS checkpoint frozen; frame counts, resolutions, and calls are not matched across access policies.The training run starts from Qwen3-VL-8B-Instruct, trains the language model and multimodal projector, and freezes the vision tower.Source: §B.3 · p. 20, C Controlled Experiments · p. 21, §C.3 · p. 22

C.1 Representation under Fixed Recent Context

• ViSS improves OVO-Bench by +1.73 points over the strict control while remaining comparable on StreamingBench; HLD improves by +8.06 points, or 15 additional correct answers out of 186.Across three paired seeds, mean OVO and HLD gaps are +1.72 and +7.71 points, respectively.Source: §C.1 · p. 21, Table 10 · p. 21

C.2 Targeted Supervision Families

• Removing sufficiency examples yields 124 OVO-Bench regressions against 67 repairs and concentrates the largest task-level decline on HLD; removing repetition-count supervision changes aggregate scores only marginally.The full ViSS scores are 84.77/65.21, versus 84.65/65.70 without repetition-count supervision.Source: §C.2 · p. 22

C.3 Adaptive Access under Frozen Representation

• At a comparable StreamingBench frame budget, TVS scores 86.33 versus 85.69 for static sampling; against exhaustive max-confidence selection it uses fewer calls and frames, improves OVO-Bench, and scores lower on StreamingBench.Against a single history call, TVS uses fewer processed frames on StreamingBench but almost the same number on OVO-Bench, while requiring more calls on both.Source: §C.3 · p. 22
• Removing the acquisition gate raises processed frames from 17.08 to 27.26 on StreamingBench and from 30.33 to 33.31 on OVO-Bench, with accuracy changes of only −0.04 and −0.05 points.Source: §C.3 · p. 22
• Removing the commit margin changes accuracy from 86.33 to 86.53 on StreamingBench and from 69.77 to 69.75 on OVO-Bench, showing no consistent accuracy advantage for the margin rule.Source: §C.3 · p. 22

C.4 Task-Wise Representation and Access Effects

• The reported paired p values test query correctness, not OVO’s task-then-group macro, and do not measure variation across training seeds.The paired accounting compares full ViSS with the removed-sufficiency model, counting repairs and regressions.Source: §C.4 · p. 23
• The fixed-view Backward gain is primarily driven by HLD, which rises from 37.63 to 75.27, while EPM improves modestly and ASI declines slightly.The passage distinguishes hallucination robustness from event recall when interpreting this group.Source: §C.4 · p. 23
• With the ViSS checkpoint frozen, TVS improves event recall and forward reasoning, particularly REC and CRR, but reduces HLD and the real-time average.REC reaches only 51.43 under oracle proposal selection, which shows limits of the tested views but not that REC is the largest source of unresolved queries.Source: §C.4 · pp. 23–24

D Error Attribution: View Recoverability and Decision Stability

• The appendix analyzes errors by whether broader views make them recoverable, TVS selects an available correct proposal, and the policy preserves a correct retained answer.Strict replay computes all three view proposals for diagnosis only and does not change deployed inference cost.Source: D Error Attribution: View… · p. 24

D.1 Recoverability, Recovery, and Preservation

• 56.3% of StreamingBench and 62.9% of OVO-Bench recent-view errors remain unresolved across all views, beyond what proposal selection alone can recover.The table reports conditional diagnostics of view availability and proposal selection, with underlying counts in parentheses.Source: §D.1 · p. 24, Table 14 · p. 24
• Recent-answer preservation is 96.9% on StreamingBench and 92.3% on OVO-Bench; recoveries and lost recent answers yield net gains of 39 and 177 correct queries.The OVO query counts are diagnostic because the official score is a task-then-group macro.Source: §D.1 · p. 24

D.2 Where Correct Proposals Are Lost

• Replacement errors dominate the remaining oracle gap; acquisition misses are second-largest, while direct rejection of an acquired correct proposal is less common.Replacement counts include initially correct recent answers and correct intermediate event answers later lost at the history stage, so they exceed recent-answer regressions.Source: §D.2 · p. 25, Table 15 · p. 25

D.3 Failures Beyond the Current Proposal Set

• An unresolved query may reflect omitted decisive frames, failed interpretation despite adequate evidence, or visual ambiguity, so the analysis calls these all-view unresolved rather than representation errors.Separating these causes would require dense-prefix evaluation or an independent evidence-coverage audit.Source: §D.3 · p. 25

E.1 A Correct Answer Acquired from History

• In the Figure 5 case, recent-4 misses the cat taking the teddy bear, while event and compact-history views support answer A and the acquire–commit policy preserves it.Recent-4 proposes D before the broader views expose the decisive evidence.Source: Figure 5 · p. 26, §E.1 · p. 26

E.2 A Correct Answer Lost during Revision

• A correct recent answer D is replaced by event proposal B and then history proposal C after both pass the calibrated relative-margin criterion.The trace is a replacement error, but does not show that a different threshold would improve aggregate performance or identify which image changed the proposal.Source: §E.2 · p. 27, Table 16 · p. 27

E.3 A Correct Historical Proposal Not Acquired

• The correct historical proposal in Figure 6 is replay-only: TVS stops after recent-4 because its margin exceeds the acquisition threshold.The broader views retain an earlier left neighbour absent from recent-4; the example is an acquisition miss, not evidence that changing the threshold improves aggregate accuracy.Source: §E.3 · p. 28, Figure 6 · p. 28

E.4 All-View Unresolved with Counting-Scope Ambiguity

• All three views propose four when the reference is three, but the counting scope remains ambiguous because the model may or may not have included the electronic display.Recent and event views agree, so TVS stops before history; the passage does not establish a representation failure.Source: §E.4 · p. 29, Figure 7 · p. 29

E.5 Additional Archived-Run Comparisons

• Figures 8 and 9 are qualitative archived-run comparisons, not isolated ablations, because checkpoints and expanded-view encodings differ.The movie-insertion example returns four after causal access exposes evidence omitted by historical sampling; the lesson example predicts triangle area from the observed prefix without future frames.Source: §E.5 · p. 30, Figure 8 · p. 30, Figure 9 · p. 31

F Theoretical Analysis

• The propositions formalize state under-specification, characterize what separating probes add to answer-level objectives, and establish limits for wider causal views and TVS decision rules.They also give conditions for the margin gate to be optimal and explain why the calibrated commitment threshold is negative.Source: F Theoretical Analysis · p. 32
• The results assume unrestricted measurable predictors and concern population quantities, so they do not describe finite-sample behavior.The analysis defines view consistency in terms of histories mapped to the same view having the same reference answer within the supervision support.Source: F Theoretical Analysis · p. 32, §F.1 · p. 32

F.1 Setting

• The setting models a predictor as a view constructor, representation, and question-conditioned readout over causal prefixes and reference answers.The supervision family is a finitely supported distribution over question-history pairs, with population risk under log loss.Source: §F.1 · p. 32

F.2 State Under-Specification

• When histories have identical supervised question–answer support, answer-level risk permits merging their representations, leaving differing counts, orders, or relations unconstrained.The objective does not require the representation to retain attributes that the supervised answers never distinguish.Source: §F.2 · p. 32
• A positive-weight separating probe makes every minimizer—and every ε-approximate minimizer with ε < 2(1−λ)w log 2—keep the two views distinct.This provides the design rule for probe families, under view-consistency.Source: §F.2 · p. 32
• If two histories produce the same view, no supervision can separate them; the view constructor must restore the distinction.This is why TVS changes the view while keeping the learned model fixed.Source: §F.2 · p. 32

F.3 Wider Views and Commitment

• A wider view has at least the Bayes accuracy of a narrower view, and no commitment rule can beat the wider-view Bayes readout.For a Bayes readout on the wider view, always taking its proposal is at least as accurate as any rule that sometimes retains the narrower-view answer; deployed readouts need not be Bayes-optimal, and the three-view oracle gap is not a Bayes-risk estimate.Source: §F.3 · p. 33

F.4 The Acquisition Gate and the Commit Threshold

• The optimal acquisition gate selects margins where the expected accuracy gain exceeds the frame cost; if net value decreases with margin, this becomes a lower-margin threshold.In the stated cascade, the history view is acquired only after the event view and only on disagreement.Source: §F.4 · p. 33
• When view-specific calibration maps differ by a shift δ, the optimal rule takes the new proposal when mk−mj > −δ, making TVS's commit threshold a calibration offset.The paper expects a negative offset when wider views yield lower raw margins at equal accuracy; the holdout estimates δ.Source: §F.4 · p. 34