PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Self-Distillation: Learning Visual State Tracking in Videos Without Supervision

Shravan Venkatraman, Wenshuai Zhao, Mohammad Hassan Vali, Arno Solin

arXiv:2609.04203v1cs.CV

TL;DR

Video-LLMs struggle to maintain cumulative visual state across a clip, a capability needed to track counts and scene changes over time. S3T uses a denser view of the same clip to teach a sparse-view student through self-generated targets, without labels or a separate teacher. It improves VSTAT accuracy and transfers state-tracking gains to real-video benchmarks.

  • Problem

    Video-LLMs remain weak at maintaining running counts and scene states as objects change or become occluded, and common objectives do not require accumulating evidence across a full clip.

    Source: §1 · pp. 1–2

  • Method

    S3T distills next-token predictions from a denser view of a clip into a sparse-view student using the same model and self-generated targets, without labels or a separate teacher.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    VSTAT accuracy rises from 34.74 to 37.44 (+2.70), with gains of +7.95 on VSTAT-YouTube cumulative-state questions and +4.50 on MVBench Action Count.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results show that temporal sampling density can provide label-free supervision for long-horizon visual state tracking, with capability learned from synthetic clips transferring to real videos.

    Source: §1 · p. 2, §6 · p. 8

    Across eleven base models from five families, only LLaVA-OV-2-8B showed a clear improvement; the tested factors behind this limited transfer remain unresolved.

    Source: §5 · p. 8

Abstract

from arXiv · show

We introduce S$^3$T (Self-Supervised Self-Distillation over Time), which, to the best of our knowledge, is the first fully self-contained framework for continuous video state tracking. Our method treats temporal sampling density as privileged information, based on the hypothesis that a denser view of the same clip recovers the running state more accurately. This view serves as the teacher, while a sparse-view student with the same weights learns to match its next-token distribution. The model generates its own target, so training requires no labels, separate teacher, or reward signal, and adds no inference cost. On LLaVA-OneVision-2-8B, S$^3$T improves VSTAT accuracy by $+1.74$ as a single model, $+2.38$ with souping, and $+2.70$ with additional vision-encoder adaptation, while prior self-evolving methods leave state tracking largely unchanged. The capability learned from unlabeled synthetic clips transfers to real videos, improving performance by $+7.95$ on VSTAT-YouTube state-tracking questions and $+4.50$ on MVBench Action Count.

1. Introduction

Video-LLMs struggle to maintain a running visual state as objects change or become occluded, while existing training objectives do not require accumulating evidence across a full clip. S3T uses denser temporal views to teach sparse-view tracking without external supervision and improves state-tracking results, including on real videos.

1. Introduction

  • Human performance on VSTAT reaches 90.5%, while the strongest open-source models score only 34–35% on cumulative state reasoning.Models may keep revising predictions as more of the video is revealed rather than maintaining a stable running state.Source: §1 · pp. 1–2
  • Existing video self-distillation still needs ground-truth labels and a judge, while self-evolving methods target general reasoning or temporal grounding rather than running visual state.Supervised fine-tuning and video–text contrastive alignment also do not require models to accumulate evidence across the full clip.Source: §1 · p. 2
  • S3T uses a denser view of a clip as teacher and distills its next-token distribution into a sparse-view student using self-generated outputs.The framework requires no labels, separate teacher, reward model, or tracker.Source: §1 · p. 2
  • S3T raises VSTAT accuracy from 34.74 to 37.44 (+2.70); gains transfer to real videos, improving VSTAT-YouTube cumulative-state questions by +7.95 and MVBench Action Count by +4.50.The authors report this transfer without retraining on real data.Source: §1 · p. 2

2. Related work

Prior video models and benchmarks emphasize action recognition, localization, and question answering more than maintaining a cumulative scene state. Existing temporal-learning and self-evolving approaches also do not directly target long-horizon state accumulation, and the closest video self-distillation method uses external supervision.

Video understanding and state tracking

  • Video-LLMs are commonly trained for action recognition, moment localization, and question answering, while VSTAT highlights weak state-tracking results: 34–35% for open-source models versus about 90.5% for humans.These tasks generally locate relevant frames rather than maintain a running scene state throughout a video.Source: §2 · p. 3
  • Temporal self-supervised objectives learn chronology, motion, or missing-frame structure, but do not require running counts or evolving scene states across long sequences and occlusions.The passage lists frame-order prediction, arrow-of-time classification, pace estimation, and masked video modeling.Source: §2 · p. 3

Self-distillation and self-evolution

  • Image self-distillation uses privileged spatial views, while video self-evolving methods target difficulty, consistency, evidence discovery, or temporal grounding rather than cumulative-state tracking.VISD is closer to video self-distillation but relies on ground-truth answers, spatio-temporal grounding annotations, and an external video-aware judge.Source: §2 · p. 3

3. Methods

S3T trains on unlabeled clips with two temporal views of the same video, using the denser view to teach the sparse view through self-generated targets. The method combines a shared adapter, a fixed state probe, and a teacher-matching objective anchored to the base model.

3. Methods

  • S3T samples each clip at two frame densities and trains the sparse view to learn from the denser view using self-generated targets and one trainable LoRA adapter.The base model is frozen, and the approach adds no inference-time cost.Source: §3 · p. 3

3.1. Synthesizing training data

  • StateGen supplies 300 unlabeled 20-second clips with 6–15 near-identical entities and 7–11 add, remove, move, swap, or recolor events.Occlusion and global camera motion continue throughout; the clips have no state annotations, question-answer pairs, or captions.Source: §3.1 · p. 3, Figure 3 · p. 3

3.2. Temporal self-distillation

  • The student samples 12 frames and the teacher 24 frames uniformly across the same full clip, with the same adapter and gradients only through the student.The two frame grids share only their endpoints, so the teacher sees additional temporal evidence rather than a deterministic superset of the student’s frames.Source: §3.2 · p. 4, Figure 4 · p. 4
  • A fixed probe question shared across views asks for a short description of the current state; the model generates the answer used for self-distillation.The probe names the domain objects and asks which balls are visible, their colors, and their count.Source: §3.2 · p. 4
  • The objective matches the teacher’s next-token distribution while anchoring the student to the base model on the same sparse input, using token-averaged JSD.The JSD is symmetric and bounded, and the base anchor is also a JSD to preserve boundedness.Source: §3.2 · pp. 4–5
  • The anchor weight β is adjusted to keep dref near a target κ, balancing drift from the base against learning from the teacher.The update clips the per-step change and β to prevent spikes in dref from pushing β to its bound.Source: §3.2 · p. 5

4. Experiments

Under the official 64-frame VSTAT protocol, S3T improves overall state-tracking accuracy, with gains concentrated on cumulative-state reasoning. Comparisons and ablations examine temporal pretexts and design choices, while gains transfer to real-video tally tasks.

4.1. Main results

  • 37.44 vs 34.74 on VSTAT (+2.70); a single S3T run gains +1.74 before souping, versus prior self-evolving methods’ −0.22 to +0.82.The table reports VSTAT accuracy under the official protocol and uses the reproduced base score of 34.74 for deltas.Source: §4.1 · pp. 5–6, Table 1 · p. 6
  • S3T gains concentrate on cumulative-state axes; its variants rank among the top two overall and on Atomic and Sequence, while SFT remains below the base overall.SFT attains the best Attribute and Set scores, but improving individual axes alone does not suffice for an overall gain.Source: §4.1 · p. 6

4.2. Comparison with temporal pretexts

  • With matched training recipe, data, and frame budget, established temporal pretexts leave the base essentially unchanged overall; only S3T improves Count, and pace prediction recovers less than half its gain.The comparison changes only the prediction target, isolating the learning signal.Source: Table 3 · p. 7, §4.2 · p. 7

4.3. Ablations

  • Only the stretch teacher gives the full gain; nested 24-frame and shifted 12→12 teachers score 34.10 and 35.14, while full temporal coverage matters more than a short-window zoom.Event-aware, multi-scale, and adaptive teachers show no consistent advantage over the fixed view.Source: §4.3 · p. 7
  • Supervised objectives and true-state distillation fall below the base, forward or reverse KLD lowers performance, and removing the reference anchor drops performance below the base.The authors report that the anchor stabilizes training and that KLD alternatives produce larger gradient spikes and less stable optimization.Source: §4.3 · p. 7

4.4. Generalization to real video

  • +7.95 on VSTAT-YouTube tally questions and +4.50 on MVBench Action Count with vision adaptation; TempCompass changes are nonsignificant and MMVU remains unchanged.The reported gains are relative to the base model; general video-understanding performance is preserved.Source: Table 4 · p. 8, §4.4 · p. 8

5. Discussion

S3T benefits most when the student and teacher use 12 and 24 frames, respectively, and temporal coverage matters more than frame count. The improvement comes mainly from the language decoder, but gains did not transfer consistently across tested models.

5. Discussion

  • At 64 frames, S3T scores 36.48 versus the base model’s 34.74; it also beats the base model at every tested budget, including the base model’s best score of 35.33 at 96 frames.Raising the base model’s evaluation budget from 24 to 64 frames changes accuracy from 34.51 to 34.74, while S3T’s rises from 34.23 to 36.48.Source: §5 · p. 8
  • S3T’s gains appear only near a 12-frame student and 24-frame teacher, with student budget more sensitive; teacher-view ablations indicate temporal coverage matters more than frame count.The denser view must provide information missing from the student while remaining similar enough for it to imitate.Source: §5 · p. 8
  • Decoder-only adaptation gains +2.38 of the total +2.70, while vision-encoder adaptation adds +0.32.With the vision encoder and vision-to-language projector frozen, the +2.38 gain comes from changing how the decoder combines information across frames.Source: §5 · p. 8
  • Across eleven base models from five families, only LLaVA-OV-2-8B [2] showed a clear improvement, and the authors did not identify a consistent pre-training signal explaining the difference.They could not separately test video post-training and adapter effects because public checkpoints do not permit matched-pretraining comparisons across adapter placements.Source: §5 · p. 8

6. Conclusion

S3T uses a denser view of a clip to distill predictions into a sparse view of the same model, without supervision. The reported gains concentrate on cumulative-state tasks, preserve general video understanding, and transfer to real videos.

6. Conclusion

  • S3T distills predictions from a denser view of a clip into a sparse view of the same model, without labels, an external teacher, a reward model, or a tracker.The conclusion characterizes temporal sampling density as label-free supervision for long-horizon visual state tracking.Source: §6 · p. 8
  • On VSTAT, S3T’s improvements concentrate on cumulative-state tasks while general video understanding is preserved.The authors interpret this pattern as better maintenance of state over time.Source: §6 · p. 8
  • The learned ability transfers to real videos on tasks requiring a running tally.Source: §6 · p. 8

Supplementary Material

The supplement details S3T’s implementation and examines its denser-view assumption, possible answer-prior effects, and held-out selection of soup strength.

Supplementary Material

  • The supplement provides implementation details, temporal-view analyses, extended ablations, additional real-video results, and backbone-generalization discussion.Source: Supplementary Material · p. 13

A. Training data and hyperparameters

  • The 300 fixed StateGen training clips contain evolving object scenes with occlusion and camera motion; their JSON state logs are used only for offline analysis, while training uses RGB video and model outputs.Each clip is 512×512, 20 seconds at 12 fps, and is sampled about ten times during 3,000 training steps.Source: A. Training data and hyperparameters · p. 13
  • The selection protocol fixes the objective, schedule, data, and training settings before VSTAT evaluation; teacher view, frame ratio, divergence, checkpoint step, and soup strength are compared as ablations, not tuned hyperparameters.The deployed configuration is reported rather than the best result for each ablation.Source: A. Training data and hyperparameters · p. 13

B. The denser view as a state estimate

C. Gains against the answer prior

  • S3T gains +1.74 on the 1,134 questions where the answer prior is wrong, with an interval excluding zero; this group supplies about half the overall gain.The gain is larger on questions agreeing with the prior (+5.68 versus +1.74), so part of the effect is a shift toward common answers, but the wrong-prior group also improves significantly.Source: C. Gains against the answer prior · p. 14, Table A2 · p. 14

D. Choosing the soup strength on held-out data

The authors selected soup strength using held-out StateGen clips before inspecting VSTAT results. The held-out choice matched the setting with the best VSTAT score, leaving the headline number unchanged.

D. Choosing the soup strength on held-out data

  • Soup strength α is an especially portable selection choice because it is a single number applied after training.Source: D. Choosing the soup strength on… · p. 14
  • The α sweep uses 100 fresh StateGen clips from seeds disjoint from training, and the procedure runs before any VSTAT number is examined.Merged models are evaluated on the student’s 12-frame view by comparing reported counts with the clips’ true counts.Source: D. Choosing the soup strength on… · p. 14
  • The held-out count error is lowest at α=1.3, which also gives the best VSTAT score, so selecting α without the benchmark returns the reported model and leaves the headline number unchanged.The table compares mean absolute count error on 100 held-out clips with the VSTAT score of the same merged model.Source: D. Choosing the soup strength on… · p. 14, Table A3 · p. 14

E. Frame budgets

The frame-budget sweep finds that gains depend on both student and teacher budgets, while evaluation with more frames alone does not recover S3T’s improvement. JSD also yields more stable optimization and better downstream performance than KLD variants.

E. Frame budgets

  • Gains concentrate near a 12-frame student and 24-frame teacher; above about 20 student frames, the objective is harmful across tested teacher budgets.The student budget is more sensitive, while teacher budgets allow a wider range; Figure A1 marks gains, losses, and the teacher-equals-student line.Source: E. Frame budgets · p. 14, Figure A1 · p. 14
  • A 2× frame ratio does not determine the gain: 24→48 loses 3.85 points, and increasing the teacher budget at a fixed student budget reduces accuracy.The student budget also matters independently.Source: E. Frame budgets · pp. 14–15
  • 36.48 at 64 frames exceeds the base model’s peak of 35.33, and S3T is higher than the base at every listed evaluation budget.The base improves up to 96 frames and declines afterward; adding frames at evaluation alone does not recover S3T’s gain.Source: Table A5 · p. 15, E. Frame budgets · p. 15

F. Choice of distillation divergence

G. Ablation setup

The ablations vary teacher views, frame budgets, and objective components to test what makes the 12→24 temporal stretch useful. Their analysis suggests that useful teacher disagreement must provide new information that remains learnable from the student’s sparse view.

G. Ablation setup

  • Teacher-view comparisons hold the 12-frame student fixed and vary teacher frame placement; the default samples 24 frames uniformly across the clip.The windowed zoom concentrates frames around one scene change and uses the student’s frame count as a placement control.Source: G. Ablation setup · pp. 15–16
  • The event-aware teacher places more frames near detected scene changes, using differences between the frozen model’s predictions on windows around candidate times.The boundary-detection evaluation covers 200 synthetic clips and uses AUROC, with 0.5 corresponding to chance.Source: G. Ablation setup · p. 16, Table A7 · p. 16
  • Adaptive teachers select density per clip, while frame-ratio runs vary student and teacher budgets to separate ratio effects from the student budget’s effect.The adaptive variants use a lightweight router or adjust density to keep dpos near a target range.Source: G. Ablation setup · p. 16
  • The objective controls compare removing the reference anchor, using identical 12-frame views, training on teacher text, or distilling toward the generator’s true state.The true-state run is the paper’s only training experiment using program state; the no-anchor setting sets β=0.Source: G. Ablation setup · pp. 15–16
  • Alternative teacher views do not meaningfully improve on the fixed 12→24 temporal stretch: the adaptive teacher’s 36.49 is within ±0.52 variation of the fixed view’s 36.48.The adaptive version also needs a per-clip decision rule that cannot be self-learned or self-distilled.Source: Table A8 · p. 16, G. Ablation setup · p. 16

H. What the teacher and student disagree about

  • Similar teacher–student disagreement can accompany different final scores; useful supervision must add information that the sparse-view student can learn.The analysis covers only four runs, and dpos for the true-state run is not directly comparable because its answer string differs.Source: H. What the teacher and student… · p. 17

I. Model souping

Model souping combines runs with different probe strengths, while the temporal-pretext comparisons test alternative unlabeled training targets. Complementary probes improve on redundant-run averaging, and none of the tested pretext baselines matches S3T’s state-tracking gains.

I. Model souping

  • The full-state run performs best across state axes at 36.88, while the alternating-probe run performs best on Count at 36.11; their soup improves beyond both parents.The two probes specialize differently: the alternating run switches between the full-state question and six fixed questions about clip changes.Source: Figure A3 · p. 17, §I · p. 17
  • At α=1.5, complementary-probe runs reach 37.12 versus 36.08 for redundant-seed merges, which remain below the 36.48 single-run best.The merge procedure and scale are held fixed; complementary runs outperform their best parent in two replications.Source: §I · pp. 17–18, Table A10 · p. 18
  • Souping averages LoRA weight updates and scales the merge by α; the reported strengths are 1.5 with a frozen vision encoder and 1.3 when adapted.The adapted setting rescales both vision-tower and downstream updates; evaluated frozen-encoder strengths remain above the 34.74 base score.Source: §I · p. 18

J. Temporal-pretext baselines

  • The pretext baselines train on clip transformations and derived text labels; masked infill instead uses S3T’s self-generated target with the two views exchanged.The four baselines receive no textual target from the clip generator.Source: J. Temporal-pretext baselines · p. 18
  • None of the pretext baselines matches S3T’s state-tracking gains; pace reaches 35.45, while masked infill scores 34.25 against the 34.74 base.Frame order raises Dictionary to 30.5 but leaves the overall score unchanged; masked infill differs from S3T in which view is degraded.Source: J. Temporal-pretext baselines · pp. 18–19

K. Additional real-video results

Additional real-video tests show that transfer is specific to cumulative-state reasoning rather than uniform across tracking tasks.

K. Additional real-video results

  • On Kinetics videos, training gives no clear overall improvement at +0.66, though cumulative-state questions improve by +4.09; VET-Bench shell-game results stay near chance.The shell game tests tracking one hidden object through swaps rather than maintaining a count.Source: Table A13 · p. 19, K. Additional real-video results · p. 19

L.1. Results across base models

Across eleven base models, S3T clearly improves only LLaVA-OV-2-8B; the tested explanations do not account for why transfer differs.

L.1. Results across base models

  • Only LLaVA-OV-2-8B shows a clear improvement; InternVL3.5-14B (+0.27) and Qwen3-VL-8B (+0.13) changes are within seed variation, and other models are unchanged or worse.The comparison uses the same objective and training data across models; changes within the ±0.52 seed spread are treated as noise.Source: §L.1 · p. 19, Table A14 · p. 19

L.2. What explains the difference?

  • None of the tested factors explains transfer differences, and teacher-view headroom does not predict S3T gains (Spearman ρ = +0.29, p = 0.39; Pearson r = +0.21, p = 0.54).Qwen3-VL-8B has greater headroom (+3.42) than LLaVA-OV-2-8B (+2.91), but gains only +0.13; models change answers across frame budgets, yet that information may not transfer to the sparse view.Source: §L.2 · p. 20
  • Molmo2-4B retuning reduces its deficit from −0.69 to −0.18, but its best checkpoint (35.82) does not clearly exceed the 35.81 base score.The checkpoint sweep peaks at 35.82 against a base of 35.81.Source: §L.1 · p. 19, §L.2 · p. 20
  • Cambrian-S-7B improves from −4.24 to −1.16 after anchor-bound tuning, but remains unstable, with final changes ranging from −1.16 to −14.02.Gradient norms range from 8.5 to 37, and halving the learning rate does not remove the instability.Source: §L.1 · p. 19, §L.2 · p. 20
  • The +0.27 and +0.13 changes on InternVL3.5-14B and Qwen3-VL-8B are not reliable; per-axis patterns differ from LLaVA-OV-2-8B.Confidence intervals reflect item-level variation only for single-seed runs; across-seed variation can exceed observed per-axis changes.Source: §L.2 · p. 20
Loading 2609.04203v1…