PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Long-Video VLM Embeddings with Query-Aware Streaming Latent Reasoning

Haozhe Chi, Song Jin, Yang Jin, Yadong Mu

arXiv:2610.04864v1cs.CVcs.AI

TL;DR

Long-video embeddings must capture sparse query-relevant evidence under a limited visual-token budget, while query-aware selection does not yield reusable corpus embeddings. QASLR streams selected clips through persistent latent memory with intermediate supervision; its full recipe improves retrieval and other evaluated tasks.

  • Problem

    Long-video embedding must capture sparse query-relevant evidence under a bounded visual-token budget, while query-conditioned frame selection generally prevents reuse as a single query-independent corpus embedding.

    Source: §1 · pp. 1–2

  • Method

    QASLR streams selected, ordered clips through persistent latent memory to produce fixed-dimensional readouts, trained with final and step-wise contrastive supervision and self-distillation.

    Source: §1 · p. 2

  • Results

    The full recipe improves HourVideo retrieval Hit@1 from 54.2 to 70.7 for 2B and from 57.4 to 72.8 for 8B Qwen3-VL-Embedding, with gains also reported on evaluated moment-retrieval and video-QA tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Query-aware selection suits bounded candidate scoring or reranking, whereas query-independent selection supports reusable video indexing.

    Source: §1 · p. 2, §5 · p. 9

    The latent memory cannot recover evidence absent from selected observations, and reusable corpus indexing requires query-independent selection and encoding.

    Source: §A.14 · p. 22

Abstract

from arXiv · show

Long-video embedding requires capturing sparse query-relevant evidence under a limited visual-token budget. Uniform sampling can miss brief events in videos spanning minutes or hours, whereas encoding more frames in a single context increases memory and computation. We introduce \textbf{Query-Aware Streaming Latent Reasoning} (QASLR), a post-training framework that accumulates evidence across clips while keeping the embedding size fixed. QASLR selects a bounded set of frames, restores their temporal order, and processes them clip by clip with a vision-language backbone. A compact set of persistent think tokens cross-attends to each clip's features, while an embed token reads out a normalized representation after every update. This design integrates evidence across multiple backbone calls without requiring all selected frames to share a single context. Training combines final contrastive learning, step-wise contrastive supervision, and final-embedding self-distillation. Intermediate supervision trains partial-video readouts for retrieval, while self-distillation regularizes them toward the final representation. Query-aware selection produces query-conditioned representations for candidate-set scoring and reranking, whereas query-independent selection enables reusable corpus indexing. Under the full training recipe, HourVideo retrieval Hit@1 increases from 54.2 to 70.7 and from 57.4 to 72.8 for 2B and 8B Qwen3-VL-Embedding backbones, respectively. Gains extend to the evaluated moment-retrieval and video-QA tasks, and the streaming head transfers to a second Qwen-family embedding backbone. These results support streaming latent aggregation as an effective approach to integrating long-video evidence into fixed-dimensional representations.

1 INTRODUCTION

Long-video embeddings must capture sparse query-relevant evidence under a bounded visual-token budget, while query-aware selection complicates reusable indexing. QASLR streams ordered clips through persistent latent memory, and its full training recipe improves retrieval and other evaluated tasks.

1 INTRODUCTION

  • Long-video embeddings must capture brief query-relevant evidence despite a bounded visual-token budget, and query-aware frame selection does not by itself solve evidence aggregation.Query-conditioned representations generally cannot be reused as a single query-independent corpus embedding.Source: §1 · pp. 1–2
  • QASLR processes selected frames as chronologically ordered clips, accumulating evidence in persistent think-token memory and emitting a fixed-dimensional embedding after each step.Query-independent selection supports reusable indexing; query-aware selection produces conditioned representations for bounded candidate scoring.Source: §1 · p. 2
  • HourVideo retrieval Hit@1 rises from 54.2 to 70.7 for 2B and from 57.4 to 72.8 for 8B Qwen3-VL-Embedding backbones; gains extend to evaluated moment-retrieval and video-QA tasks.The full recipe also improves every reported task across nine tasks and transfers within the Qwen family to RzenEmbed 7B.Source: §1 · p. 2
  • Training combines final contrastive learning, step-wise contrastive supervision, and self-distillation toward the final embedding to supervise intermediate and complete representations.The unified query–positive interface supports retrieval, moment retrieval, classification, and multiple-choice video QA.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses multimodal embeddings, long-video input constraints, and memory across observations. QASLR builds on these directions by accumulating selected clip evidence in a fixed-dimensional embedding and supervising intermediate readouts.

2.1 MULTIMODAL EMBEDDINGS AND LONG-VIDEO PROCESSING

  • Figure 1 contrasts one-shot pooling of sparsely sampled frames with QASLR’s ordered-clip processing and embeddings produced throughout the sequence.Its panels also summarize persistent memory, intermediate supervision, optional early exit, and parameter-efficient post-training.Source: Figure 1 · p. 3
  • Long-video methods extend context, compress or select visual inputs, and aggregate clip descriptions; QASLR complements these approaches by accumulating selected clip evidence into a fixed-dimensional embedding.Earlier multimodal embedding work established shared representations and temporal video aggregation.Source: §2.1 · pp. 2–3

2.2 STREAMING LATENT REASONING AND ADAPTIVE COMPUTATION

  • QASLR combines persistent latent memory across clips with contrastive supervision of intermediate embeddings; its optional early exit controls how many clips are observed.This builds on latent-query, recurrent-memory, video-streaming, and adaptive-computation approaches.Source: §2.2 · p. 3
  • Figure 2 depicts ordered clips updating persistent think-token memory, normalized readouts after each update, combined training objectives, and an optional stop gate.The illustrated configuration uses eight clips of 16 frames each.Source: Figure 2 · p. 4

3 METHOD

QASLR distinguishes reusable query-independent indexing from query-conditioned candidate scoring, then streams selected frames as ordered clips through fixed-size latent memory. Final and intermediate contrastive objectives and self-distillation train the readouts.

3.1 OVERVIEW AND RETRIEVAL SETTINGS

  • Query-independent selection yields one reusable embedding per candidate video, while query-conditioned selection encodes candidates under the query and is limited to a bounded candidate set.The model uses the same conditioning procedure for every candidate of a given query.Source: §3.1 · pp. 4–5

3.2 STREAMING VIDEO ENCODING

  • The selector returns at most Kmax frame indices; selected frames are restored to chronological order and divided into clips.At full budget, Kmax is 128 and each clip contains 16 frames; evaluation does not provide ground-truth labels or temporal boundaries to the selector.Source: §3.2 · p. 5

3.3 STREAMING LATENT MEMORY AND EMBEDDING READOUT

  • A fixed set of learnable think tokens persists across clips, attends to each clip’s evidence, and supplies an embedding readout after every update.The memory is initialized once per video and is not reset between clips; the final readout represents the video, while intermediate readouts are used for auxiliary objectives.Source: §3.3 · p. 5

3.4 TRAINING OBJECTIVE

  • Training combines final contrastive loss, step-wise supervision of valid intermediate readouts, and self-distillation using the detached final-step embedding as teacher.Mixed-task training interleaves compatible task and input formats, and only the streaming head is trainable while the backbone remains frozen.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments compare original checkpoints, single-round post-training, and QASLR across nine long-video tasks at 2B and 8B, with a separate within-family transfer evaluation. QASLR improves retrieval-style performance at both scales, while sub-batching benefits retrieval and QA results vary by task.

4.1 EXPERIMENTAL SETUP

  • The evaluation covered nine tasks across classification, retrieval, moment retrieval, and multiple-choice QA, comparing original, single-round post-training, and full QASLR at 2B and 8B.QASLR selected at most 128 frames, and only its streaming head was trainable while the backbone remained frozen.Source: §4.1 · pp. 6–7, Table 1 · p. 7

4.2 MAIN RESULTS ON QWEN3-VL-EMBEDDING

  • Mean retrieval Hit@1 rose from 45.8 to 53.5 at 2B and from 47.2 to 54.8 at 8B, with QASLR highest on all five retrieval-style tasks at both scales.The comparison is against the original checkpoints; single-round averages were 46.9 and 48.0.Source: §4.2 · p. 7, Table 1 · p. 7
  • QASLR achieved 70.7/92.5 HourVideo Hit@1/Hit@5 at 2B and 72.8/95.8 at 8B, while moment retrieval and QA also improved over original checkpoints.HourVideo Hit@1 gains over the original checkpoints were 16.5 and 15.4 points; mean QA accuracy rose from 48.3 to 50.6 at 2B and from 61.2 to 66.3 at 8B.Source: §4.2 · pp. 7, 9

4.3 GENERALIZATION TO ANOTHER VLM EMBEDDING BACKBONE

  • On RzenEmbed 7B, QASLR exceeded the original checkpoint and single-round post-training on all seven displayed tasks, with gains of 3.4–6.0 points over the original.This evaluates transfer to another Qwen-family embedding backbone.Source: Figure 3 · p. 8, §4.3 · p. 9

4.4 EFFECT OF INTERLEAVED SUB-BATCHING

  • With sub-batching, QASLR mean retrieval Hit@1 increased from 54.5 to 61.1 at 2B and from 55.7 to 63.0 at 8B; its QA advantage over single-round post-training varied by task.Sub-batching also improved single-round retrieval, and QA results were mixed across tasks and scales.Source: Table 2 · p. 8, §4.4 · p. 9

5 CONCLUSION

The conclusion presents QASLR as fixed-size latent memory updated over temporally ordered clips, with improvements across evaluated tasks and transfer to a second Qwen-family model. It also identifies query-aware use as suited to bounded candidate scoring or reranking and query-independent selection as needed for reusable indexing.

5 CONCLUSION

  • QASLR represents long videos through fixed-size latent memory updated over temporally ordered clips.The full recipe improved retrieval, moment retrieval, classification, and multiple-choice QA relative to original checkpoints at both 2B and 8B.Source: §5 · p. 9
  • Query-aware selection produces query-conditioned representations intended for bounded candidate sets or second-stage reranking, while query-independent selection is needed to index videos once for reuse across queries..Source: §5 · p. 9

AI USE STATEMENT

  • Generative AI tools were used only to improve manuscript clarity and readability, not to formulate claims, design experiments, generate results, or interpret findings..Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The authors state that the method may inherit biases and limitations from its training data and underlying models, and that real-world misuse and unintended deployment risks warrant consideration..Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The manuscript states that implementation details needed to reproduce the main results are provided in the paper and appendices..Source: REPRODUCIBILITY STATEMENT · p. 10

A APPENDIX

The appendix describes QASLR’s training configuration, evaluation data, component comparisons, and theoretical properties.

A APPENDIX

  • The appendix covers QASLR’s training configuration, evaluation data, component comparisons, and theoretical properties..Source: A APPENDIX · p. 15

A.1 DETAILED ALGORITHM

The algorithm processes selected frames in chronological clips, updating persistent think-token memory and reading out after each clip. Its training procedure combines final contrastive learning with enabled auxiliary objectives and supports optional early exit.

A.1 DETAILED ALGORITHM

  • Selected frame indices are decoded on demand, ordered chronologically, encoded clip by clip, and used to update persistent think-token memory with a readout after every clip.Memory is initialized once per video; the gate objective is only for optional early-exit training and is not required for fixed-length inference.Source: Table 3 · p. 15, §A.1 · p. 15

A.2 IMPLEMENTATION DETAILS

The implementation fixes the streaming head’s input, memory, and optimization configuration while keeping the pretrained backbone frozen. It also avoids duplicating selected image data across query–video annotations.

Backbone input and output

  • The image processor uses a pixel-area range of 4,096–1,310,720 with height and width aligned to multiples of 32; output dimensions are 2,048 for 2B and 4,096 for 8B.Source: §A.2 · p. 15

Streaming configuration

  • At the full frame budget, QASLR processes up to 128 selected frames in eight updates, using 32 think tokens and up to 256 evidence tokens per clip.With fewer selected frames, frames per clip are reduced; only valid frames and readouts contribute to attention and auxiliary losses.Source: §A.2 · p. 15

Trainable modules

  • Only the streaming latent reasoning head is updated; its trainable parameters total approximately 200 million for 2B and 600 million for 8B configurations.These counts describe the trainable head, not the total model size.Source: §A.2 · p. 15

Optimization

  • Post-training uses AdamW at a learning rate of 10−4 for approximately three to five epochs, with results averaged over three random seeds.Training ran on NVIDIA H100 GPUs; configurations used per-GPU batches of 4 or 8 and global effective batches of 256 or 512.Source: §A.2 · p. 16

Interleaved sub-batching

  • Interleaved sub-batching groups compatible examples and shares negative candidates across sub-batches and GPUs, changing both effective batch size and negative pool.For query-conditioned scoring, shared candidates retain the query-conditioning requirement.Source: §A.2 · p. 16

Frame storage

  • Selected frames are stored as indices with optional timestamps and decoded from original videos when needed, avoiding duplicated image copies for query–video annotations.Source: §A.2 · p. 16

A.3 ADDITIONAL EXPERIMENTAL DETAILS

The additional experiments draw primarily on LLaVA-Video-178K and evaluate retrieval, classification, moment retrieval, and video QA using task-specific candidate sets and scoring procedures.

Post-training data

  • Post-training primarily uses LLaVA-Video-178K, supplemented with 10,000 ActivityNet videos.Source: §A.3 · p. 16

Benchmark annotations and task construction

  • The four QA results in the main comparison are MovieChat, LVBench, EgoSchema, and TimeScope, while other tasks use sampled and filtered source-dataset subsets.Subset sizes vary from tens to thousands of examples; GPT-4o assists annotation by constructing frame-pair and question examples, and the evaluation sets are slated for public release.Source: §A.3 · p. 16

Candidate sets

  • Video-retrieval questions generally use 4 or 8 candidates, moment retrieval 8 or 16 segments, classification 4–8 categories, and QA four answer options.Retrieval scores rank within each task’s bounded candidate set; with four candidates and a valid positive, Hit@5 necessarily includes it.Source: §A.3 · p. 16

QA scoring

  • QA embeds the video–question input on the query side and the original answer options on the candidate side, selecting the option with highest embedding similarity.No additional QA prompt is introduced.Source: §A.3 · p. 16

Metrics and averaging

  • Retrieval reports Hit@1/Hit@5 where available, QA reports accuracy, and LongVideoBench moment retrieval reports Hit@1; results average three seeds.Task averages include only specified tasks and are not weighted by example count.Source: §A.3 · p. 16

A.4 EVALUATION MODES AND QUERY CONDITIONING

The evaluation contrasts reusable query-independent indexing with query-conditioned scoring of a bounded candidate set. In the two-stage HourVideo comparison, QASLR improves final Hit@1 over single-round scoring with the same first-stage candidates.

Query-aware frame selection

  • Query-aware Q-Frame selection conditions both selection and encoding of positive and negative candidates on the retrieval query, without evaluation-time labels or temporal boundaries.The resulting representation is intended for bounded candidate scoring.Source: §A.4 · p. 16

Indexing and two-stage scoring

  • Uniform selection produces reusable query-independent video representations, whereas two-stage methods apply query-aware selection and scoring only to the first-stage top-M videos.In the two-stage comparison, only the first retrieval stage is pre-indexable.Source: §A.4 · p. 16, Table 4 · p. 17

Indexable and query-conditioned retrieval

  • 70.7 vs 66.1 Hit@1 on HourVideo: two-stage QASLR + SB exceeds two-stage single-round + SB under the same first-stage candidate lists.Their reported latencies are 1.72 and 1.26 seconds per query, respectively.Source: §A.4 · p. 17, Table 4 · p. 17

First-stage recall and final accuracy

  • 95.8% first-stage R@M bounds end-to-end Hit@1; the reported values imply conditional top-one success rates of approximately 69.0% for single-round and 73.8% for QASLR.The conditional rates are calculated from rounded table values.Source: §A.4 · p. 17

A.5 ATTRIBUTION OF THE MAIN GAIN

The ablations distinguish sub-batching’s combined contribution from the additional streaming contribution, and show that QA differences vary by task while retrieval gains are more consistent.

Training configuration

  • At 2B, single-round HourVideo Hit@1 rises from 55.0 to 66.1 with sub-batching, which combines compatible grouping, shared negatives, and a larger effective batch.The comparison measures their combined contribution to post-training.Source: §A.5 · p. 17

Streaming contribution

  • QASLR improves HourVideo Hit@1 over original checkpoints by 16.5 and 15.4 points at 2B and 8B, and over sub-batched single-round training by 4.6 and 5.5 points.The latter comparison keeps sub-batching in both methods.Source: §A.5 · p. 17

Task-dependent QA gains

  • QA differences are mixed across tasks, unlike the more consistent retrieval advantage.QASLR minus sub-batched single-round differences are −3.6, −4.9, +0.6, and −1.1 points for MovieChat, LVBench, EgoSchema, and TimeScope at 2B; at 8B they are −1.2, +1.4, +0.7, and +1.5 points, respectively.Source: §A.5 · p. 17

A.6 DOES THE LATENT MEMORY USE TEMPORAL EVIDENCE?

The appendix tests whether QASLR benefits from chronological cross-clip memory, examines frame and objective choices, and states limits on what its experiments and conditional analyses establish.

A.6 DOES THE LATENT MEMORY USE TEMPORAL EVIDENCE?

  • HourVideo Hit@1 falls 6.6 points with shuffled clips, 5.7 with reversed clips, 10.4 with memory reset, and 13.8 with only the last clip; QASLR exceeds the reported Transformer aggregator by 4.5 points.These inference-time controls support the usefulness of cross-clip state and chronological inputs under the evaluated settings, while also measuring sensitivity to altered inputs or state updates.Source: §A.6 · p. 17, Table 5 · p. 18

A.7 FRAME BUDGET AND STREAMING DEPTH

  • K controls the observations supplied to the backbone, while L controls per-call context length; at fixed K, changing L also changes update count, and selected evidence cannot include omitted events.The full-budget setting uses K = 128 frames, L = 16 frames per clip, and eight updates.Source: §A.7 · p. 18, §A.9 · p. 19

A.8 OBJECTIVE ABLATIONS AND INTERMEDIATE EMBEDDING QUALITY

  • With both auxiliary objectives, final HourVideo Hit@1 reaches 70.7 versus 67.9 with final-only supervision, and mean checkpoint Hit@1 rises from 57.3 to 65.1.At the 25% checkpoint, step-wise supervision alone, distillation alone, and both together yield gains of 5.9, 7.4, and 11.8 points, respectively; early clips may lack target evidence, and distillation is a training regularizer rather than access to unseen frames.Source: Table 6 · p. 18, §A.8 · p. 19

A.9 FRAME SELECTION AND EVIDENCE COVERAGE

  • Q-Frame selects text-relevant frames within the input budget, QASLR restores chronological order, and the latent memory cannot recover evidence absent from selected observations.Source: §A.9 · p. 19

A.10 EARLY EXIT AND COMPUTATIONAL COST

  • On H100 warm-cache measurements, QASLR takes 1.72 s versus 1.26 s for two-stage single-round training, with a 4.6-point HourVideo Hit@1 gain and 0.46 s latency increase.Optional local-clip QA early exit returns the first eligible readout meeting its threshold, otherwise continuing to the final readout; fixed-length retrieval uses the complete sequence.Source: §A.10 · p. 19

A.11 DATA SEPARATION AND RESULT AGGREGATION

  • The post-training and evaluation sets have zero overlap in source videos, and reported experimental results are averaged over three random seeds.Source: §A.11 · p. 19

A.12 COMPLEXITY ANALYSIS

  • The persistent memory occupies O(ND) values per example, but the stated inference-state bounds are not training peak-memory bounds because backpropagation may retain activations across steps.Source: §A.12 · p. 20

A.13 THEORETICAL ANALYSIS

  • The theoretical claims are conditional: ranking preservation requires a positive full-step margin and bounded candidate embedding drift, while the early-exit agreement bound assumes history-conditional calibration and is not a ground-truth correctness guarantee.The architecture alone does not establish a contractive memory update, and the stated assumption is not verified by the reported experiments.Source: §A.13 · pp. 20–21
Loading 2610.04864v1…