LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perception

• Long-form AVQA evidence is scattered across brief windows, while full-recording processing consumes context and memory on largely irrelevant content.Source: §1 · p. 1
• Prior methods trade coverage and detail: coarse selection can miss needed windows, compression thins retained detail, and inference-time agents leave localization to untrained general-purpose models.Source: §1 · pp. 1–2
• LEAP’s two-resolution retrieval partitions recordings into fixed-duration blocks and candidate windows, ranks each block, then re-encodes a bounded retained set at O(1) context and working memory.Source: §1 · pp. 1–2
• 4.5–16.8%: LEAP improves accuracy over Qwen3-Omni-30B-A3B across several AVQA benchmarks, and exceeds MiniCPM-o 4.5’s published results by 3.1–13.0%.Source: §1 · p. 2
• Transcript localization can scan precomputed text without decoding frames, while raw audio-video answering preserves fine-grained visual and non-speech evidence.Source: §1 · pp. 2–3
• The block grid also supports causal-prefix queries, enabling streaming inference without streaming-specific training.Source: §1 · p. 3

• Selection, compression, and agentic systems address long-form AVQA context limits but can respectively drop needed windows, thin retained detail, or repeat media work per question.Source: §2 · p. 3
• Streaming systems answer from KV caches, fixed-size memories, or textual history; ShallowStream indexes with shallow-layer KV and reprocesses ranked frames through all layers (Hao et al., 2026).Source: §2 · p. 3
• Text retrieval ranges from captions and document indexes to transcript or subtitle search; some systems keep media indexed for fallback, while caption, document-retrieval, and long-audio planning pipelines deliver text alone to answer models.Source: §2 · p. 3

3.1 PROBLEM FORMULATION AND OVERVIEW

• LEAP partitions recordings into 600-second blocks and 75-second candidate windows, retaining up to three blocks and three windows per block for at most nine answer windows.The two-stage process localizes evidence before re-encoding selected windows for answering.Source: §3.1 · pp. 3–4

3.2 FRAMEWORK

• The frozen omni-modal backbone uses separate localization and answer LoRA adapters, which are trained on their own data and never stacked.The described setup uses Qwen3-Omni-30B-A3B and MiniCPM-o 4.5.Source: §3.2 · p. 4, §3.2.2 · p. 5

3.2.1 STAGE I: THE LOCALIZATION PASS AND WINDOW SCORING

• For each block, localization scores candidate-window option logits and ranks the block by the maximum candidate-window posterior.The pass emits option-letter logits without decoding an answer or retaining a cross-block key–value cache.Source: §3.2.1 · p. 4

3.2.2 STAGE II: BLOCK RANKING AND ANSWER PASS

• The answer pass pools the highest-scoring windows from retained blocks, restores chronological order, and jointly re-encodes their synchronized raw audio-video with timestamps.It also reads a capped transcript outline; the selected windows are bounded by the block and window limits.Source: §3.2.2 · p. 4

3.3 LOCALIZATION ADAPTER

• Localization training uses annotated evidence spans to target the candidate window that best covers the span, with cross-entropy over the option-letter logits.Training clips are adapted to the selector’s candidate structure; Qwen3-Omni also compares overlap-fraction BCE, while MiniCPM-o uses it.Source: §3.3 · p. 5

3.4 ANSWER ADAPTER

• Answer-adapter training applies cross-entropy to answer and end-of-turn tokens conditioned on evidence, including timestamped raw audio-video distractor segments.The training evidence sequence omits the transcript outline.Source: §3.4 · p. 5

3.5 ADVANTAGES

• Each pass has fixed-size context: localization reads one block, while answering reads at most the selected windows plus outline and question, keeping per-pass context and memory O(1) in recording length.A whole-recording pass costs about 47k tokens per hour and reaches the backbone’s position limit at roughly 81 minutes.Source: §3.5 · p. 5
• Separating coverage from detail lets localization scan every block while answering re-reads selected raw audio-video, preserving visual and non-speech evidence even when transcripts guide retrieval.Localization supervision marks evidence location rather than answer correctness, so the selector does not learn which inputs an answering model gets right.Source: §3.5 · pp. 5–6

4.1 SETTINGS

• The evaluation covers four main AVQA benchmarks, additional paired-ablation and video-only benchmarks, and carries the same block configuration to StreamArena.The main benchmark answer pass uses a transcript outline, while selector comparisons answer without it.Source: §4.1 · p. 6

4.2 MAIN RESULTS

• LEAP significantly leads same-stack baselines on all four main benchmarks and also leads whole-clip and frame-matched comparisons; Figure 2 extends the lead to LVBench and Video-MME.The extended comparison reads LVBench and Video-MME without audio, following their official protocols.Source: §4.2 · p. 6, Figure 2 · p. 7

4.3 RETRIEVAL STAGE

• Localization training raises evidence coverage, while the trained selector outperforms off-the-shelf retrieval and answer-confidence rankings in coverage and accuracy.Coverage gains are greatest on VideoOdyssey; gains in accuracy vary across benchmarks, with little remaining coverage to gain on TraceAV.Source: §4.3 · p. 7, Figure 3 · p. 7

4.4 ANSWER STAGE

• With the answer LoRA fixed, LEAP’s selected windows outperform whole-clip reading on all four benchmarks, retaining most of its LVOmniBench lead without clips too long to encode whole.Against frame-matched whole-clip input, LEAP leads significantly on TraceAV, LVOmniBench, and MMOU; it also transfers to MiniCPM-o 4.5.Source: §4.4 · p. 8

4.5 AUDIO AND TRANSCRIPTS

• LEAP uses audio in both passes; muting the answer pass significantly reduces accuracy on hearing-required and cross-modal questions, while transcript retrieval is a faster, lower-token alternative where evidence is spoken.Removing the transcript outline significantly costs accuracy on TraceAV, VideoOdyssey, and MMOU; a question-first outline beats uniform thinning significantly on VideoOdyssey.Source: §4.5 · p. 8

4.6 DURATION AND EVIDENCE DISTANCE

• LEAP shows no systematic accuracy collapse as videos lengthen, and significantly leads a context-filled montage in every length quintile.On MMOU, accuracy rises with duration.Source: §4.6 · p. 9, Figure 4 · p. 9
• On StreamArena, the unchanged block grid significantly outperforms whole-prefix and equally spaced-window baselines under causal access, without streaming-specific training.The answer pass includes the window ending at query time; transcript-selected windows are answered from raw media, and on MiniCPM-o 4.5 LEAP beats native streaming overall and in the second distance bin.Source: §4.6 · p. 9, Table 3 · p. 9

5 CONCLUSION

• Across four benchmarks, LEAP improves over each benchmark’s official-style same-backbone recipe by 4.5–16.8%, shows no systematic duration collapse, and transfers to a second omni-modal backbone.The transcript channel uses 4.7–6.2× fewer localization tokens than the media channel, and the block grid applies to causal prefixes without streaming training.Source: §5 · p. 10

REPRODUCIBILITY STATEMENT

• The statement directs readers to the method, adapter training, inference settings, evaluation protocol, baseline and causal-access procedures, and contamination audit.It states that all benchmarks and training corpora are public releases and that code will be released upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 THE INFERENCE ALGORITHM

• Inference ranks transcript minutes, divides video into non-overlapping 600-second blocks, and scores candidate windows within each block.Each block gets one localization pass; up to three windows per selected block contribute to a maximum of nine windows.Source: §A.1 · p. 15
• The algorithm time-sorts selected windows, packs them with the outline, question, and options, then runs one bounded answer pass.The window union is capped at nine by construction.Source: §A.1 · p. 15

A.2 TRAINING AND INFERENCE CONFIGURATION

• MiniCPM-o 4.5 uses fewer frames per second in its answer pass than localization, with ten frames per window marking its long-recording accuracy plateau.More answer-pass frames, up to one per second, do not improve accuracy on LVOmniBench or OmniVideoBench and one per second lowers it.Source: §A.2 · p. 15

A.3 DATA, PROMPTS, AND RENDERERS

• The localization corpus keeps 40,284 of 69,630 LongVALE questions after a base-model filter requires a yes on the target window and no on every alternative.The answer pass presents the transcript outline, retained raw-media windows in time order, and then the question; the transcript outline has a 4,000-token cap.Source: §A.3 · pp. 15, 17, 18

A.4 COST ACCOUNTING: A BOUNDED PEAK FOOTPRINT

• Peak context and memory remain bounded at any duration, although query work uses one localization pass per block plus one answer pass.Median pass count is 13 on VideoOdyssey and single-digit elsewhere; cumulative prefill is 2.2–2.4× a single whole-clip pass on the reported Qwen3-Omni subsets.Source: §A.4 · p. 18

A.5 RELATION TO PRIOR SEARCH OVER LONG RECORDINGS

• LEAP ranks blocks using question-conditioned option-letter logits already produced by its localization pass, adding no parameters, pass, or cost beyond M+1.Unlike Zoom-Zero’s divide-and-conquer variant, LEAP’s scan is jointly audio-visual with native-rate audio, and its score requires no second pass per block.Source: §A.5 · p. 19

B.1 SYSTEMS, CONTEXTS, AND REPORTING CONVENTIONS

• The 65,536-token position limit gives a whole-clip limit of roughly 81 minutes at 32 frames plus full audio; paired differences use source-video-resampled intervals, significant when intervals exclude zero.Resampling uses a fixed seed and at least 10,000 resamples; failures count as errors where gold labels are public.Source: §B.1 · pp. 19–20

B.2 CONCURRENT SYSTEMS ON THESE BENCHMARKS

• LEAP outperforms OmniRAG-Agent on all four benchmarks and leads OmniVideo-100K SFT on every benchmark except MMOU.On MMOU, retrieval on the OmniVideo-100K checkpoint raises accuracy from 70.1 to 71.6; Table 7 describes the concurrent-system comparison.Source: §B.2 · p. 20, Table 7 · p. 21

B.3 THE SAME-STACK OFFICIAL-RECIPE BASELINES

• Where published pipelines cannot be reproduced bit-for-bit, the authors rerun the official input recipe on their stack and compare LEAP against that same-stack baseline.On VideoOdyssey, the reimplementation uses the official audio-montage recipe; the authors note departures from the official pipeline.Source: §B.3 · p. 21

B.4 THE FRAME-MATCHED WHOLE CLIP

• Without answer training, LEAP significantly leads the frame-matched whole clip on TraceAV and MMOU on the full denominator, and LEAP leads on TraceAV, LVOmniBench, and MMOU on both denominators.The frame-matched control uses up to 288 frames; clips that do not fit are scored wrong for that control.Source: §B.4 · p. 22, Table 8 · p. 22

B.5 THE CAUSAL-ACCESS PROTOCOL ON STREAMARENA

• LEAP’s ranked windows cover annotated evidence on nearly three quarters of historical-retrospection questions, versus about half for uniform windows.Raw-media answers significantly outperform transcript-only answers in all but the farthest distance bin; transcript-index localization removes nearly all localization time but leaves decoding and answering.Source: §B.5 · pp. 23–25, Table 9 · p. 23, Figure 6 · p. 25

B.6 CONTAMINATION

• No benchmark video matches a training-set video at the audited identifier level.The audit also used perceptual-frame and audio-fingerprint checks where source media were available.Source: §B.6 · p. 25

C THE RETRIEVAL STAGE

• Replacing retrieved windows with the whole clip significantly lowers accuracy on TraceAV and VideoOdyssey; the localization LoRA and transcript outline also improve results on specified benchmarks.The localization LoRA improves over the base selector on LVOmniBench, VideoOdyssey, and MMOU; the outline gains on TraceAV, VideoOdyssey, and MMOU.Source: C THE RETRIEVAL STAGE · p. 25, Table 10 · p. 26

C.1 SELECTION INTERVENTIONS: THE RETAINED-BLOCK COUNT

• Adding blocks raises evidence coverage but not necessarily accuracy; at the deployed count, equally spaced blocks significantly reduce VideoOdyssey coverage and accuracy.The retained-block-count figure plots accuracy and evidence coverage, with paired accuracy differences against the deployed count.Source: §C.1 · p. 27, Figure 7 · p. 27

C.2 THE TRANSCRIPT CHANNEL: WHAT IT RETAINS AND WHAT IT ANSWERS

• On TraceAV, transcript selection is non-inferior to media selection within a 2.5 pp margin for evidence recall, while on CG-Bench mini it retains 8.40 pp less visual evidence and answers 1.80 pp less accurately.The TraceAV recall difference rises by +11.68 pp [ +5.80, +17.41 ] from evidence with no speech to entirely spoken evidence.Source: §C.2 · pp. 27–28, Table 12 · p. 27

C.3 THE TRANSCRIPT OUTLINE AT THE ANSWER PASS

• On MMOU, the transcript outline’s significant accuracy gain lands on questions whose annotated evidence is uncovered by selected windows; on TraceAV, gains concentrate in spatiotemporal localization.For truncated outlines, uniform allocation costs 3.57 points on VideoOdyssey under LEAP and 1.81 points on MMOU under LEAP without answer training.Source: §C.3 · pp. 28–29, Table 14 · p. 28, Table 13 · p. 28

C.4 THE WINDOW GRID

• Window-count and width sweeps show benchmark-dependent effects: Qwen3-Omni’s VideoOdyssey accuracy rises until about six windows, while 150 s windows significantly lower CG-Bench accuracy.On CG-Bench, wider windows overlap human clue intervals more at matched count yet answer less accurately; Figure 8 plots accuracy by answer-pass window count.Source: §C.4 · p. 29, Figure 8 · p. 29

C.5 WHERE THE WINDOWS SIT: THE RANDOM-PLACEMENT CONTROL

• LEAP beats random placement by 1.41 points on TraceAV and 4.99 on VideoOdyssey when window count and width are held fixed.The control changes window positions while keeping the deployed selection’s other settings.Source: §C.5 · p. 30

C.6 THE BLOCK-RANKING SCORE

• Ranking has room to act only when clips tile more than three blocks; equal spacing changes picked blocks on 95.5% of VideoOdyssey questions and costs accuracy significantly there.Elsewhere, equal spacing changes picked blocks on at most 51% of questions.Source: §C.6 · p. 30
• 23.07 points of paired evidence coverage separate ranking from equal spacing on VideoOdyssey, despite similar window seconds per question.The gap widens to 29.49 points when annotated evidence covers under a tenth of the clip.Source: §C.6 · p. 30
• Overlap-fraction BCE selects annotated evidence less often than Eq. 3 cross-entropy, while the resulting answer accuracies remain within a point on TraceAV and VideoOdyssey.The highest-ranked block contains VideoOdyssey evidence 15.09 points less often; top-three coverage is 13.87 points lower.Source: §C.6 · p. 30
• Replacing the strongest-window maximum with a mean costs tens of coverage points, while removing the shared average changes coverage by single digits.The mean substitution also costs coverage on questions without a partial tail block.Source: §C.6 · p. 31
• Trial-answer scoring adds a second pass on every block, agrees with the maximum-window-logit ranking on only 30% of eligible questions, and covers less evidence than the base selector.The trained adapter answers more accurately than trial-answer scoring on LVOmniBench, VideoOdyssey, and MMOU.Source: §C.6 · p. 31

C.7 SELECTION HITS CONVERT TO CORRECT ANSWERS: CG-BENCH AND THE BASE-SELECTOR CONTROL

• 57.50 against 49.53: LEAP significantly beats the whole clip on CG-Bench mini with the identical answer LoRA.Under the official prompt and scoring scripts, the scores are 56.57 for LEAP and 50.42 for the whole clip.Source: §C.7 · p. 31
• Selected windows overlap the human-labeled clue interval on 72.3% of questions, and LEAP’s paired advantage over the whole clip is concentrated in this hit stratum.The paired flips favor LEAP much more when windows hit than when they miss.Source: §C.7 · p. 31, Figure 9 · p. 32
• CG-Bench’s official aggregate gives LEAP CRR=0.892 versus 0.768 for the whole clip, using a clue accuracy of 64.47.CRR is defined as min(long-acc, clue-acc)/clue-acc.Source: §C.7 · p. 31
• Across MMOU, VideoOdyssey, and OmniVideoBench, accuracy gains concentrate when localization training converts a final-window miss into a hit and reverse when it loses a hit.Final-window coverage headroom spans 8.55 to 46.42 points, and paired accuracy gain per coverage point gained spans 0.18 to 0.44.Source: §C.7 · p. 32

C.8 AUDIO AT BOTH STAGES OF THE PIPELINE

• 0.43 median window-set Jaccard: silencing localization audio returns substantially different selections on TraceAV.The IQR is 0.29–0.64.Source: §C.8 · p. 32
• Muting localization audio significantly reduces retained annotated evidence, with a larger loss for hearing-borne events than for visual events.The audio-specific excess loss is significant; this comparison scores selected spans without an answer pass.Source: §C.8 · p. 32, Table 16 · p. 32
• With selected windows fixed, removing answer-pass audio and the transcript outline costs most on cross-modal and hearing-required classes and least on vision-only questions.TraceAV’s labels distinguish hearing, seeing, both, and fabricated-premise questions.Source: Table 16 · p. 32, §C.8 · p. 33
• On Video-MME, muting audio and the transcript outline costs several times more accuracy in the 30–60-minute bucket than below five minutes.The length analysis also reports audio as load-bearing on LVBench and Video-MME.Source: §C.8 · p. 33

D THE ANSWERING STAGE AND THE CROSS-BACKBONE TRANSFER

• On MiniCPM-o 4.5, localization training raises final-window coverage on VideoOdyssey and MMOU and significantly improves paired accuracy on both.On MMOU, the accuracy gain from this swap exceeds that on the larger backbone.Source: D THE ANSWERING STAGE AND THE… · pp. 33–34, Figure 11 · p. 34
• LEAP leads MiniCPM-o 4.5’s published results in every MMOU test-15K duration bucket, with longest-video accuracy near its overall accuracy.The published protocol uses five option-order shuffles and selects among prompt variants; LEAP uses neither.Source: D THE ANSWERING STAGE AND THE… · p. 34, Table 17 · p. 34, Figure 12 · p. 35
• LEAP leads both readings against two systems in its size class, including the released OmniAgent-RL-7B checkpoint under its own evaluator and TraceAV’s harness.The two readings use different evaluation harnesses and scoring setups.Source: D THE ANSWERING STAGE AND THE… · p. 34

E.1 ACCURACY ACROSS VIDEO LENGTH

• On MMOU, LEAP accuracy rises with duration; past 30 minutes it holds level while its lead over the whole clip widens.The compared systems are evaluated on the same questions.Source: §E.1 · p. 35
• LEAP leads the context-filled montage on every LVBench bucket and the whole clip on every CG-Bench and Video-MME bucket, with a larger lead at longer Video-MME durations.The Video-MME length panel covers native-audio runs; the TraceAV panel covers only questions within the whole-clip limit.Source: §E.1 · pp. 35–36, Figure 13 · p. 35

E.2 INSIDE VIDEOODYSSEY: CERTIFICATE LENGTH

• At short certificates, LEAP’s selected windows hit the annotated span far more often than random windows and answer significantly more accurately; at the longest spans it still beats the context-filled montage.The authors attribute the short-span pattern to evidence locality and the long-span pattern to context pressure; past fifteen minutes, random placement hits certificates about as often as LEAP.Source: §E.2 · pp. 36–37
• LEAP’s selected windows trail the certifying span significantly in the two shortest bins, with the gap closing as certificate length widens.Past fifteen minutes, the certifying span is longer than the eleven minutes of audio in the official-style montage.Source: §E.2 · p. 37, Figure 14 · p. 37

E.3 INSIDE VIDEOODYSSEY: THE TASK TYPES

• LEAP’s per-task accuracy exceeds the same-stack baseline on every VideoOdyssey task type except Counting.The comparison is on the benchmark’s audio-visual track; questions annotated with multiple task types are scored under each.Source: §E.3 · p. 37, Table 20 · p. 39

E.4 THE LEAD OVER THE MONTAGE ACROSS ITS AUDIO COVERAGE

• With the answer LoRA on both systems, LEAP significantly leads the context-filled montage in every audio-coverage bin; with base weights, it leads in every bin, significantly in the second and third.The bins are based on the montage’s own audio coverage, which falls with duration.Source: §E.4 · p. 37, Figure 15 · p. 38

E.5 WHERE THE EVIDENCE SITS IN THE CLIP

• On MMOU, LEAP’s lead over the whole clip grows when evidence lies past the tenth minute; on CG-Bench it leads in every bucket, and on VideoOdyssey it does not decay across hour-spanning evidence positions.Figure 16 buckets timestamped questions by whether the annotated evidence span’s start, midpoint, or end is used as its position.Source: §E.5 · p. 37, Figure 16 · p. 38