PaperlayerView exact revision on arXiv

Source-linked AI summary

LazySloth: Bounded LLM-based Lazy Tree Search for Fast Long Video Comprehension

Arka Mukherjee, Kaleen Shrestha, Larissa Zhu, Maja Matarić

arXiv:2609.37426v1cs.CVcs.AI

TL;DR

Long-video comprehension methods either spend heavily on extensive VLM captioning or retrieve efficiently using embeddings that lose temporal and fine-grained information. LazySloth searches a hierarchical video tree and captions only relevant branches; across four benchmarks, it is faster than agentic methods while matching or exceeding specialized VLMs and RAG in accuracy.

  • Problem

    Extensive VLM captioning is costly and scales with video duration, while efficient multimodal retrieval relies on lossy embeddings that discard temporal, motion, and fine-grained detail.

    Source: §1 · pp. 1–2

  • Method

    LazySloth uses a top-down tree search that lazily captions only the branches deemed relevant, bounding captioning cost by descent depth rather than video length.

    Source: §1 · p. 2

  • Results

    2.9–8.3× faster than existing agentic methods, LazySloth matched or exceeded specialized video-understanding VLMs and RAG pipelines in accuracy across four benchmarks and two open-source VLMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LazySloth speeds long-video retrieval without substantial task-performance loss, and its lazy construction remains close in accuracy to eager construction.

    Source: §5 · p. 7

    The method assumes relevant content can be localized to subtrees, but the authors observed poor backtracking and navigation to other branches; testing also covered only two base models without multiple-run uncertainty quantification.

    Source: §6 · p. 7

Abstract

from arXiv · show

Modern vision-language models (VLMs) have shown promising results in long-video understanding due to the rich semantic information they can capture. However, most methods focus on coarse captioning of extracted image frames that are computationally inefficient and require models with large context windows. While past work has explored efficient methods through multimodal retrieval-augmented generation (RAG), they rely on lossy embeddings that lose temporal context and fine-grained detail. Few works to date have investigated how VLM-based query-relevant information retrieval can be optimized. We introduce LazySloth, an efficient tree-based search method that speeds up video comprehension and retrieval tasks 2.9-8.3x (compared to existing agentic methods) through bounded captioning of portions of the video considered irrelevant by a VLM of the video. Compared to contemporary specialized video-understanding VLMs and RAG-based methods, LazySloth achieved similar or better final task accuracy across two recent open-source base VLMs--Gemma 4 31B and Qwen3.6 27B--across four benchmarks. LazySloth reduced the gap between the base open-source model and a closed-source model, GPT-4o. Ablations showed that replacing VLM scene understanding with CLIP-based retrieval cost 8.8-19.9% in accuracy, while lazy tree construction matches eager construction at a fraction of the captioning cost. With LazySloth, we demonstrate the possibility of faster long-video comprehension without substantial loss in performance.

1 Introduction

Long-video methods face a trade-off between costly, extensive VLM captioning and efficient retrieval that loses temporal and fine-grained detail. LazySloth uses query-guided hierarchical search to bound captioning, achieving faster comprehension at comparable or better accuracy across four benchmarks.

1 Introduction

  • Caption-every-frame methods require one VLM call per unit of video, so cost grows linearly with duration even when a question concerns only a brief event.The paper identifies this wasted captioning as the bottleneck because VLM calls are more expensive than other system components.Source: §1 · pp. 1–2
  • Contrastive retrieval is cheap but compresses frames into lossy vectors that discard temporal, motion, and fine-grained detail.The paper motivates optimizing VLM-based retrieval as a way to avoid this trade-off.Source: §1 · p. 2
  • LazySloth builds a top-down tree and captions only the branch the searcher deems relevant, bounding captioning cost by descent depth rather than video length.The method grows logarithmically where prior methods grow linearly.Source: §1 · p. 2
  • 2.9–8.3× faster than existing agentic methods, LazySloth matched or exceeded specialized video-understanding VLMs and RAG pipelines in accuracy across four benchmarks and two open-source VLMs.It also narrowed the accuracy gap between the tested open-source models and GPT-4o.Source: §1 · p. 2
  • The evaluation compares four long-video benchmarks and two open base VLMs with specialized models, RAG pipelines, and agentic baselines, while ablations isolate hierarchy, laziness, and scene-understanding contributions.The introduction describes these as evaluation and ablation contributions, without reporting their individual findings here.Source: §1 · p. 2

2 Related Work

Related work covers general-purpose and specialized video VLMs, long-video benchmarks, RAG, and agentic search. The paper positions its work against limits in frame sampling, retrieval representations, and optimization of VLM-based search.

General Purpose Video VLMs

  • Open VLMs process uniformly sampled frames within context limits, with aggressive truncation or down-sampling for longer or higher-resolution videos; smaller open models degrade after roughly 30 frames.Context-as-memory did not provide usable memory in the authors’ experiments, motivating retrieval systems that support hundreds of frames.Source: General Purpose Video VLMs · p. 2

Specialized VLMs for Long Videos

  • Specialized models improve video understanding through vision-centric training data or architectural, training, and temporal-grounding innovations, while frame-sampling and memory-as-context limitations remain.VideoLLaMA 3 achieved 2–3% gains on Video-MME and MLVU.Source: Specialized VLMs for Long Videos · p. 2

2.2 Long Video-Understanding Datasets and Benchmarks

  • Long-video benchmarks evaluate multiple-choice comprehension across web, egocentric, and increasingly long-form content, with the paper studying Video-MME, LongVideoBench, LVBench, and EgoSchema.The cited discussion distinguishes short-comprehension and week-long egocentric benchmarks.Source: §2.2 · pp. 2–3

RAG-Based Methods

  • Long-video RAG methods include text-based LightRAG and HippoRAG and multimodal Video-RAG; the paper contrasts their lossy embeddings with semantic LLM-based scene understanding.The cited methods are discussed by Yeo et al. (2026).Source: RAG-Based Methods · p. 3

Agentic methods

  • Agentic methods explore coarse-to-fine memory retrieval, separate memory banks, or iterative VLM search, but prior work had not evaluated optimizing VLM-based long-video search without significant performance reductions.The passage names VideoLucy (Zuo et al. 2025), Yeo et al. (2026), and Wang et al. (2025b).Source: Agentic methods · p. 3

3 LazySloth

LazySloth combines a lazily generated hierarchical caption tree, text-based search with backtracking, and multimodal answer reasoning. Its evaluation spans four long-video benchmarks, multiple baseline classes, ablations, and two base VLM configurations.

3 LazySloth

  • LazySloth uses a captioner to build a top-down tree lazily, a searcher to navigate relevant branches, and a reasoner to answer using collected textual evidence and representative frames.The evidence combines the selected caption corpus for temporal context with key frames from the region the search narrowed to.Source: §3 · pp. 3–4
  • Captioning reaches a leaf of at most ℓ frames after d = ⌈log_k(N/ℓ)⌉ levels, with each navigation captioning at most k nodes and final expansion at most ℓ frames.For fixed k and ℓ, the cost simplifies to O(k log_k N), compared with O(N) for dense frame captioning.Source: §3 · p. 4

3.1 Evaluation Benchmarks

  • The evaluation uses four multiple-choice benchmarks scored by exact match, spanning movie clips, web videos, long-form videos, and egocentric videos.The benchmark set is Video-MME, LongVideoBench, LVBench, and EgoSchema.Source: §3.1 · p. 4

3.2 Baselines

  • Baselines include general-purpose and specialized video models, text and multimodal RAG, and agentic methods such as VideoLucy and WorldMM.The general-purpose models were limited to at most 64 sampled frames to fit their context windows.Source: §3.2 · pp. 4–5

3.3 Ablations

  • Ablations replace the searcher with CLIPScore-based greedy best-first retrieval and compare lazy tree construction with an eagerly built full hierarchy while retaining backtracking.These comparisons target VLM scene understanding, lazy construction, and efficiency relative to brute-force methods.Source: §3.3 · p. 5

3.4 Experimental Setup & Models Tested

  • Experiments repeat agentic-method and LazySloth runs with Qwen3.6 27B and Gemma 4 31B, using Nemotron 3 Nano Omni 30B as captioner and 1-FPS video sampling.The models used a 79,104-token context window; experiments ran on four H100, one RTX Pro 6000, and two RTX 3090 GPUs.Source: §3.4 · p. 5

4 Results

LazySloth reports higher accuracy than specialized video-understanding models and RAG baselines on Qwen3.6 27B, alongside substantially faster LVBench inference and behavioral and ablation findings about search depth, caption budgets, and scene understanding.

4 Results

  • On Qwen3.6 27B, LazySloth exceeded the strongest specialized model by 5.2 points on Video-MME, 2.2 on LongVideoBench, 1.5 on LVBench, and 13.3 on EgoSchema.It also beat Video-RAG 7B by at least 6.0 points on the two benchmarks with available results.Source: §4 · p. 5, Table 2 · p. 5

4.1 Efficiency Analysis

  • 194 seconds per LVBench query was LazySloth’s median answer time, versus 611 seconds for VideoLucy and 1,139 seconds for WorldMM.On 140-minute videos, LazySloth was 3× faster than VideoLucy and 7.7× faster than WorldMM; the eager tree was 287× slower than LazySloth.Source: §4.1 · p. 6, Figure 3 · p. 6
  • LazySloth’s median wall-clock time was 2.9–8.3× faster than the baselines, while neither captioning nor reasoning dominated its runtime.WorldMM’s reasoning took 811 seconds; VideoLucy spent 67% of its total time on captioning.Source: §4.1 · p. 6, Figure 4 · p. 6

4.2 Behavioral Analysis

  • For Gemma 4 31B, accuracy differences were zero throughout, consistent with model-based performance differences in Section 4; multi-level tree search did not impact final accuracy.Multi-level tree search likely negatively hurt its long-video capabilities.Source: §4.2 · p. 6
  • A first tree descent raised Qwen3.6 27B accuracy by 25%, but accuracy fell beyond that on Video-MME, LongVideoBench, and LVBench.EgoSchema accuracy declined monotonically; the paper suggests short videos and harder questions as possible factors.Source: §4.2 · p. 6, Figure 6 · p. 7
  • LazySloth identified the correct LongVideoBench window 1.8–2.1× more often than random placement, yet its final window missed the referenced span in 86–91% of correctly answered questions.The analysis covers timestamp-annotated questions answered correctly.Source: §4.2 · pp. 6–7, Figure 7 · p. 7

4.3 Ablation Results

  • Replacing VLM captioning with frozen CLIP reduced accuracy by 8.8–19.9%, while lazy and eager construction with CLIPScore differed by at most 3.1% without consistent direction.The reported pairs were Video-MME 41.5 vs. 43.4, LongVideoBench 40.5 vs. 42.1, and EgoSchema 55.6 vs. 56.4.Source: §4.3 · p. 7, Table 3 · p. 7
  • A separate reasoner improved micro-averaged accuracy by 0.3%, with no clear directionality, and the searcher’s provisional answer was nearly as accurate as the final answer.The authors note that removing the reasoner call could further improve efficiency.Source: §4.3 · p. 7, Table 4 · p. 7

5 Conclusion

LazySloth bounds captioning to regions deemed relevant rather than captioning exhaustively. Across four benchmarks and two open-source VLMs, it was 2.9–8.3× faster while matching or exceeding specialized VLMs and RAG methods in accuracy.

5 Conclusion

  • LazySloth bounds VLM captioning to regions considered relevant, rather than captioning the entire video.Across four benchmarks and two open-source VLMs, it was 2.9–8.3× faster while matching or exceeding specialized VLMs and RAG methods in accuracy.Source: §5 · p. 7

6 Limitations

LazySloth assumes long-video comprehension can be localized to subtrees, but the observed method had poor backtracking and navigation to other branches.

6 Limitations

  • The subtree-localization assumption is a boundary: observed poor backtracking and navigation leave exploration of other branches constrained.The authors also rely on caption quality across resolutions and tested only two base models without uncertainty quantification across multiple runs.Source: §6 · p. 7

Appendices

Table 11 lists formalization symbols used in Section 3 and Algorithm 1.

Appendices

B FPS Sweep Results

On short video datasets, increasing frame sampling did not reliably improve accuracy. The results point to the model’s ability to use supplied frames, rather than context-window capacity, as a constraint and motivate retrieval-based design.

B FPS Sweep Results

  • Qwen 2.5 VL 7B13 fell from 47.0% at 1 FPS to 11.5% with the full frame set on RoomReader, and from 54.0% to 32.7% on SEMPI.Neither model improved monotonically as sampling increased from 1 FPS to every frame.Source: B FPS Sweep Results · p. 10
  • Gemma 4 E4B stayed essentially flat across all four sampling rates.The evaluated short clips fit entirely in VLM context windows at high sampling rates.Source: B FPS Sweep Results · p. 10
  • Because the short clips fit in context, the experiment identifies model use of the available frames—not context-window capacity—as the ceiling and motivates retrieval-based design.The passages do not claim that retrieval resolves this model limitation.Source: B FPS Sweep Results · p. 10

C Captioning Quality Tests

Candidate captioners had broadly similar CLIPScore results apart from Gemma 4 E4B, but their latency varied substantially. The authors selected Nemotron 3 Nano Omni as the fastest candidate that was not a quality outlier and used it across methods.

C Captioning Quality Tests

  • Across six viable captioners, CLIPScore ranged from 0.2331 to 0.2447, while Gemma 4 E4B scored 0.2153.The reported range was roughly 5%.Source: C Captioning Quality Tests · p. 10
  • Caption latency ranged from 0.59 to 2.70 seconds; Nemotron 3 Nano Omni was selected as the fastest captioner that was not a quality outlier.It was used across all methods so the comparison would not advantage or penalize a baseline through captioner choice.Source: C Captioning Quality Tests · p. 10
  • CLIPScore is correlated with average caption length, so Nemotron’s high score despite its second-shortest captions suggests competitive per-word quality; dataset spreads prevent a strict ranking.The passage notes that per-dataset spread within a model is comparable to spread between models.Source: C Captioning Quality Tests · p. 10

D Additional Efficiency Analysis of LazySloth

On LVBench, LazySloth’s inference time grew more slowly with video length than the agentic baselines, and its caption-call count remained comparatively bounded. VideoLucy’s similar call count did not translate to similar speed because its calls captioned dense overlapping windows.

D Additional Efficiency Analysis of LazySloth

  • LazySloth added 2.43 seconds per extra minute of video on LVBench, versus 5.53s/min for VideoLucy and 19.86s/min for WorldMM.The passage reports both baseline differences as statistically significant.Source: D Additional Efficiency Analysis of… · p. 10
  • LazySloth’s caption calls rose from 35 on 40-minute videos to 45 on 140-minute videos, compared with WorldMM’s 97 to 330 and eager trees’ 3,600 to 12,603.The passage attributes LazySloth’s growth to the O(k logk N) bound of Equation (1); the eager-tree count was 280 times higher on the longest videos.Source: D Additional Efficiency Analysis of… · p. 10
  • VideoLucy issued 27 to 54 calls, statistically indistinguishable from LazySloth’s, but ran roughly three times slower because it re-captioned dense, overlapping windows.Its multi-frame captions consumed 67% of total time; each call carried more frames and visual tokens than a LazySloth bunch caption.Source: D Additional Efficiency Analysis of… · p. 10

E Additional Benchmark-wise Behavioral Analysis

Benchmark-wise analyses show that search gains depend on the base model and question type, while first-level search choices favor the opening part of the video. The depth comparison uses matched questions within each search-depth stratum to distinguish method gains from question difficulty.

E Additional Benchmark-wise Behavioral Analysis

  • For Qwen3.6 27B, LazySloth consistently improved performance over direct inference across four benchmarks, with differences statistically significant at higher tree depths, most sharply on EgoSchema and LongVideoBench.Figure 10 scores direct inference on exactly the questions in each LazySloth depth stratum.Source: E Additional Benchmark-wise Behavioral… · p. 10, Figure 10 · p. 13
  • The first descent chose the opening quarter for 49% of Video-MME and LongVideoBench questions, 54% of LVBench, and 64% of EgoSchema—roughly twice the random rate.Figure 9 compares first-branch choices with an empirically measured random baseline.Source: E Additional Benchmark-wise Behavioral… · pp. 11–12, Figure 9 · p. 12
  • LongVideoBench gains were largest for T2O (44.7%), SOS (38.3%), and TOS (37.0%); Video-MME gains were 25.5% for Information Synopsis and 23.2% for Temporal Reasoning.Types answerable from any representative frame gained almost nothing; the passages associate larger gains with questions requiring a moment to be located and reasoned about.Source: E Additional Benchmark-wise Behavioral… · p. 12, Table 10 · p. 13

Related papers

Loading 2609.37426v1…