PaperlayerView exact revision on arXiv

Source-linked AI summary

OACap: Caption-Guided Active Observation for Detailed Video Captioning

Hao Shi, Hong Yan, Xiaoyang Zhang, Qing Cui, Chao Huang

arXiv:2610.29327v2cs.CVcs.CLcs.MAcs.MM

TL;DR

Open-ended detailed captioning must decide what evidence to acquire without a query that specifies a perceptual target. OACap uses caption gaps to direct observations and revisions, and its captions improve understanding across matched benchmarks; FSCap-Bench assesses factual coverage and semantic recoverability.

  • Problem

    Open-ended detailed captioning must determine what evidence to seek without a query specifying a perceptual target, while a global pass can miss details and text-only refinement cannot recover unseen facts.

    Source: §1 · p. 1

  • Method

    OACap treats its evolving caption as a testable account, directing targeted tool observations for missing or unsupported information and revising the caption with returned evidence.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    OACap improves caption-based understanding across five matched backbones on Video-MME, ActivityNet-QA, and VCapsBench, while CSFS responds correctly to controlled perturbations in at least 93% of examples in every condition.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across matched backbones, OACap captions preserve more information for downstream question answering and achieve higher factual comprehensiveness on FSCap-Bench, with CSFS and CSRS providing complementary views.

    Source: §7 · p. 9

    OACap uses additional observation and model calls, and the comparisons do not establish a quality–cost Pareto frontier against equal-compute alternatives.

    Source: §6 · p. 9

Abstract

from arXiv · show

Unlike question answering, where a query specifies a perceptual target, open-ended detailed captioning must determine what evidence to acquire and how to integrate it into a coherent global description. We propose Observation-Adaptive Video Captioner (OACap), a caption-centered agent that turns unsupported or missing information into targeted observation requests, uses spatiotemporal tools to acquire evidence, and iteratively verifies and revises the caption. OACap captions consistently improve performance across general video-understanding benchmarks using a fixed text-only reader. To better diagnose the information they preserve, we introduce Factual-Semantic Captioning Bench (FSCap-Bench). Its Core-Support Factuality Score (CSFS) combines claim precision with importance-aware fact coverage and responds as expected to four controlled perturbation types on 93.0-98.3% of valid caption pairs. Its Cross-Modal Semantic Round-Trip Score (CSRS) measures semantic preservation through a video-caption-video-caption round trip by matching atomic facts from source and recovered captions. Together, they connect downstream textual-memory utility with factual and semantic preservation.

1 Introduction

Detailed captioning must decide what evidence to seek without a query specifying a perceptual target, and text-only refinement cannot recover unseen facts. OACap makes the evolving caption guide targeted observation and revision, while FSCap-Bench evaluates factual and semantic preservation.

Introduction

  • Open-ended detailed captioning must preserve entities, attributes, state changes, and temporal relations, but a global pass can miss details and text-only refinement cannot recover unseen facts.Unlike VideoQA, captioning provides no query specifying what evidence to seek.Source: §1 · p. 1
  • OACap treats its evolving caption as a testable account, distinguishing gaps resolvable from existing evidence from gaps that call for targeted tool observations.Returned evidence supports a whole-caption revision, which determines what to inspect next.Source: §1 · pp. 1–2
  • FSCap-Bench evaluates atomic claims against human-verified Core and Support facts, with CSFS measuring claim precision and importance-aware coverage and CSRS measuring recoverability through a video–caption round trip.The benchmark contains 498 videos from five public benchmarks and 996 verified observation challenges.Source: §1 · p. 2, Figure 1 · p. 2
  • Across Video-MME, ActivityNet-QA, and VCapsBench, OACap improves caption-based understanding across five matched backbones, and CSFS responds correctly to controlled additions, hallucinations, and omissions in at least 93% of examples in every condition.The paper evaluates captions as question-agnostic textual memory using a fixed text-only reader.Source: §1 · pp. 1–2

2 Related Work

Prior work improves detailed captions through supervision, tool-assisted data generation, and self-improvement, while visual agents commonly use external queries or task states to guide observation. OACap instead uses gaps in the evolving caption to direct evidence acquisition and revision.

Detailed video captioning

Agentic visual understanding

Factual and semantic caption evaluation

3 OACap: Caption-Guided Active Observation

OACap interleaves caption generation with evidence acquisition: it identifies gaps in the evolving description, chooses observations to resolve them, and revises the caption using returned evidence. The updated caption then guides the next inspection.

3.1 Agent Formulation

  • OACap frames detailed captioning as interleaved observation and generation, using the evolving caption to balance broad video coverage against verification of proposed details.Claim verification and coverage inspection route gaps to spatial, temporal, textual, or externally grounded observation; returned evidence determines which claims to retain, remove, or rewrite.Source: §3.1 · p. 3

3.2 Evidence State and Gap Diagnosis

  • The evidence state tracks observations and the draft caption, and coverage inspection compares the draft with coarse context and accumulated evidence to turn missing or underspecified content into concrete evidence questions.If existing evidence resolves a gap, OACap edits the caption; otherwise it requests a discriminating observation.Source: §3.2 · p. 3

3.3 Observation-Adaptive Acquisition

  • For each prioritized gap, OACap selects a tool, temporal range, granularity, and evidence question; keyframes, crops, dense sampling, clip analysis, OCR, or retrieval address different evidence needs.External evidence is stored separately and cannot establish video actions or override contradictory visual evidence; benchmark references are not inference inputs.Source: §3.3 · p. 4

3.4 Evidence-Grounded Revision

  • Returned observations are atomized, deduplicated, temporally aligned, and checked against existing evidence; agreement supports claims, while conflict can trigger another targeted action.The loop stops when important content and temporal relations are resolved, the budget is exhausted, or expected gain is too small; unresolved claims are hedged or omitted.Source: §3.4 · p. 5

4 FSCap-Bench

FSCap-Bench evaluates caption factual comprehensiveness and cross-modal recoverability using curated, human-verified Core and Support facts. CSFS and CSRS provide complementary measures of supported coverage and semantic preservation through a reconstruction cycle.

4 FSCap-Bench

  • FSCap-Bench evaluates factual comprehensiveness with CSFS and cross-modal recoverability with CSRS.Its fact sets are curated and annotated as Core and Support facts.Source: §4 · p. 5, Figure 4 · p. 6

4.1 Construction

  • 498 videos pass four sequential filters, yielding 996 verified observation challenges across five public benchmarks.Annotators verify time-aligned, importance-stratified Core and Support facts against each retained video.Source: §4.1 · p. 5

4.2 Core–Support Factuality Score

  • CSFS combines claim precision with importance-aware Core coverage and bounded credit for complementary Support facts.Each case contains Core and Support facts, preventing incidental detail alone from earning a high aggregate score.Source: §4.2 · pp. 5–6
  • The CSFS judge applies consistent decomposition and matching rules, gives zero credit to contradicted or unsupported claims, and counts each reference fact at most once.Unsupported claims reduce precision, while omissions reduce coverage; contradiction and consistency rates remain diagnostics rather than score factors.Source: §4.2 · p. 6

4.3 Cross-Modal Semantic Round-Trip Score

  • CSRS matches atomic facts from caption C and recovered caption C′ after a text-to-video model renders C as V′.The same captioner describes C and C′; CSRS measures their agreement, not correctness against the source video, so repeated unsupported facts can still match.Source: §4.3 · p. 6

5 Experiments

The experiments test OACap captions as reusable textual memory, validate CSFS against controlled edits, and measure cross-modal recoverability with CSRS.

5 Experiments

  • The caption-as-memory protocol uses one whole-video caption, a fixed text-only reader and question prompt, and a shared output budget and abstention policy.Captions are generated without downstream questions; the reader receives only each caption and question, and abstentions count as incorrect.Source: §5 · p. 7, §5.1 · p. 7

5.1 Caption as Memory

  • OACap improves every matched Video-MME split, gains +0.11 to +1.83 on ActivityNet-QA, and raises VCapsBench AR and CR while lowering IR.On VCapsBench, OACap/Kimi-K2.6 achieves the best AR (71.37) and IR (19.57), while OACap/Qwen3.8-27B achieves the best CR (89.32).Source: §5.1 · p. 7

5.2 CSFS Validation

  • CSFS follows all intended directions: correct detail raises its score from 0.9291 to 0.9694, while hallucinations, Core deletion, and Support deletion lower it to 0.8752, 0.8836, and 0.9039.The controlled perturbations use clean captions from 300 Video-MME short videos; three edits that failed to introduce unsupported content were excluded from the hallucination condition.Source: §5.2 · pp. 7–8, Table 2 · p. 8
  • CSFS directional accuracy is 97.3%, 98.3%, 93.0%, and 93.3% across correct additions, hallucinations, Core deletions, and Support deletions, respectively.Tested PAC-S, EMScore, and FactVC configurations truncate long captions to 77 tokens; the validation does not establish human agreement or invariance to alternative atomic decompositions.Source: §5.2 · p. 8

5.3 CSRS Validation

  • OACap improves CSFS by 1.94, 5.20, and 20.75 points for Qwen3.5-27B, Qwen3.5-122B, and Qwen3.8-27B, respectively.Table 3 compares Direct Caption with OACap Pipeline under matched backbones.Source: §5.3 · p. 8, Table 3 · p. 8
  • CSRS is evaluated on 33 Video-MME-sourced FSCap-Bench videos shorter than two minutes, using a fixed video–caption–video–caption reconstruction channel.The protocol tests generator–captioner combinations at 10 seconds and, for LTX, also at 60 seconds.Source: §5.3 · p. 8, Table 4 · p. 9
  • At 10 seconds, LTX yields higher CSRS than Wan with both captioners; extending LTX to 60 seconds raises CSRS to 0.254 with Qwen3.8-27B and 0.291 with Qwen3.8-Max.LTX-60s/Qwen3.8-Max has the best overall CSRS (0.291), while LTX-60s/Qwen3.8-27B has the best precision (0.362) and lowest drift (0.638); absolute information loss remains 70.9%–83.2%.Source: §5.3 · pp. 8–9, Table 4 · p. 9

6 Limitations

OACap uses additional observation and model calls, but the comparisons do not establish a quality–cost Pareto frontier against equal-compute alternatives. Evaluation measures also have scope limits and inherit errors from their component models.

6 Limitations

  • OACap trades additional observation and model calls for caption quality, but no quality–cost Pareto frontier is established against equal-compute uniform sampling or alternative multi-stage policies.Routing and stopping also inherit perceptual and calibration errors from models that diagnose caption gaps and interpret tool outputs.Source: §6 · p. 9
  • Caption-as-memory scores depend on reader priors and question coverage, while CSFS depends on annotation completeness, importance assignments, atomic decomposition, and judge reliability.FSCap-Bench draws from datasets also used in downstream evaluation, so independent-generalization claims require a video-level overlap audit; CSRS currently covers only 33 generation-compatible videos and combines errors across its components.Source: §6 · p. 9

7 Conclusion

OACap treats captioning as active evidence acquisition, using caption deficiencies to guide observation and revision. Its captions preserve more information for downstream question answering and achieve higher factual comprehensiveness on FSCap-Bench.

7 Conclusion

  • Across matched backbones, OACap captions preserve more information for downstream question answering and achieve higher factual comprehensiveness on FSCap-Bench.CSFS measures supported claims and importance-aware coverage, while CSRS measures information recoverability through a fixed cross-modal channel.Source: §7 · p. 9
  • OACap treats a detailed caption as both the output and control state of video understanding, turning caption deficiencies into targeted tool calls and evidence-grounded revisions.It separates gaps repairable from existing evidence from gaps requiring new observation.Source: §7 · p. 9
  • CSFS and CSRS offer complementary views of caption quality: supported claims and importance-aware coverage against video-grounded facts, and recoverability through a fixed cross-modal channel.Source: §7 · p. 9

A.1 Benchmark Composition

The benchmark manifest records the provenance and filtering history of 10,716 candidate videos. The selected distribution is short-video-skewed and describes the evaluation set, not source-dataset prevalence.

A.1 Benchmark Composition

  • The manifest records 10,716 unique video IDs, item-level media metadata, stage reachability, measured gate outputs, terminal decisions, and F4 verification records.Video durations range from 2.08 to 2,029.63 seconds, and figure 3 counts are computed from the manifest.Source: §A.1 · p. 12
  • 4.6% of candidates survive all four filters; retention ranges from 2.0% for Video-MMMU to 10.0% for Video-MME across the five listed sources.The exact retained counts are 60/600 for Video-MME, 79/800 for ActivityNet-QA, 228/5,705 for VCapsBench, 125/3,310 for MVBench, and 6/301 for Video-MMMU.Source: §A.1 · p. 12
  • The selected set has a 14.7-second median duration, with 433/498 videos under two minutes, and its source-dataset retention does not estimate challenge prevalence.The largest merged label groups are Visual Puzzles, Human Activities, Time-lapse & Processes, Perception & Reasoning, and General Web Videos.Source: §A.1 · p. 12

A.2 Four-Stage Filtering

The benchmark is filtered sequentially for visual usability, semantic content, temporal coherence, and verifiable observation challenges. Passing videos receive time-aligned, importance-stratified facts verified against the source video.

A.2 Four-Stage Filtering

  • F1 rejects videos failing visual-quality thresholds for quality, blur, resolution, black frames, or invalid frames; local processing rejects 7,645 and leaves 3,071.Its score combines sharpness, detail, non-blurred fraction, bitrate per pixel per frame, and resolution.Source: §A.2 · p. 12
  • F2 removes static or image-like cases and requires VLM thresholds for density, event count, variety, and banned content, leaving 1,099 candidates.These exclusions can remove informative instructional or screen content; low retention alone does not validate rejection precision.Source: §A.2 · p. 12
  • F3 requires event density≥45, montage noise≤0.5, causality≥3, and entity continuity≥2, rejecting 202 videos and sending 897 to F4.Strong-causality gating may exclude coherent events without an explicit causal chain.Source: §A.2 · p. 12
  • F4 retains two verified challenges per passing video; 498 videos pass, yielding 996 records, most often involving state changes, local attributes, or on-screen text.Among 897 F4 candidates, 399 are rejected; challenge types may overlap within a video.Source: §A.2 · p. 12

A.3 Fact Annotation and Quality Control

  • Annotators verify automatic fact proposals against video, add missing important facts, and adjudicate ambiguous temporal boundaries and Core/Support assignments.Facts cover attributes, relations, text, actions, motion, state changes, temporal order, long-range relations, and audio-visual evidence; external knowledge requires a visible anchor.Source: §A.3 · p. 12

B Evaluation Prompts and Round-Trip Configuration

The appendix specifies a question-agnostic single-pass caption prompt and a CSRS round-trip evaluation using paired source and recovered captions. It details the prompt constraints, fact extraction, semantic matching, and score aggregation.

B.1 Single-Pass Direct-Caption Prompt

  • All single-pass captioners receive the same question-agnostic prompt for one temporally structured whole-video description.Source: §B.1 · p. 12
  • The prompt requests a faithful caption in chronological order, with a concise summary and connected prose blocks for meaningful video phases.Source: §B.1 · p. 13
  • Caption blocks use approximate absolute timestamps, remain chronological and non-overlapping, and split when meaningful phases change without creating a block for every camera cut.Brief events may be described within the surrounding phase rather than falsely extended.Source: §B.1 · p. 13
  • The prompt prioritizes information-rich developments, distinct stages, and video-supported details while excluding speculation, external knowledge, and process commentary.Source: §B.1 · p. 13

B.2 CSRS Configuration and Fact Extraction

  • Round-trip configurations use 33 Video-MME videos; the 10-second setting crosses two video generators with two captioners, while the 60-second setting uses LTX combinations.The evaluation contains six configurations and 198 source–recovered caption pairs; each source caption is rendered into a video and captioned again by the same captioner.Source: §B.2 · pp. 13–14
  • A fixed atomizer extracts exhaustive, self-contained observable facts from source and recovered captions, splitting composite descriptions into independently matchable assertions.Source: §B.2 · p. 14
  • A fixed matcher aligns semantically equivalent source and recovered facts, accepting paraphrase and granularity differences while enforcing one-to-one credit.Different objects, events, or attributes and merely inferentially related statements do not count as equivalent.Source: §B.2 · p. 14
  • Per-video CSRS is the harmonic mean of matched-fact recall and precision, with Loss and Drift reported as their respective complements and configuration values macro-averaged over 33 videos.Source: §B.2 · p. 14

Related papers

Loading 2610.29327v2…