PaperlayerView exact revision on arXiv

Source-linked AI summary

CausalChapter: Improving Long-Video Chaptering with Interventional Dependency Modeling

Xinran Duan, Guozhang Li, Yaoyao Zhong, Mei Wang, Lizhi Wang, Hua Huang

arXiv:2609.08686v1cs.CVcs.AI

TL;DR

Long-context chaptering is costly and brittle for dense instructional videos, while segment-then-caption introduces boundary error propagation and fragmented cross-chapter context. CausalChapter uses intervention-inspired dependency estimates for boundary localization and context selection, improving temporal localization and description quality on AVLecture while remaining competitive on VidChapters-7M.

  • Problem

    Content-dense instructional videos make holistic long-context chaptering costly, while segment-then-caption introduces boundary error propagation and cross-segment context fragmentation.

  • Method

    CausalChapter uses intervention-inspired masking and removal to estimate prediction-level support, with LCDS for boundary dependency shifts and CSSE for historical context selection.

  • Results

    CausalChapter improves temporal localization and description quality on AVLecture while remaining competitive on VidChapters-7M.

  • Takeaways & Limitations

    Intervention-defined dependency provides task-oriented support signals for smooth boundary detection and coherent cross-segment chapter generation.

  • Takeaways & Limitations

    Intervention-defined dependency measures prediction-level influence rather than real-world causal relations, and augmented AVLecture references may inherit annotation-pipeline stylistic regularities.

Abstract

from arXiv · show

Long-form instructional videos require automatic chaptering to support browsing, navigation, and knowledge access. Recent long-context language models can perform chaptering from textualized video inputs, but they remain costly and brittle for content-dense lecture videos with long transcripts, smooth topic transitions, and detailed chapter outputs. A scalable segment-then-caption paradigm reduces this cost, but introduces two new challenges: boundary error propagation and fragmented cross-chapter context. We propose \textbf{CausalChapter}, an intervention-inspired framework for long-video chaptering that estimates prediction-level influence through lightweight masking and removal interventions. For boundary localization, our Local Dependency Shift module detects drops in predictive dependency between adjacent temporal windows; for chapter description generation, our Cross-Segment Support Selection module reranks historical contexts according to their support for the current prediction. Experiments on long-video chaptering benchmarks show that CausalChapter improves boundary localization, chapter description quality, and cross-chapter coherence.

1 Introduction

Long-video chaptering needs scalable processing because dense instructional content makes holistic modeling costly and context-limited. CausalChapter addresses segment-level boundary errors and fragmented context with intervention-based dependency signals.

  • Motivation: Long-context chaptering becomes costly when instructional videos contain long transcripts, smooth transitions, and detailed descriptions.Context truncation, sparse sampling, or compression may discard evidence needed for boundary localization and description generation.
  • Scalable Chaptering: Segment-then-caption reduces each generation step’s input length but shifts the challenge toward identifying consequential transitions and historical segments.The paradigm first predicts boundaries and then describes each resulting segment.
  • Challenges: Boundary errors can mix adjacent chapter content or omit semantic units, while independent generation can lose definitions, assumptions, and examples needed across chapters.These failures create boundary error propagation and cross-chapter context fragmentation.
  • Context Selection: Similarity-based context retrieval may select relevant-looking segments whose surface relevance does not guarantee prediction utility.Such contexts can repeat current content or induce mechanism drift in instructional videos.
  • Proposed Framework: CausalChapter estimates prediction-level influence through interventions, using LCDS for dependency drops at boundaries and CSSE for support-based context selection.The framework is designed to mitigate boundary error propagation and context fragmentation in scalable chaptering.

2 Related Work

Prior work established long-video chaptering benchmarks, long-context modeling, and context augmentation, while CausalChapter selects context by observable predictive influence rather than surface similarity.

  • Video Chaptering: VidChapters-7M provides a large-scale user-annotated chapter dataset with chapter generation and grounding tasks.It supports evaluation of long-form video chaptering.
  • Video Chaptering: Chapter-Llama represents videos with timestamped ASR transcripts and frame captions, then jointly predicts boundaries and titles using a long-context LLM.Its single-forward-pass formulation exemplifies holistic long-context chaptering.
  • Context Augmentation: Existing context methods use memory propagation, retrieval augmentation, or evidence selection based on signals such as semantic, temporal, or visual similarity.These approaches motivate context augmentation for video-language generation.
  • Context Selection: Earlier definitions or premises may support a chapter despite low similarity, whereas highly similar segments may merely repeat current content.CausalChapter therefore treats retrieval as candidate construction and bases final selection on observable influence.
  • Causal Video Reasoning: Prior causal video reasoning studies address bias reduction, language priors, event relations, or causal structure discovery in video understanding.CausalChapter differs by using lightweight intervention-based signals for chaptering support.

3 Method

CausalChapter uses a segment-then-caption pipeline enhanced with intervention-inspired dependency estimates to localize chapter boundaries and select historically supportive contexts for generation. Its output is a temporally grounded sequence of chapter descriptions.

  • Task Formulation and Backbone: CausalChapter first predicts consecutive chapter segments, then generates a description for each segment using an LLM-based generator.The backbone encodes multimodal semantic units, predicts boundary probabilities, and constructs generation inputs from segment text, visual features, temporal prompts, and pooled representations.
  • Intervention-Based Modeling: Predictive dependency is treated as an operational measure of prediction-level support rather than a claim about the real-world causal structure of video content.The framework uses lightweight masking, perturbation, or removal interventions to estimate observable changes in model predictions.
  • Local Dependency Shift: Local Dependency Shift improves boundary localization by identifying drops in predictive dependency between neighboring temporal windows.For each candidate boundary, a dependency predictor reconstructs the following window from a preceding window; masking preceding units measures their intervention effects, which are compared with local reference dependencies.
  • Cross-Segment Support Selection: Cross-Segment Support Selection ranks historical contexts by how strongly their removal changes the generator’s predictive distribution for the current chapter.Candidate contexts include temporally neighboring and semantically retrieved segments, while intervention scoring is applied only within this compact candidate set.
  • Cross-Segment Support Selection: The final description is generated from the current segment, its first-pass description, and the top-ranked supportive historical contexts.This intervention-driven selection replaces final context choice based solely on similarity and keeps additional computation proportional to candidate-set size.

4 Experiments

CausalChapter is evaluated on AVLecture and VidChapters-7M against dense-captioning, long-video chaptering, and LLM baselines, with ablations isolating its interventional modules. It improves boundary localization and description generation, while sensitivity analyses examine window and context-selection settings.

  • Main Results: CausalChapter improves AVLecture performance over Chapter-Llama, raising CIDEr from 99.78 to 110.12 with LLaMA-3.1-8B and from 95.36 to 116.88 with Qwen2.5-7B.With Qwen2.5-7B, F1@30 and tIoU also rise from 63.93% and 62.18% to 72.97% and 69.95%.
  • Main Results: CausalChapter generalizes to VidChapters-7M, achieving competitive or better generation and localization performance than long-video LLM baselines with a compact backbone.VidChapters-7M contains more diverse open-domain videos with user-annotated chapter boundaries and titles.
  • Ablation Studies: LCDS improves F1, BS@30, and tIoU from 52.95%, 59.61%, and 66.36% to 56.72%, 63.02%, and 69.95%, respectively.The ablation uses the same Qwen2.5-3B backbone and training data across variants.
  • Ablation Studies: CSSE leaves boundary predictions unchanged while increasing CIDEr from 88.39 to 98.40 and SODA_c from 10.84 to 12.63.The full model reaches CIDEr 104.59, METEOR 17.87, and SODA_c 12.73.
  • Interventional Dependency Analysis: The intervention-defined dependency score outperforms similarity-drop and contrastive alternatives on all reported LCDS metrics, increasing F1 from 52.95% to 56.72% and tIoU from 66.36% to 69.95%.Similarity drops improve some threshold-based metrics but reduce tIoU, while contrastive learning provides only marginal gains.
  • Interventional Dependency Analysis: CSSE performs best against temporal proximity, semantic retrieval, and LLM-estimated relevance, improving CIDEr from 88.39 to 104.59 and SODA_c from 10.84 to 12.73.These comparisons use the same backbone and boundaries.

5 Conclusion

CausalChapter introduces intervention-defined dependency modeling for scalable and coherent long-video chaptering. Its modules address boundary propagation and fragmented cross-segment context, with reported improvements on AVLecture and competitive performance on VidChapters-7M.

  • Framework: CausalChapter models prediction-level influence through intervention-defined dependencies for long-video chaptering.The framework uses lightweight masking or removal interventions rather than claiming real-world causal discovery.
  • Framework: LCDS detects local dependency drops between adjacent temporal windows to provide auxiliary evidence for smooth chapter boundaries.
  • Framework: CSSE selects cross-segment contexts according to their intervention-defined support for current chapter generation.
  • Results: CausalChapter improves temporal localization and description quality on AVLecture while remaining competitive on VidChapters-7M.

Limitations

The paper identifies three limitations concerning interpretation, annotation sources, and inference cost. It distinguishes prediction-level influence from real-world causality and limits support estimation to compact candidate sets.

  • Scope: Intervention-defined dependency measures prediction-level influence, not real-world causal relations among video events or chapters.The interventions identify units or contexts affecting prediction under controlled masking or removal.
  • Data: The augmented AVLecture references may inherit stylistic regularities from the LLM-assisted annotation pipeline.The authors mitigate this by using identical references across methods and testing alternative annotation models and styles.
  • Efficiency: CSSE adds inference cost because it estimates context support through removal interventions.The implementation limits interventions to a compact candidate set from temporal neighbors and semantic retrieval.

A Method Details

The segmentation head represents each sentence-aligned semantic unit with fused visual and textual features before lightweight binary boundary prediction. Middle fusion exposes cross-modal cues to the classifier.

  • Architecture: The segmentation head uses sentence-aligned clips as basic units and predicts topic boundaries with a lightweight binary classifier.
  • Feature Fusion: Visual clip and ASR representations are projected into a shared dimension, fused through multimodal fusion layers, and concatenated into z_i.The encoders produce modality-specific features before projection and fusion.
  • Feature Fusion: Middle fusion allows transcript transitions, slide changes, and visual context shifts to inform boundary prediction before classification.The paper contrasts this with a late-fusion design.

B.1 AVLecture Benchmark

AVLecture is a large instructional-video benchmark with aligned multimodal signals and a segmented lecture subset used for boundary localization. Its chapter references are LLM-assisted, human-verified, and evaluated under controlled annotation settings.

  • Benchmark: AVLecture contains 86 courses, over 2,350 lectures, and roughly 2,200 hours spanning STEM instructional content.The benchmark includes aligned ASR transcripts and OCR signals, with many courses also providing educational resources.
  • Benchmark: A 15-course subset containing 350 lectures provides temporal segmentation boundaries and serves as the boundary-localization foundation.The work adopts this segmented subset for boundary evaluation.
  • Annotation: GPT-4o generates chapter descriptions from full transcripts, frame captions, and ground-truth segment timestamps.Descriptions are intended to summarize each segment's main topic while remaining faithful to the available inputs.
  • Annotation: All generated descriptions are manually checked and revised for genericity, unsupported content, boundary inconsistency, and redundancy.Annotators also improve descriptions spanning multiple lecture stages and normalize formatting and style.
  • Evaluation: Comparisons use the same augmented references, while annotation sensitivity varies the annotation model and description style across four reference sets.The sensitivity analysis changes the references while keeping splits, configurations, procedures, and evaluation code fixed.
  • Annotation: 88.0% of the 350 videos required no revision, while 11.1% required one revision round and 0.9% required two.All final labels were accepted by the assigned annotator group.

C.2 Backbone Scaling and Complete Baseline Results

Backbone scaling generally improves chapter-level generation, while temporal-aware optimization adds gains across LLaMA and Qwen2.5 families. Complete baseline results show that task adaptation substantially outperforms zero-shot prompting for long-video chaptering.

  • Backbone scaling: Stronger backbones generally improve chapter-level generation quality, especially on CIDEr, across the evaluated LLaMA and Qwen2.5 families.These results indicate that CausalChapter benefits from model scaling.
  • Backbone scaling: Temporal-aware optimization improves most corresponding settings beyond simply increasing backbone size.
  • Complete baseline results: Fine-tuned Chapter-Llama variants are much stronger than zero-shot counterparts, indicating that task adaptation is important for long-video chaptering.
  • Complete baseline results: Zero-shot closed-source LLMs achieve limited performance, especially on generation metrics, so general-purpose prompting alone is insufficient for this benchmark.
  • Complete baseline results: Semantic-similarity evaluation complements CIDEr because lexical metrics can undervalue semantically reasonable outputs with different wording or granularity.
  • Complete baseline results: All tested CSSE output-difference functions improve over removing CSSE, while KL divergence performs best overall.

C.5 Inference Efficiency and Scalability

CausalChapter maintains a bounded inference cost while offering tunable latency–memory trade-offs. Micro-batching and candidate scaling reduce sequential work, and K = 5 provides a balanced quality setting.

  • Micro-batching: Increasing micro-batch size from 1 to 8 reduces sequential generation calls from 11.90 to 5.30 and KL calls from 11.20 to 6.60.
  • Micro-batching: At size 8, overall latency decreases from 12.20 to 10.50 seconds per video while peak memory increases from 16.59 to 19.35 GiB.
  • Micro-batching: CSSE accounts for 4.59/10.50 (43.7%) of the complete pipeline at micro-batch size 8.
  • Candidate scaling: Increasing the candidate limit from 1 to 10 changes average KL calls only from 6.60 to 6.90, while CSSE time increases from 4.43 to 5.31 seconds per video.The practical intervention cost is governed by the compact set of available historical candidates rather than the nominal limit alone.
  • End-to-end efficiency: End-to-end time remains within 10.43–10.98 seconds per video and peak memory within 19.35–19.58 GiB across the Top-K sweep.
  • End-to-end efficiency: K = 5 gives the highest METEOR and a strong CIDEr score while using nearly all contexts available on average.It is therefore used as the balanced default.
  • Quantized inference: Under 4-bit NF4 inference, micro-batch size 1 uses 8.28 GiB for CausalChapter versus 9.90 GiB for Chapter-Llama.Increasing the size to 8 reduces latency from 47.77 to 19.41 seconds per video while using 13.80 GiB.

C.6 Qualitative Analysis

Qualitative cases show CausalChapter handling smooth transitions through intervention-defined dependency shifts and selecting historical contexts for predictive support rather than surface similarity. These behaviors yield more complete, logically aligned chapter descriptions.

  • Boundary localization: LCDS identifies smooth structural changes by detecting drops in intervention-defined predictive dependency between adjacent windows.Baseline methods often delay or miss such boundaries when topics evolve gradually.
  • Context selection: CSSE selects definitions, background, experimental setup, or reasoning premises instead of lexically close but repetitive contexts.
  • Context selection: Support-based context selection produces descriptions that are more complete and better aligned with the logical structure of the whole video.
  • Overall interpretation: The qualitative behavior is consistent with quantitative improvements in both temporal localization and chapter-level generation.
Loading 2609.08686v1…