PaperlayerView exact revision on arXiv

Source-linked AI summary

Building a Precise Video Language with Human-AI Oversight

Zhiqiu Lin, Chancharik Mitra, Siyuan Cen, Isaac Li, Yuhan Huang, Yu Tong Tiffany Ling, Hewei Wang, Irene Pi, Shihang Zhu, Ryan Rao, George Liu, Jiaxi Li, Ruojin Li, Yili Han, Yilun Du, Deva Ramanan

arXiv:2604.21718v2cs.CVcs.AIcs.CLcs.LGcs.MM

TL;DR

Video-language models still struggle with precise motion and camera descriptions, while existing annotation practices often lack consistent specification and oversight. The paper introduces a professional-grounded specification, CHAI human–AI oversight, and critique- and preference-based post-training, yielding stronger video understanding and more controllable generation, with scope remaining focused on video understanding benchmarks.

  • Problem

    Existing video-language datasets and models often lack precise, consistent supervision for visual dynamics, especially subject motion and camera behavior.

  • Method

    The paper combines a structured specification developed with professional creators, CHAI critique-based human–AI oversight, and post-training on preference and critique signals.

  • Results

    With modest expert supervision, Qwen3-VL outperforms closed-source Gemini-3.1, while explicit critique and preference signals improve SFT and RL methods across tasks.

  • Takeaways & Limitations

    Precise specifications and high-quality human–AI critiques support scalable curation for professional video understanding and more controllable video generation.

  • Takeaways & Limitations

    The work focuses on video understanding, leaving video generation benchmarks as future work.

Abstract

from arXiv · show

Video-language models (VLMs) learn to reason about the dynamic visual world through natural language. We introduce a suite of open datasets, benchmarks, and recipes for scalable oversight that enable precise video captioning. First, we define a structured specification for describing subjects, scenes, motion, spatial, and camera dynamics, grounded by hundreds of carefully defined visual primitives developed with professional video creators such as filmmakers. Next, to curate high-quality captions, we introduce CHAI (Critique-based Human-AI Oversight), a framework where trained experts critique and revise model-generated pre-captions into improved post-captions. This division of labor improves annotation accuracy and efficiency by offloading text generation to models, allowing humans to better focus on verification. Additionally, these critiques and preferences between pre- and post-captions provide rich supervision for improving open-source models (Qwen3-VL) on caption generation, reward modeling, and critique generation through SFT, DPO, and inference-time scaling. Our ablations show that critique quality in precision, recall, and constructiveness, ensured by our oversight framework, directly governs downstream performance. With modest expert supervision, the resulting model outperforms closed-source models such as Gemini-3.1-Pro. Finally, we apply our approach to re-caption large-scale professional videos (e.g., films, commercials, games) and fine-tune video generation models such as Wan to better follow detailed prompts of up to 400 words, achieving finer control over cinematography including camera motion, angle, lens, focus, point of view, and framing. Our results show that precise specification and human-AI oversight are key to professional-level video understanding and generation. Data and code are available on our project page: https://linzhiqiu.github.io/papers/chai/

1. Introduction

The paper argues that precise video language requires structured description policies, scalable human–AI oversight, and post-training methods that exploit critique and preference signals. These ingredients improve video understanding and generation, including fine-grained control over motion and cinematography.

  • Motivation: Existing video–text datasets often lack clear description policies, producing inconsistent captions that omit visual details or include hallucinated content.The paper identifies subject motion and camera dynamics as especially difficult for off-the-shelf models.
  • Contributions: The specification formalizes subjects, scenes, motion, spatial framing, and camera dynamics with professional creators.It includes hundreds of visual primitives and training guidelines for consistent application.
  • Contributions: CHAI shifts human effort from writing captions to verifying and correcting model-generated pre-captions through critique-based oversight.The framework combines model fluency with human attention to visual accuracy and produces preference pairs and natural-language critiques.
  • Post-training: Explicit preference and critique supervision improves standard SFT and reinforcement-learning methods for caption generation, reward modeling, and critique generation.The paper reports gains through SFT, DPO, and inference-time scaling.
  • Results: With modest expert supervision, Qwen3-VL outperforms closed-source Gemini-3.1, while critique quality in precision, recall, and constructiveness governs post-training success.The oversight workflow is designed to enforce these critique properties.
  • Application: Fine-tuning video generators such as Wan on re-captioned professional videos improves adherence to detailed prompts of up to 400 words and control over cinematography.The reported controls include camera dynamics, shot composition, motion, and related visual properties.

2. Related Work

Prior video–text datasets commonly suffer from incomplete coverage, visual hallucinations, inconsistent writing, and unverifiable quality. The paper adapts scalable oversight and explicit feedback-based post-training to address these limitations in video understanding.

  • Video–text datasets: Human- and model-curated video–text datasets often contain incomplete coverage, visual hallucinations, and writing problems.The paper attributes these issues largely to unclear specifications and ineffective oversight.
  • Scalable oversight: Scalable oversight studies how humans can supervise models that match or surpass them in complex task-specific skills.Models assist with error detection while humans provide oversight.
  • Scalable oversight: The paper applies this division of labor to video captions, letting models generate fluent text while humans focus on visual details.This differs from datasets relying solely on human-written or model-generated captions without cooperation.
  • Post-training strategies: Prior multimodal post-training work often uses self-critique or synthetic and image-only feedback, whereas this paper studies high-quality feedback for video understanding.The paper positions its feedback as supervision for video-specific post-training.

3. Specification For Video Captioning

The paper develops a professional-grounded specification to make video captioning more consistent, comprehensive, and objectively observable. It combines a structured taxonomy of primitives with policies for describing how those elements unfold over time.

  • Specification motivation: Without consistent annotation policies, captions may omit camera movement and include subjective descriptions that annotators need not share.The paper’s dataset review identifies missing details such as camera shake, focus changes, and tracking movement.
  • Building the specification: The specification is built with professional creators who use shared visual language to coordinate filmmaking and other creative workflows.The framework is intended to make videos vividly describable to people who have not seen them.
  • Primitives for precise specification: CameraBench-Pro defines hundreds of visual and motion primitives with rigorous definitions, examples, corner cases, and decision rules.The taxonomy spans camera motion, camera setup, and video cinematography, with over 200 primitives overall.
  • Structured captioning policy: The structured captioning policy links multiple primitives into coherent temporal descriptions of changing viewpoints, camera motion, depth of field, and focus.The policy captures how visual elements unfold rather than treating primitives as isolated labels.

4. Human-AI Oversight Framework

The CHAI oversight framework divides detailed video captioning between model-generated text and human verification, addressing recurring writing errors and visual inaccuracies. Expert screening, training, peer review, and incentives target fine-grained spatial and motion errors.

  • Motivation: Oversight is especially important because short five-second clips can contain multiple subjects, actions, camera movements, and spatial changes requiring precise ordering.Detailed descriptions can take hundreds of words and more than ten minutes to complete.
  • Motivation: Existing human and model captions both contain recurring errors, including poor writing, hallucinated visual details, and incorrect event ordering.Human captions often have typos or awkward phrasing, while model captions may invent subjects or motion.
  • Human–AI collaboration: CHAI has models generate fluent pre-captions while humans critique visual facts and guide refined post-captions.This division lets humans verify content instead of writing 200–400-word captions from scratch.
  • Workflow: The workflow separates visual-primitive labeling from captioning to reduce cognitive load and support more focused annotation.Primitive labeling uses a separate quality-controlled process before the main captioning workflow.
  • Quality control: Expert recruitment, multi-round testing, peer review, and accuracy-based bonuses are used to improve precision in subtle spatial and camera descriptions.Annotators are recruited from content-creation backgrounds and reviewers are rewarded for valid corrections.

5. Evaluation and Post-Training

The paper builds a unified benchmark and post-training pipeline around pre-caption, critique, and post-caption triplets. Results show that explicit preference and critique supervision improves performance, while critique quality governs post-training success.

  • Data and supervision: The oversight data naturally yields pre-caption, critique, and post-caption triplets for evaluating and post-training video-language models.These triplets provide both preferred-versus-rejected outputs and explicit corrective feedback.
  • Benchmarks and tasks: ∼20k triplets support a held-out benchmark and training across caption generation, reward modeling, and critique generation for five video aspects.The benchmark covers subject, scene, motion, spatial, and camera information and is substantially larger than recent public benchmarks.
  • Critique quality: Accurate, complete, and constructive critiques are essential because weakening any of these properties substantially reduces post-training performance.Constructive critiques explain how to fix identified errors rather than merely naming them.
  • Post-training: Explicit preference and critique signals consistently improve supervised fine-tuning and reinforcement-learning methods across the benchmark tasks.The training setup includes SFT, DPO, and related offline post-training methods.
  • Results: Inference-time scaling provides additional gains on reward modeling and caption generation without additional human supervision.The paper reports this as an additional result in Appendix G.
  • Results: The fine-tuned model achieves state-of-the-art performance across captioning, reward modeling, and critique generation, surpassing open- and closed-source baselines.Table 1 specifically reports comparisons with Gemini-2.5 and Gemini-3.1, with a special binary-score treatment for Gemini-3.1 reward modeling.

6. Improving Professional Video Generation

The authors apply their post-trained captioning model to large-scale professional videos and use the resulting captions to fine-tune Wan2.2. The resulting generator better follows long prompts and offers finer cinematographic control.

  • Data: About ∼150K professional videos from films, commercials, music videos, and games are manually collected for re-captioning and video-generation fine-tuning.The videos are sourced under standard non-commercial licenses.
  • Cinematographic control: The fine-tuned model demonstrates control over camera movements, angles, focus shifts, playback speed, perspective, overlays, height transitions, shot size, and subject transitions.Examples include rolling camera motion, Dutch angles, rack focus, speed ramps, and side-view game perspectives.
  • Generation results: Evaluations show that the re-captioned-data pipeline outperforms zero-shot Wan and Wan fine-tuned with zero-shot Qwen3-VL captions.The comparison is reported in Appendix H.

7. Conclusion

The paper presents an open recipe, dataset, and benchmarks for precise video captioning through clear specification, scalable oversight, and post-training. It shows that high-quality data can be curated efficiently through human–AI collaboration, while identifying video generation benchmarks as future work.

  • CHAI demonstrates that high-quality video data can be curated efficiently through human–AI collaboration with modest academic resources.
  • Future work includes scaling oversight with stronger critique models and building benchmarks for video generation.
  • The paper combines clear specification, scalable oversight, and effective post-training into an open recipe for precise video captioning.

Outline

The supplement covers dataset analysis, specification, oversight, triplets, evaluation and training details, inference-time scaling, and video-generation applications. It also provides complete annotation guidelines and pseudocode for converting labeled primitives.

  • Dataset and oversight foundations: Sections A–C cover prior-dataset error analysis, the full specification, and the complete oversight pipeline.
  • Critiques and triplets: Sections D–E present captioning triplets and methods for generating and analyzing negative critiques.
  • Training and evaluation: Section F details evaluation ablations, reward scoring, SFT preparation, adversarial critiques, and hyperparameters.
  • Scaling and generation: Sections G–H report inference-time scaling, the web-scraped video set, and text-to-video fine-tuning experiments.
  • Annotation resources: Sections I–J provide complete captioning guidelines and Python-style pseudocode for converting labeled primitives.

A. Error Analysis of Prior Datasets

Evaluations of prior video–text datasets reveal recurring failures from underspecified annotation policies and insufficient oversight, including omissions, hallucinations, terminology errors, and subjective language. These problems vary by dataset, while TUNA-Bench receives the strongest overall quality assessment among prior work.

  • Eight widely used video–text datasets were manually evaluated for systematic issues in caption accuracy, completeness, and consistency.
  • Lack of specification: Lack of specification causes imprecise terminology, missing motion or spatial information, and subjective descriptions.
  • Lack of oversight: Lack of oversight leaves writing errors, temporal-order mistakes, visual hallucinations, and inaccurate subtle details uncorrected.
  • Dataset comparisons: TUNA-Bench achieves the highest quality among existing datasets across most aspects, especially subject, scene, and motion, but cross-dataset scores are not directly comparable.
  • Crowdsourced captions: Crowdsourced captions are typically ∼100 words versus ∼400 words under structured specification and oversight, and often lack the vocabulary for cinematic or motion effects.
  • Dataset-specific findings: ShareGPT4Video, UltraVideo, and VDC exhibit recurring hallucinations or camera-detail errors, while VDC is not recommended for evaluating detailed video captioning.

B. Specification Details

The specification is developed with professional video creators and organized into five major aspects. Expert-labeled visual primitives are converted into dynamically conditioned pre-captions.

  • Professional filmmakers, motion designers, cinematographers, and game artists help refine the taxonomy, resolve ambiguities, and verify primitive definitions.
  • The specification is organized into five major aspects and their corresponding subaspects.
  • Approximately 200 labeled visual primitives are converted into pre-captions using prompts dynamically generated from each video’s labels.
  • Dynamic prompts tailor descriptions to video content, such as emphasizing a main subject or describing the environment when no main subject exists.

C. Oversight Details

The oversight framework combines professional creator roles, structured training, platform-based review, and critique-centered quality control to curate precise captions. CHAI shifts human effort from writing toward verification while retaining comparable total time and producing longer captions.

  • Human–AI team and platform: The framework uses professional creators to develop the specification, annotate videos, review captions, and curate human–AI training data.The platform supports critique refinement, expert shadowing, reviewer approval or rejection, and appeals.
  • Training and incentives: Annotators are screened and trained before production work, with promotion and compensation tied to accuracy and consistency.Reviewers can receive accuracy-based pay increases, while low-performing annotators may be stopped or assigned additional onboarding.
  • CHAI annotation statistics: 95% of samples reach an accurate post-caption after one critique-revision cycle, while spatial captions are hardest, scoring 3.3/5 initially and requiring 1.4 iterations on average.The statistics cover approximately 4k videos across five captioning aspects.
  • CHAI annotation statistics: CHAI takes 15 minutes per video, comparable to crowdsourced write-from-scratch captioning, but produces captions about 3.7× longer at 405 versus 110 words.The reported comparison characterizes effort reallocation rather than elimination and associates CHAI with substantially greater caption accuracy.
  • Scaling limitation: The main scaling bottleneck is training annotator expertise: reaching at least 90% held-out accuracy typically requires 1.5–3 months of paid training.This overhead is cited as the main reason the initial release is capped at approximately 4k videos.
  • Critique quality: Good critiques must be accurate, complete, and constructive because annotators use them to guide the final post-caption.Constructiveness requires explaining how to fix identified problems, not merely identifying errors.

E.1. Prompts for Controlled Critique Error Injection

The appendix constructs controlled critique errors and model-generated comparison critiques to isolate how accuracy, completeness, constructiveness, and visual access affect critique quality.

  • Controlled critique error injection: The ablation pipeline injects one critique-property error at a time using the caption instruction, pre-caption, and, when applicable, ground-truth feedback.The resulting variants target inaccurate, incomplete, and non-constructive critiques.
  • E.1.1. Inaccurate Critiques: Inaccurate critiques are created through insertion errors that add spurious details or replacement errors that swap correct feedback for plausible false information.Both procedures preserve critique-like form while corrupting factual content.
  • E.1.2. Incomplete Critiques: Incomplete critiques remove one necessary correction from the ground-truth critique, except when the original already says no edit is needed.The deletion prompt removes a key detail, suggestion, or explanation rather than shortening a single-item critique.
  • E.1.3. Non-Constructive Critiques: Non-constructive critiques retain error identification while removing the fix or other actionable guidance.For example, a correction specifying the replacement detail is reduced to a statement that the original detail is wrong.
  • E.2. Model-Generated Critiques: The comparison includes Gemini-2.5-Pro with video and pre-caption inputs, plus a blind Gemini-2.5-Pro setting that receives only the pre-caption.The blind model is instructed to imagine visual content, whereas the video model must inspect the actual video.
  • E.2. Model-Generated Critiques: The video-model critique prompt requires checking missing elements and factual errors against the video, then giving specific, actionable, concise corrections.It also instructs the model to remove unrelated or redundant caption content and avoid feedback outside the task instruction.

E.3. Critiques in Prior Work Are Often Non-Constructive

The analysis finds that critiques in prior benchmarks often identify errors without explaining what is correct or how to fix them, limiting their usefulness for training and evaluation.

  • Overall finding: More than half of critiques in the examined prior datasets are non-constructive, and MM-RLHF exceeds 90% among responses with critiques.The analysis combines ChatGPT classification with manual verification over approximately 200 samples per dataset.
  • Comparison with the proposed framework: Unlike prior datasets that request critique without correction, this framework writes critiques expressly to fix model outputs and thereby enforces precision, recall, and constructiveness.The paper identifies critique quality, not merely critique quantity, as essential for scalable oversight.
  • OpenAI GDC: 47% of critiques in OpenAI GDC topic-based summarization are non-constructive.The estimate is based on 1,728 test samples.
  • OpenAI GDC: 49% of critiques in OpenAI GDC text question answering are non-constructive.The estimate is based on 1,851 test samples.
  • MM-RLHF: 90% of non-empty critiques in MM-RLHF video question answering are non-constructive.Non-empty critiques comprise 58% of 8,751 test samples.

F. Post-training details

The post-training experiments evaluate captioning, critique, reward, and revision formats using preference and critique supervision. Across methods, explicit supervision improves results, while VQAScore is selected for reward modeling because it outperforms text-generated Likert scoring.

  • Post-training formats: The experiments describe SFT data formats and compare post-training strategies for caption generation, critique generation, reward modeling, and caption revision.The dataset formats include video and task instructions paired with captions, critiques, reward labels, or revised captions.
  • Evaluation metrics: LLM-Judge-Direct sends reference and candidate captions to GPT-4o and uses P(Yes) as the evaluation score.The direct prompt asks whether the candidate accurately matches the reference in content and meaning.
  • Evaluation metrics: LLM-Judge-Instruct additionally supplies the task instruction when judging whether a candidate caption follows the task and matches the reference.The judge receives the task instruction, reference caption, and candidate caption before returning Yes or No.
  • Evaluation metrics: 98.0% pairwise accuracy is achieved by LLM-Judge-Instruct with GPT-4o against human ratings, compared with 95.3% for LLM-Judge-Direct with GPT-4o.Pairwise accuracy evaluates whether metrics rank captions consistently with human 1-to-5 Likert scores.
  • Critique evaluation: Critique-Revision evaluates critiques by using them to revise captions and then comparing the revised captions with references rather than directly matching critique wording.This addresses stylistic variation among critiques.
  • Preference construction: The dataset constructs preference pairs by using pre-captions as negatives when appropriate, adversarial critiques for critique generation, flipped labels for reward modeling, and post-captions as positive revisions.Each triplet contributes training signal across the compared post-training methods.
  • Reward modeling: VQAScore substantially outperforms text-generated Likert scoring after SFT across all five aspects and is therefore used for the main reward-modeling results.VQAScore reduces reward prediction to binary classification with a continuous logit-based score, avoiding five-way probability allocation and imbalanced Likert labels.

G.1. Test-time Scaling for Reward Modeling

Inference-time critique can modestly improve reward-modeling performance beyond directly producing a Yes/No score.

  • The study evaluates reward-modeling outputs with both VQAScore and Self-Consistency.Self-Consistency is computed as the ratio of Yes to No over five rollouts.
  • Critique-Last marginally outperforms direct VQAScore, indicating that generating a critique after Yes/No can yield further gains.The evaluation compares VQAScore and Self-Consistency.

G.2. Test-time Scaling for Caption Generation

The caption-generation study compares parallel best-of-N selection with sequential refinement under explicit generation and reward costs. Parallel selection performs best at larger budgets, while sequential methods show diminishing returns.

  • Parallel approaches: Best-of-N strategies generate candidates independently and use reward-model scores to select the strongest caption or revision.Variants include caption, revision, critique-then-revision, critique-based revision, and critique selection.
  • Computational budgets: The experiments compare strategies at N ∈{1, 4, 8, 16} while separating generation cost from reward cost.Critique-based revision counts as 2× generation cost because it produces a self-critique followed by a revised caption.
  • Parallel approaches: 25.1 BLEU-4 at N = 16 is achieved by Vanilla best-of-N Caption, the strongest strategy at N ≥4.The method samples captions in parallel and selects the highest-scoring candidate with a reward model.
  • Parallel approaches: Parallel methods consistently outperform sequential iterative approaches as the computation budget increases.Parallel candidates are generated independently and selected using reward scores, whereas sequential computation accumulates linearly with N.
  • Sequential approaches: Iterative critique-based revision improves only from 16.4 to 18.0 across N = 1 to 16, showing diminishing returns.Each critique-based revision internally generates a critique and a revised caption, costing 2× generation cost per step.
  • Limitations: The study does not explore hybrid search, reward-guided early stopping, or pruning mechanisms.These extensions are left for future work.

H. Video Generation

The video-generation application re-captions diverse professional videos and uses structured captions to fine-tune Wan2.2. The resulting SFT-Caption variant receives the best human prompt-following scores across cinematographic examples.

  • Video data: The curated video collection contains approximately 150K professional and diverse YouTube videos spanning films, commercials, music videos, games, and other genres.The corpus also includes user-generated, documentary, nature, and varied gaming footage.
  • Model training: Wan2.2 is fine-tuned using captions generated either zero-shot or by an SFT-trained Qwen3-VL-8B-Instruct model.The combined caption is constrained to Wan2.2's capacity of roughly 512 tokens or up to 400 words.
  • Captioning: Captions cover five types: subject, scene, motion, spatial, and camera.The captioning pipeline merges these descriptions into one comprehensive, nonredundant caption.
  • Captioning: The spatial caption provides the base structure, while motion and camera information are merged chronologically and unique subject or scene details are added.The procedure preserves exact wording and removes redundant information.
  • Human study: The SFT-Caption variant achieves the best prompt following among the three Wan2.2 variants in a 200-prompt human study.Scores are averaged across three expert annotators using a 1-to-5 Likert scale.
  • Qualitative results: Qualitative examples demonstrate control over dolly-zoom, isometric and side-view perspectives, roll, Dutch angle, rack focus, speed ramp, overlays, height changes, shot size, and revealing shots.The examples span camera motion, angle, focus, playback speed, perspective, framing, and subject transitions.
Loading 2604.21718v2…