PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoCon: Robust Video-Language Alignment via Contrast Captions

Hritik Bansal, Yonatan Bitton, Idan Szpektor, Kai-Wei Chang, Aditya Grover

arXiv:2311.10111v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Semantically plausible caption changes expose a robustness gap in video-language alignment models. VideoCon uses LLM-generated contrast captions and explanations to finetune a generative model for entailment and explanation. It improves performance on human-generated contrasts and zero-shot downstream tasks.

  • Problem

    Despite large-scale pretraining, video-language alignment models are not robust to semantically plausible caption changes, while scaling training data presents sourcing, storage, and diminishing-returns challenges.

    Source: §1 · p. 1

  • Method

    VideoCon uses an LLM to generate plausible contrast captions and explanations, then finetunes a generative video-language model for entailment judgments and explanations.

    Source: §1 · p. 1

  • Results

    VideoCon improves robustness on human-generated contrasts and achieves SOTA zero-shot retrieval and video-question-answering performance; ROC-AUC exceeds baseline by 12 points on human-generated contrast captions.

    Source: §1 · p. 2, §8 · p. 8

  • Takeaways & Limitations

    The authors suggest contrast-caption training is a data-efficient and effective way to improve robustness compared with scaling pretraining data.

    Source: §1 · p. 2

    A current limitation is contrast-caption complexity: a single caption may contain several misalignments, requiring accurate low entailment scores and precise explanations.

    Source: §8 · p. 8

Abstract

from arXiv · show

Despite being (pre)trained on a massive amount of data, state-of-the-art video-language alignment models are not robust to semantically-plausible contrastive changes in the video captions. Our work addresses this by identifying a broad spectrum of contrast misalignments, such as replacing entities, actions, and flipping event order, which alignment models should be robust against. To this end, we introduce the VideoCon, a video-language alignment dataset constructed by a large language model that generates plausible contrast video captions and explanations for differences between original and contrast video captions. Then, a generative video-language model is finetuned with VideoCon to assess video-language entailment and generate explanations. Our VideoCon-based alignment model significantly outperforms current models. It exhibits a 12-point increase in AUC for the video-language alignment task on human-generated contrast captions. Finally, our model sets new state of the art zero-shot performance in temporally-extensive video-language tasks such as text-to-video retrieval (SSv2-Temporal) and video question answering (ATP-Hard). Moreover, our model shows superior performance on novel videos and human-crafted captions and explanations. Our code and data are available at https://github.com/Hritikbansal/videocon.

1. Introduction

The introduction identifies semantically plausible caption edits as a robustness gap for video-language alignment and presents VideoCon as targeted contrast-caption training. It reports gains on human-generated contrasts and zero-shot downstream tasks.

1. Introduction

  • Prior work reports that large-scale pretraining does not ensure robustness to plausible caption manipulations, while scaling video-language data is difficult and can yield diminishing returns.The introduction links this robustness concern to the trustworthiness of alignment models in large-scale deployment.Source: §1 · p. 1
  • VideoCon uses an LLM to generate plausible contrast captions and explanations across misalignments in objects, actions, attributes, relations, counts, event order, and hallucinations.Video-caption pairs whose alignment could be inferred from a single frame are skipped to make the dataset temporally challenging; a sample was human-verified as high-quality.Source: §1 · p. 1
  • The pipeline filters aligned pairs for temporal challenge, generates contrast captions and explanations, finetunes an alignment model on both tasks, and evaluates it against a baseline.The overview reports downstream state-of-the-art results for the finetuned model.Source: Figure 1 · p. 2
  • The evaluation includes human-generated contrast captions and explanations for videos from external datasets that were not used to develop VideoCon.This evaluation is used to assess generalization.Source: §1 · p. 2
  • 12 points higher ROC-AUC than the baseline on human-generated contrast captions; the finetuned model also outperforms baselines on unseen videos and human-generated captions and explanations.The authors attribute this to improved understanding of entities, interactions, actions, and event order.Source: §1 · p. 2
  • VideoCon achieves SOTA zero-shot performance on temporally challenging text-to-video retrieval and video question answering, improving SSv2-Temporal by 4.3 mAP, SSv2-Events by 3.6 mAP points, and ATP-Hard accuracy by 4%.The authors suggest contrast-caption training is data-efficient and effective compared with scaling pretraining data.Source: §1 · p. 2

2. Video Language Alignment

This section frames video-text alignment as a robustness problem involving plausible changes to descriptions. It defines video-language entailment and explanation tasks, which the paper addresses together with one generative model.

2. Video Language Alignment

  • The alignment approach uses video frames without the soundtrack.The paper states this follows prior works.Source: §2 · p. 2
  • Video-text alignment supports applications including retrieval, video generation, and captioning, but prior work questions its robustness to semantically plausible changes in descriptions.The paper aims to improve robustness by training on contrast captions with varied misalignments.Source: §2 · p. 2
  • The dataset represents each example as a video, aligned caption, misaligned contrast caption, and natural-language explanation of the misalignment.The contrast caption perturbs the aligned caption but does not align with the video.Source: §2 · p. 2
  • The paper jointly addresses video-language entailment and natural-language explanation, using one generative model to predict an entailment label and produce an open-ended explanation.Entailment classifies whether a video entails a text; explanation describes the discrepancy between a video and a non-entailing caption.Source: §2 · p. 2

3. VideoCon: Contrast Captions Generation for Robust Video-Language Alignment

VideoCon pairs video captions with plausible, categorized misalignments and explanations, using both LLM-generated data and human-created examples for evaluation. Its construction filters for temporal challenge and for contrasts and explanations that capture meaningful differences.

3.1. Temporally-Challenging Instance Selection

  • VideoCon draws on MSR-VTT, VaTeX, and TEMPO, retaining captions less alignable to individual frames to reduce single-frame shortcuts.After filtering, temporally challenging instances increased from 36.5% to 81.5% in MSR-VTT and from 42.6% to 71% in VaTeX.Source: §3.1 · p. 3

3.2. Categories of Contrast Captions

  • The dataset covers seven contrast types: object, action, attribute, counting, spatial relation, hallucination, and event-order flip.The reported category shares range from 8.8% to 24.2%.Source: §3.2 · p. 3, Figure 3 · p. 4

3.3. Data Generation using an LLM

  • PaLM-2 generates contrast captions and explanations for assigned misalignment types, after which NLI filters out non-contradictory contrasts and inadequate explanations.Contrasts are removed when NLI(T, C) > 0.5; explanations are removed when NLI(F(T, C), E) < 0.6.Source: §3.3 · pp. 3–4
  • 91% of sampled contrast captions and 89% of explanations were judged valid, supporting the quality of VideoCon (LLM).Human evaluation covered 500 contrast captions and explanations.Source: §3.3 · p. 4

3.4. Data Generation using Humans

  • Human workers created contrast captions and explanations for ActivityNet video segments without being given target misalignment types, yielding evaluation data from external videos.The resulting collection is used to assess generalization to out-of-distribution videos and human-generated contrasts and explanations.Source: §3.4 · p. 4
  • VideoCon (Human) contains 570 human-written tuples; a 290-instance Human-Hard subset excludes tuples with Avle(V, T) below 0.5.A 100-instance sample was 93% clean; objects and actions each account for 35% of human contrasts, while event-order flips and attribute changes each account for 8%–10%.Source: §3.4 · p. 4

4. Experimental Setup

The experiments fine-tune mPLUG-Owl-Video on VideoCon and evaluate video-language entailment, explanation generation, and downstream video tasks. Comparisons include established alignment models and a randomly sampled contrast-caption training baseline.

4. Experimental Setup

  • The evaluation is designed to measure VideoCon’s impact on video-text alignment modeling across intrinsic test-set tasks and temporal, action-focused downstream tasks.The experimental setup introduces this aim and describes the downstream assessment as complementary to intrinsic evaluation.Source: §4 · p. 4, §4.3 · p. 5

4.1. Finetuning with VideoCon

  • Owl-Base is fine-tuned on VideoCon instruction-response pairs for video-language entailment and natural-language explanation, producing the joint model Owl-Con.Both task types are trained together using maximum-likelihood loss.Source: §4.1 · p. 5

4.2. VideoCon Evaluation Metrics

  • Video-language entailment is scored by ROC-AUC from Owl-Con’s Yes/No response probabilities, while explanation quality is measured by NLI entailment against reference explanations.The explanation task uses Q2 and PaLM-2 NLI metrics, which the authors report have high agreement with human ratings.Source: §4.2 · p. 5

4.3. Video-Text Downstream Tasks

  • Downstream evaluation measures text-to-video retrieval mAP on SSv2-Temporal and SSv2-Events, and answer accuracy on ATP-Hard video question answering.For retrieval, the setup uses SSv2-Template captions; for question answering, candidate answers are converted into imperative statements and ranked by alignment.Source: §4.3 · p. 5

4.4. Baselines

  • Alignment comparisons include End-to-End VNLI, VideoCLIP, ImageBind, Owl-Base, and Owl-Rand; TACT and VFC are additional downstream baselines.Owl-Rand fine-tunes Owl-Base on VideoCon tuples with contrast captions randomly selected from other dataset captions.Source: §4.4 · p. 5

5. Experiments

VideoCon training improves entailment and explanation generation on contrast-caption evaluations, and Owl-Con achieves state-of-the-art results on the reported zero-shot retrieval and question-answering tasks.

5.1. Performance on VideoCon Entailment Task

  • VideoCon evaluation shows baseline AUC scores near random, while a VNLI-specific model reaches 67, indicating entailment requires specific training.Training with easy random negatives only slightly improves Owl-Base, to 59.7 from 57.2.Source: §5.1 · p. 5, Table 2 · p. 6
  • 84.6 AUC: VideoCon training with diverse, difficult misalignments yields the best VideoCon entailment result, while out-of-domain gains over Owl-Base are 17% on Human and 16% on Human-Hard.The reported out-of-domain gain on the LLM test is 48%; the authors note that the atemporal VNLI model also outperforms existing video-language alignment models.Source: §5.1 · p. 6

5.2. Performance on NLE Generation Task

  • 263% on Q2 score and 178% on PaLM-2 accuracy: Owl-Con’s relative in-domain NLE gains over Owl-Base are more moderate out of domain, at 40% and 20%, respectively.The authors suggest human explanations’ more diverse wording may account for the smaller out-of-domain gains.Source: §5.2 · p. 6

5.3. Performance on Video-Text Downstream Tasks

  • 7.5% on SSv2-Temporal and 9.6% on SSv2-Events: Owl-Con’s relative mAP gains over End-to-End VNLI set new state-of-the-art retrieval results.The SSv2-Events gain is 46% over TACT.Source: Table 4 · p. 6, §5.3 · p. 7
  • 5.1%: Owl-Con’s relative accuracy increase over End-to-End VNLI sets state-of-the-art performance on ATP-Hard video question answering.The passage describes the task as temporal and causal video question answering.Source: Table 5 · p. 7, §5.3 · p. 7

6. Analysis

Owl-Con outperforms the baselines across VideoCon misalignment types, with its largest reported gain on event-order flips. Qualitative examples show both accurate temporal reasoning and an object-confusion error.

6.1. Per-misalignment Entailment Results

  • Owl-Con outperforms the baselines across misalignment types, with its largest improvement on event-order flips; both Owl-Con and End-to-End VNLI outperform Owl-Base on hallucinations.The authors also report that VideoCon helps with counting and relation errors that challenge End-to-End VNLI.Source: §6.1 · pp. 7–8

6.2. Qualitative Examples

  • In a successful example, Owl-Con rejects a caption that reverses event order and generates an accurate NLE; in a failure, it confuses buns with braids and produces a wrong NLE.The authors suggest expanding video and caption variety may improve this object distinction.Source: §6.2 · p. 8

7. Related Work

Prior work studies video-language alignment models and identifies robustness to semantic caption changes as a persistent concern. VideoCon targets that concern with contrast-caption training and fine-grained explanations.

Foundation Models for Video-Language Understanding

  • Video-language foundation models use large-scale video-text pretraining, yet prior work reports that semantic caption changes, including temporal-order changes, expose robustness gaps.The paper connects these gaps to limitations on real-world alignment applications.Source: §7 · p. 8

Improving Video-Language Robustness

  • Earlier robustness work examines manipulations of verbs, actions, and entities; temporal-order-loss finetuning is also reported to have inconsistent zero-shot downstream performance.That prior method’s outcomes depend strongly on the finetuning dataset, according to the passage.Source: §7 · p. 8

Video-Language Alignment Evaluation

  • Alignment supports retrieval and generation, while prior work finds fine-grained feedback useful for evaluation and training; VideoCon pairs entailment judgments with misalignment explanations.The paper describes sparse entailment scores and dense explanations as potential feedback for future alignment work.Source: §7 · p. 8

8. Conclusion

VideoCon is designed to improve robust video-text alignment through diverse semantic misalignments and explanations. The authors identify increasing contrast-caption complexity and scaling evaluation coverage as future directions.

8. Conclusion

  • VideoCon covers varied semantic misalignments and explanations for video-text discrepancies, and finetuning on it yields state-of-the-art results on text-to-video retrieval and video question answering.The conclusion describes these as complex downstream tasks.Source: §8 · p. 8
  • A current limitation is that contrast captions may contain several misalignments, requiring accurate low entailment scores and precise explanations for those cases.The authors identify increasing the complexity of generated contrasts as a future direction.Source: §8 · p. 8
  • The VideoCon (Human) dataset needs broader coverage of visual domains, including generated videos, and more contrast captions and explanations for robust evaluation.Source: §8 · p. 9

Supplementary Material

Prior work has established video-language foundation models and applications, but their robustness to plausible caption changes remains limited. VideoCon targets robust alignment evaluation and training, including fine-grained explanations.

A. Detailed Related Work

  • Video-language foundation models trained on large web-sourced video-text collections still lack robustness to semantically plausible contrast captions, limiting their real-world use for alignment applications.Source: A. Detailed Related Work · p. 12
  • Prior work identifies failures around manipulated verbs, entities, and event order; temporal-order-loss finetuning has not consistently delivered strong zero-shot downstream performance.The cited work's performance is highly dependent on the finetuning dataset choice.Source: A. Detailed Related Work · p. 12
  • VideoCon finetunes a generative video-language model for robust entailment and fine-grained explanations, with gains on unseen human-written contrasts and downstream tasks.Source: A. Detailed Related Work · p. 12

B. Details about Video-Language Datasets

VideoCon draws on MSR-VTT, VaTeX, and TEMPO captions, filtering temporally easy examples from the first two datasets. Its construction assigns misalignment types and uses prompted LLM generation for contrast captions.

B. Details about Video-Language Datasets

  • MSR-VTT and VaTeX provide web video-caption pairs, while TEMPO contributes human-written temporal-reasoning captions; the datasets supply VideoCon's train-validation and test splits.MSR-VTT uses its train-validation and test splits; VaTeX uses halves of its training and validation sets; TEMPO contributes 11K training and 1.8K testing pairs.Source: B. Details about Video-Language Datasets · pp. 12–13
  • Filtering temporally easy MSR-VTT and VaTeX instances targets captions that can be matched from a single frame, which otherwise may not encourage temporal understanding.Figure 8 illustrates a VaTeX caption well-grounded in one frame.Source: B. Details about Video-Language Datasets · pp. 12–13, Figure 8 · p. 13

C. Misalignment Assignment

  • Relation misalignment is assigned using caption keywords from a spatial-relation list, while count misalignment is assigned when captions contain numbers from one to ten.Source: C. Misalignment Assignment · p. 13, Table 6 · p. 13
  • TEMPO examples receive five misalignment types with equal probability; MSR-VTT and VaTeX use event-order flips for challenging multi-event captions and other types for easy captions.Part-of-speech checks prevent assigning object, action, or attribute misalignments when captions lack the corresponding noun, verb, or adjective.Source: C. Misalignment Assignment · p. 13

D. LLM Prompt

E. Human Annotation for Data Quality

Human annotators assess generated-data quality and create an independent human-written contrast set. The model is finetuned with LoRA on VideoCon for two epochs.

E. Human Annotation for Data Quality

  • Mechanical Turk annotators judge whether generated contrast captions contradict originals and whether explanations describe the discrepancy, after a qualification test.One annotator handles each instance; workers were paid $18 per hour.Source: E. Human Annotation for Data Quality · p. 13

F. VideoCon (Human) Data Creation

  • For VideoCon (Human), annotators write plausible contrast captions and explanations from ActivityNet validation videos without instructions to use specific misalignment types.The set is intended to assess generalization and uses one worker per instance.Source: F. VideoCon (Human) Data Creation · p. 14

G. Finetuning Details

  • Finetuning applies LoRA with r = 32, α = 32, and dropout = 0.05 to mPLUG-Owl-Video for two epochs, using 32 sampled video frames.Training uses Adam, a 10^-4 maximum learning rate, and a total batch size of 64.Source: G. Finetuning Details · pp. 14–15

H. Human Agreement for the Generated NLE Automatic Evaluation Methods

Q2 and PaLM-2 automatic judgments show high agreement with human judgments for NLE evaluation. The appendix also describes the retrieval and video-QA evaluation setups.

H. Human Agreement for the Generated NLE Automatic Evaluation Methods

I.1. Text to Video Retrieval

  • SSv2-template is used to reduce object-recognition bias; SSv2-Temporal and SSv2-Events test temporal/action-rich retrieval, ranked by mAP.Source: §I.1 · p. 16
  • SSv2-Temporal has 18 actions and 216 candidate videos per query; SSv2-Events has 49 two-verb actions and 2,888 candidate videos per query.Source: §I.1 · p. 16

I.2. Video QA

  • ATP-Hard [9], a causal-temporal split of Next-QA validation [52], has 2,269 multiple-choice instances; each question-option pair is converted to a statement and scored for entailment.The highest-entailment statement determines the predicted answer.Source: §I.2 · pp. 16–17
Loading 2311.10111v1…