PaperlayerView exact revision on arXiv

Source-linked AI summary

IntentQA: Intent Question Answering in Videos by Cognitive Context Reasoning

Jiapeng Li, Ping Wei, Wenjuan Han, Song-Chun Zhu, Lifeng Fan

arXiv:2608.23330v1cs.CV

TL;DR

The paper addresses the difficulty of inferring latent human intentions rather than merely recognizing visible video facts. It introduces the IntentQA task and benchmark, then proposes X-CaVIR, whose context-aware and transparent LLM pipeline combines captions with VQA outputs. The reported experiments show superior performance, interpretable reasoning, and stability on contrast sets.

  • Problem

    Video models recognize observable facts but struggle to infer the unobservable mental states underlying human behavior.

  • Method

    The paper builds IntentQA and X-CaVIR around situational, contrastive, and commonsense contexts, using contrast sets and an LLM pipeline that combines captions with VQA confidence scores.

  • Results

    X-CaVIR achieves superior performance, provides transparent reasoning pathways, and remains stable against perturbations on five contrast sets.

  • Takeaways & Limitations

    IntentQA frames understanding hidden social intentions as a distinct VideoQA problem for evaluating higher-level video reasoning.

Abstract

from arXiv · show

Video understanding requires intelligent agents to transcend mere recognition of visual facts and comprehend the underlying intents behind human actions (often termed the "dark matter" of social intelligence). To bridge the gap between visual observation and intent reasoning, we introduce a novel task, IntentQA, and contribute a large-scale VideoQA dataset specifically tailored for this purpose. However, recognizing that standard metrics may overestimate capabilities due to dataset biases, we go beyond simple accuracy to rigorously evaluate model robustness. We augment the benchmark by generating five distinct contrast sets via Large Language Models (LLMs) and introducing a "Contrast Performance Decline" metric. We propose the X-CaVIR (eXplainable Context-aware Video Intent Reasoning) framework, which leverages three types of "Cognitive Context" to enhance video analysis: i) Situational Context via a cross-modal Video Query Language (VQL) module, ii) Contrastive Context via a Contrastive Learning module, and iii) Commonsense Context via a Commonsense Reasoning module. Crucially, to overcome the opacity of traditional black-box models, we refine the integration of LLMs within X-CaVIR by employing a transparent pipeline that synergizes video captions with VQA model outputs. This approach not only improves performance by effectively utilizing rich commonsense knowledge but also renders the reasoning process explicitly interpretable. Extensive experiments demonstrate the effectiveness of our components, the superiority of X-CaVIR over state-of-the-art baselines, and its stability against perturbations on the contrast sets.

I. INTRODUCTION

The paper introduces IntentQA to test inference about characters’ latent intentions in videos, addressing limits of factoid-focused VideoQA. It evaluates robustness beyond accuracy and proposes X-CaVIR, which combines cognitive contexts with interpretable LLM reasoning.

  • IntentQA asks agents to answer questions about characters’ psychological goals and distinct intentions in videos.
  • The task relies on situational, contrastive, and commonsense contexts to interpret actions whose meanings vary across settings.
  • The dataset covers Causal Why, Causal How, Temporal Previous, and Temporal Next question categories.
  • Five contrast sets and the Contrast Performance Decline metric extend evaluation beyond standard accuracy, which can be inflated by dataset biases.
  • X-CaVIR integrates VideoQA confidence scores, dense video captions, and LLMs to improve performance while exposing interpretable reasoning pathways.

II. RELATED WORK

VideoQA has progressed from recognition toward spatial, temporal, causal, and relational reasoning, while intent understanding remains comparatively underexplored. Related work also motivates incorporating contextual knowledge and LLM-based reasoning, which this study extends with detailed reasoning processes.

  • VideoQA methods: VideoQA methods include memory, attention, Transformer, neural modular, neural-symbolic, and graph-structured approaches.These techniques accompany the field’s movement toward more complex reasoning over video content.
  • VideoQA reasoning: VideoQA research has shifted from answering visual facts toward reasoning about spatial, temporal, causal, and relational information.Recent datasets and methods address increasingly complex inference among multiple entities and relations.
  • Intent understanding: Few VideoQA studies address the unobserved human mental states underlying visible actions, motivating intent-focused video question answering.The paper positions its study as the first VideoQA work targeting this gap, according to the cited passage.
  • Context and commonsense reasoning: Contextual information and commonsense knowledge are important for inference questions, with prior systems using knowledge bases or pretrained language models.Existing commonsense reasoning approaches differ in whether world knowledge is incorporated explicitly through external resources or implicitly through language models.
  • LLM integration: LLM-based vision-language systems commonly combine language models, visual encoders, and alignment layers, while earlier VQA methods used captions, captions with QA evidence, or VQA predictions.System-based methods further position LLMs as coordinators or reasoners that select models, reflect iteratively, or generate executable code.
  • This work: This work adopts an LLM-as-reasoner approach and requires detailed reasoning processes to support qualitative failure analysis and model refinement.The stated benefit is improved insight into model weaknesses and decision-making.

III. DATASET

IntentQA is built from diverse intent-focused VideoQA examples and extended with contrast sets that test sensitivity to semantic and gender-based answer perturbations.

  • Main Set: IntentQA uses causal and temporal inference questions from NExT-QA rather than factoid descriptive questions.The selected forms include Causal Why, Causal How, Temporal Previous, and Temporal Next questions.
  • Main Set: At least three annotators had to agree before a sample entered the final dataset.Filtering required physical, observable human actions and semantically and physically matching actions across contexts.
  • Main Set: 4,303 videos and 16,297 question-answer pairs comprise the main IntentQA dataset, covering 624 actions, 193 lemmatized verbs, and 162 action IDs.The data are split into training, validation, and testing subsets at an approximate 6:1:1 ratio.
  • Contrast Set: The contrast benchmark uses LLM-generated distractors to evaluate robustness beyond standard accuracy.The extension refines earlier text-based contrast-set generation with a more sophisticated LLM-based process.
  • Contrast Set: Gender contrast replaces gender-sensitive terms in correct answers with their opposite-gender counterparts and substitutes a randomly selected incorrect answer.The resulting answer set directly tests sensitivity to gender-specific linguistic alterations.
  • Contrast Set: High-Similarity and Low-Similarity contrast options are selected from the top and bottom 15% of semantic similarity candidates.High-Similarity options require fine-grained discrimination, while Low-Similarity options represent more distinct alternatives.

IV. MODEL

X-CaVIR formulates intent reasoning as answer selection for VideoQA and combines situational, contrastive, and commonsense contexts in its reasoning pipeline.

  • Task Definition: IntentQA takes a video, question, and answer set as input and outputs the correct answer from that set.Its distinguishing feature is that all questions concern intent understanding.
  • Framework Overview: X-CaVIR senses context through situational, contrastive, and commonsense components.The framework is introduced to solve intent-focused VideoQA rather than only recognize observable facts.
  • Contrastive Context: Contrastive learning uses positive and negative samples with the same action to obtain contrastive context from the situational representation.The pipeline aligns top-k attention nodes and calculates triplet loss before final feature fusion.
  • Situational Context: VQL extracts question-relevant visual context by querying video region features with BERT-derived question-and-answer features.The resulting cross-modal graph represents situational context for subsequent reasoning.
  • Feature Fusion: A multi-head self-attention transformer fuses frame/region and question-relevant graphs into a composite video representation.Commonsense context is then used in the test pipeline to predict the final result.

B. Video Query Language (VQL)

The VQL module converts question-and-answer features into question-relevant visual context by cross-modal querying and fusion with video region features.

  • Cross-Modal Querying: VQL queries BERT-derived question-and-answer features against a video region graph to compute a cross-modal similarity matrix.This operation links language content to relevant visual regions.
  • Feature Transformation: Multiplying the similarity matrix by the language features transforms them into the visual feature space.The transformed representation is then fused with the queried visual features.
  • Situational Context: The fused graph G_{r|q,A} represents the question-relevant video contexts used as situational context.This graph is the output of the VQL context-extraction stage.

C. Contrastive Learning

The contrastive-learning module forms triplets from anchor, positive, and negative QA samples selected using action and answer similarity, then trains aligned representations with triplet and classification objectives.

  • Contrastive sample selection: Positive and negative QA examples are selected by combining action-similarity levels with WUPS-based answer-similarity thresholds.Action similarity uses action consistency, lemmatized verb consistency, or action ID consistency; WUPS thresholds distinguish positive and negative answer pairs.
  • Context feature extraction: The model extracts the top-k cross-modal graph nodes with highest relevance to the question and answer set.A similarity matrix determines which graph nodes are retained as the feature representation for each triplet sample.
  • Representation alignment: Anchor, positive, and negative features are aligned through dot-product-based representations before distance computation.The positive and negative examples are aligned with the anchor representation to produce d(a,p) and d(a,n).
  • Optimization objective: The triplet objective maximizes the margin between anchor-positive and anchor-negative distances while preventing negative pairs from becoming insufficiently separated.The loss is L_triplet = max(d(a,p) - d(a,n) + margin, 0).
  • Training objective: Matching scores are computed by dot products between composite video features and language features, and combined with cross-entropy and triplet losses.The complete loss includes classification terms for anchor, positive, and negative samples together with L_triplet.

D. Commonsense Reasoning

The commonsense-reasoning pipeline converts visual-model predictions and video captions into an explicit LLM prompt, which returns both an answer and a reasoning process.

  • Confidence Prompting: Confidence Prompting formats answer options together with visual matching scores as confidence-annotated candidate inputs.The candidate representation concatenates each answer a_i with its matching score S_i.
  • Video captioning: A caption model generates a textual description of the video frames to provide additional visual context.The paper uses Qwen 7B to produce the video description D.
  • LLM reasoning: The LLM receives confidence-embedded candidates, the video description, and the question, then selects an answer and produces reasoning.The pipeline is expressed as a* , R = LLM(C, D, Q).
  • Interpretable inference: The full pipeline combines context-aware VideoQA scores with dense captions before asking the LLM for a structured response.The requested response contains an option ID, the option text, and a reasoning process; inconsistent option IDs and option text are treated as incorrect.

A. Ablation Experiments

Ablation experiments compare progressively enriched model configurations, including situational, contrastive, and commonsense components, to diagnose their contributions.

  • Commonsense reasoning analysis: Confidence Prompting and caption integration are separately indexed as Models 6 and 7, enabling direct comparison with the conference-version Late Fusion baseline.Model 6 uses Confidence Prompting without dense captions, whereas Model 7 adds caption information.
  • Model component diagnosis: The ablation study compares a blind LLM, a simplified VGT base model, and models augmented with VQL and triplet-based training.VQL adds a cross-modal graph for situational context, while the triplet loader supplies anchor, positive, and negative samples during training.
  • Overall ablation outcome: The full model achieved the best performance across all tests, with every component contributing to performance improvement.The reported ablation results identify commonsense LLM integration as the largest contributor and caption integration as another major gain.
  • Fusion strategy: Confidence Prompting reached 58.48% accuracy, exceeding Late Fusion at 57.64% under the same InstructGPT backbone without dense captions.The comparison attributes the gain to the reasoning structure rather than merely to the LLM choice or auxiliary inputs.

2) Contrastive Learning Analysis:

Additional analyses examine contrastive-sample criteria, context-node selection, information-channel ablations, and LLM substitutions to identify conditions affecting performance.

  • Contrastive learning analysis: Model 4 with Action and t1 = 1 achieved the best contrastive-learning performance, supporting stricter action and answer similarity criteria.The analysis fixes t2 at 0.5 and compares t1 values of 0.85 and 1 across action-similarity definitions.
  • Context attention analysis: Randomly masking graph nodes reduced performance from 54.5 to 51.5, showing that retained context nodes are important for solving IntentQA.The analysis also compares random masking with masking the lowest-attention nodes.
  • Commonsense reasoning analysis: Removing either captions or VideoQA likelihood reduced performance relative to the complete model, with VideoQA likelihood removal causing the larger decline.The ablation isolates the two information channels in the commonsense-reasoning component.
  • LLM backbone comparison: GPT-4 achieved 64.81% accuracy, compared with 60.73% for InstructGPT and 59.28% for ChatGPT under identical conditions.ChatGPT and InstructGPT produced comparable enhancements, while GPT-4 produced the highest overall performance.
  • Information-channel sensitivity: Removing VideoQA likelihood reduced InstructGPT and ChatGPT performance by 6.7% and 3.8%, but GPT-4 declined by only 1.6%.The reported pattern indicates that GPT-4 is more resilient to the absence of this information channel.
  • Reasoning dependence: The impact of removing VideoQA likelihood diminishes as LLM reasoning strength increases, a trend also observed in contrast-set experiments.The authors connect this pattern to greater reliance on the LLM’s reasoning capabilities.

B. Comparison with VideoQA Baselines

X-CaVIR outperforms established VideoQA baselines on IntentQA, where commonsense and contrastive context support reasoning about unobservable intent. Qualitative analyses show that VQL and LLM-based commonsense reasoning help select contextually appropriate answers.

  • Quantitative comparison: The full model performs best, while the model without GPT performs second best among established VideoQA baselines.Early factoid-oriented models perform poorly on IntentQA, and recent VGT and HQGA models still show a large performance gap.
  • Context effects: Contrastive situational context improves CW and CH question answering but only slightly improves TP&TN performance.The reported gains vary across question types.
  • Context effects: Commonsense context further significantly improves performance across all types of question answering.
  • Qualitative analysis: VQL enables the model to attend to the context corresponding to the queried person, correcting an answer selected by the base model.In the example, the base model answers incorrectly, whereas VQL predicts the correct answer; a Blind LLM without video context also fails.
  • Qualitative analysis: LLM-based commonsense reasoning distinguishes closely related intents such as a mother feeding a baby from the baby eating food.The model with GPT and Blind LLM answer correctly where basic and contrastive models choose the more obvious visible action.
  • Qualitative analysis: Contrastive learning identifies a subtle injury as the cause of crying after the girl removes a sock, correcting commonsense and situational-context errors.The three models with contrastive context answer correctly in the illustrated example.

E. Experiments on the Contrast Set

The contrast-set experiments evaluate robustness beyond standard accuracy and show that X-CaVIR improves performance across perturbed subsets while exposing trade-offs among likelihood, captions, and reasoning. Qualitative failures indicate that LLM reasoning remains limited by missing visual details and shallow intent interpretations.

  • Contrast-set evaluation: Contrast Performance Decline measures the relative accuracy decrease from the main set to the contrast set.The formula uses A_base for main-set accuracy and A_contrast for contrast-set accuracy.
  • Performance and robustness: The proposed method outperforms both Blind LLM and Traditional V-L models on the main set and all five contrast subsets.The full model improves from 54.50% with InstructGPT to 60.73%, 59.28% with ChatGPT, and 64.81% with GPT-4.
  • Ablation analysis: Removing either VideoQA likelihood or captions decreases performance, with VideoQA likelihood producing a larger performance benefit but greater sensitivity to input changes.Captions provide more consistent robustness support, while stronger reasoning models such as GPT-4 mitigate likelihood-related performance drops.
  • LLM reasoning analysis: LLM reasoning improves option selection by combining scene analysis, commonsense inference, and elimination of alternatives rather than relying on surface video-option matching.In the stroller example, the method selects “push the stroller” over the semantically similar contrast option “drive the stroller.”
  • LLM reasoning analysis: The method can fail when captions omit critical events, causing the LLM to miss that a girl fell and choose “balance herself” instead of “support herself.”
  • LLM reasoning analysis: The LLM also misses deeper social intent when it recognizes making music together but fails to infer that the guitarist intended to energize the other musician.The authors state that a significant gap from human reasoning remains.

VI. CONCLUSION

The work introduces a robustness-oriented IntentQA benchmark and the X-CaVIR framework for inferring latent video intentions with interpretable reasoning.

  • IntentQA introduces a task and dataset for answering questions about characters’ psychological goals and distinct intentions in videos.
  • The benchmark adds five Contrast Sets to evaluate robustness beyond simple accuracy and mitigate dataset bias.
  • X-CaVIR combines visual observations with Large Language Model reasoning through situational, contrastive, and commonsense contexts.
  • Integrating video captions and VQA confidence scores provides state-of-the-art performance with transparent, interpretable reasoning pathways.
Loading 2608.23330v1…