Source-linked AI summary
Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learning
Juncheng Li, Junlin Xie, Long Qian, Linchao Zhu, Siliang Tang, Fei Wu, Yi Yang, Yueting Zhuang, Xin Eric Wang
TL;DR
Temporal grounding datasets do not specifically test generalization to novel combinations of seen words. This paper introduces compositional grounding splits and VISA, which structures video and language as hierarchies and learns fine-grained correspondence; experiments report that tested state-of-the-art models fail to generalize while VISA achieves superior compositional generalizability.
Problem
Existing temporal grounding datasets do not specifically test whether models generalize to novel combinations of seen words.
Source: §1 · pp. 1–2Method
The paper introduces Charades-CG and ActivityNet-CG and proposes VISA, which decomposes video and language into semantic hierarchies and learns fine-grained cross-graph correspondence.
Source: §1 · p. 2Results
State-of-the-art models fail to generalize to novel compositions, while VISA achieves the highest performance on all proposed dataset splits and improves compositional generalizability.
Source: §1 · p. 2, §5.2 · p. 6, §6 · p. 8Takeaways & Limitations
The new splits measure generalization to novel combinations whose individual components were observed during training.
Source: §1 · p. 2VISA has failure cases distinguishing subtle adverb semantics, such as “fly close” versus “fly away.”
Source: §6 · p. 8
Abstract
from arXiv · showhide
Temporal grounding in videos aims to localize one target video segment that semantically corresponds to a given query sentence. Thanks to the semantic diversity of natural language descriptions, temporal grounding allows activity grounding beyond pre-defined classes and has received increasing attention in recent years. The semantic diversity is rooted in the principle of compositionality in linguistics, where novel semantics can be systematically described by combining known words in novel ways (compositional generalization). However, current temporal grounding datasets do not specifically test for the compositional generalizability. To systematically measure the compositional generalizability of temporal grounding models, we introduce a new Compositional Temporal Grounding task and construct two new dataset splits, i.e., Charades-CG and ActivityNet-CG. Evaluating the state-of-the-art methods on our new dataset splits, we empirically find that they fail to generalize to queries with novel combinations of seen words. To tackle this challenge, we propose a variational cross-graph reasoning framework that explicitly decomposes video and language into multiple structured hierarchies and learns fine-grained semantic correspondence among them. Experiments illustrate the superior compositional generalizability of our approach. The repository of this work is at https://github.com/YYJMJC/ Compositional-Temporal-Grounding.
1. Introduction
Temporal grounding can describe activities beyond predefined classes, but existing datasets do not specifically test whether models generalize to novel combinations of seen words. The paper introduces compositional temporal grounding benchmarks and VISA, which represents video and language hierarchically and learns fine-grained correspondence between them.
1. Introduction
- Existing temporal grounding datasets do not specifically test compositional generalization, and their training and testing splits contain almost the same compositions.The relevant novel combinations include verb-noun and adjective-noun pairs.Source: §1 · p. 1
- Charades-CG and ActivityNet-CG measure whether models generalize to novel combinations whose individual components were observed during training.The task is introduced as Compositional Temporal Grounding.Source: §1 · p. 2
- Evaluated state-of-the-art models fail to generalize to novel compositions despite progress on typical temporal grounding, and they are insensitive to shuffled word order.The authors report that shuffled word order destroys the original sentence semantics; they interpret the findings as evidence that current models rely heavily on superficial correlations.Source: §1 · p. 2
- Prior methods largely use unstructured global video and sentence representations, which do not explicitly model video structure or language composition.The paper motivates compositional reasoning through matching the individual meanings of “throws flowers” to video semantics and composing them.Source: §1 · p. 2
- VISA decomposes video and language into hierarchical graphs and learns fine-grained cross-graph semantic correspondence.The framework is named VarIational croSs-graph reAsoning; its hierarchies include global events, local actions, and atomic objects.Source: §1 · p. 2
2. Related Work
Related work covers temporal grounding methods and approaches to compositional generalization across language and vision tasks.
Temporal Grounding
Compositional Generalization
- Compositional generalization has been studied through SCAN, which tests instructions with compositions absent from training, and follow-up work on data augmentation, meta-learning, and architectural design [17] [1] [18] [34].Source: §2 · p. 2
- Related applications include image captioning, visual question answering, action recognition, and state-object recognition [33] [51] [52] [13] [14] [30].Source: §2 · p. 3
3. Compositional Temporal Grounding
The section defines Compositional Temporal Grounding to test generalization to novel combinations of seen words and introduces Charades-CG and ActivityNet-CG. Its dataset splits separate novel compositions and novel words from training while keeping videos disjoint across training and testing.
3.1. Problem Formulation
- Novel-composition and novel-word splits test generalization to unseen combinations of seen components and to queries containing unseen words, respectively.The novel-word split tests whether a model can infer potential word semantics from learned composition components in context.Source: §3.1 · p. 3
3.2. Dataset Re-splitting
- The re-splitting protocol forms training, novel-composition, novel-word, and test-trivial sets after removing instances predictable solely from videos.It identifies five composition types using lemmatization and dependency parsing, then samples training compositions so their components are observed during training.Source: §3.2 · p. 3
- Table 1 summarizes detailed statistics for Charades-CG and ActivityNet-CG.Source: Table 1 · p. 3, §3.2 · p. 3
- Queries associated with a video selected for a novel split are assigned to test-trivial, preventing video overlap between training and testing.Source: §3.2 · p. 3
4. Method
VISA represents video and language as hierarchical semantic graphs and learns fine-grained correspondence between them. It uses variational inference over latent correspondences to predict the target interval.
4. Method
- VISA decomposes video and queries into global events, local actions, and atomic objects, then learns fine-grained cross-graph correspondence to localize the target moment.Source: §4 · p. 3, §4.1 · p. 3
4.1. Hierarchical Semantic Graph
- Action and object nodes are connected by typed relations, enriched with relational and visual context, and aggregated into global event nodes.The resulting graphs provide unified structured representations that couple multiple semantic granularities across video and language.Source: §4.1 · pp. 4–5
4.2. Variational Cross-Graph Correspondence
- A latent variable represents cross-graph correspondence; training uses a target-informed posterior, while testing uses the learned prior without ground-truth boundaries.Source: §4.2 · p. 5, §4.3 · p. 6
- The model fuses graph representations, attends to segment features, pools them using the sentence feature, and predicts the temporal boundaries.Source: §4.2 · p. 6
4.3. Optimization
- The ELBO combines interval regression and KL-divergence terms to train boundary prediction and the prior’s cross-graph correspondence.The regression term also encourages the approximate posterior to learn more accurate correspondence.Source: §4.3 · p. 6
5. Experiments
Across Charades-CG and ActivityNet-CG, VISA leads existing methods, while analyses examine how contextualized learning, semantic hierarchy, and linguistic structure relate to compositional grounding.
5.1. Benchmarking the SOTA Methods
- The benchmark tests proposal-based, proposal-free, RL-based, and weakly supervised temporal grounding methods using R@n, IoU=m and mIoU.R@n, IoU=m measures the percentage of queries with at least one of the top-n predictions above the specified IoU threshold.Source: §5.1 · p. 6
5.2. Results on Compositional Temporal Grounding
- VISA achieves the highest performance across all dataset splits, with relative mIoU gains of 30.86% on Charades-CG and 23.32% on ActivityNet-CG novel-composition splits.The reported gains are relative to state-of-the-art methods.Source: §5.2 · p. 6, Table 2 · p. 7, Table 3 · p. 7
5.3. In-Depth Analysis
- Ablations indicate that contextualized learning and event-level hierarchy matter, while direct cross-graph fusion may disrupt semantic correspondence and reduce grounding performance.The study removes SCL, VCL, or HSA, or replaces VCC with cross-modal self-attention; detected semantic labels serve as unified symbols rather than directly accounting for the main gain.Source: §5.3 · pp. 6–7
- Verb-Noun compositions are the hardest type to generalize, requiring identification of corresponding actions and objects and joint reasoning about their novel combination.The composition-type analysis reports R@1, IoU=0.5 results.Source: Table 5 · p. 7, §5.3 · p. 7
5.4. Qualitative Analysis
- On shuffled queries, SOTA predictions remain nearer the original boundary, whereas VISA follows changed semantics; qualitative cases also show VISA handling event relations and novel compositions.The word-order analysis reports that VISA is most sensitive on novel-word splits and that SCL and VCC jointly promote capture of sentence structure.Source: §5.3 · p. 7, §5.4 · p. 8, Figure 3 · p. 8
- The learned hierarchical graph visualizes cross-graph semantic alignments, including visual “punching person” and “boxing ring” aligned with the phrase “perform kickboxing.”The visualization also connects “push up” and “exercise ball” with “push up (with) ball.”Source: Figure 5 · p. 8, §5.4 · p. 8
6. Conclusions
The paper introduces a task and framework for evaluating and improving compositional generalizability in temporal grounding. It reports significant improvement, while noting difficulty distinguishing subtle adverbial meanings.
- The paper introduces compositional temporal grounding and VISA, which learns fine-grained semantic correspondence between video and language across three semantic hierarchies.The task is intended to systematically evaluate temporal-grounding models’ compositional generalizability.Source: §6 · p. 8
- Experiments show significant improvement in VISA’s compositional generalizability.Source: §6 · p. 8
- VISA still fails on subtle adverb distinctions, such as “fly close” versus “fly away.”Source: §6 · p. 8