Source-linked AI summary
Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding
Zhenzhi Wang, Limin Wang, Tao Wu, Tianhao Li, Gangshan Wu
TL;DR
Temporal grounding must match language queries to video moments, while prior detection/regression pipelines underuse cross-modal negative relations. MMN reframes grounding as metric learning with mutual matching and inter-video negatives, achieving state-of-the-art performance on four video grounding benchmarks. The paper concludes that metric learning remains promising for capturing cross-modal correlation in a joint embedding space.
Problem
Prior temporal-grounding methods mainly use detection or regression on fused representations and do not fully model negative relations between moments and language queries.
Method
MMN models moment–query similarity in a joint embedding space using symmetric cross-modal mutual matching and inter-video negative pairs.
Results
MMN achieves state-of-the-art performance on Charades-STA, TACoS, ActivityNet-Captions, and HC-STVG.
Takeaways & Limitations
The results support metric learning as a promising approach for capturing essential cross-modal correlation in temporal grounding.
Takeaways & Limitations
Negative sampling assumes a sufficiently large training video corpus, and sampled negative moments are restricted to IoU no greater than 0.5.
Abstract
from arXiv · showhide
Temporal grounding aims to localize a video moment which is semantically aligned with a given natural language query. Existing methods typically apply a detection or regression pipeline on the fused representation with the research focus on designing complicated prediction heads or fusion strategies. Instead, from a perspective on temporal grounding as a metric-learning problem, we present a Mutual Matching Network (MMN), to directly model the similarity between language queries and video moments in a joint embedding space. This new metric-learning framework enables fully exploiting negative samples from two new aspects: constructing negative cross-modal pairs in a mutual matching scheme and mining negative pairs across different videos. These new negative samples could enhance the joint representation learning of two modalities via cross-modal mutual matching to maximize their mutual information. Experiments show that our MMN achieves highly competitive performance compared with the state-of-the-art methods on four video grounding benchmarks. Based on MMN, we present a winner solution for the HC-STVG challenge of the 3rd PIC workshop. This suggests that metric learning is still a promising method for temporal grounding via capturing the essential cross-modal correlation in a joint embedding space. Code is available at https://github.com/MCG-NJU/MMN.
1 Introduction
Temporal grounding seeks the video interval matching a language query, but prior detection/regression methods underuse cross-modal negative relations. MMN reframes the task as metric learning with mutual matching and inter-video negatives to learn stronger joint representations.
- Temporal grounding identifies the start and end times of the moment that best corresponds semantically to a verbal description.
- Prior detection/regression approaches use fused multimodal features but largely ignore relations among all moment–query pairs.They commonly supervise only with IoU-based scores between a sentence and moments from the same video.
- MMN adds cross-modal mutual matching, contrasting the matched query with unmatched descriptions for a given video moment.The auxiliary task introduces negative textual relations alongside existing grounding supervision.
- MMN adopts symmetric matching directions so sentences and moments each help select the correct instance in the other modality.This creates more supervision than one-directional matching.
- The framework combines mutual matching with a separate similarity space that uses IoU supervision to rank moments and distinguish hard negatives.The binary mutual-matching signal is complemented by finer scalar similarity estimation.
- The contributions include late-fusion metric learning with inter-video negatives, a direct mutual-matching auxiliary task, and evaluation on four grounding benchmarks.
2 Related Works
Related work covers regression-based temporal grounding, auxiliary tasks, spatio-temporal grounding, and metric learning. MMN extends these directions by measuring visual-language similarity in a late-fusion joint space and adapting to spatio-temporal grounding.
- Temporal Grounding Methods: Regression-based temporal grounding predicts target boundaries from fused multimodal features using boundary classification or direct boundary regression.
- Auxiliary Tasks in Temporal Grounding: Prior auxiliary-task approaches explored query reconstruction by adding video-captioning losses to temporal grounding.
- Metric Learning: MMN measures cross-modal similarity in a late-fusion joint visual-language space and shares video features among sentences from the same video during training.The paper reports this design as advantageous for training computational cost.
- Spatio-Temporal Video Grounding: Spatio-temporal video grounding extends temporal grounding to include spatial action localization.
- Spatio-Temporal Video Grounding: Adapting MMN to linked human bounding boxes for temporal trimming yields significant gains over a previous transformer-based spatio-temporal grounding method.
- Metric Learning: Metric learning typically selects positives and negatives by class or augmentation, whereas MMN selects them according to temporal-grounding ground truth.
3 Model
MMN formulates temporal grounding as metric learning, using late fusion and bidirectional cross-modal matching to learn joint embeddings for moments and sentences. It combines mutual matching with IoU-based regression so negative pairs improve discrimination while IoU scores support finer moment ranking.
- Architecture: MMN uses late modality fusion, independently embedding video moments and language queries before combining them through cross-modal similarity.The architecture uses a Siamese-like design with a simple inner product in a joint visual-language space.
- Architecture: The video branch samples clips, builds a 2D candidate-moment feature map, and uses sparse sampling to reduce highly overlapping candidates and computation.Moment features are aggregated with max pooling and modeled using 2D convolutions based only on visual features.
- Loss Functions: MMN learns embeddings with two complementary objectives: BCE regresses scaled IoU values, while mutual matching discriminates matched from unmatched moment-sentence pairs.The final loss is their linear combination, and the final candidate score is the product of the IoU and mutual-matching scores.
- Loss Functions: Mutual matching adds negative sentence samples and inter-video negatives, contrasting them with positive pairs to learn more discriminative modality representations.Negative moments come from low-IoU moments within videos or other videos; negative sentences come from other sentences in the same or other videos.
- Loss Functions: The mutual-matching objective maximizes bidirectional cross-modal matching likelihood, with normalized embeddings and conditional distributions over sampled moments and sentences.The training corpus is assumed to be large enough for the sampling scheme.
4 Experiments
Experiments evaluate MMN on temporal and spatio-temporal grounding benchmarks using standard retrieval metrics, ablations, comparisons, visualizations, and computational-cost measurements. Results show that mutual matching with diverse negatives improves performance, while MMN remains competitive across datasets and extends successfully to action-tube localization.
- Evaluation Setup: Rank n@m measures the percentage of queries with at least one moment of IoU ≥m among the top-n retrieved moments.Experiments use dataset-specific IoU thresholds and n ∈{1, 5}.
- Ablation Study: Mutual matching improves performance over the baseline, with the strongest results often using both intra-video and inter-video negatives.Inter-video negatives alone tend to outperform intra-video negatives on smaller datasets, while combining both commonly performs best.
- Ablation Study: MMN consistently outperforms 2D-TAN across moment-length intervals on Charades-STA.The dataset contains substantially more short than long ground-truth moments, so the intervals are unbalanced.
- Comparison with the State of the Art: MMN achieves highly competitive results on ActivityNet Captions, Charades-STA, and TACoS, outperforming or matching most prior methods.The comparisons include strong baselines and methods using related metric-learning or multimodal architectures.
- Comparison with the State of the Art: Training converges in 10 GPU hours for MMN versus 36 GPU hours for the baseline on ActivityNet-Captions.MMN shares video features among sentences in the same video, reducing training cost; testing can also benefit when videos contain many sentences.
- Qualitative Analysis: Combining IoU-regression and pair-discrimination scores improves final performance by about 0.5% in the reported experiments.The authors describe IoU scores as sharp near the ground truth but error-prone, while pair-discrimination scores better cover the location but are less sharp among hard negatives.
- Spatio-Temporal Video Grounding: On HC-STVG, MMN outperforms several baselines and transformer-based methods and ranks first in the 3rd PIC workshop challenge.The extension uses candidate action tubes and does not rely on extra-large-scale video multimodal pretraining.
5 Conclusion
MMN applies metric learning to temporal grounding by using mutual matching and inter-video negatives, achieving state-of-the-art performance on four challenging video grounding benchmarks.
- MMN uses mutual matching to select the correct sentence for video moments and exploits inter-video negatives to learn more discriminative cross-modal features.
- MMN achieves state-of-the-art performance on Charades-STA, TACoS, ActivityNet-Captions, and HC-STVG.