Source-linked AI summary
MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment
Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, Larry S. Davis
TL;DR
Natural language moment retrieval must handle multiple candidate moments and temporal dependencies that misalign language with video. MAN combines language-aligned single-shot candidate encoding with iterative graph-based temporal reasoning, and reports state-of-the-art performance on Charades-STA and DiDeMo.
Problem
The task is to retrieve the temporal segment matching a natural-language query despite semantic and structural misalignment in untrimmed videos.
Method
MAN uses a single-shot fully convolutional network for language-aligned multi-scale candidate moments and an Iterative Graph Adjustment Network for learned temporal relations.
Results
MAN achieves state-of-the-art performance on the Charades-STA and DiDeMo benchmarks.
Takeaways & Limitations
The framework explicitly combines semantic alignment with moment-wise structural reasoning in an end-to-end model.
Abstract
from arXiv · showhide
This research strives for natural language moment retrieval in long, untrimmed video streams. The problem is not trivial especially when a video contains multiple moments of interests and the language describes complex temporal dependencies, which often happens in real scenarios. We identify two crucial challenges: semantic misalignment and structural misalignment. However, existing approaches treat different moments separately and do not explicitly model complex moment-wise temporal relations. In this paper, we present Moment Alignment Network (MAN), a novel framework that unifies the candidate moment encoding and temporal structural reasoning in a single-shot feed-forward network. MAN naturally assigns candidate moment representations aligned with language semantics over different temporal locations and scales. Most importantly, we propose to explicitly model moment-wise temporal relations as a structured graph and devise an iterative graph adjustment network to jointly learn the best structure in an end-to-end manner. We evaluate the proposed approach on two challenging public benchmarks DiDeMo and Charades-STA, where our MAN significantly outperforms the state-of-the-art by a large margin.
1. Introduction
Natural language moment retrieval must localize queried temporal segments in untrimmed videos despite semantic misalignment from multiple occurrences and structural misalignment from complex temporal dependencies. MAN addresses these challenges by aligning language and video representations while modeling relations among candidate moments in a graph.
- Natural language moment retrieval localizes the start and end of the temporal segment that best matches a verbal description.
- Multiple moments of interest make ordinal queries require searching across the whole video for the correct occurrence.The example query asks for the child touching the ground the second time.
- Complex temporal dependencies create structural misalignment when the language order differs from the video order.A moment of being close to the camera can occur before the described running away.
- Existing methods sample sliding-window candidates and compare each moment independently, making semantic and structural relations difficult to model.
- MAN generates language-aligned candidate moments across temporal locations and scales and models their temporal relationships in a graph-structured network.Its hierarchical convolutional network encodes the video, dynamic filters align language and visual features, and IGAN models candidate-moment relations.
- MAN is evaluated on Charades-STA and DiDeMo, where the paper reports significantly better performance than the state of the art.
2. Related Work
Earlier temporal activity detection methods generally target fixed activity classes, while natural language moment retrieval methods use shared embeddings and sliding-window or heuristic matching. MAN instead integrates language-video alignment with learned graph-based temporal reasoning.
- Temporal activity detection predicts activity start and end times together with activity labels in untrimmed videos.
- Most temporal activity detection methods focus on fixed activity classes without language queries.
- Natural language moment retrieval methods learn a shared embedding space for video segments and sentences, then compare them using sliding windows or handcrafted heuristics.
- Language-video fusion methods improve alignment through multilevel feature fusion or frame-by-word interactions but do not reason about complex cross-modal relations.
- MAN combines semantic alignment and structural reasoning in one unified network, using graph-structured moment relations to model temporal dependencies.
- Unlike fixed-graph graph convolutional networks and hand-coded temporal structures, MAN jointly learns temporal relations from data.
3. Model
MAN combines language-conditioned candidate moment encoding with graph-based temporal reasoning in an end-to-end single-shot framework. Its IGAN component jointly updates moment representations and the graph structure connecting candidate moments.
- Overall architecture: MAN directly generates candidate moment representations aligned with language semantics instead of processing each candidate independently.The framework integrates language and video information in a single-shot architecture.
- Language-conditioned encoding: Dynamic filters generated from encoded language features are convolved with video features to produce semantically aligned visual representations and matching scores.The filters are input-dependent and derive from the encoded sentence representation.
- Single-shot video encoding: A hierarchical convolutional video encoder produces multi-scale temporal feature maps whose cells correspond to candidate moments at particular locations and lengths.Temporal pooling progressively reduces temporal dimensions to create variable-length candidates.
- Temporal graph reasoning: Candidate moments are represented as graph nodes, enabling graph convolutions to compute each node’s response from neighboring moments defined by learned relations.The node feature matrix contains N candidate moments represented by d-dimensional vectors.
- Iterative graph adjustment: IGAN overcomes fixed-graph reasoning by iteratively learning adjacency weights and updating node representations in a fully differentiable, end-to-end process.It decomposes adjacency into preserving and residual components, then accumulates residual signals across recurrent cells.
4. Experiments
MAN is evaluated on DiDeMo and Charades-STA using standard retrieval metrics, implementation settings, benchmark comparisons, ablations, and qualitative visualizations. Results show consistent gains from the proposed architecture, feature alignment, iterative graph adjustment, and TAN visual features.
- Evaluation setup: MAN is evaluated on DiDeMo and Charades-STA using Rank@1, Rank@5, mIoU, and R@n, IoU@m metrics.For Charades-STA, R@n, IoU@m measures the percentage of queries with at least one correct retrieval among the top n moments at IoU threshold m.
- Implementation details: The model uses fixed 300-dimensional GloVe word2vec embeddings, a 512-unit LSTM, TAN video features, and end-to-end training from video frames and language queries.Sentences are truncated to 15 words, and the video feature dimension is reduced to 512.
- Comparison with state of the art: MAN improves over previous state-of-the-art methods by around 3% on all DiDeMo metrics and raises Charades-STA R@1, IoU=0.5 by over 10% in absolute values.On Charades-STA, MAN tops the compared methods across all evaluation metrics.
- Ablation studies: 23.56% on Rank@1 and 77.66% on Rank@5 are achieved by the single-shot video encoder alone on DiDeMo.This Base model removes feature alignment and IGAN cells while retaining the hierarchical single-shot video encoder.
- Ablation studies: Feature alignment improves the Base model, and word-level dynamic filters further improve performance through finer-grained video-language interactions.The evaluated variants use sentence-level filters from the final LSTM state or word-level filters from all LSTM hidden states.
- Ablation studies: Rank@1 decreases monotonically from 27.02% to 25.67% with fewer IGAN cells, while IGANx3 converges faster and generalizes better with smaller variance.The authors attribute the gains from additional cells to iterative optimization of temporal relations in the moment graph.
- Qualitative results: Qualitative results show MAN retrieving diverse moments, including queries requiring strong temporal dependencies such as identifying when a woman shows her face for the first time.The visualization marks ground-truth moments in black and retrieved moments in green for examples from Charades-STA and DiDeMo.
5. Conclusion
MAN unifies candidate moment encoding with temporal structural reasoning in a single-shot framework for natural language moment retrieval. It addresses semantic and structural misalignment through cross-modal alignment and iterative graph-based temporal reasoning, achieving state-of-the-art performance on Charades-STA and DiDeMo.
- MAN unifies candidate moment encoding and temporal structural reasoning in a single-shot structure for natural language moment retrieval.
- The framework addresses semantic and structural misalignment through a fully convolutional network for cross-modal alignment and an iterative graph adjustment network for moment-wise temporal relations.
- MAN achieves state-of-the-art performance on the challenging Charades-STA and DiDeMo benchmarks.