Source-linked AI summary
Object Relational Graph with Teacher-Recommended Learning for Video Captioning
Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, Zhengjun Zha
TL;DR
Video captioning needs richer visual representations and better training for content-specific words because existing methods overlook object interactions and face long-tailed caption vocabularies. The paper proposes ORG-TRL, combining relational graph encoding with teacher-recommended learning from an external language model. It reports state-of-the-art performance on MSVD, MSR-VTT, and VATEX, with ablations and visualizations supporting the methods' effectiveness.
Problem
Video captioning lacks adequate object-interaction representation and sufficient training for content-specific words in long-tailed caption corpora.
Method
ORG-TRL uses learnable object relational graphs with GCN-based reasoning and TRL, which adds external language-model word proposals to teacher-enforced learning.
Results
The ORG-TRL system achieves state-of-the-art performance on MSVD, MSR-VTT, and VATEX.
Takeaways & Limitations
Relational reasoning enriches detailed object features, while TRL exposes additional potential words and adds no sentence-generation computation at inference.
Abstract
from arXiv · showhide
Taking full advantage of the information from both vision and language is critical for the video captioning task. Existing models lack adequate visual representation due to the neglect of interaction between object, and sufficient training for content-related words due to long-tailed problems. In this paper, we propose a complete video captioning system including both a novel model and an effective training strategy. Specifically, we propose an object relational graph (ORG) based encoder, which captures more detailed interaction features to enrich visual representation. Meanwhile, we design a teacher-recommended learning (TRL) method to make full use of the successful external language model (ELM) to integrate the abundant linguistic knowledge into the caption model. The ELM generates more semantically similar word proposals which extend the ground-truth words used for training to deal with the long-tailed problem. Experimental evaluations on three benchmarks: MSVD, MSR-VTT and VATEX show the proposed ORG-TRL system achieves state-of-the-art performance. Extensive ablation studies and visualizations illustrate the effectiveness of our system.
1. Introduction
Video captioning must bridge visual and linguistic information, but existing approaches underrepresent object interactions and undertrain content-specific words. ORG enriches object representations through relational reasoning, while TRL supplements ground-truth supervision with external language knowledge; together they achieve state-of-the-art results on three benchmarks.
- Video captioning must represent both spatial appearance and temporal dynamics while matching visual features to captions.
- Existing visual features capture global information but miss detailed object dynamics and spatial-temporal relationships between objects.
- The caption corpus is long-tailed: function and common words greatly outnumber content-specific words, causing insufficient training for meaningful vocabulary.
- ORG connects objects through partial and complete relational graphs, then uses GCNs to enhance object representations during relational reasoning.
- TRL supplements teacher-enforced learning by using an external language model to recommend additional word proposals alongside ground-truth targets.
- The ORG-TRL system achieves state-of-the-art performance on MSVD, MSR-VTT, and VATEX.
2. Related Works
Prior video-captioning work uses global, temporal, object-level, or language-model-based methods, but important limitations remain in modeling object interactions and integrating external language models. ORG-TRL addresses these gaps with a temporal-spatial object graph and training-only teacher recommendations.
- Recent video-captioning methods commonly use encoder-decoder sequence learning with temporal attention, frame reconstruction, or object-level information.
- Existing object-level approaches mainly model global information or salient-object temporal structure without interactions among all objects in frames.
- Visual relational reasoning has been applied to image captioning, VQA, and action recognition using explicit, predefined, or learnable graphs.
- External language models have supported sequence-generation tasks such as machine translation and speech recognition through shallow, deep, and cold fusion.
- Shallow fusion can introduce distributional bias, while deep and cold fusion add inference-time dependence or architectural complexity.
- ORG-TRL constructs a learnable temporal-spatial object graph and uses TRL during training to integrate external linguistic knowledge without extra sentence-generation computation at inference.
3. Methodology
The system combines an object-relational-graph visual encoder with a hierarchical temporal-spatial decoder and teacher-recommended learning. ORGs model object interactions, while an external language model supplies soft word targets alongside ground-truth supervision.
- Object-relational-graph visual encoder: The encoder extracts appearance, motion, and object features, then uses a learnable ORG with GCNs to enhance object representations through relational reasoning.The system uses an encoder-decoder framework in which the ORG dynamically learns interactions among objects.
- Object-relational-graph visual encoder: P-ORG connects objects within each frame, whereas C-ORG connects all objects across the video using a complete graph.P-ORG shares graph parameters across frames; C-ORG is noisy when directly connecting a center node to every object.
- Hierarchical decoder: The decoder aligns objects across frames, applies temporal and spatial attention, and uses Attention LSTM and Language LSTM modules to generate captions step by step.Aligned object features provide local context, while appearance and motion features provide global context.
- Teacher-recommended learning: Teacher-recommended learning supplements teacher-enforced learning by matching caption-model distributions to top-k soft targets generated by a fixed external language model.The external model produces semantically reasonable proposals that supplement ground-truth words, while top-k selection removes low-probability words.
- Teacher-recommended learning: The combined objective balances cross-entropy training on hard targets with KL-divergence training on external-model soft targets using λ.TRL participates only during caption-model training and adds no computational burden during sentence generation at inference.
4. Experiments
Experiments evaluate ORG-TRL on MSVD, MSR-VTT, and VATEX using standard captioning metrics, comparisons, ablations, and qualitative analyses. Results show improvements over baselines and support the roles of relational object modeling and teacher-recommended learning.
- Experimental Setup: Evaluation covers MSVD, MSR-VTT, and VATEX using BLEU-4, METEOR, CIDEr, and ROUGE-L.MSVD and MSR-VTT use benchmark comparisons, while VATEX results come from its online evaluation system.
- Experimental Setup: The authors align feature extractors and preprocessing with recent models, while noting that different extraction methods prevent completely fair comparison.The comparison includes visual modalities such as appearance, motion, and object features.
- Performance Comparison: Significant improvements on MSVD and MSR-VTT, especially under CIDEr, are reported against models using the same features.The authors associate the CIDEr gains with generating novel words and capturing detailed video information while using linguistic knowledge from the ELM.
- Performance Comparison: ORG outperforms OA-BTG on MSR-VTT, supporting the benefit of modeling relationships between objects.The authors note that OA-BTG and GRU-EVE use detailed object information but ignore object relationships.
- Ablation Experiments: Adding ORG or TRL alone improves the baseline, and combining both produces further performance gains on MSVD and MSR-VTT.The baseline uses appearance and motion features with the same encoder-decoder architecture but without the object encoder.
- Qualitative Analysis: Inference examples show TRL learning common word matches and content-related words, while qualitative examples show richer captions with detailed objects, relations, and positions.The examples include recognizing the relation “mixing” between “person” and “food” and the position “in a bowl.”