Source-linked AI summary
Open-book Video Captioning with Retrieve-Copy-Generate Network
Ziqi Zhang, Zhongang Qi, Chunfeng Yuan, Ying Shan, Bing Li, Ying Deng, Weiming Hu
TL;DR
Video captioning relies on visual content alone, which can yield generic captions, while a model's knowledge domain cannot be expanded after training without retraining. The paper proposes retrieving video-relevant sentences as guidance and dynamically copying their expressions or generating words from video content. It reports state-of-the-art results on VATEX and superior performance on MSR-VTT.
Problem
Video captioning methods rely on video content alone, which can leave generation without adequate guidance and produce generic sentences; the model's knowledge domain also cannot be expanded after training without retraining.
Source: §1 · p. 1Method
The Retrieve-Copy-Generate network retrieves video-relevant sentences from a text corpus and uses a copy-mechanism generator to copy expressions or generate words from video content.
Source: §1 · pp. 1–2Results
The proposed approach achieves state-of-the-art results on VATEX and superior performance on MSR-VTT.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that copying knowledge beyond retrieved sentences, including subtitles and text from video or speech, is practical for comprehensive information acquisition and better generation.
Source: §5 · p. 8The visual encoder is specified under an appearance-feature assumption.
Source: §3.1 · p. 4
Abstract
from arXiv · showhide
Due to the rapid emergence of short videos and the requirement for content understanding and creation, the video captioning task has received increasing attention in recent years. In this paper, we convert traditional video captioning task into a new paradigm, \ie, Open-book Video Captioning, which generates natural language under the prompts of video-content-relevant sentences, not limited to the video itself. To address the open-book video captioning problem, we propose a novel Retrieve-Copy-Generate network, where a pluggable video-to-text retriever is constructed to retrieve sentences as hints from the training corpus effectively, and a copy-mechanism generator is introduced to extract expressions from multi-retrieved sentences dynamically. The two modules can be trained end-to-end or separately, which is flexible and extensible. Our framework coordinates the conventional retrieval-based methods with orthodox encoder-decoder methods, which can not only draw on the diverse expressions in the retrieved sentences but also generate natural and accurate content of the video. Extensive experiments on several benchmark datasets show that our proposed approach surpasses the state-of-the-art performance, indicating the effectiveness and promising of the proposed paradigm in the task of video captioning.
1. Introduction
The introduction frames video captioning as limited by reliance on video content alone, which can produce generic captions, and by a knowledge domain that cannot be expanded after training. It proposes open-book captioning, using retrieved sentences as guidance that the generator can copy from or build on; the authors report state-of-the-art results on VATEX and superior performance on MSR-VTT.
1. Introduction
- Existing captioning methods rely on video content alone, which can leave generation without adequate guidance and produce generic sentences.The introduction also notes that a model's memory or knowledge domain is fixed after training unless it is retrained.Source: §1 · p. 1
- Open-book video captioning retrieves sentences relevant to a video and uses them as extra hints or guidance for caption generation.At inference, the generator can produce words from video content or copy expressions from retrieved sentences.Source: §1 · pp. 1–2
- The flexible retriever and changeable corpus allow the model to be extended or revised.The introduction contrasts using a changeable corpus at inference with traditional semi-supervised learning from fixed examples during training.Source: §1 · p. 2
- The proposed Retrieve-Copy-Generate network pairs an improved cross-modal retriever with a copy-mechanism generator that dynamically copies expressions and generates words.The introduction describes retrieving sentences from a training-set text corpus, then passing the retrieved sentences and visual features to the generator.Source: §1 · p. 2
- The proposed approach achieves state-of-the-art results on VATEX and superior performance on MSR-VTT.The authors attribute benefits to combining cross-modal retrieval with copy-mechanism generation.Source: §1 · p. 2
2. Related Works
Prior video-captioning work either retrieves descriptions or generates them from visual features, while this paper’s related-work framing combines retrieval with encoder-decoder generation. Its retriever builds on the efficiency of bi-encoders while addressing the complementary strengths of retrieval and generation.
2. Related Works
- Retrieval-based description methods transfer retrieved captions but can struggle to produce entirely correct descriptions, while encoder-decoder methods offer limited expression diversity and controllability.The paper positions its approach as coordinating both methods to seek diverse expressions and accurate video content.Source: §2 · p. 2
- Bi-encoders efficiently embed queries and candidates separately, whereas cross-encoders offer richer interactions and higher accuracy at steep computational cost.The authors choose to improve a bi-encoder-based model as more applicable to their paradigm.Source: §2 · p. 2
- Retrieval-augmented NLP work spans question answering, translation, and dialogue; this paper extends that pattern to cross-modal video captioning.The related works cite REALM [10], ORQA [17], DPR [15], RAG [18], SEG-NMT, RER, and DeepCopy [32] as examples.Source: §2 · pp. 2–3
3. Retrieve-Copy-Generate Network
Retrieve-Copy-Generate first retrieves video-relevant sentences, then generates a caption using those sentences alongside visual features. Its retriever ranks corpus sentences in a joint embedding space, while its generator combines visual decoding with dynamic copying and supports separate or iterative end-to-end training.
3. Retrieve-Copy-Generate Network
- RCG retrieves top-k sentences for a video, then predicts each caption token from retrieved text, visual information, and previously generated tokens.Retrieved sentences can provide related expressions, while their retrieval probabilities act as a soft threshold for copying.Source: §3 · p. 3
3.1. Effective Video-to-Text Retriever
- The bi-encoder maps corpus sentences and videos into a joint semantic space, ranks them by dot-product similarity, and returns the top-k closest sentences.Textual embeddings can be computed offline for efficient evaluation; the retrieval corpus may contain nearly 0.3 million sentences.Source: §3.1 · pp. 3–4
- A bi-LSTM encodes sentence words, and multiplicative attention aggregates discriminative word features; appearance and motion features are aggregated separately.The visual features are combined by averaging appearance and motion similarities.Source: §3.1 · p. 4
3.2. Copy-mechanism Caption Generator
- The hierarchical decoder attends to visual features to produce fixed-vocabulary words, while dynamic multi-pointers attend to each retrieved sentence and select whether to copy or generate.The pointer module uses the decoder state to distribute attention over each sentence’s words; copying probabilities depend on retrieved-sentence context and decoder state.Source: §3.2 · pp. 4–5
3.3. Training
- The retriever and generator can be trained separately or jointly; joint training adds ranking loss to generation loss and refreshes retrieval periodically for stability.The paper performs retrieval once per epoch in its iterative training procedure.Source: §3.3 · p. 5
4. Experiments
RCG is evaluated on MSR-VTT and VATEX against state-of-the-art methods and through analyses of retrievers, retrieved-sentence counts, corpus quality, generalization, and qualitative outputs.
4.1. Experimental Settings
- Experiments use MSR-VTT and VATEX; most are reported on VATEX, whose annotations are longer and more accurate, with CIDEr emphasized among caption metrics.Caption metrics include BLEU-4, Meteor, Rouge-L, and CIDEr; retrieval is measured with R@K, MedR, and MnR.Source: §4.1 · p. 5
4.2. Quantitative Analysis
- Training with three retrieved sentences helps; more test-time hints help until saturation, while too many noisy training sentences hurt and the model adaptively selects useful cues.The model is tested using top-10 retrieved sentences in the training-count comparison.Source: Table 2 · p. 6, Table 3 · p. 6, §4.2 · p. 6
- Larger retrieval corpora improve generation, and an Oracle test corpus assumed to contain correct descriptions yields further significant improvement; Oracle is not used for training.Corpus quality is simulated through corpus size.Source: Table 4 · p. 6, §4.2 · pp. 6–7
- Better-performing retrievers significantly improve generation; with a randomly initialized worst retriever, results resemble the no-retriever baseline, indicating generator robustness.Source: §4.2 · p. 6
- Jointly trained retrieval improves accuracy and convergence speed over fixed retrieval, but trains at 127.2 versus 137.6 videos/sec, an 8.2% slowdown.Figure 4 plots CIDEr scores during VATEX training.Source: Figure 4 · p. 7, §4.2 · p. 7
- VATEX-trained RCG performs better than the baseline on unseen MSR-VTT; a higher-quality corpus and an MSR-VTT-trained retriever further improve cross-dataset performance.The experiments train on VATEX and test on MSR-VTT.Source: Table 5 · p. 7, §4.2 · p. 7
4.4. Qualitative Analysis
- Attention focuses on retrieved-sentence keywords, while RCG produces more diverse expressions and can correct descriptions such as “a game of frisbee” to “playing catch.”Source: Figure 5 · p. 8, §4.4 · p. 8
4.3. Comparison to State-of-the-Arts
- RCG exceeds ORG-TRL by 3.9% relative CIDEr on MSR-VTT and 15.7% on VATEX, with best VATEX results and comparable other MSR-VTT metrics.The comparison uses standard motion and appearance features without reinforcement learning or audio features.Source: Table 6 · p. 8, §4.3 · p. 8
5. Conclusion
The conclusion presents RCG as an open-book captioning system that retrieves and copies multiple sentence cues, and reports advantages on two large-scale datasets.
5. Conclusion
- RCG retrieves relevant sentences, copies cues from multiple retrievals, and generates captions; the system can be optimized separately or end-to-end.Source: §5 · p. 8
- Experiments on two large-scale video-caption datasets demonstrate advantages for the proposed method.Source: §5 · p. 8
- The authors suggest subtitles, on-screen text, and speech text as additional knowledge sources for comprehensive information acquisition and better generation.Source: §5 · p. 8
- Future work will explore those text sources and more advanced retrieval to improve model efficiency.Source: §5 · p. 8