Source-linked AI summary
MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning
Jie Lei, Liwei Wang, Yelong Shen, Dong Yu, Tamara L. Berg, Mohit Bansal
TL;DR
Video paragraph captioning must preserve relevance and discourse coherence across multiple sentences while limiting repetition. MART adds a summarized external memory to a shared encoder-decoder transformer, using video-segment and sentence history to predict subsequent sentences. Across ActivityNet Captions and YouCookII, it produces more coherent and less redundant paragraphs without degrading relevance, although repetition within individual sentences remains.
Problem
Video paragraph captioning requires relevant, nonredundant, and coherent multi-sentence descriptions, while transformers can suffer context fragmentation across segments.
Method
MART augments a shared encoder-decoder transformer with an external memory module summarizing previous video segments and caption history.
Results
MART shows better overall performance than baseline methods, generating more coherent and less redundant paragraphs without degradation in relevance on two standard datasets.
Takeaways & Limitations
MART's sentence-level recurrence improves paragraph coherence and reduces redundancy while maintaining visual relevance across ActivityNet Captions and YouCookII.
Takeaways & Limitations
Repetition still occurs within individual sentences, leaving single-sentence repetition reduction as an area for further effort.
Abstract
from arXiv · showhide
Generating multi-sentence descriptions for videos is one of the most challenging captioning tasks due to its high requirements for not only visual relevance but also discourse-based coherence across the sentences in the paragraph. Towards this goal, we propose a new approach called Memory-Augmented Recurrent Transformer (MART), which uses a memory module to augment the transformer architecture. The memory module generates a highly summarized memory state from the video segments and the sentence history so as to help better prediction of the next sentence (w.r.t. coreference and repetition aspects), thus encouraging coherent paragraph generation. Extensive experiments, human evaluations, and qualitative analyses on two popular datasets ActivityNet Captions and YouCookII show that MART generates more coherent and less repetitive paragraph captions than baseline methods, while maintaining relevance to the input video events. All code is available open-source at: https://github.com/jayleicn/recurrent-transformer
1 Introduction
Video paragraph captioning requires descriptions that remain relevant, nonredundant, and coherent across multiple event segments. MART addresses transformer context fragmentation with a shared encoder-decoder and external memory, producing more coherent and less redundant paragraphs without losing relevance.
- Task and challenge: Dense video captioning decodes coherent multi-sentence descriptions from temporal event segments, while paragraph captioning can assume the segments are provided.The simplified task focuses on decoding better paragraph captions rather than generating event segments.
- Task and challenge: Generating relevant, less redundant, and coherent sentences across a paragraph remains challenging.
- Task and challenge: Transformer captioning models face context fragmentation because separated segments lack information flow across segments and surrounding context.This limits their ability to capture longer-range dependencies needed for coherent video paragraphs.
- MART: MART augments a shared encoder-decoder transformer with an external memory module that models previous video segments and sentence history.The memory state summarizes prior video and caption information, while the shared design uses common transformer layers for encoding and decoding.
- MART: MART generates more coherent paragraphs, including improved coreference and ordering, with less redundancy and no loss of paragraph accuracy.
2 Related Work
Related work includes sentence-wise concatenation, recurrent LSTM approaches, adversarial reranking, and transformer-based captioning. These methods differ in how they model cross-sentence context, but transformer context fragmentation motivates recurrence-based alternatives.
- Video paragraph captioning: Some methods decode each video event segment separately and concatenate the resulting sentences, which can produce inconsistent and redundant paragraphs.
- Recurrent and adversarial methods: LSTM-based approaches pass hidden states through the decoding process or use discriminators for relevance, linguistic diversity, and inter-sentence coherence.
- Recurrent and adversarial methods: Adversarial inference reranks generated sentences using separately trained discriminators as an orthogonal way to improve caption quality.
- Transformer methods: Transformers use self-attention rather than recurrent structure to model dependencies and have shown stronger performance than RNNs in sequential tasks.
- Transformer methods: Context fragmentation occurs when transformer language segments are modeled individually without surrounding context, motivating Transformer-XL's recurrence mechanism.
3 Methods
MART extends a shared encoder-decoder transformer with external memory to model video-segment and sentence history for coherent paragraph captioning. Its summarized memory updates retain important information while reducing redundancy across generated sentences.
- Task: Video paragraph captioning generates sentences for temporally ordered video segments, with each sentence describing its corresponding segment.
- Vanilla Transformer: The vanilla transformer uses separated encoder and decoder networks, multi-head attention, feed-forward layers, residual connections, and layer normalization.Masked multi-head attention prevents the decoder from seeing future words.
- MART: MART instead shares transformer layers between encoder and decoder, concatenating encoded video and text embeddings with token-type indicators.Loss calculation and word generation use only text-token positions.
- MART: MART augments the unified transformer with external memory that integrates previous video segments and caption history when generating the next sentence.At each layer and decoding step, multi-head attention aggregates the current intermediate states with the previous memory state.
- MART: The memory updater uses an update gate to control retained information from the previous memory state, reducing redundancy and maintaining paragraph coherence.Its multi-head-attention-based update supports multiple memory slots rather than the single state used by LSTM and GRU.
- Transformer-XL Comparison: Unlike Transformer-XL, which directly carries previous hidden states with relative positional encoding, MART uses same-layer recurrence and highly summarized memory states.The authors suggest summarization may discard less relevant and repetitive information while preserving important history.
4 Experiments
MART is evaluated on ActivityNet Captions and YouCookII using automatic metrics, human judgments, ablations, and qualitative analyses. Results indicate improved coherence and reduced redundancy while preserving relevance, with a remaining issue of within-sentence repetition.
- Evaluation setup: Experiments use ActivityNet Captions and YouCookII, with paragraph-level automatic metrics and human evaluations of relevance and coherence.The automatic metrics include BLEU@4, METEOR, CIDEr-D, and R@4 repetition.
- Automatic evaluation: MART achieves the best CIDEr-D and R@4 scores on ActivityNet Captions, with comparable BLEU@4 and METEOR performance to competing models.The strongest BLEU@4 and METEOR comparison models use additional detection features or grounding supervision, limiting fairness of those comparisons.
- Ablation: Ablations show that recurrence improves overall performance, while two hidden layers and memory state length 1 provide a balance between performance and computation.The ablation compares recurrent MART models with a variant without recurrence.
- Qualitative analysis: Qualitative examples associate MART’s recurrence with more coherent, less redundant paragraphs and better pronoun consistency, but repetition still occurs within individual sentences.The authors attribute reduced cross-sentence redundancy to highly summarized previous memory states that remove redundant information.
- Memory analysis: Memory-state nearest-neighbor retrieval produces mostly related activities despite the memory not being explicitly trained for retrieval.The experiment uses the last-step first-layer memory state and cosine similarity on 10K ActivityNet Captions training videos.
5 Conclusion
MART introduces an auxiliary memory module that enables recurrence in transformers for video paragraph captioning. Across two standard datasets, it improves overall performance while producing more coherent and less redundant paragraphs without degrading relevance.
- MART augments a transformer with an auxiliary memory module to enable recurrence for video paragraph captioning.
- MART shows better overall performance than baseline methods on ActivityNet Captions and YouCookII.
- MART generates more coherent, less redundant paragraphs without degradation in relevance.
A.1 Additional Qualitative Examples
Qualitative examples compare paragraph captions from multiple models, highlighting coherence, repetition, and content errors. Models with sentence-level recurrence tend to be more coherent, while MART is usually less repetitive than Transformer-XL, though all models miss fine-grained details in some cases.
- Models with sentence-level recurrence, including MART and Transformer-XL, tend to generate more coherent captions.
- MART captions are usually less repetitive than Transformer-XL captions in the qualitative comparisons.
- All three models sometimes fail to recognize fine-grained video details, such as gender and fine-grained objects or actions.
- Figure 5 marks pronoun errors in red/bold, repetitive patterns in blue/italic, and content errors with underlining.