Source-linked AI summary
Long-term Recurrent Convolutional Networks for Visual Recognition and Description
Jeff Donahue, Lisa Anne Hendricks, Marcus Rohrbach, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, Trevor Darrell
TL;DR
Visual recognition and description need models that handle variable-length sequential inputs and outputs rather than fixed temporal processing. This paper introduces end-to-end trainable Long-term Recurrent Convolutional Networks, which consistently improve prior methods across visual sequence tasks.
Problem
Existing visual sequence models offered limited handling of variable-length inputs and outputs, motivating architectures that model temporal structure beyond fixed processing or simple pooling.
Method
LRCN combines convolutional layers with long-range recurrent units in an end-to-end trainable architecture jointly learning visual representations and temporal dynamics.
Results
Across video recognition, image description, retrieval, and narration tasks, LRCN consistently improves previous methods and outperforms strong baselines on several evaluations.
Takeaways & Limitations
Deep recurrent sequence modeling is broadly applicable to vision tasks with sequential inputs or outputs and can integrate readily with existing visual recognition pipelines.
Abstract
from arXiv · showhide
Models based on deep convolutional networks have dominated recent image interpretation tasks; we investigate whether models which are also recurrent, or "temporally deep", are effective for tasks involving sequences, visual and otherwise. We develop a novel recurrent convolutional architecture suitable for large-scale visual learning which is end-to-end trainable, and demonstrate the value of these models on benchmark video recognition tasks, image description and retrieval problems, and video narration challenges. In contrast to current models which assume a fixed spatio-temporal receptive field or simple temporal averaging for sequential processing, recurrent convolutional models are "doubly deep"' in that they can be compositional in spatial and temporal "layers". Such models may have advantages when target concepts are complex and/or training data are limited. Learning long-term dependencies is possible when nonlinearities are incorporated into the network state updates. Long-term RNN models are appealing in that they directly can map variable-length inputs (e.g., video frames) to variable length outputs (e.g., natural language text) and can model complex temporal dynamics; yet they can be optimized with backpropagation. Our recurrent long-term models are directly connected to modern visual convnet models and can be jointly trained to simultaneously learn temporal dynamics and convolutional perceptual representations. Our results show such models have distinct advantages over state-of-the-art models for recognition or generation which are separately defined and/or optimized.
1 INTRODUCTION
The paper introduces Long-term Recurrent Convolutional Networks (LRCNs), which combine CNN visual representations with recurrent sequence models for variable-length visual inputs and outputs. It argues that recurrent models can improve visual time-series recognition and support end-to-end image-to-sentence generation.
- Proposed architecture: LRCNs combine a CNN with stacked recurrent sequence models to process possibly variable-length visual inputs and produce variable-length predictions.The architecture feeds CNN outputs into LSTMs, which generate the final sequence prediction.
- Motivation: The paper advocates temporal recurrence of latent variables instead of fixed-window processing or simple temporal pooling for video recognition and description.Recurrent neural networks provide representations that are deep in time and implicitly compositional.
- Motivation: LSTM units address the vanishing-gradient limitation of simple RNNs by enabling learning over long-range temporal intervals.Simple RNNs become increasingly difficult to optimize as errors are backpropagated through long temporal intervals.
- Contributions: Convolutional networks with recurrent units are presented as broadly applicable to visual time-series modeling, improving recognition and enabling end-to-end visual-to-language optimization.The paper argues that LSTM-style RNNs can improve tasks previously using static or flat temporal models when ample training data are available.
- Experimental settings: The experiments connect visual convolutional models to deep LSTMs for video recognition, use a visual convnet plus LSTM decoder for image-to-sentence mapping, and drive LSTM decoders from predicted discriminative labels.These settings address video activity recognition, natural-language generation, and semantic video-role-based narration.
2 BACKGROUND: RECURRENT NETWORKS
RNNs process sequences by recursively mapping inputs to hidden states and outputs, but long-term learning is hindered by vanishing and exploding gradients. LSTMs address sequential modeling with gated memory, support stacking and end-to-end fine-tuning, and handle variable-length inputs and outputs.
- Traditional RNNs: Traditional RNNs sequentially map each input x_t to a hidden state h_t and output z_t through a nonlinear recurrence.For a length-T sequence, updates proceed from h_1, z_1 through h_T, z_T with h_0 = 0.
- Traditional RNNs: RNNs can model tasks such as speech recognition and text generation, but long-term dynamics are difficult to learn because gradients may vanish or explode across recurrent layers.The difficulty is attributed in part to propagating gradients through the many layers corresponding to recurrent steps.
- LSTMs: LSTMs use input, forget, output, and modulation gates with a memory cell that combines the previous cell state and gated current-input information.The cell state is updated as c_t = f_t ⊙ c_t−1 + i_t ⊙ g_t, while h_t = o_t ⊙ tanh(c_t).
- LSTMs: LSTMs can be stacked to add depth and have achieved strong results on language tasks while remaining end-to-end fine-tunable.Examples include speech recognition and machine translation, including a deep bidirectional LSTM mapping spectrograms to text.
- LSTMs for vision: In vision, LSTMs integrate readily with existing systems and model sequential data with variable-length inputs or outputs, including text and video.These properties motivate combining recurrent models with deep convolutional networks in unified end-to-end trainable networks.
3 LONG-TERM RECURRENT CONVOLUTIONAL NETWORK (LRCN) MODEL
The LRCN model combines a deep CNN-based visual feature extractor with a recurrent sequence-learning module to model temporal dynamics for sequential visual, linguistic, and other tasks. Its flexible formulations support static or sequential inputs and outputs, with visual and recurrent parameters jointly optimized end-to-end.
- Core architecture: LRCN combines a deep hierarchical visual feature extractor with a recurrent model that learns temporal dynamics for sequential inputs or outputs.Each visual input is transformed by φV before entering the recurrent sequence-learning module.
- Core architecture: The recurrent module processes inputs sequentially, updating hidden states from h1 through hT and optionally stacking multiple LSTMs.Inference computes each time step from the preceding hidden state, beginning with h0 = 0.
- Task formulations: LRCN adapts to sequential-input/static-output, static-input/sequential-output, and sequence-to-sequence tasks with potentially different input and output lengths.It uses late fusion for sequence classification, input duplication for image captioning, and an encoder-decoder formulation for mismatched sequence lengths.
- Training: The visual and sequential parameters are jointly optimized end-to-end by maximizing ground-truth likelihood and using stochastic-gradient backpropagation.End-to-end learning lets the visual extractor select input features relevant to the sequential classification problem.
- Applications: The framework is evaluated on activity recognition, image captioning, and video description as applications of end-to-end trainable hybrid convolutional-recurrent networks.These applications instantiate the model’s broader sequential-learning task classes.
4 ACTIVITY RECOGNITION
On UCF101 activity recognition, LRCN uses recurrent convolutional processing of RGB and flow sequences and improves over single-frame CNN predictions. Its gains hold across most classes and produce accuracy comparable to or exceeding other deep models.
- Architecture and inputs: LRCN processes each frame with tied convolutional weights over time and supports both RGB and optical-flow inputs.Flow is converted into flow images by scaling and shifting x and y values to [−128, +128] and adding a third channel.
- Training: The model is trained on 16-frame clips with 227 × 227 crops and mirroring, while UCF101 videos typically contain about 100 frames.Short-clip training serves as a form of data augmentation for the longer videos.
- Overall results: 0.83% and 2.91%: LRCN improves over the single-frame baseline for RGB and flow inputs, respectively, across UCF101’s three standard test splits.LRCN achieves the best results for both input types; combining RGB and flow uses weighted averaging of network scores.
- Per-class results: For the majority of activity classes, LRCN outperforms the single-frame model, with gains such as BoxingPunchingBag and HighJump outweighing losses on classes including Knitting and Mixing.Consequently, overall accuracy is higher despite some class-level degradations.
5 IMAGE CAPTIONING
For image captioning, LRCN combines image features with an LSTM language model to generate variable-length descriptions without explicit language-model constraints. Factored sequence architectures, LSTM units, stronger or fine-tuned CNNs, and improved generation strategies yield stronger retrieval and captioning performance.
- Model: LRCN feeds image features and the previous word into a stack of LSTMs to model the time-varying natural-language output sequence.The LSTMs each have 1000 hidden units.
- Training and inference: During training, LRCN uses ground-truth previous words; during inference, it samples predicted words until generating an <EOS> token.The first input is the <BOS> token, and each subsequent prediction conditions on the generated caption prefix and image features.
- Model: The factored architecture injects visual features into a higher LSTM layer, separating lower-layer caption processing from visual-language fusion.In the factored case, lower-layer hidden states are conditionally independent of the image given the partial caption.
- Evaluation: LRCN consistently outperforms strong Flickr30k retrieval baselines, while LSTM units outperform vanilla RNNs and unfactored two-layer models perform worst.The results attribute the LSTM advantage to gating mechanisms that support long-term memory; LRCN2f and LRCN1u perform similarly.
- Retrieval: Finetuning improves R@k by 3-5% for CaffeNet and 5-8% for VGGNet, while switching from CaffeNet to VGGNet improves retrieval by around 8-12% or roughly 11-17%, depending on direction.The reported gains apply across caption-to-image and image-to-caption retrieval tasks.
- Caption generation: Fine-tuning improves CIDEr-D by roughly 0.04 points for CaffeNet and 0.07 points for VGGNet, while switching from finetuned CaffeNet to VGGNet adds 0.13 points.The strongest generation setup uses a fine-tuned VGGNet with the two-layer factored LRCN2f architecture and best-of-100 sampling.
6 VIDEO DESCRIPTION
The video-description framework treats videos as variable-length inputs and uses LSTMs to generate sentences from visual representations. On TACoS multilevel, the simpler decoder variants outperform the encoder-decoder design and achieve a BLEU-4 score of 28.8%, exceeding the prior best of 26.9%.
- Framework: The framework models video as a variable-length input stream but uses LSTMs specifically for sentence generation because available datasets limit input processing.Visual inputs are based on activity, tool, object, and location predictions from traditional recognition systems.
- Architectures: The evaluated architectures use either CRF maximum predictions or probability distributions, with LSTM encoder-decoder and decoder-only variants.The probability-based decoder replaces maximum predictions with distributions, allowing the LSTM to learn uncertainty in visual generation.
- Results: On TACoS multilevel, the LSTM outperforms an SMT-based video-description approach, while decoder variants (b) and (c) outperform encoder-decoder variant (a).The authors suggest decoder variants perform better because their inputs do not need to be memorized.
- Results: 28.8% BLEU-4 is achieved on TACoS multilevel, exceeding the best reported 26.9% from.The evaluation uses 44,762 video/sentence pairs, with about 40,000 allocated to training and validation.
- Implications: The architecture integrates cleanly with fixed- or variable-length inputs from vision systems beyond deep networks.This demonstrates that the approach is not restricted to inputs produced by deep networks.
7 RELATED WORK
Related work spans activity recognition, image captioning, and video description, contrasting fixed-frame, shallow, template-based, and temporally pooled approaches with convolutional-recurrent models. Subsequent work extends these combinations to improved recognition, captioning, grounding, and object retrieval.
- Activity Recognition: Shallow activity-recognition models capture low-level temporal changes but lose higher-level features and temporal relationships through bag-of-words or Fisher-vector encoding.Deep alternatives instead stack fixed numbers of frames or fuse RGB and optical-flow streams.
- Image Captioning: Early image-captioning systems used templates, trees, or caption fragments, whereas later multimodal models used recurrent networks for retrieval and next-word generation from image features.Related CNN-recurrent models differ in when image features enter the sequence model, and attention mechanisms target the currently generated word.
- Video Description: Video-description approaches used templates, retrieval, language models, or phrase-based SMT, while later end-to-end models pooled CNN features across frames but thereby lost temporal information.The paper presents the first application of deep models to video description and replaces phrase-based SMT with LSTMs.
- Activity Recognition: Contemporaneous CNN-LSTM activity-recognition work pretrained on Sports-1M and achieved 88.6% on UCF101, while other systems predicted multiple frame-level activities with attention.These approaches emphasize temporal information beyond a single-frame baseline.
- Extensions: Subsequent convolutional-recurrent models addressed visual grounding by attending to image regions and natural-language object retrieval by combining local proposal features with spatial configurations.The object-retrieval system returns a bounding box rather than an entire image.
8 CONCLUSION
The conclusion presents LRCN as a spatially and temporally deep model class for varied vision tasks with sequential inputs and outputs. It emphasizes consistent improvements over prior approaches and the practical importance of deep sequence modeling for time-varying visual problems.
- Conclusion: LRCN is spatially and temporally deep, flexible, and applicable to vision tasks involving sequential inputs and outputs.The model class is designed for varied vision tasks with sequential structure.
- Conclusion: Learning sequential dynamics with a deep sequence model consistently improves upon methods that learn depth only in the visual domain.The conclusion contrasts LRCN with approaches focused solely on visual parameter hierarchies.
- Conclusion: Deep sequence modeling tools like LRCN are increasingly central for vision systems addressing sequential structure and time-varying visual input or sequential outputs.They can be incorporated into existing visual recognition pipelines with little input preprocessing and no hand-design.