Source-linked AI summary
RNN Fisher Vectors for Action Recognition and Image Annotation
Guy Lev, Gil Sadeh, Benjamin Klein, Lior Wolf
TL;DR
Conventional Fisher Vectors are order-invariant, limiting their use for sequences whose meaning depends on event order. The paper derives Fisher Vectors from RNN gradients, achieving state-of-the-art or competitive results across video action recognition and image annotation, including transfer from image annotation to action recognition.
Problem
Conventional Fisher Vectors are insensitive to ordering, although sequence meaning can depend on the order of events.
Method
RNN-FV represents sequences using gradients from an RNN trained to predict the next element or its embedding.
Results
RNN-FV surpasses state-of-the-art action-recognition results on two challenging datasets and achieves state-of-the-art or competitive results for image annotation and image search.
Takeaways & Limitations
The representation preserves sequence ordering and supports transfer learning from image annotation to video action recognition.
Takeaways & Limitations
Mean pooling can blur multiset content, and one author-supported scope boundary is that the broader generality claim was not previously established.
Abstract
from arXiv · showhide
Recurrent Neural Networks (RNNs) have had considerable success in classifying and predicting sequences. We demonstrate that RNNs can be effectively used in order to encode sequences and provide effective representations. The methodology we use is based on Fisher Vectors, where the RNNs are the generative probabilistic models and the partial derivatives are computed using backpropagation. State of the art results are obtained in two central but distant tasks, which both rely on sequences: video action recognition and image annotation. We also show a surprising transfer learning result from the task of image annotation to the task of video action recognition.
1. Introduction
The paper introduces RNN-FV, a Fisher Vector representation for sequences that preserves element ordering through RNN-derived gradients. It applies this representation to video action recognition and image annotation, achieving strong benchmark performance and transfer from image annotation to action recognition.
- Method: RNN-FV represents sequences with gradients derived from an RNN rather than hidden or output-layer activations.The RNN predicts the next sequence element, and backpropagation provides the gradients used in the Fisher Vector.
- Motivation: Unlike conventional Fisher Vectors, RNN-FV is sensitive to element ordering and addresses the order invariance of probabilistic-mixture Fisher Vectors.This is especially relevant for video, where event order contributes to meaning.
- Applications: The method trains RNNs to predict next-element representations for image annotation and next-frame representations for video action recognition.Image tasks use word prediction approaches, while video uses VGG or C3D sequential features and regression.
- Transfer learning: A transfer-learning approach from image annotation to video action recognition uses CCA to project VGG frame embeddings into an image-sentence-aligned space.The projection is learned from matching images and sentences.
- Results: RNN-FV achieves state-of-the-art action recognition on HMDB51 and UCF101, state-of-the-art image annotation on Flickr8K, and competitive results on other benchmarks.These results span both video and sentence-based sequence tasks.
2. Previous Work
Prior work develops separate deep-learning pipelines for video features, image representations, sentence representations, and image-sentence matching. These approaches include convolutional, recurrent, embedding-based, and correlation or contrastive methods.
- Action Recognition: Video action-recognition research has moved from hand-crafted spatiotemporal descriptors toward deep-learned features.Early deep-learning attempts underperformed hand-crafted features, while later methods achieved state-of-the-art results.
- Image Annotation and Image Search: Image annotation and image search systems commonly combine image representation, sentence representation, and image-sentence matching.Pre-trained CNN activations are typically used for image representation.
- Sentence Representation: Sentence representations have used recursive networks, TF-IDF, word embeddings pooled with Fisher Vectors, matching CNNs, and recurrent networks.RNN-based approaches incrementally encode words into a semantic sentence vector.
- Matching Images and Sentences: Image-sentence matching methods project modalities into a common space or learn scores from matching and unmatching image-sentence pairs.CCA, Deep CCA, and contrastive-loss approaches are represented in this prior work.
3. Baseline pooling methods
The baseline methods pool vectors without preserving their order, while standard Fisher Vectors encode distributions through likelihood gradients. RNN-FV is proposed to address order insensitivity, and normalization is applied across Fisher Vector variants.
- Baseline limitation: Baseline pooling methods represent a multiset of vectors as one vector, so sequences become insensitive to ordering.RNN-FV is introduced as a pooling method intended to address this limitation.
- Mean Vector: Mean pooling computes the average of input vectors, producing a representation in the original D-dimensional space.The input is a multiset X = {x1, x2, . . . , xN} in R^D.
- Mean Vector: Mean pooling blurs multiset content by combining vectors into a location within the convex hull of the represented words.This can obscure distinctions among the words in semantic embedding space.
- Fisher Vector of a GMM: A standard Fisher Vector is the gradient of a multiset’s log-likelihood with respect to the parameters of a pretrained diagonal-covariance GMM.The usual representation retains derivatives with respect to means and standard deviations while omitting mixture-weight derivatives.
- Fisher Vector of a GMM: GMM Fisher Vector pooling is linear because multiset likelihoods multiply and their log-likelihoods add, unlike the RNN model’s history-dependent probabilities.In an RNN, the probability of each element depends on previous elements.
- Normalization: All Fisher Vector types use element-wise power normalization followed by L2 normalization.The power transform is f(z) = sign(z)|z|^α with 0 ≤ α ≤ 1.
4. RNN-Based Fisher Vector
The paper replaces order-insensitive mixture-model Fisher Vectors with RNN-based Fisher Vectors that encode sequence order through gradients of an RNN sequence model. Regression and classification variants are defined, with gradients computed by backpropagation and optionally normalized using the Fisher Information Matrix.
- RNN-FV motivation: RNN-FV replaces the order-insensitive pooling model with an RNN sequence model that predicts each next element from preceding elements.This preserves gradient-based Fisher Vector representations while making them sensitive to sequence order.
- RNN-FV variants: The method defines regression and classification RNN-FVs; regression is directly useful for video analysis and outperforms classification for image annotation.The regression variant predicts vectors, whereas the classification variant predicts symbols.
- Regression formulation: The RNN receives a sequence preceded by a start vector and is trained to predict the next vector at every time step.For input X = (x0, ..., xN−1), the target sequence is Y = (x1, ..., xN), with x0 = xstart.
- Sequence likelihood: For a new sequence, the RNN outputs a prediction at each time step, and the loss compares each prediction with the corresponding next element.The RNN is treated as a generative model assigning likelihood to the next sequence element given its history.
- Gradient representation: The unnormalized RNN-FV is the gradient of the sequence loss with respect to the RNN weights, obtained through forward and backward passes.This applies both to regression and classification formulations, where the loss equals the corresponding RNN training loss.
- Normalization: Fisher Information Matrix normalization was approximated from training-sequence gradients but produced no empirical improvement, so reported experiments omit it.The paper notes that a closed-form expression for the required FIM term is unavailable for the RNN probabilistic model.
5. Action recognition pipeline
The action-recognition pipeline combines frame or fixed-block appearance features, RNN-FV sequence encoding, and a linear SVM. Video representations use VGG frame descriptors, while the RNN predicts subsequent elements and the resulting gradients are pooled and reduced.
- Pipeline: The action-recognition pipeline consists of appearance features, RNN-FV sequence encoding, and an SVM classifier.The classifier is applied after the sequence has been converted into a fixed-dimensional RNN-FV representation.
- Appearance features: VGG extracts a 4096-dimensional representation from each video frame using ten crops, whose feature vectors are averaged into one image representation.The video is subsampled by encoding one frame in every ten, with comparable recognition performance to using all frames.
- RNN training: The video RNN is trained to predict the next element of the representation sequence from preceding elements.The action-recognition experiments use the regression formulation.
- RNN-FV construction: Gradients with respect to the last fully connected layer are averaged over time, producing a 100500-dimensional representation that PCA reduces to 1000D before power and L2 normalization.Derivatives for other layers did not improve recognition empirically.
- Classification and validation: A linear SVM with C = 1 is selected using validation accuracy and early stopping, then retrained on the combined training and validation samples for testing.The validation split is constructed to avoid placing similar segments from the same original video in both training and validation sets.
6. Image-sentence retrieval
For image-sentence retrieval, RNN-FVs represent sentences as ordered word sequences and are matched with VGG image representations in a shared CCA space. The paper compares classification and regression RNN training, with regression offering better scalability and results.
- Retrieval pipeline: Image-sentence retrieval represents images and sentences separately, maps both representations into a common space, and matches them there.The RNN-FV replaces the GMM-FV sentence representation in this pipeline.
- Sentence representation: A sentence is encoded as an ordered sequence beginning with a start token, while the RNN predicts each following word from the preceding words.The final word is treated as a period or end token.
- Training data: Sentence RNNs can be trained on benchmark sentences or on a large external corpus such as Wikipedia to obtain a more generic representation.The choice of training data is application dependent.
- RNN alternatives: The classification and regression RNN alternatives use gradients from the output layer as sentence representations because hidden-layer derivatives did not improve performance.Classification predicts dictionary words, whereas regression predicts the next word embedding.
- Classification versus regression: Classification has a dictionary-sized softmax and cannot handle words absent from training, whereas regression has output dimension determined by the embedding size and avoids these issues.The regression output is a 300-dimensional predicted word embedding.
- Cross-modal matching: VGG image vectors and sentence RNN-FVs are matched with regularized CCA, using cosine similarity in the shared space.The CCA regularization parameter is selected on the validation set.
7. Experiments
Experiments evaluate RNN-FV across action recognition and image-sentence retrieval, comparing pooling methods, feature choices, model variants, and combinations. RNN-FV outperforms alternative pooling methods, while regression-based variants and selected combinations provide strong retrieval performance.
- Action recognition: RNN-FV pooling outperformed mean and GMM-FV pooling by a sizable margin on HMDB51 and UCF101.The comparison used VGG-PCA, VGG-CCA, and C3D features, with recognition average accuracy as the metric.
- Action recognition: CCA consistently outperformed PCA with VGG frame representations across all pooling methods.The compared feature configurations were VGG coupled with PCA, VGG projected by CCA, and C3D.
- Action recognition: 66.99% accuracy was obtained on HMDB51 when RNNs trained on UCF101 encoded HMDB51 videos, including idt features.Without idt, the comparable result was 54.47, and the cross-dataset result was reported as above current state of the art.
- Image-sentence retrieval: Regression-based RNN-FV was preferred over classification-based RNN-FV because it produced better results, lower-dimensional representations, and natural handling of unseen words.Classification was practical only for Flickr8K; larger-vocabulary datasets made its large softmax layer and weight count impractical.
- Image-sentence retrieval: Wikipedia-trained RNN-FV remained competitive on image-sentence retrieval, while CCA embeddings and combining reverse-order models improved results in supported settings.Combining RNN-FV with the best prior model outperformed the state of the art on Flickr8K and was competitive on other datasets.
8. Conclusions
The paper introduces an order-sensitive RNN-derived Fisher Vector representation for sequences and reports strong results across video action recognition and image-language tasks. It also demonstrates transfer learning from image annotation to video action recognition.
- RNN-derived Fisher Vectors provide a richer sequence representation that is sensitive to element ordering rather than using the additive bag model of conventional Fisher Vectors.
- The representation surpasses state-of-the-art results for video action recognition on two challenging datasets.
- RNN-FV achieves state-of-the-art results on Flickr8K image annotation and competitive results on other image annotation and image search benchmarks.
- Transfer learning from image annotation to video action recognition produces a performance boost using CCA-projected image embeddings.
- The paper cautions that the representation's generality was not established as broadly as presented.