Source-linked AI summary
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings
Michael Wray, Diane Larlus, Gabriela Csurka, Dima Damen
TL;DR
Fine-grained action retrieval must match videos and free-form captions beyond predefined action labels. The paper disentangles captions into part-of-speech-specific multimodal embeddings and unifies them for retrieval. JPoSE outperforms the evaluated baselines and supports zero-shot fine-grained retrieval, while PoS disentangling also benefits general video retrieval.
Problem
Fine-grained actions require cross-modal retrieval between videos and free-form captions rather than classification over predefined labels.
Method
JPoSE learns separate multimodal embeddings for caption parts of speech and jointly feeds their representations into a final PoS-agnostic embedding.
Results
JPoSE outperforms MMEN variants and baselines for both video-to-text and text-to-video retrieval, improving zero-shot fine-grained action retrieval and benefiting MSR-VTT video retrieval.
Takeaways & Limitations
Disentangling captions provides specialised views for acts and objects that can be combined for fine-grained and general video retrieval.
Abstract
from arXiv · showhide
We address the problem of cross-modal fine-grained action retrieval between text and video. Cross-modal retrieval is commonly achieved through learning a shared embedding space, that can indifferently embed modalities. In this paper, we propose to enrich the embedding by disentangling parts-of-speech (PoS) in the accompanying captions. We build a separate multi-modal embedding space for each PoS tag. The outputs of multiple PoS embeddings are then used as input to an integrated multi-modal space, where we perform action retrieval. All embeddings are trained jointly through a combination of PoS-aware and PoS-agnostic losses. Our proposal enables learning specialised embedding spaces that offer multiple views of the same embedded entities. We report the first retrieval results on fine-grained actions for the large-scale EPIC dataset, in a generalised zero-shot setting. Results show the advantage of our approach for both video-to-text and text-to-video action retrieval. We also demonstrate the benefit of disentangling the PoS for the generic task of cross-modal video retrieval on the MSR-VTT dataset.
1. Introduction
The paper targets fine-grained action retrieval between videos and free-form captions, where actions such as “cracking eggs” require more detailed descriptions than predefined labels. It disentangles captions by part of speech, learns specialised embeddings, and combines them for retrieval and zero-shot generalisation.
- Motivation: Fine-grained action retrieval searches short action segments using either video sequences or textual captions.Free-form descriptions provide subtler action characterisation than predefined labels.
- Method: Captions are split by part of speech so verbs, nouns, and other relevant components receive separate embedding spaces.Verb embeddings focus on shared acts, whereas noun embeddings focus on shared objects.
- Method: Multiple PoS embeddings provide different views of the data and contribute collaboratively to a final action-retrieval embedding.PoS parsing injects additional information automatically, without requiring extra annotation.
- Motivation: PoS-specific embeddings can generalise across varied actions involving the same verb, including actions not seen during training.This generalisation addresses the broader action variety encountered in retrieval.
- Evaluation: The approach reports the first fine-grained action-retrieval results on EPIC and evaluates PoS-aware embeddings for general video retrieval on MSR-VTT.EPIC experiments use released free-form narrations as supervision.
2. Related Work
Prior work established multimodal embedding and retrieval methods for videos, text, and actions, but fine-grained action recognition largely used closed vocabularies. This paper instead formulates fine-grained action search as cross-modal retrieval using open-vocabulary captions.
- Multimodal retrieval: Cross-modal retrieval commonly learns joint embeddings for images and text, videos and audio, or videos and text.Learning-to-rank methods extend across two or more modalities.
- Text representation: Text representations progressed from TF-IDF and bag-of-words models to RNN, GRU, and LSTM-based features and embedding networks.These approaches aggregate variable-length captions into representations usable for retrieval.
- Action embedding and retrieval: Action-retrieval research includes semantic, activity, object, Word2Vec, mixture-of-experts, and weakly paired video-text embedding approaches.Prior systems address zero-shot recognition, multimodal retrieval, or large-scale weak supervision.
- Research gap: Fine-grained action datasets generally use closed-vocabulary class labels, while this work uses open-vocabulary captions and treats the task as retrieval rather than recognition.The authors state that prior cross-modal retrieval on fine-grained actions had not been attempted.
3. Method
The method builds shared video–text representations while disentangling caption parts of speech into specialised embedding spaces. Their outputs are pooled and unified by a final PoS-agnostic embedding trained jointly with PoS-aware and PoS-agnostic objectives.
- 3.1. Multi-Modal Embedding Network (MMEN): MMEN maps videos and captions into a common space using cross-modal and within-modal triplet losses.Cross-modal losses bring relevant cross-modal items closer than non-relevant items, while within-modal losses preserve neighbourhood structure.
- 3.2. Disentangled Part of Speech Embeddings: Captions are split into PoS tags, and a separate PoS-MMEN embeds each tag’s words with the video representation.The experiments focus on verbs and nouns for fine-grained action recognition, while other PoS types are explored for generic video retrieval.
- 3.2. Disentangled Part of Speech Embeddings: PoS-specific relevance makes each embedding specialise in its tag, so shared verbs can be relevant despite different nouns, whereas shared nouns can remain irrelevant in a verb space.For example, ‘cut carrots’ is relevant to ‘cut tomato’ in a verb-focused space but not in a noun-focused space.
- 3.3. PoS-Aware Unified Action Embedding: The outputs of multiple PoS-MMENs are combined by concatenation, max pooling, or average pooling to form PoS-aware representations.Max and average pooling require the individual embedding spaces to have the same dimensionality.
- 3.3. PoS-Aware Unified Action Embedding: JPoSE applies a final PoS-agnostic MMEN to the combined representations for action retrieval, using relevance defined at the overall action level.The final model learns the PoS-specific and final embedding networks jointly through weighted PoS-aware and PoS-agnostic losses.
4. Experiments
Experiments evaluate JPoSE on fine-grained action retrieval with EPIC and general video retrieval with MSR-VTT, including cross-modal, within-modal, ablation, zero-shot, and qualitative analyses. JPoSE outperforms the compared MMEN variants and baselines across reported EPIC retrieval settings, improves zero-shot performance, and consistently surpasses full-caption embeddings on MSR-VTT.
- Experimental details: EPIC’s open-vocabulary narrations are parsed into PoS tags, with nouns and verbs emphasized because the dataset contains mainly these categories.A large English spaCy parser identifies PoS tags; dataset statistics guide the PoS choices.
- Experimental scope: EPIC experiments evaluate cross-modal and within-modal action retrieval, ablations, zero-shot settings, and qualitative retrieval results.The study uses open-vocabulary narrations and reports both video-to-text and text-to-video tasks.
- Cross-modal retrieval: JPoSE outperforms all MMEN variants and standard baselines for video-to-text and text-to-video retrieval on both EPIC test sets.The comparison includes caption, single-PoS, and concatenated PoS representations, while MMEN ([Verb, Noun]) improves over other MMEN variants but remains below full JPoSE.
- Within-modal retrieval: JPoSE also outperforms all MMEN variants on EPIC within-modal text-to-text and video-to-video retrieval.The reported within-modal comparison covers both tt and vv retrieval.
- Ablation study: The ablation confirms that JPoSE’s proposed encoding and joint training configuration is the best tested option, while open-vocabulary training benefits performance.The study compares encoding functions and independent versus joint training with identity or learned functions.
- Zero-shot and general video retrieval: JPoSE clearly improves generalized zero-shot settings and consistently outperforms full-caption single embeddings on MSR-VTT cross-modal retrieval.The zero-shot gains are attributed to multiple PoS views specialized to acts and objects; on MSR-VTT, nouns are more informative than verbs alone.
5. Conclusion
The paper proposes disentangled PoS embeddings for fine-grained action retrieval and reports generalisation to zero-shot cases on EPIC, alongside applicability to video retrieval on MSR-VTT.
- The model learns distinct PoS embeddings and combines them into a space designed for fine-grained action retrieval.
- On EPIC, the method uses open-vocabulary labels and demonstrates generalisation to zero-shot cases.
- Disentangling captions by PoS is also applicable to general video retrieval on MSR-VTT.
A. Individual Part-of-Speech Retrieval (Sec. 3.3)
Individual verb and noun retrieval experiments show that JPoSE matches or improves competing PoS-specific embeddings across cross-modal and within-modal searches, with complementary PoS information supporting the individual spaces.
- Verb retrieval: JPoSE produces better or equivalent results than a verb-only PoS-MMEN for fine-grained verb retrieval in cross-modal and within-modal searches.
- Noun retrieval: Fine-grained noun retrieval shows similar mAP increases for cross-modal and within-modal searches.
- Noun retrieval: Complementary information from other PoS embeddings and the PoS-aware action embedding helps define individual embedding spaces.
B. Closed vs Open Vocabulary Embedding
Training the embedding with EPIC’s full open vocabulary provides a sizeable benefit when testing with open-vocabulary actions, compared with training on the closed vocabulary.
- Using the full open vocabulary during training yields a sizeable benefit over closed-vocabulary training when testing with open-vocabulary actions.
C. Text embedding Using RNN
Replacing the caption perceptron with a GRU improves some mAP results over the caption-based MMEN, but using the entire caption still performs worse than MMEN or JPoSE with verb and noun information.
- The RNN uses capped, zero-padded captions of up to 15 words and was not tested for individual PoS-MMENs.
- The GRU-based caption embedding improves mAP for tv, vv, and tt compared with MMEN (caption).
- Despite those gains, full-caption RNN retrieval performs worse than MMEN ([Verb,Noun]) or JPoSE (Verb,Noun) for cross-modal and within-modal retrieval.
D. Additional MSR-VTT Experiments (Sec. 4.2)
The MSR-VTT experiments compare multiple PoS combinations, including mean pooling instead of NetVLAD, using averaged retrieval results across ten runs. Noun disentanglement achieves the best performance among the tested options.
- Experimental setup: The expanded experiment tests varied PoS combinations and replaces MMEN’s NetVLAD layer with mean pooling in an AVG variant.Each row reports an average of 10 runs.
- PoS combinations: Nouns disentangled from the rest of the caption achieve the best results among the tested JPoSE options.The alternatives are (Verb, Noun), (Caption\Verb, Verb), and (Capiton\Noun, Noun).
- Evaluation: MSR-VTT video-caption retrieval is evaluated with recall@k, where higher is better, and median Rank, where lower is better.The table reports results for video-caption retrieval and includes a text-to-video comparison from [22].