Source-linked AI summary
OmniNet: A unified architecture for multi-modal multi-task learning
Subhojeet Pramanik, Priyanka Agrawal, Aman Hussain
TL;DR
Existing neural architectures usually specialize in one task or modality, so the paper introduces OmniNet as a unified multi-modal, multi-task extension of Transformer. Its spatio-temporal cache supports shared representations across tasks, while joint training compresses the four-task model and enables limited zero-shot transfer to unseen video tasks.
Problem
Existing architectures generally specialize in one task and input domain, motivating shared representations across multiple modalities and concurrent tasks.
Method
OmniNet combines modality-specific peripheral networks with a Central Neural Processor and spatio-temporal caches for asynchronous multi-task decoding.
Results
Training POS tagging, VQA, image captioning, and HMDB together reduces total parameters by three times, while the shared model produces zero-shot predictions for video captioning and video question answering.
Takeaways & Limitations
Shared spatio-temporal representations can be reused across related multi-modal tasks and unseen tasks within the demonstrated scope.
Abstract
from arXiv · showhide
Transformer is a popularly used neural network architecture, especially for language understanding. We introduce an extended and unified architecture that can be used for tasks involving a variety of modalities like image, text, videos, etc. We propose a spatio-temporal cache mechanism that enables learning spatial dimension of the input in addition to the hidden states corresponding to the temporal input sequence. The proposed architecture further enables a single model to support tasks with multiple input modalities as well as asynchronous multi-task learning, thus we refer to it as OmniNet. For example, a single instance of OmniNet can concurrently learn to perform the tasks of part-of-speech tagging, image captioning, visual question answering and video activity recognition. We demonstrate that training these four tasks together results in about three times compressed model while retaining the performance in comparison to training them individually. We also show that using this neural network pre-trained on some modalities assists in learning unseen tasks such as video captioning and video question answering. This illustrates the generalization capacity of the self-attention mechanism on the spatio-temporal cache present in OmniNet.
1 Introduction
Existing sequence and deep-learning architectures generally specialize in one task and input domain, motivating a unified model for shared representations across modalities and concurrent tasks.
- Transformers and generic deep-learning architectures perform strongly but typically handle one task and one input domain at a time.
- Self-attention captures global temporal dependencies, raising whether Transformer-like models can share representations across domains for concurrent tasks.
- Prior multi-task architectures support selected modality or task combinations, but not a unified architecture for arbitrary multi-modal inputs.
- OmniNet extends Transformer with spatio-temporal caching, enabling one model to process multiple modalities and asynchronously decode tasks such as POS tagging, captioning, VQA, and video recognition.Peripheral-specific encoders store shared spatial and temporal representations for subsequent task decoding.
2 Related Work
Multi-task and multi-modal learning span many applications, but existing systems commonly assume a fixed task set or specific modality combinations.
- Multi-task learning has been applied across NLP, speech, vision, and combinations such as captioning, translation, and parsing.
- Most existing architectures assume the tasks are known in advance, limiting generic support for varied multi-modal task sets.
3 Proposed Model
OmniNet combines domain-specific peripheral networks with a Central Neural Processor that stores spatio-temporal representations and decodes task-specific outputs through spatial and temporal attention.
- 3 Proposed Model: OmniNet connects modality-specific peripheral networks to a shared Central Neural Processor, allowing image, text, and video inputs to become standardized spatio-temporal tensors.Each tensor has temporal dimension t, spatial dimension s, and model dimension dmodel.
- 3.2 Central Neural Processor (CNP): The CNP repeatedly encodes each modality with domain-aware representations, stores spatial and temporal information in separate caches, and links their dimensions.The encode routine can process multiple inputs from the same or different domains.
- 3.1 Peripheral networks: The vision peripheral converts images and videos into projected feature tensors, using t = 1 for images and t = F for videos.The experiments use ResNet-152 features after removing its final fully connected and average-pooling layers.
- 3.2 Central Neural Processor (CNP): The encoding routine appends spatial vectors when s > 1 and averages spatial features before a TemporalEncoder appends temporal embeddings to the temporal cache.The algorithm records each input’s t-to-s relationship in the link array.
- 3.2 Central Neural Processor (CNP): During decoding, task-specific output embeddings and classifiers produce predictions while attention jointly accesses the temporal and spatial caches.The decoder uses a two-step attention mechanism that incorporates the link array connecting temporal and spatial structures.
4 Tasks and Setup
OmniNet is evaluated on four tasks spanning temporal, spatial, multimodal, and spatio-temporal input configurations. The setup uses task-specific peripherals and a shared encoding/decoding architecture across these modalities.
- Task coverage: Four tasks cover distinct spatio-temporal configurations: POS tagging, image captioning, visual question answering, and video activity recognition.The tasks vary in temporal dimension t and spatial dimension s, covering temporal-only, spatial-only, multimodal, and combined spatio-temporal inputs.
- Task coverage: POS tagging uses text sequences with t > 1 and s = 1, leaving the spatial cache empty and using Transformer-like temporal decoding.Sentences are encoded through the language peripheral into x ∈ R^t×1×dmodel.
- Task coverage: Image captioning uses a resized image with t = 1 and s > 1, producing 49 spatial embeddings through a pretrained ResNet-152 vision peripheral.The image is resized to 224 × 224 and represented as x ∈ R^1×49×dmodel.
- Task coverage: Visual question answering combines image and question inputs through vision and language peripherals, then predicts answers in one decoding step.The image produces x1 ∈ R^1×49×dmodel, while the question produces x2 ∈ R^t×1×dmodel.
- Task coverage: Video activity recognition represents the t > 1 and s > 1 setting using 16 resized frames from HMDB, which contains over 5000 clips across 51 action classes.Evaluation uses HMDB train-test split 1, with each frame resized to 224 × 224.
5 Results and Discussion
Experiments assess OmniNet on individual tasks, joint multi-task training, zero-shot transfer, and architectural ablations. Joint training substantially reduces parameters while preserving similar performance, and shared representations support meaningful unseen-task predictions with scope limitations.
- Evaluation scope: The evaluation covers diverse individual tasks, multi-task models, unseen-task reuse, and architectural ablations.Table 1 compares independent and multi-task settings, while Figure 3 and Table 2 address zero-shot transfer and component effects.
- Architecture: OmniNet can extend to new domains without modifying the CNP when a peripheral converts domain inputs into spatio-temporal tensors.This peripheral-based extensibility supports applicability across multiple multimodal tasks.
- Multi-task learning: Over 450 × 10^6 parameters become over 149 × 10^6 under multi-task sharing, a three-times reduction while achieving similar performance.MULT-3 performs similarly or sometimes better than independent models, whereas MULT-4 is slightly reduced, attributed to HMDB’s skewed dataset size.
- Zero-shot transfer: Without video captioning or video question-answering training, MULT-4 transfers information from image captioning, VQA, and HMDB to generate meaningful video predictions.The model can describe concepts learned from related tasks, but for unseen concepts it may describe the environment or substitute alternate known concepts.
- Architectural ablations: Ablations show that independently trained captioning collapses when spatial cache is removed, while VQA retains some performance by using temporal question information.POS tagging is unaffected because its input contains only temporal modality; multi-task captioning retains some performance through spatial averages stored in temporal cache.
6 Conclusions and Future Work
The paper presents OmniNet as a unified architecture for varying input modalities and multi-task learning over spatio-temporal data. It concludes with extensions toward new peripherals, richer data structures, and improved task scheduling.
- Conclusions: OmniNet learns tasks with multiple varying modalities and supports multi-task learning across spatio-temporal inputs.Sharing one model reduces total parameters and supports reusable representations for unseen tasks.
- Future work: Future work includes peripherals for domains such as speech, modeling graphs and relational data, and scheduling mechanisms for multi-task optimization.These directions aim to extend the architecture beyond spatial and temporal data dimensions.