Source-linked AI summary
Task-Driven Modular Networks for Zero-Shot Compositional Learning
Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, Marc'Aurelio Ranzato
TL;DR
The paper tackles sample-inefficient compositional zero-shot recognition, where models must classify novel object-attribute combinations despite scarce examples. It proposes a task-driven modular network that gates semantic-space modules using the task and jointly models image-object-attribute compatibility. Under generalized evaluation on two benchmarks, the modular network outperforms existing approaches, while naive baselines often match or exceed the prior state of the art.
Problem
Long-tailed visual concepts and contextual image-object-attribute interactions make reliable recognition difficult without examples for every category.
Method
TMN uses task-conditioned gating to reconfigure small semantic-space modules and learns compatibility from the joint image, object, and attribute triplet.
Results
TMN outperforms all previous approaches under generalized evaluation on two widely used benchmarks, while baseline approaches often outperform the current state of the art.
Takeaways & Limitations
Generalized evaluation gives a more precise view of compositional generalization, and modular task reweighting supports transfer, sample efficiency, computational efficiency, and interpretability.
Takeaways & Limitations
The framework’s stated extension to novel objects and attributes assumes structured object and attribute representations, while the proposed dataset splits differ from prior work.
Abstract
from arXiv · showhide
One of the hallmarks of human intelligence is the ability to compose learned knowledge into novel concepts which can be recognized without a single training example. In contrast, current state-of-the-art methods require hundreds of training examples for each possible category to build reliable and accurate classifiers. To alleviate this striking difference in efficiency, we propose a task-driven modular architecture for compositional reasoning and sample efficient learning. Our architecture consists of a set of neural network modules, which are small fully connected layers operating in semantic concept space. These modules are configured through a gating function conditioned on the task to produce features representing the compatibility between the input image and the concept under consideration. This enables us to express tasks as a combination of sub-tasks and to generalize to unseen categories by reweighting a set of small modules. Furthermore, the network can be trained efficiently as it is fully differentiable and its modules operate on small sub-spaces. We focus our study on the problem of compositional zero-shot classification of object-attribute categories. We show in our experiments that current evaluation metrics are flawed as they only consider unseen object-attribute pairs. When extending the evaluation to the generalized setting which accounts also for pairs seen during training, we discover that naive baseline methods perform similarly or better than current approaches. However, our modular network is able to outperform all existing approaches on two widely-used benchmark datasets.
1. Introduction
The paper addresses compositional visual recognition under long-tailed categories, where models must capture interactions among images, objects, and attributes. It proposes Task-driven Modular Networks (TMN), which reconfigure semantic-space modules for new object-attribute tasks and outperform prior approaches under generalized evaluation.
- Motivation: Long-tailed visual concepts make collecting sufficient training examples for every possible category impractical.The paper contrasts this data requirement with humans’ ability to recognize novel concepts without observing instances.
- Motivation: Compositional reasoning can transfer attributes across objects, enabling recognition of unseen combinations such as “wrinkled envelope.”The motivating example combines knowledge of white envelopes, white paper, and wrinkled paper.
- Challenge: Correct inference requires modeling contextual interactions among the image, object, and attribute because attribute effects depend on both object category and input image.The same attribute can alter different objects, or different images of the same object, in distinct ways.
- Approach: TMN jointly predicts the object and attribute while learning high-level transferable sub-tasks and rich image-object-attribute interactions.These two ingredients motivate the proposed architecture for compositional learning.
- Approach: Task-driven modularity reweights semantic-space modules on the fly for new object-attribute pairs, improving transfer, sample efficiency, computational efficiency, and interpretability.Transfer requires learning module gates rather than relearning module parameters, while modules operate in smaller sub-spaces.
- Results: Under generalized evaluation on two widely used benchmarks, TMN outperforms previous approaches, while baseline methods often outperform the prior state of the art.Generalized evaluation measures both seen and unseen object-attribute pairs.
2. Related Work
Prior compositional zero-shot methods embed object-attribute pairs in image feature space, whereas this paper learns input-dependent joint compatibility and introduces task-driven modularity for CZSL.
- Compositional zero-shot learning: Compositional zero-shot learning extends zero-shot learning by predicting object-attribute pairs, including combinations not observed during training.Generalized CZSL evaluates both seen and unseen pairs, unlike the conventional unseen-only setting.
- Prior approaches: Most prior CZSL methods embed object-attribute pairs in image feature space, while this work learns compatibility representations dependent on the full input triplet.The triplet-dependent representation is intended to capture dependencies between the image and the object-attribute pair.
- Modular networks: Using modular networks for CZSL is presented as a major novelty relative to prior applications in other learning settings.The paper interprets these networks as task-conditioned generalizations of hierarchical mixture-of-experts architectures.
- Related task-driven approach: A concurrent task-driven approach gates factorized convolutional layers component-wise, which the paper contrasts with modules intended to represent high-level sub-tasks.The paper characterizes component-wise gating as degenerate modularity because individual feature dimensions may not model high-level sub-tasks.
- Meta-learning interpretation: The gating network can also be viewed as meta-learning because it predicts task-specific gates for the recognition network on the fly.This interpretation concerns the gating mechanism rather than the full modular architecture.
3. Approach
TMN formulates compositional zero-shot classification as joint compatibility scoring while configuring modular computation from object-attribute concepts. It trains on seen concepts and evaluates generalization to seen and unseen pairs.
- Images are compositions of objects and attributes, and TMN extrapolates classifiers to novel combinations without training examples for those classes.
- Generalized evaluation includes both seen and unseen concepts in validation and test sets, unlike standard evaluation focused only on unseen concepts.
- TMN scores the joint compatibility of an image, object, and attribute, assigning high scores to correct triplets and low scores to incorrect ones.
- A pretrained ResNet maps images into semantic space, where small differentiable modules are composed through concept-conditioned gating.
- Training jointly optimizes the gating and feature-extraction networks with cross-entropy, while negative sampling reduces the cost of score normalization.
4. Experiments
Experiments evaluate TMN on two compositional zero-shot benchmarks using generalized metrics, ablations, efficiency comparisons, and qualitative analyses. TMN consistently outperforms tested baselines, improves the seen–unseen trade-off, trains efficiently, and learns visually organized task-dependent modular structure.
- Quantitative Analysis: TMN consistently outperforms the tested baselines on both compositional zero-shot benchmark datasets under generalized evaluation.Generalized evaluation tests performance on both seen and unseen object-attribute pairs.
- Quantitative Analysis: Generalized evaluation exposes a trade-off between seen-pair and unseen-pair accuracy that is summarized more meaningfully by harmonic mean and AUC.Changing the calibration bias improves unseen accuracy while reducing seen accuracy, making comparisons at different operating points inconclusive.
- Quantitative Analysis: TMN reaches an AUC of 3.5 on MIT-States within 4 epochs, whereas AttributesAsOperators and LabelEmbed+ require 400–800 epochs to reach their best AUC values.The comparison uses the same minibatch size; TMN’s modular extractor also uses substantially fewer parameters than a comparable fully connected network.
- Ablation Study: Task-driven gates and joint image–object–attribute representations are important: task-agnostic gating deteriorates performance, while moving the pair embedding to the output causes a drastic drop.The ablation compares shared task-agnostic gates and task-agnostic output features against TMN’s task-conditioned architecture.
- Ablation Study: AUC is fairly robust to the number of layers and modules per layer except when the modular network is very shallow.This robustness is reported for the MIT-States validation set.
- Qualitative Analysis: Gating patterns are organized mainly by visual similarity, with edge overlap depending strongly on similarity between object-attribute pairs.t-SNE reveals smaller clusters for the same object across attributes, while contextual changes can separate visually dissimilar or appearance-changing pairs.
5. Conclusion
The paper presents TMN for zero-shot compositional learning in heavy-tailed visual concept spaces and advocates generalized evaluation covering both unseen and seen concepts.
- TMN combines joint image–object–attribute processing with task-dependent modular gating.The architecture is designed to account for contextuality while reweighting modules for different input pairs.
- TMN outperforms the baseline approaches considered while remaining efficient and interpretable.
- The generalized evaluation protocol measures accuracy on both unseen and seen concepts.
A. Hyperparameter tuning
The main-paper results use the best validation-selected hyperparameters, with the same cross-validation procedure applied to every method.
- Hyperparameters were selected using the best validation performance.
- The same cross-validation procedure was used for all methods, including TMN.
- The appendix reports both the grid-search ranges and the selected hyperparameter values.
A.1. Task Driven Modular Networks
The TMN hyperparameter search varies sampled negatives, ConceptDrop, module output dimensions, and network depth across candidate settings.
- For MIT States, sampled negatives were 200, 400, or 600, with 600 selected; UT-Zappos used all negatives.
- Module output dimensions were 8 or 16, with 16 selected.
- The number of layers ranged from 1 to 5; 3 were selected for MIT States and 2 for UT-Zappos.
A.2. LabelEmbed+
The LabelEmbed+ hyperparameter search includes multiple ConceptDrop rates and selects a 5% training-concept drop fraction.
- The selected ConceptDrop rate was 5%.
- The selected setting drops a fraction of training concepts during the search.
A.3. RedWine
The RedWine section reports the ConceptDrop setting used in the experiment: no training concepts were dropped.
- 0% of training concepts were dropped in ConceptDrop.
A.4. Attributes as Operators
The Attributes as Operators experiments used a 5% ConceptDrop rate and followed the original paper’s hyperparameter choices and implementation.
- 5% of training concepts were dropped in ConceptDrop.
- Learning rate, batch size, and regularization weights were taken from the original paper using its implementation.
2. Additional Topology Visualizations
This section provides additional examples of task-driven topologies learned in TMN.
- Figure 8 shows additional examples of task-driven topologies learned in TMN.