Source-linked AI summary
Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, Alan Yuille
TL;DR
The paper addresses how to learn novel visual concepts from a few sentence-described images without retraining an entire captioning model. It adapts a pretrained captioner using linguistic and visual information, weight sharing, and overfitting controls. Experiments show comparable performance to full retraining, with better performance when only a few novel-concept images are available.
Problem
Existing methods learn new categories from few examples, but the paper targets learning novel visual concepts from sentences without repeatedly retraining on all data.
Method
The method adapts a pretrained m-RNN captioning model, expands its word dictionary, uses transposed weight sharing, and controls overfitting while learning from few examples.
Results
The method performs comparably to retraining from scratch on all data when novel-concept images are plentiful, and better when only a few are available.
Takeaways & Limitations
The framework learns and uses novel concepts while preserving previously learned concepts, supported by three released datasets for future research.
Takeaways & Limitations
NC-3 is challenging because its concepts resemble known categories, are absent from ImageNet's 1,000 categories, and use a different sentence style from the base data.
Abstract
from arXiv · showhide
In this paper, we address the task of learning novel visual concepts, and their interactions with other concepts, from a few images with sentence descriptions. Using linguistic context and visual features, our method is able to efficiently hypothesize the semantic meaning of new words and add them to its word dictionary so that they can be used to describe images which contain these novel concepts. Our method has an image captioning module based on m-RNN with several improvements. In particular, we propose a transposed weight sharing scheme, which not only improves performance on image captioning, but also makes the model more suitable for the novel concept learning task. We propose methods to prevent overfitting the new concepts. In addition, three novel concept datasets are constructed for this new task. In the experiments, we show that our method effectively learns novel visual concepts from a few examples without disturbing the previously learned concepts. The project page is http://www.stat.ucla.edu/~junhua.mao/projects/child_learning.html
1. Introduction
The paper introduces NVCS, where models learn novel visual concepts from a few image descriptions and use them in captions. It proposes adapting an existing captioning model without extensive retraining, supported by new datasets and experiments.
- The task matters because adding a few novel-concept images should not require retraining an entire model with all previous data.
- NVCS asks models to learn novel visual concepts from a few sentence-described images and then use those concepts to describe new images.
- The proposed framework enlarges a pretrained model's word dictionary using a few examples without retraining the model from scratch on all data.
- The method adapts m-RNN for captioning and introduces transposed weight sharing, halving learned parameters while enabling richer word and multimodal representations without overfitting.
- Three datasets cover novel man-made objects, animals, and activities, and few-example training performs as well as retraining the entire model on all examples.
2. Related Work
Related work covers deep neural networks for language and vision and few-shot learning of new object categories. The paper distinguishes NVCS because prior methods learn words or attributes rather than sentence-grounded visual concepts.
- The project provides an updated dataset containing eight additional novel concepts beyond its original three concepts.
- RNNs and LSTMs achieve strong results in language tasks, while CNNs improve object classification and detection in computer vision.
- One-shot learning methods learn new object categories from few examples, but focus on words or attributes rather than sentences.
- The paper positions NVCS as distinct from prior methods because it learns novel visual concepts from sentence descriptions.
3. The Image Captioning Model
The image-captioning model combines word and image inputs to predict the next word, using LSTM-based language modeling and transposed weight sharing to reduce parameters and support novel concept learning.
- Captioning architecture: The model takes the current word index and image as inputs, then predicts the next word index through shared sub-model weights.The language component uses word embeddings and an LSTM, while the model combines language, vision, and multimodal components.
- Captioning architecture: The architecture replaces the original recurrent layer with an LSTM to address gradient explosion and vanishing problems.This modification also improves performance on the original image-captioning task.
- Captioning architecture: Because each input word is represented by a one-hot vector, only its corresponding column of UD is needed during forward and backward propagation.This avoids calculating the full matrix multiplication for every word.
- Transposed weight sharing: UD encodes a one-hot word vector into a dense semantic vector, while UM decodes multimodal activations into a pseudo one-hot word vector.The two matrices therefore perform complementary encoding and decoding operations in different semantic spaces.
- Novel-concept adaptation: During novel-concept training, the model updates the sub-matrix connected to new-word nodes in the one-hot and SoftMax layers.The figure identifies this trainable sub-matrix as UDn.
- Transposed weight sharing: Transposed weight sharing decomposes UM and shares part of it with UD in transposed form, reducing parameters without increasing the word-embedding dimension.The strategy enables a larger word-embedding layer while preserving parameter efficiency.
4. The Novel Concept Learning (NVCS) Task
The NVCS task adapts a pretrained captioning model to learn new visual concepts from few images without retraining the full model. The approach fixes or normalizes parameters that could disturb prior concepts or bias new-word probabilities, while combining linguistic and visual cues.
- Task motivation: Retraining the full model is costly or impossible, whereas fine-tuning on only new data can overfit and reduce accuracy on previously learned concepts.These constraints motivate incremental learning from a small set of novel-concept images.
- Preventing interference: The method fixes weights associated with original words while learning separate parameters for new words during incremental training.UD is partitioned into UDo for original words and UDn for new words.
- Preventing bias: Training only on new examples can inflate new-word baseline probabilities because the new data provide unreliable estimates of their output biases.The bias vector is separated into original-word and new-word components.
- Preventing bias: Baseline Probability Fixation sets every new-word bias element to the average value of the new-word bias estimates after centralizing intermediate-layer activations.This addresses the additional baseline shift caused by nonzero mean intermediate activations.
- Language and vision: Sentence descriptions provide weak labels that let the language component infer new-word properties and relations to known words, such as linking cat with dog.The example shows how linguistic context transfers semantic and syntactic information from familiar vocabulary.
- Language and vision: The vision component is pretrained on ImageNet and supplies visual attributes, while combining language and vision enables learning new concepts from few examples.The pretraining uses 1.2 million images and 1,000 categories.
5. Datasets
The paper constructs three novel-concept datasets spanning object, activity, and uncommon concepts, then separates them into training, validation, and test sets to assess learning and overfitting.
- Three datasets cover five novel visual concepts, including cat, motorcycle, quidditch, t-rex, and samisen.
- NC-3 contains quidditch, t-rex, and samisen, with 100 images and five sentence annotations per concept.Its annotations are designed to vary sentence descriptions across images.
- The NC-3 annotations average 13.5 words per sentence versus 10.7 words for MS COCO.The dataset was constructed with richer and more diversified sentence descriptions than the two COCO-derived datasets.
- The NC datasets are randomly divided into training, validation, and testing sets, with separate Novel Concept and Base test sets.The Base test set evaluates whether adaptation harms performance on previously learned concepts.
6. Experiments
Experiments evaluate caption quality and novel-word accuracy, test architectural strategies and baselines, and examine performance with full and few-shot training data. Deep-NVCS approaches match strong retraining results more efficiently while preserving base-concept performance, though NC-3 remains challenging.
- Evaluation Metrics: BLEU, METEOR, and novel-word f scores evaluate sentence quality and explicit accuracy on newly learned concepts.The f score balances precision and recall for novel words, with high recall and low precision indicating overfitting and the reverse indicating underfitting.
- Effectiveness of TWS and BPF: 2.5% increase in f comes from TWS, while BPF adds a 2.4% increase on NewObj-Cat.These comparisons are Deep-NVCS-BPF-TWS against Deep-NVCS-BPF-noTWS and Deep-NVCS-UnfixedBias, respectively.
- Using all training samples: Deep-NVCS performs much better than Model-base on novel-concept tests while reaching comparable performance on base tests.It also outperforms Model-word2vec and approaches Model-retrain while requiring less than 2% of the time.
- Using a few training samples: Deep-NVCS performs comparably with full novel-concept training after training on only 10 to 50 images.With about five training images, it already shows a nontrivial improvement over the base model and runs faster than Model-retrain.
- Qualitative Results: Nearest-neighbor word embeddings capture semantic information from both language and vision for the learned novel concepts.The paper also presents generated sentence descriptions from the base and Deep-NVCS models as qualitative evidence.
7. Conclusion
The paper introduces NVCS, learning novel visual concepts from sentence descriptions of a few images. Its method performs comparably to full retraining with many novel-concept images and better with only a few, supported by three released datasets.
- 7. Conclusion: NVCS requires learning novel concepts from sentence descriptions of a few images and using them in image descriptions.The paper frames this as a distinct task and releases three datasets to support future research.
- 7. Conclusion: The proposed method performs comparably with retraining from scratch when many novel-concept images are available and better when only a few are available.The conclusion reports validation on three novel-concept datasets.