Source-linked AI summary
Modulating early visual processing by language
Harm de Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, Aaron Courville
TL;DR
The paper addresses whether language should influence visual processing earlier than the high-level fusion used in many language-vision models. It conditions a pretrained ResNet’s batch-normalization parameters on a language embedding, producing MODERN. MODERN improves strong baselines on two VQA tasks, with ablations indicating that modulating the entire visual signal, including early stages, gives the best performance.
Problem
Language-vision models commonly process modalities independently, despite evidence that language can influence visual processing from its beginning.
Method
The paper uses Conditional Batch Normalization to condition a pretrained ResNet’s normalization parameters on linguistic input, forming MODERN.
Results
MODERN significantly improves baseline models on VQAv1 and GuessWhat?!, and performance decreases when modulation is restricted to later ResNet stages.
Takeaways & Limitations
Modulating the entire visual processing pipeline, including early stages, is important for obtaining maximum performance gains.
Takeaways & Limitations
Because MODERN backpropagates through all convolutional layers, its experiments use ResNet-50 to limit GPU-memory requirements.
Abstract
from arXiv · showhide
It is commonly assumed that language refers to high-level visual concepts while leaving low-level visual processing unaffected. This view dominates the current literature in computational models for language-vision tasks, where visual and linguistic input are mostly processed independently before being fused into a single representation. In this paper, we deviate from this classic pipeline and propose to modulate the \emph{entire visual processing} by linguistic input. Specifically, we condition the batch normalization parameters of a pretrained residual network (ResNet) on a language embedding. This approach, which we call MOdulated RESnet (\MRN), significantly improves strong baselines on two visual question answering tasks. Our ablation study shows that modulating from the early stages of the visual processing is beneficial.
1 Introduction
Visual question answering tests how language and vision should be integrated. Rather than processing modalities independently and fusing only high-level representations, the paper proposes language-conditioned modulation throughout visual processing.
- Motivation: VQA asks how visual and textual representations should be fused in computational models.The paper focuses on visual question answering as a testbed for integrating language and vision.
- Motivation: Current systems typically extract high-level image features and language embeddings independently before fusing them downstream.Fusion commonly uses concatenation, element-wise products, bilinear pooling, or related mechanisms.
- Motivation: Independent processing is motivated by overfitting concerns and the view that language refers primarily to high-level visual concepts.Using a pretrained convolutional network as a fixed feature extractor reduces the risk of overfitting.
- Motivation: Neuroscience findings indicate that words can establish visual priors and modulate low-level P1 signals before image recognition.These findings motivate investigating fusion at early stages rather than only after high-level visual features are extracted.
- Approach: The paper proposes Conditional Batch Normalization to condition a pretrained ResNet’s normalization parameters on linguistic input, creating MODERN.The stated contributions include modulating visual processing from early stages and improving state-of-the-art models on two VQA tasks.
2 Background
The background reviews ResNet and batch normalization, then describes how questions are converted into language embeddings for visual question answering. It also contrasts these components with the paper’s computation-graph framing.
- 2.1 Residual networks: ResNet refines representations by adding residuals, enabling very deep convolutional networks to train without suffering as much from vanishing gradients.Residual blocks use a residual function composed of three convolutional layers.
- 2.1 Residual networks: A ResNet begins with one convolutional layer followed by four computation stages, with transitions that halve spatial dimensions and double feature-map count.ResNet-50, ResNet-101, and ResNet-152 differ in their numbers of residual blocks per stage.
- 2.2 Batch Normalization: Batch normalization normalizes convolutional feature maps using batch statistics and trainable affine parameters γc and βc.For convolutional layers, mean and variance are computed across batch and spatial dimensions; inference uses population statistics.
- 2.3 Language embeddings: Questions are token sequences whose learned or pretrained word embeddings are processed recurrently, with the final LSTM hidden state used as the question embedding eq.The embedding sequence is repeatedly transformed by a recurrent transition function.
- 2.2 Batch Normalization: Figure 2 contrasts the computation graphs of standard batch normalization and conditional batch normalization.The figure is intended to be viewed in color.
3 Modulated Residual Networks
The proposed architecture predicts language-conditioned changes to batch-normalization parameters in a pretrained ResNet. These changes modulate feature maps throughout the residual network while keeping the ResNet parameters frozen.
- 3 Modulated Residual Networks: Conditional batch normalization predicts changes to frozen β and γ parameters from the question embedding, preserving the pretrained network at initialization.The predicted deltas are designed to have zero mean and small variance.
- 3 Modulated Residual Networks: One-hidden-layer MLPs predict per-feature-map parameter deltas from the question embedding, producing vectors whose size equals the layer’s channel count.The predicted values are added to β and γ before batch normalization.
- 3 Modulated Residual Networks: Updated β and γ parameters are used in batch normalization, with CBN applied to all three convolutional layers in each residual block.All ResNet parameters, including the original γ and β, remain frozen during training.
- 3 Modulated Residual Networks: CBN lets language embeddings manipulate entire feature maps by scaling, negating, or shutting them off, while its batch-normalization parameters comprise less than 1% of a pretrained ResNet.The paper presents this as more scalable than conditionally predicting weight matrices or their low-rank approximations.
4 Experimental setting
The experiments evaluate MODERN and related baselines on VQA and GuessWhat?!, including architectural variants, stage-wise modulation, and representation analyses. MODERN conditions pretrained ResNet processing on language and improves performance across the reported tasks.
- Experimental scope: The study evaluates conditional batch normalization on two VQA tasks and also applies MODERN to the MLB architecture.Experiments cover VQA and the Oracle task of GuessWhat?!, with an additional MLB-based VQA architecture.
- Architectures: MODERN modulates batch normalization parameters throughout all residual blocks, conditioning the pretrained ResNet on a language embedding.The architecture is evaluated alongside an MLB variant with multiple image glimpses.
- VQA architecture: The VQA baseline extracts ResNet-50 feature maps and uses question-conditioned spatial attention before fusing visual and linguistic embeddings.The fused representation is computed by element-wise product after projecting both modalities and applying tanh nonlinearities.
- VQA results: MODERN improves VQA accuracy from the 58.05% baseline to 60.82% when evaluated with 224x224 input images.Fine-tuning only the final stage reaches 56.91%, while fine-tuning batch-normalization parameters reaches 58.98%.
- GuessWhat?! results: MODERN improves the GuessWhat?! Oracle test error to 25.06% with object-crop input and to 19.52% when spatial and category information are included.The corresponding improvements are reported relative to fine-tuned or raw-feature baselines in the two settings.
- Ablations and representations: Ablations show that performance decreases when conditional batch normalization is restricted to later ResNet stages, although the two last stages are recommended when resources are limited.The representation analysis finds that MODERN groups features by three VQA answer types, unlike raw and fine-tuned features.
5 Related work
MODERN follows work on conditional normalization but differs by using a differentiable question-conditioned mapping while keeping pretrained convolutional filters fixed.
- MODERN builds on VQA systems that independently process ImageNet-based visual features and questions before fusing them.
- Unlike prior top-level fusion methods, MODERN conditions visual normalization on the question embedding.
- MODERN uses a differentiable mapping from the question embedding rather than CIN’s non-differentiable lookup table.
- MODERN predicts normalization changes for a pretrained convolutional network while keeping its convolutional filters fixed.
- The paper presents CBN-based conditional modulation of vision processing as novel relative to the cited prior work.
6 Conclusion
The conclusion presents CBN as a mechanism for language-conditioned modulation across a visual network and reports gains from modulating the full visual signal. It also identifies broader applications beyond text and images.
- CBN modulates all layers of a visual processing network and underlies the proposed MODERN architecture when applied to a pretrained ResNet.
- MODERN can be incorporated into existing architectures, and the experiments report significant improvements over baseline models.
- The experiments find that modulating the entire visual signal is important for obtaining maximum performance gains.
- CBN may extend to networks processing sound or video and to tasks such as reinforcement learning, NLP, or adversarial training.
A Hyperparameters
This appendix lists the hyperparameters used for the GuessWhat?! Oracle and VQA experiments and notes that the experimental code will be released.
- The appendix lists the architectures’ hyperparameters used in the experiments.
- The authors state that TensorFlow code will be released to replicate the experiments.
- Table 4 is the hyperparameter table for the GuessWhat?! Oracle task.
- Table 5 is the hyperparameter table for VQA.
B T-SNE visualization
The appendix visualizes MODERN feature-map projections from four ResNet stages and compares projections colored by question type or answer type.
- The MODERN projections cover feature maps from Stage4, Stage3, Stage2, and Stage1.
- Figure 6 colors t-SNE points by question type for image/question pairs from the VQA dataset.
- Figure 7 shows t-SNE projections of finetuned ResNet feature maps before attention, colored by VQA answer type.
- Figure 7 reports that no answer-type clusters can be observed in either case.