FiLM: Visual Reasoning with a General Conditioning Layer

Ethan Perez; Florian Strub; Harm de Vries; Vincent Dumoulin; Aaron Courville

Source-linked AI summary

FiLM: Visual Reasoning with a General Conditioning Layer

Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, Aaron Courville

arXiv:1709.07871v2Sep 22, 2017cs.CVcs.AIcs.CLstat.ML

TL;DR

Visual reasoning requires answering image questions through multi-step, high-level processes, motivating whether general-purpose neural architectures can learn it. FiLM uses conditioned feature-wise affine transformations and achieves strong visual reasoning across tasks, with robust models and generalization from limited data.

Problem
Whether general-purpose neural architectures can learn multi-step visual reasoning remains an open question important for broadly applicable artificial agents.
Method
FiLM conditions neural-network intermediate features through feature-wise affine transformations, enabling language inputs to modulate visual computation.
Results
FiLM achieves state-of-the-art performance across varied visual reasoning tasks, remains robust to ablations, and generalizes from little data to substantially different data.
Takeaways & Limitations
FiLM provides a general-purpose conditioning approach for visual reasoning that supports diverse multi-step tasks and applications beyond normalization-heavy settings.
Takeaways & Limitations
On zero-shot generalization, FiLM’s accuracy differs substantially across conditions, suggesting it memorizes some attribute combinations from training data.

Abstract

from arXiv · show

We introduce a general-purpose conditioning method for neural networks called FiLM: Feature-wise Linear Modulation. FiLM layers influence neural network computation via a simple, feature-wise affine transformation based on conditioning information. We show that FiLM layers are highly effective for visual reasoning - answering image-related questions which require a multi-step, high-level process - a task which has proven difficult for standard deep learning methods that do not explicitly model reasoning. Specifically, we show on visual reasoning tasks that FiLM layers 1) halve state-of-the-art error for the CLEVR benchmark, 2) modulate features in a coherent manner, 3) are robust to ablations and architectural modifications, and 4) generalize well to challenging, new data from few examples or even zero-shot.

1 Introduction

The section introduces FiLM as a general-purpose conditioning method for visual reasoning and presents evidence that it achieves strong, coherent, robust, and generalizing performance across visual reasoning tasks.

FiLM method: FiLM applies a feature-wise affine transformation to intermediate neural-network features, conditioned on an arbitrary input.For visual reasoning, an RNN processes the question and influences CNN computation over the image.
Key findings: FiLM models achieve state-of-the-art performance across a variety of visual reasoning tasks, often by significant margins.The claimed results concern visual reasoning tasks evaluated in the paper.
Key findings: FiLM learns underlying structure coherently, selectively manipulates conditioned features, and enables the CNN to localize question-referenced objects.These findings characterize both feature modulation and spatial behavior during visual reasoning.
Key findings: FiLM remains robust under many ablations, with several variants outperforming prior state-of-the-art methods.The paper also reports that normalization is not closely linked to the success of conditioned affine transformations, relaxing application requirements.
Key findings: FiLM models learn from little data and generalize to more complex or substantially different data, including through a novel zero-shot generalization method.The zero-shot method further improves and validates FiLM’s generalization capabilities.

2 Method

FiLM conditions neural-network computation by applying input-dependent, feature-wise affine transformations to intermediate activations. In the visual-reasoning model, a question-processing GRU generates modulation parameters for a visual residual network, using only this affine conditioning to influence visual processing.

FiLM conditioning: FiLM learns input-dependent γ and β parameters that apply feature-wise affine transformations to intermediate neural-network features.The functions producing γ_i,c and β_i,c may be arbitrary functions, including neural networks.
FiLM conditioning: A single FiLM generator can output shared (γ, β) vectors for a target FiLM-ed network, enabling independent feature-map modulation.FiLM layers can scale, negate, shut off, or selectively threshold feature maps, providing fine-grained activation control.
FiLM conditioning: FiLM uses two parameters per modulated feature map, and its computational cost does not scale with image resolution.This makes the conditioning method scalable and computationally efficient.
Visual reasoning model: The visual pipeline extracts 128 14 × 14 feature maps from a resized 224 × 224 image and processes them with FiLM-ed residual blocks.Image features come from either a CNN trained from scratch or a fixed pre-trained extractor with a learned 3 × 3 convolutional layer.
Visual reasoning model: The model relies solely on feature-wise affine conditioning to inject question information into visual processing, rather than fusing image and language into a single embedding.The paper contrasts this approach with fusion methods based on element-wise products, concatenation, attention, and more advanced mechanisms.

3 Related Work

FiLM generalizes conditional normalization by applying conditioning-dependent feature-wise affine transformations without requiring placement directly after normalization. It also connects to conditional-bias, feature-gating, hypernetwork, and visual-reasoning methods.

Conditional normalization: FiLM can be viewed as a generalization of Conditional Normalization, which replaces normalization affine parameters with a learned function of conditioning information.Conditional normalization has proven effective across multiple domains.
Conditional normalization: Prior conditional-normalization work did not examine affine-transform placement; FiLM finds that placement directly after normalization is unnecessary and unifies these methods.This connects FiLM and normalization without requiring strict adjacency between normalization and modulation.
Other conditioning methods: Concatenating conditioning feature maps or vectors with layer inputs is less parameter-efficient and amounts to a feature-wise conditional bias.This applies to convolutional and fully connected layers.
Other conditioning methods: Input-dependent gating methods amount to feature-wise conditional scaling restricted to values between 0 and 1, whereas FiLM uses a separate conditioning input.Examples include LSTMs, Convolutional Sequence to Sequence, and Squeeze and Excitation Networks.
Visual reasoning: In visual reasoning, Program Generator plus Execution Engine models use compositional neural modules, while Relation Networks impose comparison-based pairwise reasoning over image features.These approaches explicitly encode structural priors for reasoning.

4 Experiments · 4.1 CLEVR Task · 4.2 What Do FiLM Layers Learn?

Experiments show that FiLM achieves state-of-the-art visual reasoning performance on CLEVR while producing interpretable, question-conditioned feature modulation. Analyses indicate that FiLM localizes relevant objects, supports reasoning across the pipeline, and organizes modulation by reasoning function across network depth.

4 Experiments: The experiments evaluate FiLM on CLEVR, CLEVR-Humans, and compositional few-shot and zero-shot generalization settings.CLEVR is a synthetic dataset of 700K image-question-answer-program tuples with multi-step, compositional questions.
Baselines: The CLEVR comparison includes question-only, CNN-plus-LSTM, stacked-attention, module-network, program-execution, and relation-network baselines.These baselines range from predicting by question category to explicitly modeling pairwise relations or question-dependent subfunctions.
Results: FiLM achieves a new overall state-of-the-art on CLEVR, outperforming humans and prior methods with explicit reasoning, program supervision, or data augmentation.
Results: 4.5% to 2.3%: FiLM roughly halves state-of-the-art error among methods without extra supervision.FiLM performs equally well with raw pixels instead of pre-trained image features.
Activation Visualizations: FiLM localizes answer-related or question-related objects, and its feature modulation indirectly produces spatial modulation.The model can isolate the answer-referenced object or retain multiple question-referenced objects before final classification.
FiLM Parameter Histograms: γ values range from -15 to 19 and β values from -9 to 16, with γ sharply peaking at 0.This indicates that FiLM can upregulate, downregulate, or shut off feature maps based on conditioning information.
FiLM Parameters t-SNE Plot: FiLM parameter vectors cluster by low-level reasoning functions in early layers and high-level reasoning functions in late layers.This pattern is observed in a deeper 6-ResBlock model with six FiLM layers and similar validation accuracy to the 4-ResBlock model.

4.3 Ablations

Ablations show that FiLM is robust across conditioning choices, depth, and architectural changes, while both scaling and biasing contribute to visual reasoning. The results also indicate that scaling, repeated feature modulation, residual placement, and spatially persistent features are important to model performance.

Conditioning parameters: A 1.5% accuracy drop with γ = 1 and a .5% drop with β = 0 show that either scaling or biasing alone supports reasoning, but γ matters more.Conditioning both together performs best.
Conditioning parameters: Replacing β with its training-set mean drops accuracy by 1.0%, whereas replacing γ drops it by 65.4%, indicating that FiLM relies primarily on γ at test time.These are test-time ablations of a model trained with both γ and β.
Scaling functions: Restricting γ to (0, 1), (−1, 1), or (0, ∞) hurts performance, showing that large-magnitude scaling and the ability to negate or zero feature maps contribute to FiLM’s success.The sigmoid and tanh restrictions perform roughly as poorly as removing γ conditioning entirely.
FiLM depth: Models with fewer FiLM layers, even one, remain close to the best model, showing that a single modulation can support diverse visual reasoning.This result examines the contribution of repetitive conditioning.
Spatial reasoning: A model using only 1 × 1 convolutions without coordinate features still achieves 95.3% accuracy, indicating that FiLM can reason about space without transferring information across spatial positions.Global max-pooling near the network’s end prevents spatial information transfer in this model.
Architecture and depth: Removing residual connections causes one of the larger accuracy drops, while FiLM inside residual connections performs better; varying depth is robust except with only 1 ResBlock.These findings support reasoning through features that remain important across multiple levels of the network.

4.4 CLEVR-Humans: Human-Posed Questions

On CLEVR-Humans, FiLM achieves state-of-the-art generalization to realistic, diverse, free-form questions both before and after fine-tuning. Fine-tuning only its linguistic FiLM-generating pipeline substantially improves accuracy, and FiLM surpasses PG+EE by 9.3% through flexible feature-map modulation.

Dataset: CLEVR-Humans contains human-posed questions on CLEVR images designed to be difficult for a smart robot, using more diverse vocabulary and complex concepts.The dataset has 18K training, 7K validation, and 7K test samples.
Method: Fine-tuning updates only FiLM’s linguistic pipeline while keeping the visual pipeline fixed to mitigate overfitting on the small CLEVR-Humans training set.The method starts from the best CLEVR-trained FiLM model.
Results: FiLM achieves state-of-the-art generalization on CLEVR-Humans both before and after fine-tuning, with a considerably improved final accuracy after fine-tuning.Before fine-tuning, FiLM outperforms prior methods by a smaller margin.
Results: FiLM surpasses PG+EE by 9.3% after fine-tuning on CLEVR-Humans.PG+EE struggles when questions cannot be well approximated with its module inventory.
Analysis: FiLM’s feature-map modulation provides flexible, fine-grained operations for reasoning about novel concepts beyond PG+EE’s fixed module inventory.Figure 8 examples include obstruction, superlatives, and reflections, while hypothetical scenarios remain difficult.

4.5 CLEVR Compositional Generalization Test

On CLEVR-CoGenT, FiLM learns stronger compositional generalization than other visual reasoning models but retains some memorized attribute biases. A FiLM-parameter analogy method improves zero-shot performance on unseen combinations without specialized training, though it applies to only part of the questions and has limitations.

Compositional generalization: FiLM surpasses other visual reasoning models, including compositionality-supervised PG+EE, at learning general concepts on CLEVR-CoGenT.The model is trained on Condition A and evaluated on Conditions A and B before and after fine-tuning.
Compositional generalization: 1/3 as much fine-tuning data lets FiLM achieve prior state-of-the-art accuracy, but fine-tuning still causes catastrophic forgetting.
Compositional generalization: FiLM’s much higher Condition A accuracy than Condition B indicates memorization of attribute combinations, including a bias that cubes are not cyan.
Zero-shot generalization: The zero-shot method derives FiLM parameters by adding and subtracting parameters for related questions, enabling correct counting of unseen attribute combinations.For cyan cubes, it combines questions about cyan spheres, brown cubes, and brown spheres.
Zero-shot generalization: 3.2% overall accuracy gain results from zero-shot generalization on B; among the 1/3 of B questions it supports, accuracy rises from 71.5% to 80.7%.The method uses a parser to automatically generate the required question combinations.
Zero-shot generalization: The zero-shot improvements require no specialized training and exploit concept disentanglement, while convex FiLM-parameter combinations often interpolate predicted answers between endpoint questions.The implemented method has limitations, motivating future work on direct optimization and learned parsing.

5 Conclusion

FiLM layers enable strong visual reasoning by selectively and meaningfully manipulating intermediate neural-network features. The approach lets language modulate visual processing for diverse, multi-step image reasoning and remains resilient to several architectural and test-time changes.

FiLM layers support strong visual reasoning through selective, meaningful manipulation of intermediate neural-network features.The conclusion presents FiLM as a general-purpose conditioning mechanism for visual reasoning.
A recurrent neural network can use language to modulate a convolutional neural network through FiLM for diverse, multi-step reasoning over images.
FiLM is resilient to architectural modifications and test-time ablations, according to the ablation study.

Occlusion

Partial occlusion accounts for many model errors, and higher-resolution CNNs may address these errors because FiLM’s computational cost is independent of resolution.

Occlusion: Many model errors are due to partial occlusion, which higher-resolution CNNs may fix because FiLM’s computational cost is independent of resolution.The passage presents higher-resolution processing as feasible under FiLM’s resolution-independent computational cost.

Counting

FiLM’s counting mistakes are overwhelmingly off-by-one errors, suggesting it learned underlying counting concepts such as relationships between close numbers.

Counting: 96.1% of counting mistakes are off-by-one errors, indicating learned counting concepts such as relationships between close numbers.The passage identifies close relationships between close numbers as an underlying concept behind counting.

Logical Consistency

The model can produce logically inconsistent answers even when it counts object categories correctly. The paper identifies direct minimization of logical inconsistency as a future-work direction orthogonal to FiLM.

Logical Consistency: The model sometimes gives mutually inconsistent relational answers despite correctly counting objects.In one case, it counts one gray and two cyan objects but says their counts are equal, and says gray objects are both fewer than and equal to yellow blocks.
Logical Consistency: Directly minimizing logical inconsistency could prevent these errors and is proposed as future work orthogonal to FiLM.The proposed direction targets consistency constraints rather than changing the FiLM conditioning method itself.
Logical Consistency: A representative failure combines correct counting with an incorrect comparison between counts.The failure is described as inconsistent with the model’s other answers.