Source-linked AI summary
Dynamic Filter Networks
Bert De Brabandere, Xu Jia, Tinne Tuytelaars, Luc Van Gool
TL;DR
Fixed convolutional filters cannot adapt their filtering operations to individual inputs or motion patterns. Dynamic Filter Networks generate input-conditioned filters and apply them through convolutional or local filtering layers. The framework performs effectively on video and stereo prediction, including state-of-the-art moving-MNIST performance with a much smaller model, while learned filters expose unsupervised flow-like information.
Problem
Existing image-generation methods apply the same filtering operations to every input, despite differing motion patterns across video clips.
Method
Dynamic Filter Networks generate sample-specific filters from an input and apply them through dynamic convolution or position-specific local filtering.
Results
The framework is effective on video and stereo prediction, and achieves state-of-the-art moving-MNIST performance with a much smaller model.
Takeaways & Limitations
Visualized dynamic filters reveal flow-like or depth-related information learned from unlabeled data, suggesting unsupervised pretraining for optical flow and disparity estimation.
Abstract
from arXiv · showhide
In a traditional convolutional layer, the learned filters stay fixed after training. In contrast, we introduce a new framework, the Dynamic Filter Network, where filters are generated dynamically conditioned on an input. We show that this architecture is a powerful one, with increased flexibility thanks to its adaptive nature, yet without an excessive increase in the number of model parameters. A wide variety of filtering operations can be learned this way, including local spatial transformations, but also others like selective (de)blurring or adaptive feature extraction. Moreover, multiple such layers can be combined, e.g. in a recurrent architecture. We demonstrate the effectiveness of the dynamic filter network on the tasks of video and stereo prediction, and reach state-of-the-art performance on the moving MNIST dataset with a much smaller model. By visualizing the learned filters, we illustrate that the network has picked up flow information by only looking at unlabelled training data. This suggests that the network can be used to pretrain networks for various supervised tasks in an unsupervised way, like optical flow and depth estimation.
1 Introduction
Dynamic Filter Networks address the limitation of fixed filtering operations by generating parameters adapted to each input. The framework is demonstrated across video and stereo prediction, with filters also revealing unsupervised motion information.
- Existing view-synthesis and video-prediction methods apply the same filtering operations to every input image.
- Different motion patterns across video clips motivate generating future frames with parameters adapted to each sample.
- The dynamic filter module combines a filter-generating network with a differentiable dynamic filtering layer.
- The framework is evaluated on video prediction and stereo prediction, while its explicitly computed filters can be visualized without groundtruth maps.
2 Related Work
Related work explores flexible neural transformations for spatial warping, dynamic parameters, view synthesis, and information routing. Dynamic Filter Networks differ by applying generated filters directly to image filtering operations.
- Deep learning architectures: Spatial Transformer Networks condition feature-map transformations on the input through a localization network and transformation parameters.
- Deep learning architectures: Dynamic parameter layers have been used to generate fully connected-layer parameters from information in another domain, such as a question representation.
- New view synthesis: Prior view-synthesis methods include recurrent, convolutional recurrent, multi-scale adversarial, and neighboring-view architectures for predicting or reconstructing images.
- Shortcut connections: Shortcut connections in highway and residual networks regulate information flow or learn residual functions relative to identity mappings.
3 Dynamic Filter Networks
Dynamic Filter Networks generate sample-specific filters and apply them through dynamic convolution or position-specific local filtering. This supports local transformations while requiring fewer model parameters than traditional locally connected layers.
- Core framework: Model parameters are shared across samples and learned during training, whereas dynamically generated parameters are produced on the fly for each sample.
- Filter-generating network: The filter-generating network maps input IA to filters Fθ, which operate on input IB to produce output G.
- Dynamic convolution: Dynamic convolution applies one generated filter across all input positions, unlike traditional convolution whose filter weights are fixed model parameters.
- Dynamic local filtering: Dynamic local filtering applies different generated filters at different positions, making the operation both sample-specific and position-specific.
- Dynamic local filtering: Softmax-constrained local filters can select elements from local neighborhoods, providing a way to model local spatial deformations.
- Dynamic local filtering: Dynamic local filtering can include a dynamic pixel-wise bias to address photometric changes.
- Model size: The dynamic local filtering layer uses fewer model parameters than a traditional locally connected layer, producing a smaller model suitable for embedded systems.
- Relationship with other networks: Compared with Spatial Transformer Networks, Dynamic Filter Networks are more suitable for local transformations, while Spatial Transformers naturally support global transformations.
4 Experiments
The experiments apply Dynamic Filter Networks to steerable filtering, video prediction, and stereo prediction. Across these tasks, dynamically generated filters learn transformations and motion-related structure, including unsupervised flow- and depth-like information.
- Learning steerable filters: The steerable-filter experiment maps an input orientation to a 9x9 convolution filter and applies it to produce the transformed image.The trained network learns the expected filters and applies the correct transformation.
- Video prediction: Moving MNIST predicts 10 future frames from 10 input frames containing two moving digits.The model uses dynamically generated local filters to model individual digit motion, bouncing behavior, and separation after overlap.
- Video prediction: The method outperforms the state-of-the-art on Moving MNIST with a much smaller model, although predictions become blurrier over time.The reported degradation is partly attributed to imperfect digit separation after overlap and accumulated errors.
- Video prediction: On Highway Driving, the model achieves a test loss of 13.54 with 368,122 parameters, while qualitatively predicting lanes, a bridge, and an approaching car.The dataset is structured and predictable, but accumulated errors eventually place the network in an out-of-domain regime.
- Video and stereo prediction: Flow-like visualizations of dynamically generated filters reveal motion structure learned from unlabeled video data, while stereo prediction produces depth-related information from a single image.For stereo, the network predicts the right view from the left view and is reported to estimate depth information without ground-truth maps.
5 Conclusion
Dynamic Filter Networks apply sample-specific dynamically generated filters, with versions for dynamic convolution and dynamic local filtering. The framework is validated on steerable filters, video prediction, and stereo prediction, with future applications proposed for pose-adaptive classification and image deblurring.
- Dynamic Filter Networks apply dynamically generated filters to images in a sample-specific way.
- The paper presents two versions: dynamic convolution and dynamic local filtering.
- The framework is validated for steerable filters, video prediction, and stereo prediction.
- Future applications include filters adapted to object pose for fine-grained classification and image structure for deblurring.
A Video Prediction
Additional experiments examine moving MNIST and highway driving, including filter visualization and out-of-domain moving-digit inputs. A model trained with two moving digits is evaluated on inputs containing one or three moving digits.
- Additional experiments evaluate the model on moving MNIST and highway driving datasets.
- The moving MNIST experiment visualizes the dynamically generated filter for the first predicted frame in Figure 9.
- A model trained on two moving digits is tested on out-of-domain inputs containing one or three moving digits.
A.1 Moving MNIST
The moving MNIST results compare input sequences, ground-truth future frames, and model predictions for two moving digits, while also visualizing dynamically generated filters. A separate figure reports out-of-domain results for one and three moving digits.
- Figure 8 compares input sequences, ground-truth future frames, and model predictions for two moving digits.
- Figure 9 visualizes previous frames, dynamically generated filters, predicted current frames, and ground-truth frames.
- Figure 10 reports out-of-domain results for inputs with one moving digit and three moving digits.
A.2 Highway driving
Figure 11 presents results on the highway driving dataset.
- Figure 11 reports results on the highway driving dataset.
B Stereo Prediction
Stereo prediction results are presented as a video covering the whole test sequence, with each frame showing the input view, filter visualization, prediction, and ground truth.
- The stereo prediction results are shown for the whole test sequence.
- Each frame begins with the left-view image and then displays a visualization of the filters.
- The final two frame elements are the predicted right view and the ground truth.