Source-linked AI summary
AutoFlow: Learning a Better Training Set for Optical Flow
Deqing Sun, Daniel Vlasic, Charles Herrmann, Varun Jampani, Michael Krainin, Huiwen Chang, Ramin Zabih, William T. Freeman, Ce Liu
TL;DR
Optical-flow pre-training relies on synthetic data because reliable ground truth is difficult or impossible to obtain, but effective dataset design remains unclear. AutoFlow learns rendering hyperparameters by optimizing trained-model performance on a target dataset. Its 2D-rendered data matches or exceeds established synthetic-data baselines in the reported pre-training comparisons.
Problem
Optical-flow ground truth is difficult or impossible to obtain reliably, making synthetic data important, while the principles of effective optical-flow datasets remain unclear.
Source: §1 · p. 1Method
AutoFlow jointly optimizes rendering and model training, learning hyperparameters for a layered 2D rendering pipeline according to model performance on a target dataset.
Source: §1 · p. 2Results
Takeaways & Limitations
AutoFlow offers another option for optical-flow pre-training using 2D rendering rather than datasets generated with 3D models.
Source: §6 · p. 8The comparisons with FlyingChairs and FlyingThings3D are not strictly fair because implementations and hyperparameters differ.
Source: §5.2 · p. 8
Abstract
from arXiv · showhide
Synthetic datasets play a critical role in pre-training CNN models for optical flow, but they are painstaking to generate and hard to adapt to new applications. To automate the process, we present AutoFlow, a simple and effective method to render training data for optical flow that optimizes the performance of a model on a target dataset. AutoFlow takes a layered approach to render synthetic data, where the motion, shape, and appearance of each layer are controlled by learnable hyperparameters. Experimental results show that AutoFlow achieves state-of-the-art accuracy in pre-training both PWC-Net and RAFT. Our code and data are available at https://autoflow-google.github.io .
1. Introduction
Synthetic data supports optical-flow pre-training because reliable ground truth is difficult or impossible to label, yet the principles of an effective dataset remain unclear. AutoFlow learns rendering hyperparameters by optimizing a trained model’s performance on a target dataset, and the paper reports gains over FlyingChairs-based pre-training.
1. Introduction
- Optical-flow labels are difficult or impossible to obtain reliably, so synthetic datasets such as FlyingChairs and FlyingThings3D are used before fine-tuning on limited target data such as Sintel and KITTI.More realistic datasets have not proven more effective for pre-training, and the principles behind effective optical-flow datasets remain unclear.Source: §1 · pp. 1–2
- AutoFlow makes rendering-data generation a joint optimization of rendering and model training, evaluating learnable hyperparameters by performance on a target dataset.Its layered 2D pipeline needs neither manual labels nor 3D models; layer motion and shape are generated according to hyperparameters.Source: §1 · p. 2
- Both RAFT and PWC-Net improve on Sintel and KITTI over the same models trained on FlyingChairs; four augmented AutoFlow examples also outperform 22,872 augmented FlyingChairs examples for RAFT on Sintel.final.With enough AutoFlow examples, the performance gap between PWC-Net and RAFT becomes small.Source: §1 · p. 2
- AutoFlow’s motion statistics differ from Sintel’s and underrepresent small motions, although the authors suggest tiny motions may contribute little to overall error.The motion-statistics comparison is illustrated in Fig. 8.Source: §1 · p. 2
- The paper presents learning to render optical-flow training data as a new approach and includes an analysis of which data-generation features matter.Source: §1 · p. 2
2. Related Work
Prior work explored synthetic optical-flow datasets, rendering methods, and network architectures, but dataset design was largely heuristic. AutoFlow instead learns rendering parameters against target-task performance, and its learned settings also apply to PWC-Net.
2. Related Work
- FlyingChairs and FlyingThings3D became standard pre-training datasets, while other synthetic datasets had not proven more effective for pre-training.Source: §2 · p. 2
- Heuristically designed synthetic datasets and 2D rendering techniques informed AutoFlow, whose distinction is learning parameters through joint rendering and training for target-dataset performance.Meta-Sim can optimize task performance but relies on valid scene structures from a grammar; other cited methods focus on single-image generation [37].Source: §2 · pp. 2–3
- Optical-flow architectures advanced from FlowNet and FlowNet2 through PWC-Net to RAFT, but models generally retained the same FlyingChairs-then-FlyingThings3D training procedure.Source: §2 · p. 3
- The paper reports that learning to render can achieve accuracy similar to or better than FlyingChairs and FlyingThings3D for pre-training, with RAFT-learned rendering parameters also applying to PWC-Net.Source: §2 · p. 3
- Changing datasets and incorporating recent training practices significantly improves PWC-Net and narrows its performance gap from RAFT.Source: §2 · p. 3
3. Generating training data
AutoFlow renders optical-flow training pairs with layered images, masks, and motion, then applies visual effects and augmentation. Its rendering choices are controlled by learnable hyperparameters.
3. Generating training data
- Layers are built from sampled images, masks, and flow fields; flow warps each layer into the second frame before back-to-front alpha compositing and visual effects.Source: §3 · p. 3
3.1. Object Masks
- Foreground masks can be random polygons with holes, smoothed edges, and blurred boundaries, or manual OpenImages segmentations placed at random locations and sizes.Source: §3.1 · pp. 3–4, Figure 3 · p. 4
3.2. Motion Model
- Layer motion combines rigid transformations, perspective distortion, and bilinear grid warps; foregrounds use all modalities, while backgrounds receive only moderate perspective distortion.The grid warp interpolates flow vectors from grid vertices and permits more complex forward flow with an analytic warping solution.Source: §3.2 · p. 4
3.3. Visual Effects
- Motion blur and fog simulate video effects by changing image data without changing ground-truth flow.Blur strength is tied to average absolute flow within a mask; the generated fog remains stationary between frames.Source: §3.3 · p. 4
3.4. Data Augmentation
- Augmentation randomly selects rotation, scale, squeeze, translation, and additive noise, with the number and strength of transformations treated as learnable hyperparameters.Source: §3.4 · p. 5
3.5. Hyperparameters
- Learned ranges govern the uniformly sampled mask shape, size, and position, motion complexity and magnitude, and visual effects used during training-data generation.Source: §3.5 · p. 5
4. Learning to Render Training Data
AutoFlow searches for rendering hyperparameters that yield training data whose trained model performs well on a target optical-flow dataset. It combines population-based training with CMA-ES to search hyperparameter subgroups.
4. Learning to Render Training Data
- For a target dataset, rendering hyperparameters define a training set, and the objective is to optimize the trained model's target-dataset metric.Source: §4 · p. 5
4.1. Problem Formulation
- The objective selects rendering hyperparameters λ to minimize metric Ω on model θ, which is trained on the corresponding rendered dataset; by default, the metric is average end-point error.Source: §4.1 · p. 5
4.2. Hyperparameter Search Algorithm
- A hybrid population-based training and CMA-ES algorithm searches hyperparameter subgroups, updating a multivariate Gaussian sampling distribution using metric-ranked evaluations.The algorithm runs N iterations with M trainings in parallel; time complexity grows linearly with search iterations and training steps per search.Source: §4.2 · p. 5
5. Experimental Results
AutoFlow pre-training is comparable to or more accurate than established synthetic-data pre-training, and fine-tuning AutoFlow-pre-trained models improves results on challenging benchmarks. Ablations examine which rendering choices and training settings affect performance.
5.1. AutoFlow Versus the State of the Art
- AutoFlow-pre-trained models are comparable to or more accurate than models pre-trained on FlyingChairs or FlyingChairs→FlyingThings3D.RAFT trained on AutoFlow also tends to outperform FlyingChairs-trained RAFT at medium-to-large motion ranges.Source: §5.1 · p. 5, Table 2 · p. 6
- One-cycle learning rates and gradient clipping improve PWC-Net pre-training, while clipping also stabilizes training.Two of eight runs diverged without gradient clipping.Source: Table 3 · p. 6, §5.1 · p. 6
- RAFT-A is more accurate than TF-RAFT on the challenging Sintel.final and KITTI benchmarks.Table 4 summarizes benchmark errors for these comparisons.Source: Table 4 · p. 6, §5.1 · p. 6
5.2. Ablation Study
- Removing motion blur or fog reduces accuracy on Sintel.final and KITTI, but neither effect significantly changes Sintel.clean performance.Gaussian and box filters give similar results for approximating motion blur.Source: §5.2 · p. 6
- Smooth-edged masks outperform sharp-edged masks; three or four foreground objects tend to work best, and removing grid warping degrades Sintel and KITTI accuracy.Davis appearance images at 720p are the default; 1080p degrades performance, while 540p is similar. More than four foreground objects bring no further gain.Source: §5.2 · p. 7
- AutoFlow's augmented motion distribution emphasizes middle-to-large motions and has little small motion, unlike Sintel and FlyingChairs.The authors suggest tiny motion contributes little to the overall learning metric.Source: §5.2 · p. 7
- Both models converge from one augmented AutoFlow example, and four examples yield lower Sintel.final errors for RAFT than 22,872 augmented FlyingChairs examples.Removing spatial augmentation sharply worsens the four-example result; augmented four-example motion statistics are similar to full AutoFlow's.Source: §5.2 · pp. 7–8
6. Conclusions
AutoFlow learns optical-flow pre-training data with a 2D renderer. It can match or exceed established 3D-generated datasets, including with only four augmented examples for RAFT on Sintel.final.
6. Conclusions
- AutoFlow uses 2D rendering yet achieves results comparable to or better than FlyingChairs and FlyingThings3D, which were generated using 3D models.The conclusion also reports that AutoFlow significantly improves PWC-Net, sometimes bringing it on par with RAFT.Source: §6 · p. 8
- Four augmented AutoFlow examples yield more accurate RAFT results on Sintel.final than 22,872 augmented FlyingChairs examples.The comparison concerns the final pass of Sintel.Source: §6 · p. 8
- The authors present AutoFlow as another option for optical-flow pre-training and hope it will enable further progress and innovation.Their analysis notes that augmented AutoFlow motion statistics are close between the four-example and full datasets.Source: §6 · p. 8, Figure 9 · p. 8
A. Rendering Hyperparameters
AutoFlow learns ranges of rendering hyperparameters that control layered masks, motion, visual effects, and augmentation. These ranges determine the random settings used to generate training data.
A. Rendering Hyperparameters
- The rendering hyperparameter ranges govern sampled data-generation settings, including mask shape, size, and position, motion complexity and magnitude, and visual effects.The paper defines U as a uniform random number from −1 to 1.Source: A. Rendering Hyperparameters · p. 11
- Mask hyperparameters control polygon complexity, including side counts, holes, and subdivisions.The settings also include object size and center ranges, and the probability and strength of mask blur.Source: A. Rendering Hyperparameters · p. 11
- Motion hyperparameters include rotation, translation, grid-warp strength, and grid size.The rendering ranges also include probabilities and strengths for motion blur and fog, plus average fog density.Source: A. Rendering Hyperparameters · p. 11
- RandAugment hyperparameters set the number of augmentations per iteration and their strength level.These parameters specify the augmentation settings used during training.Source: A. Rendering Hyperparameters · p. 11
B. More Samples
The appendix presents AutoFlow sample image pairs and their visualized flow fields for scenes with one through five foreground objects.
B. More Samples
- The one-object examples show first and second images alongside visualized optical flow.The first and fourth rows are first images, the second and fifth rows are second images, and the third and sixth rows show flow.Source: Figure B.1 · p. 12
- The two-object examples use the same first-image, second-image, and flow-field row arrangement.The appendix caption identifies scenes with two foreground objects.Source: Figure B.2 · p. 13
- The three-object examples show paired images and visualized flow in the same row arrangement.The appendix caption identifies scenes with three foreground objects.Source: Figure B.3 · p. 14
- The four-object examples show paired images and visualized flow in the same row arrangement.The appendix caption identifies scenes with four foreground objects.Source: Figure B.4 · p. 15
- The five-object examples show paired images and visualized flow in the same row arrangement.The appendix caption identifies scenes with five foreground objects.Source: B. More Samples · p. 11, Figure B.5 · p. 16