Source-linked AI summary
FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation
Xiaoyu Shi, Zhaoyang Huang, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, Hongsheng Li
TL;DR
FlowFormer’s cost-volume encoder is trained from scratch despite the benefits of pretraining for transformer representations. FlowFormer++ introduces MCVA with block-sharing masking and task-specific reconstruction to pretrain that encoder, achieving generalization and dataset-specific performance improvements over FlowFormer and state-of-the-art benchmark performance.
Problem
FlowFormer’s cost-volume encoder is trained from scratch, while neighboring cost maps are redundant and correlated, making naive masked autoencoding prone to information leakage and pretraining–finetuning discrepancy.
Method
MCVA pretrains FlowFormer’s cost-volume encoder using block-sharing masking and a task-specific reconstruction task aligned with finetuning.
Results
FlowFormer++ obtains all-sided improvements over FlowFormer and sets new state-of-the-art performance on public benchmarks.
Takeaways & Limitations
MCVA ensures semantic integrity between pretraining and finetuning while encouraging long-range cost-volume aggregation.
Takeaways & Limitations
The image encoder must be frozen during pretraining to keep reconstruction targets static and avoid training collapse.
Abstract
from arXiv · showhide
FlowFormer introduces a transformer architecture into optical flow estimation and achieves state-of-the-art performance. The core component of FlowFormer is the transformer-based cost-volume encoder. Inspired by the recent success of masked autoencoding (MAE) pretraining in unleashing transformers' capacity of encoding visual representation, we propose Masked Cost Volume Autoencoding (MCVA) to enhance FlowFormer by pretraining the cost-volume encoder with a novel MAE scheme. Firstly, we introduce a block-sharing masking strategy to prevent masked information leakage, as the cost maps of neighboring source pixels are highly correlated. Secondly, we propose a novel pre-text reconstruction task, which encourages the cost-volume encoder to aggregate long-range information and ensures pretraining-finetuning consistency. We also show how to modify the FlowFormer architecture to accommodate masks during pretraining. Pretrained with MCVA, FlowFormer++ ranks 1st among published methods on both Sintel and KITTI-2015 benchmarks. Specifically, FlowFormer++ achieves 1.07 and 1.94 average end-point error (AEPE) on the clean and final pass of Sintel benchmark, leading to 7.76\% and 7.18\% error reductions from FlowFormer. FlowFormer++ obtains 4.52 F1-all on the KITTI-2015 test set, improving FlowFormer by 0.16.
1. Introduction
FlowFormer++ pretrains FlowFormer’s transformer-based cost-volume encoder with MCVA, addressing redundancy, masked-information leakage, and pretraining–finetuning discrepancy. Its task-specific designs support long-range aggregation and improve optical-flow estimation on public benchmarks.
- Motivation: FlowFormer’s cost-volume encoder remained trained from scratch, motivating pretraining to improve correspondence estimation.FlowFormer’s ImageNet-pretrained image encoder improved accuracy, while the cost-volume encoder lacked analogous pretraining.
- Method: MCVA pretrains the cost-volume encoder by masking cost values and reconstructing masked cost patches.The scheme adapts masked autoencoding to FlowFormer’s cost volume.
- Method: Block-sharing masking assigns common mask patterns to source pixels within large blocks, preventing reconstruction through neighboring correlated cost maps.This design encourages the encoder to aggregate information from far-away source pixels.
- Method: The reconstruction task crops 9 × 9 cost patches as queries and reconstructs 15 × 15 patches at matching locations, aligning pretraining with FlowFormer finetuning.The task explicitly encourages long-range information aggregation and reduces pretraining–finetuning discrepancy.
- Impact: MCVA encourages holistic cost-volume representations, reasoning over masked information, and task-specific decoding for harder cases including noise, large displacement, motion, and occlusion.These designs are intended to improve flow estimation.
- Results: FlowFormer++ obtains all-sided improvements over FlowFormer and sets new state-of-the-art performance on public benchmarks.The contribution summary attributes these improvements to the proposed pretraining technique.
2. Related Work
Optical-flow research progressed from optimization-based and data-driven methods toward neural architectures and transformer-based models. Related work emphasizes architectural design and masked-autoencoding pretraining as routes to stronger visual representations.
- Optical Flow: Traditional optical-flow methods formulate estimation through optimization, whereas data-driven methods learn flow directly from labeled data.Neural optical-flow methods have continued advancing since FlowNet.
- Optical Flow: Network architecture design has become a central route to improving optical-flow accuracy as neural methods progress.Prior work introduced increasingly capable network modules.
- Vision Transformers: Transformers provide long-range information encoding through self-attention but require large-scale training data to learn unbiased representations.Masked autoencoding has been applied across image, video, and point-cloud modalities without extra labeled data.
- Vision Transformers: FlowFormer applies a transformer-based cost-volume encoder to optical flow, while this work extends masked-autoencoding pretraining to that encoder.The related-work discussion positions MCVA as an adaptation of MAE to cost-volume encoding.
3. Method
FlowFormer++ pretrains FlowFormer's cost-volume encoder with masked cost-volume autoencoding, using task-specific masking, tokenization, and reconstruction to preserve pretraining–finetuning consistency. The method blocks information leakage from correlated neighboring cost maps while training the encoder to aggregate long-range cost information.
- FlowFormer and MCVA: FlowFormer++ applies MCVA to pretrain FlowFormer's transformer-based cost-volume encoder, which was previously trained from scratch.The 4D cost volume is compressed into cost memory and processed by a transformer-based alternate-group transformer before flow decoding.
- Block-sharing Cost Volume Masking: Block-sharing masking assigns one mask pattern to all source pixels within each large block, preventing neighboring cost maps from copying masked values.Blocks use heights and widths of 32–120 pixels, making local information leakage less effective.
- Pretraining Architecture: During pretraining, frozen image and context encoders keep raw cost-volume reconstruction targets static and avoid training collapse.The reconstruction targets are raw cost values generated from the frozen ImageNet-pretrained image encoder.
- Masked Cost-volume Tokenization: Masked cost-volume tokenization prevents masked features from entering later aggregation layers while retaining the standard FlowFormer semantics for finetuning.Masked convolutions remove masked-region features during pretraining; finetuning removes the masks and reuses the pretrained convolution and cross-attention parameters.
- Pre-text Reconstruction: The reconstruction task retrieves cost-memory features with small local patches and reconstructs larger patches at the same locations, encouraging long-range aggregation.This target preserves the cross-attention decoding structure and aligns the pretraining prediction target with finetuning.
4. Experiments
FlowFormer++ is evaluated on Sintel and KITTI-2015, where it improves generalization and dataset-specific performance over FlowFormer, with ablations supporting its masking and reconstruction designs.
- Quantitative Experiments: FlowFormer++ ranks 1st on both Sintel and KITTI-2015 benchmarks among published methods.The evaluation covers generalization with C+T and dataset-specific finetuning with C+T+S+K+H.
- Quantitative Experiments: 4.52 F1-all is achieved on the KITTI-2015 online benchmark, improving FlowFormer by 0.16 and S-Flow by 0.12.The model is finetuned on KITTI-2015 after the Sintel stage.
- Qualitative Experiments: FlowFormer++ preserves clearer details and better global consistency than FlowFormer on Sintel and KITTI test examples.Examples include sharper bird and leaf boundaries and more consistent flow across separated parts of an occluded car.
- Ablation Study: Block-sharing masking yields larger performance gains than random masking, supporting its role in MCVA.The ablation evaluates models pretrained and finetuned on C+T.
- Ablation Study: Masking 50% of cost values gives the best overall results, while the pre-text reconstruction design outperforms conventional MAE variants.The reconstruction design also uses random locations and local patches as query features to align pretraining with finetuning.
- Ablation Study: FlowFormer++ converges faster and reaches lower validation error than FlowFormer on FlyingChairs.The result indicates more effective feature relationships during pretraining and benefits supervised finetuning.
5. Conclusion
MCVA addresses the failure of naive MAE adaptation to cost volumes by using task-specific masking and reconstruction designs. These designs preserve pretraining-finetuning consistency, encourage long-range aggregation, and improve FlowFormer performance across datasets.
- MCVA enhances FlowFormer by pretraining its cost-volume encoder with a masked autoencoding scheme.
- Block-sharing masking and a novel reconstruction task address redundant cost volumes and pretraining-finetuning discrepancy.
- The proposed designs encourage long-range information aggregation while maintaining semantic integrity between pretraining and finetuning.
- Experiments demonstrate clear generalization and dataset-specific performance improvements.