Source-linked AI summary
FlowFormer: A Transformer Architecture for Optical Flow
Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, Hongsheng Li
TL;DR
Optical flow requires accurate per-pixel correspondence estimation, motivating a way to combine transformer long-range modeling with cost-volume representations. FlowFormer tokenizes and encodes a 4D cost volume into latent cost memory with alternate-group transformers, then recurrently decodes it using dynamic positional cost queries. It achieves state-of-the-art performance on Sintel and KITTI, including strong Sintel generalization without Sintel training.
Problem
FlowFormer addresses the challenge of combining transformers’ long-range modeling with the cost volumes widely used for optical-flow estimation.
Method
FlowFormer tokenizes a 4D cost volume into latent cost tokens, encodes them with alternate-group transformer layers, and recurrently decodes cost memory using dynamic positional cost queries.
Results
FlowFormer ranks 1st among compared methods on Sintel and KITTI, achieving 1.01 and 2.40 AEPE on Sintel clean and final passes.
Takeaways & Limitations
FlowFormer demonstrates that deeply integrating transformers with cost volumes can deliver strong optical-flow accuracy and generalization.
Abstract
from arXiv · showhide
We introduce optical Flow transFormer, dubbed as FlowFormer, a transformer-based neural network architecture for learning optical flow. FlowFormer tokenizes the 4D cost volume built from an image pair, encodes the cost tokens into a cost memory with alternate-group transformer (AGT) layers in a novel latent space, and decodes the cost memory via a recurrent transformer decoder with dynamic positional cost queries. On the Sintel benchmark, FlowFormer achieves 1.159 and 2.088 average end-point-error (AEPE) on the clean and final pass, a 16.5% and 15.5% error reduction from the best published result (1.388 and 2.47). Besides, FlowFormer also achieves strong generalization performance. Without being trained on Sintel, FlowFormer achieves 1.01 AEPE on the clean pass of Sintel training set, outperforming the best published result (1.29) by 21.7%.
1 Introduction
FlowFormer addresses the challenge of combining transformers’ long-range modeling with optical flow’s cost-volume representation. It introduces a transformer encoder-decoder that compactly encodes cost information and recurrently decodes flows.
- Optical flow estimates per-pixel correspondences as a 2D displacement field and supports dense correspondence reasoning in downstream video tasks.
- Recent optical-flow systems improve performance through cost-volume encoding and decoding, with PWC-Net using hierarchical local costs and RAFT using a global 4D volume with local retrieval.
- Transformers model long-range relations, but Perceiver IO operates directly on image-pair pixels, ignores cost volumes, and requires ∼80× training examples.
- FlowFormer combines transformer architectures with cost volumes to aggregate similarity information for optical-flow estimation.
- Its cost-volume encoder embeds the 4D volume into compact latent cost tokens, while its recurrent decoder estimates flows from encoded latent cost features.
- Two-step tokenization converts 2D cost maps into patches and further projects them to reduce the tokenization burden of the 4D cost volume.
- The decoder uses dynamic positional cost queries to retrieve cost information based on current flow estimates and iteratively refine flow residuals.
- FlowFormer reports state-of-the-art flow-estimation performance and validates that an ImageNet-pretrained transformer can benefit optical-flow estimation.
2 Related Work
Related work established cost volumes as central to optical flow and transformers as powerful long-range modeling tools. FlowFormer combines these strengths by globally aggregating cost-volume information in a latent space.
- Optical-flow research progressed from optimization with similarity and regularization terms to end-to-end deep convolutional networks.
- Cost volumes encode visual similarity between pixel pairs and act as a core component supporting optical-flow estimation.
- FlowFormer aggregates cost-volume information in a latent space with transformers while retaining the cost volume as a compact similarity representation.
- Perceiver IO demonstrated transformer-based optical-flow estimation but ignored cost volumes and required ∼80× more training examples.
- Global aggregation of similarity information is described as especially beneficial for hard cases involving large displacement and occlusion.
3 Method
FlowFormer builds a 4D cost volume, compresses it into latent cost memory with alternate-group attention, and recurrently decodes that memory into dense optical flow. Its encoder alternates spatial and latent-token information propagation, while its decoder uses dynamic positional queries for iterative refinement.
- Architecture: The architecture has three stages: building the cost volume, encoding it into cost memory, and recurrently decoding cost memory with source-image context features into flows.
- Cost Volume Construction: FlowFormer constructs an H × W × H × W 4D cost volume from dot-product similarities between all source-target feature pairs.
- Cost Volume Encoder: Each source pixel’s cost map compares that pixel with all target pixels, but repeated patterns and non-discriminative regions make correspondence identification difficult.
- Cost Volume Encoder: The encoder patchifies each 2D cost map, embeds cost patches, and encodes the resulting tokens into cost memory.
- Latent Tokenization: Positional embeddings are concatenated with patch features before attention produces K latent representations of dimension D for each source pixel.
- Alternate-Group Transformer: The alternate-group transformer alternates grouping tokens by source pixel and by latent representation, enabling information propagation within cost maps and across source pixels.
- Alternate-Group Transformer: Spatially separable self-attention incorporates source-image context features when generating queries and keys for cross-source-pixel cost propagation.
- Cost Memory Decoder: The decoder cross-attends from dynamic positional cost queries to cost memory, reusing keys and values across iterations and predicting flow residuals recurrently.
4 Experiment
FlowFormer is evaluated on Sintel and KITTI for both generalization and dataset-specific accuracy, with ablations isolating its cost representation, decoder, image encoder, and AGT layers.
- Quantitative Experiment: 1.01 and 2.40 AEPE on Sintel clean and final training sets place FlowFormer first among compared methods for generalization.Trained on FlyingChairs and FlyingThings only, it reduces errors versus GMA by 22.3% and 12.4%.
- Quantitative Experiment: 1.16 and 2.09 AEPE on Sintel clean and final test sets are 16.5% and 15.5% lower than GMA∗.FlowFormer does not use the warm-start strategy required by RAFT∗ and GMA∗.
- Qualitative Experiment: FlowFormer reduces flow leakage at object boundaries and recovers details such as hair and holes on the box compared with GMA.The qualitative comparison attributes this to contextual information from cost encoding over a large receptive field.
- Ablation Study: Replacing RAFT’s decoder with latent cost tokens and a cost memory decoder improves performance, with K = 8 and D = 128 achieving the best ablation setting.The latent tokens summarize whole-image cost information, while the decoder absorbs information through co-attention.
- Ablation Study: Adding AGT layers progressively decreases errors from no attention to AGT×3, demonstrating benefits from intra-cost-map and inter-cost-map attention.The final FlowFormer uses three AGT layers.
- Ablation Study: FlowFormer’s advantage over GMA is not explained by parameter count: GMA-L has comparable size yet 33% larger Sintel clean error, while GMA-Twins remains 15% worse.GMA-Twins also gives no significant reductions on other metrics and is worse on KITTI-15.
5 Conclusion
FlowFormer integrates transformers with optical-flow cost volumes through compact cost tokens, AGT encoding, and dynamic positional-query decoding. The design achieves state-of-the-art accuracy and strong cross-dataset generalization.
- Architecture: FlowFormer summarizes the H × W × H × W 4D cost volume as H × W × K tokens of length D.These compact tokens support efficient cost encoding.
- Architecture: Alternate-group transformer layers encode cost tokens into compact cost memory that captures essential information from the cost volume.
- Architecture: Dynamic positional cost queries decode the cost memory for residual flow regression without the limitation of local windows.
- Contribution: FlowFormer is presented as the first method to deeply integrate transformers with cost volumes for optical flow estimation.
- Results: FlowFormer achieves state-of-the-art accuracy and strong cross-dataset generalization.