Source-linked AI summary
Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereo
Lukas Mehl, Jenny Schmalfuss, Azin Jahedi, Yaroslava Nalivayko, Andrés Bruhn
TL;DR
Existing benchmarks do not adequately represent or evaluate fine structures while also supplying large, high-quality data for modern dense-matching methods. Spring introduces a large computer-generated benchmark with HD imagery, super-resolved UHD ground truth, and region-focused evaluation across scene flow, optical flow, and stereo. Initial results show that fine-detail estimation remains challenging for recent methods.
Problem
Existing benchmarks lack a combination of high-quality data, many frames, and fine-detail coverage, while image-based scene flow has only one benchmark with limited variability.
Method
Spring provides rendered HD stereo sequences, super-resolved UHD ground truth, a public evaluation website, and metrics for multiple image regions and dense-matching tasks.
Results
Initial evaluations of 15 non-finetuned baselines show that small details remain difficult across optical flow, stereo, and scene flow.
Takeaways & Limitations
Spring enables high-detail assessment and broadens benchmark coverage for developing methods across optical flow, stereo, and image-based scene flow.
Takeaways & Limitations
The reported rankings are initial results from non-finetuned baselines and are intended as starting points pending finetuned submissions.
Abstract
from arXiv · showhide
While recent methods for motion and stereo estimation recover an unprecedented amount of details, such highly detailed structures are neither adequately reflected in the data of existing benchmarks nor their evaluation methodology. Hence, we introduce Spring $-$ a large, high-resolution, high-detail, computer-generated benchmark for scene flow, optical flow, and stereo. Based on rendered scenes from the open-source Blender movie "Spring", it provides photo-realistic HD datasets with state-of-the-art visual effects and ground truth training data. Furthermore, we provide a website to upload, analyze and compare results. Using a novel evaluation methodology based on a super-resolved UHD ground truth, our Spring benchmark can assess the quality of fine structures and provides further detailed performance statistics on different image regions. Regarding the number of ground truth frames, Spring is 60$\times$ larger than the only scene flow benchmark, KITTI 2015, and 15$\times$ larger than the well-established MPI Sintel optical flow benchmark. Initial results for recent methods on our benchmark show that estimating fine details is indeed challenging, as their accuracy leaves significant room for improvement. The Spring benchmark and the corresponding datasets are available at http://spring-benchmark.org.
1. Introduction
Existing benchmarks do not jointly provide high-quality, large-scale data for evaluating fine details and training modern neural methods. Spring addresses this gap with a large high-resolution dataset, UHD ground truth, region-focused evaluation, and an online benchmark.
- Dense correspondence estimation supports numerous computer-vision tasks, including driver assistance, robot navigation, visual odometry, stereo reconstruction, and structure-from-motion.
- Existing benchmarks lack a combination of high-quality data, many frames, and fine-detail coverage for training and evaluating current neural methods.
- Spring renders 6000 HD stereo pairs from 47 sequences and provides 12000 stereo and 23812 motion ground-truth frames.
- Spring computes ground truth at UHD resolution, enabling pixel-level evaluation of small details.
- The benchmark website supports result upload, analysis, comparison, and region-specific statistics including high-detail and large-displacement areas.
- 15 non-finetuned baselines show that small details remain difficult and leave significant room for improvement across optical flow, stereo, and scene flow.
2. Related work
Prior benchmarks vary in scene type, resolution, dataset size, and evaluation focus, but generally do not combine large-scale high-resolution data with fine-detail assessment. Spring extends this landscape for general-scene dense matching and image-based scene flow.
- Existing datasets span automotive and general scenes, with additional variation in motion complexity and illumination conditions.
- Resolution: Most benchmarks use QHD resolution or below, while Spring is the first general optical-flow and scene-flow dataset with HD input images and higher-resolution ground truth.
- Dataset size: Spring is the first general-scene benchmark with several thousand stereo and scene-flow samples for training and testing.
- Benchmark evaluation: Current evaluations emphasize occlusions, discontinuities, large displacements, and non-rigid regions, whereas Spring adds focused assessment of high-detail and thin-structure regions.
- Scene flow benchmarking: KITTI 2015 is the only image-based scene-flow benchmark but has only 400 frames, limiting dataset diversity for developing robust methods.
3. Spring dataset
Spring is a large, computer-generated dataset for scene flow, optical flow, and stereo, built from stereoscopic video sequences with extensive ground truth. Its design combines high-resolution rendering, super-resolution ground truth, and focused evaluation maps to assess fine structures and diverse motion regions.
- Spring provides stereoscopic video sequences and ground truth scene flow with reference-frame disparity, target-frame disparity, and optical flow for left/right views and forward/backward time directions.
- The dataset is generated from the open-source Blender movie “Spring,” whose scenes include varied shot sizes, animated and simulated motions, motion blur, depth of field, and camera zooms.
- The generation pipeline renders image data with visual effects, separately computes ground-truth data, handles sky regions, and derives additional maps for focused evaluation.
- Spring addresses thin structures that prior pixel-level representations could not adequately represent without altering their visual appearance.
- Super-resolution ground truth uses four values per pixel to identify high-detail regions, while matching, rigidity, sky, and displacement-size maps support region-specific analysis.
4. Benchmark
Spring combines a public evaluation website with region-specific metrics and high-resolution ground truth to assess dense matching methods. Initial non-finetuned results show persistent difficulties in detailed and challenging regions, while subsampling preserves rankings.
- Spring provides a public website for uploading test results, automatic evaluation, and optional public rankings.
- The benchmark evaluates scene flow, optical flow, and stereo using multiple error measures, including task-specific 1px outlier rates.The main scene-flow measure is a union 1px error over reference disparity, target disparity, and optical flow.
- 15 non-finetuned methods are evaluated using author-provided checkpoints trained on established datasets.The set includes 8 optical-flow, 4 stereo, and 3 scene-flow methods.
- Optical flow: Optical-flow errors remain large in high-detail, unmatched, non-rigid, sky, and large-displacement regions.MS-RAFT+ ranks first, whereas PWCNet ranks last; architecture capability for high-detail, high-resolution input is especially important on Spring.
- Stereo: Stereo results are generally worse than optical-flow results because methods often assume clean data and strictly positive disparity, excluding challenging sky regions.The benchmark argues that dense disparity methods should also handle zero-disparity or infinite-depth sky regions.
- Scene flow: Scene-flow results inherit stereo-related difficulties, while inconsistent results across training schedules indicate limited generalization from KITTI-focused training.
- Subsampling: Subsampling produces similar results and rankings to evaluation on all ground-truth pixels, independent of the error measure.This experiment makes the influence of subsampled evaluation transparent.
5. Conclusion
Spring is a large, high-resolution, high-detail benchmark for dense matching across optical flow, stereo, and image-based scene flow. Its initial evaluation shows that recent methods remain challenged by detailed and difficult image regions.
- Spring provides 6000 photo-realistic HD stereo pairs with 23812 motion and 12000 stereo UHD ground-truth frames.
- Spring is the first deep-learning-era benchmark to evaluate image-based scene flow alongside optical flow and disparity estimation.
- Initial results from 15 non-finetuned baselines show that Spring is challenging, particularly in high-detail, non-rigid, unmatched, and sky regions.