Source-linked AI summary
Distortion-Aware Spatiotemporal Learning for Real-Time Wide-Angle Video Rectification
Lang Nie, Yihan Yang, Wenbo Nie, Hongguang Zhu, Yang Wei, Chunyu Lin, Bin Xiao
TL;DR
Wide-angle lenses distort people near image boundaries, while existing approaches often lack real-time efficiency, temporal consistency, or full-body correction. FastVC combines distortion-aware control-point remapping with spatiotemporal learning and a dedicated dataset. It reports state-of-the-art rectification performance, including lower face and body distortion errors than the compared methods.
Problem
Wide-angle lenses stretch people near image boundaries, while existing methods often lack real-time efficiency, temporal smoothness, or balanced full-body correction.
Source: Introduction · p. 1Method
FastVC combines peripheral-aware thin-plate-spline control-point remapping with neighborhood spatiotemporal attention and sparse supervision for video rectification.
Source: Introduction · pp. 1–2Results
FastVC reports state-of-the-art performance, with 3.04% Avg. Diff. on face regions and 2.35% Avg. Diff. on body regions.
Source: Comparison Result · p. 6, Conclusion · p. 8Takeaways & Limitations
The wide-angle full-body video dataset provides a dedicated benchmark for training and standardized evaluation.
Source: Introduction · p. 2
Abstract
from arXiv · showhide
While wide-angle lenses provide an expanded field of view (FoV), they exhibit severe nonlinear distortion near image boundaries, unnaturally stretching people in these regions. Existing methods mostly focus on facial regions, require high computational costs, and lack temporal modeling, resulting in incomplete full-body rectification, impractical real-time deployment, and noticeable inter-frame shakes. To tackle these issues, we propose \textbf{FastVC}, an efficient full-body video rectification framework driven by distortion-aware spatiotemporal learning. Grounded in lens physics, FastVC introduces a distortion-aware thin-plate spline (DA-TPS) module that nonlinearly remaps control points into a peripheral-dense layout, enabling more accurate geometric fitting without additional computational cost. We further develop a dual-level spatiotemporal learning scheme for causal and stable video rectification. At the model level, neighborhood spatiotemporal attention (NST) aggregates local context and selectively fuses historical features with low overhead. At the optimization level, sparse spatiotemporal supervision applies spatial alignment only to anchor frames, decoupling frame-wise accuracy from temporal stability. Besides, we contribute the first benchmark dataset for wide-angle full-body video correction. Extensive experiments show that FastVC achieves state-of-the-art rectification fidelity and visual stability at 57.0 FPS.
Introduction
Wide-angle distortion, costly inference, and weak temporal consistency leave existing approaches short of real-time full-body video correction. FastVC combines distortion-aware geometric fitting with spatiotemporal learning and a dedicated dataset.
Introduction
- Existing methods are often too costly for mobile real-time use and, being designed for static images or faces, leave video smoothness and body correction inadequately addressed.Wide-angle distortion also stretches people near frame edges, degrading photo and video quality.Source: Introduction · p. 1
- DA-TPS remaps control points into a peripheral-dense layout, concentrating geometric capacity where distortion is strongest without extra computational overhead.The layout is motivated by the spatially non-uniform increase in distortion from the optical center toward the periphery.Source: Introduction · pp. 1–2
- FastVC combines neighborhood spatiotemporal attention with sparse supervision to support causal, stable video rectification.NST aggregates local features from historical frames; sparse supervision separates frame-wise fitting from temporal stability, reducing jitter and training costs.Source: Introduction · p. 2
- The authors build a wide-angle full-body video dataset spanning diverse smartphones and scenes, and present it as a dedicated benchmark for training and evaluation.They use per-frame pseudo-labels and report using the dataset to validate sparse spatiotemporal supervision.Source: Introduction · p. 2
Related Work
Prior portrait-correction research moved from iterative geometric optimization to learned image warping, then extended correction to video. The reviewed approaches retain limitations in computational cost, temporal stability, or full-body coverage.
Image Portrait Correction
- Early image-correction methods used mesh-based energy optimization; later learning-based methods estimated warping fields, including cascaded, Transformer, coupled-TPS, and multitask approaches.The reviewed optimization methods could depend on line extraction, manual annotations, or camera intrinsics, and iterative computation was costly.Source: Related Work · p. 2
- Image-level methods improved portrait correction but remained inadequate for maintaining temporal stability across video sequences.The review describes MOWA and UniRect as using Mamba-based architectures and prompt-guided mechanisms for multitask distortion rectification.Source: Related Work · p. 2
Video Portrait Correction
- Video-correction methods introduced spatiotemporal smoothing or unsupervised adaptation, but remained computationally intensive and focused on facial regions.The optimization-based smoothing approach was described as too slow for real-time mobile processing.Source: Related Work · p. 2
Method
FastVC rectifies wide-angle video through distortion-aware control-point placement and a dual-level strategy that combines local spatiotemporal feature aggregation with sparse spatial supervision and temporal regularization.
Overview
- FastVC uses a sliding window to rectify each current frame online, passing shared-backbone features through NST aggregation and a regression head that estimates a TPS warp for resampling.The window contains historical frames and the current frame; outputs are generated causally rather than from a full-video input.Source: Overview · pp. 2–3
Distortion-Aware TPS Transformation
- DA-TPS redistributes control points from the near-rectilinear center toward distorted boundaries, improving geometric representation without adding points.A concave radial remapping creates sparse center sampling and dense peripheral sampling; boundary anchors preserve rectangular integrity, and the resulting layout also favors inter-frame consistency.Source: Distortion-Aware TPS Transformation · p. 3
Dual-Level Spatiotemporal Learning Scheme
- NST first attends to local patches within each frame, then uses current-frame queries to selectively fuse features from historical frames at corresponding locations.Restricting attention to a w × w neighborhood reduces intra-frame complexity from O(M^2) to O(M · w^2); a learnable Gaussian prior controls the influence of frames at different temporal distances.Source: Neighborhood Spatiotemporal Attention · p. 4
- Sparse supervision applies spatial losses only to the two anchor frames while intermediate frames receive temporal motion regularization, avoiding fitting independent noisy pseudo-labels on every frame.The temporal term guides smooth transitions between anchors, while anchor spatial losses use a body mask and multi-scale pixel, gradient, and feature components.Source: Sparse Spatiotemporal Supervision · pp. 4–5
Dataset and Metrics
The paper builds a full-body wide-angle video dataset with frame-level annotations and evaluates rectification using measures designed to capture aspect-ratio errors and temporal stability.
Dataset Construction
- 32 1080P clips at 30fps provide 17,029 frames, captured across five smartphone brands and split into 22 training and 10 test sequences.The test split contains 5,215 frames; the training split contains 11,814.Source: Dataset Construction · p. 5
- Each dataset annotation combines a wide-angle image, rectified ground truth, flow ground truth, and full-body mask.Full-body labels expand image-level rectification from face regions to full bodies.Source: Dataset Construction · p. 5
Evaluation Metric
- Existing image-level metrics miss temporal consistency and full-body deformation, motivating the paper’s Avg. Diff. metric for face and body regions.ShapeAcc considers facial landmarks only and image-level metrics require spatially aligned static references.Source: Evaluation Metric · p. 5
- Avg. Diff. measures predicted-versus-ground-truth person aspect-ratio differences, while threshold-error rates and Stability Score assess robustness and inter-frame smoothness.Aspect-ratio comparisons are invariant to minor spatial shifts; the reported thresholds are 5% and 10%.Source: Evaluation Metric · pp. 5–6
Experiment
FastVC achieves low face and body distortion errors, leading temporal stability, and real-time speed. Ablations favor sparse anchor-frame supervision and the combination of NST and DA-TPS.
Implementation Details
- Training used 120 epochs on one NVIDIA RTX 5090 GPU, with frames resized to 512 × 384 and 172 TPS control points.The implementation uses Adam, and RAFT estimates optical flow for temporal regularization.Source: Implementation Details · p. 6
Comparison Result
- FastVC achieves 3.04% Avg. Diff. on faces and 2.35% Avg. Diff. on bodies, reducing face ¯e10 from 4.41% to 2.63%.Qualitative results across smartphone brands further show simultaneous face and body restoration.Source: Comparison Result · p. 6, Figure 6 · p. 7
- FastVC records a 0.8265 translation stability score, addressing high-frequency flickering seen in image-based baselines.The cited baselines are MOWA and CoupledTPS.Source: Comparison Result · p. 6
- In a blind study of 20 participants and 170 clips, FastVC records 86.47% Correction Quality, 54.71% Stability, and 61.77% Background Fidelity.The clips came from 10 testing videos across five brands.Source: Comparison Result · p. 6, Table 3 · p. 6
- FastVC runs at 57.0 FPS with 8.7M parameters and 3.64 GFLOPs, while VideoPC requires 4515.16 GFLOPs.Without fine-tuning, it also restores proportions and background lines on unseen OPPO, vivo, and Xiaomi devices.Source: Comparison Result · p. 6, Table 4 · p. 6
Ablation Study
- Sparse supervision retains comparable stability (0.8542 vs. 0.8555), while dense supervision yields 3.84% Face AR and 7.52% ¯e10.Temporal-only training collapses without spatial anchors; Z = 3 gives the best trade-off, while larger windows accumulate error.Source: Ablation Study · pp. 6–7, Table 5 · p. 7
- DA-TPS reduces Face AR from 3.61% to 2.74% in the single-frame setup by aligning control-point density with lens priors.The reported gain requires no extra parameters.Source: Ablation Study · p. 7, Table 6 · p. 7
- NST improves Body AR from 2.67% to 2.33% and Face AR to 3.17% over three-frame concatenation; NST plus DA-TPS reaches 3.04% Face AR and 0.8542 Stability Score.Naive temporal concatenation worsens Body AR from 2.45% to 2.67%.Source: Ablation Study · p. 7
Conclusion
FastVC combines distortion-aware geometric modeling with dual-level spatiotemporal learning for causal, real-time wide-angle video rectification. The paper reports state-of-the-art performance, high temporal stability, and computational efficiency.
Conclusion
- FastVC uses DA-TPS to align non-uniform control points with physical lens priors for efficient geometric modeling.Source: Conclusion · p. 7
- Its dual-level scheme combines NST attention at the model level with sparse supervision at the optimization level.Source: Conclusion · p. 8
- Together with a specialized full-body video dataset, the components support high temporal stability, state-of-the-art performance, and computational efficiency for mobile deployment.Source: Conclusion · p. 8