PaperlayerView exact revision on arXiv

Source-linked AI summary

PhasorNet: Learning Structure from Frequency for Real-Time Stereo Matching

Md Raqib Khan, Santosh Kumar Vipparthi, Subrahmanyam Murala

arXiv:2608.29819v1cs.CV

TL;DR

Accurate stereo matching remains difficult in fine structures and reflective or transparent regions because appearance cues can be ambiguous or unreliable. PhasorNet addresses this with Fourier phase-aware attention, geometry-context refinement, and multi-scale edge/high-error supervision; with 5.3M parameters, it achieves state-of-the-art ETH3D performance and strong cross-dataset results.

  • Problem

    Stereo matching must recover precise pixel correspondences despite ambiguous appearance cues in fine structures and reflective or transparent regions.

  • Method

    PhasorNet combines Fourier phase-aware attention, convolutional and lightweight WQA/CDGA refinement streams, and multi-scale EHR supervision for structure-preserving disparity estimation.

  • Results

    PhasorNet achieves state-of-the-art ETH3D performance and strong cross-dataset generalization, using only 5.3M parameters.

  • Takeaways & Limitations

    PhasorNet provides an efficient stereo-matching framework that balances accuracy, generalization, and detail preservation for challenging surfaces.

Abstract

from arXiv · show

Accurate stereo matching remains challenging in ill-posed regions such as fine structures, reflective, or transparent objects, where appearance cues are often ambiguous or unreliable. To tackle this, we propose PhasorNet, a lightweight yet powerful framework that boosts geometric discrimination via frequency-domain cues. At its core, the Phase-Augmented Transformer (PAT) injects Fourier-derived phase information into the attention mechanism, yielding photometrically robust, structure-preserving features that prioritize structural consistency in difficult areas. Additionally, we develop a Geometry-Context Fusion Refinement Module (GCFRM) that combines a full-resolution convolutional stream with a lightweight attention-based stream (leveraging WQA and CDGA blocks) to efficiently preserve fine details and object boundaries without excessive overhead. Training is further enhanced by a multi-scale Edge-guided High-Error Region (EHR) loss that adaptively focuses optimization on high-error and edge regions, guiding hierarchical cost volume refinement. With only 5.3M parameters, PhasorNet achieves state-of-the-art performance on the challenging ETH3D benchmark while exhibiting excellent cross-domain generalization on KITTI, delivering an efficient and practical solution for accurate real-time stereo matching.

1 Introduction

PhasorNet addresses stereo matching failures in reflective, transparent, weak-textured, and fine-structure regions with frequency-aware structural features and efficient refinement. Its compact design combines phase-augmented attention, geometry-context refinement, and edge/high-error supervision to balance accuracy, efficiency, and detail preservation.

  • Motivation: Stereo matching remains difficult in ill-posed regions because photometric inconsistencies and weak texture undermine reliable pixel correspondences.These challenges are especially relevant to 3D perception applications requiring precise scene geometry.
  • Phase-Augmented Transformer: PAT injects Fourier phase information into attention to capture structural cues that are robust to intensity variations and encode spatial displacement across views.Phase-aware features emphasize geometric consistency over appearance similarity, including in reflective and weakly textured regions.
  • EHR Loss: The EHR Loss supervises disparity maps at multiple scales while emphasizing object boundaries and high-error regions during hierarchical cost-volume refinement.This approach avoids heavy post-processing and supports refinement without computationally expensive 3D convolutions.
  • Geometry-Context Fusion Refinement: GCFRM combines a full-resolution convolutional stream with lightweight WQA and CDGA attention to preserve fine details and object boundaries during disparity refinement.FFT supplies global structural cues, while DWT provides localized multi-scale frequency decomposition for boundary preservation.
  • Efficiency and Evaluation: 5.3M parameters make PhasorNet a compact stereo network designed to balance accuracy, efficiency, and detail preservation.Experiments across Scene Flow, KITTI 2012/2015, and ETH3D demonstrate a compelling accuracy-efficiency trade-off for low-resource deployment.

2 Related Work

Prior stereo methods improved cost aggregation, correspondence features, and refinement, but recurring challenges include computational cost, weak boundary preservation, and limited robustness in difficult regions. PhasorNet’s architecture is presented against this background as a phase-aware, coarse-to-fine alternative.

  • Cost-Volume Methods: Early deep stereo systems popularized 3D convolutions for cost-volume aggregation, while later adaptive 2D and cascaded methods reduced but did not eliminate memory and computation costs.These approaches include GC-Net, PSMNet, AANet, and CFNet.
  • Feature Representation: Learned correspondence features can become oversmoothed or semantically misaligned, while transformer encoders risk losing high-frequency boundary and textureless-region details.Recent methods add geometric cues or monocular priors, but heavy cost-volume post-processing remains common.
  • Foundation-Model Approaches: Foundation-model stereo methods improve robustness and generalization through monocular depth priors, but some achieve benchmark strength with higher memory and computation demands.MonSter uses confidence-guided mutual refinement, whereas DEFOM-Stereo uses a recurrent framework with combined encoders and scale updates.
  • Disparity Refinement: Recurrent refinement methods such as RAFT-Stereo and CREStereo achieve high accuracy but can increase inference time and smooth object edges.DLNR addresses over-smoothing with decoupled LSTM streams and disparity normalization.
  • Proposed Direction: PhasorNet’s pipeline uses phase-augmented feature extraction, coarse-to-fine cost-volume refinement, and geometry-context refinement to address these related-work trade-offs.The architecture overview identifies multi-scale feature extraction, coarse cost construction, and final disparity enhancement as successive stages.

3 Methodology

PhasorNet combines phase-augmented cross-attention, multi-scale 2D cost-volume refinement, and lightweight disparity refinement for stereo matching. Its phase features preserve structural cues while reducing sensitivity to appearance variations.

  • Overall Pipeline: The pipeline extracts multi-scale stereo features, constructs a normalized-inner-product cost volume, and progressively refines it using lightweight 2D operations.The cost volume is refined from quarter resolution through half and full resolution, avoiding computationally expensive 3D convolutions.
  • Phase-Augmented Cross-Attention: Phase-only Fourier features preserve high-frequency edges and contours while suppressing appearance-dependent variations.Magnitude normalization uses ε = 10^-5 for numerical stability before inverse transformation.
  • Phase-Augmented Cross-Attention: PAC injects phase features into query and key embeddings, while leaving value features unmodified for contextual aggregation.The asymmetric residual augmentation targets correspondence matching and preserves richer value information.
  • Phase-Augmented Cross-Attention: Cross-attention weights compare phase-augmented left queries with right keys using scaled dot-product attention and a learnable temperature.Queries, keys, and values are generated through lightweight pointwise and depthwise convolutions.
  • Phase-Augmented Cross-Attention: Residual attention outputs are projected and added back to the original features, followed by a lightweight feed-forward refinement network.Residual connections preserve the input features while incorporating cross-view information.

3.3 Geometry-Context Fusion Refinement Module

GCFRM refines initial disparity by coupling a full-resolution convolutional stream with a multi-resolution attention stream. WQA enhances query structure and CDGA selectively gates encoder–decoder fusion to preserve details efficiently.

  • Module Overview: GCFRM takes the cost volume, left-image features, and initial disparity as inputs and produces a refined disparity estimate.The module combines geometric and contextual cues for disparity refinement.
  • Convolutional Stream: Stream-1 preserves local details through six convolutional stages that exchange features with the second stream.Each stage uses convolutional layers with ReLU activations and receives information from prior and corresponding attention stages.
  • Transformer-Based Encoder–Decoder Stream: Stream-2 uses three encoder and three decoder stages, coupling each transformer block with one of Stream-1’s six multi-scale feature maps.The encoder reaches a bottleneck at one-eighth of the original resolution before symmetric decoding.
  • Wavelet-Based Query Attention: WQA applies wavelet decomposition to the query, reassembles processed subbands, and preserves contextual information in keys and values.This asymmetric design is described as improving robustness to noisy regions and the accuracy-efficiency trade-off.
  • Cross-Dimensional Gated Attention: CDGA uses cross-dimensional attention to emphasize informative features and suppress unreliable responses during encoder–decoder fusion.Its attention weights span kernels, filters, channels, and spatial dimensions, supporting refinement in difficult regions with minimal overhead.
  • Disparity Prediction: The final full-resolution decoder feature predicts a residual that is added to the initial disparity, while asymmetric gating supports edge sharpness and lightweight computation.The complete design explicitly combines convolutional localisation with global attention.

3.4 Proposed EHR Loss

EHR Loss combines edge-aware and high-error-region supervision to focus disparity learning on boundaries and difficult pixels across hierarchical refinement stages. Its visualization varies the selected error fraction and shows complementary boundary sharpening and hard-region emphasis.

  • Loss design: EHR Loss combines edge-aware disparity supervision with high-error-region disparity supervision for challenging occlusions, boundaries, and textureless areas.The edge term weights disparity errors using ground-truth and predicted Canny edges, while the high-error term selects pixels with the largest absolute disparity errors.
  • Visualization: Figure 4 compares k ∈{3, 5, 7, 10, 15}% and visualizes L_her’s error focus alongside L_edge’s boundary enhancement.The heatmaps illustrate how the two components emphasize different spatial properties.
  • High-error focus: The high-error component selects the top k% pixels by absolute disparity error and propagates gradients only through those selected errors.The mask is detached during backpropagation, and k = 7% is chosen from sensitivity analysis as the best accuracy–stability trade-off.
  • Loss design: The combined loss is L_EHR = λ_1 · L_edge + λ_2 · L_her, with λ_1 = 1.0 and λ_2 = 0.5.These weights balance boundary sharpening against focus on hard regions.
  • Hierarchical supervision: The total objective sums L1 and EHR losses across hierarchical refinement stages, while Stage 2 trains the refinement module with multi-stage regression.This hybrid supervision is described as supporting stable geometry-aware learning in Stage 1 and precise high-resolution disparity in Stage 2.

3.5 Implementation Details

PhasorNet is trained on Scene Flow with AdamW, OneCycle scheduling, augmentation, and random cropping following the RAFT-Stereo protocol. Scene Flow visualizations compare disparity before and after refinement against ground truth, highlighting boundary and detail changes.

  • Training setup: PhasorNet is trained on all 35,454 Scene Flow stereo pairs using AdamW with specified β values, weight decay, and a OneCycle schedule.The schedule uses warmup and cosine annealing, peaks at 2 × 10^-4, and training uses batch size 4.
  • Training setup: Data augmentation and random cropping follow the RAFT-Stereo protocol.
  • Refinement visualization: Figure 6 compares Scene Flow disparities before and after refinement with ground truth, using a boxed region to highlight boundary accuracy and detail.

3.6 Datasets and Evaluation Metrics

The evaluation spans synthetic Scene Flow and real-world KITTI and ETH3D benchmarks, using dataset-specific disparity-error metrics. The reported tables distinguish benchmark accuracy, cross-dataset generalization, efficiency, and zero-shot performance on reflective and edge regions.

  • Datasets: Scene Flow provides 35,454 synthetic training pairs, while KITTI-2012, KITTI-2015, and ETH3D provide real-world evaluation settings with occlusions, reflections, sparse labels, and high-resolution pairs.KITTI-2012 has 194 training and 195 testing images, KITTI-2015 has 200 training and 200 testing images, and ETH3D has 27 training and 20 testing pairs.
  • Metrics: Scene Flow and KITTI use D1 Error, defined by disparity errors exceeding 3 px or 5% of ground truth.
  • Metrics: ETH3D uses Bad 1.0, AvgErr, and RMS to emphasize large and boundary-related disparity errors.
  • Cross-dataset evaluation: Table 2 reports D1 error across Scene Flow, KITTI-2012, and KITTI-2015, alongside runtime, parameter count, and memory usage.The table evaluates models trained on Scene Flow and marks best and second-best results.
  • Targeted evaluation: Table 3 evaluates zero-shot KITTI-2012 performance on reflective-region and edge masks using D1_ref and D1_edge.Both metrics are lower-is-better D1 errors restricted to the specified regions.
  • Single-pass evaluation: Table 4 evaluates single-pass zero-shot Bad3 error on KITTI-2012 and KITTI-2015 under a constrained one-iteration setting.This setting assesses robustness when iterative refinement is limited.

3.7 Results Analysis

PhasorNet reports strong ETH3D accuracy with a compact model and competitive cross-dataset and targeted zero-shot performance. The evaluation emphasizes both benchmark results and performance on reflective and edge regions.

  • ETH3D results: 0.41% and 0.58% Bad 1.0 errors on ETH3D non-occluded and all pixels, respectively, surpass MonSter’s 0.44% and 0.70%.PhasorNet also ranks second in non-occluded AvgErr and RMS while using 5.3M parameters.
  • Cross-dataset results: 1.95% D1-all error on Scene Flow after synthetic training outperforms MonSter at 2.02% and Mocha-Stereo at 2.35%.
  • Cross-dataset results: 4.30% and 4.19% D1-all errors are achieved without fine-tuning on KITTI-2012 and KITTI-2015, respectively.
  • Targeted results: Zero-shot KITTI-2012 performance reaches D1_ref = 4.95% and D1_edge = 4.47%, ranking second on reflective regions and best on edges.
  • Efficiency comparison: PhasorNet is included in a comparison of lightweight stereo matching methods using lower-is-better error metrics.

3.8 Complexity Analysis

PhasorNet combines state-of-the-art ETH3D accuracy with low latency, a compact parameter count, and modest memory use. Its frequency-based modules contribute limited latency relative to standard convolutions.

  • 108 ms inference, 5.3M parameters, and 2.08 GB memory establish PhasorNet’s compact efficiency on KITTI-2015.On the same image-pair setting, MonSter uses 388.6M parameters, 595 ms, and 7.64 GB.
  • 98% fewer parameters and over 5× faster inference distinguish PhasorNet from MonSter on the KITTI-2015 efficiency benchmark.
  • 29.3% of total inference time comes from PAT, WQA, and CDGA, while standard convolutional layers account for 70.6%.FFT and DWT operations are parameter-free and memory-bound, contributing negligible computational cost compared with convolutions.
  • PhasorNet achieves better ETH3D accuracy than HITNet while maintaining a favorable accuracy-efficiency trade-off.

3.9 Ablation Study

Ablations show that PAC, GCFRM, WQA, CDGA, and EHR Loss each improve stereo matching, with PAC and GCFRM providing complementary gains. Design choices within WQA and the high-error loss also affect accuracy and latency.

  • Adding GCFRM reduces refined EPE from 0.58 to 0.52, while adding PAC further lowers it to 0.48 on Scene Flow.The progressive gains indicate complementary rather than redundant contributions from PAC and GCFRM.
  • The full model reaches refined D1 of 1.95% and EPE of 0.48; removing PAC, WQA, or CDGA consistently worsens both metrics.Removing PAC yields 2.10% D1 and 0.52 EPE, WQA 2.05% and 0.50, and CDGA 2.00% and 0.49.
  • Asymmetric WQA improves D1 from 2.05 to 1.98 over the no-WQA baseline and outperforms symmetric WQA at 2.01 with lower latency.
  • Setting k = 7% produces the best refined D1 of 1.95% in the high-error-region loss sensitivity study.
  • Replacing L1 supervision with full EHR Loss reduces refined D1 from 1.98% to 1.95%, with edge-aware and high-error terms adding incremental gains.

4 Conclusion

PhasorNet combines Fourier phase information, wavelet-based attention, detail-preserving refinement, and multi-scale EHR supervision for robust and efficient stereo matching. It achieves strong benchmark performance, but uniform areas remain ambiguous for correspondence-based methods.

  • PhasorNet integrates Fourier phase cues and wavelet-based attention to improve robustness on reflective, transparent, and weakly textured surfaces.
  • PAT injects phase-derived structural cues into cross-attention, while GCFRM combines convolutional, WQA, and CDGA streams for efficient refinement.
  • 5.3M parameters and 108 ms inference accompany ETH3D Bad 1.0 values of 0.41% non-occluded and 0.58% all, plus strong zero-shot KITTI generalisation.
  • Perfectly uniform areas remain ambiguous for correspondence-based methods, including PhasorNet.
Loading 2608.29819v1…