PaperlayerView exact revision on arXiv

Source-linked AI summary

Hybrid Mamba-Transformer Network for Single-Shot 3D Reconstruction from Fringe Patterns

Xinjie Zhou, XiaoLei Li

arXiv:2610.33988v1cs.CVphysics.optics

TL;DR

Recovering accurate surface height from one high-frequency fringe image requires both long-range consistency and fine geometric detail. The paper combines a Swin-based U-Net with intermediate-resolution state-space modeling and an axial- and dilation-aware module; it reports lower errors and higher SSIM than four compared methods, including an RMSE of 0.548 mm.

  • Problem

    Accurate single-shot fringe-to-depth reconstruction must preserve long-range consistency and fine details, while CNNs have limited long-range interaction and windowed Transformers build distant interactions progressively.

    Source: §1 · p. 2

  • Method

    A Swin-based U-shaped network places visual state-space blocks at an intermediate encoder stage and uses ADVSS convolutions before four-directional selective scanning.

    Source: §4 · p. 22

  • Results

    The method outperforms hNet, SwinConvUNet, UHRNet, and HiDNet in MAE, RMSE, AbsRel, and SSIM, with 0.548 mm RMSE and 0.9985 SSIM.

    Source: §4 · p. 22

  • Takeaways & Limitations

    The reported findings support combining selective scanning with hierarchical window-based feature processing for single-shot fringe-based surface reconstruction.

    Source: §4 · p. 22

    Evaluation used data acquired with a fixed imaging configuration and fringe spatial frequency, so generalization across systems, distances, fringe densities, and materials remains to be established.

    Source: §4 · p. 23

Abstract

from arXiv · show

Deep learning facilitates rapid single shot three dimensional (3D) reconstruction using fringe projection profilometry (FPP), but recovering accurate surface height from a single high frequency fringe image remains challenging because both long-range spatial consistency and fine geometric details must be preserved. Convolutional networks are limited in modeling long range dependencies, whereas window-based Transformers establish global interactions only progressively with increasing depth. To address these limitations, a hybrid Mamba Transformer network is proposed for fringe to height regression. The network adopts a Swin based U shaped architecture and introduces visual state space modeling at the intermediate resolution encoder stage. An axial and dilated vision state space (ADVSS) block is further developed to enhance prescan feature encoding through local, dilated, and axial depthwise convolutions while retaining four directional selective scanning. This design combines linear complexity long range modeling with direction aware preservation of local fringe structures. Experiments on a publicly available FPP dataset show that the proposed method outperforms hNet, SwinConvUNet, UHRNet, and HiDNet in mean absolute error(MAE), root mean square error (RMSE), absolute relative error, and structural similarity index measure (SSIM). Using only one deformed fringe image, the proposed method achieves an RMSE of 0.548 mm and an SSIM of 0.9985, reducing RMSE by 65.2% and 59.6% relative to UHRNet and HiDNet, respectively. These results demonstrate the effectiveness of combining selective state space modeling with hierarchical window based attention for accurate single-shot FPP reconstruction.

1. Introduction

Single-shot FPP seeks to recover 3D surface geometry from one fringe image, but accurate reconstruction requires both long-range consistency and fine-detail preservation. The proposed hybrid combines a Swin-based U-Net with intermediate-resolution visual state-space blocks and an axial- and dilation-aware module.

1. Introduction

  • Single-shot FPP reconstructs 3D geometry from one fringe image, while conventional multistep phase shifting and temporal unwrapping require sequential images.FPP is described as a noncontact, high-precision optical metrology technique for 3D surface measurement.Source: §1 · p. 1
  • CNNs model local features, but limited long-range interaction can leave depth estimates inconsistent across high-frequency fringes, especially near discontinuities, steep slopes, shadows, and low-modulation regions.Recovering metric depth from one high-frequency fringe image requires both long-range consistency and accurate local-detail preservation.Source: §1 · p. 2
  • Swin attention reduces global self-attention’s quadratic cost, yet distant regions interact progressively across shifted-window blocks, and square windows do not explicitly exploit fringe directionality and periodicity.Source: §1 · p. 2
  • The network retains a Swin-based U-Net and adds visual state-space blocks at intermediate encoder stages, where reduced resolution balances computational efficiency with long-range feature modeling.VSS scanning propagates fringe information over extended ranges while Swin components aggregate local features and preserve fine geometric detail.Source: §1 · pp. 2–3
  • ADVSS applies axial and dilated convolutions before multidirectional selective scanning to enhance orientation-dependent and multiscale local features in high-frequency fringe patterns.The convolutions adapt feature encoding without changing the selective-scan mechanism.Source: §1 · p. 3
  • The proposed method outperforms hNet, SwinConvUNet, UHRNet, and HiDNet across MAE, RMSE, AbsRel, and SSIM, achieving 0.548 mm RMSE and 0.9985 SSIM.Its RMSE is reduced by 65.2% relative to UHRNet and 59.6% relative to HiDNet.Source: §1 · p. 3

2. Method

This section presents a method for mapping a single fringe image to a metric depth map using a Swin-based U-shaped network with intermediate visual state-space modeling and a compound MSE–SSIM objective. It evaluates the hybrid architecture through baseline comparisons and component and input-channel ablations, including reconstruction accuracy and preservation of fine surface structure.

3.1 Dataset and training

  • Offline training learns fringe-to-depth regression from dataset images and ground-truth depth maps, while online inference feed-forwards one captured fringe image to produce a metric depth map for 3D surface recovery.The method uses the dataset reported in Ref., which contains objects with diverse geometries, dimensions, and surface characteristics.Source: §2 · p. 4, §3.1 · p. 12

2. Method

  • The encoder combines an initial Swin stage, four ADVSS blocks, and a six-block Swin bottleneck; a two-stage Swin decoder restores resolution using attention-gated skip features.The output head produces a single-channel depth map normalized to [0,1].Source: §2 · pp. 5–6

2.2 Details of Swin-Transformer Block

  • Alternating W-MSA and SW-MSA enables local and cross-window feature exchange, while decoder-guided attention gates suppress less relevant encoder features before skip fusion.With fixed window size, hierarchical Swin attention has approximately linear computational complexity in the number of image patches.Source: §2.2 · pp. 8, 9, 11
  • ADVSS combines local, dilated, and axial depth-wise convolutions with four-directional selective scanning to model long-range fringe dependencies while preserving fine-scale details.Its parallel branches include a 3×3 convolution, a dilated 3×3 convolution with dilation rate 2, and 1×7 and 7×1 axial convolutions; the local response is also injected into the scan output.Source: §2.2 · pp. 9–11

2.5 Compound Loss Function

  • The compound objective combines MSE for point-wise height accuracy with SSIM loss for local structural consistency, using weights α = 0.80 and β = 0.20.SSIM uses a data range of 1.0 for normalized height maps.Source: §2.5 · pp. 11–12

3.1 Dataset and training

  • The dataset provides 3,070 left-camera fringe–depth pairs; the selected input is the ninth-frame 128-fringe pattern, and 74 samples are held out from optimization and checkpoint selection.The remaining samples are divided into 2,697 training and 299 validation samples, with the lowest-validation-loss model evaluated on the independent test set.Source: §3.1 · p. 13

3.3 Ablation experiment

  • Five-channel inputs improve all metrics for all-Swin Mode A but degrade VSS Mode B and ADVSS Mode E; single-channel Mode E remains the strongest overall configuration.From Mode E to Mode L, MAE increases from 0.195 to 0.205 mm and RMSE from 0.548 to 0.613 mm; the added channels contribute only 0.008 M parameters.Source: §3.3 · pp. 17–18
  • Mode E, with ADVSS at intermediate-resolution L1, performs best overall; removing its axial, dilated, or post-scan local-residual component worsens reported metrics.Removing the axial branch raises MAE from 0.195 to 0.205 mm and RMSE from 0.548 to 0.580 mm; removing the dilated branch raises AbsRel from 0.043 to 0.055, while removing the local residual yields the highest RMSE and AbsRel among the four variants, at 0.608 mm and 0.058.Source: §3.3 · pp. 16–18, Table 3 · p. 16

3.4 Quantitative and qualitative comparisons with representative methods

  • The proposed method reaches the lowest final RMSE on both training and validation sets, at 0.236 and 0.607 mm, respectively, with a comparatively stable validation curve.All methods broadly approach convergence after approximately 250 epochs, though baseline curves show pronounced early fluctuations.Source: §3.4 · p. 18, Figure 7 · p. 19
  • MAE 0.195 mm, RMSE 0.548 mm, AbsRel 0.043, and SSIM 0.9985 make the proposed method outperform hNet, SwinConvUNet, UHRNet, and HiDNet on the test set.Its RMSE is 72.0%, 67.6%, 65.2%, and 59.6% lower than hNet, SwinConvUNet, UHRNet, and HiDNet, respectively.Source: Table 5 · p. 19, §3.4 · p. 19
  • For two spatially isolated objects, the proposed method reports RMSE 0.549 mm and SSIM 0.9983, with reduced residual errors and better-preserved separation and fine surface structure.In a representative single-object profile, it produces the lowest error over most of the sampled line and preserves fine surface details more faithfully.Source: §3.4 · p. 20, Figure 8 · p. 20, Figure 9 · p. 21
  • The proposed method better preserves relative object positions, boundaries, and fine geometric structures than the baselines, which show detail loss, particularly near depth discontinuities.hNet and SwinConvUNet show especially weakened fine details.Source: §3.4 · p. 22
  • 0.543 mm RMSE and 0.9987 SSIM were achieved by the proposed method, reducing RMSE by 72.2% versus hNet and 60.0% versus HiDNet.Its absolute-error maps also show the smallest residual errors.Source: §3.4 · p. 22

4. Discussions and conclusion

The paper presents a hybrid Mamba–Transformer network for single-shot surface-height reconstruction, combining ADVSS-based selective scanning with hierarchical window attention. It reports stronger reconstruction metrics than four baselines, while noting computational and generalization limits.

4. Discussions and conclusion

  • At the intermediate-resolution L1 stage, ADVSS combines four-directional selective scanning with local, dilated, and axial depth-wise convolutions, while Swin blocks provide complementary feature refinement.The design aims to model long-range dependencies while retaining directional and multiscale fringe characteristics.Source: §4 · p. 22
  • Replacing L1 Swin blocks with VSS improves all four metrics, extending VSS to high-resolution L0 adds no benefit, and replacing L1 VSS with ADVSS gives the best overall configuration.Removing axial, dilated, or post-scan local residual components degrades performance; five-channel input does not improve VSS- or ADVSS-based models under the stated training protocol.Source: §4 · p. 22
  • MAE 0.195 mm, RMSE 0.548 mm, AbsRel 0.043, and SSIM 0.9985; the method outperforms hNet, SwinConvUNet, UHRNet, and HiDNet on all four metrics.RMSE is reduced by 65.2% versus UHRNet and 59.6% versus HiDNet.Source: §4 · p. 22
  • The proposed network uses 30.225 M parameters and takes 0.0246 s per image on an RTX 4090, versus hNet's 8.631 M parameters and 0.0092 s.The authors identify further optimization as necessary for resource-constrained platforms and high-speed measurement applications.Source: §4 · p. 23
  • Experiments use a fixed imaging configuration and fringe spatial frequency, leaving generalization across measurement systems and object materials unestablished; difficult surfaces and fringe conditions may still produce locally elevated errors.Future work will test more diverse real-world datasets and measurement systems and explore reliability and efficiency improvements.Source: §4 · p. 23

CRediT authorship contribution statement

The statement lists the authors' contributions, declares no competing interests, and identifies the dataset source and planned release of research materials.

CRediT authorship contribution statement

  • Xinjie Zhou is credited with the original draft, visualization, software, methodology, investigation, conceptualization, formal analysis, and data curation.Source: CRediT authorship contribution statement · p. 23
  • XiaoLei Li is credited with review and editing, software, project administration, methodology, conceptualization, formal analysis, and data curation.Source: CRediT authorship contribution statement · p. 23
  • The authors declare no known competing financial interests or personal relationships that could have influenced the reported work.The dataset is identified as reference, and the source code, trained models, and processed dataset are planned for public release after acceptance.Source: CRediT authorship contribution statement · p. 23

Related papers

Loading 2610.33988v1…