ActiveStereoNet: End-to-End Self-Supervised Learning for Active Stereo Systems

• Depth sensors have technology-specific limits: time-of-flight systems suffer motion artifacts and multipath interference, while structured light is vulnerable to ambient illumination and multi-device interference.Source: §1 · p. 1
• Active stereo combines patterned infrared projection with stereo cameras, and has only recently become commercially available, leaving little targeted research and no large-scale ground-truth training data.Source: §1 · pp. 1–2
• Active stereo must handle occlusions, high-resolution projected patterns, pattern-alignment local minima, and luminance differences, without access to a large ground-truth depth dataset.Source: §1 · p. 2
• ActiveStereoNet is a fully self-supervised end-to-end method that introduces a locally normalized reconstruction loss, adaptive window-based aggregation, and occlusion masking.The paper reports that these components improve convergence and test-time depth accuracy.Source: §1 · p. 2

• Time-of-flight systems can suffer motion artifacts and multipath interference, while structured light is vulnerable to multi-device interference and strong outdoor illumination.Source: §2 · p. 3
• Deep passive-stereo methods learn matching jointly but rely on strong supervision, limiting their advantage over handcrafted optimization when ground-truth depth data is unavailable.Source: §2 · p. 3
• Self-supervised passive stereo uses image reconstruction without ground-truth depth, but reported methods produce blurry results and struggle in textureless and dark regions.Source: §2 · p. 3
• The Intel D400 family provides 1280 × 720 images, but its handcrafted CENSUS and semi-global matching pipeline still has common stereo-matching issues, including edge fattening, occlusions, and holes.Source: §2 · p. 3
• Earlier learning-based active-stereo methods used shallow features, disparity regression, or direct intensity-to-depth mappings; reported problems include poor general-scene performance and textureless-region failures.Source: §2 · pp. 3–4
• The ActiveStereoNet architecture builds a low-resolution cost volume, upsamples and refines disparity, and trains an invalidation network to predict confidence.Source: Figure 2 · p. 4

3 Method

• The network takes rectified, synchronized active-illumination images and predicts full-resolution disparity maps; known focal length and baseline convert disparity to depth.The experiments use Intel RealSense D435 images at 1280 × 720 and 30 fps.Source: §3 · p. 4

3.1 Network Architecture

• A Siamese feature network builds a low-resolution cost volume, upsamples its disparity estimate, and refines high-frequency details; an invalidation mask is estimated alongside disparity.The system is reported to run at 60 Hz on a high-end GPU.Source: §3.1 · p. 5

3.2 Loss Function

• Photometric residuals can overweight bright, nearby pixels and occlusions; weighted local contrast normalization reduces brightness dependence and reweights residuals using local standard deviation.The local statistics are computed over a 9 × 9 patch; the weighted loss is reported not to be biased in bright or low-texture regions.Source: §3.2 · pp. 5–7

3.3 Window-based Optimization

• Adaptive support weights aggregate neighboring costs to smooth ambiguous cost functions while preserving edges, avoiding the detail loss associated with large uniform windows.The implementation uses a 32 × 32 window during training; graduated optimization produced results similar to using a single-pixel loss throughout.Source: §3.3 · pp. 7–8, Figure 4 · p. 7

3.4 Invalidation Network

• A left-right disparity consistency check invalidates pixels with disagreement above one disparity unit, and a constant-label cross-entropy regularizer discourages invalidating every pixel.The learned invalidation network produces and refines a mask from Siamese features, avoiding the need to predict disparity from both viewpoints at inference.Source: §3.4 · p. 8

4 Experiments

• The evaluation includes real and synthetic data, with additional experiments testing whether the self-supervised loss also applies to passive RGB stereo.The authors report improved generalization and compelling results on multiple passive-stereo benchmarks.Source: §4 · pp. 8–9, §4.1 · p. 9

4.1 Dataset and Training Schema

• Real training uses 10,000 office images and tests on 100 images from unseen scenes; synthetic training uses 10,000 images and tests on 1,200 frames from new scenes.The model is not fine-tuned on test data because doing so reduces real-application generalization capability.Source: §4.1 · p. 9

4.2 Stereo Matching Evaluation

• ASN achieves 1/30th-pixel disparity precision, one order of magnitude lower bias than the compared methods, and the lowest jitter at almost every depth.Its disparity maps are complete and preserve sharp boundaries; real-sequence outputs retain detail with low noise.Source: §4.2 · pp. 10–11, Figure 5 · p. 10

4.3 Ablation Study

• On synthetic data, supervision yields more pixels below 5-disparity error, while self-supervision does better below 2; on real test images, the supervised model produces blurrier results.The authors suggest the supervised model may overfit its training set.Source: §4.3 · p. 11
• On synthetic passive-stereo data, the supervised model has more pixels below 5-disparity error, but self-supervision does better below 2; on real test images, supervision produces blurrier results.The authors suggest the supervised model may overfit its training set.Source: §4.3 · p. 11
• With the same network settings, WLCN produces less noisy disparities, crisper edges, and fewer outliers than photometric or perceptual losses.Across over 10M evaluated pixels, WLCN has the lowest reconstruction error at low intensities, though the losses are similar over the rest of the intensity range.Source: §4.3 · pp. 11–12, Figure 7 · p. 12
• Using an invalidation mask outperforms omitting it across tested disparity-error thresholds and reduces gross errors near occlusion boundaries and flying pixels.The invalidation network reaches 80.7% average precision for occlusion detection, compared with 80.9% for a left-right check and 51.3% for photometric error.Source: §4.3 · pp. 12–13, Figure 9 · p. 14
• Window-based optimization with adaptive support weights recovers thin structures with higher precision while smoothing results and reducing edge fattening.The figure comparison reports more recovered thin structures and less edge fattening than single-pixel loss.Source: §4.3 · p. 13, Figure 10 · p. 14

• 0.03-pixel subpixel precision is reported, one order of magnitude better than other active stereo matching methods.The method also reports state-of-the-art results on challenging scenes.Source: §5 · p. 14
• ActiveStereoNet avoids oversmoothing and produces complete depth maps, crisp edges, and no flying pixels.Source: §5 · p. 14
• The invalidation network also estimates disparity confidence for applications that require occlusion handling.Source: §5 · p. 14
• Runtime is 15 ms per frame on an NVidia Titan X.Source: §5 · p. 14
• Transparent objects and thin structures remain problematic because of the cost volume's low resolution.Source: §5 · p. 14

• The supplement provides implementation details and further ablation studies of important framework components.Source: ActiveStereoNet: End-to-End… · p. 19
• Its passive-stereo evaluation shows the proposed approach can also be applied to RGB images.Source: ActiveStereoNet: End-to-End… · p. 19

• The section states that it provides the details needed to re-implement the proposed architecture.Source: A Additional Implementation Details · p. 19

A.1 Architecture

• The Siamese tower runs residual blocks at full resolution before downsampling later, preserving more high-frequency input signals.The architecture diagram shows the Siamese tower, invalidation network, and disparity refinement network.Source: §A.1 · p. 19, Figure 1 · p. 20
• A two-stream refinement network processes the image and upsampled disparity separately before merging features, producing fewer dot artifacts.Source: §A.1 · p. 19
• The invalidation network predicts disparity confidence and refines its upsampled invalidation map with the image and full-resolution disparity.Source: §A.1 · pp. 19–20
• Invalidation learning uses a reweighted L1 loss, assigning valid pixels 1 and invalid pixels -10 to address class imbalance.Source: §A.1 · p. 20

Resolution

• Because full-resolution 1280 × 720 training exceeds 12 GB memory, training crops matching 1024 × 256 regions from both images.The passage states that this crop does not change disparity, so models trained on small disparity can be tested at full resolution.Source: Resolution · p. 20
• The qualitative comparison indicates that active illumination has higher-level noise, passive illumination struggles in textureless regions, and using both yields high-quality disparity maps.Source: Figure 2 · p. 21
• Full resolution is needed for thin structures and crisp edges.Source: Figure 3 · p. 21
• Training generates invalidation targets through a left-right consistency check, enabling the invalidation network only after 20,000 disparity-training iterations.The staged schedule is reported to help disparity convergence and prevent invalidation from affecting the disparity network.Source: Resolution · p. 21

B Additional Evaluations

• The additional evaluation section presents further ablations, qualitative examples, and comparisons with other methods.Source: B Additional Evaluations · p. 21, Figure 4 · p. 22

B.1 Active + Passive Illumination

• Combining active and passive illumination produces smooth disparities with sharp edges, unlike active-only noise and passive-only failures in texture-less regions.Active-only results show dot spikes and lose object edges; passive-only matching fails on regions such as walls.Source: §B.1 · p. 22

B.2 Image Resolution

• Half-resolution inputs substantially degrade results, removing thin structures and boundary details compared with full-resolution inputs.The authors cite preserving high-frequency signals from the active pattern as the reason for retaining full resolution, while designing a compact network to limit memory use.Source: §B.2 · p. 22

B.3 Performance w.r.t. dataset size

• On Kitti 2012 and Kitti 2015, the method outperforms Godard et al.’s unsupervised method and generalizes better than the supervised loss.Source: Figure 5 · p. 23
• The algorithm produces reasonable depth maps on unseen images even with only 100 training images; increasing training data yields sharper results, but no additional improvement was found above 10,000 images.Source: §B.3 · pp. 22–23

B.4 More Qualitative Results

• The method estimates more complete disparity maps than local stereo matching algorithms and the sensor output in additional challenging scenes.Source: §B.4 · p. 23
• Its disparity maps show less edge fattening and oversmoothing in the additional challenging scenes.Source: §B.4 · p. 23
• Further results show predicted disparities together with invalidation masks on more diverse scenes.Source: §B.4 · p. 23

C Self-Supervised Passive Stereo

• The data-size evaluation reports sharper depth maps as training data increases, but no significant improvement beyond 10,000 examples.The figure caption says the overall error does not improve much as dataset size increases.Source: Figure 6 · p. 24
• The proposed loss achieves the lowest EPE on all evaluated datasets and outperforms the SceneFlow-trained supervised method on Kitti 2012 and Kitti 2015.Evaluation includes SceneFlow validation and unseen Middlebury, Kitti 2012, and Kitti 2015 datasets without fine-tuning; the authors say the Kitti results suggest the supervised loss is prone to overfitting.Source: C Self-Supervised Passive Stereo · pp. 23–24
• Qualitatively, the method shows less edge fattening and fewer frequent outliers than Godard et al. on every dataset, and avoids gross errors on Kitti images.The supervised model's Kitti errors are described as overfitting the original dataset.Source: C Self-Supervised Passive Stereo · p. 24
• The qualitative RGB comparison includes SceneFlow validation examples and unseen-dataset examples, with no fine-tuning on the latter.The compared networks were trained from scratch on SceneFlow using supervised, Godard et al., and proposed losses.Source: C Self-Supervised Passive Stereo · p. 23, Figure 8 · p. 26