PaperlayerView exact revision on arXiv

Source-linked AI summary

Disparity Has a Sign: Stereo Matching Beyond the Zero-Disparity Plane

Jian Shi, Xinge Yang, Chaoyang Wang, Wolfgang Heidrich, Peter Wonka

arXiv:2609.06809v1cs.CV

TL;DR

Stereo matching models and benchmarks have largely excluded negative disparity, despite its presence in stereoscopic content. ZDPShift measures this regime, and signed training restores performance across six backbones, while frozen-feature results indicate that the sign convention is lost at the output layer.

  • Problem

    Stereoscopic content includes behind-screen negative disparities, but existing benchmarks contain no negative-disparity pixels and models cannot represent or measure them.

    Source: §1 · p. 2

  • Method

    ZDPShift benchmarks stereo across multiple zero-disparity-plane positions, while SceneFlow supervision and signed hypothesis ranges enable negative-disparity estimation.

    Source: §1 · pp. 2–3, §4 · p. 6

  • Results

    Across six backbones, EPE rises 4.6–37× with negative disparities, while signed training removes the degradation.

    Source: §1 · p. 3, §5.2 · p. 7

  • Takeaways & Limitations

    Frozen matching features perform within 0.2 px of full tuning, indicating that correspondence features support negative disparity and the sign convention resides in the output layer.

    Source: §6 · p. 8

    Signed ground truth exists only for rendered content, so real delivered stereo performance can be validated only qualitatively.

    Source: §6 · p. 9

Abstract

from arXiv · show

Modern stereo matching models fail when disparity crosses zero, with end-point error (EPE) rising by 4.6-37$\times$. Yet stereoscopic content, from cinema 3D to VR, routinely contains objects behind the zero-disparity plane (ZDP), corresponding to negative disparities. The blind spot cascades through datasets, architectures, and evaluation protocols, all of which inherit the non-negative geometry. Rectified parallel cameras place ZDP at infinity, so every finite depth yields $d=fB/z \ge 0$ by construction, and nothing within the standard pipeline can violate, or even measure, a negative disparity. To measure it, we propose \textit{ZDPShift}, a benchmark of $21{,}495$ stereo pairs from seven cinematographer-authored open movies, each frame rendered at five zero-disparity-plane positions with dense signed ground truth. Six state-of-the-art image and video stereo matching models collapse once the plane moves. On identical scene content, FoundationStereo goes from $2.24$ px EPE to $75.33$px, with every backbone leaving roughly half of all pixels exceeding a three-pixel disparity error. What is missing, however, is not the underlying matching capability. % The capability itself, however, is already present. Training on supervision synthesized from SceneFlow, which adds no new data or parameters, keeps the error flat across the signed range. Training only the decoder, with the pretrained matching features frozen, performs comparably across all six backbones, with EPE jittering within $0.2$px. Thus, the pretrained features already extend to the negative regime they were never trained on, and only the output convention discarded it. Meanwhile, positive-regime accuracy on KITTI, Middlebury, ETH3D, and Sintel is largely preserved.

1 INTRODUCTION

Stereoscopic content places objects on both sides of the zero-disparity plane, but standard stereo matching models cannot represent behind-screen disparities. The paper introduces ZDPShift and signed models to measure and address that gap.

Motivation

  • Released 3D films can place more than 90% of pixels behind the screen, where zero-shot FoundationStereo and Stereo-Any-Video cannot represent the content but the signed model recovers it.Figure 2 shows four stereo films with red indicating in front of the ZDP and blue indicating behind it.Source: §1 · p. 2, Figure 2 · p. 2

Sign convention

  • The stereo-matching sign convention calls content beyond the zero-disparity plane negative, corresponding to the behind-the-screen regime in stereoscopic production.This differs from the display-parallax convention, where behind-screen content has positive parallax.Source: §1 · p. 2

Main findings

  • At ∆= +32, backbones reach 13–150 px EPE, while signed versions stay within 2.9–5.2 px across the signed range; overall, failures rise 4.6–37×.On identical scene content, FoundationStereo rises from 0.94 px EPE at ∆= 0 to 58.1 px at ∆= +32.Source: §1 · pp. 2–3, Figure 3 · p. 3

The gap

  • Standard stereo datasets, model architectures, and evaluation protocols enforce or measure only non-negative disparities, leaving negative-disparity failures unobservable.Rectified parallel rigs guarantee non-negative disparities; one-sided cost-volume ranges can make negative hypotheses inexpressible.Source: §1 · p. 3

Contributions

  • ZDPShift is a controlled multi-ZDP benchmark from open-movie content, and the paper adapts SceneFlow supervision and disparity hypotheses to cover signed disparities.The proposed changes restore negative-disparity estimation across six backbones without adding training parameters, while preserving positive-regime accuracy on standard benchmarks.Source: §1 · p. 3

2 RELATED WORK

The related work frames ZDP placement as a control over screen-relative depth, identifies stereo matching architectures as one-sided, and notes that existing datasets omit multi-ZDP coverage.

2.1 STEREOSCOPIC PERCEPTION AND ZDP PLACEMENT

  • The zero-disparity plane maps scene depth onto the display’s comfort budget, and automating its placement requires measuring disparity on both sides.Content too far in front of or behind the plane can be fatiguing.Source: §2.1 · pp. 3–4

2.2 STEREO MATCHING ARCHITECTURES

  • Cost-volume methods use one-sided disparity ranges that make negative disparity inexpressible, while recurrent methods initialize at zero and train on non-negative corpora.The paper targets this regime boundary rather than broadening competence within the non-negative regime.Source: §2.2 · p. 4

2.3 DATASETS

  • Common stereo datasets use rectified parallel rigs, so negative-disparity pixels cannot occur; no prior dataset renders the same scene at multiple ZDP positions.Table 1 compares datasets by capture/render convention, ZDP coverage, disparity sign, and scale.Source: §2.3 · p. 4, Table 1 · p. 4

3 MEASURING THE SIGNED AXIS

ZDPShift measures signed disparity by rendering open-movie scenes at multiple zero-disparity-plane positions while retaining rectified stereo geometry. Its 21,375 pairs progressively cover negative disparities, with coverage varying by scene.

3 MEASURING THE SIGNED AXIS

  • ZDPShift renders 32 scenes from seven open movies at five shifts, offsetting camera principal points while keeping physical positions and parallel alignment unchanged.Per-pixel ground-truth disparity is computed analytically from rendered depth.Source: §3 · pp. 4–5

3.1 COLLECTION PROTOCOL

  • A narrow f/64 aperture and disabled view-dependent compositor effects keep rendered views sharp and photometrically consistent for correspondence measurement.Lighting, materials, composition, camera path, and per-shot focal length remain inherited from the original scene files.Source: §3.1 · p. 5

3.2 DATASET STATISTICS

  • 21,375 stereo pairs span five ZDP shifts; at +32, 40% of pixels are negative, reaching −32 px disparities across 78% of frames.Coverage grows monotonically with shift for every scene, from essentially all-positive Agent 327 to 93% negative Project Gold.Source: §3.2 · p. 5, Table 2 · p. 6

4 ENABLING NEGATIVE DISPARITY ESTIMATION

The method addresses one-sided training data and disparity hypotheses with lightweight changes. It also tests whether pretrained matching features already support negative disparities by freezing them and training only the decoder.

4 ENABLING NEGATIVE DISPARITY ESTIMATION

  • Shifting SceneFlow pairs and relabeling disparity as d′ = d − ∆ synthesizes signed supervision, while half of each batch remains unshifted.For video models, each clip receives one shift; Dynamic Replica and CREStereo are also included in video training.Source: §4 · p. 6
  • Extending the GWC index range to [−dneg, dpos] and offsetting signed-bin soft-argmax enables negative hypotheses without adding training parameters.This change applies to IGEV-Stereo and FoundationStereo; RAFT-Stereo and the three video models need no architectural change.Source: §4 · p. 6
  • A frozen matching-feature pathway with decoder-only training tests how much of the negative regime pretrained correspondences already cover.The variant is designed to distinguish a need for new correspondences from a need for an output convention that admits signed disparity.Source: §4 · p. 6

5 RESULTS

Across six backbones, zero-shot error rises as the ZDP moves into negative disparity, while signed training keeps performance nearly flat. The frozen-feature variant is comparable, and positive-regime benchmark accuracy remains competitive.

5.1 EXPERIMENT SETTING

  • Inference is bounded to [−128, +384] at 960 px width, excluding 1.5% of valid pixels; temporal EPE uses one 16-frame clip per scene.All models use the stated training setup, including 30k iterations and a single H200 GPU.Source: §5.1 · p. 6

5.2 QUANTITATIVE RESULTS

  • At ∆=+32, zero-shot models degrade by around 4–37×, with roughly half of pixels exceeding three-pixel disparity error; signed variants stay nearly flat.The comparison uses the same scenes, and the table reports unchanged temporal consistency for both signed variants.Source: §5.2 · p. 7, Table 3 · p. 7
  • Frozen matching features match full training within 0.2 px throughout the signed range, and comparable performance appears within 500 steps.Figure 6 plots performance across training steps; its axes are logarithmic and step 0 is the pretrained model.Source: §5.2 · p. 7, Figure 6 · p. 7
  • Signed training retains competitive positive-regime accuracy and temporal consistency across the standard benchmarks evaluated for image and video models.The reported trade against negative-regime gains is favorable for every backbone.Source: §5.2 · p. 8, Table 4 · p. 8
  • With ∆=+32 withheld during training, models achieve comparable performance on the unseen shift rather than merely memorizing training offsets.The experiment trains on the other four shifts for 20k steps; Table 5 reports mean EPE over specified shifts.Source: §5.2 · p. 8, Table 5 · p. 8

6 DISCUSSION

Measurements on released stereo films find negative disparity in most sampled frames, and frozen-feature results place the sign convention in the output layer rather than the matching representation. Signed outputs also make disparity-based comfort assessment computable, while real-film validation remains limited by missing ground truth.

6 DISCUSSION

  • Across 81 clips from ten released films, 89% of frames contain negative disparity and 67% have more than half their pixels in that regime.The rectified parallel rig fixes the ZDP at infinity, producing only positive disparity by construction in the standard pipeline.Source: §6 · p. 8
  • Frozen matching features remain within 0.2 px of full tuning, supporting the interpretation that correspondence is represented while sign convention resides in the output layer.The features are not updated despite supporting correspondences of a sign they were never trained on.Source: §6 · p. 8
  • Signed disparity makes stereo comfort computable because the comfort budget is defined over content both in front of and behind the screen.The cited comfort budget bounds parallax angle to about 1° on either side of the screen.Source: §6 · p. 9
  • For ten clips and 50 generations, MEt3R degrades from 0.081 at t=0 to 0.091 at t=1 (+12.5%), with 9/10 clips worse at t=1.The comparison uses StereoCrafter outputs synthesized with entirely positive or entirely negative disparities.Source: §6 · p. 9, Figure 7 · p. 9
  • Low-texture regions can flip sign and jitter, while released films lack disparity ground truth, limiting real-film validation to qualitative assessment.The authors identify sign jitter as accounting for most of the small residual cost on real stereo films.Source: §6 · p. 9

7 CONCLUSION

The paper argues that negative disparity is a regular part of stereoscopic content, but one-sided training data and hypothesis ranges have made it unobservable to standard stereo matching. It reports that signed supervision and a signed search range can be added cheaply, and that frozen matching features still recover most of the gain.

7 CONCLUSION

  • Negative disparity is a regular operating regime of stereoscopic displays and movies, not an edge case.ZDPShift makes this regime measurable and is intended to make signed disparity a standard evaluation axis.Source: §7 · p. 9
  • One-sided training data and inexpressible negative-disparity hypotheses make the regime unobservable, rather than inherently difficult to match.Both constraints can be lifted cheaply; the paper reports that freezing matching features still recovers most of the gain.Source: §7 · p. 9
  • The pretrained representation could already match across the signed range, but lacked a way to express the sign.The conclusion attributes this gap to the output convention.Source: §7 · p. 9

A DATASET GENERATION DETAILS

The dataset shifts the zero-disparity plane through symmetric horizontal sensor offsets on parallel camera copies, while retaining shot-native focal lengths and finite per-pixel labels.

A DATASET GENERATION DETAILS

  • The ZDP shift is implemented as a symmetric horizontal sensor offset on two parallel copies of the artist’s camera.The left and right shifts are +∆/(2W) and −∆/(2W) in normalized sensor units, where W is image width in pixels.Source: A DATASET GENERATION DETAILS · p. 13
  • At ∆ = 0 the ZDP is at infinity; negative ∆ places it virtually behind the viewer, while focal lengths span 1,234–12,000 px.For f = 2,700 px and ∆ = +32, the stated ZDP depth is 8.44 m.Source: A DATASET GENERATION DETAILS · p. 13
  • Cycles shots use 32 samples with adaptive sampling and denoising, while EEVEE-authored shots render in EEVEE-Next.The forced deep aperture is f/64, and the reported empirical sampler-noise floor is approximately 0.005/255 per pixel.Source: A DATASET GENERATION DETAILS · p. 13
  • Disparity is stored as signed float32 with a finite label for every pixel; frames containing infinite-depth pixels are excluded.No pixels are masked.Source: A DATASET GENERATION DETAILS · p. 13

B NEGATIVE DISPARITIES IN PRODUCTION STEREO MOVIES

The section documents negative disparity in production films, evaluates how stereo generation and geometry models respond to ZDP shifts, and compares practical adaptation strategies.

B NEGATIVE DISPARITIES IN PRODUCTION STEREO MOVIES

  • The median production clip is 73.3% behind the screen; negative disparity appears in 91.4% of clips and covers over half the frame in 66.7%.The distribution varies widely across films, including films authored without negative disparity and documentaries that are almost entirely behind the screen.Source: B NEGATIVE DISPARITIES IN PRODUCTION… · p. 13

C THE BLIND SPOT ON THE GENERATION SIDE

  • MEt3R rises from 0.0808 at t = 0 to 0.0909 at t = 1, a 12.5% increase, and nine of ten clips score worse when content moves behind the screen.The comparison uses identical source frames and depth budget across ZDP positions.Source: C THE BLIND SPOT ON THE GENERATION SIDE · p. 14

D.1 NON-STEREO GEOMETRY PARADIGMS UNDER ZDP SHIFT

  • VGGT and MASt3R remain nearly flat across ZDP shifts but are imprecise: about 17.5 px and 12 px EPE, respectively, with 35–45% correct signs on behind-screen pixels.VGGT uses oracle alignment; MASt3R is evaluated on the pixels it matches.Source: §D.1 · p. 16, Table C · p. 16

D.2 REAL-FILM DEMONSTRATIONS ACROSS ALL BACKBONES

  • On the same real-film frames, zero-shot backbones force the scene into the positive regime, while the adapted versions recover behind-screen content.The demonstrations also show the bow correctly popping out in front.Source: §D.2 · p. 16

D.3 VERSUS THE NUMBER OF ADAPTED PARAMETERS

  • One trainable parameter gives FoundationStereo 7.06 px mean EPE across the signed spectrum, about 4.3× better, while capacity saturates early.The reported full recipe reaches 4.14 px with 1.13% of parameters trainable for FoundationStereo and 4.02 px with 9.6% for StereoAnyVideo.Source: §D.3 · pp. 16–17, Table D · p. 16

D.4 CAN THE ZDP SIMPLY BE MANUALLY RE-POSITIONED?

  • With manual repositioning limited to a fixed 2.9% image-width shift, FoundationStereo gains over 30% on negative regimes, while StereoAnyVideo is comparable.The cited example reaches 3.8% negative disparity, exceeding the 2.9% comfortable-viewing guideline.Source: §D.4 · p. 17

D.5 ABLATIONS

The ablations compare data-only adaptation, signed disparity search, and SceneFlow rehearsal to identify their contributions to signed-range performance.

D.5 ABLATIONS

  • The data-only recipe helps one-sided cost-volume backbones but leaves a large positive-∆ residual gap because their correlation volume is sampled out of bounds.The ablation omits the signed cost volume.Source: §D.5 · p. 17
  • FoundationStereo with signed search reaches 0.76 px mean EPE and 1.8% bad-3 at ∆ = +32; IGEV-Stereo reaches 0.92 px mean EPE.FoundationStereo is sub-pixel at every ∆, and RAFT-Stereo’s per-∆ profile is flat to 0.05 px.Source: §D.5 · p. 18
  • SceneFlow rehearsal contributes +0.62 px, or +64%, to the full recipe’s mean EPE when removed, with the cost spread roughly uniformly across all five shifts.This pattern is consistent with rehearsal acting as a regularizer across signed and positive disparity regimes.Source: §D.5 · p. 18

D.6 DETAILED PERFORMANCE BREAKDOWN

Released-model error tracks how much content lies behind the plane, while signed decoding reverses the source ranking. Remaining errors cluster in scenes with unreliable transparent-element geometry, and performance varies within a source movie.

D.6 DETAILED PERFORMANCE BREAKDOWN

  • Tables H and I report source- and scene-level results for image and video matchers at ∆= +32, under released-weight, full-training, and frozen-feature conditions.Results use 960-px inference, native-pixel errors, and predictions restricted to the declared search range.Source: §D.6 · pp. 18–19, Table I · p. 20
  • Project Gold, 90% behind the plane, is worst at 711 px for RAFT-Stereo, while Agent 327, 1% behind, is barely affected at 2.42 px.For released models, the ordering follows the share of each source behind the plane, not its content.Source: §D.6 · p. 18
  • Restoring the signed decoder removes that dependence and reverses the ordering, with Project Gold ending at 0.75 px.Charge and Sprite Fright remain hardest because smoke, foliage, and hair make the renderer’s geometry reference unreliable; five such scenes account for half the remaining error across all six backbones.Source: §D.6 · p. 18
  • Sprite Fright ranges from 1.50 to 15.53 px, while Agent 327’s RAFT-Stereo error rises from 2.42 to 3.93 px after the change.Agent 327 was never restricted and therefore had nothing to regain; the small cost matches the trade-off measured on external benchmarks.Source: §D.6 · p. 18
Loading 2609.06809v1…