PaperlayerView exact revision on arXiv

Source-linked AI summary

SCOPE-4D: Endoscopic 4D Geometry Foundation Models

Chaoyi Zhou, Zhongpai Gao, Anwesa Choudhuri, Meng Zheng, Benjamin Planche, Run Wang, Terrence Chen, Siyu Huang, Ziyan Wu

arXiv:2610.02343v1cs.CV

TL;DR

Endoscopic geometry learning faces scarce annotations and ambiguity between camera motion and tissue deformation. SCOPE-4D combines geometric supervision from SCOPE-5K with Common–Residual Motion and trajectory supervision to predict geometry and tissue tracks. It improves geometry and tracking across reported benchmarks, and the study favors its real-video reconstructions; the experiments do not establish clinical deployment safety.

  • Problem

    Endoscopic geometry annotations are scarce, and camera motion can be difficult to distinguish from tissue deformation.

    Source: §1 · p. 2

  • Method

    SCOPE-4D combines geometric fine-tuning on SCOPE-5K with Common–Residual Motion and trajectory supervision to jointly predict cameras, dense geometry, and 3D tissue tracks.

    Source: §1 · p. 2

  • Results

    Experiments report accurate geometry across in-domain and out-of-domain benchmarks, improved 3D tracking, more stable long-sequence reconstruction, and expert preference for real clinical-video reconstructions.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The results support large-scale endoscopic supervision and motion constraints for joint geometry estimation and tissue tracking.

    Source: §6 · p. 10

    The reported reconstruction-quality experiments do not establish safety for clinical deployment.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Geometric understanding supports endoscopic navigation and robotic assistance, but learning reliable endoscopic geometry faces two challenges: scarce geometric annotations and ambiguity between camera motion and tissue deformation. We present SCOPE-4D, an endoscopic 4D geometry foundation model that jointly predicts camera parameters, dense geometry, and 3D tissue trajectories from monocular RGB video in a single forward pass. Our curation and annotation pipeline constructs SCOPE-5K, a collection of approximately 5,000 clips spanning real and synthetic gastrointestinal endoscopy and laparoscopy. The collection provides rich geometric supervision and includes newly collected phantom and real-colonoscopy evaluation sets. Geometric supervised fine-tuning on SCOPE-5K learns endoscopic priors that improve camera and depth estimation. Common--Residual Motion (CRM) further constrains local deformation relative to common tissue movement. Together with geometric supervision, CRM and trajectory supervision further improve camera and depth estimation over geometric fine-tuning alone while enabling dense 3D tissue tracking. Evaluations on public and newly collected benchmarks demonstrate strong in-domain and out-of-domain geometry, superior 3D tracking, and more stable long-sequence colon reconstruction. A blinded user study further supports the perceived reconstruction quality on real clinical video. Together, these results demonstrate the value of large-scale endoscopic supervision and motion constraints for joint geometry estimation and tissue tracking.

1 INTRODUCTION

Endoscopic geometry learning is constrained by scarce annotations and ambiguity between camera motion and tissue deformation. SCOPE-4D pairs broad geometric supervision with explicit motion modeling to jointly estimate geometry and tissue trajectories.

1 INTRODUCTION

  • Endoscopic geometry annotations remain scarce, while camera motion, coherent tissue movement, and local deformation can produce similar image changes and distort reconstructed geometry.Weak texture, reflections, liquids, bleeding, and illumination changes also make reliable SfM correspondences difficult to obtain at scale.Source: §1 · p. 2
  • SCOPE-5K combines approximately 5,000 real and synthetic GI endoscopy and laparoscopy clips with new phantom and real-colonoscopy evaluation sets.The phantom set provides externally tracked camera references, and the real-colonoscopy set supports a blinded reconstruction-quality study.Source: §1 · p. 2
  • SCOPE-4D jointly predicts camera parameters, dense geometry, and 3D tissue trajectories from monocular RGB video in one forward pass, combining geometric SFT with CRM.CRM composes common tissue movement with point-dependent residual motion; trajectory supervision further improves camera and depth estimation over geometric fine-tuning alone.Source: §1 · p. 2
  • The reported evaluation finds leading geometry performance across in-domain and held-out benchmarks, improved 3D tracking, and more accurate long-sequence camera reconstruction.Prior geometry foundation models predict cameras and geometry feed-forward, while endoscopic methods face limited endoscopic coverage and ambiguity under weak texture and deformation.Source: §1 · pp. 2–3

2 RELATED WORK

Geometry foundation models extend feed-forward reconstruction to dynamic scenes, but endoscopic reconstruction remains constrained by motion ambiguity and limited data coverage. SCOPE-4D combines geometric supervision with common–residual motion, while new phantom and clinical benchmarks broaden evaluation.

2 RELATED WORK

  • General geometry foundation models learn feed-forward reconstruction, and dynamic-scene models extend it to joint geometry and motion prediction.DUSt3R predicts paired pointmaps, VGGT jointly predicts cameras and geometry, and D4RT extends feed-forward prediction to dynamic reconstruction.Source: §2 · p. 3
  • Endoscopic reconstruction methods include per-sequence neural fields, self-supervised camera-and-depth estimation, and adaptations of pretrained general-domain models, each with coverage or motion-ambiguity constraints.SCOPE-4D combines geometric SFT on SCOPE-5K with CRM to improve geometry estimation and constrain tissue deformation.Source: §2 · p. 3
  • Existing endoscopic datasets provide controlled geometry, clinical video, or synthetic and stereo data, but their coverage or acquisition setups limit representation of routine clinical conditions.SCARED and C3VD offer controlled reference geometry; EndoMapper broadens clinical coverage but lacks dense real-video ground truth, while SimCol3D and StereoMIS have specialized data characteristics.Source: §2 · p. 3
  • SCOPE-5K combines existing datasets with curated and annotated clinical video, while phantom and real-colonoscopy benchmarks assess camera accuracy and reconstruction quality, respectively.Figure 2 depicts the curation and annotation pipeline alongside joint prediction and the common–residual motion composition.Source: §2 · p. 4, Figure 2 · p. 4

3 SCOPE-5K: ENDOSCOPIC GEOMETRY DATA

SCOPE-5K expands endoscopic geometric supervision through curated existing sources and new annotations, with separate phantom and clinical benchmarks for reconstruction evaluation.

3 SCOPE-5K: ENDOSCOPIC GEOMETRY DATA

  • SCOPE-5K contains 4,855 in-domain clips across seven sources and 55 out-of-domain clips spanning real and synthetic GI endoscopy and laparoscopy.The collection adds camera and dense-geometry annotations to EndoMapper-Real and dense depth to StereoMIS.Source: §3 · p. 4

3.1 GEOMETRY ANNOTATION

  • For monocular annotation, the pipeline screens clinical video for visible anatomy and camera movement, filters unreliable frames, and splits videos at temporal discontinuities.MFT tracks become pairwise matches for incremental SfM, while independent LoMa and SIFT-LightGlue reconstructions cross-check camera trajectories.Source: §3.1 · p. 4
  • Recovered poses and intrinsics condition MapAnything to generate dense depth and point clouds, avoiding conventional photometric MVS under endoscopic imaging challenges.Quality control checks multi-view consistency and independent camera reconstructions; masks exclude depth-edge artifacts and regions without multi-view support.Source: §3.1 · pp. 4–5
  • StereoMIS camera poses from robot kinematics allow depth annotation without camera reconstruction, using rectified-stereo disparity and known focal length and baseline.Clip selection and quality control follow the monocular pipeline's principles, with checks tailored to stereo geometry.Source: §3.1 · p. 5

3.2 SCOPE-COLON BENCHMARK

  • SCOPE-colon-phantom synchronizes phantom video with a five-degree-of-freedom tracker for camera-position and viewing-direction references; SCOPE-colon-real supports blinded clinical reconstruction assessment.Together, the sets support controlled camera evaluation and clinical validation.Source: §3.2 · p. 5

4 SCOPE-4D: ENDOSCOPIC 4D GEOMETRY FOUNDATION MODEL

SCOPE-4D combines geometric fine-tuning with common–residual motion to predict camera parameters, dense geometry, and tissue tracks from monocular video. Its training jointly supervises geometry and motion, with motion losses also updating shared features.

4 SCOPE-4D: ENDOSCOPIC 4D GEOMETRY FOUNDATION MODEL

  • SCOPE-4D combines geometric fine-tuning on SCOPE-5K with Common–Residual Motion to jointly predict cameras, dense geometry, and 3D tissue tracks from monocular RGB video.Geometry-only SCOPE-4DSFT is distinguished from the full geometry–motion model, SCOPE-4D.Source: §4 · p. 5, §4.1 · p. 5

4.1 GEOMETRY SUPERVISED FINE-TUNING

  • Geometry SFT uses camera and depth annotations from SCOPE-5K to learn endoscopic priors and decode intrinsics, camera poses, depth maps, and reference-frame pointmaps.Depth maps are back-projected with intrinsics and transformed by camera poses to produce dense pointmaps.Source: §4.1 · p. 5

4.2 COMMON–RESIDUAL MOTION

  • CRM composes a shared rigid transform with point-dependent residual motion, constraining local deformation relative to common tissue movement.A fixed zero-residual anchor adds no local deformation; increasing its assignment tightens the displacement bound while preserving common motion.Source: §4.2 · pp. 5–6

4.3 JOINT GEOMETRY AND MOTION PREDICTION

  • A motion transformer combines spatial and temporal information from shared geometry features, and CRM applied to the reference pointmap produces dense tissue tracks separate from camera poses.Common-motion, residual-motion, and assignment heads predict the corresponding transforms and reference-frame logits.Source: §4.3 · p. 6

4.4 TRAINING

  • Training combines geometry and tracking losses, with motion supervision also updating the shared backbone used for camera and depth prediction.Tracking labels combine synthetic Sano trajectories and MFT correspondences lifted into 3D; common-motion fitting and assignment targets encourage points with small residuals to follow common motion.Source: §4.4 · p. 6

5 EXPERIMENTS

SCOPE-4D performs strongly on endoscopic geometry and tissue tracking, while ablations indicate benefits from diverse geometric supervision and common–residual motion modeling. Long-sequence results and blinded expert rankings also favor the full model.

5.1 3D GEOMETRY RECONSTRUCTION AND TRACKING

  • 29.44 versus 7.81 for in-domain pose AUC@5◦ and 0.042 versus 0.093 for depth AbsRel show gains from geometric SFT; the full model further improves pose and depth results.On held-out benchmarks, the full model leads in camera-stability ATE, ARE, and δ1.25 on Hamlyn and has the lowest ATE and viewing-direction error on SCOPE-colon-phantom.Source: §5.1 · p. 7
  • 19.8% lower EPE than the best competing method on StereoMIS accompanies the highest APD on both Sano and StereoMIS.SCOPE-4D also outperforms competing tracking methods in camera and depth metrics on both datasets.Source: §5.1 · p. 7

5.2 ABLATION STUDIES

  • 22.8% higher pose AUC@5◦ and 13.0% higher tracking APD over SoM accompany comparable depth accuracy for full CRM on StereoMIS.A zero-motion anchor and assignment supervision improve EPE and APD; common–residual composition and its supervision further improve pose AUC and APD.Source: §5.2 · pp. 8–9, Table 3 · p. 9
  • Both added data sources improve depth accuracy on unseen Hamlyn and provide complementary phantom-camera gains; together they yield the strongest overall depth results.Table 3(a) compares geometry-only SFT by data source.Source: §5.2 · p. 8, Table 3 · p. 9

5.3 DOWNSTREAM TASKS AND USER STUDY

  • Both SCOPE-4D variants outperform baselines on long-sequence reconstruction, and the full model achieves lower ATE than geometric SFT.Evaluation aligns overlapping chunks sequentially with Sim(3) transforms, without loop closure or global alignment.Source: Table 4 · p. 9, §5.3 · p. 9
  • SCOPE-4D earns the best mean expert rank and ranks first in 70% of evaluations for camera trajectories, point clouds, and depth maps.In paired comparisons with pretrained VGGT-Ω, participants prefer SCOPE-4D in 75.0%, 75.0%, and 76.7% of evaluations, respectively.Source: Table 5 · p. 9, §5.3 · p. 9

6 CONCLUSION

SCOPE-4D jointly predicts camera parameters, dense geometry, and 3D tissue tracks from monocular RGB video, combining broader endoscopic supervision with explicit tissue-motion modeling. The conclusion reports accurate geometry, improved tracking, more stable long-sequence reconstruction, and favorable expert rankings on real clinical video.

6 CONCLUSION

  • SCOPE-4D combines geometric supervision and tissue-motion modeling, improving geometry over geometric SFT alone while enabling dense tracking and more stable long-sequence reconstruction.The paper also reports accurate geometry across in-domain and out-of-domain benchmarks, improved 3D tracking, and favorable blinded rankings on real clinical video.Source: §6 · p. 10

APPENDIX

  • The appendix documents data splits, annotation checks, motion-target construction, training settings, and dataset-specific evaluation protocols.It also supplies per-domain geometry and tracking-threshold results alongside the main comparisons.Source: REPRODUCIBILITY STATEMENT · p. 11, APPENDIX · p. 15

6 CONCLUSION

  • The experiments assess reconstruction quality retrospectively and do not establish safety for clinical deployment.The study uses existing datasets, in-house phantom recordings, and clinical video; six clinical and biomedical experts took part in the blinded ranking study.Source: ETHICS STATEMENT · p. 11

A.1 DATA COMPOSITION AND SPLITS

SCOPE-5K combines diverse endoscopic data and geometric annotations with new phantom and clinical-colonoscopy evaluation sets. The appendix describes how real-video clips and geometry are curated, annotated, and quality-checked.

A.1 DATA COMPOSITION AND SPLITS

  • SCOPE-5K spans real and synthetic gastrointestinal endoscopy and laparoscopy, with source-specific splits and annotation coverage.It includes SCARED, C3VD, SimCol3D, EndoMapper-Sim, Sano, EndoMapper-Real, and StereoMIS; the tables summarize sources, splits, and sampling proportions.Source: §A.1 · p. 15, Table 6 · p. 15, Table 7 · p. 16

A.2 ENDOMAPPER-REAL CURATION AND ANNOTATION

  • EndoMapper-Real curation screens video, filters and segments frames, then reconstructs cameras and annotates dense geometry using independent correspondence checks.The screening criteria include tissue visibility, camera travel, obstruction, and image quality; retained clips are 50–300 frames.Source: §A.2 · pp. 16–17

A.3 STEREOMIS CURATION AND ANNOTATION

  • StereoMIS clips use provided camera poses and stereo-derived depth, with non-white-light segments excluded and source indices preserving alignment.Depth is computed from disparity using focal length and stereo baseline; instrument masks use the rectified image grid.Source: §A.3 · p. 18

A.4 QUALITY CONTROL

  • Quality control combines hard gates and soft flags; training uses Accept and Borderline clips, whereas held-out evaluation uses Accept clips.Checks include sparse–dense depth discrepancy, scale drift, cloud connectivity, and cross-source camera residuals; unavailable comparisons do not affect the QC label.Source: §A.4 · p. 18, Table 8 · p. 18
  • Human verification rejected 28 of 2,379 retained clips (1.18%), after 11 Borderline clips from held-out recordings were excluded.Table 9 reports the curation counts; Figure 5 illustrates borderline and rejected cases.Source: §A.4 · p. 18, Table 9 · p. 19

A.5 SCOPE-COLON BENCHMARK

The SCOPE-colon benchmarks evaluate camera accuracy on phantom recordings and reconstruction quality on clinical colonoscopy clips. The appendix also specifies training and ablation configurations for geometric fine-tuning and motion modeling.

A.5 SCOPE-COLON BENCHMARK

  • SCOPE-colon-phantom provides 14 physical phantom clips with 128 evaluated views each and tracked camera positions and viewing directions, but not roll.SCOPE-colon-real provides ten clinical clips with 128 views per clip for specialist assessment of reconstruction quality.Source: §A.5 · p. 19
  • Clinical recordings are assessed retrospectively, without changing clinical procedures or patient care.The study was conducted under institutional review board approval.Source: §A.5 · p. 19

B.1 OPTIMIZATION AND TRAINABLE MODULES

  • The full model trains for 29,000 optimizer steps on eight B200 GPUs, using a motion-branch learning rate of 2 × 10^-5 and 10^-5 for the image encoder and attention backbone.Its geometry heads remain fixed; geometric SFT updates all parameters.Source: §B.1 · p. 19
  • Motion-design ablations compare SoM, an anchor variant, and full CRM; the anchor denotes zero residual motion, so assigned points follow the common transform.Motion variants use StereoMIS training for 8,000 steps; the anchor variant adds static/moving assignment supervision, while full CRM adds common–residual composition and corresponding supervision.Source: §B.1 · p. 20

B.2 ARCHITECTURE AND SUPERVISION

The method predicts geometry and tissue motion from shared features, representing tracks in the reference-camera frame separately from camera poses. Geometry and tracking supervision use aligned annotations, robust common-motion targets, and losses defined over valid observations.

B.2 ARCHITECTURE AND SUPERVISION

  • The motion branch projects backbone features into a transformer that predicts per-frame twists and temporally shared reference-frame assignments.Trajectories use first-camera coordinates, and both common and residual transforms are initialized to identity by subtracting their first-frame twists.Source: §B.2 · p. 20
  • StereoMIS tracks are lifted into 3D using stereo depth and camera poses, while Sano trajectories combine rendered optical flow and depth across views.StereoMIS annotations exclude occlusions and instruments.Source: §B.2 · p. 20
  • A robustly fitted common SE(3) transform defines common-motion targets for valid observations, using a 3ϵ inlier threshold and rigid refitting; ϵ is a source-specific annotation noise floor.Residuals are measured in millimeters before training normalization; ϵ is 0.03 mm for Sano and 1.32 mm for StereoMIS.Source: §B.2 · p. 20
  • Geometry supervision normalizes points, depth, and camera translations by mean valid point distance and handles the two equivalent quaternion representations by minimizing over quaternion signs.The passage identifies pointmaps in first-camera and local camera frames, camera translation and rotation, camera-ray directions, depth, and predicted confidence.Source: §B.2 · pp. 20–21
  • Common-motion loss compares points transformed by predicted and fitted transforms, while assignment loss trains anchor probabilities from track residuals.Motion losses are averaged over the batch, with unlabeled samples contributing zero; geometry supervision covers pointmaps, depth, and camera quantities.Source: §B.2 · pp. 20–21

C PROOF OF THE CRM RESIDUAL BOUND

The CRM residual bound relates point displacement to residual motion around a common transform, and the protocol defines metrics for evaluating geometry and tracking. The compared-method description clarifies which methods use geometric fitting at test time.

C PROOF OF THE CRM RESIDUAL BOUND

  • As q0(x) approaches 1, displacement relative to common motion vanishes and the point follows Tc(t)p; zero mixed twist in SoM instead suppresses all point motion.The bound is stated for fixed residual bases.Source: C PROOF OF THE CRM RESIDUAL BOUND · pp. 21–22

D.1 COMPARED METHODS

  • Test-time geometric fitting differs across methods: CUT3R fits focal length, Endo3R fits focal length and uses PnP for poses, and D4RT fits cameras in closed form from predicted points.D4RT∗ denotes the open-source reproduction.Source: §D.1 · p. 22

D.2 IN-DOMAIN PROTOCOL

  • Depth is evaluated with AbsRel and δ1.25 after sequence-level median scale alignment; camera evaluation uses pose AUC, ATE, RTE, and RRE.For 3D tracking, EPE measures Euclidean position error and APD averages the percentages within 1, 2, 5, and 10 mm.Source: §D.2 · p. 22

D.3 OUT-OF-DOMAIN PROTOCOL

The out-of-domain protocol evaluates depth and camera stability on Hamlyn, camera accuracy against tracker measurements on the phantom set, and camera trajectories, point clouds, and depth in a real-colonoscopy user study. The study results are interpreted as sample-specific preferences, not population-level superiority.

D.3 OUT-OF-DOMAIN PROTOCOL

  • Hamlyn depth is assessed with AbsRel and δ1.25, while camera stability is measured on stationary-camera clips because camera-pose ground truth is unavailable.For the phantom set, camera estimates are evaluated against electromagnetic tracker measurements; long RealSynCol sequences use ATE, RTE, and RRE.Source: §D.3 · pp. 22–23
  • Table 10 identifies the compared methods, and Figure 6 depicts the synchronized user-study interface with reference video, masked depth, point clouds, and ranking controls.The figure caption says no rankings are selected in the displayed interface.Source: Table 10 · p. 23, Figure 6 · p. 24
  • Six blinded clinical and biomedical experts completed 180 four-method ranking tasks across ten clips, assessing camera motion, colon-wall coherence, and temporally stable depth.All four methods received the same 128 input frames per clip; rankings ran from 1 (best) to 4 (worst), without ties.Source: §D.3 · p. 23
  • The user study’s exploratory sign tests did not reach significance after Holm correction, so its rankings are interpreted as sample-specific visual preferences rather than population-level superiority.Mean per-clip Kendall’s W was 0.780 for camera trajectories, 0.728 for point clouds, and 0.698 for depth.Source: §D.3 · p. 23

E.1 ADDITIONAL QUANTITATIVE RESULTS

The additional quantitative results report camera-pose and depth results across five datasets, plus tracking-threshold results for Sano and StereoMIS. Additional figures provide geometry and depth comparisons, while noting a shared tissue-centered crop for each case.

E.1 ADDITIONAL QUANTITATIVE RESULTS

  • Camera-pose and depth results are reported for SCARED, SimCol3D, C3VD, EndoMapper-Sim, and EndoMapper-Real, with Sano and StereoMIS results in Table 2.Tables 11–15 contain the additional per-dataset results.Source: §E.1 · p. 24, Table 11 · p. 24, Table 12 · p. 25
  • Table 16 supplements mean APD with APD values at each distance threshold for Sano and StereoMIS.Table 16 reports 3D tracking APD at multiple thresholds.Source: §E.1 · p. 26, Table 16 · p. 26

E.2 ADDITIONAL QUALITATIVE RESULTS

  • Additional figures compare reconstructed geometry and depth using the same tissue-centered crop across methods within each case; distant camera frusta may fall outside the display.The figures include geometry comparisons, depth comparisons, and a long-sequence reconstruction on RealSynCol Colon 2.Source: §E.2 · p. 27, Figure 7 · p. 27, Figure 8 · p. 28
Loading 2610.02343v1…