PaperlayerView exact revision on arXiv

Source-linked AI summary

DART: Depth-as-Target Pretraining for Surgical Vision Foundation Models

John J. Han, Adam Schmidt, Muhammad Abdullah Jamal, Jie Ying Wu, Omid Mohareri

arXiv:2609.04555v1cs.CV

TL;DR

Surgical systems need visual representations that capture semantics, fine detail, and spatial structure, while encoder-input depth methods create a clinical deployment constraint. DART adds pseudo-depth reconstruction to masked iBOT patches during DINOv2 pretraining, and improves performance across dense and image-level surgical tasks.

  • Problem

    Prior RGB-D methods use depth as an encoder input, requiring depth at inference in clinical pipelines where depth sensors may be unavailable.

    Source: Introduction · p. 2

  • Method

    DART adds a masked-patch pixel-wise depth reconstruction objective to DINOv2, using pseudo-labeled depth only as a pretraining target.

    Source: Introduction · p. 3, §3.3 · p. 7

  • Results

    DART delivers consistent gains over comparable baselines across segmentation, depth estimation, classification, and recognition, while maintaining strong image-level performance.

    Source: Introduction · p. 3, §5 · p. 14

  • Takeaways & Limitations

    The resulting model uses RGB at fine-tuning and inference, avoiding a downstream requirement for depth maps.

    Source: Introduction · p. 3

    Monocular depth estimation is ill-posed, and a minority of pseudo-depth samples were inaccurate in unclean, blurred, or bloody scenes.

    Source: S1.1 LEMON and RGB-D Dataset Generation · p. 21

Abstract

from arXiv · show

Vision foundation models (VFMs) are valuable in data-scarce domains such as surgery, where a single pretrained backbone can provide rich representations for many downstream tasks. Yet the dominant self-supervised pretraining paradigm uses only RGB images, leaving readily available complementary signals, such as depth maps, unused. This is a particular missed opportunity in surgery, where natural-image VFMs transfer poorly while the scene geometry is rich and informative. With strong off-the-shelf models now able to produce pseudo-labeled dense depth for any image corpus, we hypothesize that such signals can be folded into pretraining to learn better representations. We present DART, an RGB-D pretraining recipe that builds on DINOv2 with a simple modification: a pixel-space depth reconstruction objective applied to masked iBOT patches, supervised by pseudo-labeled depth. Depth is used only during pretraining, so fine-tuning and inference remain RGB-only. We find that this pixel-level reconstruction head improves representation quality rather than disrupting it. We further show that depth, which encodes scene geometry, is more effective as a target than alternative dense signals such as Canny edges, confirming that the gains stem from depth rather than added supervision alone. Across eight surgical benchmarks spanning segmentation, depth estimation, and image-level recognition, DART outperforms both natural-image and in-domain baselines, including a vanilla DINOv2 trained on identical data, improving dense prediction while also strengthening image-level understanding. More broadly, DART shows that freely available geometric pseudo-labels can strengthen foundation model pretraining without extra labels or added inference cost, pointing toward stronger backbones for surgery.

Introduction

DART adds pseudo-depth supervision to DINOv2 pretraining to address weak transfer of off-the-shelf models and the limits of prior RGB-D approaches in surgery. Across surgical benchmarks, it improves dense prediction while maintaining strong image-level performance, and depth outperforms alternative auxiliary targets.

Introduction

  • Surgical systems need visual understanding that captures image semantics, fine-grained detail, and spatial structure across tasks including segmentation, tracking, and skill assessment.A generalist backbone is especially valuable because surgical data is costly to label and requires domain expertise.Source: Introduction · p. 2
  • Off-the-shelf DINOv2 and DINOv3 transfer poorly to surgical tasks, while prior MAE-family depth-input methods are impractical without depth at inference and underperform in the surgical setting.Source: Introduction · p. 2
  • DART adds a lightweight head that reconstructs pseudo-labeled depth from masked iBOT patch tokens while retaining DINOv2’s other objectives and requiring only RGB downstream.Depth is a supervisory target, not an encoder input, and is not required at fine-tuning or inference.Source: Introduction · p. 3
  • Across five segmentation, one depth-estimation, and two image-level benchmarks, DART outperforms baselines on dense prediction while maintaining strong image-level performance.It also outperforms other models trained on the same LEMON data and publicly released checkpoints.Source: Introduction · p. 3
  • Depth maps generally lead to better results than alternative auxiliary targets such as Canny edges.Source: Introduction · p. 3

2 Related Work

Surgical vision foundation models draw on varied self-supervised recipes, while DART builds on DINOv2 and adds geometric depth supervision. Prior RGB-D pretraining often makes depth an inference input; DART instead uses depth as a target during pretraining.

2.1 Surgical Vision Foundation Models

  • Surgical foundation-model work spans masked-image and video modeling, next-frame prediction, and language-aligned pretraining, with models adapted to tasks such as segmentation and depth estimation.Examples include EndoViT, EndoFM, EndoFM-LV, GSViT, SurgeNetXL, and SurgFM.Source: §2.1 · p. 3
  • DART differs from related surgical SSL methods by building on DINOv2 rather than MAE or DINO and adding auxiliary depth supervision to use surgical-scene geometry.Its figure shows a depth reconstruction head applied to masked student patch features within the DINOv2 recipe.Source: §2.1 · p. 4, Figure 1 · p. 4

2.2 RGB-D Pretraining

  • Prior RGB-D pretraining commonly combines depth as an encoder input and reconstruction target, so the resulting backbone expects depth at inference—a poor fit for clinical pipelines without depth sensors.MultiMAE accepts and predicts RGB, depth, and semantic segmentation; related approaches also combine contrastive and reconstructive objectives or tokenize many modalities.Source: §2.2 · p. 4
  • The paper reports that no prior work puts depth into the DINOv2 pretraining objective as a target while keeping depth out of the encoder input.Other described approaches instead use depth-only DINO-style pretraining, modify a frozen backbone post hoc, or use pseudo-depth without modifying the backbone’s pretraining objective.Source: §2.2 · pp. 4–5

3 Methods

DART adds a pseudo-depth reconstruction objective to DINOv2 pretraining on surgical RGB images, while keeping downstream inputs RGB-only. Its depth target supervises masked patch features so they recover geometric as well as semantic content.

3 Methods

  • The method builds on DINOv2 [33] with a depth decoder, using pseudo-depth only as pretraining supervision and RGB images as inputs for pretraining and downstream fine-tuning.The encoder is intended to support dense surgical prediction and image-level tasks without depth at inference.Source: §3 · p. 5

3.1 Data Preprocessing

  • The pretraining set samples LEMON surgical video at 1 Hz into 3.4M images, uses standardized pseudo-depth from Depth Anything V2-Large, and has no downstream-dataset overlap.Depth sensors are not currently available for collecting ground-truth depth in surgical scenes; pseudo-depth is discarded at fine-tuning and inference.Source: §3.1 · pp. 5–6

3.2 Preliminaries: DINOv2

  • DINOv2 supplies the unchanged self-distillation and iBOT framework, with multi-view consistency and latent-space objectives rather than pixel-level reconstruction.Its KoLeo regularizer encourages spread among student class-token representations.Source: §3.2 · p. 6

3.3 Depth-Aware Pretraining

  • DART adds a lightweight decoder that predicts pseudo-depth from iBOT-masked student patch features, while depth remains absent from encoder inputs and downstream inference.The same masked positions support semantic distillation through iBOT and geometric reconstruction through the depth objective.Source: §3 · p. 5, §3.3 · pp. 6–7

3.4 Implementation Details

  • The final pretraining recipe uses a 3-layer, 512-hidden-dimension depth decoder and sets the depth-loss weight to 0.1.ViT-B/16 is pretrained from scratch for 187,500 iterations at batch size 1024; downstream backbones are frozen while task-specific heads are trained.Source: §3.4 · p. 7, §3.3 · p. 7

Results

DART’s strongest reported gains are in dense prediction, while its image-level results vary by task. Depth-target and decoder choices also affect performance, and DART’s features appear more spatially coherent than those of matched DINOv2.

4.1 Experimental Design

  • Evaluation freezes each backbone and trains task-specific heads on downstream tasks that are out-of-domain relative to pretraining.A vanilla DINOv2 trained on the same LEMON data with identical parameters provides the controlled comparison for the depth objective.Source: §4.1 · p. 8

4.2 Dense Pixel Tasks

  • +9.2 mIoU on CholecInstanceSeg (+ms.) separates DART from matched RGB-only DINOv2, and DART also surpasses larger LEMON-FM on dense prediction.The matched comparison isolates depth supervision; LEMON-FM uses a ConvNeXt-L backbone with over twice as many parameters.Source: §4.2 · p. 9
  • Despite RGB-D pretraining, MultiMAE trails MAE in most RGB-only settings; DART beats RGB-D MultiMAE in 8 of 10 settings and produces sharper instrument boundaries.The authors also report that natural-image MAE and MultiMAE do not outperform their in-domain counterparts on surgical segmentation.Source: §4.2 · pp. 9–10
  • DART attains δ1 0.653 versus 0.630 for the next-best model on SCARED-C, where DINOv3 is also competitive.SCARED-C uses ex-vivo porcine scenes; the authors suggest this may widen the gap from surgical-video pretraining data.Source: §4.2 · pp. 10–11

4.3 Image-level Tasks

  • On CholecT50 triplet recognition, LEMON-FM leads DART, 7.9 vs 6.9 APivt, while DART exceeds comparable-size baselines without domain-specific sampling.The authors attribute LEMON-FM’s advantage partly to its larger backbone and temporally augmented distillation strategy.Source: §4.3 · pp. 11–12

4.4 Auxiliary Targets

  • Depth is best in 7 of 10 segmentation settings, beating Canny or grayscale by up to 2.5 mIoU points, with two CholecInstanceSeg exceptions.Plain DINOv2 leads in the multiscale setting, while Canny leads under linear probing.Source: §4.4 · pp. 12–13

4.5 Ablations and Sweeps

  • An MLP yields best Cholec80 phase recognition (65.5 F1), while a transformer decoder yields best EndoVis18 segmentation (31.2 mIoU); λdepth=0.25 destabilizes training.The authors adopt the MLP for simplicity and identify 0.1 as a robust depth-loss weight.Source: §4.5 · p. 13

4.6 PCA Visualization

  • DART’s PCA features form more coherent object regions and sharper boundaries than matched DINOv2 features, including around specularities and bright spots.The authors connect these observations to depth reconstruction at masked patch positions.Source: §4.6 · p. 13, Figure 5 · p. 14

5 Conclusion and Future Work

DART adds masked-patch pixel-wise depth reconstruction to surgical DINOv2 pretraining and reports gains across dense and image-level tasks. The authors note limitations in depth quality and the scope of their evaluation.

5 Conclusion and Future Work

  • DART’s masked-iBOT depth reconstruction improves performance on dense-pixel and image-level tasks, with gains over comparable baselines across segmentation, depth estimation, classification, and recognition.The authors also report that depth outperforms alternative spatial targets such as Canny edges and blurred grayscale.Source: §5 · p. 14
  • DART uses RGB at fine-tuning and inference, which the authors say improves practicality for deployment.Source: §5 · p. 14
  • Depth supervision relies on imperfect pseudo-labels from off-the-shelf monocular estimators, and the authors do not assess sensitivity to depth quality or estimator choice.They report gains despite noisy depth targets.Source: §5 · p. 14
  • Evidence is limited to surgical data and ViT-S/ViT-B backbones, leaving effectiveness at larger scales and in other domains open.Source: §5 · p. 14
  • The authors propose scaling to larger backbones and more data, studying robustness to the depth source, and adding modalities such as semantic maps or robot kinematics.They also identify combining latent-space objectives with pixel-space reconstruction as a promising direction, especially in specialized domains where public backbones generalize poorly.Source: §5 · p. 14

S1 Implementation Details

The supplementary material describes pretraining constraints, generation of the LEMON RGB-D data, and downstream evaluation resources and configurations.

S1 Implementation Details

  • Compute constraints prevented matching prior pretraining scale, including MAE’s 1,600 epochs, while the authors report that their undertrained models outperform off-the-shelf models because of the surgical–natural image domain gap.Pretraining and fine-tuning configurations are listed in Tables S1 and S2.Source: S1 Implementation Details · p. 20

S1.1 LEMON and RGB-D Dataset Generation

  • The LEMON dataset contains 938 hours of YouTube surgical video; sampling at 1 Hz yields 3.4M images, with ViT-L Depth Anything V2 used to generate depth maps.The RGB-D pairs are stored in WebDataset shards; depth is retained as inverse depth, with higher values closer to the camera, and physical depth values are not required by the framework.Source: S1.1 LEMON and RGB-D Dataset Generation · pp. 20–21
  • The authors report that raw depth is used in black-border regions and that a minority of samples are inaccurate due to unclean, blurred, or bloody images.They nevertheless report that noisy depth maps remain useful for representation quality.Source: S1.1 LEMON and RGB-D Dataset Generation · p. 21

S1.2 Downstream Task Datasets

S2 ViT-B vs ViT-S Experiments

The appendix reports ViT-S experiments and notes that their batch size differs from the main ViT-B setup.

S2 ViT-B vs ViT-S Experiments

  • ViT-S experiments use batch size 256 rather than 1024, which the authors identify as a reason ViT-B results in the supplementary table differ from the main results.Source: S2 ViT-B vs ViT-S Experiments · p. 22
  • The authors observed that batch size was a major contributor to DINOv2 performance because centering and KoLeo regularization depend on large-minibatch statistics.Source: S2 ViT-B vs ViT-S Experiments · p. 22
  • The appendix compares ViT-S and ViT-B semantic-segmentation results in Table S4.The ViT-S results use frozen features with linear and multiscale heads and are averaged over three seeds.Source: S2 ViT-B vs ViT-S Experiments · p. 22

S3 ViT-S Depth Decoder Sweeps

The authors tuned the depth decoder for ViT-S after observing that the ViT-B decoder size did not perform as expected at the smaller scale.

S3 ViT-S Depth Decoder Sweeps

S4 Full Segmentation Results

The segmentation results report means across three runs; standard deviations omitted from the main results are provided in Tables S6 and S7.

S5 Depth Anything v2 Comparison Study

The comparison finds DAv2 competitive on segmentation and depth estimation, but DART remains superior to it on both task types.

  • The DAv2 comparison uses a frozen ViT-B backbone with its depth head discarded and task-specific heads trained under the original configuration.Source: S5 Depth Anything v2 Comparison Study · pp. 22–23
  • Table S5 describes a depth-head hidden-dimension sweep using frozen-feature mIoU on four surgical segmentation benchmarks.Source: Table S5
  • Tables S6 and S7 report frozen-feature linear and multiscale segmentation results as mean ± standard deviation over three seeds.Table S8 covers frozen Depth Anything v2 ViT-B on segmentation and depth estimation, using H1 linear and H2 multiscale segmentation or DPT depth heads.Source: Table S6 · p. 23, Table S7 · p. 24, Table S8 · p. 24
  • DART outperforms DAv2 on segmentation, while DAv2 is competitive and exceeds ImageNet-trained MAE and MultiMAE.Off-the-shelf DINOv2 and DINOv3 slightly outperform DAv2, while surgical-data-trained DINOv2 and DART far outperform it.Source: S5 Depth Anything v2 Comparison Study · p. 23
  • On SCARED-C depth estimation, DAv2 outperforms surgical-data-trained DINOv2, but DART remains superior to DAv2.Source: S5 Depth Anything v2 Comparison Study · p. 23

S6 Additional Qualitative Results

The supplement presents qualitative segmentation and depth-estimation comparisons across the evaluated models, and notes that sparse SCARED-C depth limits performance.

  • The section presents additional qualitative results for segmentation and depth estimation and includes all nine evaluated models.Source: S6 Additional Qualitative Results · p. 24
  • SCARED-C provides sparse depth, and the lack of dense supervision limits the models’ depth-estimation performance ceiling.Source: S6 Additional Qualitative Results · p. 24
  • EndoVis18 segmentation panels compare ground truth with nine pretrained models, including DART trained on LEMON.The panels list the models from MAE (IN-1k) through DART (LEMON).Source: Figure S1 · p. 25
  • CholecSeg8k segmentation panels use the same ground-truth and nine-model lineup as the EndoVis18 comparison.Source: Figure S2 · p. 25
  • SCARED-C depth panels compare RGB and ground truth with the same nine-model lineup.Source: Figure S3 · p. 26

Related papers

Loading 2609.04555v1…