PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth-Guided Contrastive Learning for 2D Representations with 3D Spatial Awareness

Liang Zeng, Maarten Vergauwen

arXiv:2609.28159v1cs.CV

TL;DR

Pixel-level contrastive learning needs meaningful local similarity, and semantic similarity alone does not capture 3D structure. DGCL uses depth-based relative 3D proximity as auxiliary supervision for 2D representations. The authors report consistent transfer improvements across frameworks, datasets, and downstream tasks.

  • Problem

    Defining fine-grained pixel similarity for dense prediction remains challenging, while existing pixel relationships may not correspond to physical structure.

    Source: §1 · p. 1

  • Method

    DGCL adds an auxiliary objective that compares randomly sampled pixels’ relative 3D distances and treats the nearest anchor as positive for each query.

    Source: §1 · pp. 1–2

  • Results

    DGCL consistently improves transfer performance across contrastive frameworks, pretraining datasets, and downstream tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Depth-derived geometric relationships can complement semantic contrastive learning and benefit semantic downstream tasks.

    Source: §1 · p. 1, §7 · p. 8

    Performance drops mainly for small or thin objects and small objects in cluttered contexts, which may be sensitive to depth errors and downsampling.

    Source: §4 · p. 14

Abstract

from arXiv · show

Standard contrastive learning frameworks are mainly designed from a semantic perspective, yet learning 2D visual representations that preserve 3D spatial structure is also important for scene understanding. In this work, we propose Depth-Guided Contrastive Learning (DGCL), a simple auxiliary objective that injects 3D spatial awareness into 2D contrastive representation learning. Our key idea is to use depth to convert local 3D proximity into contrastive similarity: pixels that are closer in 3D space are encouraged to have more similar representations than pixels that are farther apart. Instead of relying on absolute depth values, DGCL formulates supervision through relative 3D distance comparisons among randomly sampled pixels, making the objective invariant to depth scale, efficient to compute, and easy to integrate into existing contrastive frameworks. Experiments across different datasets and models show that DGCL consistently improves 2D representation learning and benefits semantic downstream tasks by stronger spatial and geometric understanding. The code is available on https://github.com/LeungTsang/DGCL.

1. Introduction

Pixel-level contrastive learning can address dense tasks’ need for local relationships, but defining fine-grained pixel similarity remains a challenge. DGCL uses relative 3D proximity from depth as complementary geometric supervision for 2D features.

1. Introduction

  • Pixel-level contrastive learning suits dense prediction better than image-level pairs, but defining meaningful fine-grained similarity between pixels remains difficult.Traditional image-level contrastive learning uses augmentations of one image as positives and other images as negatives; pixel-level methods instead contrast local features.Source: §1 · p. 1
  • DGCL assumes nearby 3D pixels are often physically or structurally related, not that they share a semantic label.The authors argue that encouraging correlated representations for such pixels can support local scene organization and semantic downstream tasks.Source: §1 · p. 1
  • DGCL introduces an auxiliary pixel-level contrastive task that selects the 3D-nearest sampled anchor as positive and treats other anchors as negatives.It uses relative 3D distance comparisons to encourage 2D features to reflect local 3D structure.Source: Figure 1 · p. 2, §1 · p. 2
  • Relative distance comparisons make DGCL robust to depth scale and allow it to use depth from sensors, synthetic environments, stereo, or monocular estimators.It directly supervises RGB features without a separate depth encoder and can be added to existing contrastive frameworks with minimal architectural modification.Source: §1 · p. 2
  • The authors report that DGCL consistently improves transfer performance across contrastive frameworks, pretraining datasets, and downstream tasks.The supplied introduction states this broad result without giving specific benchmark values.Source: §1 · p. 2

2. Related Work

Prior work extends contrastive learning beyond images, while local similarity remains a key design choice. DGCL uses relative depth-based proximity as a lightweight geometric alternative to semantic grouping, RGB feature similarity, and other 3D-guided approaches.

2.1. Contrastive learning

  • Dense contrastive learning makes local similarity a key design choice; existing approaches use learned features or external priors such as proposals and segmentation masks.External priors may require extra models, preprocessing, task-specific assumptions, or supervised annotations.Source: §2.1 · pp. 2–3
  • DGCL differs from prior pixel-level methods by deriving geometric similarity from depth-based 3D proximity rather than semantic grouping or RGB feature similarity.Its lightweight auxiliary objective can be combined with existing contrastive frameworks.Source: §2.1 · p. 3

2.2. 2D representation learning with 3D information

  • 3D-guided methods use structured data, multimodal alignment, or geometric reconstruction, but may require constrained data or additional networks and often learn 3D awareness implicitly.Examples include multi-view correspondences, RGB-depth alignment, and joint geometric reconstruction.Source: §2.2 · p. 3
  • PixDepth [46] uses depth to filter positive pairs in PixPro [58], whereas DGCL is a standalone objective that selects positives by relative 3D proximity.DGCL does not threshold absolute depth differences or align RGB features with a depth encoder.Source: §2.2 · p. 3
  • DGCL requires paired RGB images and depth maps, not camera poses, multi-view sequences, point clouds, or reconstructed scenes.It uses relative rather than absolute metric depth and directly supervises 2D visual features with geometric relationships.Source: §2.2 · p. 3

3. Methodology

DGCL back-projects depth into approximate 3D coordinates, then uses relative distances among sampled pixels to provide geometric contrastive supervision. Repeated sampling gives closer pixels a higher chance of becoming positives, while the method avoids dependence on absolute depth scale.

3.1. Back-Projection from Depth to 3D

  • Approximate 3D coordinates from depth provide the geometric cue for comparing sampled pixels, so coarse reconstruction suffices even with estimated or scale-ambiguous depth.The method uses simplified camera intrinsics when accurate intrinsics are unavailable; the coordinates are not intended as precise metric scene reconstruction.Source: §3.1 · p. 3

3.2. From 3D Proximity to Contrastive Similarity

  • DGCL selects the nearest sampled anchor as each query’s positive, treats the remaining anchors as negatives, and jointly optimizes its loss with the base contrastive objective.The overall depth-guided loss uses all pixels as queries and averages across sampled anchor groups.Source: §3.2 · p. 4
  • Relative distance comparisons avoid absolute-depth thresholds, so multiplying all 3D coordinates by a positive scale factor leaves the selected positive unchanged.This supports using sensor, synthetic, or monocularly estimated depth.Source: §3.2 · p. 4

3.3. Distribution of positive samples

  • Repeated random sampling gives closer pixels a higher chance of becoming positives, and the number of anchors K controls the distribution’s concentration around the query.Monte Carlo sampling is used in practice to avoid computing pairwise distances between all valid pixels.Source: §3.3 · p. 4, Figure 2 · p. 4

4. Experiments

Experiments test DGCL across contrastive frameworks, pretraining datasets, and backbones. DGCL consistently improves transfer performance, including with scene-based pretraining and ViT backbones.

4. Experiments

  • The experiments integrate DGCL into MoCo v2, MoCo v3, and SlotCon, using a pixel-level objective alongside the base methods.Reported results average three independent runs; DGCL uses six anchors, 64 repeated samplings, and temperature 0.1, with MiDaS Swin2L-384 depth.Source: §4 · p. 5

4.1. Pre-training on object-centric ImageNet

  • DGCL consistently improves MoCo v2 and SlotCon transfer after ImageNet pretraining, with gains on detection, instance segmentation, keypoints, and semantic segmentation.DGCL-S, pretrained for 200 epochs, reaches comparable or better performance than 1000-epoch DetCon on most metrics.Source: §4.1 · p. 5, Table 1 · p. 6

4.2. Pre-training on scene-based COCO

  • DGCL improves both MoCo v2 and SlotCon after COCO pretraining, and DGCL-S achieves the best performance across all reported metrics.Gains are larger and more consistent than with ImageNet pretraining, suggesting particular benefit from scene-centric images with richer spatial layout.Source: §4.2 · p. 5, Table 2 · p. 6

4.3. Pre-training Vision Transformers

  • DGCL-M3 consistently improves MoCo v3 on ViT-S/16 transfer tasks, including COCO instance segmentation and Pascal VOC and ADE20K semantic segmentation.DGCL-M3 outperforms DINO on Pascal VOC but remains below it on COCO instance segmentation and ADE20K; the stated main comparison is with MoCo v3 under the same contrastive framework.Source: §4.3 · pp. 5–6, Table 3 · p. 6

5. Ablation and Analysis

The analyses compare DGCL with other ways to use depth and segmentation masks, then test its sensitivity to depth quality. DGCL performs best among the compared guidance methods, while coarse estimated depth is already useful.

5.1. Comparison with other methods that use depth

  • All tested depth-based objectives improve over MoCo v2, but DGCL achieves the strongest transfer performance without a depth encoder or regression decoder.The comparison includes depth regression and RGB-depth embedding alignment (CMC).Source: §5.1 · p. 7, Table 4 · p. 7

5.2. Comparison with segmentation mask guidance

  • Mask guidance improves MoCo v2, stronger masks improve transfer, and DGCL still achieves the best performance.Mask-based objectives group pixels within regions; DGCL instead uses relative 3D proximity, producing smoother maps and relationships not limited to object boundaries.Source: §5.2 · p. 7, Table 5 · p. 7, Figure 4 · p. 7

5.3. Sensitivity to Depth Quality

  • Constant depth yields only a small improvement, while estimated depth substantially improves the baseline; even MiDaS Small256 raises APb by 2.2.Higher-quality depth generally helps, but gains saturate quickly: stronger MiDaS models reach APb around 39.0 to 39.4 with relatively small differences. Coarse depth can suffice because feature-map downsampling removes fine detail.Source: §5.3 · p. 8, Table 6 · p. 8

6. Visualizations

The visualizations compare dense feature similarities around a query pixel. DGCL maps concentrate more around geometrically and semantically related regions than the contrastive baselines shown.

6. Visualizations

  • DGCL-M2 and DGCL-S produce similarity maps more concentrated around geometrically and semantically related regions than PixPro, MoCo v2, and SlotCon.PixPro maps can spread across 2D-near but structurally less-related areas; SlotCon grouping can be overly coarse or misaligned with object parts.Source: §6 · p. 8, Figure 6 · p. 8

7. Conclusions

The conclusion presents DGCL as a relative-depth auxiliary objective for adding 3D awareness to 2D contrastive learning. Experiments show improved downstream transfer, while future work targets fine-grained depth cues and higher-level scene relationships.

7. Conclusions

  • DGCL uses relative 3D distances among randomly sampled pixels to guide 2D contrastive representations, avoiding dependence on absolute depth scale.The conclusion reports transfer improvements across datasets and models and identifies finer depth cues and higher-level 3D scene relationships as future directions.Source: §7 · p. 8

1. Hyperparameter Analysis

DGCL performs best with about 8 anchors and gains transfer performance as anchor groups increase, reaching a plateau around 64 groups with modest overhead.

1.1. Number of anchors K

  • Around K = 8 gives the best performance, with results relatively stable across pretraining datasets and downstream tasks of different semantic granularity.Very small K makes nearest-anchor prediction less informative; very large K can make it too easy.Source: §1.1 · p. 13, Figure 7 · p. 13

1.2. Number of anchor groups N

  • N = 64 reaches the transfer-performance plateau while increasing iteration time by only about 10%.Memory use is almost unnoticeable for N below 256, and N = 64 is used as the default setting.Source: §1.2 · p. 13, Figure 8 · p. 13

2. Convergence

DGCL improves convergence for both MoCo v2 and SlotCon, with the clearest advantage early in pretraining.

2. Convergence

  • DGCL achieves higher COCO detection AP and VOC mIoU than the corresponding baselines at the same pretraining epochs, especially early in training.The authors report this faster convergence as useful when the pretraining budget is limited.Source: §2 · p. 13, Figure 9 · p. 13

3. Standard Deviations

Three-run results show stable gains, and DGCL improves most COCO instance-segmentation categories across object sizes, with some exceptions among small or thin objects.

3. Standard Deviations

  • Small standard deviations across three independent runs support the stability of the main downstream results.Tables 7 and 8 report standard deviations for models pretrained on ImageNet and COCO, respectively.Source: §3 · p. 14, Table 7 · p. 14, Table 8 · p. 14

4. Per-Category Performance Gain

  • DGCL improves most COCO instance-segmentation categories across small and large objects, while drops mainly occur for small, thin, or cluttered objects.The authors suggest depth errors and downsampling may mix object pixels with nearby background or supporting surfaces, so local 3D proximity can conflict with instance boundaries.Source: §4 · p. 14, Figure 10 · p. 14

Related papers

Loading 2609.28159v1…