PaperlayerView exact revision on arXiv

Source-linked AI summary

Teaching DINOv3 About Partial 3D Geometry: A Self-Supervised Geometry-Aware Approach

Viktoria Ehm, Dongliang Cao, Riccardo Marin, Daniel Scholz, Weikang Wang, Florian Bernard, Daniel Cremers

arXiv:2610.21802v1cs.CV

TL;DR

Partial observations make dense correspondence between deformable 3D shapes difficult, while image-foundation features lack 3D grounding. GeoLoRA refines 2D foundation features using self-supervised learning on synthetic full and partial shapes, and the authors report improved matching and left-right disambiguation.

  • Problem

    Dense correspondence between deformable 3D shapes remains an open problem, especially when occlusions, acquisition noise, or reconstruction errors leave one or both shapes partial; image-foundation models also lack 3D grounding.

    Source: §1 · pp. 1–2

  • Method

    GeoLoRA trains a LoRA module on synthetic full and partial shape pairs, aligning their features with a geodesic-based contrastive loss.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    GeoLoRA features outperform handcrafted and 2D foundation alternatives across datasets and partiality classes, improve downstream partial-to-full and partial-to-partial matching, and strengthen left-right disambiguation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors suggest partial-to-full matching can serve as an auxiliary task for improving foundation models’ geometrical understanding.

    Source: §1 · p. 2

    Training datasets primarily contain upright shapes, and failure cases often involve rotated or side-lying poses.

    Source: §13 · p. 13

Abstract

from arXiv · show

Partial shape matching is a crucial yet underexplored problem in 3D vision, with significant relevance to real-world scenarios where shapes are often only partially observed. Existing feature descriptors face difficulties in this setting, as traditional representations either struggle with the boundaries of partial shapes or heavily depend on the shape's spatial position. While existing approaches have employed DINO features for partial shape matching, these features are not inherently suited for handling partial observations. In this work, we propose a method to refine DINO features using LoRA-based self-supervised learning, enabling the generation of feature descriptors that are robust to partiality. Our features substantially improve performance on partial shape matching compared to traditional or vision foundation features. Additionally, when integrated into existing partial shape matching pipelines, we achieve state-of-the-art results on partial shape matching and left-right prediction benchmarks.

1. Introduction

Dense correspondence between deformable 3D shapes remains difficult, especially with partial observations; image-foundation features also lack 3D grounding. GeoLoRA addresses these gaps by injecting geometry into 2D foundation features and reports stronger partial-shape matching results.

1. Introduction

  • Dense correspondence for deformable shapes is still open, and partial observations commonly arise from occlusions, acquisition noise, or reconstruction errors.Such correspondences support downstream information transfer and learning statistical models.Source: §1 · p. 1
  • GeoLoRA injects 3D awareness into 2D foundation features using self-supervised learning on synthetic full and partial shape pairs.The method uses known correspondences between each generated partial shape and its source full shape to supervise a lightweight LoRA module with a geodesic penalty.Source: §1 · p. 2
  • GeoLoRA features outperform handcrafted and 2D foundation alternatives across partiality classes and datasets, in partial-to-full and partial-to-partial matching, and improve downstream matching and left-right disambiguation.The authors also report that the features improve state-of-the-art performance in downstream matching pipelines.Source: §1 · p. 2

2. Related Work

  • The related-work overview covers partial shape matching, shape-matching features, and self-supervised learning approaches.Source: §2 · p. 2

2.1. Partial Shape Matching

  • Partial-to-partial matching must find correspondences and overlapping regions when both shapes have missing parts that may differ; prior approaches include learned and integer-program methods.DPFM predicts overlap and correspondences, while Echo-Match improves its overlap predictor and uses 2D image-foundation features.Source: §2.1 · p. 2

2.2. Features for Shape Matching

  • Handcrafted descriptors can be fragile around missing parts and boundaries, while extrinsic coordinates can bias matching toward aligned shapes; foundation features offer semantic cues but retain geometric ambiguity.Prior work has used image flipping or optimal transport, whereas incorporating 3D information has been less explored.Source: §2.2 · p. 3

2.3. Self-Supervised Learning for 3D Shapes

  • Prior self-supervised 3D-shape methods address static scenes, rigid-scene pretraining, or point-cloud matching, but not large-scale partial non-rigid matching with vision foundation models.Source: §2.3 · p. 3

3. Method

GeoLoRA learns geometry-aware foundation features by training a LoRA module on synthetic full and partial shape pairs. Its training pipeline renders multiple views, backprojects image features to vertices, and uses a geodesic-aware contrastive loss to align corresponding features.

3. Method

  • GeoLoRA generates partial shapes from full ones and trains a LoRA module to align their features with a geodesic-based contrastive loss.Source: §3 · p. 3

3.1. Self-Supervised Training

  • Partial shapes are generated by cutting with a random plane, removing random holes, or simulating a partial camera observation; each is a subset of its full shape with known correspondences.Source: §3.1 · p. 3
  • The method keeps DINOv3 frozen, adds a shared-weight LoRA module, renders multiple views, and averages backprojected image features into vertex-wise descriptors.The training setup uses shapes in the same pose; the authors describe this as matching the goal of recovering a partial shape’s semantics as part of a full geometry.Source: §3.1 · pp. 3–4
  • Figure 2’s pipeline renders 9 views of each full shape and 4 of each partial shape before backprojecting and averaging features, then applies the geodesic-aware contrastive loss.Source: Figure 2 · p. 4
  • The loss pulls corresponding full and partial vertices together and weights mismatches by their geodesic distance on the full shape.Partial vertices are indexed through their known correspondences on the full shape; the geodesic distance matrix is shifted by 0.5 for the weighting.Source: §3.1 · p. 4
  • Geodesic cues distinguish ambiguous geometry, including symmetric points far apart on the surface, and the authors report gains over vanilla PointInfoNCE in left-right disambiguation and geodesic error.Source: §3.1 · p. 5

4. Experiments

The experiments evaluate GeoLoRA features on partial-shape benchmarks, compare them with alternative descriptors, and test their use in matching pipelines. They also assess design choices and applications, including left-right prediction and a real-world partial scan.

4.1. Implementation Details

  • Training uses a DINOv3 ViT-B/16 backbone with LoRA in all attention layers, while inference uses 49 views.Training uses 9 views per full shape and 4 per partial shape, with two partial observations per full shape and 50,000 iterations.Source: §4.1 · p. 5

4.2. Datasets

  • The evaluation spans SHREC16 CUTS and HOLES, PFAUST, CP2P24, PSMAL, and BECOS, with train/test splits described for several datasets.BECOS includes full-to-full, partial-to-full, and partial-to-partial instances; the study also tests non-isometric correspondence on PSMAL.Source: §4.2 · p. 5

4.3. Metrics

  • Evaluation uses mean geodesic error, overlap-region IoU for partial-to-partial matching, and left-right chirality accuracy.Geodesic error is scaled by full-shape diameter; partial-to-partial error is measured on the overlapping region.Source: §4.3 · p. 5

4.4. Matching with Raw Features

  • GeoLoRA improves feature quality over alternatives across partial datasets, particularly on BECOS, with better left-right disambiguation than DINOv3.The comparison reports mean geodesic error; Table 1 covers partial datasets and identifies BECOS* as BECOS without SHREC16 data.Source: §4.4 · pp. 5–6, Table 1 · p. 6

4.5. Integration with Matching Methods

  • GeoLoRA features improve partial-to-full matching over DINOv3 and improve partial-to-partial correspondence and overlap prediction on BECOS and PSMAL.On CP2P24, partial-to-partial results are similar to DINOv3; the authors suggest supervision may prevent the method from benefiting from improved descriptors.Source: §4.5 · pp. 6–7, Table 2 · p. 7, Table 3 · p. 7

4.6. Ablation Studies

  • The view-count ablation finds that 49 views balance runtime and geodesic error, which saturates before runtime stops increasing.The same view count is used for GeoLoRA, DINOv2, and DINOv3.Source: Table 4 · p. 7, §4.6 · p. 7
  • GeoLoRA’s loss formulation significantly reduces BECOS validation geodesic error relative to PointInfoNCE.The loss weights mismatches by geodesic distance rather than penalizing them all equally.Source: §4.6 · p. 7, Table 5 · p. 8

4.7. Applications

  • GeoLoRA improves chirality accuracy on partial BECOS shapes and shows robustness on an illustrative misaligned real-world scan where DINOv3 struggles.The scan maps a partial observation from a depth-sensor body scanner to a full template.Source: Table 6 · p. 8, §4.7 · p. 8, Figure 6 · p. 8

5. Discussion and Conclusion

GeoLoRA uses synthetic data and self-supervised tasks to learn robust features for partial 3D shapes, with stronger geometric understanding and matching performance than DINOv3. Its results remain weaker on upside-down poses than on upright shapes, and future work could extend to clutter and man-made objects.

5. Discussion and Conclusion

  • GeoLoRA learns robust partial-shape features from synthetic data and self-supervised tasks, outperforming DINOv3 on upside-down poses while both approaches perform worse than on upright shapes.The authors identify DINOv3's natural-image pretraining and resulting upright-orientation bias as a likely inherited limitation.Source: §5 · p. 8
  • The authors report GeoLoRA as the best available features for partial-to-partial and partial-to-full matching, with better geometric understanding than 2D foundation backbones.They characterize low-rank adaptation of vision foundation models as a promising step toward bridging image representations and 3D geometry.Source: §5 · p. 8
  • Future work could extend GeoLoRA to external clutter and man-made objects.Source: §5 · p. 8

Summary

The supplementary material collects notation, extended ablations, implementation details, comparisons, failure and bias analyses, and additional examples.

Summary

  • The supplement provides method notation and extended ablations, including runtime comparisons and dataset-wide chirality accuracy.Source: Summary · p. 12
  • It also includes implementation details, comparisons with prior feature representations, failure cases, and upright-bias analysis.Source: Summary · p. 12
  • Additional materials include real-world scans, qualitative visualizations, 2D/3D application examples, and rendering comparisons with and without texture.Source: Summary · p. 12

7. Notation Table

The supplement presents notation and further ablations of LoRA settings, inference time, and backbone scale.

7. Notation Table

8. Further Ablation Studies

  • Rank r = 16 and α = 32 yield the best geodesic-error results on the BECOS validation set.Source: §8 · p. 12, Table 8 · p. 12
  • GeoLoRA, DINOv2, and DINOv3 have similar inference runtimes, while Diff3f takes longer because it also runs Stable Diffusion.Source: Table 9 · p. 12, §8 · p. 12
  • Replacing ViT-B with ViT-L brings no significant geodesic-error improvement and requires 50% more training time and twice the GPU VRAM.Source: §8 · p. 12, Table 10 · p. 12

9. Dataset-wide Chirality Accuracy Definition

The section defines chirality accuracy for a full dataset containing M partial shapes and then introduces a final chirality accuracy.

9. Dataset-wide Chirality Accuracy Definition

  • The definition aggregates chirality accuracy for a full dataset U containing M partial shapes, followed by a final chirality accuracy definition.Source: §9 · p. 12

10. Additional Implementation Details

The supplementary sections report implementation settings, comparisons with original features, and the pose conditions associated with failures. GeoLoRA features outperform XYZ inputs and the compared DINOv2-plus-Stable-Diffusion features on their respective evaluations.

10. Additional Implementation Details

  • Training uses τ = 0.07 and one A40 GPU; the longest BECOS run takes approximately five days for 50,000 iterations.Training duration varies from two to five days across datasets, depending on vertex counts.Source: §10 · pp. 12–13

11. ULRSSM: Comparison to Original Features

  • GeoLoRA features outperform XYZ features on all partial-to-full datasets in the ULRSSM comparison.The comparison reports geodesic error (×100); XYZ inputs are sensitive to shape orientation, which is especially ambiguous for partial shapes.Source: Table 11 · p. 13, §11 · p. 13

12. Left-Right Prediction: Comparison to Original Features

  • GeoLoRA features show higher left-right accuracy than the DINOv2-plus-Stable-Diffusion features used in.The comparison is reported for chirality prediction.Source: Table 12 · p. 13, §12 · p. 13

13. Failure Cases

  • Most failure cases involve poses unlike the primarily upright shapes in the training datasets.Examples include a person in a pike position or lying on their side; the method primarily struggles when shapes are not upright.Source: §13 · p. 13, Figure 7 · p. 13

14. Upright Bias

The supplementary evaluations test GeoLoRA on upside-down shapes, real-world scans, qualitative matching, and additional 2D and 3D tasks. Results include lower geodesic error than DINOv3 on the reported upside-down and annotated real-scan evaluations.

14. Upright Bias

  • 6.69 geodesic error for GeoLoRA versus 63.84 for DINOv3 on upside-down partial shapes.The evaluation assumes upright orientation for consistency with prior work, trains on FAUST with random rotations along all axes, and reports DINOv3 as 10× worse.Source: §14 · p. 13

15. Additional Real-World Experiments

  • 5.92 vs. 16.26 geodesic error: GeoLoRA significantly reduces error compared with DINOv3 on the annotated SHREC20 real scan.Quantitative evaluation on real-world partial scans is limited by the lack of ground-truth data availability; additional statue examples are described as qualitatively better with GeoLoRA.Source: §15 · pp. 13–14

16. Additional Qualitative Results

  • The paper presents qualitative comparisons with other features in Figure 11 and further feature-prediction examples in Figure 12, alongside qualitative partial-to-partial results in Figure 13 and additional partial-to-full results in Figure 14.Source: §16 · p. 14

17. Further 2D/3D Applications

  • GeoLoRA trained on BeCoS provides better SPair-71k image matching than DINOv3 in several cases, especially when foreground elements differ in view, and is also used for 3D keypoint matching.The paper shows SPair-71k image examples and BeCoS test-set keypoint examples.Source: §17 · p. 14, Figure 9 · p. 14, Figure 10 · p. 14

18. Texture & Rendering

The section tests texture sensitivity and presents qualitative matching comparisons. Adding texture degrades the reported SMPL performance, while the qualitative comparisons vary by dataset and shape region.

18. Texture & Rendering

  • 3.65 vs. 1.32 geodesic error: adding texture degrades performance in the SMPL texture-sensitivity test.The benchmarks are textureless, so meshes are rendered without texture, using a single colour with surface shadow.Source: §18 · p. 14
  • GeoLoRA features show improved qualitative performance, especially for left-right and front-back predictions.The feature-quality comparison is shown with colour-coded correspondences.Source: Figure 11 · p. 15
  • GeoLoRA improves qualitative partial-to-partial matching on BECOS and PSMAL, while CP2P24 shows similar quality; partial-to-full examples improve especially at extremities.Shown extremities include a human left arm and horse legs.Source: Figure 13 · p. 17, Figure 14 · p. 18

Related papers

Loading 2610.21802v1…