PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth-guided Geometric Adaptation for Few-shot Image Classification

Zhiqi Miao, Estefanía Talavera, Weichuan Zhang, Jiapan Guo

arXiv:2610.28856v1cs.CV

TL;DR

FSIC must recognize novel classes from few labeled examples despite appearance ambiguity and background clutter that complicate patch matching. DAFIC uses monocular depth for RGB-D pre-training and geometric patch adaptation, then matches support and query patches bidirectionally. Across four benchmarks it achieves strong performance, with reported gains over CPEA and SMKD.

  • Problem

    FSIC must classify novel classes from few labeled examples, while background clutter and RGB appearance similarities make reliable patch matching difficult.

    Source: §1 · pp. 1–2

  • Method

    DAFIC combines RGB-D self-supervised pre-training, depth-guided adaptation of patch features with geometry-enriched class context, and non-parametric bidirectional support-query matching.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across four benchmarks, DAFIC achieves strong performance; in 5-way 1-shot MiniImageNet, it improves accuracy by +5.74% over CPEA (77.71% vs. 71.97%).

    Source: §4.2 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    The paper identifies coupling geometry-conditioned episodic adaptation with bidirectional matching as the central contribution, rather than simply combining existing components.

    Source: §5 · p. 8, E. Discussion of DAFIC · p. 19

    The discussion notes that segmentation-based approaches typically require an additional prompt or mask-selection mechanism to identify class-relevant masks for each few-shot task.

    Source: E. Discussion of DAFIC · p. 19

Abstract

from arXiv · show

Few-shot image classification (FSIC) frequently suffers from visual ambiguity, where background clutter and RGB appearance similarities hinder discriminative feature matching in data-scarce settings. To mitigate this, we propose DAFIC, a geometry-guided framework that incorporates monocular depth into representation learning, episodic adaptation, and dense support-query matching. DAFIC first learns geometry-aware token representations through RGB-D self-supervised pre-training. Subsequently, the Depth-guided Geometric Adaptation (DGA) module uses depth-derived geometry to refine patch representations with geometry-enriched class context while suppressing background interference. Spatial Bidirectional Patch Matching (SBiM) then performs bidirectional dense matching on the refined features, reducing sensitivity to spurious correspondences and background clutter. Extensive experiments on four benchmarks show that DAFIC achieves strong performance against existing methods, with ablation and robustness analyses supporting the effectiveness of coupling geometry-conditioned episodic adaptation with bidirectional matching for few-shot visual recognition.

1. Introduction

Few-shot image classification must recognize novel classes from very few labeled examples, where RGB appearance and background clutter make reliable matching difficult. DAFIC introduces monocular depth as a geometric prior for representation learning, adaptation, and support-query matching.

1. Introduction

  • Extreme data scarcity makes it difficult for FSIC models to suppress background clutter and ambiguity that large-scale models can learn to isolate using abundant data.FSIC requires recognition of novel classes disjoint from training classes using only one or a few labeled samples per class.Source: §1 · p. 1
  • RGB-based patch methods lack explicit geometric cues, leaving them vulnerable to appearance shifts, noisy feature aggregation, and misalignment between local patches and global semantics.These effects are amplified in episodic FSIC, resulting in class-inconsistent matching.Source: §1 · p. 1
  • Unidirectional dense matching can repeatedly assign query patches to one salient support patch, while bidirectional matching can penalize unmatched background patches.Global aggregation also discards patch-to-patch correspondence, while flattening dense similarities destroys their 2D layout.Source: §1 · pp. 1–2
  • DAFIC combines RGB-D self-supervised pre-training, depth-guided geometric adaptation, and bidirectional dense matching to refine patch features and reduce sensitivity to background clutter and spurious matches.DGA preserves local cues in salient patches, adds geometry-enriched class context, and steers low-attention regions toward consistent global descriptors.Source: §1 · p. 2

2. Related work

Prior metric-based FSIC work increasingly uses ViTs and patch-level correspondences, but most approaches rely on RGB appearance without explicit geometric guidance. DAFIC brings monocular depth into episodic learning to provide spatial structure for target-focused representations.

2.1. Few-shot Image Classification

  • Metric-based FSIC compares query and support samples in a representation space, and recent frameworks combine pre-training with episodic meta-learning.Related approaches include optimization-based, transfer learning-based, and metric-based paradigms.Source: §2.1 · p. 2
  • Recent ViT-based methods refine patch representations through online optimization, masked distillation, class-aware embeddings, or attention-driven filtering.The approaches discussed include FewTURE, SMKD, CPEA, CPES, SAFF, and SPFF.Source: §2.1 · p. 2
  • Existing patch-refinement methods largely depend on 2D RGB appearance rather than explicit depth guidance, while dense matching approaches include nearest-neighbor aggregation, optimal transport, and cross-attention.DAFIC pairs geometry-aware representation and adaptation with lightweight symmetric patch matching.Source: §2.1 · p. 2

2.2. Geometric Priors and Depth Estimation

  • Depth remains relatively underexplored in FSIC; prior geometric cues include language priors, frequency-domain cues, and depth streams used in few-shot video recognition.The passage notes limitations of language priors, frequency cues, and the additional computational overhead of the video approach.Source: §2.2 · p. 3
  • DAFIC uses depth as an annotation-free structural prior that can be optimized episodically into geometry-salient attention for isolating target entities.The cited structural cues include boundaries, layering, and occlusion.Source: §2.2 · p. 3

3. Method

DAFIC builds geometry-aware token representations from RGB-D inputs, adapts patches using depth-derived attention and class context, then classifies with non-parametric bidirectional patch matching. Episodic training combines local matching and auxiliary global supervision.

3. Method

  • FSIC trains on base classes and evaluates on a disjoint novel-class set using an N-way K-shot support set and query samples.The label spaces are disjoint, and each class contributes K labeled support samples.Source: §3 · p. 3

3.1. Self-supervised Pre-training with Depth Cues

  • RGB-D self-supervised pre-training extends teacher–student self-distillation and masked image modeling with depth cues to learn geometry-aware token representations.Spatial transformations are shared across RGB and depth, while appearance transformations are applied only to RGB; the objective combines class-token consistency and masked-patch prediction.Source: §3.1 · p. 3

3.2. Few-shot Learning with DGA Module

  • DGA converts normalized depth maps into patch attention, modulates patch embeddings, and adds geometry-enriched class context to refine local-to-global alignment.Its depth-guided attention emphasizes geometry-salient regions and suppresses background distractors; geometry-aware adaptation preserves salient local cues while steering low-attention patches toward a consistent global representation.Source: §3.2 · pp. 4–5

3.3. Few-shot Classification via Bidirectional Patch Matching

  • SBiM retains patch-level correspondence by matching each query patch to its nearest support patch and each support patch to its nearest query patch, then fusing the two scores.The symmetric score captures complementary patch coverage and helps mitigate spurious one-way correspondences and many-to-one matching issues.Source: §3.3 · p. 5
  • An auxiliary global similarity objective is jointly optimized with the local matching loss, while inference uses the fused dense-matching score for prediction.The global signal is derived from spatially pooled refined patch representations.Source: §3.3 · p. 5

4. Experiments

Across four FSIC benchmarks, DAFIC performs strongly against recent methods, with especially large gains in 1-shot settings. RGB-D pre-training, geometric adaptation, and bidirectional matching each contribute to the framework’s results, which remain robust under evaluated depth corruptions.

4.1. Experimental Settings

  • Evaluation uses 1,000 random 5-way episodes with 15 queries per class across MiniImageNet, TieredImageNet, CIFAR-FS, and FC100.The experiments follow established class splits; depth maps are generated with DepthAnything V2-base and cached.Source: §4.1 · pp. 6–7

4.2. Comparison results

  • DAFIC exceeds SMKD by +3.43%/+4.22% in 1-shot and +1.53%/+1.90% in 5-shot on MiniImageNet/CIFAR-FS, and improves over CPEA by +5.74%/+6.48% in 1-shot on MiniImageNet/CIFAR-FS.The reported comparisons cover four benchmark datasets and show particularly strong gains in the data-scarce 1-shot setting.Source: §4.2 · p. 7
  • After adaptation, DAFIC better localizes class-relevant regions and suppresses background noise than the baseline; its RGB-D-pretrained raw patches already show stronger class-aware focus.Figure 3 compares patch-to-class alignment, with red for higher class relevance and blue for lower relevance.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3. Ablation Study

  • RGB-D pre-training gains +3.25% and +2.48% over RGB-only pre-training on 1-shot and 5-shot tasks, respectively, using a simple prototypical classifier.Figure 4 compares average MiniImageNet test accuracy under 5-way 1/5-shot settings.Source: §4.3 · p. 7, Figure 4 · p. 7
  • On MiniImageNet, RGB-D pre-training raises 1-shot accuracy from 69.12% to 73.85%; adding DA, GA, SBiM, and Global Supervision yields further gains of +0.79%, +2.05%, +0.73%, and +0.29%.Across benchmarks, the DGA+SBiM variant improves 1-shot results by +3.69% on MiniImageNet and +5.25% on CIFAR-FS over the baseline.Source: Table 4 · p. 7, §4.3 · p. 8
  • On CIFAR-FS, DAFIC outperforms the baseline by 6.03% in 1-shot and 3.23% in 5-shot with comparable cached-depth latency; it also uses fewer FLOPs and lower latency than SMKD.The reported latency uses cached depth maps; separate supplementary results cover on-the-fly depth estimation.Source: §4.3 · p. 8, Table 5 · p. 8
  • On MiniImageNet, DAFIC retains 73.06% 1-shot accuracy under object-region occlusion and 74.53% with low-quality depth, exceeding CPEA by 0.81% and 2.28%, respectively.Background masking causes only a 0.61% drop in 5-way 1-shot performance.Source: §4.3 · p. 8, Table 6 · p. 8, Figure 5 · p. 8

5. Conclusion

DAFIC integrates monocular depth into representation learning, episodic adaptation, and dense support-query matching. Experiments on four benchmarks show strong performance against existing methods.

5. Conclusion

  • DAFIC combines RGB-D self-supervised pre-training, Depth-guided Geometric Adaptation, and Spatial Bidirectional Patch Matching to mitigate appearance ambiguity and background clutter in FSIC.DGA refines geometry-salient patches with geometry-enriched class context, while SBiM performs bidirectional matching on the refined features.Source: §5 · p. 8

Supplementary Material

The supplementary material covers datasets, RGB-D pre-training, additional few-shot results and ablations, discussion and future work, and integration of depth cues into existing FSIC methods.

Supplementary Material

  • The supplement includes further evaluations and ablations, discusses future work, and examines depth-prior integration across existing FSIC methods.Its sections also recap datasets and detail RGB-D pre-training.Source: Supplementary Material · p. 13

A. Dataset

The experiments use four standard FSIC benchmarks. The supplement also documents the RGB-D pre-training configuration, including its architecture, augmentation, and optimization settings.

A. Dataset

  • The four benchmarks are MiniImageNet, TieredImageNet, CIFAR-FS, and FC100.The study follows prior FSIC works on these standard datasets.Source: A. Dataset · p. 13, Table 7 · p. 13

B.1. RGB-D Pre-training Setups

  • RGB-D pre-training uses a ViT-Small backbone with 16×16 patches and a shared 8192-dimensional projection head for class and patch tokens.The student and teacher networks use the shared projection head.Source: §B.1 · p. 13
  • Training applies synchronized RGB-depth multi-crop augmentation and block-wise masked image modeling, with 1,600 epochs for MiniImageNet, CIFAR-FS, and FC100 and 800 for TieredImageNet.The downstream checkpoints are selected at epochs 1,000 and 500, respectively.Source: §B.1 · p. 13

B.2. Pre-training Comparison Analysis

RGB-D self-supervised pre-training converges faster and improves several training metrics over RGB-only pre-training, though its clustering metrics are generally lower. The RGB-D checkpoints are then used for downstream few-shot classification.

Pre-training setup

  • The study uses checkpoints from 1,000 epochs on most datasets and 500 epochs on TieredImageNet for downstream few-shot classification.Pre-training runs for 1,600 epochs on most datasets and 800 epochs on TieredImageNet, with a smaller total batch size of 384 rather than 512 due to computational constraints.Source: §B.2 · pp. 14–15

Pre-training comparison

  • RGB-D pre-training converges faster and yields lower total and class-level losses and higher training accuracy across four benchmarks.The comparison tracks six training-dynamics metrics; RGB-D clustering metrics are generally lower.Source: §B.2 · pp. 13–14, Figure 6 · p. 14
  • RGB-D pre-training is generally competitive on patch-level loss, but its advantage is less consistent on TieredImageNet.Its NMI and F-score are generally lower than those of RGB-only pre-training.Source: §B.2 · p. 14, Figure 6 · p. 14

C. Few-shot Learning with DAFIC

DAFIC is meta-trained with its DGA adaptation module and SBiM classifier. It outperforms the baseline and performs strongly against prior methods across datasets and both evaluated shot settings, while visualizations show more reliable attention to class-relevant regions.

Training protocol

  • The full DAFIC framework, including DGA and SBiM, is trained under a meta-training paradigm for cross-task generalization.Both training and inference use a single NVIDIA V100 GPU.Source: C. Few-shot Learning with DAFIC · p. 15

Qualitative Visualizations Compared to the Baseline

  • DAFIC focuses more reliably on class-relevant regions than the baseline in scenes with distracting backgrounds, multiple entities, and clutter.The visualizations indicate improved patch-to-class alignment over the baseline, with further improvement after applying the full framework.Source: C. Few-shot Learning with DAFIC · p. 15, Figure 7 · p. 15

Quantitative Comparison with Prior FSIC Methods

D. Additional Ablation Studies of DAFIC

The additional studies examine DAFIC’s attention design, matching strategy, component contributions, training stability, latency, and sensitivity to depth-prior quality. They support the selected CNN-based attention and bidirectional matching designs, while showing that weaker depth priors reduce—but do not eliminate—the reported advantage over CPEA on MiniImageNet.

Choices of Depth-guided Attention (DA)

  • The CNN-based depth-guided attention design performs best in both RGB and RGB-D settings on TieredImageNet 5-way 1-shot.The authors attribute its advantage to integrating local and global geometric cues, unlike fixed patch-average weights or patch-restricted MLP predictions.Source: D. Additional Ablation Studies of DAFIC · p. 17, Table 11 · p. 17

Importance of SBiM

Ablations of β and λ

Ablation of α and γ for the final loss

Independent contributions of DGA and SBiM

Variance over multiple training runs

Latency comparison

Sensitivity to depth-prior quality

  • On MiniImageNet, a weaker depth prior lowers DAFIC from 77.71/90.35% to 76.16/89.05% for 5-way 1-/5-shot, yet remains 3.91/1.81% above CPEA.The alternative prior is used throughout pre-training, meta-training, and evaluation; the authors conclude depth quality affects performance but DAFIC remains effective with a weaker prior.Source: D. Additional Ablation Studies of DAFIC · p. 18, Table 19 · p. 18, Figure 8 · p. 18

E. Discussion of DAFIC

DAFIC’s novelty is its unified geometry-guided formulation, which integrates geometry across representation learning, episodic adaptation, and dense matching. The discussion contrasts its soft, task-agnostic depth prior with segmentation-based approaches and externally pretrained semantic representations.

Innovation beyond existing patch-level FSIC methods

  • DAFIC’s contribution is the coupling of geometry-conditioned episodic adaptation with bidirectional matching, rather than simply adding depth or combining existing components.Its pipeline integrates geometry into representation learning, episodic patch adaptation, and dense support-query matching to address appearance ambiguity and background clutter.Source: E. Discussion of DAFIC · pp. 18–19

Geometry-conditioned adaptation and matching

  • DGA uses depth to preserve local cues for geometry-salient patches, incorporate geometry-enriched class context, and steer background regions toward global representations.SBiM then performs bidirectional matching on the refined features to reduce spurious correspondences.Source: E. Discussion of DAFIC · p. 19

Segmentation methods for future work

  • Segmentation is a future direction, but may require task-specific prompt or mask selection and does not by itself resolve few-shot matching ambiguities.Hard masks may also remove informative object parts, shape cues, or surrounding regions; the authors describe depth as a softer, task-agnostic alternative.Source: E. Discussion of DAFIC · p. 19

Foundation models and the strict FSIC protocol

  • Frozen DINOv2 and CLIP representations provide a diagnostic reference for the few-shot capability of large-scale external semantic pre-training.The paper distinguishes these externally pretrained representations from DAFIC’s learnable backbone, trained only on the standard meta-train split with fixed, task-agnostic depth predictions as auxiliary inputs.Source: E. Discussion of DAFIC · p. 19

F. Effect of Depth Cues on FSIC Methods

Matched-prior controls show that estimated depth benefits many FSIC methods, while DAFIC remains above the directly matched CPEA+Depth baseline across the reported benchmarks. The comparison indicates that adding depth alone does not explain DAFIC’s full performance gain.

F. Effect of Depth Cues on FSIC Methods

Loading 2610.28856v1…