PaperlayerView exact revision on arXiv

Source-linked AI summary

ReCalMatch:Reliability-Calibrated Semantic Guidance for Semi-Supervised Fine-Grained Recognition

Yundi Hong, Hongyang He, Zheng Fang, Xuanyu Liu, Victor Sanchez

arXiv:2609.29678v1cs.CVcs.AI

TL;DR

Fine-grained SSL can admit visually confident but incorrect pseudo-labels, and its visual-only reliability signals do not assess semantic compatibility. ReCalMatch calibrates pseudo-labels with multi-aspect visual–semantic agreement, confidence, and entropy; it reports consistent gains across fine-grained benchmarks, largest in low-label regimes.

  • Problem

    In fine-grained SSL, visually confident predictions can be incorrect, while common reliability criteria rely on visual predictions and do not assess semantic compatibility.

    Source: §1 · p. 2

  • Method

    ReCalMatch combines multi-aspect visual–semantic agreement with confidence and entropy to calibrate pseudo-label reliability and reweight consistency learning.

    Source: §3 · p. 5

  • Results

    ReCalMatch reports consistent gains across four fine-grained benchmarks, with the largest improvements in low-label regimes.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The approach reduces overconfident pseudo-label errors and improves pseudo-label learning in fine-grained SSL.

    Source: §6 · p. 14

    The setup considers a K-class semi-supervised setting.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Semi-supervised fine-grained visual recognition is highly vulnerable to overconfident pseudo-label errors: visually similar categories frequently produce high-confidence yet incorrect predictions, and consistency regularization then reinforces these errors throughout training. Existing semi-supervised learning (SSL) methods estimate pseudo-label reliability almost entirely from the visual classifier itself---maximum probability, adaptive thresholds, or entropy---signals that remain blind to whether a predicted class is \emph{semantically} compatible with the visual representation. We propose \textbf{ReCalMatch}, a reliability-calibrated semantic framework for semi-supervised fine-grained recognition. Rather than treating textual semantics as auxiliary supervision, ReCalMatch uses multi-aspect semantic prototypes as \emph{calibration evidence} for pseudo-label learning. We construct class-conditioned semantic prototypes from class names and domain-specific semantic aspects, and measure a \emph{visual--semantic agreement} score between each unlabeled embedding and its pseudo-label prototype. This agreement is combined with prediction confidence and entropy into a single reliability weight that down-weights pseudo-labels that are visually confident but semantically inconsistent. A semantic consistency term and a semantic margin regularizer further sharpen prototype separability under limited labels. Extensive experiments on CUB-200-2011, Stanford Dogs, NABirds, and iNaturalist18 show that ReCalMatch consistently improves strong SSL baselines, with the largest gains in low-label regimes where pseudo-label noise is most severe.

1 Introduction

Fine-grained categories are difficult for SSL because visually confident predictions can still be wrong, and consistency training can reinforce those errors. ReCalMatch uses multi-aspect semantic agreement alongside confidence and uncertainty to calibrate pseudo-label reliability, with reported gains across fine-grained benchmarks, especially in low-label regimes.

1 Introduction

  • Fine-grained SSL is vulnerable to corrupted pseudo-labels because subtle category cues and variable appearance make confident predictions unreliable, while consistency regularization can reinforce errors.Source: §1 · p. 2
  • Visual-only confidence, threshold, and entropy criteria can accept overconfident errors, while existing semantic-guided pipelines use semantics mainly as additional supervision.A single class-level text view can also be too coarse for fine-grained distinctions spread across multiple semantic aspects.Source: §1 · p. 2
  • ReCalMatch combines visual–semantic agreement, confidence, and entropy to reweight the unsupervised consistency loss, with semantic consistency and margin regularization sharpening prototype separability.The margin regularizer reduces cross-class prototype redundancy so agreement remains discriminative.Source: §1 · pp. 2–3
  • ReCalMatch uses multi-aspect semantic evidence to calibrate pseudo-label reliability rather than treating semantics only as auxiliary supervision.The framework assesses whether an unlabeled embedding is compatible with the semantic prototype of its predicted class.Source: §1 · pp. 2–3
  • ReCalMatch reports consistent gains across four fine-grained benchmarks, with the largest improvements in low-label regimes.Source: §1 · p. 3

2 Related Work

Prior SSL methods commonly estimate pseudo-label reliability from visual predictions, while fine-grained recognition and semantic-guided approaches leave open how to assess trustworthiness using fine-grained semantic evidence. ReCalMatch addresses this gap with multi-aspect prototypes and visual–semantic agreement as an explicit calibration signal.

2.1 Semi-Supervised Learning and Pseudo-Label Reliability

  • SSL methods use confidence thresholds, adaptive thresholds, or entropy to estimate pseudo-label reliability from visual predictions, but gains can be unstable when inter-class margins are small.Examples include FixMatch [48], FlexMatch [65], Dash [64], FreeMatch [58], and SoftMatch [6].Source: §2.1 · p. 3

2.2 Fine-Grained Visual Classification

  • Supervised FGVC methods model discriminative parts, regions, or tokens, while dedicated FGVC-SSL approaches stabilize learning without assessing whether an unlabeled prediction is trustworthy.ReCalMatch is complementary: it adds an external semantic reliability signal rather than redesigning the visual pipeline.Source: §2.2 · pp. 3–4

2.3 Vision-Language Semantics for Recognition

  • Prior semantic-guided SSL uses text chiefly as auxiliary supervision and often relies on a single class-level view, whereas ReCalMatch uses multi-aspect prototypes and visual–semantic agreement to calibrate reliability.Class-name text embeddings have been used to guide pseudo-label learning in SemiVisBooster.Source: §2.3 · pp. 4–5

3 ReCalMatch

ReCalMatch adds semantic evidence to standard semi-supervised learning so pseudo-label reliability reflects visual confidence, uncertainty, and agreement with class prototypes. Its training objective also uses semantic consistency and prototype-margin losses to keep the calibration signal discriminative.

3 ReCalMatch

  • ReCalMatch couples a standard visual SSL branch with multi-aspect semantic prototypes, a reliability-calibration block, and two auxiliary losses.The calibration weight gates the unsupervised consistency loss; the auxiliary terms are semantic consistency and prototype margin regularization.Source: §3 · p. 5

3.1 Baseline SSL and Pseudo-Label Reliability

  • Visual confidence masks and prediction statistics alone cannot distinguish correct predictions from overconfident errors on ambiguous fine-grained samples.The method therefore adds an external reliability signal from multiple semantic aspects of the predicted class.Source: §3.1 · p. 6

3.2 Multi-Aspect Semantic Prior Construction

  • The semantic prior supplies separate per-aspect class prototypes for calibration and a fused, class-conditioned prototype as the auxiliary-loss consensus target.The fused prototype’s gate is image-independent, while per-sample adaptivity comes from the separate agreement scores.Source: §3.2 · pp. 6–7

3.3 Semantic Bank Instantiation and Cross-Dataset Protocol

  • The semantic bank is precomputed offline and instantiated with domain-specific prompts, using color, shape, and part aspects for the listed biological FGVC datasets.CUB uses attribute and part annotations; the other described datasets use class-name-driven templates or bird-oriented vocabulary.Source: §3.3 · p. 7

3.4 Per-Aspect Agreement and Cross-Aspect Disagreement

  • Per-aspect agreement scores are summarized by their mean and cross-aspect disagreement, which identifies pseudo-labels with uneven semantic support.The construction does not assume that the underlying aspects are statistically independent.Source: §3.4 · pp. 7–8

3.5 Reliability-Calibrated Consistency Learning

  • The reliability weight combines visual statistics with mean semantic agreement and an exponential disagreement gate that suppresses inconsistent semantic support.The gate can reduce a pseudo-label’s weight even when the other factors individually accept it.Source: §3.4 · p. 8, §3.5 · p. 8

3.6 Auxiliary Semantic Consistency and Margin Regularization

  • Semantic consistency draws embeddings toward class prototypes, while margin regularization penalizes excessive correlations between fused prototypes to preserve separability.The shared class and aspect representations let the prototype penalty also improve the separability of per-aspect agreement scores.Source: §3.6 · pp. 8–9

3.7 Overall Objective and Training Protocol

  • The full objective combines supervised, reliability-calibrated consistency, semantic-consistency, and margin losses in a lightweight branch that adds cosine-similarity computations.The baseline pseudo-labeling and consistency machinery is preserved.Source: §3.7 · p. 9

4 Experiments and Results

Across four fine-grained benchmarks and three label settings, ReCalMatch reports its strongest performance in low-label regimes. Visualizations show more separated embeddings and compact, object-aligned activations.

4.1 Implementation Details

  • Experiments use ViT-B/16 at 224 × 224, exclude CLIP to isolate calibration, and follow the FreeMatch training protocol with shared hyperparameters across datasets and label ratios.Training uses AdamW, cosine decay, warm-up, and 200 epochs; the text encoder remains frozen while the projector is adapted.Source: §4.1 · pp. 9–10

4.2 Datasets

  • Evaluation covers iNaturalist18, NABirds, CUB-200-2011, and Stanford Dogs at 1%, 10%, and 25% labels, spanning varied degrees of class imbalance.The benchmarks include severely and highly long-tailed distributions as well as moderately imbalanced datasets.Source: §4.2 · p. 10

4.3 Comparison with the State of the Art

  • ReCalMatch achieves its strongest performance in low-label regimes, improves accuracy over SemiVisBooster across all four benchmarks, and remains competitive at 25% labels.Table 1 reports top-1 accuracy under 1%, 10%, and 25% labeled-data settings.Source: §4.3 · pp. 10–11, Table 1 · p. 11

4.4 Visualization Results

  • On CUB-200-2011, t-SNE embeddings progress from overlapping FreeMatch clusters through diffuse SemiVisBooster structure to more compact, separated ReCalMatch geometry.The figure compares the visual-only baseline, class-level text guidance, and reliability-calibrated semantic guidance.Source: Figure 3 · p. 11, §4.4 · p. 11
  • Activation maps progress from scattered, background-distracted FreeMatch responses and diffuse SemiVisBooster activations to compact, object-aligned ReCalMatch regions.The reported comparison associates calibration by cross-aspect disagreement with attention to regions supported by independent semantic aspects.Source: Figure 4 · p. 12, §4.4 · p. 12

5 Ablation Studies

The ablations attribute gains to complementary semantic aspects and disagreement-aware reliability calibration, with the disagreement gate improving pseudo-label precision. Sensitivity tests report robust settings and a runtime trade-off, while alternative semantic-bank sources show the method is not restricted to curated metadata.

5.1 Component and Calibration Analysis

  • Adding multi-aspect prototypes, the margin regularizer, and reliability calibration progressively improves the FreeMatch-based model, with disagreement gating adding the final gain, largest at 1% labels.The comparison is on CUB-200-2011 under 1%, 10%, and 25% labeled-data settings.Source: Table 2 · p. 12, §5.1 · p. 12

5.2 Aspect and Gating Sensitivity

  • Accuracy rises as semantic aspects are added; part is the strongest single aspect, and color+shape+part performs best, while ζ peaks at 2 and is stable from 1 to 3.Setting ζ = 0 disables disagreement gating and recovers the configuration without disagreement.Source: Figure 5 · p. 13, Figure 6 · p. 13, §5.2 · pp. 13–14

5.1 Component and Calibration Analysis

  • At 1% labels, class-conditioned aspect gating of Eq. 10 achieves 54.2% accuracy, outperforming uniform averaging (52.3%) and global learnable fusion (53.0%) in consensus-target construction.Source: §5.1 · p. 13
  • At 1% labels, disagreement gating raises accepted-pseudo-label precision from 71.3% to 78.6% while retaining 92% of correct pseudo-labels.The reported change indicates the gate removes overconfident errors without simply reducing pseudo-label coverage.Source: §5.1 · p. 13

5.3 Sensitivity, Stability, and Efficiency

  • Both confidence and entropy factors improve over the unweighted variant, their combination is strongest, and multi-seed results show gains stable across random splits.Temperatures τg and τc peak at 0.07 and remain stable within [0.05,0.10].Source: §5.3 · p. 14
  • ReCalMatch uses 17.6 GFLOPs at 0.120 s/step, close to SemiVisBooster’s 17.5 GFLOPs and faster than its 0.198 s/step runtime, but exceeds FreeMatch’s 4.2 GFLOPs and 0.060 s/step.This trade-off is most favorable in the low-label regimes, where ReCalMatch provides its largest accuracy gains.Source: §5.3 · p. 14

5.4 Semantic Bank Source

  • On CUB-200-2011 at 1% labels, an LLM-generated semantic bank achieves 53.6% accuracy, and combining generated descriptions with metadata reaches 54.8%.The results suggest curated attributes are not strictly required, though they provide a cleaner semantic-bank instantiation.Source: §5.4 · p. 14

6 Conclusion

ReCalMatch uses multi-aspect semantic evidence to calibrate pseudo-label reliability alongside semantic consistency and prototype margin regularization. Its semantic bank's quality and domain relevance bound the approach, and broader domains and prediction settings remain future work.

6 Conclusion

  • Table 4 compares semantic-bank sources on CUB-200-2011 under the 1% labeled-data setting.Source: Table 4 · p. 14
  • ReCalMatch calibrates pseudo-label reliability with class-conditioned semantic prototypes and cross-aspect disagreement, alongside semantic consistency and prototype margin regularization.The conclusion states that this design reduces overconfident pseudo-label errors and improves pseudo-label learning in fine-grained SSL.Source: §6 · p. 14
  • Performance depends on semantic-bank quality and domain relevance; atypical coloration may raise disagreement enough to over-suppress a correct pseudo-label.Domain-specific or LLM-generated aspects can mitigate reliance on curated attributes.Source: §6 · p. 14
  • Extending semantic reliability calibration to domains with substantially different cues and to dense or open-vocabulary prediction remains future work.Source: §6 · p. 14

Related papers

Loading 2609.29678v1…