PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Mismapping in Generalized Category Discovery: Evaluation and Mitigation

Zitai Wang, Qianqian Xu, Cong Hua, Xi Chen, Zhiyong Yang, Xiaochun Cao, Qingming Huang

arXiv:2610.39356v1cs.CVcs.AIcs.LG

TL;DR

Parametric GCD globally matches outputs even though old indices are supervised and anchored, allowing matching to hide prediction errors. The paper introduces SPE to preserve those semantics and MMO to mitigate underlying prediction errors; experiments show mismapping can inflate new-category accuracy substantially and MMO generally improves recognition and balance.

  • Problem

    Parametric GCD uses global matching despite old outputs being semantically anchored, and this can hide errors and overestimate new-category accuracy by up to 25.63 points on Herbarium 19.

    Source: §1 · p. 2

  • Method

    SPE preserves old-output semantics and matches only within the new region, while MMO combines region-balance and novel-diversity losses to mitigate prediction errors.

    Source: §1 · p. 2

  • Results

    Experiments show that GH overestimation is negligible on coarse-grained benchmarks but reaches 17.40–25.63 points in new accuracy on Herbarium 19; MMO generally improves new-category recognition and old/new balance.

    Source: §6.2 · p. 9, §7 · p. 13

  • Takeaways & Limitations

    The findings support using SPE alongside existing GCD evaluation to preserve anchored output semantics and report old-category recognition and novel-category discovery faithfully.

    Source: §7 · p. 13

    The analysis assumes the total number of target categories is known, although it can be estimated with off-the-shelf methods when unknown.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Generalized Category Discovery (GCD) aims to perform old-category recognition and novel-category discovery in the same unlabeled set. Early methods address both objectives through clustering and evaluate the exchangeable predicted indices with global Hungarian (GH) matching. Recent methods adopt trainable parametric heads, using supervision for old-category recognition and discovery-oriented objectives for new categories. In this way, the output space becomes partially exchangeable: old indices are semantically anchored, while new indices remain exchangeable. However, these methods still apply global matching, which can reassign anchored old indices to other categories and hide some prediction errors. Empirically, we are surprised to observe that this reassignment can induce an overestimation of new-category accuracy by up to 25.63 points. We term this phenomenon semantic mismapping and further derive a permutation-cycle condition that characterizes when semantic mismapping can increase the global matching score. To fix this issue, we propose Semantics-Preserving Evaluation (SPE), which preserves the supervised semantics of old indices and restricts Hungarian matching to the new region. Furthermore, we propose Mismapping-Mitigating Optimization (MMO) to alleviate the prediction errors that can give rise to semantic mismapping, combining a region-balance loss (RBL) for old/new routing with a novel-diversity loss (NDL) that encourages orthogonality among new prototypes. Theoretically, we show that minimizing the two losses can tighten an upper bound on spurious accuracy gains from semantic mismapping, with controllable residual terms. Empirical results under both GH and SPE demonstrate the effectiveness of our proposed method.

1 INTRODUCTION

Generalized Category Discovery combines recognition of labeled old categories with discovery of new categories in an unlabeled set. In parametric methods, global Hungarian matching can override old indices’ supervised semantics, hiding prediction errors; the paper proposes evaluation and optimization remedies.

1 INTRODUCTION

  • Parametric GCD makes the output space partially exchangeable: old indices are supervised and anchored, whereas new indices remain exchangeable.This differs from early clustering-based GCD, where all induced cluster indices are exchangeable and are aligned to categories by Hungarian matching.Source: §1 · p. 1
  • On Herbarium 19, semantic mismapping overestimates new-category accuracy by up to 25.63 points, while its effect is negligible on CIFAR-10 and ImageNet-100.Global matching can reassign anchored old indices, making some false-old predictions appear correct; a permutation-cycle condition characterizes when this raises the matching score.Source: §1 · p. 2
  • Semantics-Preserving Evaluation (SPE) preserves old-output semantics, matches only within the new region, and counts predictions crossing the old/new boundary as errors.The protocol is also extended to non-parametric GCD and settings with an estimated number of new categories.Source: §1 · p. 2
  • Mismapping-Mitigating Optimization (MMO) combines region-balance loss for old/new prediction mass with novel-diversity loss that encourages orthogonality among new prototypes.The paper derives bounds showing that these losses can suppress spurious accuracy gains when residual errors are controlled, with sufficient conditions for GH and SPE to yield the same overall accuracy.Source: §1 · p. 2

2 RELATED WORK

The related work situates GCD among category-discovery settings, evaluation protocols, and methods addressing prediction bias. It distinguishes the paper’s focus on mixed old/new GCD evaluation from prior settings and approaches.

2.1 Category-Discovery Settings

  • GCD places old and new categories together in the unlabeled set, unlike NCD’s new-only discovery set; this paper studies faithful evaluation and joint improvement of old recognition and novel discovery.The paper focuses on the standard GCD setting.Source: §2.1 · p. 3

2.2 Evaluation Protocols for Category Discovery

  • Prior GCD evaluation globally matches the unlabeled set, while FRoST preserves old-category semantics in NCD but leaves mixed-set GCD evaluation unaddressed.µGCD observes nonzero new-category accuracy when no samples are predicted to new outputs, but still uses global matching without preserving anchored old-output semantics.Source: §2.2 · p. 3

2.3 GCD Methods and Prediction Bias

  • Existing GCD methods address prediction bias at prediction, feature, and gradient levels; this work instead examines how global matching hides resulting errors and designs prediction- and feature-level regularizers.Prior approaches include entropy or class-wise regularization, prototype separation, and gradient coordination.Source: §2.3 · p. 3

3 PRELIMINARIES

GCD trains from labeled old-category data and an unlabeled set containing both old and new categories. Its standard global Hungarian evaluation treats every predicted index as exchangeable, despite parametric heads anchoring old indices.

3.1 Generalized Category Discovery

  • GCD learns from labeled old-category data and an unlabeled set containing old and new categories, with the total target-category count assumed known unless estimated by an off-the-shelf method.At evaluation, the model recognizes old-category samples and assigns new-category samples to category indices.Source: §3.1 · p. 3
  • Early GCD uses non-parametric clustering with fully exchangeable outputs, whereas parametric heads partition outputs into supervised, anchored old indices and exchangeable new indices.The parametric head is trained through labeled-data supervision for old categories and discovery-oriented objectives on the unlabeled set for new categories.Source: §3.1 · p. 3

3.2 Standard GCD Evaluation Protocol

  • Standard GCD evaluation applies one global Hungarian assignment to the full unlabeled set and reuses it to report old, new, and overall accuracy.This protocol suits fully exchangeable outputs, but its use with partially exchangeable parametric heads can induce semantic mismapping.Source: §3.2 · pp. 3–4
  • In the illustrated confusion matrix, GH makes 46 false-old predictions and 44 old-to-new errors appear correct, whereas SPE keeps both cross-region groups as errors.The matrix places predicted outputs in rows and ground-truth categories in columns.Source: Figure 2 · p. 4

4 SEMANTIC MISMAPPING AND SEMANTICS-PRESERVING EVALUATION

Global Hungarian matching can override the meanings of supervised old outputs, allowing prediction errors to count as correct. SPE preserves those meanings while retaining local matching for new categories, and can be extended to additional GCD settings.

4.1 Semantic Mismapping under Global Matching

  • Semantic mismapping occurs when matching assigns an old output to a category other than its supervised category.It can occur within the old region or across the old–new boundary.Source: §4.1 · pp. 4–5
  • A negative cycle margin means a reassignment raises the global matching score; the semantics-preserving assignment is globally optimal exactly when msem ≥ 0.The margin compares correct predictions gained and lost along a permutation cycle.Source: §4.1 · p. 4

4.2 Semantics-Preserving Evaluation

  • SPE checks old outputs against their fixed meanings, locally matches new outputs to new categories, and counts cross-region predictions as errors.This prevents both within-old and cross-boundary semantic mismapping while retaining permutation invariance for new-category discovery.Source: §4.2 · p. 5

4.3 Extensions of SPE Protocol

  • For non-parametric GCD, SPE identifies old-output correspondence from labeled data before evaluating unlabeled predictions.The extension applies to most non-parametric methods that use semi-supervised k-means.Source: §4.3 · pp. 5–6
  • When the new-category count is estimated, SPE pads only the new region with dummy outputs or categories before matching.Dummy assignments leave the corresponding real output or category unmatched.Source: §4.3 · p. 6

5 MMO

MMO adds region-balance and novel-diversity losses to parametric GCD training. The losses target old/new routing and discrimination among new categories, and theory bounds how they can limit spurious gains from semantic mismapping.

5 MMO

  • MMO adds two plug-and-play losses to existing parametric GCD training: RBL for old/new routing and NDL for discrimination within the new region.The combined objective retains the baseline method’s supervised and discovery-oriented objectives.Source: §5 · p. 6, §5.1 · pp. 6–7

5.1 Objectives of MMO

  • RBL encourages batch-average probability mass for the old and new regions to match their estimated sample proportions.The experiments use a tolerance-based variant to accommodate minibatch proportions that differ from the overall proportions.Source: §5.1 · p. 6
  • NDL penalizes correlations between normalized new-category prototypes; exact orthogonality is attainable when Cnew ≤ d.When Cnew exceeds d, it can still reduce prototype correlations without requiring zero loss.Source: §5.1 · pp. 6–7

5.2 Theoretical Analysis

  • With residual errors controlled, reducing the two losses tightens an upper bound on matching gains and raises a lower bound on semantic cycle margins.The residual includes additional errors such as old-category misclassification and inaccurate prior estimates; the global guarantee gives equal GH and SPE overall accuracy when 2B ≤ π*.Source: §5.2 · p. 7

6 EXPERIMENTS

Experiments compare seven method–backbone configurations under GH and SPE, and evaluate MMO across benchmarks, parameter settings, and SPE extensions. The results show substantial GH–SPE differences in some settings and generally improved new-category accuracy with MMO.

6.2 Empirical Evidence of GH Overestimation (RQ1)

  • On Herbarium 19, Δnew reaches 17.40–25.63 across configurations, and method rankings change between GH and SPE; summed cycle margins match Δall in four SimGCD configurations.These observations connect GH overestimation to the cycle-margin analysis.Source: Table 3 · p. 8, §6.2 · p. 9, Table 4 · p. 9

6.3 Effectiveness and Analysis of MMO (RQ2)

  • MMO improves new-category accuracy under both GH and SPE in 23 of 28 configurations, with average gains of 2.55 points under GH and 6.48 points under SPE.On CUB, SimGCD–CLIP gains 5.57 points under GH and 6.10 points under SPE.Source: §6.3 · pp. 9–10, Table 5 · p. 10
  • On Herbarium 19, MMO raises NewSPE by 22.17, 30.93, and 26.76 points for SimGCD–DINO, SimGCD–CLIP, and GET–CLIP, respectively.Their full-set GH–SPE gaps fall from 9.03, 12.13, and 10.44 points to 0.85, 0.56, and 0.75 points, respectively.Source: §6.3 · pp. 10–11
  • On ImageNet-1K, MMO improves overall and new-category accuracy under both protocols with both backbones.For DINO, overall/new gains are 1.65/2.40 points under GH and 2.07/3.02 under SPE; for CLIP, they are 1.04/1.27 and 1.37/2.13 points.Source: §6.3 · p. 11, Table 6 · p. 11
  • With SimGCD–CLIP on Herbarium 19, setting bρ = 0.40 yields the highest reported AllSPE and HSPE among manual settings: 50.30% and 51.04%.The TIcE estimate also improves AllSPE, NewSPE, and HSPE over baseline, though it falls below the best manual setting.Source: Figure 4 · p. 11, §6.3 · p. 12
  • Moderate λRB and λND settings favor old–new balance; the largest tested weights reduce accuracy, while t-SNE views show several clusters becoming more compact and less overlapping with MMO.For GET and GET-TAR, NewSPE reaches 43.86% and 44.80% with MMO.Source: §6.3 · p. 12, Figure 5 · p. 12, Figure 6 · p. 12

6.4 Applicability of SPE Extensions (RQ3)

  • With estimated new-category counts, SPE pads only the new region and remains applicable; on Stanford Cars, MMO reduces SimGCD–CLIP’s Δnew and Δall gaps to 0.48 and 0.05 points.AllSPE rises from 72.51% to 76.22%.Source: §6.4 · p. 13, Table 7 · p. 13
  • For non-parametric GCD, labeled-set calibration enables SPE, and GH–SPE differences remain observable after calibration across the reported benchmarks.On CUB, calibration accuracy is 99.93%, yet GH exceeds SPE by 0.30 points in new accuracy and 0.24 points overall.Source: §6.4 · p. 13, Table 8 · p. 13

7 CONCLUSION

The paper identifies semantic mismapping as a source of misleading GH accuracy gains and introduces SPE and MMO to address evaluation and prediction errors. Its cycle analysis characterizes when reassignment can improve the global matching score.

7 CONCLUSION

  • Global matching can overestimate GCD performance by overriding supervised old-output meanings, while SPE preserves those meanings during evaluation.The paper reports that semantic mismapping can also affect method rankings.Source: §7 · p. 13
  • The paper proposes MMO to mitigate prediction errors that global matching can hide, and reports that experiments support SPE’s applicability across different GCD settings.MMO generally improves new-category recognition and the balance between old and new categories.Source: §7 · p. 13

A.1 Semantic Mismapping Margin

  • A reassignment cycle can raise the global matching score when its margin is negative; cycles involving only new categories cannot improve on the best assignment that preserves old meanings.Therefore, any improvement must involve a cycle containing an old category.Source: §A.1 · p. 19

A.2 Theoretical Analysis of MMO

The analysis bounds semantic-mismapping gains under evaluation and prediction–similarity assumptions, decomposing prediction errors into residual terms that RBL and NDL do not eliminate. It also identifies when score-increasing reassignments require old-category involvement and when GH and SPE must agree.

A.2 Theoretical Analysis of MMO

  • The bounds assume an evaluation set covering every category, at least two new categories, and prediction–similarity consistency for new outputs.The consistency condition requires similarity-order agreement, not correct predictions or well-separated prototypes.Source: §A.2 · p. 20
  • RBL controls aggregate old-region probability mass, but the bound retains prior-estimation and soft-to-hard routing discrepancies as residual errors.RBL uses an estimated old-category proportion as its fixed target; it does not itself correct that estimate or enforce individual argmax decisions.Source: §A.2 · pp. 21–22
  • NDL bounds within-new-category errors through prototype correlations, with the largest new-category proportion scaling the correlation term.The derivation combines weighted Cauchy–Schwarz with the average squared cosine similarity between distinct new prototypes.Source: §A.2 · pp. 20, 22
  • Any score-increasing reassignment cycle must include an old category; cycles involving only new categories cannot increase the score under the semantics-preserving assignment.The analysis also bounds cycle margins using the total SPE error.Source: §A.2 · pp. 23–24
  • When 2B < π*, GH and SPE have the same accuracy; when 2B = π*, semantic mismapping may occur without changing accuracy.The matching-gain bound also depends on residual errors, including old-category, prior-estimation, routing, and feature–prototype alignment terms.Source: §A.2 · p. 23

B.1 Datasets

The experiments span eight GCD datasets, compare seven baseline configurations with MMO, and report protocol comparisons across coarse- and fine-grained benchmarks.

B.1 Datasets

  • Eight datasets cover coarse-grained, fine-grained, and large-scale category discovery settings.The suite includes CIFAR-10/100, ImageNet-100/1K, CUB, Stanford Cars, FGVC-Aircraft, and Herbarium 19.Source: §B.1 · p. 24

B.2 Evaluated Methods

  • Seven DINO- and CLIP-based configurations represent classifier, prototype, regularized, and multimodal approaches, with MMO applied while preserving each baseline pipeline.The evaluated configurations include LegoGCD, SimGCD-BIA, ProtoGCD, SimGCD, GET, and GET-TAR.Source: §B.2 · p. 25

B.3 Implementation Details

  • Training follows baseline preprocessing for 200 epochs, uses final checkpoints and original seeds, and selects MMO weights by grid search.The grid searches λRB over {0.25, 0.5, 0.75} and λND over {0.01, 0.05, 0.1}.Source: §B.3 · p. 25

C.1 Additional Comparison between GH and SPE (RQ1)

  • On coarse-grained benchmarks GH and SPE usually agree, whereas on Herbarium GH raises new and overall scores for every method and SPE usually raises old-category accuracy.The new-category gap can have either sign because GH optimizes overall accuracy, while the overall GH–SPE gap is nonnegative.Source: §C.1 · p. 26, Table 9 · p. 27

C.2 Additional Results on the Coarse-Grained Benchmark (RQ2)

On ImageNet-100, MMO improves performance across five configurations even though GH and SPE produce identical scores for every reported run.

C.2 Additional Results on the Coarse-Grained Benchmark (RQ2)

  • MMO gains 0.15–3.05 overall-accuracy points and 0.02–3.30 new-accuracy points across all five configurations, while GH and SPE scores coincide.The reported results cover the coarse-grained ImageNet-100 benchmark.Source: Table 10 · p. 28, §C.2 · p. 28

C.3 Complete Results on the Large-Scale Benchmark (RQ1 and RQ2)

On ImageNet-1K, MMO improves new and overall accuracy under both protocols, while the GH–SPE discrepancy appears across all three reproduced baselines.

C.3 Complete Results on the Large-Scale Benchmark (RQ1 and RQ2)

  • Across all three baselines, GH raises new accuracy by 5.19–7.36 and overall accuracy by 1.41–2.23 points, while lowering old accuracy by 6.12–8.20 points versus SPE.The protocol discrepancy persists on ImageNet-1K and is not tied to one backbone.Source: Table 11 · p. 28, §C.3 · p. 28
  • Under GH/SPE, MMO raises New by 2.40/3.02 points for DINO and 1.27/2.13 for CLIP, and raises All by 1.65/2.07 and 1.04/1.37 points, respectively.The comparison covers SimGCD and its +Ours variant on ImageNet-1K.Source: Table 12 · p. 28, §C.3 · p. 28

C.4 Complete ρ-Sensitivity Results on an Imbalanced Benchmark (RQ2)

MMO improves results across protocol and cardinality settings, while the experiments expose sensitivity to the old/new trade-off and show that visual compactness alone does not establish correct category alignment.

C.4 Complete ρ-Sensitivity Results on an Imbalanced Benchmark (RQ2)

  • On Herbarium, raising bρ from 0.40 to 0.70 raises NewSPE from 41.04 to 45.37 but lowers AllSPE from 50.30 to 42.76; TIcE-estimated ρ lets MMO gain 10.75 AllSPE and 20.09 NewSPE points.Source: §C.4 · p. 29

C.5 Old–New Trade-offs and Overall Accuracy (RQ2)

  • Under SPE, MMO raises SimGCD New accuracy by 30.93 points on Herbarium; on Stanford Cars, SimGCD gains 12.38 NewSPE and 6.80 AllSPE points.The SPE trade-offs often pair a small old-category decrease with a larger new-category gain.Source: §C.5 · p. 29, Figure 8 · p. 30, Figure 10 · p. 30

C.7 Additional Results of the t-SNE Embedding Visualization (RQ2)

  • Across three methods, MMO generally tightens and reduces cluster mixing as Intra falls and Inter rises; t-SNE is qualitative, with distances measured in the original embeddings.Smaller Intra indicates greater within-category compactness, while larger Inter indicates greater separation between categories.Source: §C.7 · pp. 30–31

C.6 Ablation of MMO Components (RQ2)

  • Under SPE, RBL raises HSPE by 1.76, 2.59, and 5.62 points on CUB, Aircraft, and Cars; adding NDL raises NewSPE by 2.83, 4.38, and 0.46 points.With NDL added, HSPE gains are 1.51, 1.33, and 0.65 points on those datasets, respectively.Source: §C.6 · p. 29, Table 14 · p. 31

C.8 Detailed Results with Estimated Category Numbers (RQ3)

  • With estimated category numbers, MMO reduces the Stanford Cars SimGCD–CLIP New GH–SPE gap from 3.91 to 0.48 points and raises NewSPE, AllSPE, and HSPE by 4.99, 3.71, and 3.57 points.The results also report improved new-category accuracy and harmonic mean for both DINO configurations.Source: §C.8 · p. 31, Table 15 · p. 32

APPENDIX D IMPLEMENTATION DETAILS OF REGION-BALANCE LOSS

The appendix specifies a minibatch-tolerant implementation of RBL and a procedure for estimating an unknown labeling fraction to obtain its old-region sample prior.

D.1 Minibatch Tolerance and Numerical Stability

  • The tolerance variant replaces RBL’s fixed regional prior with an effective target and applies no corrective gradient inside the allowed interval, pushing batch-average probabilities toward its nearest boundary otherwise.The underlying prior remains fixed; setting δ = 0 recovers the fixed-prior loss, subject to numerical clipping.Source: §D.1 · p. 33
  • RBL clips pold and probabilities inside logarithms to [ϵ, 1 − ϵ], with ϵ = 10^-6, and uses δ = 0.05 by default.The new-region prior is pnew = 1 − pold.Source: §D.1 · p. 33

D.2 Estimating an Unknown Labeling Fraction

  • When the labeling fraction ρ is unknown, TIcE estimates it from frozen features and labeled/unlabeled status, without hidden category labels, to obtain RBL’s old-region sample prior.The estimate assumes old-category samples are selected for labeling independently of features with probability ρ; for any feature-defined subset A, Pr(S = 1 | X ∈ A) = ρ Pr(Z = 1 | X ∈ A) ≤ ρ.Source: §D.2 · p. 33
  • TIcE uses decision-tree subsets dominated by old-category samples, separate folds for subset selection and labeled-fraction estimation, and uncertainty-adjusted subset scores.It initializes ρ0 = 0.5, takes the maximum subset score and overall labeled fraction per estimation fold, averages across folds, repeats once, and substitutes the resulting estimate into Eq.(19) to obtain pold.Source: §D.2 · p. 33
Loading 2610.39356v1…