PaperlayerView exact revision on arXiv

Source-linked AI summary

C-Score: Beyond Accuracy for Robustness Assessment in Semi-Supervised Learning under Open-World Unlabeled Contamination

Tsao-Lun Chen, Chi-Cheng Fu, Han-Yi E. Chou, Shun-Feng Su

arXiv:2608.20667v1cs.LGcs.AI

TL;DR

Pseudo-label-based SSL can receive misleading supervision from confident OOD predictions, while clean accuracy remains stable. The paper introduces C-Score to diagnose prediction, representation, and optimization degradation. Across contaminated settings, its internal signals reveal degradation that clean accuracy alone may miss.

  • Problem

    Open-world OOD contamination can make confident pseudo-labels misleading, while conventional clean-accuracy evaluation may hide internal SSL degradation.

  • Method

    C-Score evaluates contaminated pseudo-label-based SSL through prediction, feature-representation, and optimization diagnostics.

  • Results

    C-Score signals reveal hidden degradation that clean test accuracy alone may fail to detect across CIFAR-10 and CIFAR-100 contamination experiments.

  • Takeaways & Limitations

    Clean accuracy alone is insufficient for evaluating pseudo-label-based SSL robustness under open-world unlabeled contamination.

Abstract

from arXiv · show

Pseudo-label-based semi-supervised learning has achieved strong performance due to its simplicity and scalability. However, it is typically developed under a closed-world assumption that unlabeled data are drawn from the same distribution as labeled data. In practical deployment, unlabeled data are often collected from open environments and may contain OOD samples. Under such contamination, OOD samples may still receive high-confidence predictions and be incorporated into training as if they were valid target examples. This creates an important evaluation problem: clean in-distribution test accuracy may appear stable even when the internal learning dynamics of SSL have already deteriorated. To address this issue, we study hidden collapse in pseudo-label-based SSL under open-world unlabeled contamination from a diagnostic evaluation perspective. We present C-Score, a compact framework that evaluates training behavior in three complementary spaces: prediction, feature representation, and optimization. C-Score includes PLE and CCI for unlabeled prediction behavior, Sem-Drift for deviation from labeled semantic anchors, and Grad-Align for the compatibility between labeled and unlabeled optimization. Experiments on CIFAR-10 and CIFAR-100 with multiple OOD sources, varying contamination ratios, and four pseudo-label-based SSL algorithms show that C-Score metrics reveal hidden degradation that clean accuracy alone fails to detect: under SVHN contamination, CCI rises over 280% while best-accuracy remains within 3% of the uncontaminated baseline; near-OOD sources (CIFAR-100, STL-10) cause up to 14.9% accuracy collapse (FlexMatch, r=0.5). The results suggest that clean accuracy alone is insufficient for evaluating SSL robustness in open-world environments, and that internal diagnostic signals are necessary for more reliable robustness assessment under unlabeled contamination.

I. INTRODUCTION

Pseudo-label-based SSL is attractive for its simplicity and scalability but assumes unlabeled data share the labeled distribution. Open-world OOD contamination can create accuracy masking, motivating C-Score’s multi-space diagnostics.

  • Pseudo-label-based SSL is attractive because it is simple and scalable.
  • Open-world deployment can violate the assumption that labeled and unlabeled samples share a distribution.Unlabeled pools may contain OOD samples.
  • High-confidence predictions can turn OOD samples into misleading training targets while clean accuracy remains stable.This evaluation blind spot is called accuracy masking.
  • C-Score assesses training behavior across prediction, feature, and optimization perspectives.
  • C-Score signals reveal hidden degradation that clean test accuracy alone may not show.
  • The paper identifies accuracy masking as an evaluation blind spot in pseudo-label-based SSL under open-world contamination.

II. PROBLEM FORMULATION

The paper studies pseudo-label-based SSL when unlabeled data mix in-distribution and OOD samples. Its formulation emphasizes confidence-derived supervision and distinguishes near-OOD from far-OOD contamination.

  • A. Pseudo-Label-Based Semi-Supervised Learning: SSL combines a small labeled dataset with a much larger unlabeled dataset to learn a target-task classifier.
  • A. Pseudo-Label-Based Semi-Supervised Learning: Pseudo-label-based SSL constructs hard or soft targets from model predictions on unlabeled samples.Confidence-based selection or weighting makes unlabeled influence depend directly on prediction confidence.
  • A. Pseudo-Label-Based Semi-Supervised Learning: The approach relies on confident predictions being semantically valid when unlabeled samples match the target distribution.
  • B. Open-World Unlabeled Contamination: Open-world unlabeled contamination occurs when the unlabeled pool contains both in-distribution and OOD samples unknown to the model during training.
  • B. Open-World Unlabeled Contamination: OOD samples can receive confident predictions and enter training like in-distribution samples, despite semantically invalid supervision.
  • B. Open-World Unlabeled Contamination: The paper distinguishes near-OOD samples, which are semantically closer to the target distribution, from more clearly unrelated far-OOD samples.The taxonomy follows OpenOOD.

C. Accuracy Masking

Under open-world contamination, clean test accuracy can obscure deterioration in prediction behavior, feature organization, and optimization compatibility. Accuracy masking therefore motivates diagnostics that compare these internal learning signals.

  • Clean in-distribution accuracy may no longer sufficiently measure reliability under open-world unlabeled contamination.
  • Confident OOD pseudo-labels can alter predictions, features, and optimization before clean accuracy visibly changes.
  • Accuracy masking describes similar observable accuracy alongside degraded underlying learning dynamics.
  • C-Score diagnoses hidden degradation by checking pseudo-label behavior, alignment with labeled class structure, and compatibility with supervised optimization.
  • C-Score separates macro-level unlabeled prediction metrics from micro-level comparisons with labeled anchors or supervised optimization signals.PLE and CCI are macro-level; Sem-Drift and Grad-Align are micro-level.

A. Prediction Space Metrics: PLE and CCI

PLE and CCI summarize global unlabeled prediction behavior. PLE measures predictive uncertainty, while CCI measures concentration of predictions across target classes.

  • PLE measures the average entropy of unlabeled predictive distributions.
  • Low PLE indicates sharp predictions, whereas high PLE indicates diffuse and uncertain predictions.PLE is computed from softmax probabilities without threshold filtering.
  • CCI quantifies whether unlabeled predictions concentrate on a few classes.
  • CCI is defined as the KL divergence from the average predicted class distribution to the uniform distribution.
  • Small CCI indicates balanced class usage, while large CCI indicates concentration in a small subset of classes.Under contamination, such concentration suggests class-wise sinks.

B. Feature Space Metric: Semantic Drift

Sem-Drift detects whether pseudo-labeled unlabeled representations remain aligned with labeled semantic anchors, complementing prediction-space statistics.

  • B. Feature Space Metric: Semantic Drift: Prediction sharpness can coexist with representation drift away from labeled class anchors.This motivates evaluating feature structure separately from prediction distributions.
  • B. Feature Space Metric: Semantic Drift: Sem-Drift measures deviation between labeled semantic anchors and masked pseudo-labeled unlabeled representations in logit space.It compares class-wise centroids and averages distances over classes with both centroids defined.
  • B. Feature Space Metric: Semantic Drift: For each class, the metric compares a labeled centroid with a masked pseudo-labeled unlabeled centroid.The confidence mask determines which unlabeled samples contribute to the pseudo-labeled centroid.
  • B. Feature Space Metric: Semantic Drift: Small Sem-Drift indicates alignment with labeled anchors, whereas large Sem-Drift indicates feature-space corruption.The implementation uses per-class L2 distances between the two logit centroids.

C. Optimization Space Metric: Gradient Alignment

Grad-Align probes whether unlabeled optimization remains compatible with the supervised objective by comparing labeled and unlabeled gradients.

  • C. Optimization Space Metric: Gradient Alignment: Grad-Align measures compatibility between optimization signals induced by unlabeled supervision and the supervised objective.It is used as a lightweight probe of supervised-unlabeled gradient compatibility.
  • C. Optimization Space Metric: Gradient Alignment: The metric compares gradients of labeled and unlabeled losses with respect to the final fully connected classifier layer.The relevant parameters are denoted W_fc.
  • C. Optimization Space Metric: Gradient Alignment: Grad-Align is defined as the cosine similarity between flattened labeled and unlabeled gradients.This follows the implementation’s gradient extraction and similarity computation.
  • C. Optimization Space Metric: Gradient Alignment: Positive Grad-Align indicates reinforcement, near-zero values weak coupling, and negative values direct gradient conflict.These interpretations describe how unlabeled optimization relates to supervised learning.

D. Oracle Metric: OOD Filtration Failure

OOD-FF is an oracle diagnostic that measures whether OOD samples pass the confidence mask relative to in-distribution samples.

  • D. Oracle Metric: OOD Filtration Failure: The metric is used to verify whether OOD contamination enters training through the confidence filter.C-Score reports OOD-FF for controlled contamination analysis rather than as a standard deployable signal.
  • D. Oracle Metric: OOD Filtration Failure: OOD-FF quantifies the relative rate at which OOD and ID unlabeled samples pass the confidence mask.It requires ground-truth ID/OOD partition labels and is therefore an oracle-only metric.
  • D. Oracle Metric: OOD Filtration Failure: A small OOD-FF indicates effective OOD suppression, whereas a large value indicates masking failure relative to ID samples.The interpretation follows the relative pass rates of OOD and ID samples.

IV. EXPERIMENTAL RESULTS

The experiments evaluate pseudo-label-based SSL across datasets, OOD sources, contamination protocols, algorithms, and repeated training runs.

  • IV. EXPERIMENTAL RESULTS: CIFAR-10 and CIFAR-100 provide the primary and harder 100-class ID settings, respectively.CIFAR-10 uses 40 labeled samples and 50,000 unlabeled ID images; CIFAR-100 uses 400 labeled samples and 50,000 unlabeled training images.
  • IV. EXPERIMENTAL RESULTS: Six OOD sources are tested, with CIFAR-100 and STL-10 classified as near-OOD and four others as far-OOD.The far-OOD sources are SVHN, MNIST, Gaussian Noise, and Textures.
  • IV. EXPERIMENTAL RESULTS: The contamination protocol fixes the ID unlabeled pool and varies OOD samples across r∈{0.1, 0.2, 0.3, 0.4, 0.5}.This controls the amount of ID unlabeled data while changing contamination.
  • IV. EXPERIMENTAL RESULTS: Four pseudo-label-based methods are compared: FixMatch, FlexMatch, SoftMatch, and DS3L.They represent fixed-threshold, class-adaptive, soft-weighted, and other pseudo-label control mechanisms described in the study.

B. Main Evidence of Accuracy Masking

Under OOD contamination, internal diagnostic metrics expose degradation that clean accuracy may obscure. CCI provides the clearest separation, while diagnostic behavior also differentiates SSL algorithms beyond end-task accuracy.

  • Main evidence: CCI most consistently reveals hidden degradation under severe OOD contamination, even when accuracy changes remain limited.Near-OOD sources generally produce larger accuracy drops, while far-OOD sources are more often suppressed by confidence thresholding.
  • Diagnostic trends: CCI shows the clearest and most stable separation across contamination levels in FixMatch on CIFAR-10 with SVHN contamination.Sem-Drift and PLE provide complementary evidence, whereas Grad-Align remains close to zero and separates weakly.
  • Algorithm comparison: All four SSL algorithms exhibit increasing CCI as contamination rises, although their absolute levels and rates of increase differ.FlexMatch and SoftMatch maintain lower absolute CCI than FixMatch and DS3L, while DS3L increases most sharply at high contamination.
  • Algorithm comparison: Clean-accuracy ranking can be incomplete because methods with comparable accuracy may differ substantially in internal stability under contamination.The comparison supports evaluating robustness with both accuracy and diagnostic signals.

E. Effect of OOD Source Type

OOD source type affects both accuracy degradation and whether internal deterioration remains visible through clean accuracy alone. Near-OOD sources cause larger accuracy losses, while some far-OOD sources strongly increase CCI despite moderate accuracy changes.

  • Source-type effects: SVHN, MNIST, and Gaussian Noise can strongly increase CCI despite only moderate accuracy loss.Thus, source type influences whether degradation remains hidden from clean accuracy alone.
  • Evaluation on CIFAR-100: On CIFAR-100 classification under SVHN contamination, clean accuracy changes from 43.21% to 39.22% while CCI rises from 0.161 to 1.430, an approximately 787% relative increase.This result extends accuracy masking to a harder 100-class setting.
  • Implication: C-Score supports moving beyond accuracy-only evaluation by revealing hidden degradation across prediction, feature, and optimization perspectives.CCI shows the most consistent sensitivity across settings.

A. Full OOD-Source × Contamination-Ratio Sweep (FixMatch, CIFAR-10)

The appendix expands the diagnostic evaluation with full contamination-ratio sweeps, covering OOD-source variation, CIFAR-100 SVHN contamination, and algorithm comparisons under SVHN contamination.

  • OOD source × contamination ratio: Table A1 reports FixMatch on CIFAR-10 across the full metric sweep by OOD source and contamination ratio.
  • CIFAR-100 SVHN sweep: Table A2 adds PLE, Sem-Drift, Grad-Align, and OOD-FF to the CIFAR-100 FixMatch SVHN sweep alongside best accuracy and CCI.
  • Algorithm comparison: Table A3 reports all six metrics for FixMatch, FlexMatch, SoftMatch, and DS3L under SVHN contamination on CIFAR-10.
  • Algorithm comparison: The SVHN algorithm comparison spans contamination ratios r ∈ {0.0, ..., 0.5}, with r = 0.0 denoting each algorithm’s clean baseline.
Loading 2608.20667v1…