PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Full Conformal Image Classifiers

Julio Silva-Rodríguez, Ender Konukoglu

arXiv:2609.37298v1cs.LGcs.CV

TL;DR

Split conformal prediction is data-inefficient, while full conformal prediction is computationally costly in large label spaces because it refits for candidate labels. The paper combines VLM-guided label pruning with an efficient LDA adaptation solver; experiments report modest ImageNet latency and more stable empirical coverage than split conformal alternatives.

  • Problem

    Split conformal prediction uses calibration-data splits, whereas full conformal prediction requires candidate-specific refits that make it computationally burdensome at scale.

    Source: §1 · pp. 1–2

  • Method

    T-FCP uses a zero-shot VLM-guided inductive conformal predictor to prune labels before FCP, while SO-LDA accelerates the repeated adaptation fits.

    Source: §1 · p. 3

  • Results

    Across benchmarks, T-FCP yields more stable coverage distributions than split conformal procedures, with ∼30 ms/image latency on ImageNet.

    Source: §1 · p. 3, §4.2 · p. 7

  • Takeaways & Limitations

    The experiments support using data-efficient full conformal procedures for more stable empirical coverage in conformal image classification.

    Source: §5 · p. 9

    The SO-LDA solver is less performant than iterative gradient-based methods, and combining it with T-FCP does not retain a theoretical coverage guarantee.

    Source: §5 · p. 9

Abstract

from arXiv · show

Conformal prediction provides set-valued predictions with distribution-free coverage guarantees, making it attractive for high-stakes image classification. However, split conformal prediction is data-inefficient, while full conformal prediction (FCP), despite its stronger statistical efficiency, is computationally prohibitive at scale because it requires candidate-specific model refits at test time. We address this limitation by leveraging zero-shot vision-language models (VLMs) to guide scalable FCP in large label spaces. We introduce Targeted Full Conformal Prediction (T-FCP), which uses a lightweight inductive conformal predictor to prune unlikely labels and applies FCP only to the remaining candidates, reducing computation while retaining the formal guarantee of the combined conformal procedure. We further propose Stabilized Online LDA (SO-LDA), an efficient VLM adaptation solver based on rank-one inverse-covariance updates. Across multiple benchmarks, including ImageNet, T-FCP enables practical full-conformal image classification with modest test-time overhead, yielding efficient prediction sets and more stable empirical coverage than split conformal alternatives.

1 Introduction

Conformal prediction returns label sets with distribution-free coverage guarantees, but split calibration can waste data and full conformal prediction requires candidate-specific refits that are costly in large label spaces. The section introduces VLM-guided Targeted Full Conformal Prediction and SO-LDA to make full conformal image classification more practical.

1 Introduction

  • Split conformal prediction is data-inefficient because it reserves only a fraction of available data for training and quantile estimation, and small calibration sets can yield noticeable coverage deviations.At α = 0.1, empirical coverage may be closer to 85% than the 90% target.Source: §1 · p. 2
  • Full conformal prediction uses all available labeled calibration data for fitting and quantile estimation, but tests every candidate label and refits a classifier for each at test time.The fitting function must be symmetric on the augmented dataset to maintain score exchangeability.Source: §1 · p. 2
  • Zero-shot VLM class prototypes provide label-space proxies that can guide FCP toward likely labels and exclude dissimilar categories from per-image testing.The motivation is that large label spaces include both weakly related classes and groups of similar categories.Source: §1 · p. 3
  • T-FCP applies FCP to an ICP-guided subset of labels, with a combined coverage guarantee and no calibration-data split; SO-LDA uses incremental class-prototype and rank-one inverse-covariance updates.SO-LDA is reported to approach gradient-based solver performance while running orders of magnitude faster.Source: §1 · p. 3
  • T-FCP and SO-LDA enable full conformal image classification on large-scale datasets, with ∼30 ms/image latency on ImageNet and more stable coverage distributions than split conformal procedures.T-FCP prunes unlikely labels, while SO-LDA reduces the cost of repeated model fitting.Source: §1 · p. 3

2 Related Work

Prior image-classification conformal work mostly follows split conformal prediction, while full conformal classification and VLM adaptation have been explored in more limited settings. The section situates this work among computational approaches to FCP and methods combining conformal predictors.

Full conformal prediction

  • Classification FCP work has used online kNN and SVM updates or influence-function approximations, while ordering-based reductions developed for regression do not directly transfer to classification.Regression approaches exploit natural response ordering through discretization or homotopy/continuation.Source: §2 · p. 3

Conformal prediction for image classification

  • Most neural-network image-classification conformal methods use split conformal prediction, emphasizing set efficiency, adaptiveness, class-conditional coverage, or robustness rather than calibration-data efficiency and coverage variability.These methods typically rely on in-domain training data and separate calibration splits.Source: §2 · p. 3

Conformal prediction for zero-shot models

  • Prior VLM conformal work includes split-conformal adaptation and a full-conformal closed-form linear solver, but the latter focused on datasets with C < 50 and its SS-Text solver does not scale well to large label spaces and fine-grained tasks.The passage identifies [49] as the small-scale full-conformal work.Source: §2 · p. 3

Combining conformal predictors

  • Conformal cascades prune candidate sets progressively, while other combined-predictor methods obtain guarantees through independence-based bounds or error-rate allocation such as Bonferroni corrections.The cited applications include label-dependent guarantees for object detection and joint guarantees across multiple outputs.Source: §2 · p. 3

3 Methods

The methods use zero-shot vision-language prototypes to guide full conformal prediction, pruning unlikely labels while retaining a combined coverage guarantee. Stabilized Online LDA then makes candidate-specific classifier updates more efficient and stable.

3.1 Background on zero-shot image classifiers

  • A visual encoder maps an image to a feature vector, while a text encoder turns class descriptions into prototype embeddings used with normalized image embeddings to produce class probabilities.The class prototypes are formed from textual embeddings of class descriptions.Source: §3.1 · pp. 3–4

3.2 Targeted Full Conformal Prediction

  • The combined coverage bound can be conservative: with overlapping miscoverage events, intersection coverage may exceed 1 − (α_1 + α_2).The bound is tight when the two miscoverage events are disjoint and have probabilities α_1 and α_2.Source: §3.2 · p. 4
  • T-FCP intersects an ICP set based on zero-shot weights with an FCP set fitted on each candidate-augmented dataset, preserving the combined marginal coverage guarantee.The ICP stage prunes excluded labels so the more expensive FCP stage runs only on surviving candidates; α_ICP controls the computational-statistical tradeoff.Source: §3.2 · p. 4

3.3 SO-LDA: Stabilized Online LDA

  • Standard gradient-based softmax cross-entropy solvers are generally better-performing, but their O(TNFC) cost makes them infeasible for full conformal prediction.The method therefore builds on a training-free LDA solver for VLM embeddings.Source: §3.3 · p. 5
  • SO-LDA uses rank-one covariance updates to reduce each candidate-label update to O(F^2), rather than repeated O(F^3) matrix inversion or recomputing the LDA solution from scratch.Parameters are estimated from calibration data and updated for each augmented dataset.Source: §3.3 · p. 5
  • SO-LDA centers features using zero-shot prototypes, keeping residual centering invariant under candidate augmentation, and adds a text-guided prior to stabilize updated class means, especially with little data.The text-guided prior is weighted by λ_TEXT.Source: §3.3 · p. 5

4 Experiments

Across 11 CLIP-based datasets, T-FCP improves median set size and singleton rate over SCP while producing more consistent empirical coverage. Its pruning and SO-LDA solver make ImageNet-scale full conformal prediction practical, with further studies examining data regimes, solver performance, and backbone generalization.

4.1 Setup

  • Experiments span 11 heterogeneous datasets, using CLIP ViT-B/16 prototypes and label-marginal calibration sets of N = C × 16 by default, repeated over 50 random seeds.The study also evaluates other CLIP and MetaCLIP backbones and smaller calibration samples.Source: §4.1 · p. 6

4.2 Main results

  • T-FCP achieves valid coverage in 81.8% of runs at α = 0.10 and 88.7% at α = 0.05, while improving median set size and singleton rate versus SCP.Table 1 averages results across 11 datasets; Figure 1 analyzes empirical coverage and set-size distributions.Source: §4.2 · pp. 6–7, Table 1 · p. 7, Figure 1 · p. 7
  • On ImageNet, full conformal latency falls from ∼500 s per image with iterative GD to ∼0.33 s with SO-LDA and ∼30 ms with T-FCP.Testing 100 labels in parallel uses a peak ∼16 Gb of GPU memory.Source: §4.2 · p. 7, Figure 2 · p. 8

4.3 In-depth studies

  • With αICP near 1%, pruning removes most labels; gains track zero-shot accuracy and are especially large on heterogeneous, large-scale label spaces.Figure 3 examines pruning across datasets.Source: §4.3 · pp. 7–8, Figure 3 · p. 8
  • Increasing αICP accelerates FCP but sacrifices set efficiency; all settings reduce finite-sample coverage variance similarly to ICP, while set-efficiency gains over SCP require αICP < 2.5%.The choice of αICP lets users trade latency against set efficiency.Source: §4.3 · p. 8, Figure 3 · p. 8
  • Across smaller calibration regimes, T-FCP has more consistent coverage than SCP and more efficient sets; at N = C × 4, FCP shows slight undercoverage and SCP has ∼±4.5 coverage deviation.Coverage dispersion decreases for all procedures as calibration data increase.Source: §4.3 · p. 8, Figure 4 · p. 9
  • SO-LDA’s zero-shot-guided centering beats vanilla LDA and comes within ∼1.5% of iterative GD; keeping prototypes near initialization stabilizes low-data gains.Simple-shot, which relies solely on class centers, falls short of the best-performing GD method.Source: §4.3 · pp. 8–9, Figure 4 · p. 9
  • Across CLIP and MetaCLIP backbones, T-FCP coverage variance is ∼30% smaller than SCP; on MetaCLIP ViT-L/14, median set size falls 31% without sacrificing coverage stability.The reported benefits also hold for smaller backbones such as ViT-B/32.Source: Figure 5 · p. 9, §4.3 · p. 9

5 Discussion

T-FCP and SO-LDA enable full conformal prediction on large-scale image datasets, and the experiments report more stable empirical coverage than split-based alternatives. The discussion also notes limits to SO-LDA's guarantee and cautions that marginal coverage does not ensure subgroup coverage or protection under distribution shift.

5 Discussion

  • Data-efficient procedures yield more stable empirical coverage than split-based alternatives, while enabling full conformal prediction on large-scale image datasets.The authors identify data efficiency as important for conformal image classification.Source: §5 · p. 9
  • SO-LDA's diagonal-loading approximation means T-FCP's theoretical coverage guarantee does not carry over to the combined solver, despite negligible differences from exact implementations in ablations.SO-LDA is less performant than iterative gradient-based methods, and T-FCP sets are not guaranteed to be more efficient than SCP sets.Source: §5 · p. 9

A Broader Impacts

  • Marginal coverage does not imply conditional or subgroup coverage, and guarantees need not hold under distribution shift.Finite-sample empirical coverage can also fall below the nominal level in real-world deployment, especially with data-inefficient pipelines.Source: A Broader Impacts · p. 14

B Proofs

The proof of Proposition 1 uses the marginal coverage guarantees for the individual procedures to establish the stated result.

B Proofs

  • The proof of Proposition 1 begins from Eq. 9.Source: B Proofs · p. 14
  • The argument uses the marginal coverage guarantees for each individual procedure in Eq. 8.Source: B Proofs · p. 14
  • The proof then states that this implies the result.Source: B Proofs · p. 14

C Stability of SO-LDA

The SO-LDA stability analysis compares solver configurations and finds that asymmetric residual centering is the dominant source of instability. It also reports K = 4 under-coverage in both online and exact recomputation variants.

C Stability of SO-LDA

  • The analysis examines mild asymmetry from diagonal covariance regularization, whose diagonal is estimated only from calibration data for efficient rank-one updates.Source: C Stability of SO-LDA · p. 14
  • The compared variants include fixed-Ridge loading, full recomputation of class means and covariance, and an Unstable variant that omits zero-shot prototypes for residual estimation.The Non-online variant has per-label cost O(NF^2 + F^3).Source: C Stability of SO-LDA · p. 14
  • The Unstable configuration has the largest marginal coverage gap and lower valid-occurrence ratio, while SO-LDA's performance is similar to more expensive exact Non-online updates.Fixed-Ridge gives the cleanest symmetric online updates but lower discriminative performance than SO-LDA; both have similar coverage distributions.Source: C Stability of SO-LDA · p. 14
  • K = 4 under-coverage occurs with both online and exact recomputation, suggesting diagonal-loading approximation is not its primary cause.The authors note similar instability in prior work and leave complete characterization of this regime to future work.Source: C Stability of SO-LDA · p. 14

D Datasets

The appendices describe the dataset overview and the scope of the solver-configuration study, then give implementation details for three baselines.

D Datasets

  • Table 3 summarizes the datasets, class counts, tasks, and test-partition sample counts; its balance labels use “b” and “i”.Source: D Datasets · p. 14, Table 3 · p. 15
  • The SO-LDA configuration study averages results over 10 datasets, uses 20 random seeds, and excludes ImageNet.Source: Table 2 · p. 15

E Baselines

  • The gradient-descent linear probe initializes class weights with zero-shot prototypes and trains for 300 full-batch iterations using cross-entropy.It uses SGD with momentum 0.9 and cosine annealing from a base learning rate of 0.1.Source: E Baselines · p. 15
  • The Conf-OT baseline estimates label-marginal distributions from calibration data and applies optimal transport to joint calibration and test sets.Its entropic weight is τ = 1 and TOT = 3; the TIM baseline instead optimizes mutual information on joint calibration and test samples for 300 iterations.Source: E Baselines · p. 15

F Detailed Results

This section points readers to detailed conformal-prediction results across datasets and additional configurations. Table 4 covers configurations omitted from Table 1, while Table 5 reports per-dataset averages for the main procedures.

F.1 Additional configurations

  • Table 4 reports CLIP ViT-B/16 conformal-procedure results for configurations not included in Table 1.The configurations use N = C × 16.Source: Table 4 · p. 15
  • Table 5 gives per-dataset average results for the main conformal procedures reported in Table 1 using CLIP ViT-B/16.Source: Table 5 · p. 16, §F.1 · p. 16

F.3 Results using APS

With APS as the nonconformity score, T-FCP retains more stable empirical coverage than ICP and SCP across the reported studies, while remaining consistent on CCV. Adaptive-score sorting also raises latency substantially for large label spaces.

F.3 Results using APS

  • T-FCP with APS has smaller coverage dispersion and a larger ratio of coverage-satisfying realizations than ICP and SCP, while remaining consistent on CCV.Results average 10 datasets, use 20 random seeds, and exclude ImageNet.Source: §F.3 · p. 16, Table 6 · p. 17
  • Nearly 220 ms/image for ImageNet T-FCP with APS, versus nearly 30 ms/image with LAC, makes APS less appealing at the largest scale and vanilla FCP unfeasible.Adaptive APS/RAPS scores require ranking or sorting over the label space within the candidate-wise FCP loop.Source: §F.3 · p. 16

F.4 Extension to unimodal models

T-FCP is model-agnostic in principle when a valid low-cost pruning predictor is available. A ResNet-50 pretrained on ImageNet transfers to ImageNet-V2, where T-FCP prunes many labels and yields more stable empirical coverage than split-conformal alternatives.

F.4 Extension to unimodal models

  • T-FCP can use unimodal models when an initial pruning proxy exists, such as a pretrained classifier over the target label space.The construction allows any valid low-cost conformal predictor for pruning and any valid or efficient FCP scoring rule for the second stage.Source: §F.4 · p. 17
  • 65% of labels are pruned by T-FCP on ImageNet-V2 using an ImageNet-pretrained ResNet-50, with more stable empirical coverage than split-conformal alternatives.The datasets share a label space; Table 7 reports results using LAC and 8 labeled calibration samples per class across 20 seeds.Source: §F.4 · p. 17, Table 7 · p. 17
Loading 2609.37298v1…