Source-linked AI summary
G2D: Generative-to-Discriminative Collaborative Inference for Zero-Shot Image Classification
Zehua Hao, Fang Liu, Qinliang Wang, Yaoyang Du, Xinyan Huang, Puhua Chen
TL;DR
Zero-shot classification must combine efficient label retrieval with fine-grained visual reasoning, but CLIP and generative VLMs fail in complementary ways. G2D therefore uses a generative VLM to verify CLIP-retrieved candidates, reaching 68.85% average top-1 accuracy across eight benchmarks with fixed, label-free deployment routing.
Problem
CLIP often retains the correct class in its top-K despite a wrong top-1, while standalone generative VLMs face large label spaces, unconstrained outputs, and no candidate structure.
Method
G2D is a training-free framework that uses CLIP’s candidate prior and confidence routing to direct a generative VLM’s constrained, image-grounded verification.
Results
68.85% average top-1 accuracy across eight benchmarks, with strict G2D(1θ) improving over CLIP and matching or exceeding Qwen3-VL throughout.
Takeaways & Limitations
The framework combines efficient shortlist recall with image-grounded ambiguity resolution while preserving valid predictions without task-specific training.
Takeaways & Limitations
G2D requires both CLIP and a generative VLM, incurs VLM latency, and assumes a useful discriminative shortlist within a closed vocabulary.
Abstract
from arXiv · showhide
Zero-shot classification needs efficient label retrieval and fine-grained visual reasoning, yet discriminative and generative vision-language models fail in complementary ways.When CLIP's top-1 prediction is wrong, the correct label often remains in its top-$K$ shortlist, making disambiguation rather than recall the key challenge.Standalone generative models, however, are hindered by large label spaces and unconstrained outputs.This complementarity motivates separating broad candidate retrieval from fine-grained, image-grounded verification.We propose G2D, a training-free framework that uses a generative VLM to verify CLIP-retrieved candidates against the image.Candidate names and CLIP probabilities provide a structured prior for resolving visually similar classes.Fixed confidence routing, entropy-adaptive candidate sizing, and trie-constrained decoding focus generative reasoning on uncertain samples and ensure one valid output for each input at test time.Across eight benchmarks, G2D achieves 68.85% average accuracy, versus 59.35% for CLIP and 63.11% for the standalone VLM.Across seven generator configurations, candidate-set verification improves average accuracy by 1.08--27.42 percentage points.G2D also transfers to DCLIP, WaffleCLIP, and CuPL, supporting a practical interface between discriminative proposal and generative visual reasoning. Code: https://github.com/Harzva/G2D
1 Introduction
G2D addresses zero-shot classification by separating CLIP’s broad candidate retrieval from generative, image-grounded verification. Its routing and constrained decoding mechanisms use complementary model strengths while restricting generation to valid labels.
- CLIP’s top-K often contains the ground-truth label when top-1 is wrong, making residual disambiguation the central challenge.
- Generative VLMs can reason from fine-grained image details, but large label spaces, unconstrained outputs, and absent candidate structure hinder standalone classification.Over 30% of unconstrained outputs fall outside the valid label set on fine-grained datasets.
- G2D uses CLIP to define a candidate space and confidence prior while a generative VLM verifies shortlisted classes against the image.
- Entropy-adaptive candidate sizing adjusts K to uncertainty, CLIP probabilities enter the prompt, and trie-constrained decoding restricts outputs to valid class names.These mechanisms target uncertain samples and reduce semantic drift from unconstrained generation.
- G2D’s main result uses one fixed hyperparameter configuration across datasets without few-shot filtering, validation tuning, or training-time adaptation.Selected two-threshold routing uses benchmark labels and is reported separately as sensitivity analysis.
2 Related Work
Prior work improves discriminative matching, evaluates generative recognition, or adapts foundation models with additional data. G2D differs by combining CLIP shortlist structure with constrained generative verification in a training-free zero-shot protocol.
- Prompt-enrichment methods refine CLIP text prototypes but do not give the language model access to the test image.
- G2D constrains a generative VLM to CLIP’s candidate labels and uses image-grounded verification rather than open-ended label generation.
- Unlike few-shot adaptation and collaborative inference methods using labeled support images, G2D uses no support images or parameter updates.
3 Method
G2D implements collaboration as a cascaded pipeline: CLIP builds the candidate prior and routes samples, while a generative VLM verifies within that constrained space.
- The framework consists of three cascaded stages that operationalize complementarity between discriminative and generative vision-language models.
- CLIP constructs the candidate prior and routes each sample before generative verification occurs within the constrained candidate space.
3.1 Problem Setup
The problem setup keeps the label space unchanged and frames zero-shot prediction as selecting the class with the highest CLIP image-text similarity.
- Given an image x and label set C, zero-shot classification predicts a class from C using CLIP image-text similarity.
- G2D changes the inference strategy without changing the underlying label space.
3.2 G2D Pipeline
G2D uses CLIP to extract a confidence-aware candidate shortlist, then applies a constrained generative verifier to uncertain samples. The verifier receives the image, candidate names, and CLIP priors, while constrained decoding keeps the output within valid class names.
- CLIP provides class probabilities and a candidate set S_K(x) for each image.
- The shortlist reduces the effective generative label space from |C| to K and preserves CLIP’s ranked confidence prior.
- Samples with CLIP top-1 probability above θ_high use the CLIP route; others invoke the collaborative verifier.
- The verifier uses the image, candidate names, and CLIP probabilities to decide among shortlisted classes rather than searching the full label space.
- +0.06 points from implicit thinking came with about 1.8× greater wall-clock time per sample, so direct-answer mode is used by default.
3.3 Three Mechanisms for Effective Collaboration
G2D combines adaptive candidate sizing, CLIP-prior injection, and constrained decoding to make generative verification focused and label-valid. These mechanisms adjust the interface to uncertainty, preserve useful discriminative information, and prevent semantic drift.
- The naive cascade of running the VLM only after CLIP failures is insufficient; G2D instead uses three collaboration mechanisms.
- Entropy-Adaptive Candidate Sizing: Entropy-adaptive sizing uses smaller candidate sets at low entropy and larger sets at high entropy to maintain ground-truth coverage.
- CLIP Probability Injection: CLIP probability injection gives the verifier a distributional prior that it can follow when informative or override when image evidence contradicts it.
- Trie-Constrained Decoding: 31.3% semantic drift on FGVC-Aircraft and 30.2% on Stanford Cars occur without constrained decoding.
- Trie-Constrained Decoding: Exact, fuzzy, and trie-based projection ensures outputs are valid labels in S_K(x) and enables standard exact-match evaluation.
3.4 Unified Decision Rule
G2D defines one-threshold and two-threshold routing variants that assign samples to CLIP, a standalone generator, or a verifier according to confidence. The two-threshold variant is calibrated analysis, while the one-threshold variant is the simplest strict deployment form.
- G2D(1θ): G2D(1θ) uses a single θ_high threshold to split samples between a CLIP route and a verifier route.
- G2D(2θ): G2D(2θ) uses θ_high and θ_low to partition samples among CLIP, standalone generator, and verifier experts.
- G2D(2θ): Route A trusts CLIP at high confidence, Route B uses the standalone generator at very low confidence, and Route C invokes verification otherwise.
- G2D(2θ): Selected two-threshold pairs use benchmark labels and are therefore excluded from strict no-label claims.
- Verifier Output: Route C projects decoded outputs to valid labels using exact matching followed by fuzzy or trie fallback.
3.5 Default Zero-Shot Configuration
The default strict zero-shot configuration uses shared settings across datasets and routes samples between CLIP and the verifier with a fixed threshold. A separate two-threshold analysis uses benchmark labels and is not a no-label deployment protocol.
- Shared Configuration: Adaptive K∈[3, 10], τ_low=0.5, τ_high=2.0, beam size=5, and no validation tuning are shared across datasets.
- Strict Deployment: G2D(1θ) uses θ_high=0.70 fixed before deployment and sends samples to either CLIP or the verifier without validation or test-label selection.
- Calibrated Analysis: G2D(2θ) thresholds are selected by grid search over cached predictions using benchmark labels, making the analysis calibrated rather than no-label deployment.
4 Experiments
Experiments evaluate G2D under a shared pure zero-shot protocol across eight datasets, comparing standalone generation, candidate verification, routing, complementarity, and deployment cost. The results show gains from combining CLIP’s candidate prior with generative verification, while exposing an accuracy–cost trade-off and remaining practical bottlenecks.
- Experimental setting: Eight zero-shot datasets are evaluated, with a four-dataset subset used for routing diagnostics and matched intermediate analyses.The main benchmarks are EuroSAT, DTD, Oxford-Pets, CUB200, Food-101, ImageNet, ImageNetV2, and Places365.
- Main results: 68.85% average accuracy is achieved by strict G2D(1𝜃), improving over CLIP on all eight datasets and matching or improving Qwen-only on all eight.With Qwen3-VL-8B fixed, Gene-only reaches 63.11%, Verifier-only 64.19%, and G2D(1𝜃) 68.85%.
- Complementarity analysis: 56.4% average Complementarity Index and 10.3% top-20 confusion-pair overlap indicate substantial recoverable disagreement between CLIP and Qwen across eight benchmarks.The oracle reaches 74.62% average accuracy, compared with 59.35% for CLIP and 63.11% for Qwen.
- Protocol and attribution: The main no-label configuration fixes 𝜃high = 0.70 before deployment without validation or test-label feedback, while G2D(2𝜃) is reported only as label-calibrated sensitivity analysis.The controlled attribution orders Gene-only, Verifier-only, and fixed routing using the same Qwen3-VL-8B.
- Routing and deployment: Strict G2D invokes the VLM on 22.9–85.8% of images in the four-dataset deployment analysis, framing routing as an accuracy–cost trade-off.High-confidence samples go to CLIP, while remaining samples go to the verifier.
- Limitations and interpretation: G2D’s realized accuracy remains below the oracle because CLIP may omit the ground-truth label from top-𝐾, routing may retain ambiguous samples, and verification may still fail.These bottlenecks motivate interpreting results together with routing diagnostics rather than treating complementarity as sufficient for oracle-level performance.
4.5 Ablation
Ablations show that candidate-set verification remains beneficial across generator and discriminator variants, while routing performance is generally stable around the shared threshold. The method still carries inference-cost and proposal-quality constraints.
- Verifier and routing ablations: +1.08 to +27.42 accuracy points are gained by candidate-set verification across tested generator configurations.Qwen3-VL-8B gains +1.08, while Qwen3.5-0.8B gains +27.42; LLaVA-1.5-7B gains +21.11 on six jointly supported datasets.
- Verifier and routing ablations: Every evaluated vanilla-CLIP configuration improves with both routed and no-route G2D variants.On DTD, gains reach +18.97 routed / +19.91 without routing; on EuroSAT, +13.84 / +13.96.
- Discriminator modularity: G2D remains effective when the discriminator prior changes to DCLIP, WaffleCLIP, CuPL, or Comparative-CLIP.WaffleCLIP produces the largest reported DTD gain: +21.75 routed / +22.57 without routing.
- Threshold sensitivity: Threshold-sensitivity curves are generally smooth, with most optimal thresholds clustered between 0.7 and 0.9.The main comparison retains the shared θ_high=0.7 rather than selecting a threshold per method or dataset.
- Limitations: G2D requires both CLIP and a generative VLM, and its latency increases when the verifier-route ratio is high.The method also assumes closed-vocabulary recognition and a useful discriminative proposal prior; missing shortlist labels or misleading priors can cause verification failure.
5 Conclusion
G2D is a training-free framework that combines discriminative shortlist retrieval with generative, image-grounded verification. With a fixed threshold and no labeled validation or test feedback, it reaches 68.85% average top-1 accuracy across eight benchmarks and supports valid predictions.
- Conclusion: G2D uses a generative VLM as an image-grounded verifier over a discriminator-defined shortlist.Its mechanisms include entropy-adaptive candidate sizing, fixed confidence routing, prior injection, and trie-constrained decoding.
- Conclusion: 68.85% average top-1 accuracy is achieved across eight benchmarks with θ_high=0.70 fixed before deployment.Strict G2D(1θ) improves over CLIP and matches or exceeds Qwen3-VL throughout without labeled validation or test feedback.
- Conclusion: CLIP supplies efficient shortlist recall while the verifier resolves residual ambiguity through image-grounded reasoning.The division combines discriminative retrieval and constrained generation without task-specific training while preserving valid predictions.