PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Fine-Grained Visual Classification with Text as Privileged Information via Multimodal Distillation

Aishwarya Venkataramanan, Christian Ickler, Joachim Denzler

arXiv:2610.32442v1cs.CV

TL;DR

Fine-grained recognition must distinguish highly similar categories, while textual descriptions that could guide recognition are often unavailable at test time. FG-LUPI uses those descriptions to train a multimodal teacher and distills its knowledge into an image-only student. Across six benchmarks, it improves classification performance and lowers ECE compared with vanilla ViT.

  • Problem

    Fine-grained visual classification must distinguish highly similar categories, while multimodal methods typically require textual inputs at inference, which can be infeasible in practical deployment settings.

    Source: §1 · p. 1

  • Method

    FG-LUPI uses textual attributes as training-time privileged information for a multimodal teacher, then distills its knowledge to an image-only student.

    Source: §1 · p. 2

  • Results

    FG-LUPI consistently improves performance over strong ViT-based baselines and competitive methods, and achieves lower ECE than vanilla ViT across all six benchmarks.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results suggest textual attribute supervision can benefit recognition and confidence calibration while preserving image-only inference.

    Source: §1 · p. 2, §5 · p. 9

    The effectiveness of training-time textual supervision depends on the specificity and quality of available descriptions.

    Source: D. Limitations and Future Work · p. 16

Abstract

from arXiv · show

Fine-grained visual classification (FGVC) requires distinguishing between highly similar categories by focusing on subtle, localized visual differences. Recent multimodal methods have demonstrated that textual attribute descriptions can provide valuable semantic supervision by explicitly describing discriminative object characteristics. However, these descriptions are typically unavailable at test time, limiting their direct use in deployment. We propose FG-LUPI, a Learning Using Privileged Information (LUPI)-based framework that leverages attribute descriptions during training. We train a multimodal teacher that integrates image patches and textual attributes via gated cross-attention, allowing semantic attributes to condition visual representations. We distill this knowledge into an image-only student using multi-level distillation at the prediction, patch, and attention levels, aligning global predictions, patch representations, and attention patterns with the teacher. This encourages the student to inherit attention structure learned from multimodal supervision while maintaining image-only inference. Experiments on multiple FGVC benchmarks show consistent improvements in classification accuracy over strong ViT-based baselines. Furthermore, evaluation using Expected Calibration Error (ECE) shows that FG-LUPI also produces better-calibrated predictions across all six benchmarks. These results suggest that textual attribute supervision can benefit not only fine-grained recognition performance but also the reliability of model confidence, while preserving the efficiency and practicality of image-only inference.

1. Introduction

FG-LUPI treats textual attributes as training-only privileged information, transferring multimodal teacher knowledge to an image-only student. Across fine-grained benchmarks, it improves classification performance and produces lower ECE than vanilla ViT.

1. Introduction

  • FGVC is difficult because categories are highly similar and vary within categories, requiring models to localize subtle visual differences.Text can describe discriminative attributes such as color patterns, beak shape, and wing structure.Source: §1 · p. 1
  • Because multimodal encoders typically require text at inference, the paper asks whether language can instead shape visual representations only during training.The motivation also asks whether this supervision can improve confidence reliability, which accuracy alone does not measure.Source: §1 · pp. 1–2
  • The paper formulates FGVC as a LUPI problem: textual attributes guide a multimodal teacher during training, and its knowledge is distilled into an image-only student.The authors describe this as the first investigation of semantic textual supervision for deep FGVC under LUPI.Source: §1 · p. 2
  • A multimodal teacher combines textual attributes and visual representations through gated cross-attention, then transfers prediction, patch, and attention knowledge to an image-only student.The student retains image-only inference while learning from the teacher's language-guided attention patterns.Source: §1 · p. 2
  • FG-LUPI consistently improves performance over strong ViT-based baselines and competitive methods across multiple benchmarks, with lower ECE than vanilla ViT on all six benchmarks.Attention maps also show the student focusing on discriminative image regions.Source: §1 · p. 2

2. Related Works

Prior FGVC work localizes discriminative regions through part modeling and attention, while multimodal methods use text but often retain it at inference. FG-LUPI combines training-only text supervision with multi-level distillation into an image-only student.

2. Related Works

  • FGVC methods localize discriminative regions through part annotations [21] [74], weakly supervised learning [12] [26], and attention mechanisms [37] [48].ViT methods also use attention to select informative tokens and refine representations across layers [17] [62].Source: §2 · p. 2
  • Vision-language models and cross-modal fusion use textual descriptions or metadata for FGVC, but text generally remains part of multimodal recognition.FG-LUPI instead transfers text-guided knowledge to an image-only model.Source: §2 · p. 3
  • Knowledge distillation transfers predictions or intermediate representations, with later methods also distilling attention and other relations [20] [42] [71] [52] [50] [78].FG-LUPI distills prediction-, patch-, and attention-level knowledge from a teacher conditioned on textual attributes into an image-only student.Source: §2 · p. 3
  • LUPI uses auxiliary information available during training but unavailable at inference, including prior applications to attention supervision in fine-grained recognition.FG-LUPI applies this setting to textual attributes and transfers the resulting visual knowledge to an image-only student.Source: §2 · p. 3
  • FG-LUPI combines text as training-only privileged information with multi-level distillation to transfer semantic guidance into an image-only model.This bridges multimodal FGVC, which uses text directly, and conventional distillation from visually trained teachers.Source: §2 · p. 3

3. Method

FG-LUPI treats textual attribute descriptions as privileged training-time information and transfers multimodal teacher knowledge to an image-only student. The student aligns predictions, patch representations, and attention patterns, with EMA teacher targets and image-only inference.

3.1. Problem Setting

  • Textual descriptions accompany images and class labels as privileged information during training, while the student is designed to classify images without text.Source: §3.1 · p. 3

3.2. Multimodal Teacher Network

  • The multimodal teacher projects text representations into the image embedding space and uses visual tokens as queries to attend to textual keys and values.Gated cross-attention injects this interaction into the visual stream.Source: §3.2 · pp. 3–4

3.3. Image-Only Student Network

  • Student training aligns teacher and student predictions, patch embeddings, and attention patterns, using patch-wise cosine alignment and row-wise KL for attention.Ground-truth labels also supervise the student with cross-entropy.Source: §3.3 · pp. 4–5

3.4. Training and Inference

  • The teacher and student train simultaneously; an EMA teacher stabilizes distillation targets for all three objectives, and only the student is retained for inference.Source: §3.4 · p. 5

4. Experiments

Across six benchmarks, FG-LUPI improves image-only classification accuracy and calibration over vanilla ViT; experiments also examine attention transfer and the effects of its distillation components.

4.1. Experimental Setup

  • Experiments use six benchmarks, with human-authored CUB descriptions and Gemma-generated descriptions for the other datasets as training-time privileged semantic supervision.Both models use ViT-B/16; the teacher encodes descriptions with a frozen pretrained BERT encoder.Source: §4.1 · p. 5

4.2. Results

  • FG-LUPI improves over vanilla ViT-B/16 by 2.5 percentage points on CUB, 3.5 on Stanford Dogs, and 2.7 on NABirds; it leads on FGVCAircraft and reaches 89.1% on Stanford Cars.On iNaturalist 2021, top-1 accuracy rises from 74.08% to 76.46%.Source: §4.2 · p. 6, Table 2 · p. 7, Table 3 · p. 7
  • With the same teacher and student architectures, FG-LUPI outperforms representative distillation methods by jointly transferring prediction-, patch-, and attention-level information.The compared methods include FitNet, Attention Transfer, Similarity-Preserving, DKD, and CRD.Source: Table 4 · p. 7, §4.2 · p. 7
  • FG-LUPI achieves lower ECE than vanilla ViT across all evaluated benchmarks, indicating better calibration of its image-only classifier.The paper defines lower ECE as better calibration and reports the cross-dataset comparison in Table 5.Source: Table 5 · p. 7, §4.2 · p. 7
  • The image-only student closely replicates the multimodal teacher’s spatial focus, while FG-LUPI maps emphasize object parts and suppress background clutter across datasets.Examples include focus on birds rather than branches or water, and on dogs in images with multiple objects or human-dog interactions.Source: Figure 3 · p. 6, §4.2 · p. 7, Figure 4 · p. 8

4.3. Ablation Study

  • CUB accuracy largely saturates above a loss weight of approximately 0.4, and performance varies little among evaluated pretrained text encoders.BERT performs best, followed closely by RoBERTa and DeBERTa.Source: Figure 5 · p. 8, §4.3 · p. 8, Table 7 · p. 9
  • Combining prediction, patch, and attention distillation yields the best CUB accuracy at 93.03%; prediction distillation alone is the strongest individual objective at 91.64%.Combining prediction with patch or attention supervision reaches 92.42% or 92.15%, respectively.Source: Table 6 · p. 8, §4.3 · p. 8
  • Random vectors reduce accuracy from 93.0% to 91.4%, while removing the EMA teacher or using an image-only teacher also lowers accuracy to 91.2%.The paper attributes the additional gain from meaningful descriptions to complementary semantic supervision.Source: §4.3 · pp. 8–9

5. Conclusion

FG-LUPI uses training-time textual attributes as privileged semantic supervision while keeping test-time inference image-only. Across benchmarks, it improves classification and calibration over the vanilla ViT baseline.

5. Conclusion

  • FG-LUPI consistently improves performance over strong ViT-based baselines and competitive methods across multiple fine-grained recognition benchmarks.It also achieves lower Expected Calibration Error than the vanilla ViT baseline across all six evaluated benchmarks.Source: §5 · p. 9
  • FG-LUPI uses textual attribute descriptions as privileged training-time supervision through a multimodal teacher, while the student uses images alone at test time.The student benefits from complementary semantic supervision without requiring text at inference.Source: §5 · p. 9
  • Qualitative analyses show that the student focuses on meaningful, discriminative image regions, indicating transfer of knowledge acquired from textual supervision.Source: §5 · p. 9
  • Future work will investigate how different privileged information and training objectives influence predictive performance and confidence calibration in fine-grained recognition.Source: §5 · p. 9

A.1. Dataset Description

The evaluation covers six fine-grained benchmarks across birds, dogs, aircraft, cars, and biodiversity, with dataset sizes ranging from 10,000 to 1,000,000 images.

A.1. Dataset Description

  • CUB-200-2011 contains 11,788 images from 200 bird species, while NABirds contains 48,562 images from 555 North American bird categories.CUB-200-2011 has 5,994 training and 5,794 test images; NABirds has 23,929 training and 24,633 test images.Source: §A.1 · p. 13
  • Stanford Dogs contains 20,580 images across 120 breeds, FGVC-Aircraft 10,000 across 100 variants, and Stanford Cars 16,185 across 196 categories.Their train/test splits are 12,000/8,580, 6,667/3,333, and 8,144/8,041 images, respectively.Source: §A.1 · p. 13
  • The iNaturalist 2021 Mini subset spans 10,000 species categories and uses 500,000 training images and 500,000 test images.Source: §A.1 · p. 13

A.2. Privileged Information

The privileged descriptions come from human annotations for CUB-200-2011 and generated text for other datasets, with image-level or class-level generation depending on the dataset.

A.2. Privileged Information

  • For CUB-200-2011, each image has ten human-authored descriptions, and training randomly samples one description per image.The descriptions are provided by Reed et al. [40].Source: §A.2 · p. 13
  • For datasets lacking human-authored descriptions, the authors generate privileged text with multimodal language models; Gemma 4 E4B generates image-level descriptions.Prompts for FGVCAircraft, Stanford Cars, and NABirds request concise structural and morphological features, excluding paint schemes, markings, damage, and environmental context.Source: §A.2 · p. 13
  • For iNaturalist 2021, one class-level description is synthesized per category by Gemma 3 27B from three randomly selected training images.The prompt requests about 50 words describing visible features that distinguish the class from others in its supercategory.Source: §A.2 · p. 13
  • All generated descriptions are used only as privileged information during training and are not required at inference.Source: §A.2 · p. 13

B. Evaluation Metrics

The evaluation uses Top-1 Accuracy for predictive correctness and Expected Calibration Error (ECE) for confidence reliability.

B. Evaluation Metrics

  • Top-1 Accuracy measures the proportion of test samples whose highest-confidence predicted class matches the ground-truth label.The metric is computed over a test set of size N; p_i,c denotes the predicted probability for class c, y_i the ground-truth label, and 1(·) the indicator function.Source: B. Evaluation Metrics · p. 13
  • Expected Calibration Error measures the discrepancy between prediction confidence and empirical accuracy across M confidence bins.Each bin B_m contains samples whose predicted confidence falls in that bin; its accuracy and confidence are averaged over those samples.Source: B. Evaluation Metrics · pp. 13–14
  • Lower ECE indicates better calibration, and an ECE of zero corresponds to perfectly calibrated predictions.Source: B. Evaluation Metrics · p. 14

C. Additional Results

Additional analyses find that FG-LUPI’s attention localizes relevant object regions, including across datasets, and that performance is similar with human- and LLM-generated descriptions. The image-only student also compares favorably with evaluated vision-language models, while the multimodal teacher adds training-time cost but no inference-time computation.

Cross-dataset attention generalization

  • On NABirds images unseen during training, FG-LUPI concentrates attention on the bird and discriminative anatomical regions, unlike vanilla ViT’s frequent focus on background context.The models were trained exclusively on CUB-200-2011.Source: C. Additional Results · p. 14

Text-conditioned attention visualization

  • Different attribute queries produce distinct cross-attention responses on the same image, such as “black eyes” focusing on the head and “yellow breast” on the chest.Some attributes partially activate adjacent regions because token-level semantics can combine multiple cues.Source: C. Additional Results · p. 14, Figure 7 · p. 14

Additional Qualitative Results

  • Across birds, dogs, aircraft, and cars, FG-LUPI’s student shows more concentrated attention on foreground objects and less on surrounding backgrounds than vanilla ViT.The comparison is shown in the qualitative attention maps.Source: C. Additional Results · p. 15, Figure 8 · p. 15

Impact of Text Source

  • 92.8 ± 0.2% accuracy with human descriptions and 92.6 ± 0.1% with Gemma-generated descriptions on CUB-200 indicate comparable performance across text sources.The authors interpret this as robustness to the source of semantically relevant textual supervision.Source: C. Additional Results · p. 15

Comparison with Vision-Language Models

  • FG-LUPI student achieves the best performance on CUB, Dogs, NABirds, and FGVCAircraft among evaluated vision-language models, with 86.7M parameters.The comparison covers zero-shot and supervised fine-tuning settings.Source: C. Additional Results · p. 15, Table 8 · p. 16

Computational Overhead Analysis

  • 13.39 ± 0.92 ms training latency for the combined framework versus 7.02 ± 0.29 ms for ViT-B/16 is an approximately 1.9× increase; the deployed student adds no inference-time computation.Measurements use 448 × 448 resolution and batch size 1; deployment retains the 86.7M-parameter image-only student.Source: C. Additional Results · p. 16

D. Limitations and Future Work

FG-LUPI depends on the specificity and quality of textual descriptions, and most benchmark descriptions are synthetically generated from training images rather than independent visual information. The multimodal teacher adds training cost, while future work may explore richer supervision and other forms of training-only information.

Limitations

  • FG-LUPI’s textual supervision depends on description specificity and quality, particularly for fine-grained classes whose localized, model-specific differences may be missed by generated descriptions.The paper identifies Stanford Cars as an example.Source: D. Limitations and Future Work · p. 16
  • CUB descriptions are human-authored, whereas descriptions for other datasets are generated from training images and should be treated as semantic supervision, not independent visual information.The authors describe this information as guiding the teacher and then transferring to the image-only student.Source: D. Limitations and Future Work · p. 16

Future Work

  • Future work may investigate richer human annotations, learned attribute representations, other training-only information, and settings with varying text quality.Source: D. Limitations and Future Work · p. 16

Related papers

Loading 2610.32442v1…