PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning

Hulingxiao He, Zijun Geng, Yuxin Peng

arXiv:2602.07605v3cs.CVcs.AI

TL;DR

MLLMs struggle with knowledge-intensive FGVR, especially when adapting with limited annotations and recognizing unseen categories. Fine-R1 combines CoT SFT with TAPO to improve reasoning and discrimination, and with 4-shot training it outperforms MLLMs and contrastive CLIP models on seen and unseen sub-categories.

  • Problem

    MLLMs perform poorly on knowledge-intensive FGVR, while adapting them requires costly labeled data and often causes overfitting to seen categories.

  • Method

    Fine-R1 uses a two-stage framework combining CoT SFT with TAPO, which augments policy optimization using positive and negative samples to address intra-class and inter-class variation.

  • Results

    Fine-R1 achieves superior closed-world and open-world FGVR performance, surpassing general and reasoning MLLMs and contrastive CLIP models with few-shot training.

  • Takeaways & Limitations

    Fine-R1 improves FGVR knowledge deployment and generalization to unseen categories while requiring only limited training data.

Abstract

from arXiv · show

Any entity in the visual world can be hierarchically grouped based on shared characteristics and mapped to fine-grained sub-categories. While Multi-modal Large Language Models (MLLMs) achieve strong performance on coarse-grained visual tasks, they often struggle with Fine-Grained Visual Recognition (FGVR). Adapting general-purpose MLLMs to FGVR typically requires large amounts of annotated data, which is costly to obtain, leaving a substantial performance gap compared to contrastive CLIP models dedicated for discriminative tasks. Moreover, MLLMs tend to overfit to seen sub-categories and generalize poorly to unseen ones. To address these challenges, we propose Fine-R1, an MLLM tailored for FGVR through an R1-style training framework: (1) Chain-of-Thought Supervised Fine-tuning, where we construct a high-quality FGVR CoT dataset with rationales of "visual analysis, candidate sub-categories, comparison, and prediction", transition the model into a strong open-world classifier; and (2) Triplet Augmented Policy Optimization, where Intra-class Augmentation mixes trajectories from anchor and positive images within the same category to improve robustness to intra-class variance, while Inter-class Augmentation maximizes the response distinction conditioned on images across sub-categories to enhance discriminative ability. With only 4-shot training, Fine-R1 outperforms existing general MLLMs, reasoning MLLMs, and even contrastive CLIP models in identifying both seen and unseen sub-categories, showing promise in working in knowledge-intensive domains where gathering expert annotations for all sub-categories is arduous. Code is available at https://github.com/PKU-ICST-MIPL/FineR1_ICLR2026.

1 INTRODUCTION

Fine-R1 addresses MLLMs’ difficulty with knowledge-intensive fine-grained visual recognition, costly adaptation, and poor unseen-category generalization through two-stage reasoning and policy optimization. With few-shot training, it improves closed-world and open-world FGVR performance over MLLMs and contrastive CLIP models.

  • FGVR requires distinguishing subtle differences among visually similar categories, but MLLMs show substantial performance drops on this knowledge-intensive task.
  • Extensive labeled data is costly for adapting general-purpose MLLMs, which also tend to overfit seen categories and generalize poorly to novel concepts.
  • Fine-R1 combines CoT SFT with reinforcement learning through TAPO to deploy intrinsic knowledge for FGVR and improve generalization with limited data.TAPO uses positive samples from the same sub-category and negative samples from similar sub-categories to address intra-class and inter-class variation.
  • Fine-R1 surpasses general MLLMs, reasoning MLLMs, and contrastive CLIP models in both closed-world and open-world FGVR evaluations.Reported gains include +8.51% closed and +23.75% open over Qwen2.5-VL-7B, and +4.27% closed over SigLIP-L.
  • The framework improves seen-category FGVR while generalizing to unseen categories with few-shot data, and analyses indicate better knowledge deployment rather than substantially changed visual features or category knowledge.

2 RELATED WORK

Related work adapts MLLMs to FGVR through classification fine-tuning or training-free methods, while CoT and reinforcement learning provide complementary routes for improving reasoning with less annotation.

  • MLLMs for FGVR: Prior MLLM approaches to FGVR fine-tune classification data or use training-free methods, with explicit object mentions and classification-focused data supporting object-centric reasoning.
  • Reinforcement Learning: Reinforcement learning has enhanced reasoning in LLMs and MLLMs while reducing reliance on large annotated datasets.
  • CoT Reasoning with MLLMs: Chain-of-thought reasoning supplies explicit intermediate steps, with multimodal implementations using prompting or supervised fine-tuning.

3 PRELIMINARIES

The preliminaries distinguish MLLM-based FGVR from CLIP-based classification: MLLMs generate sub-category names directly or select among candidates, whereas CLIP selects the most similar known label embedding.

  • FGVR with MLLMs: An MLLM maps an image and text query to a text output, which can represent a predicted fine-grained sub-category.
  • FGVR with MLLMs: In open-world FGVR, the MLLM freely generates a sub-category name from its original output space.
  • FGVR with MLLMs: In closed-world FGVR, the query supplies a predefined candidate set and the MLLM must select one candidate sub-category.
  • FGVR with CLIP models: CLIP models operate only in closed-world settings, selecting the candidate class whose text embedding has the highest cosine similarity with the image embedding.

4 METHODOLOGY

Fine-R1 uses progressive CoT supervised fine-tuning and Triplet Augmented Policy Optimization to build open-world FGVR reasoning that is robust to intra-class variation and discriminative across similar sub-categories.

  • CoT SFT: CoT SFT trains a structured “visual analysis, candidate subcategories, comparison, and final prediction” procedure for open-world FGVR.The curated reasoning data integrates domain knowledge into candidate generation and comparison before prediction.
  • TAPO: TAPO follows CoT SFT and optimizes reasoning for robustness to intra-class variance and discrimination across inter-class similarities.It extends DAPO with FGVR-specific augmentation and policy objectives.
  • Intra-class Augmentation: Intra-class Augmentation mixes predicted-answer trajectories from anchor and positive images to model diverse visual perspectives within a sub-category.The policy update remains conditioned only on the anchor image and question, while positive trajectories enrich the rollout pool.
  • Inter-class Augmentation: Inter-class Augmentation encourages distinct responses for visually similar images from different sub-categories.Its ratio measures output-distribution change when the image is replaced by a near-neighbor from another sub-category; higher values indicate stronger use of category-specific cues.
  • Policy objective: TAPO adds a KL-divergence objective between outputs conditioned on anchor or positive images and outputs conditioned on negative images.The combined intra-class and inter-class objective is further constrained by entropy regularization and a rollout condition requiring some, but not all, responses to include the ground truth.

5 EXPERIMENTS

Across six FGVR datasets, Fine-R1 is evaluated under few-shot base-to-new generalization, achieving strong closed- and open-world performance while ablations and analyses examine its training framework and mechanism.

  • 5.1 EXPERIMENT SETTINGS: Experiments use six FGVR datasets, with 60% of categories treated as seen, 40% unseen, and 4-shot training per seen category.Evaluation reports closed-world accuracy and open-world relative semantic similarity, with results averaged over three trials.
  • 5.2 MAIN RESULTS: 91.71% accuracy on seen categories and 85.70% on unseen categories show Fine-R1 outperforming Qwen2.5-VL-7B and SigLIP-L in closed-world evaluation.The seen-category gains are +7.73% over Qwen2.5-VL-7B and +3.38% over SigLIP-L; unseen-category gains are +9.28% and +5.16%, respectively.
  • 5.2 MAIN RESULTS: 74.80% relative semantic similarity on average gives Fine-R1-7B a 23.75% improvement over Qwen2.5-VL-7B in open-world evaluation.The evaluation uses only 4-shot training samples per sub-category and reports base-to-new category generalization.
  • 5.3 ABLATION STUDY: SFT improves seen-category accuracy by 3.98% but reduces unseen-category accuracy by 6.12%, whereas the two-stage framework improves over SFT by 7.56% on seen categories and over No-Thinking-RL by 10.05% on unseen categories.CLS-RL alone remains below the zero-shot baseline, while the combined framework addresses the contrasting seen- and unseen-category behavior.
  • 5.3 ABLATION STUDY: Direct CoT prompting changes the baseline from 74.68% to 74.79%, while Fine-R1 surpasses Qwen2.5-VL-3B with ICL by 17.90%.Fine-R1 also outperforms CLIP-like models despite prompt-ensembling optimization.
  • 5.4 PERFORMANCE GAIN ANALYSIS: Combining Intra-class and Inter-class Augmentation achieves the best ablation result of 67.32%, while CoT SFT and DAPO contribute 13.30% and 1.60%, respectively.Additional analyses find the best anchor-to-positive rollout ratio at n1:n2 = 1, increased performance with more CoT data, higher data quality than quantity, and consistent TAPO gains across architectures.
  • 5.4 PERFORMANCE GAIN ANALYSIS: Fine-R1 yields negligible differences in linear-probed visual features and subcategory-knowledge similarity, but positive and negative image-category pairs are more linearly separable in its representations.The result supports improved deployment of existing fine-grained subcategory knowledge rather than fundamentally different visual embeddings or stored subcategory knowledge.

6 CONCLUSION

Fine-R1 addresses FGVR data inefficiency and base-to-new generalization by combining CoT SFT with TAPO, using triplet augmentation to handle intra- and inter-class variation. It achieves state-of-the-art closed- and open-world results, surpassing contrastive CLIP models.

  • 6 CONCLUSION: Fine-R1 strengthens intrinsic-knowledge use for FGVR through CoT SFT and Triplet Augmented Policy Optimization.TAPO uses anchor, positive, and negative images from the same or different subcategories to address high intra-class and low inter-class variance.
  • 6 CONCLUSION: Fine-R1 achieves state-of-the-art performance in both closed-world and open-world FGVR evaluations.The conclusion reports that it outperforms contrastive CLIP models dedicated to discriminative tasks.

REPRODUCIBILITY STATEMENT

The reproducibility statement directs readers to the sections and appendices containing implementations, metrics, prompts, and implementation details.

  • REPRODUCIBILITY STATEMENT: Model implementations are described in Sections 4.2 and 4.3.
  • REPRODUCIBILITY STATEMENT: Evaluation metrics are presented in Section 5.1, while evaluation prompts and implementation details appear in Appendices B and C.

A QUALITATIVE RESULTS OF VISUAL CONCEPTS

The paper illustrates fine-grained visual concepts and specifies prompt formats for closed- and open-world recognition, alongside the compared CLIP, general MLLM, and reasoning MLLM families.

  • A QUALITATIVE RESULTS OF VISUAL CONCEPTS: Figure 5 depicts different image-level visual concepts for objects sharing the same subcategory.
  • A QUALITATIVE RESULTS OF VISUAL CONCEPTS: Fine-R1 prompts require explicit thinking and answer tags for both closed-world and open-world FGVR evaluation.The closed-world prompt requests fine-grained categories, while the open-world prompt conditions the final choice on provided options.
  • A QUALITATIVE RESULTS OF VISUAL CONCEPTS: The comparison includes CLIP models, general MLLMs, and the reasoning MLLM DeepPerception-7B.Listed CLIP baselines include CLIP-L, EVA-G, SigLIP-L, and SigLIP2-L; the general MLLM group includes multiple LLaVA, Idefics, InterVL, Qwen, and related models.
  • A QUALITATIVE RESULTS OF VISUAL CONCEPTS: CLIP-L, EVA-G, and SigLIP-L serve as vision encoders in several compared MLLM families.The paper notes that these MLLMs should theoretically have competitive or better FGVR capacity than their corresponding vision models.

E OPEN-WORLD EVALUATION RESULTS WITH TEXT INCLUSION

Table 6 reports open-world FGVR evaluation using text inclusion, with results averaged over three trials.

  • E OPEN-WORLD EVALUATION RESULTS WITH TEXT INCLUSION: Table 6 reports open-world FGVR results in terms of text inclusion (%), averaged over 3 trials.

F EXPERIMENTS ON MORE BASE MODELS

Applying CoT SFT and TAPO to openPangu-VL-7B consistently improves closed-world FGVR performance, indicating the training pipeline transfers across base models.

  • CoT SFT and TAPO consistently improve closed-world FGVR performance when applied to openPangu-VL-7B.Table 7 reports results averaged across three trials.

G GENERAL CAPABILITY

Fine-R1 improves classification-based and general visual-question-answering capabilities after FGVR post-training, while qualitative cases show structured reasoning producing more accurate answers than direct pattern matching.

  • Fine-R1 raises ImageWikiQA performance from 54.85% to 58.45% after post-training solely on FGVR data.The evaluation also reports improvements on MMBench and SEED-Bench.
  • Fine-R1 demonstrates competitive general-purpose performance on MME, MMBench, and SEED-Bench without general instruction-tuning data.The reported results are presented as evidence that reinforcement learning gains are not merely answer memorization.
  • On Stanford Car-196 and FGVC-aircraft, Fine-R1-3B is compared with Qwen2.5-VL-3B through qualitative case studies.The comparison is illustrated in Figure 6.
  • Fine-R1 generates accurate answers through visual analysis, candidate sub-categories, comparison, and prediction, unlike the baseline’s superficial direct responses.The structured process integrates domain-specific knowledge with visual observations.
Loading 2602.07605v3…