Source-linked AI summary
Fine-Grained Visual Classification via Progressive Multi-Granularity Training of Jigsaw Patches
Ruoyi Du, Dongliang Chang, Ayan Kumar Bhunia, Jiyang Xie, Zhanyu Ma, Yi-Zhe Song, Jun Guo
TL;DR
FGVC must handle subtle intra-class variation, while prior work pays less attention to identifying useful granularities and fusing information across them. PMG progressively trains granularity-specific stages and uses a random jigsaw patch generator, achieving state-of-the-art results on two of three datasets and a competitive result on the other.
Problem
Prior FGVC methods leave which granularities are most discriminative and how to fuse cross-granularity information insufficiently addressed, despite subtle intra-class variation.
Method
PMG combines progressive training that fuses different granularities with a random jigsaw patch generator that encourages granularity-specific feature learning.
Results
PMG achieves state-of-the-art performance on two standard FGVC datasets and a competitive result on the third.
Takeaways & Limitations
The framework learns complementary multi-granularity features end-to-end using only category labels and one network propagation during testing.
Takeaways & Limitations
PMG assumes discriminative information lies naturally across visual granularities rather than explicitly mining fine-grained representations from parts.
Abstract
from arXiv · showhide
Fine-grained visual classification (FGVC) is much more challenging than traditional classification tasks due to the inherently subtle intra-class object variations. Recent works mainly tackle this problem by focusing on how to locate the most discriminative parts, more complementary parts, and parts of various granularities. However, less effort has been placed to which granularities are the most discriminative and how to fuse information cross multi-granularity. In this work, we propose a novel framework for fine-grained visual classification to tackle these problems. In particular, we propose: (i) a progressive training strategy that effectively fuses features from different granularities, and (ii) a random jigsaw patch generator that encourages the network to learn features at specific granularities. We obtain state-of-the-art performances on several standard FGVC benchmark datasets, where the proposed method consistently outperforms existing methods or delivers competitive results. The code will be available at https://github.com/PRIS-CV/PMG-Progressive-Multi-Granularity-Training.
1 Introduction
FGVC must distinguish subtle variations within object sub-classes, while existing methods leave the most discriminative granularities and cross-granularity fusion insufficiently addressed. PMG combines progressive multi-granularity training with a random jigsaw patch generator and achieves state-of-the-art or competitive benchmark performance.
- FGVC identifies sub-classes such as bird species and vehicle models, where subtle intra-class variations make classification difficult.
- Existing weakly supervised methods locate discriminative local regions, but leave which granularities are most informative and how to fuse them insufficiently explored.
- PMG hypothesizes that discriminative information lies across visual granularities and jointly encourages multi-granularity learning with feature fusion.
- The jigsaw generator forms different granularity levels by shuffling evenly divided patches, restricting learned part granularity at each step.
- Progressive training starts with finer granularities, gradually moves to coarser ones, and fuses features from trained stages to cultivate complementary information.
- PMG obtains state-of-the-art or competitive performances on all three standard FGVC benchmark datasets.
2 Related Work
Related FGVC work shifted from strongly supervised part localization toward weakly supervised learning, but multi-part fusion remains challenging. PMG adapts progressive training and jigsaw augmentation to learn granularity-specific information in a single FGVC network.
- Weakly supervised FGVC methods use category labels while focusing on discriminative parts, complementary parts, or parts at varied granularities.
- Existing fusion approaches make predictions sequentially or concatenate features from detected parts, often using learned fully connected fusion layers.
- Fusion across parts remains challenging, motivating progressive learning from stable local details toward larger image structures.
- Jigsaw splitting has been used for self-supervised representation learning, spatial-relationship exploitation, and augmentation.
- Progressive training traditionally shifts from coarse structure to finer details in generative modeling, whereas PMG applies staged multi-granularity learning to FGVC.
- PMG increases patch counts while adding and training higher-level network stages, with each stage's classifier contributing to loss computation.
3 Approach
PMG progressively trains network stages from local to global information and uses shuffled patches to impose stage-specific granularities. It concatenates stage representations for complementary classification and supports efficient inference using one propagation.
- PMG trains shallower layers on stable fine-grained information before shifting deeper layers toward abstract, larger-granularity information.
- Intermediate backbone feature maps are reduced to vector representations, classified by stage-specific modules, and concatenated across the last S stages.
- Progressive training optimizes one stage output per step while using cross-entropy loss for stage and concatenated predictions.
- The jigsaw generator divides each image into n × n patches, randomly shuffles them, and uses n to control patch granularity.
- Patch counts follow n = 2^(L−l+1), subject to patch sizes remaining below the corresponding receptive field and increasing with it.
- Random cropping changes patches across iterations, so small discriminative parts are not always split and the model is encouraged to find parts at a target granularity.
- At inference, original images are used without jigsaw augmentation; using only concatenated features permits removal of other stages' fully connected layers.
- Combining single-stage outputs with equal weights can improve performance because each stage captures unique, complementary granularity information.
4 Experiment Results and Discussion
Experiments on CUB, Stanford Cars, and FGVC-Aircraft evaluate PMG against state-of-the-art methods and examine its progressive-training and jigsaw-generator components. Results show strong benchmark performance, benefits from suitable progressive settings, and meaningful earlier-stage attention.
- Experimental setup: PMG is evaluated on CUB, Stanford Cars, and FGVC-Aircraft using VGG16 and ResNet50 backbones, with category labels as the only training annotations.The experiments include state-of-the-art comparisons, ablations, and visualization analyses.
- Benchmark comparisons: PMG achieves competitive CUB performance while requiring only one network and one propagation during testing, outperforming RA-CNN by 4.3% and MGE-CNN by 1.1%.Stacked LSTM reports better performance but requires a two-phase pipeline with additional models and computation.
- Benchmark comparisons: PMG achieves state-of-the-art performance on Stanford Cars with ResNet50, outperforming MA-CNN by 2.3% and NTS-Net by 1.2%.Combining multi-stage outputs provides limited additional improvement on this dataset because yconcat is already strong.
- Benchmark comparisons: PMG achieves state-of-the-art performance on FGVC-Aircraft and surpasses S3N by 0.6% with the same ResNet50 backbone.It remains competitive when using VGG16 as the base network.
- Ablation study: Increasing the number of progressive stages improves performance significantly when S < 4, while larger settings do not provide the same benefit.The ablation varies S from 1 to 5 without the jigsaw generator and keeps yconcat for all runs.
- Ablation study: The jigsaw generator improves progressive training when S < 4 but lowers performance at S = 5 because patches become too small to preserve meaningful information when n > 8.The generator provides no advantage at S = 4.
- Visualization: Activation maps show more meaningful target-object concentration in PMG than in the baseline, with useful attention appearing at earlier stages under intermediate supervision.PMG’s third-to-fifth stages use jigsaw-generated inputs with n = {8, 4, 2} sequentially, while the baseline shows correct attention mainly at the last stage.
5 Conclusions
The paper introduces PMG Training, combining progressive multi-granularity feature fusion with a jigsaw generator. It is trained end-to-end using category labels and achieves state-of-the-art performance on two datasets and competitive performance on another.
- Conclusions: PMG Training combines progressive fusion of multi-granularity features with a jigsaw puzzle generator that forms images containing different granularity levels.The framework is presented as the paper’s two-component contribution.
- Conclusions: PMG can be trained end-to-end without manual annotations beyond category labels and requires only one network with one propagation during testing.This summarizes the stated training and inference procedure.
- Conclusions: PMG obtains state-of-the-art performance on two widely used fine-grained datasets and a competitive result on the third.The conclusion presents this pattern as evidence of the method’s effectiveness.