PaperlayerView exact revision on arXiv

Source-linked AI summary

VICAL: Vicinal Consistency Alignment for Long-Tailed Visual Recognition

Jiangang Zhu, Zheng Wang, Bin Zhu, Yi-Ping Phoebe Chen, Jingjing Chen

arXiv:2609.04948v1cs.CVcs.AI

TL;DR

Long-tailed multi-expert recognition has questioned whether greater expert diversity reliably improves ensemble accuracy. VICAL instead reduces prediction variance through self-consistency learning and deep ensemble distillation, achieving state-of-the-art performance across CIFAR-LT, ImageNet-LT, and iNaturalist 2018.

  • Problem

    Whether expert diversity improves ensemble accuracy in long-tailed recognition remains insufficiently established because diversity and accuracy are coupled rather than independent.

  • Method

    VICAL reduces prediction variance using Self-Consistency Learning for local prediction stability and Deep Ensemble Distillation for low-frequency cross-expert semantic agreement.

  • Results

    VICAL consistently achieves state-of-the-art performance across CIFAR-LT, ImageNet-LT, and iNaturalist 2018.

  • Takeaways & Limitations

    The findings support prioritizing variance reduction over explicit diversity maximization in long-tailed multi-expert recognition.

  • Takeaways & Limitations

    VICAL’s multi-expert architecture with vicinal views increases training time compared with single-model baselines, although inference cost remains comparable.

Abstract

from arXiv · show

Multi-expert models have become the dominant paradigm for long-tailed learning, largely attributed to their presumed ability to benefit from expert diversity. However, we revisit this central assumption and reveal that diversity induced by logit adjustment or explicit regularizers does not guarantee better ensemble accuracy. Our work suggests that multi-expert models benefit more from variance reduction than diversity maximization. We introduce \textbf{VICAL}, a \textbf{VI}cinal \textbf{C}onsistency \textbf{AL}ignment framework that improves long-tailed recognition not by enforcing expert diversity, but by reducing prediction variance. Specifically, our approach comprises two key components: Self-Consistency Learning and Deep Ensemble Distillation. Self-Consistency Learning discourages reliance on unstable high-frequency information, smoothing the local loss landscape and mitigating overfitting, especially for tail classes. Deep Ensemble Distillation promotes cross-expert low-frequency semantic agreement using a low-resolution view, thereby sidestepping optimization conflicts with established knowledge. Extensive experiments on CIFAR-LT, ImageNet-LT, and iNaturalist 2018 show that VICAL consistently outperforms state-of-the-art methods, validating the effectiveness of our consistency-driven design. Our code is available at \href{https://github.com/FlamieZhu/Vicinal-Consistency-Alignment}{VICAL}.

1 Introduction

Long-tailed data cause head-class bias and prediction uncertainty, while greater expert diversity does not reliably improve ensemble accuracy. VICAL therefore targets variance reduction through self-consistency and low-frequency cross-expert alignment, achieving gains on major long-tailed benchmarks.

  • Motivation: Long-tailed distributions create severe tail-class scarcity, causing deep networks to favor head classes and limiting applications such as medical diagnostics and autonomous driving.
  • Revisiting Diversity: Explicitly increasing expert diversity through logit adjustment or regularizers does not reliably improve ensemble accuracy and can compromise individual expert representation quality.Varying logit-adjustment strength changes expert agreement without notable ensemble-accuracy improvement; explicit regularization can degrade individual expert representations.
  • VICAL: VICAL reframes multi-expert learning around implicit variance reduction, seeking stable predictions under localized high-frequency perturbations and global resolution shifts, especially for tail classes.
  • VICAL Components: Self-Consistency Learning discourages reliance on unstable high-frequency patterns, while Deep Ensemble Distillation aligns experts through low-frequency semantic agreement.
  • Experiments: VICAL achieves significant gains over existing state-of-the-art methods on CIFAR-LT, ImageNet-LT, and iNaturalist 2018.

2 Related Work

Long-tailed visual recognition commonly addresses class imbalance through re-sampling, re-weighting, and logit adjustment. These strategies respectively modify class frequencies, loss contributions, or decision boundaries to better accommodate tail classes.

  • Long-Tailed Visual Recognition: Conventional long-tailed recognition methods include re-sampling, re-weighting, and logit adjustment.Re-sampling changes the class distribution, re-weighting emphasizes tail-class losses, and logit adjustment modifies decision boundaries for class imbalance.

3 Do More Diverse Experts Yield Better Performance?

This section tests the assumption that more diverse experts improve long-tailed recognition. It evaluates diversity induced by logit adjustment and explicit regularization, finding no positive correlation between expert diversity and ensemble performance.

  • Preliminaries: Long-tailed recognition trains on skewed data but evaluates performance on a balanced test set using an ensemble of M experts.The common setup uses M = 3 experts unless otherwise specified.
  • Empirical evaluation: Expert diversity induced by logit adjustment or explicit loss regularization has no positive correlation with ensemble performance.The result challenges the premise underlying methods that make experts focus on different categories.
  • Empirical evaluation: The study varies logit-adjustment intensities for long-tailed, uniform, and reversed experts on CIFAR-100-LT with IF=100.The expert settings are (τ1, τ2, τ3) = (1 −λ, 1, 1 + λ), with λ ∈ {0, 0.5, 1, 1.5, 2}, and diversity is measured using Q-statistics.

4 Vicinal Consistency Alignment

VICAL reduces multi-expert prediction variance through vicinal consistency alignment rather than diversity maximization, combining Self-Consistency Learning with Deep Ensemble Distillation. SC suppresses unstable high-frequency reliance, while DED and CKF preserve complementary expertise through low-frequency semantic agreement and filtered knowledge transfer.

  • 4 Vicinal Consistency Alignment: VICAL targets variance reduction with two components: Self-Consistency Learning and Deep Ensemble Distillation.The framework constructs vicinities by interpolating two strongly augmented full-resolution views.
  • Self-Consistency Learning: Self-Consistency Learning aligns predictions on interpolated, artifact-laden views with EMA-teacher predictions from the original views.This penalizes unstable high-frequency reliance and provides stronger regularization for data-scarce tail classes.
  • Deep Ensemble Distillation: DED distills ensemble consensus through a low-resolution view, restricting cross-expert alignment to low-frequency semantics and avoiding conflicts with SC.The low-resolution student logits align with averaged target-model logits from full-resolution views.
  • Conflicting Knowledge Filter: The Conflicting Knowledge Filter excludes samples where the teacher fails but the student is correct, preserving experts’ specialized knowledge during semantic alignment.This prevents incorrect transfers and uncertain global consensus from degrading individual experts.
  • Training and Inference: Training combines cross-entropy, self-consistency distillation, and ensemble distillation, while inference retains only the EMA target network.The online network is discarded because interpolated-sample exposure degrades its Batch Normalization statistics.

5 Experiments

VICAL is evaluated on four long-tailed benchmarks under standard balanced-test top-1 evaluation, with results reported by head, medium, and tail class frequency. It consistently achieves state-of-the-art performance and supports its variance-reduction design through frequency, conflict, and bias–variance analyses.

  • Evaluation Protocol: Evaluation uses top-1 accuracy on balanced test sets, reporting separate results for many-shot, medium-shot, and few-shot classes.The groups are defined as more than 100, 20–100, and fewer than 20 training images, respectively.
  • Comparison with state-of-the-art: VICAL consistently achieves state-of-the-art performance across long-tailed CIFAR-10, CIFAR-100, ImageNet-LT, and iNaturalist 2018.The comparisons include NCL++ [33], MDCS, and BalPoE, with results reported in Tables 1 and 2.
  • Hyperparameter and Inference Analysis: The target model is used for inference because its clean inputs yield superior performance, whereas corrupted online-network inputs degrade Batch Normalization statistics.The reported best hyperparameters are η = 0.8 for the target model, η = 0.6 for the online network, and β = 1 for the target model.
  • Frequency-based Analysis: Frequency analyses show that removing unstable high-frequency patterns mitigates cross-expert conflicts and improves VICAL’s robustness in the low-frequency region, especially for tail classes.Resolution-based DED views and spatial-filter comparisons support the frequency-decoupled mechanism; Fourier-noise analysis shows only slight increased sensitivity for head classes at extreme high frequencies.
  • Identifying Conflicting Knowledge: Conflicting teacher–student pairs in which the teacher is wrong and the student is correct comprise only 2% of samples but cause substantial performance degradation, motivating their removal.The conflicting knowledge set is analyzed with alternative filters and a terminal-phase histogram.
  • Variance and Consistency Analysis: VICAL reduces variance and bias on tail classes while preserving head performance, and Self-Consistency Learning lowers interpolation entropy particularly for tail samples.The method’s regularization effect is assessed through mean accuracy, bias, and variance comparisons and entropy profiles between full-resolution views.

6 Conclusion

The paper challenges the assumed positive relationship between model diversity and ensemble accuracy in long-tailed learning, finding that diversity alone does not universally improve ensembles and motivating VICAL’s variance-reduction focus.

  • The paper re-evaluates the prevailing assumption that greater model diversity positively correlates with ensemble accuracy in long-tailed learning.
  • Diversity induced by logit adjustment or explicit regularizers does not universally guarantee better ensemble accuracy.
  • Motivated by these findings, VICAL shifts the focus from explicit diversity constraints to implicit variance reduction.

1 Datasets and Training Details

Experiments use long-tailed CIFAR-10/100, ImageNet-LT, and iNaturalist 2018 under standard splits and backbone configurations tailored to each dataset. CIFAR-LT evaluations vary the imbalance factor from 10 to 200.

  • Datasets: Experiments cover CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018 using standard long-tailed splits where specified.CIFAR-10-LT and CIFAR-100-LT contain 10 and 100 classes, respectively.
  • Datasets: CIFAR-LT experiments use imbalance factors of 200, 100, 50, and 10, with imbalance defined as Nmax/Nmin.The splits follow common practice for fair comparison.
  • Backbone: CIFAR-LT uses a shared-block ResNet-32 backbone, while ImageNet-LT and iNaturalist 2018 use shared-block ResNeXt-50 and ResNet-50 backbones, respectively.The CIFAR-LT backbone matches prior setups used to reproduce BalPoE and MDCS fairly.

2 Experimental Setup of Section 3

The study evaluates expert diversity using Q-statistics, Pearson correlation ρ, and diversity factor σ, while varying logit-adjustment intensity and an explicit RIDE-style diversity constraint. Contrary to diversity-maximization assumptions, negative κ increases prediction diversity but degrades ensemble performance.

  • Diversity Metrics: The experiments measure pairwise expert diversity with Q-statistics and Pearson correlation ρ, averaged across classifier pairs.Q-statistics range from −1 to 1, with positive values indicating shared correctness or errors and negative values indicating complementary correctness.
  • Diversity Metrics: The diversity factor σ is an auxiliary accuracy-based metric that reaches its maximum when every test sample is correctly classified by at least one expert.When all experts’ outputs are identical, σ equals their top-1 accuracy.
  • Explicit Diversity Constraint: The explicit diversity loss uses κ alongside Balanced Softmax loss, with positive κ encouraging consensus and negative κ enforcing divergence from the ensemble prediction.The constraint is based on each expert’s prediction probability and the ensemble prediction probability.
  • Explicit Diversity Constraint: Negative κ lowers Q-statistics and ρ, indicating increased prediction diversity but degraded ensemble performance.Ensemble accuracy also shows a slight negative correlation with diversity accuracy σ.

3 Extended Experiments

Extended experiments show that VICAL reduces inter-expert prediction variance, remains effective across expert counts, test distributions, and matched computation budgets, and benefits from carefully chosen consistency mechanisms. Additional ablations and visualizations support its self-consistency design and semantic agreement between full- and low-resolution views.

  • Inter-Expert Variance Analysis: VICAL achieves the lowest inter-expert logit variance, 1.21, versus 1.85 for the baseline, 1.49 with DED, and 1.31 with SC.Both components improve cross-expert consistency, with their combination producing the strongest variance reduction.
  • Consistency Ablations: Equal teacher averaging minimizes head- and tail-class loss, performance remains stable for α ∈[0.25, 1.0], and SC performs best among tested self-distillation variants.The experiments use α = 1.0, while extrapolative averaging weights outside [0, 1] increase loss.
  • Visualization Results: Low-resolution views enforce semantic consistency while full-resolution views preserve diverse details, with Grad-CAM visualizations showing experts capture different patterns.The visualizations use three experts on iNaturalist 2018.
  • Computation Cost: At equivalent computation cost, VICAL consistently surpasses MDCS and NCL++.The comparison measures forward FLOPs for the online network per sample with multiple views as input.
  • Effect of the Number of Experts: VICAL consistently outperforms NCL++ across expert counts and provides superior head- and medium-class prediction stability.The framework requires no additional hyperparameter tuning, although NCL++ achieves higher tail-class accuracy at a cost to many- and medium-shot performance.
  • Results on Multiple Test Distributions: VICAL surpasses other methods across CIFAR-100-LT and ImageNet-LT multiple test distributions by a large margin.These results support the benefit of vicinal consistency alignment under varied test priors.

4 Broader Impact and Limitations

VICAL shows promise for fairer, more reliable, and robust vision systems, while applying directly to semi-supervised recognition. Its multi-expert vicinal-view design increases training time, and its consistency–variance relationship remains theoretically underexplored.

  • Broader Impact: VICAL’s stability and variance-reduction focus may improve fairness, reliability, and trustworthiness in critical real-world vision applications.The framework demonstrates superiority in imbalanced classification and may support robust deployments.
  • Broader Impact: VICAL can be directly applied to semi-supervised visual recognition without complex modifications.
  • Limitations: The multi-expert vicinal-view architecture increases training time over single-model baselines, although inference cost remains comparable.The precise theoretical relationship between consistency learning and variance reduction in long-tailed settings also requires further formal analysis.
Loading 2609.04948v1…