Source-linked AI summary
SciLT: Long-tailed Image Classification under Scientific Image Domains
Jiahao Chen, Bing Su
TL;DR
Scientific long-tailed recognition asks whether foundation-model fine-tuning remains effective under domain shifts from natural to scientific images. The paper investigates this question, introduces SciLT to fuse penultimate- and final-layer representations with complementary supervision, and reports consistent benchmark improvements while identifying remaining scope boundaries.
Problem
Existing long-tailed benchmarks mainly use natural images within similar pre-training and fine-tuning domains, leaving foundation-model effectiveness in scientific long-tailed settings uncertain.
Method
SciLT adaptively fuses penultimate- and final-layer representations and trains them with Logit Adjustment and cross-entropy supervision, respectively.
Results
SciLT consistently outperforms existing methods on Blood, ISIC, and NIH-Chest, with an overall ISIC BScore of 74.5.
Takeaways & Limitations
The findings support exploiting intermediate representations and class-prior information when adapting foundation models to imbalanced scientific data.
Takeaways & Limitations
The current design primarily exploits penultimate-layer representations, while richer interactions across multiple layers may further improve representation learning.
Abstract
from arXiv · showhide
Long-tailed recognition has benefited from foundation models and fine-tuning paradigms, yet existing studies and benchmarks are mainly confined to natural image domains, where pre-training and fine-tuning data share similar distributions. In contrast, scientific images exhibit distinct visual characteristics and supervision signals, raising questions about the effectiveness of fine-tuning foundation models in such settings. In this work, we investigate scientific long-tailed recognition under a purely visual and fine-tuning paradigm. Experiments on three scientific benchmarks show that fine-tuning foundation models yields limited gains, and reveal that penultimate-layer features play an important role, particularly for tail classes. Motivated by these findings, we propose SciLT, a framework that exploits multi-level representations through adaptive feature fusion and dual-supervision learning. By jointly leveraging penultimate- and final-layer features, SciLT achieves balanced performance across head and tail classes. Extensive experiments demonstrate that SciLT consistently outperforms existing methods, establishing a strong and practical baseline for scientific long-tailed recognition and providing valuable guidance for adapting foundation models to scientific data with substantial domain shifts.
1 Introduction
Scientific long-tailed recognition remains underexplored because scientific images differ visually from natural images, challenging the transferability of foundation-model fine-tuning. SciLT addresses this through adaptive multi-level feature fusion and dual supervision, improving balance across head and tail classes.
- Motivation: Scientific datasets combine severe long-tailed distributions with domain characteristics that differ from natural-image benchmarks.Natural-image benchmarks typically keep pre-training and fine-tuning within the same domain.
- Motivation: Fine-tuning foundation models provides limited benefits when downstream scientific tasks deviate from the pre-training paradigm.The paper investigates this behavior under a purely visual, parameter-efficient fine-tuning setting.
- Findings: Penultimate-layer features can significantly contribute to scientific long-tailed learning and sometimes outperform deeper layers, particularly for tail classes.A substantial Wasserstein-distance gap indicates that penultimate and final layers capture markedly different information.
- SciLT: SciLT adaptively fuses penultimate- and final-layer features to construct more expressive representations, especially for tail classes.The fused features use Logit Adjustment criteria, while final-layer features retain standard cross-entropy supervision.
- SciLT: SciLT jointly optimizes multi-level representations under different criteria to balance head- and tail-class performance.This design is presented as a principled balance between representation depth and class imbalance.
- Results: SciLT consistently outperforms existing methods on Blood, ISIC, and NIH-Chest, establishing a practical baseline for scientific long-tailed recognition.The results also provide guidance for fine-tuning foundation models under scientific domain shifts.
2 Related work
Prior long-tailed and scientific representation-learning research has developed imbalance remedies and domain-specific pre-training, but existing work gives limited attention to scientific domain characteristics under foundation-model fine-tuning.
- Long-tailed learning: Long-tailed learning commonly mitigates class imbalance through re-weighting, re-sampling, class-balanced loss, or logit adjustment.These methods aim to emphasize minority classes and reduce bias toward head classes.
- Benchmark comparisons: Table 1 compares long-tailed benchmarks using relative improvements over the best of cRT, MiSLAS, PaCo, and LiVT.Gray rows identify methods fine-tuned from a foundation model.
- Benchmark comparisons: Figure 2 reports relative gains on Places365-LT and iNaturalist2018 across Many, Medium, and Few classes.RAC and LPT results for iNaturalist2018 are only partially available.
- Scientific representation learning: Existing work mainly studies natural images and downstream distribution bias, while scientific domains receive limited attention.This paper targets imbalanced scientific datasets under domain shift and explores strategies for foundation-model adaptation.
- Scientific representation learning: Scientific representation learning increasingly uses large-scale pre-training and self-supervised learning to improve generalization with limited supervision.Earlier work adapted convolutional networks to medical imaging, while newer work emphasizes transferable domain-specific features.
3 Motivation
The motivation study tests whether foundation-model fine-tuning transfers consistently across domains and semantic gaps. Natural-image results are generally favorable, whereas iNaturalist2018 shows marginal gains and slight head-class degradation.
- Empirical motivation: Fine-tuning foundation models yields substantial gains on ImageNet-LT and Places365-LT but only marginal improvements on iNaturalist2018.The comparison covers fine-tuned approaches alongside methods trained from scratch.
- Empirical motivation: Fine-tuning benefits tail classes more than head classes on ImageNet-LT and Places365-LT, based on relative gains across Many, Medium, and Few classes.iNaturalist2018 instead shows slightly degraded head-class accuracy relative to training from scratch.
- Research question: The central question is how fine-tuning effectiveness depends on domain and semantic-granularity gaps between pre-training and target datasets.This frames scientific long-tailed recognition as a domain-shifted adaptation problem.
4 Exploration on Scientific datasets
Experiments across scientific long-tailed datasets show that fine-tuning benefits depend strongly on the dataset and domain shift, while penultimate-layer representations can better support tail recognition. Feature-distribution analysis motivates using complementary representations rather than relying only on final-layer features.
- Fine-tuning gains are dataset-dependent: Fine-tuning foundation models yields only marginal improvements on scientific long-tailed datasets, with performance depending on the dataset and domain shift.On NIH-Chest, fine-tuning can underperform training from scratch, whereas ISIC shows clearer fine-tuning advantages.
- Fine-tuning gains are dataset-dependent: Re-balancing objectives such as LA improve tail-class performance, highlighting their role in mitigating severe class imbalance.Figure 3 reports this pattern across the NIH-Chest and ISIC performance curves.
- Penultimate-layer representations: Penultimate-layer features achieve comparable or superior performance to final-layer features, with particularly larger improvements for tail classes on NIH-Chest.The authors attribute this potential advantage to tail classes’ scarce training data and greater sensitivity to feature initialization.
- Feature-space analysis: Wasserstein-distance analysis finds a distributional discrepancy between penultimate- and final-layer features, indicating that the layers encode distinct representation structures.The discrepancy suggests that some information preserved in the penultimate layer is not fully retained after the final projection.
- Feature-space analysis: Different feature distributions and class-wise errors motivate feature fusion, but these observations alone do not prove that fusion improves performance.The paper presents the relationship as an empirical motivation rather than a demonstrated causal explanation.
5 Method
SciLT addresses uncertainty in representation depth and class-prior bias by fusing the two deepest layers and training predictors for balanced and observed-prior settings. Its theoretical analysis characterizes depth-dependent risk, fusion gains, and retention-constrained gate estimation.
- SciLT: SciLT fuses penultimate- and final-layer representations, then combines balanced-prior and observed-prior predictors to address representation selection and class-prior bias.The two branches target class balance and observed-prior accuracy, respectively.
- SciLT: Both representation depths retain nonzero adaptive weights, while their relative contributions can vary with the input.The normalized weights lie in (1/4, 3/4), preventing either depth from being prematurely suppressed.
- SciLT: SciLT trains a balanced-prior branch with τbal > 0 and an observed-prior branch with τobs = 0, then interpolates their scores at inference.The inference score is s(β) = βsobs + (1 − β)sbal.
- Theoretical Analysis: A class-specific oracle depth can be preferable because deeper features may reduce approximation error while increasing transfer bias or complexity.The n−1/2 term amplifies this competition for rare classes.
- Theoretical Analysis: Fusion strictly improves squared surrogate risk when the two depths have complementary errors, although the optimal class-specific weight is difficult to estimate for tail classes.Theorem 5.2 requires cy < min{vy,N, vy,N−1}; its optimal weight is a population quantity.
- Theoretical Analysis: Retention-constrained gate learning trades controlled oracle-clipping bias for protection against finite-sample gate collapse.The bound separates gate-estimation uncertainty from the bias introduced by clipping the oracle.
6 Experiment
Experiments across ISIC, Blood, and NIH-Chest show that SciLT improves class balance, especially for minority groups, while producing dataset-dependent OvAcc–Macro trade-offs. Ablations further support the contribution of feature fusion and selected representation depths.
- Results on ISIC: 74.5 overall BScore: SciLT achieves the highest score on ISIC and improves several minority classes over LA and CE.SciLT reaches 67.8 on MEL, improving over LA by +9.4 points and over CE by +7.7; on AK, it exceeds CE by +21.9 points, reaching 64.1.
- Results on Blood: 97.8 BScore: SciLT maintains strong Blood performance while improving Mono. accuracy by +4.3 points over CE to 93.6.The method also maintains high accuracy on Baso., Eosino., Lympho., and Neutro., indicating minority gains without sacrificing head-class performance.
- Results on NIH-Chest: +15.5 and +6.1 points: SciLT improves NIH-Chest Medium and Few accuracy over CE, while increasing Macro by +7.7 points at a 3.4-point OvAcc cost.Against LA, SciLT improves OvAcc by +16.6 points but remains 2.0 points lower in Macro, yielding the highest BScore of 24.8.
- Ablation study of the fusion model: +1.4 overall-accuracy points and +3.7 macro-accuracy points: the proposed fusion strategy outperforms the no-fusion configuration on the derived NIH-Chest task.The ablation compares direct penultimate-feature classification and output ensembling against the proposed fusion strategy.
- Ablation study on the selected depths: 19.7 OvAcc: the final-layer-only setting provides the baseline for evaluating progressively incorporated backbone depths on NIH-Chest.The current SciLT configuration uses two depths, while the experiment keeps backbone, objectives, schedule, inference rule, and seed fixed.
7 Discussion
The discussion identifies remaining limits in SciLT’s representation design and clarifies its intended scientific-image setting. The framework is general rather than domain-specific and does not assume multi-source or unlabeled target-domain data.
- Limitations: SciLT primarily exploits penultimate-layer representations, leaving richer interactions across multiple layers as a possible direction for improvement.The limitation concerns the current representation design rather than the framework’s overall scientific-image scope.
- Scope and assumptions: SciLT is a general fine-tuning framework for scientific long-tailed images, not a method tailored specifically to medical imaging.The evaluated datasets are presented as representative scientific benchmarks rather than restrictions to particular domains.
- Scope and assumptions: The method uses labeled data from the target scientific domain and assumes neither multiple source domains nor unlabeled target-domain data.This distinguishes the setting from conventional domain adaptation or domain generalization.
- Limitations: VL-LTR and RAC are not compared because they rely on auxiliary textual information that may be difficult to obtain or reliably verify in scientific applications.The omission limits comparisons with methods requiring additional textual information.
8 Conclusion
The paper concludes that scientific long-tailed recognition benefits from jointly addressing representation depth and class-prior information. SciLT demonstrates this approach across ISIC, NIH-Chest, and Blood while supporting analysis of depth uncertainty and retention trade-offs.
- Conclusion: SciLT jointly addresses representation depth and class prior through retention-constrained fusion and prior-complementary supervision.The framework is evaluated on ISIC, NIH-Chest, and Blood.
- Conclusion: Results support exploiting intermediate representations and class-prior information for fine-tuning long-tailed medical images.The conclusion connects these design choices with analyses of depth uncertainty, complementary errors, and retention trade-offs.