PaperlayerView exact revision on arXiv

Source-linked AI summary

FIDAL: Diversity-Aware Federated Active Learning Under Real-World Distribution Shifts

David Dueñas Gaviria, Shadi Albarqouni

arXiv:2609.31637v1cs.LGcs.CV

TL;DR

Federated clinical active learning must contend with annotation costs, domain shifts, class imbalance, and OOD samples that existing methods often overlook. FIDAL combines calibrated uncertainty, diversity-aware selection, and adaptive OOD rejection in an open-set federated framework. Across three medical-imaging benchmarks, it achieves accuracy–purity trade-offs not matched by the evaluated baselines in the tested settings.

  • Problem

    Many active-learning methods assume in-distribution data and overlook unknown samples in federated clinical settings, where annotation budgets and distribution shifts matter.

    Source: §1 · p. 1

  • Method

    FIDAL combines global–local evidential uncertainty, support-set diversity weighting, and adaptive OOD rejection in open-set federated active learning.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    FIDAL is the only method on the accuracy–ID purity Pareto front across all three benchmarks and is closest to the fully supervised reference on each.

    Source: §4.3 · p. 6

  • Takeaways & Limitations

    Across three benchmarks, FIDAL balances predictive performance and annotation efficiency while reaching 81.9–92.3% ID purity.

    Source: §6 · p. 12

    LfOSA reaches 90.7% versus 83.8% purity on FedCamelyon and 92.6% versus 92.3% on FedEMBED, with lower classification performance; experiments do not isolate why.

    Source: §5.4 · p. 11

Abstract

from arXiv · show

Federated learning enables collaborative model training across institutions without centralizing data, yet high annotation costs, domain shifts, and class imbalance remain major obstacles, especially when irrelevant out-of-distribution (OOD) samples dilute the labeled data. Existing active learning methods target uncertainty or diversity within in-distribution (ID) data and overlook unknown samples in federated clinical settings. We propose FIDAL, an open-set federated active learning framework that combines calibrated global-local evidential uncertainty, support-set diversity weighting, and adaptive OOD rejection. The rejection gate thresholds a foundation-model Gaussian-coverage signal per client and per round with Otsu's criterion, so that highly informative ID samples are queried while irrelevant outliers are excluded without any hand-tuned threshold. Evaluated on three multi-center medical imaging benchmarks (dermatology, histopathology, and mammography with organically occurring artifacts) in realistic open-set scenarios, FIDAL outperforms detector-based open-set methods by up to about 12 percentage points of balanced accuracy and is the only method on the accuracy-ID purity Pareto front of all three benchmarks. At an equal query budget it spends at least 1.3 times fewer annotations on OOD samples than every accuracy-matched baseline, saving an estimated 7-29 hours of expert reading on the mammography benchmark. By labeling only a fraction of the data pool, it matches or exceeds fully supervised performance across modalities. These results highlight the value of integrating uncertainty, diversity, and OOD rejection in open-set federated active learning for medicine.

1. Introduction

The introduction frames open-set federated active learning as a response to annotation costs, heterogeneous clinical data, and OOD samples, which existing active-learning approaches often do not address together. FIDAL combines calibrated uncertainty, diversity-aware selection, and adaptive OOD rejection for this setting.

1. Introduction

  • Clinical pools contain OOD samples and rare, informative edge cases, while many active-learning methods assume closed-set, in-distribution data.The paper identifies heterogeneous, privacy-sensitive clients and strict annotation budgets as additional constraints.Source: §1 · p. 1

2. Related work

  • Existing federated active-learning approaches address heterogeneity and uncertainty but still assume a pure in-distribution pool, leaving OOD samples and under-represented classes insufficiently handled.The related-work discussion also notes that open-set active-learning approaches are studied centrally and rely on assumptions that may not hold across clients.Source: §1 · p. 1, §2 · p. 2

1. Introduction

  • FIDAL unifies federated training, open-set rejection before annotation, and budgeted active acquisition in one medical-imaging framework.Its stated components include calibrated evidential uncertainty, support-set diversity weighting, and adaptive client-specific OOD rejection.Source: §1 · p. 2

2. Related work

  • Prior active-learning methods use uncertainty, diversity, or both, but clinical data challenge their closed-set, balanced-pool assumptions with domain shifts, rare classes, and redundant routine cases.The introduction describes uncertainty and diversity methods as susceptible to miscalibration under federated data heterogeneity.Source: §1 · p. 1, §2 · p. 2
  • FIDAL avoids sharing OOD samples or pool-wide statistics, and adapts rejection to each client’s OOD examples and changing distribution across rounds.Its local similarity filter is complemented by a foundation-model coverage term.Source: §2 · pp. 2–3

3. Methodology

FIDAL formulates client-level querying as selecting a diverse, informative set that favors in-distribution samples and rejects OOD candidates. It combines global–local evidential uncertainty, support-weighted similarity, and a foundation-model coverage gate to rank eligible samples.

3.1. Problem formulation

  • FIDAL selects a budgeted query set intended to maximize ID inclusion, minimize OOD selection, and produce diverse, informative labels across federated clients.Clients alternate federated training with local annotation and dataset updates across rounds.Source: §3.1 · p. 3

3.2. Uncertainty calibration

  • Global epistemic uncertainty modulates global and local aleatoric uncertainty, calibrating the score against model ignorance associated with domain shift.The method uses aleatoric uncertainty from both models to capture ambiguity and client noise, but global epistemic uncertainty alone for domain shift.Source: §3.2 · p. 3

3.3. Similarity-based diversity, OOD penalty, and ranking score

  • Support-set count gives less-represented ID regions greater diversity weight, while similarity to labeled OOD examples supplies a client-specific OOD penalty.The adaptive threshold counts neighbors within 20% of a candidate’s maximum ID similarity; a small count indicates a rare region worth exploring.Source: §3.3 · pp. 3–4
  • The task-feature ranking score combines calibrated uncertainty, support-weighted diversity, and OOD rejection.A frozen foundation-model encoder provides a per-client Gaussian coverage score derived from the labeled-ID manifold.Source: §3.3 · p. 4, §3.4 · p. 4

3.4. Foundation-model coverage and adaptive gate

  • Otsu thresholding adapts the foundation-model coverage gate per client and round, hard-rejecting candidates below threshold and weighting survivors by coverage.The query set comprises the B survivors with the highest final scores.Source: §3.4 · pp. 4–5

4. Experiments and results

Across three multi-center medical imaging benchmarks, FIDAL combines active selection with open-set rejection and achieves strong accuracy while maintaining high ID purity. Its coverage gate and diversity-related components contribute differently across datasets.

4.1. Datasets and open-set protocol

  • The evaluation uses dermatology, histopathology, and mammography benchmarks, with injected OOD samples in FedISIC and FedCamelyon and organically occurring artifacts in FedEMBED.FedISIC and FedCamelyon pools contain 38.7% and 38.5% OOD samples, respectively; FedEMBED artifact prevalence varies from 4.8% to 13.1% across clients.Source: §4.1 · p. 5

4.2. Experimental setup

  • Methods are reported at their own best round using BMA for FedISIC and FedEMBED, ACC for FedCamelyon, and cumulative ID purity; each benchmark uses a fixed per-round budget.The comparison includes standard federated active-learning baselines, open-set methods, and a fully supervised clean-ID reference.Source: §4.2 · p. 5

4.3. Results across the three benchmarks

  • Across all three datasets, FIDAL queries fewer OOD samples than accuracy-matched baselines, with reported reductions of 1.3–3.8×.The comparison counts OOD samples sent for annotation at FIDAL’s headline round, avoiding credit for savings caused by an accuracy collapse.Source: §4.3 · p. 6
  • FIDAL remains competitive under budget and local-epoch sweeps, including at low budgets and few local epochs, and retains high purity on FedEMBED.On FedISIC it tops every reported sweep cell; on FedCamelyon it is best except at B=50, while on FedEMBED it averages 66.88% BMA versus 66.73% for OpenPath* at the final round.Source: §4.3 · pp. 6–7
  • FIDAL is the only method on the accuracy–ID purity Pareto front across all three benchmarks, exceeding the fully supervised reference on FedISIC and matching it on FedEMBED.It reaches 71.14 ± 0.54% BMA at 81.9% ID purity on FedISIC, 97.34 ± 0.12% ACC at 83.8% on FedCamelyon, and 69.49 ± 0.36% BMA at 92.3% on FedEMBED.Source: §4.3 · p. 6, Table 1 · p. 7, Figure 3 · p. 7

4.4. Why FIDAL works: the coverage gate and the ablation

  • Otsu thresholds the foundation-model coverage signal per client: it catches ≈91% of OOD at ≈1% ID loss on FedISIC and ≈92% at ≈2% on FedCamelyon, but struggles on overlapping FedEMBED modes.On FedEMBED, Otsu reaches 57% recall at 23% ID loss, so the labeled-OOD SOOD penalty carries more of the purity burden.Source: §4.4 · p. 7, Figure 4 · p. 8
  • On FedISIC, removing the coverage gate lowers purity by ≈13 pp, while removing diversity costs 4.88 pp BMA; on FedEMBED, removing SOOD drops purity from 92.3% to 87.6%.The full combination is Pareto-optimal on FedISIC at 71.14% BMA and 81.9% purity.Source: §4.4 · pp. 7, 9, Figure 5 · p. 9

5. Discussion

FIDAL combines calibrated uncertainty, support-based diversity, and adaptive OOD rejection to balance predictive performance with annotation efficiency. The discussion reports budget and training-regime results, gate configuration guidance, qualitative examples, and scope boundaries.

5. Discussion

  • Across three benchmarks, FIDAL achieves accuracy–purity trade-offs not matched by evaluated baselines in the tested settings.The design combines a calibrated global–local evidential score, support-based diversity for rare-class coverage, and adaptive per-client OOD rejection.Source: §5 · p. 9

5.1. Impact of FIDAL’s budget efficiency in real-world heterogeneous settings

  • At equal budgets, FIDAL acquires ≈1.7× more ID labels than uncertainty baselines on FedISIC and saves an estimated 7–29 hours of expert reading on FedEMBED.The reading-time estimate assumes each avoided query takes approximately 59 seconds; dermoscopy and histology results are reported only as avoided counts because no validated per-image reading time maps to those query units.Source: §5.1 · pp. 9–10
  • At E=2, FIDAL reaches 61.47% BMA, 1.5 pp above BADGE and 22.9–28.6 pp above evidential and detector-based methods.The proposed explanation—that the frozen FM-coverage gate remains usable when learned local representations are weak—is plausible but not isolated experimentally; full fine-tuning is more epoch-sensitive than the frozen-encoder FedEMBED probe.Source: §5.1 · p. 10

5.2. Configuring the gate in practice

  • Gate configuration depends on FM–OOD separability: the tested pilot heuristic varies λOOD by gate performance, while its parameter ranges are not universal cutoffs.On FedISIC, the headline cell (0.5, 0.5) reaches (71.14 ± 0.54)% BMA; on FedEMBED, (0.5, 5.0) lies on the ID-purity Pareto frontier.Source: §5.2 · p. 10

5.3. Qualitative assessment of what FIDAL selects and rejects

  • FIDAL selects canonical Easy ID and atypical Diverse ID examples, but its gate also rejects some true-ID cases near the coverage boundary.The SOOD penalty handles OOD samples that pass the gate, without another hard threshold.Source: §5.3 · pp. 10–11

5.4. Scope and limitations

  • At B=50, FEAL leads on FedCamelyon ACC and OpenPath* on FedEMBED BMA, while FIDAL retains the highest ID purity; the authors favor a joint frontier and per-dataset pilot.At R=10, LfOSA also reaches equal or higher ID purity on FedCamelyon and FedEMBED, but with lower classification performance; the experiments do not isolate why.Source: §5.4 · p. 11
  • Client-local scoring supports data minimization but does not establish GDPR or HIPAA compliance; secure aggregation was not evaluated.Compliance depends on deployment infrastructure, institutional policies, and jurisdiction.Source: §5.4 · p. 11
  • The evaluation does not model ID/OOD-boundary drift and lacks sex- and gender-stratified results, limiting generalizability.EMBED contains approximately 20% ambiguous ROIs, and the aleatoric term does not explicitly model label noise.Source: §5.4 · pp. 11–12

6. Conclusion

FIDAL combines an adaptive client-specific FM-coverage gate with calibrated uncertainty, support-aware diversity, and similarity to identified OOD samples. Across three benchmarks, it balances predictive performance and annotation efficiency, while future work targets more heterogeneous settings and realistic label noise.

6. Conclusion

  • Across FedISIC, FedCamelyon, and FedEMBED, FIDAL reaches 81.9–92.3% ID purity and reduces OOD queries by factors of 1.3–3.8 versus accuracy-competitive baselines.Future work will extend FIDAL to diverse modalities, variable annotation budgets, realistic label noise, and settings where the ID/OOD boundary drifts across institutions and time.Source: §6 · pp. 12–13

CRediT authorship contribution statement

The contribution statement assigns conceptualization, methodology, software, formal analysis, visualization, and original drafting to David Dueñas Gaviria, and supervision, funding acquisition, and review to Shadi Albarqouni.

CRediT authorship contribution statement

  • David Dueñas Gaviria is credited with conceptualization, methodology, software, formal analysis, visualization, and the original draft; Shadi Albarqouni with supervision, funding, and review.Source: CRediT authorship contribution statement · p. 13

Data availability

The study uses previously published public datasets and releases split files for FedISIC and FedCamelyon; access-controlled FedEMBED data and split files cannot be redistributed.

Data availability

  • FedISIC and FedCamelyon split files are released, while FedEMBED split generation is supported by released code because its images are access-controlled.FedEMBED is available to credentialed users under a standard data-use agreement; the study generated no new primary imaging data.Source: Data availability · p. 13

Supplementary material

The supplementary material collects additional benchmark composition, implementation, trajectory, pool-composition, stress-test, and per-class recall details.

Supplementary material

  • Sections S1–S6 provide per-client OOD composition, implementation and prompt-protocol details, per-client Pareto trajectories, round-wise labeled-pool composition, budget and epoch variation tables, and per-class recall caveats.These materials supplement the main-text analyses and figures.Source: Supplementary material · p. 13

S1. Open-set benchmark composition

The open-set benchmarks use injected OOD artifacts in FedCamelyon and organically occurring, expert-annotated artifacts in FedEMBED, with client-level composition details supplied for both.

S1. Open-set benchmark composition

S2. Implementation details

The supplement documents implementation choices and examines client-level trajectories, round-wise OOD composition, and robustness to acquisition-budget and local-epoch variation.

S2. Implementation details

  • FIDAL adds no optimizer-side hyperparameters beyond (λdiv, λOOD); its gate and diversity thresholds are adaptive, with no manual percentile cut-off.Preprocessing follows each benchmark’s public protocol, and models generally use AdamW; EOAL uses SGD.Source: S2. Implementation details · p. 16
  • OpenPath* runs client-local frozen BiomedCLIP scoring against GPT-4-generated prompts, following the original M=8 OOD-prompt protocol.The prompt sets specify ID and plausible OOD categories for FedISIC, FedCamelyon, and FedEMBED.Source: S2. Implementation details · pp. 16–17

S3. Per-client Pareto trajectories

  • On all four FedISIC clinics, FIDAL tops or matches the best baseline by the final round despite marked label- and class-distribution heterogeneity.The per-client trajectories indicate that the aggregate gain is not an artifact of averaging over a favorable clinic.Source: S3. Per-client Pareto trajectories · p. 17, Figure S1 · p. 18

S4. Per-round composition of the labeled pool

S5. Budget and local-epoch variation

S6. Extended analysis of per-class recall on FedISIC (main-text Fig. 6)

The FedISIC per-class analysis compares recall at a common round across budgets and identifies caveats in interpreting rare-class advantages.

S6. Extended analysis of per-class recall on FedISIC (main-text Fig. 6)

  • At R=3, rare-class leads are largest at lower budgets, but the pattern does not establish that baselines rejected rare-class images specifically as artifacts.Recall is averaged over three seeds at E=100 for budgets B ∈{50, 150, 300, 500}; gaps narrow as budgets grow and baselines recover scarce classes.Source: S6. Extended analysis of per-class… · p. 23
Loading 2609.31637v1…