PaperlayerView exact revision on arXiv

Source-linked AI summary

InfoTaxa: Information-Calibrated Label-Free Clustering for Fine-Grained Visual Taxonomy

David Ahmedt-Aristizabal, Mohammad Ali Armin, Lars Petersson

arXiv:2609.17218v1cs.CV

TL;DR

Fine-grained image clustering can support scalable biodiversity monitoring, yet image-only methods often plateau at species level. InfoTaxa combines label-free clustering with probe-based information calibration and a paired-DNA audit, finding that the tested species-level plateau is both clustering-limited and representation-limited.

  • Problem

    Image-only fine-grained taxonomy recovers broad structure but degrades at species level, raising whether failure reflects clustering, embedding geometry, or missing representation signal.

  • Method

    InfoTaxa combines image-only clustering, cross-validated probes estimating exposed image information, and held-out image+DNA late fusion used only as an audit.

  • Results

    In BIOSCAN-5M, BioCLIP 2 with UMAP and HDBSCAN reaches 0.79 family AMI and 0.67 genus AMI, while species-level DNA fusion provides a 1.99-bit held-out gain.

  • Takeaways & Limitations

    At order and family, clustering recovers most probe-estimated image information; at species, additional clustering may help but cannot close the DNA-audited gap alone.

  • Takeaways & Limitations

    Probe estimates are lower bounds, clustering efficiency is pipeline-specific rather than a formal impossibility bound, and the audit does not prove raw images lack further species cues.

Abstract

from arXiv · show

Label-free clustering of frozen pretrained visual embeddings offers a scalable route to biodiversity monitoring, but image-only fine-grained taxonomy exhibits a consistent coarse-to-fine failure mode: clusters recover broad taxonomic structure yet plateau at species level. We study this behaviour on BIOSCAN-5M through an information-calibrated clustering analysis. BioCLIP~2 features with UMAP and HDBSCAN reach $0.79$ AMI at family and $0.67$ at genus, substantially improving over the prior image baseline and remaining competitive with oracle-$K$, graph-based, and learned clustering heads on the same frozen features. To diagnose whether the remaining plateau is method-limited or information-limited, we introduce InfoTaxa, which combines clustering efficiency---the fraction of probe-estimated image information recovered by an unsupervised partition---with paired DNA as an audit signal only, not an inference input. The density pipeline recovers approximately $0.90$ and $0.81$ of the image-available information at order and family, respectively. Held-out late-fusion probes show that adding DNA to the image embedding reduces species-level prediction error by approximately two bits. Robustness analyses cover multiple image encoders, described-species and rare-class subsets, probe diagnostics, and held-out-species coarse-rank generalisation and same-species retrieval. Thus, in the tested setting, species-level label-free clustering is both clustering-limited and representation-limited: improved clustering may recover additional image-exposed structure, but cannot close the DNA-audited information gap alone.

1 Imaging and Computer Vision Group, CSIRO, Australia

The section identifies the Imaging and Computer Vision Group at CSIRO, Australia.

  • The Imaging and Computer Vision Group is affiliated with CSIRO, Australia.

1. Introduction

Image-only clustering is attractive for scalable biodiversity monitoring but consistently loses fine-grained taxonomic resolution at species level. InfoTaxa diagnoses whether this plateau reflects clustering limitations, representation limitations, or both.

  • Image-only fine-grained taxonomy recovers broad taxonomic structure but degrades sharply at the finest ranks, especially species, where visual boundaries may be subtle or cryptic.BIOSCAN-5M pairs specimen images, DNA barcodes, and Linnaean labels, making it suitable for auditing this failure mode.
  • 0.79 family AMI and 0.67 genus AMI are achieved by BioCLIP 2 with UMAP and HDBSCAN on BIOSCAN-5M, improving over the prior image baseline.
  • InfoTaxa combines image-only clustering AMI, probe-estimated image information, and held-out image+DNA late-fusion gain to separate clustering headroom from signal beyond image.Clusters are formed without BIOSCAN labels or paired DNA; DNA is used only post hoc as an audit signal.
  • 0.90 and 0.81 of probe-estimated image information are recovered at order and family, respectively, indicating limited clustering headroom on frozen features.
  • At species, clustering efficiency falls to 0.54 while adding DNA yields a 1.99-bit held-out late-fusion gain, indicating both unrecovered image structure and representation-limited signal.Improved clustering may recover more image-exposed structure but cannot remove the held-out DNA-audited signal beyond image alone.
  • InfoTaxa separates information content from clusterability: DNA exposes at least as much taxonomic information as images at every rank, while images cluster better at coarse ranks.

2. Related work

Related work spans biological vision representations, category discovery, and information-based diagnostics. InfoTaxa targets benchmark-label-free inference by combining probe-based information budgets with clustering scores.

  • Fine-grained visual recognition and biodiversity: Biological vision foundation models use large-scale organism-image pretraining and taxonomic structure, with BioCLIP 2 receiving biological supervision and DINOv2 serving as a non-biological visual control.
  • Figure 2 contrasts InfoTaxa’s image-only clustering branch with post-hoc label-based evaluation and paired-DNA auditing of late-fusion gain beyond image.
  • Category discovery and generalized discovery: Generalized category-discovery methods such as GCD and SimGCD often assume labelled examples or the true number of classes, making them upper-bound references rather than matches for benchmark-label-free inference.
  • InfoTaxa combines probe-based bit budgets with clustering scores to diagnose fine-grained visual taxonomy rank by rank.

3. Method

InfoTaxa separates image-only clustering from post-hoc audits of exposed taxonomic information and DNA-audited signal. The method evaluates clustering headroom across taxonomic ranks while preserving label-free, image-only inference.

  • InfoTaxa combines image-only clustering, probe-estimated image information, and held-out image+DNA late-fusion gain to diagnose clustering headroom and beyond-image signal.
  • The inference pipeline uses frozen image embeddings, dimensionality reduction, and unsupervised clustering without BIOSCAN labels or paired DNA.The default pipeline uses UMAP followed by HDBSCAN, which discovers the number of clusters from embedding geometry.
  • Held-out cross-validated probes estimate a practical lower bound on taxonomic information exposed by each frozen representation.Linear, wider, and non-parametric probes serve as diagnostics, while the primary probe is a shallow MLP evaluated out of sample.
  • DNA is excluded from cluster construction and instead audits incremental predictive signal through held-out late-fusion gain.A positive Gfuse indicates that adding DNA reduces held-out taxonomic prediction error; the measure is operational rather than an exact conditional-mutual-information estimate.
  • Clustering efficiency compares cluster–label mutual information with probe-estimated image information to quantify recovered structure and remaining headroom.It is a pipeline-specific diagnostic under transductive UMAP/HDBSCAN, not an exact fraction of individual image information or an impossibility theorem.
  • Robustness tests vary encoders, label provenance, rare classes, probe families and capacities, and closed-label versus unseen-species evaluation.Held-out-species probes assess family and genus generalisation, while same-species nearest-neighbour retrieval provides species-level open-set evidence.

4. Experiments

On BIOSCAN-5M, BioCLIP 2 with UMAP and HDBSCAN provides a strong label-free coarse-taxonomy operating point, but fine-rank performance reflects both clustering headroom and information beyond images. InfoTaxa separates these factors using clustering efficiency and a held-out paired-DNA audit, with robustness analyses supporting the diagnosis.

  • 4.2. Image-only clustering on BIOSCAN-5M: 0.79 family AMI and 0.67 genus AMI improve substantially over the prior image baseline, while image-only methods still decline at species and BIN.The headline pipeline remains competitive with non-semi-supervised alternatives, although several methods recover more species structure than HDBSCAN H.
  • 4.3. Clustering efficiency: At order and family, clustering efficiency reaches 0.90 and 0.81, indicating limited headroom because the density partition captures image-probe-accessible taxonomic association close in scale.Efficiency falls to 0.61 at genus and 0.54 at species, where improved clustering can still recover probe-predictive image structure.
  • 4.4. DNA audit of the species-level plateau: 1.99 bits of held-out species-level late-fusion gain shows that adding paired DNA to images reduces prediction cross-entropy beyond the frozen image embedding.At order and family, gains are only 0.06 and 0.35 bits; the marginal probe difference is supporting evidence rather than a formal conditional-information decomposition.
  • 4.4. DNA audit of the species-level plateau: The combined diagnostics indicate that species-level clustering is both clustering-limited and representation-limited: better clustering may recover image-exposed structure but cannot remove the held-out DNA-audited signal.Figure 4 explicitly presents these quantities as a diagnostic comparison, not an additive mutual-information decomposition.
  • 4.5. Robustness of the DNA audit: The DNA audit remains positive across six external image encoders, including a 1.10-bit species gain for the in-domain CLIBD probe, and persists for described-species and better-sampled subsets.Across external encoders, species-level gains range from 1.99 to 4.77 bits; restricting species to at least 20 specimens still leaves a 0.97-bit gain.
  • 4.5. Robustness of the DNA audit: Held-out-species analyses distinguish coarse-rank transfer from fine-grained retrieval: images predict family and genus better, whereas DNA leads same-species retrieval at 0.990 versus 0.780.Permutation and wider-probe diagnostics provide no evidence that the main audit result is caused by a weak image probe or detectable positive bias.

5. Discussion and limitations

InfoTaxa separates clustering headroom from information unavailable to the frozen image embedding. Its diagnosis is informative but bounded: DNA is an audit signal, probes provide lower bounds, and the evidence does not establish an absolute limit on visual taxonomy.

  • Discussion and limitations: At fine ranks, lower clustering efficiency indicates unrecovered image-exposed structure that better partitions may recover, while paired DNA adds predictive signal beyond the image embedding.The two diagnostics jointly support a clustering-limited and representation-limited interpretation of the species plateau.
  • Discussion and limitations: InfoTaxa is diagnostic rather than absolute because DNA is used only as a paired audit signal, probes are lower bounds, and efficiency is pipeline-specific.The framework does not provide a formal impossibility bound.
  • Discussion and limitations: The audit localises the gap to the frozen image embedding as read out by held-out probes, but does not prove raw images contain no further species cues.It also cannot separate missing visual evidence from information discarded by the encoder.
  • Discussion and limitations: The described-species control reduces, but cannot fully eliminate, possible dependence between barcode evidence and fine-rank labels.
  • Discussion and limitations: The full empirical validation is on BIOSCAN-5M, while FishNet supplies only a DNA-free check of coarse-to-fine image falloff.Broader validation and richer representations or sensing may expose further species cues.

6. Conclusion

The paper introduces InfoTaxa to diagnose label-free clustering of frozen visual embeddings and applies it to distinguish clustering headroom from representation limits. On BIOSCAN-5M, the tested species-level image-only setting is both clustering-limited and representation-limited.

  • Conclusion: 0.79 family and 0.67 genus AMI are achieved by BioCLIP 2 with UMAP and HDBSCAN, competitively with representative graph-based, oracle-K, and learned alternatives.
  • Conclusion: At order and family, clustering efficiency shows that the density pipeline recovers most probe-estimated image information, leaving limited headroom.
  • Conclusion: At species, the current partition leaves image-exposed structure unrecovered, so improved clustering may help.
  • Conclusion: Approximately two bits of held-out prediction cross-entropy are reduced when paired DNA is added to image, even though DNA is used only as an audit signal.
  • Conclusion: In the tested frozen embeddings, stronger clustering alone cannot eliminate the held-out signal beyond image, which may require richer visual representations or additional sensing.

A. Clustering and probe configurations

The clustering comparison fixes and documents representation, cluster-count, supervision, selection, and transductive choices across methods. Robustness analyses use frozen encoder features and show stable rank-wise behaviour across UMAP seeds, with no label-free method best at every rank.

  • A. Clustering and probe configurations: Table 7 specifies each method’s input representation, cluster-count regime, label usage, selection rule, and transductive status for the clustering comparison.Discovered-K density and graph methods use fixed configurations on UMAP-50 embeddings; oracle-K methods receive the true rank cardinality.
  • A. Clustering and probe configurations: All image features are extracted once and cached without encoder fine-tuning; CLIBD adds an in-domain multimodal image encoder pretrained with paired DNA and text supervision.
  • A. Clustering and probe configurations: Standard deviations are ≤0.015 across five UMAP seeds, indicating stable rank-wise behaviour under the fixed BioCLIP 2 + UMAP-50 protocol.FINCH is evaluated at the hierarchy level closest to the rank cardinality.
  • A. Clustering and probe configurations: No label-free method is best at every rank, so the comparison does not identify a universally dominant clustering approach.

B. Rare-class control

The rare-class control tests whether BIOSCAN-5M’s long tail explains the late-fusion DNA gain, while held-out-species evaluations separate coarse-rank transfer from species open-set retrieval. The fine-rank signal persists after removing rare classes, with different modalities favoured for coarse prediction and same-species retrieval.

  • B. Rare-class control: 0.97 bits of species-level late-fusion gain remain with at least 20 specimens per species, showing the fine-rank gap is not solely a few-shot artefact.The gain decreases as rare classes are removed, so the long tail amplifies but does not fully explain the effect.
  • B. Rare-class control: Held-out-species probes evaluate family and genus transfer beyond seen species, whereas same-species nearest-neighbour retrieval supplies the separate species-level open-set test.
  • B. Rare-class control: Image embeddings predict family and genus of unseen species better than DNA embeddings, while same-species retrieval strongly favours DNA.

D. Probe diagnostics

Probe diagnostics estimate how much taxonomic information frozen BioCLIP 2 image embeddings expose and test whether those estimates are biased by probe choice or label permutations.

  • The MLP provides the tightest tested lower bound for species-level image information, while kNN is weaker in the high-class-count regime.At family, MLP and kNN agree closely; the wider MLP is near the MLP result, while linear and kNN probes are generally looser.
  • Held-out cross-entropy is reported alongside MI because late-fusion gain is defined as the reduction in cross-entropy.The reported quantity is MI = H(Y)−CE in bits under held-out evaluation.
  • The image–DNA cross-entropy difference widens toward species and BIN, indicating a larger fine-rank separation between the modalities’ probe-exposed information.
  • Permutation controls yield zero estimated MI at every tested rank, showing no detectable positive bias under the held-out label-shuffling checks.

E. Image–DNA clusterability crossover

The image–DNA comparison separates information content from clusterability: image geometry is more clusterable at coarse ranks, whereas DNA is stronger from genus through BIN. Additional analyses show that DNA-supervised image projections do not improve raw image-only species clustering, and multi-resolution clustering remains a triage mode rather than exact taxonomic recovery.

  • E. Image–DNA clusterability crossover: Image embeddings cluster better at order and family, whereas DNA embeddings perform better from genus through BIN under oracle-K agglomerative clustering.DNA exposes at least as much probe-based taxonomic information as image at every rank, so the crossover is geometric rather than informational.
  • F. DNA-supervised image projections: None of the tested DNA-supervised image-only projections improves species AMI over raw BioCLIP 2.The evaluated projections include CCA, DNA-latent MLP distillation, and relational contrastive learning; DNA-at-inference rows are reference comparisons.
  • G. Multi-resolution Leiden sweep: The multi-resolution Leiden sweep produces full-coverage partitions across progressively finer granularities, but selected cluster counts need not match taxonomic class counts.The species and BIN operating points each use K = 3369 clusters, below 4,363 species and 5,296 BINs in the evaluation set.
  • G. Multi-resolution Leiden sweep: Leiden provides a single-resolution triage mode, whereas rank-tuned HDBSCAN achieves higher family and genus AMI.Protocol R is a diagnostic reference because it selects HDBSCAN minimum cluster size independently at each rank rather than using the fixed headline protocol.
  • H. External representation check: Probe-estimated image information is high at order and family and decreases toward finer ranks for both BioCLIP 2 and DINOv2 on FishNet.FishNet supplies an external DNA-free check of the coarse-to-fine falloff, but its sparse species labels prevent interpreting the species estimate as an information ceiling.
  • H. External representation check: The FishNet result supports a coarse-to-fine falloff on a second dataset but cannot establish a representation-limited component without the paired DNA audit.
Loading 2609.17218v1…