PaperlayerView exact revision on arXiv

Source-linked AI summary

InfoTaxa: Information-Calibrated Label-Free Clustering for Fine-Grained Visual Taxonomy

David Ahmedt-Aristizabal, Mohammad Ali Armin, Lars Petersson

arXiv:2609.17218v1cs.CV

TL;DR

Image-only visual clustering recovers broad taxonomic structure but degrades at fine ranks, especially species. InfoTaxa compares clustering with probe-estimated image information and audits it with paired DNA; at species, it finds unrecovered image structure alongside a 1.99-bit held-out image+DNA late-fusion gain.

  • Problem

    Image-only fine-grained taxonomy remains difficult because frozen visual embeddings recover broad taxonomic structure but degrade sharply at the species rank.

    Source: §1 · p. 1

  • Method

    InfoTaxa compares label-free cluster AMI with probe-estimated image information and uses paired DNA only as a held-out late-fusion audit signal.

    Source: §1 · p. 2

  • Results

    At species, clustering efficiency is 0.54 and held-out image+DNA late-fusion gain is 1.99 bits; at order and family, density clustering recovers approximately 0.90 and 0.81 of probe-estimated image information.

    Source: §1 · p. 2

  • Takeaways & Limitations

    In the tested frozen representations, species clustering is both clustering-limited and representation-limited: better clustering may recover image-exposed structure but cannot close the DNA-audited gap alone.

    Source: §1 · p. 2, §6 · p. 8

    The audit is diagnostic, not absolute: it cannot establish that raw images lack further species cues or distinguish missing visual evidence from information discarded by the encoder.

    Source: §5 · p. 8

Abstract

from arXiv · show

Label-free clustering of frozen pretrained visual embeddings offers a scalable route to biodiversity monitoring, but image-only fine-grained taxonomy exhibits a consistent coarse-to-fine failure mode: clusters recover broad taxonomic structure yet plateau at species level. We study this behaviour on BIOSCAN-5M through an information-calibrated clustering analysis. BioCLIP~2 features with UMAP and HDBSCAN reach $0.79$ AMI at family and $0.67$ at genus, substantially improving over the prior image baseline and remaining competitive with oracle-$K$, graph-based, and learned clustering heads on the same frozen features. To diagnose whether the remaining plateau is method-limited or information-limited, we introduce InfoTaxa, which combines clustering efficiency---the fraction of probe-estimated image information recovered by an unsupervised partition---with paired DNA as an audit signal only, not an inference input. The density pipeline recovers approximately $0.90$ and $0.81$ of the image-available information at order and family, respectively. Held-out late-fusion probes show that adding DNA to the image embedding reduces species-level prediction error by approximately two bits. Robustness analyses cover multiple image encoders, described-species and rare-class subsets, probe diagnostics, and held-out-species coarse-rank generalisation and same-species retrieval. Thus, in the tested setting, species-level label-free clustering is both clustering-limited and representation-limited: improved clustering may recover additional image-exposed structure, but cannot close the DNA-audited information gap alone.

1. Introduction

InfoTaxa diagnoses why image-only clustering of frozen visual embeddings recovers broad taxonomy but plateaus at species level, using clustering efficiency and paired DNA only as an audit signal. On BIOSCAN-5M, the density pipeline recovers most probe-estimated image information at order and family, but species-level error remains both clustering-limited and representation-limited.

  • Image-only fine-grained taxonomy recovers broad taxonomic structure but degrades sharply at species, motivating tests of clustering quality, embedding geometry, and missing taxonomic signal.The setting is relevant to insect biodiversity monitoring, where specimen volumes are large, expert labels are expensive, and many categories are long-tailed.Source: §1 · p. 1
  • BioCLIP 2 features with UMAP and HDBSCAN reach 0.79 AMI at family and 0.67 AMI at genus on BIOSCAN-5M, substantially improving over the prior image baseline.This is the paper’s reported image-only operating point.Source: §1 · p. 2
  • InfoTaxa combines rank-wise clustering AMI, probe-estimated image information, and held-out image+DNA late-fusion gain to quantify clustering headroom and audit signal beyond frozen image embeddings.Clustering efficiency is the fraction of probe-estimated image information recovered by an unsupervised partition; DNA is not an inference input.Source: §1 · p. 2
  • At order and family, density clustering recovers approximately 0.90 and 0.81 of probe-estimated image information; at species, efficiency is 0.54 and DNA adds a 1.99-bit held-out late-fusion gain.The marginal DNA–image probe difference is supporting evidence, not conditional mutual information.Source: §1 · p. 2
  • DNA embeddings expose at least as much taxonomic information as image embeddings at every rank, while image embeddings cluster better at coarse ranks, making the image–DNA crossover geometric rather than informational.InfoTaxa is a calibrated diagnostic combination of standard clustering, held-out probes, and paired-modality audit—not a new clustering algorithm or mutual-information estimator.Source: §1 · p. 2

2. Related work

Related work spans fine-grained biodiversity recognition, frozen-feature clustering, category discovery, DNA barcoding, and information-theoretic diagnostics. InfoTaxa combines probe-estimated image information with clustering scores and a paired-DNA audit to distinguish clustering inefficiency from limits of image representations.

Fine-grained visual recognition and biodiversity

  • BioCLIP and BioCLIP 2 use biological supervision, while BIOSCAN-5M pairs specimen images, DNA barcodes, and taxonomy for studying image-exposed taxonomic information.Prior work primarily evaluates classification or clustering performance; InfoTaxa examines taxonomic information by rank and whether the species-level plateau reflects clustering or missing image-exposed information.Source: §2 · p. 2

Label-free clustering with frozen vision features

  • SCAN learns a clustering head, FINCH and graph methods use nearest-neighbour structure, and Lowe et al. [10] benchmark conventional clusterers on frozen features.InfoTaxa calibrates unsupervised partitions against probe-estimated image information and paired DNA; “label-free” applies to clustering, not necessarily foundation-model pretraining.Source: §2 · p. 2

Category discovery and generalized discovery

  • GCD and SimGCD use partial supervision and often assume labelled examples or the true class count, unlike benchmark-label-free BIOSCAN clustering.InfoTaxa compares methods according to the information available to them, including discovered K, oracle K, learned oracle-K heads, or semi-supervised labels.Source: §2 · p. 3

DNA barcoding as a calibration signal

  • DNA barcoding can identify species when morphology is ambiguous or cryptic, while InfoTaxa uses paired DNA only to audit fine-rank signal absent from image embeddings.HyenaDNA provides sequence embeddings, and CLIBD aligns image, DNA, and text representations; neither changes InfoTaxa’s DNA-free clustering stage.Source: §2 · p. 3

Information-theoretic diagnostics

  • Because direct mutual-information estimation is statistically delicate, InfoTaxa uses held-out discriminative probes as practical lower bounds and combines probe-based bit budgets with clustering scores.Information-theoretic precedents include the information bottleneck principle [20] and mutual-information objectives in self-supervised learning [22].Source: §2 · p. 3

3. Method

InfoTaxa separates image-only clustering of frozen visual embeddings from a post-hoc audit using probes and paired DNA. The framework is instantiated on BIOSCAN-5M, with broader validation across taxa and settings left for future work.

3.1. Problem setup

  • Clusters use images only; BIOSCAN evaluation spans order, family, genus, species, and barcode index number (BIN), with species the main fine-rank biological endpoint and BIN diagnostic.Taxonomic labels and paired DNA are used only after clustering for evaluation, information measurement, and audit experiments.Source: §3.1 · pp. 3–4

3.2. Image-only clustering pipeline

  • Frozen embeddings feed label-free dimensionality reduction and clustering; default UMAP [13] followed by HDBSCAN discovers cluster count from geometry, alongside alternative partition methods.Encoders are not fine-tuned on BIOSCAN-5M; label-free inference does not imply unsupervised encoder pretraining, since BioCLIP 2 is biologically supervised and DINOv2 is a non-biological control.Source: §3.2 · p. 4

3.3. Probe-based information audit

  • Cross-validated shallow-MLP probes estimate taxonomic information, while held-out late fusion measures predictive signal gained by adding DNA to image.Probe estimates are lower bounds, not exact mutual information; late-fusion gain is operational, whereas the marginal DNA–image probe difference is not conditional mutual information or primary evidence of added signal.Source: §3.3 · p. 4

3.4. Clustering efficiency

  • Clustering efficiency compares cluster–label mutual information with probe-estimated image information to diagnose unrecovered structure across taxonomic ranks.It is a pipeline-specific empirical diagnostic, not an information bound or impossibility theorem: clustering is transductive, the denominator is a probe lower bound, and plug-in cluster MI can be positively biased.Source: §3.4 · p. 4

3.5. Robustness protocols

  • Robustness protocols vary encoders, described-species and rare-class subsets, probe families and capacities, and label permutation; unseen-species tests assess coarse-rank transfer and same-species retrieval.Held-out-species probes predict family and genus, while species-level open-set evidence comes from nearest-neighbour retrieval rather than a closed-set classifier.Source: §3.5 · p. 5

4. Experiments

On BIOSCAN-5M, BioCLIP 2 image clustering recovers strong coarse-rank structure but declines at species and BIN. InfoTaxa finds both remaining image-exposed clustering headroom and a substantial held-out taxonomic signal from paired DNA beyond the image embedding.

4.1. Experimental setup

  • The evaluation uses 47,260 BIOSCAN-5M specimens with labels at five taxonomic ranks and frozen encoders; clustering uses neither taxonomic labels nor paired DNA at inference.The headline image representation is BioCLIP 2 ViT-L/14 with UMAP-50 and HDBSCAN; its biological pretraining supervision means label-free refers to inference-time clustering, not pretraining.Source: §4.1 · p. 5

4.2. Image-only clustering on BIOSCAN-5M

  • BioCLIP 2 + UMAP-50 + HDBSCAN raises family AMI from approximately 0.36 to 0.79 and genus AMI from approximately 0.29 to 0.67 versus the prior baseline; FINCH reaches 0.594 species AMI versus HDBSCAN’s 0.500.The prior reference used DINO-v1, UMAP-50, and Ward; the headline pipeline remains competitive with non-semisupervised alternatives at family and genus, while image-only methods decline at species and BIN.Source: §4.2 · pp. 5–6, Figure 3 · p. 6, Table 2 · p. 6

4.3. Clustering efficiency

  • At order and family, headline HDBSCAN recovers 0.90 and 0.81 of probe-estimated image information, but efficiency falls to 0.61 at genus and 0.54 at species.The diagnostic indicates limited clustering headroom at coarse ranks but substantial probe-predictive image structure unrecovered at fine ranks.Source: §4.3 · p. 6, Table 3 · p. 6

4.4. DNA audit of the species-level plateau

  • Adding paired DNA to image reduces held-out cross-entropy by 1.99 bits at species, while gains are 0.06 bits at order, 0.35 at family, and 1.02 at genus.The species-level supporting marginal probe difference is 2.00 bits, but it is not interpreted as conditional mutual information or a formal decomposition of DNA-exclusive information.Source: §4.4 · pp. 6–7, Table 4 · p. 6

4.5. Robustness of the DNA audit

  • The species-level DNA audit persists across controls: described-species late-fusion gain is 1.84 bits, and gains across six external image encoders range from 1.99 to 4.77 bits.For described species, the supporting marginal difference is 1.85 bits versus 2.00 bits for all species; in-domain CLIBD narrows the gain to 1.10 bits, while the audit gap remains above one bit at species and BIN.Source: §4.5 · p. 7, Table 5 · p. 7, Table 6 · p. 7
  • The species-level late-fusion gain decreases from 2.00 bits for all species to 0.97 bits among species with at least 20 specimens; same-species retrieval on held-out species favors DNA, 0.990 versus 0.780.Image embeddings predict family and genus of held-out species better than DNA embeddings; probe diagnostics show zero estimated MI under permutation and no species-level image-estimate improvement from a wider MLP.Source: §4.5 · pp. 7–8

4.6. Additional validation and operating modes

  • Oracle-K comparisons show an image–DNA clusterability crossover despite DNA exposing at least as much probe-based taxonomic information; CCA reaches 0.510 species AMI versus 0.562 for raw images.The comparison finds image clusters better at order and family, DNA clusters better from genus to BIN, and rank-tuned HDBSCAN provides a stronger family/genus diagnostic reference than the full-coverage Leiden sweep.Source: §4.6 · p. 8

5. Discussion and limitations

Across taxonomic ranks, InfoTaxa finds strong family/genus clustering but species remains limited by both clustering efficiency and the frozen image representation. Its paired-DNA audit adds predictive signal, while the evidence is diagnostic and scoped to BIOSCAN-5M.

Discussion and limitations

  • BioCLIP 2 with UMAP and HDBSCAN is a strong family/genus operating point, but species clustering remains both clustering-limited and representation-limited.Lower clustering efficiency indicates unrecovered image-exposed structure, while held-out late fusion shows that paired DNA adds substantial predictive signal beyond the frozen image embedding.Source: §5 · p. 8
  • “Label-free” refers to inference-time clustering: BIOSCAN labels and paired DNA do not form clusters, although BioCLIP 2 received biological pretraining supervision.The marginal DNA–image difference supports the audit but is not interpreted as conditional mutual information.Source: §5 · p. 8
  • The audit diagnoses the frozen image embedding as read by held-out probes, not an absolute limit on species information in raw images.DNA is only a paired audit signal, probes are lower bounds, and clustering efficiency is pipeline-specific rather than an impossibility bound; the described-species control reduces but cannot eliminate possible barcode-label dependence. Diagnostics find no evidence of probe underfitting or positive estimator bias, and CLIBD narrows but does not close the gap. Empirical paired-DNA validation is on BIOSCAN-5M; FishNet checks only coarse-to-fine image falloff, so broader validation remains future work.Source: §5 · p. 8

6. Conclusion

InfoTaxa diagnoses label-free clustering of frozen visual embeddings as effective at recovering image-exposed structure at coarser ranks, but limited at species by both unrecovered image structure and information beyond the image. Paired DNA serves only as an audit signal, not a partition input.

6. Conclusion

  • 0.79 family and 0.67 genus AMI: BioCLIP 2 with UMAP and HDBSCAN is competitive with representative graph-based, oracle-K, and learned alternatives on BIOSCAN-5M.Label-free means that neither BIOSCAN labels nor paired DNA form the partition.Source: §6 · p. 8
  • At order and family, the density pipeline recovers most probe-estimated image information, leaving limited headroom for clustering improvement.Clustering efficiency measures the fraction of probe-estimated image information recovered by an unsupervised partition.Source: §6 · p. 8
  • At species, the current partition leaves image-exposed structure unrecovered, while adding paired DNA reduces held-out prediction cross-entropy by approximately two bits.Paired DNA is used only as an audit signal; stronger clustering may recover more image-exposed structure, but cannot eliminate held-out signal beyond the image alone.Source: §6 · p. 8
  • The species-level finding is robust across encoder, label-provenance, rare-class, and probe controls in the tested frozen embeddings.The results indicate that image-only species clustering is both clustering-limited and representation-limited; additional progress beyond image may require richer visual representations or additional sensing.Source: §6 · p. 8

Supplementary Material · A. Clustering and probe configurations

This supplement documents clustering and encoder operating points, then assesses seed stability and CLIBD robustness. Across five UMAP seeds, rank-wise behavior was stable, while CLIBD narrowed but did not close the fine-rank DNA-audited information gap.

A. Clustering and probe configurations

  • Operating points distinguish discovered-K from oracle-K methods; oracle methods receive rank cardinality only, while learned heads use self-supervision and oracle K.SimGCD alone uses BIOSCAN labels as a semi-supervised upper-bound reference, and no label-free operating point is selected using test-set AMI.Source: A. Clustering and probe configurations · p. 11, Table 7 · p. 11
  • Discovered-K density and graph methods use fixed configurations on UMAP-50 embeddings, and the comparison specifies selection rules and whether inference is transductive.Table 7 documents the clustering configurations and operating-point selection.Source: A. Clustering and probe configurations · p. 11, Table 7 · p. 11
  • Encoder robustness uses cached frozen features without fine-tuning; CLIP-L denotes MetaCLIP ViT-L/14 [27], while CLIBD [6] adds an in-domain multimodal encoder.CLIBD’s released image tower was pretrained on BIOSCAN-5M with paired DNA and text supervision; its evaluation-distribution overlap makes it an in-domain robustness probe rather than the headline operating point.Source: A. Clustering and probe configurations · p. 11, Table 8 · p. 11
  • Across five UMAP seeds, standard deviations were ≤0.015, but no label-free method led every rank; Leiden and Ward exceeded fixed HDBSCAN at species.FINCH was evaluated at the hierarchy level with cluster count closest to rank cardinality.Source: Table 9 · p. 11, A. Clustering and probe configurations · p. 11
  • With CLIBD under protocol H, AMI(global) was 0.563±.013 at order, 0.776±.001 at family, 0.634±.003 at genus, 0.489±.002 at species, and 0.474±.002 at BIN.These values were comparable to the BioCLIP 2 headline operating point in Table 9.Source: A. Clustering and probe configurations · p. 11
  • CLIBD exposed 10.54 bits of species-level image information; its held-out species late-fusion gain was Gfuse=1.10±.01 bits (∆=1.11), and BIN Gfuse was 1.12±.01 bits.The gap remained above one bit at both fine ranks.Source: A. Clustering and probe configurations · p. 11

B. Rare-class control · C. Held-out-species coarse-rank generalisation and same-species retrieval

The rare-class control shows that long-tailed sampling amplifies, but does not fully explain, the species-level late-fusion gain. On held-out species, image embeddings better predict family and genus, while DNA strongly favors same-species retrieval.

B. Rare-class control

  • The rare-class control repeats the held-out late-fusion audit after retaining species with at least 5, 10, or 20 specimens.The gain decreases across these increasingly restrictive subsets.Source: B. Rare-class control · p. 12, Table 10 · p. 12
  • 0.97 bits of species-level late-fusion gain remain among species with at least 20 specimens, so the long-tail effect is not solely a few-shot artefact.The gain decreases as rare species are removed.Source: Table 10 · p. 12, B. Rare-class control · p. 12

C. Held-out-species coarse-rank generalisation and same-species retrieval

  • Random specimen folds measure closed-label representation information because species can occur in both probe-training and evaluation folds.The held-out-species evaluation instead excludes every specimen of selected species from probe training and tests family and genus prediction.Source: C. Held-out-species coarse-rank… · p. 12, Table 11 · p. 12
  • Held-out-species probes evaluate coarse-rank transfer only, not species classification.Table 11 reports family and genus top-1 accuracy on species excluded from training.Source: C. Held-out-species coarse-rank… · p. 12, Table 11 · p. 12
  • Image embeddings predict the family and genus of unseen species better than DNA embeddings, whereas same-species retrieval strongly favours DNA.Table 11 reports the coarse-rank entries as held-out top-1 accuracy and retrieval as kNN@1; Joint denotes late fusion of image and DNA representations.Source: Table 11 · p. 12, C. Held-out-species coarse-rank… · p. 12
  • Same-species nearest-neighbour retrieval tests whether unseen-species specimens remain close to conspecifics in the frozen embedding space, without fitting a closed-set species classifier.The test evaluates species-level open-set evidence separately from the family and genus probes.Source: C. Held-out-species coarse-rank… · p. 12, Table 11 · p. 12

D. Probe diagnostics

Probe diagnostics find that MLPs provide the tightest tested species-level image lower bound, while permutation controls yield zero estimated MI. Across representations, the image–DNA cross-entropy difference widens toward species and BIN.

D. Probe diagnostics

  • At species, kNN is weaker than MLP while the probes agree closely at family; the permutation control yields zero estimated MI.This indicates no detectable positive bias under the tested permutation check.Source: D. Probe diagnostics · p. 12, Table 12 · p. 12
  • Table 13 reports held-out cross-entropy and implied MI in bits because late-fusion gain is defined as a held-out cross-entropy reduction; image-probe permutation MI is zero at every rank.The shuffled-label control's held-out cross-entropy does not improve on the marginal predictor.Source: D. Probe diagnostics · p. 12, Table 13 · p. 12
  • Across tested probe families, the MLP ties for or provides the largest lower bound in each setting; the wider MLP is close, while linear and kNN probes are generally looser.The image–DNA cross-entropy difference widens toward species and BIN.Source: D. Probe diagnostics · p. 12, Table 13 · p. 12

E. Image–DNA clusterability crossover

Under oracle-K agglomerative clustering, image embeddings perform better at order and family, while DNA embeddings perform better from genus through BIN; this crossover reflects embedding geometry, not probe-exposed information. DNA exposes at least as much probe-based taxonomic information as image at every rank.

E. Image–DNA clusterability crossover

  • DNA exposes at least as much probe-based taxonomic information as image at every rank.Thus, the clustering crossover does not mean DNA lacks coarse-rank information.Source: E. Image–DNA clusterability crossover · pp. 12–13
  • Image embeddings cluster better at order and family, whereas DNA embeddings perform better from genus through BIN under oracle-K agglomerative clustering.The crossover reflects embedding geometry rather than probe-exposed information.Source: E. Image–DNA clusterability crossover · p. 13, Figure 5 · p. 13
  • None of the DNA-supervised image projections improves species clustering over raw BioCLIP 2 on held-out test unseen data.Rows below the table divider use DNA at inference and are reference comparisons.Source: Table 14 · p. 13

F. DNA-supervised image projections · G. Multi-resolution Leiden sweep · H. External FishNet coarse-to-fine falloff

Image-only DNA-supervised projections did not improve species AMI over raw BioCLIP 2, while multi-resolution Leiden offers full-coverage coarse-to-fine partitions that are not exact taxonomic-count recovery. FishNet also shows a coarse-to-fine image-information falloff, but its sparse species labels and lack of paired DNA preclude interpreting this as an information ceiling.

F. DNA-supervised image projections

  • None of the tested image-only DNA-supervised representations—CCA, DNA-latent MLP distillation, or relational contrastive mapping—improves species AMI over raw BioCLIP 2.The two DNA and image–DNA fusion rows use DNA at inference and are reference comparisons, not image-only candidates.Source: F. DNA-supervised image projections · p. 13

G. Multi-resolution Leiden sweep

  • A single Leiden resolution parameter produces full-coverage partitions with progressively finer granularity on the BioCLIP 2 + UMAP-50 shared-nearest-neighbour graph.The sweep supplies candidate operating points from family through fine ranks.Source: Figure 6 · p. 13, G. Multi-resolution Leiden sweep · p. 13

F. DNA-supervised image projections

  • DNA and image–DNA fusion yield stronger fine-rank clustering in reference comparisons, but do not change the image-only result.These comparisons use DNA at inference and are not image-only candidates.Source: F. DNA-supervised image projections · p. 13

G. Multi-resolution Leiden sweep

  • The selected Leiden species and BIN operating points each have K = 3369 clusters, below the evaluation-set counts of 4,363 species and 5,296 BINs.The sweep is a low-tuning, full-coverage triage mode rather than exact recovery of Linnaean class counts.Source: G. Multi-resolution Leiden sweep · p. 13

H. External FishNet coarse-to-fine falloff

  • On FishNet, probe-estimated image information as a fraction of label entropy is high at order and family and decreases toward finer ranks for BioCLIP 2 and DINOv2.The species-level estimate is not interpreted as evidence of an information ceiling because FishNet species are extremely sparse.Source: H. External FishNet coarse-to-fine… · p. 13, Figure 7 · p. 14

G. Multi-resolution Leiden sweep

  • Leiden provides full coverage, but rank-tuned HDBSCAN protocol R achieves higher family and genus AMI on BioCLIP 2 + UMAP-50.Protocol R tunes minimum cluster size independently at each rank using AMI and is a diagnostic reference, not the fixed headline protocol H.Source: G. Multi-resolution Leiden sweep · p. 13, Table 15 · p. 14

H. External FishNet coarse-to-fine falloff

  • FishNet supports coarse-to-fine falloff on a second biological dataset, but cannot establish an information ceiling or representation-limited component.FishNet lacks the paired DNA audit, and its sparse species labels make the species-level estimate confounded by few-shot sampling.Source: H. External FishNet coarse-to-fine… · p. 14

Related papers

Loading 2609.17218v1…