Source-linked AI summary
Multimodal Taxonomic Conditioning for Generative Plankton Imagery
Daniela Ivanova, Ozgu Goksu, Nicolas Pugeault
TL;DR
Rare plankton taxa are underrepresented in automated-imaging datasets, limiting reliable recognition of ecologically important classes. The paper learns taxonomic structure with a ranked contrastive CLIP adaptation for deep, ragged lineages and uses frozen embeddings to condition a parameter-efficient diffusion transformer. The approach improves synthetic-data fidelity and replacement-based classifier utility, while its rare-class augmentation advantage is not distinguished from simpler baselines.
Problem
Severely long-tailed plankton datasets can leave ecologically important rare taxa with fewer than ten images, limiting reliable classifier training and evaluation.
Method
The method adapts CLIP with ranked contrastive supervision for deep, ragged taxonomies, then freezes its multimodal embeddings to condition a parameter-efficient diffusion transformer.
Results
FID improves from 22.43 to 19.17, while generated-only classifier training raises macro-F1 from 0.603 to 0.664; rare-class augmentation does not distinguish generators from duplication.
Takeaways & Limitations
Taxonomy-aware conditioning supports higher-fidelity synthetic plankton imagery and better substitution of real training data in the reported evaluation.
Takeaways & Limitations
Image preprocessing compresses aspect ratios, inflating all FID scores equally, and frozen semantic embeddings cannot transfer from specimens whose appearance mismatches their text-nearest neighbors.
Abstract
from arXiv · showhide
Automated plankton imaging produces severely long-tailed datasets, where the rare taxa of greatest ecological interest have too few images to train or evaluate classifiers reliably. We generate synthetic plankton imagery conditioned on taxonomy: a CLIP encoder is adapted on a large plankton corpus with a ranked contrastive objective extended to deep, ragged taxonomies, then frozen to condition a parameter-efficient diffusion transformer. We evaluate synthetic sample quality on distributional fidelity and downstream classifier utility.
1 Introduction
The paper targets severely long-tailed plankton datasets, where rare ecologically important taxa may have fewer than ten images. It proposes learning taxonomic structure in embeddings before using those embeddings to condition a parameter-efficient diffusion model.
- Rare plankton taxa may have fewer than ten images, while abundant taxa dominate automated-imaging datasets.
- Taxonomy can transfer visual information from abundant relatives to rare species because related taxa share more morphology.
- A frozen taxonomy-aware CLIP encoder conditions a parameter-efficient diffusion model, separating hierarchical representation learning from generation.
- The method adapts ranked contrastive learning to deep, ragged taxonomies with truncation-aware depth matching.
- Evaluation covers both distributional fidelity and downstream classifier utility.
2 Related Work
Prior work addresses plankton recognition, fine-grained conditional generation, and hierarchical contrastive learning, but each leaves limitations for deep, unevenly annotated plankton taxonomies. These gaps motivate embedding-based taxonomic conditioning and a deeper ranked objective.
- Plankton recognition has used oversampling, self-supervised pretraining, and large multi-instrument corpora to address imbalanced classification.
- Fine-grained diffusion methods improve conditioning for many visually similar classes through parameter-efficient adaptation or hierarchical guidance.
- Flat InfoNCE treats all nonmatching samples equally, whereas RINCE grades negatives by taxonomic rank.
- Existing ranked-contrastive experiments assume two ranks and complete labels, unlike plankton taxonomies that are deep and often identified only to genus or family.
3 Method
The method adapts CLIP using rank-aware contrastive supervision over deep, ragged taxonomies, then uses the frozen multimodal embeddings to condition a parameter-efficient diffusion transformer. It handles incomplete lineage annotations through shared-depth ranks and coverage-based weighting.
- The method forms cumulative lineage strings at each rank and directly supervises graded similarity so taxonomic structure enters embedding geometry before generation.
- A CLIP text encoder is adapted on a large plankton corpus, then its frozen text and image embeddings replace a diffusion model’s learned class-embedding table.
- Ranked contrastive learning: RINCE partitions positives by decreasing similarity and recursively treats coarser ranks as negatives at each level.
- Taxonomy-aware text encoder: Ragged taxonomies are handled by assigning pairwise rank from shared lineage depth, including matching truncation.
- Taxonomy-aware text encoder: Each rank is weighted by its batch coverage rather than uniformly, because different ranks constrain different portions of a ragged batch.
- Taxonomy-conditioned generation: The diffusion transformer trains only its conditioning embedder, biases, and normalization layers, totaling 2.5M of 676M parameters.
4 Experiments
Experiments use a disjoint, long-tailed IFCB dataset to test synthetic replacement and rare-class augmentation. Fidelity is measured with FID, while classifier utility is measured by macro-F1 on a shared real test split.
- The evaluation dataset contains 74,181 IFCB images across 145 taxonomic classes and is disjoint from the corpus used to adapt the text encoder.
- Synthetic replacement creates one generated image per real training image, while augmentation tops up 58 underrepresented classes to 100 images.
- FID compares each model’s replacement set with the real training split after identical resizing, while classifier utility uses macro-F1 on a shared real test split.
- Generation compresses aspect ratios relative to real imagery, inflating all methods’ FID equally; real-versus-real calibration is 10.98.
5 Results
The evaluation compares taxonomy-conditioned generators with baselines on qualitative fidelity, distributional fidelity, and classifier utility across rare and common plankton classes. The method improves replacement quality overall but is weaker for the smallest classes and does not clearly outperform duplication in rare-class augmentation.
- Qualitative comparison: Figure 1 compares real specimens with one sample from each generator for three rare classes containing 2, 13, and 7 images and three common classes containing 3591, 1492, and 3950 images.The qualitative comparison directly contrasts generator outputs across highly imbalanced training-set sizes.
- Overall results: FID 19.17 versus 22.43, while generated-only training raises macro-F1 from 0.603 to 0.664.The three generators are ordered identically by FID and replacement utility, with FineDiffusion the stronger baseline.
- Smallest classes: The conditioning loses 0.113 macro-F1 to duplication across four classes with at most two training images.These are the smallest classes evaluated in the reported rare-class comparison.
- Specimen-conditioned generation: Figure 2 illustrates specimen-conditioned generation by producing four samples from each real image’s CLIP embedding rather than from a class prototype.This demonstrates conditioning at the individual-specimen level.