Source-linked AI summary
Automated pipeline for herbarium label digitization
Hiba Abbad, Hanane Ariouat, Eva Perez Pimpare, Nicolas Turenne, Eric Chenin, Abderrazak Sebaa, Edi Prifti, Jean-Daniel Zucker, Youcef Sklab
TL;DR
Herbarium label metadata remains difficult to access at scale despite extensive image digitization. HERBIOME combines detection, text localization, TrOCR recognition, and GPT-4o Mini structuring; evaluation on 450 French specimens finds training-strategy trade-offs, with taxonomic fields the main bottleneck. The pipeline also supports paired image-text datasets, although its training data and French-only scope remain limitations.
Problem
Herbarium labels contain valuable specimen metadata, but manual and existing automated transcription methods do not scale reliably across heterogeneous historical collections.
Method
HERBIOME integrates component detection, word-level localization, fine-tuned TrOCR, and GPT-4o Mini semantic structuring into an end-to-end pipeline.
Results
Hybrid Sampling achieves higher semantic fidelity than Random (SMA: 0.445 vs. 0.440), while taxonomic fields remain the principal accuracy bottleneck.
Takeaways & Limitations
Structured outputs enable paired image-text datasets that capture specimen individuality for multimodal biodiversity AI.
Takeaways & Limitations
The study is limited by a relatively small herbarium-specific training corpus and an exclusive focus on French collections.
Abstract
from arXiv · showhide
Digitized herbarium collections, now comprising over 100 million freely accessible specimen images, have become a critical resource for addressing fundamental questions in ecology and evolutionary biology. Yet the rich metadata encoded in herbarium labels (collector identities, geographic localities, collection dates, and ecological observations) remains largely inaccessible at scale, constraining both biodiversity informatics and the construction of specimen-specific image-text corpora for multimodal AI. We present HERBIOME, a modular end-to-end pipeline for automated herbarium label digitization, integrating YOLOv8-based component detection, CRAFT Hezar word-level text localization, fine-tuned TrOCR for recognition of mixed handwritten and printed text, and GPT-4o Mini for semantic metadata structuring into standardized fields. TrOCR was trained on a multi-source dataset combining general transcription corpora (CREMMA-AN, PictoCatalogs) with herbarium-specific data (RéColNat), achieving a Character Error Rate of 4.05-4.10%. End-to-end evaluation on 450 French herbarium specimens, using a dual-metric framework of Maximum Window Similarity (MWS: 0.614-0.618) and Semantic Metadata Accuracy (SMA: 0.440-0.445), reveals that hybrid training strategies improve semantic fidelity while random sampling maximizes surface similarity, with taxonomic fields remaining the principal bottleneck. By automating the extraction of structured metadata from complex, heterogeneous labels, HERBIOME reduces transcription burden, enables the construction of paired image-text datasets that faithfully capture specimen individuality, which is a prerequisite for next-generation multimodal biodiversity AI systems.
DIGITIZATION ∗
The paper presents HERBIOME as a contribution to intelligent information extraction for herbarium label digitization and multimodal biodiversity AI.
- The authors are affiliated with institutions in Algeria and France.
- HERBIOME addresses intelligent information extraction from herbarium labels.
- The work focuses on OCR and LLM-based metadata structuring.
1 Introduction
Herbarium digitization has opened large-scale research opportunities, but label metadata remains difficult to extract because manual and existing automated methods do not scale reliably. HERBIOME addresses this gap with a modular pipeline and reports TrOCR performance on French herbarium text.
- More than 100 million high-resolution herbarium specimen images are freely accessible online.
- Herbarium labels record taxonomic, geographic, temporal, collector, habitat, and historical determination information.
- Manual transcription is labor-intensive and existing OCR systems struggle with handwritten, degraded, multilingual, and heterogeneous labels.
- HERBIOME combines component detection, word localization, TrOCR recognition, and GPT-4o Mini metadata structuring.
- 4.05-4.10% Character Error Rate was achieved by training TrOCR on general and herbarium-specific transcription corpora.
- The study evaluates the pipeline on 450 real French herbarium specimens using a dual-metric framework for transcription and metadata quality.
2 Related work
Related work spans specimen analysis, component detection, OCR, and metadata structuring, while HERBIOME connects label recovery to specimen-specific multimodal herbarium datasets.
- Herbarium AI supports species identification, morphological trait extraction, and phenological research linked to climate change.
- Label and component detection: Component detectors face herbarium-specific challenges from overlapping plant material, irregular layouts, and low-resolution scans.
- Optical character recognition (OCR): OCR methods must handle mixed handwriting and print, historical degradation, and multilingual content.
- Post-OCR structuring: Post-OCR methods structure recognized text into biodiversity-standard metadata, with NER, LLM, and hybrid approaches offering different trade-offs.
- From label digitization to multimodal herbarium AI: Structured label metadata can produce paired image-text corpora grounded in specimen-specific ecological and historical context.
3 Proposed methodology: HERBIOME pipeline
HERBIOME converts raw herbarium sheets into structured metadata through sequential detection, word localization and ordering, TrOCR recognition, and GPT-4o Mini structuring.
- HERBIOME transforms raw herbarium sheet images into structured, machine-readable metadata through four sequential stages.
- Stage 1: Component detection and classification: YOLOv8-x detects eleven herbarium component types and retains high-confidence text-related crops using a 0.7 threshold.
- Stage 1: Component detection and classification: Primary label detection achieves F1 = 98.5%, while other component types achieve F1 scores between 95% and 97%.
- Stage 2: Word-level text detection: CRAFT Hezar localizes individual words, and DBSCAN groups and sorts them to restore reading order in irregular layouts.
- Stage 3: Text recognition: Fine-tuned TrOCR maps ordered word-image crops directly to character sequences and uses beam search during inference.
- Stage 4: Metadata structuring: GPT-4o Mini extracts seven metadata fields from noisy OCR text, while post-processing normalizes dates, countries, and localities and preserves missing fields as empty strings.
4 Case study and experimental evaluation
HERBIOME combines multi-source OCR training, modular preprocessing, and dual-metric evaluation on French herbarium specimens. Random sampling maximized raw OCR similarity, whereas Hybrid Sampling improved semantic metadata fidelity, with taxonomic fields remaining difficult.
- Experimental setup: Four datasets supported training and evaluation, with IRIS reserved for end-to-end assessment across all four pipeline stages.PictoCatalogs and CREMMA-AN provided printed and handwritten French material, ReColNat supplied herbarium-specific word images, and IRIS provided end-to-end ground truth.
- Experimental setup: Three training strategies differed by sampling level: Random and Hybrid operated on word-level merged data, while Two-Stage Training progressed from line images to word images.Preprocessing included RGB conversion, contrast and sharpening adjustments, aspect-ratio padding, tokenization, and corrupted-sample replacement.
- Evaluation and comparison: MWS and SMA measure complementary surface and semantic properties, so their divergent rankings expose weaknesses that a single metric would conceal.HERBIOME’s results are not directly comparable with HESPI because the systems use different corpora, fields, and metric definitions.
- OCR results: CER: 4.05% and EM: 69.3% were achieved by Random Sampling, while Two-Stage Training reached CER: 5.64% on mixed scripts.Hybrid Sampling produced more stable training dynamics at a marginal cost in raw accuracy.
- End-to-end results: SMA: 0.445 vs. 0.440 favored Hybrid over Random Sampling, despite similar surface scores on 450 IRIS specimens.The results indicate that curriculum-based training better handled format and style variation across heterogeneous labels.
- Field-level results: Collection dates were most reliably extracted, with MWS > 0.81 and accuracy > 0.66, whereas family and genus accuracy remained < 0.55.The taxonomic bottleneck reflects the sensitivity of Latin nomenclature to minor OCR errors.
5 Conclusion
HERBIOME integrates modular detection, transformer OCR, and LLM structuring to automate metadata extraction from heterogeneous herbarium labels. The evaluation framework identifies taxonomic fields as the main bottleneck, while the structured outputs support paired image-text datasets for multimodal biodiversity AI.
- HERBIOME combines detection, transformer-based OCR, and LLM-driven structuring to automate structured metadata extraction from complex herbarium labels.
- The dual-metric framework combines MWS for surface robustness with SMA for semantic fidelity and reveals training-strategy trade-offs concealed by a single metric.
- Taxonomic fields remain the principal bottleneck because Latin nomenclature is sensitive to minor recognition errors.
- The herbarium-specific training corpus is relatively small, and the evaluation focuses exclusively on French collections.
- Structured label outputs enable paired image-text datasets that capture specimen individuality for next-generation multimodal biodiversity AI systems.