PaperlayerView exact revision on arXiv

Source-linked AI summary

A Visual Classification Dataset and Model Evaluation for Historical Manuscript Illustrations

Yoav Evron, Michal Bar-Asher Siegal, Michael Fire

arXiv:2609.33449v1cs.CVcs.LG

TL;DR

Large digitized manuscript collections remain difficult to organize and explore, and extraction alone does not classify illustration content. The paper builds a labeled dataset and benchmarks classification approaches; fine-tuned image classifiers perform best overall, while taxonomy overlap remains a challenge.

  • Problem

    Digitized manuscript collections are difficult to explore at scale, and extraction systems locate illustrations without organizing their semantic content.

    Source: §1 · pp. 1–2

  • Method

    The paper creates a manually labeled dataset with a compact taxonomy and compares supervised classifiers, embedding-based methods, zero-shot approaches, and vision-language models.

    Source: §1 · p. 3, §3 · p. 4

  • Results

    Fine-tuned classifiers perform best overall, with ConvNeXt reaching 88.9% accuracy and 81.3% macro-F1; embedding-based methods are strong alternatives, while zero-shot approaches remain below supervised classifiers.

    Source: §6 · p. 7, §7 · p. 10

  • Takeaways & Limitations

    The benchmark identifies visually separable categories and cases that remain challenging because of genuine taxonomy overlap.

    Source: §7 · p. 11

    The single-label protocol supports consistent model comparison but simplifies the multi-label nature of manuscript imagery.

    Source: §7 · p. 9

Abstract

from arXiv · show

Historical manuscript illustrations preserve rich visual evidence of past cultures. They depict people, animals, plants, diagrams, music notations, and decorative forms. Although large digitization projects have made many manuscripts available online, the material itself remains difficult to explore at scale. Extraction systems can find illustrations on manuscript pages, but without meaningful categories, large collections remain hard to search and explore. We address this gap by introducing a manually labeled dataset of 15,000 illustrations from manuscripts dating back hundreds of years across 22 categories, and evaluating modern vision models for image classification on this task. The problem is challenging due to stylistic diversity, degradation, and semantic ambiguity, with many images that fit more than one category. We compare fine-tuned CNN and Transformer-based classifiers, zero-shot CLIP, embedding-based classifiers, and direct vision-language models. Results show that fine-tuned image classifiers perform best overall, with ConvNeXt reaching 88.9% accuracy and 81.3% macro-F1. Using CLIP embeddings with XGBoost provides a strong alternative. In contrast, zero-shot CLIP and direct vision-language classification perform substantially worse, highlighting the limits of general-purpose models in this domain. Beyond overall performance, the analysis reveals which categories are visually separable and where errors reflect genuine semantic overlap, suggesting that some limitations arise from the taxonomy itself.

1 Introduction

Digitized manuscript collections are difficult to organize and explore at scale, and extraction alone does not classify their visual content. The paper introduces a labeled dataset and categorization approach to support systematic exploration, while highlighting ambiguity in both the images and their categories.

1 Introduction

  • Digitization makes millions of manuscript pages available, but their visual material exceeds what can be explored manually, and extraction alone does not organize it.Categorical organization complements search by supporting browsing and comparison when relevant content is unknown in advance.Source: §1 · pp. 1–2
  • Generated captions can be incomplete or unreliable for degraded, culturally specific pages, while current vision-language training data largely reflects modern web imagery [9].Existing classification work focuses mainly on page elements rather than the semantic content of illustrations.Source: §1 · p. 2
  • The task is difficult because illustrations may combine categories, such as animals within decorated initials, and historical images may be degraded [13] [14].The taxonomy is intended to remain stable yet flexible enough for large-scale analysis.Source: §1 · p. 2
  • The proposed taxonomy assigns each image a dominant category and uses annotation rules to handle ambiguity, balancing historical relevance with usability.It was developed through data-driven analysis, alignment with art-historical systems, and iterative manual refinement.Source: §1 · p. 2
  • Fine-tuned classifiers perform best overall, while CLIP-embedding classifiers offer an accuracy-efficiency trade-off when embeddings are already available.The analysis also finds that many errors reflect semantic overlap between categories.Source: §1 · p. 3
  • 15,000 manually verified illustrations across 22 categories and a unified model benchmark address the limited evaluation of semantic classification for manuscript illustrations.The paper also analyzes class-level performance, semantic overlap, embedding structure, and taxonomy-related ambiguity.Source: §1 · pp. 2–3

2 Related work

Prior work has detected or extracted manuscript illustrations, but has generally not organized their semantic content into fine-grained categories. The paper compares model families because their suitability for visually detailed and semantically rich manuscript images remains uncertain.

2 Related work

  • Earlier historical document work focused on text tasks such as OCR and layout analysis [13], with newer work detecting and extracting visual elements [25].These efforts primarily locate illustrations rather than organize or analyze their content.Source: §2 · p. 3
  • Prior systems detect or extract illustrations but often collapse distinct semantic content into a broad image category, and may not use manuscript-level splits or audit false-positive extractions.S-VED and AnnoPage jointly localize visual elements and assign broad classes; docExtractor [25] performs page-level segmentation with a general illustration label.Source: §2 · p. 3
  • CNNs and Transformer architectures are established image-classification families [18], but their relative suitability for manuscript illustrations is unclear.The images combine fine-grained visual detail with higher-level semantic content [30].Source: §2 · p. 3
  • CLIP [19] supports zero-shot classification through shared image-text embeddings, while LLaVA [20], Qwen2-VL [21], and InstructBLIP [22] extend vision-language modeling to description and visual question answering.Their effectiveness in specialized historical domains remains unclear, particularly for unfamiliar, degraded, or semantically ambiguous images.Source: §2 · pp. 3–4

3 Dataset

The dataset comprises extracted illustration crops from Vatican Library manuscripts and uses a compact taxonomy designed for consistent, practical classification. A single dominant-category label simplifies multi-content images for comparable evaluation, while labeling combines model-assisted retrieval with human verification.

3 Dataset

  • The dataset contains 15,000 manually labeled illustrations across 22 categories, with a compact taxonomy, annotation protocol, and manuscript-level splits.The domain includes stylistic diversity, historical specificity, and images combining multiple visual functions.Source: §3 · p. 4
  • Images are extracted regions rather than full pages, isolating individual elements that may otherwise appear alongside text, decorated initials, marginalia, or diagrams.The crops vary considerably in size; median resolution is 697×749 pixels and median width-to-height ratio is 0.95.Source: §3 · p. 4
  • The compact 22-category taxonomy spans object types, visual configurations, and artifact or non-illustration classes, balancing research meaning with annotation feasibility.It is informed by data-driven exploration, art-historical concepts, and manual inspection; artifact classes can support filtering in extraction pipelines.Source: §3 · p. 4
  • Each image receives one category for its dominant semantic content rather than labels for every element, with guidelines resolving cases such as animals in decorated initials.The single-label design trades representational richness for evaluation stability, reproducibility, and practical collection organization.Source: §3 · pp. 4–5
  • Generated captions guide category-specific retrieval of candidates, but annotators manually verify labels and select examples across manuscripts, styles, layouts, and visual configurations.Captions are retrieval aids rather than ground-truth labels because they can omit, confuse, or invent visual details.Source: §3 · p. 5
  • Every image receives a second human verification, and an independent labeling check of 250 images yielded 82.4% raw agreement and Cohen’s κ = 0.79.A third annotator adjudicated flagged cases, producing 69 revised labels incorporated into the dataset and experiments.Source: §3 · p. 5

4 Experiments

The experiments compare supervised classifiers, zero-shot vision-language models, and classifiers trained on fixed embeddings using a common class set and data split. Manuscript-level splits are repeated across five seeds to assess variation across splits.

4 Experiments

  • All model families use the same class set and a 70/15/15 train-validation-test split grouped by manuscript within each category.The procedure is repeated five times with seeds 42–46, reporting mean and standard deviation of test metrics.Source: §4 · p. 5
  • The supervised comparison fine-tunes EfficientNetV2-S, ConvNeXt-Tiny, and ViT-B/16 [18] with class-weighted cross-entropy and validation macro-F1 early stopping.Training runs for up to 12 epochs with patience of 4 epochs.Source: §4 · pp. 5–6
  • Zero-shot experiments evaluate short- and long-prompt CLIP [19] and direct classification by LLaVA [20], Qwen2-VL [21], and InstructBLIP [22].Direct models receive the full category list and generate one category name using deterministic decoding.Source: §4 · p. 6
  • Embedding-based experiments train XGBoost on CLIP image embeddings or on fused CLIP image and LLaVA-caption text embeddings.Both use a multi-class soft-probability objective with 300 trees, depth 6, and inverse-frequency sample weights.Source: §4 · p. 6

5 Evaluation

The evaluation compares model families under a common setup, using aggregate and category-level metrics alongside runtime. Manuscript-level splits are used to assess generalization to unseen manuscripts.

5 Evaluation

  • Macro-F1 gives equal weight to all categories, offering a more reliable performance view under the dataset’s strong class imbalance.The evaluation also reports accuracy, macro-averaged precision and recall, per-class results, and confusion patterns.Source: §5 · pp. 6–7
  • Runtime is measured in seconds per test image because the intended application involves collections with hundreds of thousands of illustrations.Timings include the full model-specific inference pipeline, including embedding extraction or caption generation where applicable.Source: §5 · p. 7
  • Images from each manuscript are assigned exclusively to training, validation, or test sets within each category, preventing style-specific leakage.This split is intended to provide a more realistic measure of generalization to unseen manuscripts.Source: §5 · p. 7
  • The CLIP embedding analysis uses t-SNE and hierarchical clustering to examine category structure and help explain confusion among visually or semantically similar classes.These analyses provide a qualitative view of category organization in the embedding space.Source: §5 · p. 7

6 Results

The results establish a benchmark on 15,000 labeled illustrations and show that fine-tuned classifiers lead overall. Category-level analyses also reveal overlaps and distinct patterns across model families.

6 Results

  • 15,000 manually labeled illustrations across 22 visual categories form the curated dataset used for evaluation.The categories range from people, animals, and plants to diagrams, decorative elements, and non-illustration classes.Source: §6 · p. 7
  • 88.9% accuracy and 81.3% macro-F1: ConvNeXt leads overall and is the fastest end-to-end model.Embedding-based classifiers, particularly CLIP-XGBoost and CLIP-LLaVA-XGBoost, are competitive and outperform the Vision Transformer baseline; off-the-shelf methods generally perform worse than supervised approaches.Source: §6 · p. 7, Table 1 · p. 8
  • Longer, descriptive CLIP prompts outperform short category labels, while vision-language approaches are slower and harder to scale than fine-tuned classifiers.CLIP-LLaVA-XGBoost also incurs substantially higher inference time; embedding-based methods remain relatively fast.Source: §6 · p. 8
  • ConvNeXt confuses Fight with Multiple Figures, and also confuses decorated elements and some Architecture scenes with visually related categories.The reported examples include crowded scenes, groups around a sick or treated person, and buildings within outdoor scenes.Source: §6 · p. 8
  • Music Notations, Map, and Decorated Initial form relatively coherent CLIP embedding clusters, while overlapping categories are more mixed.Hierarchical clustering also places related categories near one another, including Person, Multiple Figures, and Fight.Source: §6 · p. 8, Figure 2 · p. 9

7 Discussion

The discussion finds that model performance depends on task-specific training, annotation choices, and category structure. It also identifies dataset scope and single-label annotation as important boundaries on the conclusions.

7 Discussion

  • The Vatican Library is the benchmark’s sole institutional source, and candidate collection relied on model-assisted retrieval that may introduce selection bias.The study describes deliberate sampling across manuscripts, styles, layouts, and visual configurations.Source: §7 · p. 9
  • Single-label annotation simplifies manuscript imagery’s multi-label nature, so some classification errors reflect that simplification rather than clear model failures.The discussion points to multi-label or hierarchical annotation, or decomposing images into multiple objects, as possible extensions.Source: §7 · pp. 9–10
  • The per-class F1 heatmap compares model performance across 22 categories, highlighting consistently easy or difficult categories and differences between model families.The discussion’s error and embedding analyses find that many misclassifications occur between semantically overlapping categories.Source: Figure 3 · p. 10, §7 · p. 10
  • Errors between semantically overlapping categories indicate that performance is closely tied to taxonomy structure, not only visual recognition.The paper says category definitions and handling ambiguous or multi-content images remain open challenges.Source: §7 · pp. 10–11
  • Task-specific training is highly effective: ConvNeXt has the highest mean performance, while embedding-based methods outperform the Vision Transformer baseline.Direct vision-language models remain below supervised classifiers, even though Qwen2-VL substantially outperforms LLaVA and InstructBLIP.Source: §7 · p. 10
  • Manuscript-specific decorative classes are underrepresented in general-purpose training data, while short and long CLIP prompts produce markedly different zero-shot performance.The discussion also identifies inference time as a practical factor in choosing between ConvNeXt and CLIP-based classifiers.Source: §7 · p. 11
  • The benchmark identifies visually separable and persistently challenging categories, and points toward dataset expansion and multi-label or hierarchical approaches.The paper presents large-scale classification as feasible while describing key remaining challenges as conceptual.Source: §7 · p. 11

8 Conclusions

The paper introduces a dataset and practical taxonomy for classifying historical manuscript illustrations, finding that fine-tuned image classifiers perform best overall. Its results also show that taxonomy structure and semantic overlap matter alongside model choice.

8 Conclusions

  • 15,000 illustrations labeled across 22 categories form a dataset and evaluation setting for organizing historical manuscript imagery at scale.The work also provides a practical taxonomy and annotation protocol.Source: §8 · p. 11
  • Fine-tuned image classifiers provide the most reliable performance, with ConvNeXt achieving the best overall balance of accuracy, macro-F1, and inference speed.Embedding-based approaches remain useful when CLIP representations are already part of the pipeline, but are not the most efficient end-to-end solution in this setting.Source: §8 · p. 11
  • Many errors reflect semantic overlap and data ambiguity, so performance depends on taxonomy structure and evaluation protocol as well as model choice.The authors conclude that progress requires clearer problem formulation alongside better models.Source: §8 · p. 11

10 Appendix

The appendix describes a practical taxonomy built from visual clustering, art-historical concepts, and manual validation, then gives caption-generation settings and supplementary visual analyses.

10.1 Taxonomy Construction

  • The taxonomy combines embedding-space structure, art-historical interpretability, and corpus coverage into a practical classification scheme rather than a complete ontology.The process used k = 40 CLIP-embedding clusters, comparison with systems such as IconClass, and manual inspection of 10,000 illustrations to refine coverage and category boundaries; practical non-iconographic categories were also included.Source: §10.1 · pp. 14–15

10.2 Caption Generation Configuration

  • Dataset-construction captions were generated with LLaVA-1.5-7B using a detailed image-description prompt and an 80-token maximum.The prompt requested elements, colors, objects, relationships, and emotions.Source: §10.2 · p. 15

10.3 Additional Results

  • Figure 4 presents the test-set confusion matrix for the best-performing ConvNeXt run, selected by macro-F1 for qualitative error analysis.Source: Figure 4 · p. 16
  • Figure 5 shows hierarchical clustering of the 22 categories from CLIP-embedding correlations, summarizing category proximity in embedding space.Source: Figure 5 · p. 17

Related papers

Loading 2609.33449v1…