PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Hierarchy-Aware Deep Learning for the Recognition of Tironian Notes

Yule Kang, Thomas Gorges, Janne van der Loop, Franziska Marske, Nikolaus Weichselbaumer, Tino Licht, Vincent Christlein

arXiv:2609.35277v1cs.CV

TL;DR

Tironian note recognition is difficult because the symbol inventory is fine-grained and visually similar, while flat classifiers do not explicitly encode relationships among related signs. The paper compares flat classification with coarse-to-fine and hierarchy-aware models across visual domains and adaptation settings; hierarchical models perform competitively without adaptation, while flat models perform strongest after few-shot adaptation. The authors conclude that hierarchy can support recognition, particularly for error analysis and non-adapted settings.

  • Problem

    Tironian notes have fine-grained, visually similar symbols, and flat classifiers do not explicitly encode relationships between related signs.

    Source: §1 · p. 2

  • Method

    The study compares flat classification, HD-CNN-style coarse-to-fine classification, and hierarchy-aware routing under comparable training and few-shot adaptation conditions.

    Source: §6 · p. 16

  • Results

    Without few-shot adaptation, HD-CNN achieves the best Top-1 accuracy; after adaptation, flat models achieve the strongest manuscript-domain performance.

    Source: §6 · p. 17

  • Takeaways & Limitations

    Hierarchical structure can support recognition, especially for error analysis and non-adapted recognition, though final accuracy depends on tree construction, routing depth, backbone, and supervision.

    Source: §6 · p. 17

    The study is limited by few annotated manuscript-domain samples, a single manuscript source, and non-flat models’ dependence on visual clustering-tree quality.

    Source: §6 · p. 17

Abstract

from arXiv · show

Tironian notes are generally regarded as the first Latin shorthand system and are notable for their large, fine-grained symbol inventory. Their high visual similarity and large class set make manual reading time-consuming, leaving manuscripts that contain Tironian notes inaccessible to many researchers. Automatic recognition is also challenging because models must distinguish subtle differences in stroke shape and sign structure while realistic training data remain scarce. However, standard flat classifiers do not explicitly use visual or structural relations between related signs. This paper investigates whether structural relationships between Tironian notes can support automatic recognition. We use the Supertextus Notarum Tironianarum (SNT) by Martin Hellmann, which provides idealized sign forms and a hierarchical organization of Tironian notes. We compare flat ResNet18, ConvNeXt, Shifted Window Transformer (Swin), and Vision Transformer (ViT) classifiers with Hierarchical Deep Convolutional Neural Network (HD-CNN)-style coarse-to-fine models and hierarchy-aware routing models based on visual class cleaning and similarity-based re-clustering. The models are evaluated on handwritten samples and manuscript-domain samples from Vergilius Turonensis, both with and without limited few-shot adaptation to the manuscript domain. The results show that the relative performance of flat and hierarchical models depends on adaptation. On Vergilius Turonensis, HD-CNN achieves the best non-adapted Top-1 result with 45.43%, while flat classification reaches the best Top-1 result after few-shot adaptation with 82.09%. Overall, the results indicate that hierarchical structure can support Tironian note recognition, especially under non-adapted conditions.

1 Introduction

Tironian notes use compact symbols for words, word parts, or phrases, and their fine-grained, visually similar forms make recognition difficult. This study compares flat classification with two strategies that use visually generated hierarchies, across handwritten and manuscript data and with or without few-shot adaptation.

1 Introduction

  • Tironian symbols typically represent complete words, word parts, or phrases rather than single letters, and interpreting them requires specialized expertise.Consequently, manuscripts containing them remain inaccessible to many researchers.Source: §1 · p. 2
  • The large, fine-grained inventory and subtle visual differences make recognition challenging, compounded by the gap between clean, handwritten, and historical manuscript images and scarce annotated manuscript data.Manuscript crops may also have textured backgrounds, degraded strokes, low contrast, and incomplete details.Source: §1 · p. 2
  • Flat classifiers map each image directly to a label but do not explicitly encode relationships between visually or structurally related symbols.Coarse-to-fine and hierarchy-aware routing instead organize labels into broader groups before fine-grained prediction.Source: §1 · p. 2
  • HD-CNN uses the generated tree’s leaf labels as fine classes and immediate parent nodes as coarse groups, while hierarchy-aware routing makes sequential decisions through deeper tree levels.Both strategies use visually generated clustering trees, but use their structure differently.Source: §1 · p. 3
  • The study compares flat, HD-CNN, and hierarchy-aware routing recognition on clean standard symbols, augmented data, modern handwritten notes, and expert-cropped Vergilius Turonensis symbols, including limited few-shot adaptation.The evaluation focuses on recognition and Top-k performance while accounting for possible instability from additional routing decisions.Source: §1 · p. 3

2 Related Work

Prior work supplies visual-recognition architectures, augmentation strategies for limited handwritten data, and neural approaches that incorporate label hierarchies. This study builds on work in visually grounded tree construction and hierarchical inference to compare flat classifiers with two hierarchical strategies.

2 Related Work

  • CNNs learn local visual patterns, while ResNet uses residual connections, ConvNeXt updates convolutional design, and ViT and Swin use self-attention; Swin adds shifted windows and hierarchical features.Local patterns such as strokes and curves can distinguish visually similar symbols.Source: §2 · p. 3
  • Data augmentation can increase training variability in low-resource settings, but handwritten-symbol transformations must preserve stroke structure because small changes can alter meaning.Chen et al. propose script-level augmentation that preserves writing structure.Source: §2 · pp. 3–4
  • Hierarchical classifiers replace a single all-label prediction with broader and finer decisions; HD-CNN uses coarse categories and specialist fine-level classifiers, while tree-CNN and B-CNN integrate label structure differently.These approaches represent distinct ways to incorporate label structure into neural recognition.Source: §2 · p. 4
  • Hierarchical classification depends on both model architecture and label-tree construction, with prior work spanning label-tree learning, visually grounded clustering, and methods for combining local tree decisions.This study compares flat classifiers, two-level HD-CNN-style models, and routing models based on visually generated trees.Source: §2 · p. 4

3 Data

The experiments use clean SNT forms, modern handwritten reproductions, and historical manuscript crops, with augmented SNT variants added for training. The sources differ in coverage and image condition, and the manuscript crops provide the manuscript-domain evaluation and adaptation data.

3 Data

  • The evaluation spans clean SNT forms, modern handwritten samples, and cropped historical notes from Vergilius Turonensis, with augmented SNT variants used as additional training data.Figure 1 presents visually corresponding examples from the three domains.Source: §3 · p. 4, Figure 1 · p. 5
  • SNT provides clean standard symbols and metadata with broad class coverage; augmented variants increase training data for classes with few or even one standard sample.The generated variants are for training and validation, not an independent evaluation domain.Source: §3 · pp. 4–5
  • The internal modern handwritten dataset varies stroke shape, thickness, and arrangement on clean backgrounds, but covers only a subset of SNT labels, mainly frequent symbols.It is used for training and controlled-domain evaluation.Source: §3 · p. 5
  • Vergilius Turonensis crops contain textured backgrounds, uneven contrast, ink variation, degraded strokes, and incomplete details; 24 crops without SNT labels were excluded from evaluation.The crops are used for manuscript-domain evaluation and few-shot adaptation; Figure 2 shows representative samples.Source: §3 · p. 6, Figure 2 · p. 6

4 Evaluation Pipeline and Recognition Strategies

The evaluation pipeline cleans and augments the SNT data, trains flat classifiers, and uses their features to build visual clustering trees. HD-CNN and hierarchy-aware routing use those trees differently, with routing making decisions across more levels.

4 Evaluation Pipeline and Recognition Strategies

  • The pipeline trains flat classifiers as baselines and feature extractors, then uses their features to build trees for two hierarchical recognition strategies.K-nearest-neighbor is also evaluated as a non-parametric baseline.Source: §4 · p. 6

4.1 Preprocessing and Data Augmentation

  • SNT labels that are visually near-identical after normalization are conservatively deduplicated, and each standard symbol receives 16 augmented training variants.Augmented samples address limited per-class training data and are not a separate evaluation domain.Source: §4.1 · p. 7

4.2 Flat Classification Baselines

  • Flat baselines use four ImageNet-pretrained backbones and predict directly over all cleaned leaf labels.The backbones are ResNet18, ConvNeXt-Tiny, Swin-Tiny, and ViT-Small.Source: §4.2 · pp. 7–8

4.3 Visual Clustering for Hierarchical Grouping

  • The clustering tree is built from normalized features of representative SNT symbols using pairwise cosine distances and average-linkage agglomerative clustering.Tree construction uses a distance threshold for ResNet18 features and dynamic branching for the other backbones.Source: §4.3 · pp. 8–9

4.4 hd-cnn Coarse-to-Fine Classification

  • HD-CNN uses leaf labels as fine classes and their immediate parents as coarse groups, combining group and specialist probabilities into global leaf scores.This two-level structure uses hierarchical information while limiting sequential decisions compared with deeper routing.Source: §4.4 · pp. 9–10, Figure 5 · p. 9

4.5 Hierarchy-Aware Routing Classification

  • Hierarchy-aware routing assigns classifiers to internal tree nodes and predicts through local decisions; path-normalized loss reduces depth-related imbalance, while beam search retains candidate paths.Routing is also evaluated with group-level scores to assess whether incorrect leaf predictions remain within related visual groups.Source: §4.5 · pp. 10–11, Figure 6 · p. 10

5 Results and Discussion

On Vergilius Turonensis, the strongest approach depended on few-shot adaptation: HD-CNN led without adaptation, while flat classifiers performed better after adaptation. Results also show domain-specific gaps, differences between routing strategies, and errors associated with ambiguous manuscript crops.

5.1 Flat Classification

  • Without adaptation, flat Top-1 was 98.99–99.36% handwritten versus 36.28–43.82% on Vergilius; adapted Swin reached 51.13% balanced accuracy alongside 82.09% Top-1.The lower balanced accuracy indicates that rare classes remained more challenging.Source: §5.1 · pp. 11–12
  • Few-shot fine-tuning raised manuscript Top-1 for all flat models: ResNet18 36.28% to 68.73%, ConvNeXt 40.37% to 79.14%, Swin 43.82% to 82.09%, and ViT 42.62% to 81.77%.Handwritten results remained close to maximum, with the main effect of adaptation visible on Vergilius Turonensis.Source: §5.1 · p. 12
  • 90.12% handwritten versus 4.20% Vergilius: pixel-level KNN; feature-based KNN reached 38.83% on Vergilius.Feature-based KNN improved on the manuscript score but remained limited.Source: §5.1 · p. 12

5.2 Hierarchy-Aware Routing Classification

  • With self-generated trees after adaptation, ConvNeXt achieved the highest Vergilius Top-1 among hierarchy-aware models.The handwritten results remained high for all models.Source: Figure 7 · p. 13, §5.2 · p. 13
  • Few-shot fine-tuning improved hierarchy-aware results across all four backbones, but did not consistently beat flat classification at the leaf level; group scores were usually higher.The authors identify early branch errors as one possible source of error propagation in deeper routing.Source: §5.2 · p. 13

5.3 hd-cnn Coarse-to-Fine Classification

  • 45.43% Top-1: non-adapted HD-CNN with Swin led on Vergilius; after few-shot fine-tuning, flat models outperformed HD-CNN in the manuscript domain.The HD-CNN advantage without adaptation held for ResNet18, Swin, and ViT, with ConvNeXt as the exception.Source: Table 5 · p. 14, §5.3 · p. 14
  • HD-CNN’s two-level design was more stable than deeper routing in the reported manuscript results; before adaptation, it led Swin at Top-1 and Top-5, while flat Swin led at Top-10 and Top-30.The authors associate the shallower design with fewer sequential decisions and a more limited effect from early routing errors.Source: §5.3 · pp. 14, 16, Figure 9 · p. 16

5.4 Error Analysis

  • In the first two challenging examples, hierarchy-aware models ranked the true label higher than flat models; in the last, both missed it from their Top-30 candidates.The examples illustrate ambiguity between manuscript crops and standard symbols.Source: Table 6 · p. 15, §5.4 · p. 16

6 Conclusion and Outlook

Across the evaluated settings, the strongest strategy depends on few-shot adaptation: HD-CNN leads without it, while flat models lead after adaptation. The study concludes that hierarchy can support recognition, particularly without adaptation, while noting limits in data and scope.

6 Conclusion and Outlook

  • The comparison covers flat, HD-CNN-style coarse-to-fine, and hierarchy-aware routing strategies under comparable training and few-shot adaptation conditions.Source: §6 · p. 16
  • Recognition is easier on the cleaner handwritten test set than on Vergilius Turonensis, where background texture and degraded strokes make recognition more difficult.Source: §6 · pp. 16–17
  • Without few-shot adaptation, HD-CNN achieves the best Top-1 accuracy, while flat models achieve the strongest manuscript-domain performance after adaptation.The study also reports that limited target-domain supervision is highly effective.Source: §6 · p. 17
  • Hierarchy-aware routing does not consistently improve final leaf-level recognition, and early incorrect branch decisions may introduce additional instability.Source: §6 · p. 17
  • Group-level results remain useful for analyzing whether errors stay among visually or structurally related symbols.Source: §6 · p. 17
  • The study is limited by scarce annotated manuscript samples, one manuscript source, and non-flat models' dependence on visual clustering-tree quality.Source: §6 · p. 17
  • Overall, hierarchical structure can support recognition, but final accuracy depends on tree construction, routing depth, backbone architecture, and manuscript-domain supervision.Source: §6 · p. 17

S1 Supplementary Results

The supplementary results compare the SNT reference hierarchy with generated visual clustering trees and report additional balanced-accuracy and retrieval-baseline evaluations.

S1 Supplementary Results

  • Generated visual clustering trees are more compact than the SNT reference tree, which expands strongly at Levels 3 and 4.The SNT tree follows the source glossary, whereas generated trees group symbols by visual feature similarity for recognition-oriented routing.Source: S1 Supplementary Results · p. 21, Table S1 · p. 21

S1.1 Balanced Accuracy

S1.2 Additional KNN Retrieval Baselines

  • The additional KNN baselines use binarized pixels or trained-backbone features, cosine-similarity ranking, and unique class labels; they are excluded from few-shot fine-tuning.The reference gallery combines cleaned SNT symbols with handwritten training and validation samples.Source: S1.2 Additional KNN Retrieval Baselines · p. 25, Table S12 · p. 25

Related papers

Loading 2609.35277v1…