Source-linked AI summary
Unsupervised Domain Adaptation for Symbol Spotting in Historical Encrypted Manuscripts
Giuseppe De Gregorio, Alicia Fornés, Lei Kang, Beáta Megyesi
TL;DR
Identifying symbols in unknown encrypted manuscripts is difficult because clean font queries differ from degraded handwritten glyphs, and target-script labels may be unavailable. The paper combines domain-invariant representation learning with retrieval-time style adaptation, and reports a P@1 gain over the strongest zero-shot foundation-model baseline.
Problem
Symbol spotting must locate candidate alphabet symbols in unseen manuscripts despite the domain gap between clean font glyphs and degraded handwriting, often without target-script labels.
Source: §1 · p. 2Method
A three-stage pipeline extracts and renders symbols, trains a joint SimCLR+DANN encoder for domain-invariant embeddings, and adapts font queries at retrieval time without re-training.
Source: §1 · pp. 2–3, §3.2 · p. 5Results
+0.138 P@1 over CLIP ViT-L/14, the strongest zero-shot foundation-model baseline.
Source: §5 · p. 15Takeaways & Limitations
Raw-Cover@k, computed fully unsupervised, can serve as a script-family signature for identifying the alphabet underlying an unknown document.
Source: §1 · p. 3The query-driven approach can detect only symbols resembling a Unicode-renderable candidate alphabet, not scripts unrelated to attested writing systems.
Source: §5 · p. 16
Abstract
from arXiv · showhide
The decipherment of historical encrypted manuscripts poses a fundamental challenge in Digital Humanities: before any transcription can begin, the symbol inventory of the underlying cipher alphabet must first be identified and characterized. We address this challenge through symbol spotting: given a candidate alphabet specified as a set of rendered font glyphs, the task is to determine whether and where its characters appear in an unseen handwritten document, without any labeled examples from the target script. The main difficulty lies in the domain gap between clean, digitally rendered font queries and degraded handwritten manuscript symbols. We propose a three-stage pipeline that bridges this gap without manual annotation, combining a joint SimCLR+DANN encoder for domain-invariant glyph representations with an embedding-space style-adaptation mechanism applied at retrieval time, requiring no re-training. Experiments on fourteen pages from seven encrypted manuscript collections show that our method outperforms zero-shot foundation models, including CLIP and DINOv2, by a large margin ($+0.194$ P@1 over CLIP ViT-L/14), and surpasses task-specific trained baselines by $+0.138$ P@1. We further demonstrate that the Raw-Cover metric, computed in a fully unsupervised setting, provides a meaningful script-family fingerprint that identifies the underlying alphabet of an unknown document. This capability is of direct practical relevance to palaeographers, historians, and other researchers working with undeciphered manuscripts.
1 Introduction
Symbol spotting in unknown or encrypted manuscripts must bridge clean font-rendered queries and degraded handwritten glyphs without labeled target examples. The paper proposes a three-stage pipeline and uses Raw-Cover@k as a script-family signature.
1 Introduction
- Unknown and rare scripts often lack labeled examples and prior knowledge of their symbol systems, limiting the applicability of standard recognition approaches.The DECODE database archives hundreds of unresolved manuscripts spanning cipher alphabets, personal notations, and invented writing systems.Source: §1 · pp. 1–2
- The central challenge is matching clean, digitally rendered candidate glyphs to degraded handwritten manuscript symbols when the target script may be unknown.The approach queries candidate alphabets without manual annotation, but the font-to-manuscript domain gap makes retrieval cross-domain.Source: §1 · p. 2
- A three-stage pipeline combines symbol extraction and rendering, joint SimCLR+DANN representation learning, and retrieval-time style adaptation without re-training.The encoder is trained to produce domain-invariant embeddings, while style adaptation shifts font queries toward the target document’s local style manifold.Source: §1 · pp. 2–3
- Embedding-space style adaptation operates on the fly and consistently improves retrieval across all tested configurations.It closes the residual domain gap at inference time without re-training.Source: §1 · p. 3
- The paper evaluates on fourteen pages from seven encrypted manuscript collections, comparing zero-shot baselines with task-specific trained models and conducting an ablation study.The evaluation isolates the contribution of each pipeline component.Source: §1 · p. 3
- Raw-Cover@k, computed in a fully unsupervised setting, can serve as a script signature for identifying the alphabet family of an unknown document.Source: §1 · p. 3
2 Related Work
Prior spotting work ranges from supervised retrieval to weakly supervised and self-supervised methods, while zero-shot foundation models offer another route. The paper positions its approach against these methods’ annotation requirements and limitations on isolated glyph crops.
Supervised Symbol and Word Spotting
- Supervised spotting methods can achieve strong results but require labeled examples from the target collection, an assumption that encrypted manuscripts with unknown alphabets cannot satisfy.The literature includes handcrafted descriptors, deep models with PHOC embeddings, Siamese networks, and a few-shot cipher pipeline.Source: §2 · p. 3
Weakly Supervised and Self-Supervised Approaches
- Weakly supervised and self-supervised approaches relax annotation needs through pseudo-labeling or pretraining, but some methods still use minimal annotations for fine-tuning.SimCLR learns representations from unlabeled augmentation pairs, while DANN penalizes features that reveal domain origin.Source: §2 · pp. 3–4
Zero-Shot Foundation Models
- CLIP and DINOv2 underperform task-specific approaches on isolated glyph retrieval, with image-text alignment and patch-based features poorly suited to small crops.The paper frames its domain-specific training strategy as complementary to foundation models.Source: §2 · p. 4
3 Method
The method combines manuscript-symbol extraction and font-glyph rendering with domain-adaptive representation learning and retrieval-time style adaptation. It aims to make font queries useful for spotting symbols in an unlabeled handwritten document.
3.1 Symbol Extraction and Rendering
- Manuscript glyphs are isolated from binarized pages as connected components, while candidate alphabet characters are rendered from Unicode code points using handwriting-approximating fonts.The manuscript and rendered glyphs receive the same white-background, black-ink normalization, although a texture and stroke-regularity gap remains.Source: §3.1 · pp. 4–5
3.2 Domain-Adaptive Representation Learning
- Font and manuscript symbols are jointly embedded using supervised contrastive learning for character discrimination and domain-adversarial learning for font-to-manuscript invariance.The shared representation is intended to preserve character identity while reducing domain shift.Source: §3.2 · p. 5
- Font anchors use all same-class font samples in the batch as contrastive positives, while manuscript anchors use their paired augmented view.All samples, regardless of domain, contribute to the contrastive denominator as negatives.Source: §3.2 · p. 6
- A class-balanced sampler includes unlabeled manuscript symbols as hard negatives, and adversarial training begins only after an initial stabilization period.Training uses early stopping based on classification loss and domain-discriminator entropy.Source: §3.2 · p. 7
3.3 Retrieval with Embedding-Space Style Adaptation
- At retrieval time, each font query is shifted toward the local manuscript style using its nearest-neighbor centroid, without retraining.The centroid summarizes the average stroke style in the neighborhood; the adapted query is then normalized before retrieval.Source: §3.3 · pp. 7–8
- Final retrieval ranks all gallery embeddings by cosine similarity to Q; matches above the empirically determined threshold τret are accepted, with at most kret = 25 results retained per query.Experiments use k = 50 neighbours for style estimation and α = 0.7, which consistently outperforms α = 0 and α = 1 across all tested documents.Source: §3.3 · p. 8
4 Results
Across fourteen pages from seven encrypted-manuscript collections, the experiments evaluate retrieval accuracy and whether unsupervised Raw-Cover can indicate script families. The full method leads the baselines, while threshold and script-identification analyses clarify its behavior and scope.
4.1 Datasets and Experimental Setup
- The evaluation covers fourteen detached-script pages from seven DECODE collections, using alphabets filtered to symbols present in each document.A manuscript-cropped Copiale alphabet serves as a same-domain query baseline; Cuneiform is a negative control.Source: §4.1 · pp. 8–9
4.2 Threshold Sensitivity and Optimization.
- τ = 0.70 balances Precision@5 and Cover@5; above that threshold, style adaptation maintains higher scores than the unadapted method.The sensitivity analysis varies τ from 0.50 to 1.00 and reports results aggregated across collections.Source: §4.2 · pp. 10–11, Figure 3 · p. 11
4.3 Comparative Analysis and Ablation Study
- P@1=0.595 and MRR=0.661; the full method exceeds CLIP ViT-L/14 by +0.194 P@1 and +0.201 MRR.Table 1 compares the full method with zero-shot and task-specific baselines at τ = 0.70.Source: Table 1 · p. 12, §4.3 · p. 12
- SimCLR+DANN reaches P@1=0.595, above SimCLR alone (0.484), DANN alone (0.525), and the ResNet50-trained baseline (0.457).Style adaptation improves every evaluated configuration, with gains from +0.009 P@1 on the full model to +0.055 on SimCLR alone.Source: Table 2 · p. 12, §4.3 · pp. 12–13
4.4 Script Identification via Raw Cover
- Raw-Cover@5 without labels aligns with script-family patterns: Runicus scores 0.175 for Runic, while Copiale scores 0.686 for Greek and 0.550 for Copiale.Cuneiform, a negative control, yields near-zero Raw-Cover; ASV has a small residual of 0.050.Source: §4.4 · pp. 13–14, Figure 4 · p. 14
- Missing frequent symbols on test pages and visually ambiguous pairs limit Raw-Cover, with some retrievals crossing alphabet classes despite high similarity.Figure 5 illustrates such cross-class matches; the text describes the ambiguity as intrinsic to the manuscripts.Source: §4.4 · p. 14, Figure 5 · p. 15
5 Conclusion
The paper presents a zero-shot symbol-spotting pipeline for encrypted manuscripts and reports gains over zero-shot foundation models. Raw-Cover@k offers an unsupervised script-family fingerprint, while the evaluation and query-driven design have stated scope limits.
5 Conclusion
- +0.138 P@1 over the strongest zero-shot baseline, CLIP ViT-L/14, is reported for the proposed method.The paper also reports that large-scale pretraining alone cannot substitute for task-specific domain adaptation in this setting.Source: §5 · p. 15
- SimCLR+DANN reaches P@1=0.595, and the ablation confirms all three components contribute independently and complementarily.Source: §5 · p. 15
- The zero-shot pipeline requires no labelled target-document examples and combines domain-invariant encoding with retrieval-time style adaptation without re-training.Source: §5 · p. 15
- Raw-Cover@k, computed without labels, provides a script-family fingerprint consistent with expert paleographic knowledge and is proposed as a practical tool for initial collection exploration.Source: §5 · p. 15
- Experiments target detached-script collections that simplify segmentation, while query-driven retrieval only detects symbols resembling Unicode-renderable alphabets and offers no traction on unrelated scripts.The paper identifies open-set or clustering-based formulation as a requirement for unrelated scripts.Source: §5 · p. 16
- The longer-term goal is identifying the underlying script family from candidate alphabets and an unseen manuscript as a practical first step toward decipherment.Source: §5 · p. 16