PaperlayerView exact revision on arXiv

Source-linked AI summary

From Global Alignment to Local Grounding: Zero-Shot Chinese Character Recognition with Radical Verification

Yu-Heng Shih, Bing-Chen Wu, Tsz-To Wong, Ting-En Yen, Hong-Han Shuai, Bin-Hua Hsieh, Chien-An Chen, Yi-Ren Yeh, Ching-Chun Huang

arXiv:2610.09449v1cs.CV

TL;DR

CLIP-style ZS-CCR methods rely on global image–IDS similarity that omits explicit radical geometry and can misrank characters differing by a subtle local radical. STG-CLIP enriches IDS prototypes with tree-position and canonical geometric priors, while a margin gate invokes RVM to verify candidate radicals locally. On the character setting, STG-CLIP+RVM reaches 83.06% accuracy at 2755 seen classes.

  • Problem

    CLIP-style ZS-CCR methods rely on global image–IDS similarity that omits explicit radical geometry and can misrank characters differing by a subtle local radical.

    Source: §1 · p. 2

  • Method

    STG-CLIP enriches IDS prototypes with tree-position and canonical geometric priors, while a margin gate invokes RVM to verify candidate radicals locally.

    Source: §1 · p. 2, §3.5 · p. 11

  • Results

    On the character setting, STG-CLIP+RVM reaches 83.06% accuracy at 2755 seen classes.

    Source: §4.3 · p. 12

  • Takeaways & Limitations

    The framework achieves state-of-the-art accuracy in character zero-shot scenarios, with ablations reporting additional gains from geometry and RVM.

    Source: §4.4 · p. 13, §5 · p. 14

    At 500 and 1,000 seen classes, STG-CLIP trails CCR-CLIP; the authors attribute this to its ViT encoder needing more data to train well.

    Source: §4.3 · p. 13

Abstract

from arXiv · show

Zero-shot Chinese character recognition (ZS-CCR) aims to recognize characters whose categories are never observed during training, and typically relies on the compositional structure shared between seen and unseen characters. Recent CLIP-style methods represent this structure with the Ideographic Description Sequence (IDS) and align it with glyph images in a shared embedding space. However, they rely on a single global image--IDS similarity that discards the spatial layout of radicals and, being learned only implicitly from seen classes, generalizes poorly to unseen ones; moreover, global matching often retrieves the correct character within the top candidates yet fails to rank it first when characters differ only in subtle local radicals. To address these issues, we propose a global-to-local two-stage framework. In the first stage, STG-CLIP augments the IDS with explicit tree-position and radical-level geometric priors, yielding a spatial-aware prototype that provides a consistent spatial description across seen and unseen categories for high-recall global retrieval. In the second stage, the Radical Verification Module (RVM) uses the radical instances of each retrieved candidate as queries to verify whether the corresponding radicals can be matched to spatially compatible regions in the input glyph. A margin-based gating rule activates the RVM only when the leading global candidates receive similar similarity scores. Experiments on the ICDAR2013 benchmark demonstrate that our method achieves state-of-the-art performance under the character-level zero-shot setting, obtaining 83.06% top-1 accuracy with 2,755 seen classes. Ablation studies further show that the explicit geometric priors and radical-level verification provide complementary improvements.

1 Introduction

ZS-CCR addresses the practical challenge of recognizing rare Chinese characters without handwritten training samples by transferring compositional knowledge from seen to unseen classes. The paper proposes spatially informed IDS prototypes for global retrieval and candidate-conditioned radical verification for resolving local ambiguities.

1 Introduction

  • ZS-CCR targets rare and unseen Chinese characters because collecting handwritten samples for every category is practically impossible.Its class descriptions decompose characters into radical and structural-operator tokens shared across seen and unseen classes.Source: §1 · p. 2
  • Flattened IDS encodings discard tree relations and lack explicit radical geometry, so learned spatial priors extrapolate poorly to unseen layouts.Global similarity may retrieve the right character within the top-k yet fail to rank it first when a local radical differs.Source: §1 · p. 2
  • STG-CLIP injects tree-position and canonical geometric priors into IDS tokens to provide consistent spatial descriptions across seen and unseen categories.The geometric attributes include normalized centroid, bounding box, and area ratio.Source: §1 · pp. 2–3
  • The framework separates global candidate retrieval from local radical verification, invoking verification only for ambiguous decisions.RVM uses candidate radicals to assess their presence and glyph coverage, supplying localized evidence for subtle ambiguities.Source: §1 · p. 3

2 Related Work

Related work approaches ZS-CCR through contrastive alignment, sequence generation, radical localization, and radical representation learning. The paper distinguishes its approach through explicit geometry and candidate-conditioned verification.

2.1 Contrastive Learning-based Methods for ZS-CCR

  • CLIP-style methods align glyphs with IDS, with CCR-CLIP, FT-CLIP’s Formation Tree, and local/global alignment by Cai and Zhu.These methods model symbolic or relative structure but lack explicit absolute geometric awareness of radical instances.Source: §2.1 · p. 3

2.2 Generative Sequence Recognition Methods

  • DenseRAN and SD/SLD generate radical or stroke sequences autoregressively, but early errors can propagate and valid-label parsing requires post-processing.The proposed method instead retrieves classes through image–prototype alignment without sequence generation.Source: §2.2 · p. 4

2.3 Radical Detection and Localization Methods

  • JRED and LERRNet localize radicals, whereas RVM verifies candidate hypotheses using localized visual evidence to resolve global prediction ambiguities.The cited detection and localization models predict component sequences directly rather than verifying candidate hypotheses.Source: §2.3 · p. 4

2.4 Radical Representation Learning

  • FaRE improves radical embeddings with rendered-font visual features, but IDS methods still overlook character-dependent radical location, scale, and aspect ratio.The paper models radical instances by attaching canonical geometric attributes to IDS tokens.Source: §2.4 · p. 4

3 Proposed Method

The method treats IDS and canonical geometry as class-side information for recognizing handwritten characters from unseen categories. It combines spatially enriched global retrieval with local verification of candidate radicals.

3.1 Problem Formulation and Framework Overview

  • ZS-CCR trains on handwritten images from seen categories and predicts unseen-category labels; unseen printed glyphs and radical masks are excluded from optimization.IDS descriptions and canonical geometric descriptors remain available as class-side information for all categories.Source: §3.1 · pp. 4–5
  • STG-CLIP retrieves a high-recall top-K shortlist using tree- and geometry-enriched IDS prototypes, while RVM checks candidate radicals against localized visual features.A margin gate either invokes local re-ranking or trusts the global top-1 prediction.Source: §3.1 · p. 5, Figure 1 · p. 5

IDS Tree-Position Encoding

  • Tree-position encoding assigns each IDS token a depth index and sibling index to preserve hierarchical cues lost by flattening.Depth measures distance from the root; sibling index records left-to-right order among nodes with the same parent.Source: IDS Tree-Position Encoding · p. 6

Radical Geometry Augmentation

  • Radical geometry augmentation derives token-level spatial descriptors from canonical printed glyph masks, including centroid, bounding-box dimensions, and area ratio.Structural operators use the enclosing box and union of masks for their subtree; radical tokens use grouped stroke paths rendered as a component mask.Source: Radical Geometry Augmentation · p. 6

IDS Prototype Encoding

The IDS prototype serves both as STG-CLIP’s global retrieval target and as the structural basis for RVM’s local radical hypotheses. STG-CLIP aligns image and prototype embeddings, with style regularization drawing handwritten and canonical printed glyph embeddings closer.

IDS Prototype Encoding

  • The text encoder maps the spatial-aware token sequence Sgeo(y) to class prototype ty, which also supplies radical hypotheses for local verification.The prototype is the global retrieval target; its underlying symbolic decomposition is passed to RVM.Source: IDS Prototype Encoding · p. 7

3.3 Global Image–Prototype Alignment (STG-CLIP)

  • STG-CLIP learns a shared embedding space between handwritten images and spatially enriched IDS prototypes for coarse retrieval.Image embeddings are computed for handwritten and canonical printed glyphs during training.Source: §3.3 · p. 7, Image and Prototype Representation · p. 7

Image and Prototype Representation

  • The canonical printed-glyph branch is an auxiliary training regularizer and is omitted at inference; embeddings are L2-normalized before similarity calculation.The image encoder produces separate handwritten and printed-glyph embeddings.Source: Image and Prototype Representation · p. 7

Contrastive Alignment with Style Regularization

  • A symmetric InfoNCE objective aligns image–prototype pairs, while style regularization encourages handwritten embeddings to match corresponding canonical printed-glyph embeddings.The global objective adds the style loss to the image–text contrastive loss, weighted by λstyle.Source: Contrastive Alignment with Style… · p. 7

Top-K Candidate Retrieval

At inference, a handwritten query is compared with class prototypes across the recognition vocabulary. The highest-scoring classes form the shortlist for later processing.

Top-K Candidate Retrieval

  • The search covers all character categories in the recognition vocabulary.The query's visual feature vector is extracted before scoring.Source: Top-K Candidate Retrieval · p. 8
  • The global score compares the query image feature vector with each class prototype using a similarity function.The comparison is computed for every class in the recognition vocabulary.Source: Top-K Candidate Retrieval · p. 8
  • The shortlist contains the K classes with the highest global similarity scores.These classes are selected from the scores computed over the full vocabulary.Source: Top-K Candidate Retrieval · p. 8

3.4 Radical Verification Module

The Radical Verification Module (RVM) checks whether candidate radicals are grounded in spatially compatible regions of the input glyph. It uses localized visual features to preserve character layout for this verification.

3.4 Radical Verification Module

  • RVM verifies whether each retrieved candidate’s radicals are visually grounded in the input image through localized structural alignment.It targets ambiguities between characters with similar overall layouts but a differing radical.Source: §3.4 · p. 8

Localized Visual Features

  • A separate ResNet-50 backbone extracts local visual features for RVM and is trained independently of STG-CLIP.The feature maps are flattened and projected into patch tokens.Source: Localized Visual Features · p. 8
  • Two-dimensional sinusoidal position embeddings are added to patch tokens to preserve the glyph’s spatial layout.The tokens are derived from projected visual feature maps.Source: Localized Visual Features · p. 8

Decoder-based Structural Alignment

For each candidate, the decoder uses its IDS-derived radical occurrences as queries over localized visual features. The resulting representations support parallel radical-existence and spatial-mask predictions.

Decoder-based Structural Alignment

  • Each candidate’s IDS decomposition provides an ordered sequence of radical occurrences, initialized as learnable decoder-query embeddings.The number of occurrences is candidate-specific.Source: Decoder-based Structural Alignment · p. 8
  • Unlike STG-CLIP text representations, the radical queries omit canonical geometric priors because handwritten and printed radicals are not pixel-aligned.Source: Decoder-based Structural Alignment · p. 8
  • A cascade of transformer decoder layers refines radical queries through cross-attention to localized visual features, followed by feed-forward layers.Source: Decoder-based Structural Alignment · p. 8
  • The terminal radical representations feed parallel heads for component-existence probabilities and soft spatial attention masks.The masks are reshaped and upsampled to input resolution to delineate predicted radical locations.Source: Decoder-based Structural Alignment · pp. 8–9

Multi-Task Optimization

RVM training supervises radical presence and, for positive queries, spatial localization. A combined objective weights the match, focal, and Dice losses.

Multi-Task Optimization

  • Positive training queries come from radicals present in a character, while negative queries are sampled from absent categories.Source: Multi-Task Optimization · p. 9
  • Binary cross-entropy supervises radical existence using the ground-truth presence label for each query.Source: Multi-Task Optimization · p. 9
  • For spatial localization, positive queries use focal loss to suppress easy background pixels and Dice loss to optimize region overlap.The focal-loss probability is selected according to the ground-truth mask pixel label.Source: Multi-Task Optimization · p. 9
  • The total multi-task objective combines match, focal, and Dice losses for positive queries using balancing weights.Negative queries bypass the spatial losses.Source: Multi-Task Optimization · p. 9

3.5 Training and Inference Pipeline

The pipeline trains the RVM first on printed glyphs with radical masks, then adapts it to handwriting without mask labels. At inference, it locally verifies retrieved candidates only when the leading global scores are close.

Training

  • Printed-glyph pretraining uses radical-mask supervision; handwritten fine-tuning freezes the mask head and trains the remaining module with character-level occurrence loss.The printed stage activates matching, focal, and Dice losses, while handwritten samples require no radical-level pixel annotations.Source: §3.5 · pp. 9–10

Inference

  • For each shortlisted candidate, the RVM queries its radicals and combines radical-presence and glyph-coverage scores into a local verification score.Presence averages queried-radical existence probabilities; coverage measures how much of the observed foreground strokes are covered by predicted radical masks.Source: §3.5 · p. 10

Gated inference

  • When the top-two global-score margin is at most δ, the system reranks the shortlist by local score; otherwise, it returns the global top-1 candidate.The threshold δ is tuned on the validation set, and a larger margin bypasses local verification for efficiency.Source: §3.5 · pp. 10–11

4 Experiments

Experiments evaluate character- and radical-level zero-shot recognition, comparing the framework with prior methods and assessing its components and inference settings. The strongest reported character-zero-shot result is 83.06% at 2,755 seen classes.

Dataset.

  • CASIA-HWDB1.0-1.1 supplies 2,678,424 samples across 3,881 classes for training, while ICDAR2013 tests on 224,419 samples from 60 disjoint writers.Printed-glyph radical masks are constructed for RVM pretraining; the mask head is frozen during handwritten fine-tuning, which needs no mask labels.Source: Dataset · p. 11

Implementation Details.

  • The global image encoder uses 32×32 inputs with λstyle=1, while the RVM backbone uses 64 × 64 inputs and λ1=5, λ2=1.The gating threshold is selected per setting on the validation set.Source: Implementation Details · p. 11

4.3 Comparison with Existing Methods

  • 83.06% at 2,755 seen classes makes STG-CLIP+RVM best on character zero-shot, beating FT-CLIP by 4.10 points and CCR-CLIP by 10.08.RVM improves STG-CLIP across seen-class counts, by 4.02 points at 1,500; it yields little gain when test characters contain radicals unseen during training. STG-CLIP trails CCR-CLIP at 500 and 1,000 seen classes.Source: Table 1 · p. 12, §4.3 · pp. 12–13

4.2 Experiments on Unseen Character Recognition

  • Evaluations cover character-zero-shot and radical-zero-shot settings, with all categories in each experiment used as test-time candidates.Zero-shot learning is evaluated as generalization to categories unseen during training.Source: §4.2 · p. 12

Character Zero-shot Settings.

  • Character-zero-shot training uses the first m GB2312-ordered characters, with m in {500, 1000, 1500, 2000, 2755}; ICDAR2013 tests the last 1,000.The training samples come from CASIA-HWDB.Source: Character Zero-shot Settings · p. 12

Radical Zero-shot Settings.

  • In radical-zero-shot evaluation, a character is held out if it contains a radical below the frequency threshold n, tested at n in {50, 40, 30, 20, 10}.This split tests generalization to characters containing rare radical compositions.Source: Radical Zero-shot Settings · p. 12

4.4 Ablation Studies

  • 82.26% (+4.48) follows canonical geometry; tree topology lifts 76.27% to 77.78% (+1.51), and RVM adds 0.80 to reach 83.06%.The cumulative ablation uses the character-zero-shot setting with 2,755 seen classes; component gains depend on addition order and do not fully separate encoding from geometric information.Source: §4.4 · p. 13, Table 2 · p. 13

Selection of K and

  • K=2 and δ=0.03 are selected on validation and fixed at test time; with K=2, accuracy fluctuates by less than 0.4% across evaluated thresholds.Smaller shortlists perform better (K=2 > K=3 > K=4), although larger K improves raw retrieval recall while admitting distractors.Source: Selection of K and · p. 13, Figure 2 · p. 14

5 Conclusion

The paper proposes a coarse-to-fine zero-shot Chinese character recognition framework that combines spatial-aware global retrieval with margin-gated local radical verification. Experiments report state-of-the-art character-zero-shot accuracy, while identifying unseen radicals as a limitation for verification.

Framework

  • The framework combines STG-CLIP, which injects tree topology and geometric priors into IDS, with RVM local candidate reranking triggered by a margin gate.The stated aim is a consistent spatial description across categories and mitigation of the seen-to-unseen spatial gap.Source: §5 · p. 14

Findings

  • Experiments report state-of-the-art accuracy in character-zero-shot scenarios.The conclusion also identifies ViT encoder scaling properties as part of the paper's analysis.Source: §5 · p. 14

Scope

  • Extending RVM to entirely unseen radicals remains a future direction.The conclusion describes radical-zero-shot settings as a limitation.Source: §5 · p. 14
Loading 2610.09449v1…