Source-linked AI summary
A statistical approach to bias in zero-shot learning: the lens of handwriting recognition
Clarence Chew, Gim Siang Chia, Sukalpa Chanda, Subhroshekhar Ghosh, Soumendu Sundar Mukherjee
TL;DR
GZSL systems exhibit seen-class bias and face scalability challenges in zero-shot handwritten word recognition with very large vocabularies. The paper introduces a statistical two-stage debiasing architecture and reports improved unseen-class detection, overall GZSL accuracy, and effective low-dimensional representations.
Problem
Existing GZSL frameworks persistently favor seen classes, while large handwritten-word vocabularies create dense class manifolds and subtle visual differences.
Method
The method treats a classical GZSL learner as a black box and uses lightweight Monte Carlo classifiers with calibrated thresholds to estimate training status and rectify bias.
Results
Unseen-class detection accuracy improves from 0.77 to 0.931, while overall GZSL accuracy h improves from 0.84 to 0.917.
Takeaways & Limitations
The strongest ensemble achieves h = 0.917, while the results show that large-vocabulary word recognition can use approximately 15-dimensional representations.
Takeaways & Limitations
A random-forest variant has intensive computational cost that grows with tree height.
Abstract
from arXiv · showhide
Generalized zero-shot learning (GZSL) has emerged as an important paradigm for visual recognition systems that must generalize to classes that were not observed during training. Traditional GZSL techniques are limited by their applicability to a relatively small number of such unseen classes, scalability beyond which is challenging due to its well-known misclassification bias towards classes observed during training. In this work, we investigate the GZSL paradigm through the lens of zero-shot handwritten word recognition over extremely large vocabularies. We propose a statistical approach to rectifying this bias, which views any classical GZSL feature learner as a black box mechanism whose intrinsic bias in identifying the training status (seen vs. unseen) of a typical data point we aim to correct, similar to an out of distribution inferential problem. Our method leverages a simple two-stage hierarchical architecture, combining a classical GZSL blackbox in the first stage and an ensemble of lightweight Monte Carlo bias-correctors in the second. Once debiased, the classification of test data is undertaken only restricted to its predicted training status via well-founded statistical methods (eg nearest neighbour, logistic regression and random forests). We achieve relative accuracy improvements of over 20% in the classification of unseen words compared to established techniques. A key outcome is that word recognition over large scale vocabularies is amenable to a much lower dimensional representation (~15 dimensions). Our approach is underpinned by mathematical analysis that captures the essence of the statistical approach to bias correction. Our approach to bias rectification can be combined in a turn-key fashion with any classical GZSL learner as a blackbox, thereby suggesting a wide scope of applicability of this method for a wide variety of GZSL implementations in different domains.
1 Introduction
The paper addresses seen-class bias in generalized zero-shot learning for large-vocabulary handwritten word recognition. It proposes statistical bias correction combined with structured embeddings and reports improved recognition of unseen words.
- Problem: Existing GZSL frameworks exhibit persistent bias toward classes whose data were used during training.This bias is especially problematic when recognition must cover extremely large vocabularies containing unseen words.
- Problem: Large-vocabulary handwritten word recognition is harder than conventional object-level zero-shot classification because classes are numerous, visually similar, and highly variable in writing style.Words may differ by a single character or small stroke modification, while arbitrary character combinations create thousands of classes.
- Motivation: CTC-based sequence models are structurally misaligned with zero-shot word recognition because of segmentation difficulties, accumulated character errors, and limited focus on global word structure.The paper instead uses structured word embeddings that encode positional and visual shape information.
- Approach: The proposed method treats a classical GZSL learner as a black box and statistically predicts whether each input belongs to a seen or unseen class.A two-stage architecture combines the original feature learner with Monte Carlo bias-correctors and threshold calibration.
- Approach: Recognition is restricted to the predicted seen or unseen vocabulary before classification with nearest neighbours, logistic regression, or random forests.This design dynamically limits candidate classes according to the predicted training status.
- Results: PCA shows that the 769-dimensional word embedding can be represented using 15 dimensions while retaining information for distinguishing thousands of word classes.The reduced representation supports lightweight classifiers for training-status prediction.
- Results: Relative improvements of over 20% are reported for identifying unseen classes in zero-shot handwritten word recognition.The framework is presented as compatible with classical GZSL feature learners used as black boxes.
2 Problem Formulation and Methodology
The methodology uses embedding models to represent handwritten words, then estimates training status with an ensemble of biased binary classifiers. Classification is performed within the predicted seen or unseen vocabulary, optionally after dimensionality reduction.
- Problem formulation: The framework partitions the vocabulary into disjoint seen and unseen classes and maps handwritten word images to structured embedding vectors.The embedding function supports nearest-neighbour recognition against word-class embeddings.
- Phase I: Embedding models: Multiple first-stage embedding models are trained on partial vocabulary subsets, creating diverse predictions for each input word image.Each model produces a structured embedding and a corresponding predicted training status.
- Phase II: Seen/unseen classifier: Second-stage classifiers estimate whether each embedding corresponds to a class observed during training.Their outputs are aggregated across an ensemble to address the embedding models’ tendency to predict seen classes.
- Phase II: Seen/unseen classifier: The system declares an input seen only when the ensemble’s seen predictions exceed a threshold calibrated above a simple majority.The threshold can be selected through cross-validation.
- Low-dimensional approximation: PCA-based low-dimensional projections make the second-stage classifiers computationally light enough to deploy in large ensembles.The trade-off is approximation from using the projected representation instead of the full embedding.
- Inference: After training-status prediction, nearest-neighbour matching is performed within the corresponding seen or unseen vocabulary subset.Random forests or logistic regression can optionally replace nearest-neighbour matching.
3 The Algorithmic Pipeline
The pipeline trains multiple embedding models, constructs training data for seen/unseen classifiers and word predictors, generates per-model predictions, and aggregates them into a final word prediction.
- Seen/unseen ensemble: The second-stage ensemble is trained from data not used for the embedding model together with synthetic examples treated as unseen.Each classifier uses a random 80% subset so ensemble outputs differ and can be combined.
- Word prediction: Optional random forest or logistic regression predictors can replace nearest-neighbour matching within the relevant vocabulary subset.These predictors are trained using held-out and synthetic data associated with the embedding model.
- Prediction aggregation: The final word prediction is the most frequent label across embedding models, with nearest-neighbour distance resolving ties.Algorithm 3 compares embedding distances among tied candidate labels.
- Embedding construction: The pipeline represents words with 769-dimensional Pho(SC)Net vectors composed of shape, letter, and bigram features.The components contribute 165 shape, 504 letter, and 100 bigram dimensions.
- Training embedding models: Each embedding model is trained after randomly splitting labels and examples into model-training, held-out, and synthetic or unseen-designated subsets.Algorithm 1 uses label and per-class data splits before training the embedding model.
- Per-model prediction: For each embedding model, the inference algorithm counts ensemble classifiers predicting that the feature corresponds to a seen class.If the count reaches the threshold, prediction occurs within the model’s seen vocabulary; otherwise it uses the unseen vocabulary.
4 Results & Discussion
On the IAM dataset, the proposed ensembles substantially improve generalized zero-shot recognition, especially unseen-class detection, while dimensionality reduction preserves useful information and lowers computational demands.
- Dataset and metrics: The IAM task evaluates recognition among 8,407 words, with 7,898 seen and 509 unseen classes.The dataset contains 250 × 50 pixel handwritten-word images.
- Baseline and metrics: Pho(SC)Net shows seen-class bias, with unseen accuracy 0.77 versus seen accuracy 0.93.Performance is assessed using seen and unseen accuracies and their harmonic mean.
- Bias correction: The two-stage method classifies inputs as seen or unseen using an ensemble threshold, then restricts prediction to the corresponding vocabulary with NN, RF, or LR.The threshold is chosen roughly to maximize the harmonic mean of seen and unseen accuracy.
- Dimensionality and ensemble analysis: The top 15 eigenvalues explain 0.6224 of total covariance variance, while the top 30 explain 0.7592.The results indicate substantial low-dimensional structure in the embedding space.
- Performance: 0.931 unseen-class detection accuracy is achieved from 0.77, while overall GZSL harmonic-mean accuracy reaches 0.917 from 0.84.The 21M[1] + 50M[2] ensemble without dimension reduction achieves h = 0.917.
- Computational considerations: Low-dimensional ensemble variants are more computationally lightweight alternatives, although RF cost grows with tree height and LR has numerical-stability limitations.The most expensive 769-dimensional experiment took approximately one week of compute; other experiments took up to three days.
5 Theoretical Underpinnings of Ensembling for Bias Reduction
The paper analyzes ensemble bias correction using binary predictors that classify whether a word belongs to seen or unseen classes. Under arbitrary or random class allocations, thresholded ensembles can achieve high accuracy when separation and ensemble-size conditions hold.
- Model and assumptions: The theoretical model uses L binary predictors trained on restricted class subsets to identify whether a test word is seen or unseen.The predictors may be independent through sample splitting, while exchangeable predictors are also considered.
- Model and assumptions: A predictor is characterized by Type 1 and Type 2 errors, with α_k + β_k ≤ 1 ensuring performance no worse than random.A Gaussian mixture model provides one example of a binary predictor by labeling low-density observations as unseen.
- Ensemble construction: The ensemble E_m labels an example as seen when at least m predictors output seen, and the analysis evaluates arbitrary and random class-allocation mechanisms.The threshold m is selected relative to the expected number of predictors identifying a class as seen.
- Theoretical results: Under arbitrary allocation, E_m classifies seen and unseen classes with high accuracy when Δ is bounded away from zero and μ = Θ(L).Theorem 1 uses m = ⌈μ(1 + δ)⌉ under positive predictor separation.
- Theoretical results: Under random allocation, E_m achieves high accuracy when μ is large, δ is bounded away from zero, and the number of predictors is large.The result assumes independence between predictor error rates and the allocation mechanism.
- Interpretation: The analysis also notes that multiple predictions provide robustness against incorrect predictions for unseen words outside an individual predictor’s training vocabulary.This contrasts with the tendency of a single model to favor words in its own seen-class subset.
6 Concluding Remarks
The paper presents an ensemble-based statistical methodology for correcting seen-class bias in generalized zero-shot handwritten word recognition. It combines a two-stage architecture with theoretical analysis and is proposed as adaptable to classical GZSL learners.
- Contribution: The method uses a two-stage architecture whose second stage explicitly focuses on mitigating bias toward seen classes.The authors describe the methodology as an ensemble-based statistical approach for handwritten word recognition.
- Scope: The framework can in principle augment any classical GZSL learner in a turn-key fashion for bias correction.The paper focuses on handwritten word recognition while identifying other GZSL domains as follow-up applications.
- Future directions: The work provides a foundational theoretical structure for the methodology and identifies broader applications and weaker assumptions as future research directions.These extensions are presented as natural directions rather than established results.
A Supporting Algorithms
The supporting algorithms generate synthetic seen and unseen data, prepare training sets, and classify feature vectors using random forests, logistic regression, or nearest-neighbor fallback.
- Synthetic data generation: The synthetic unseen-data procedure assigns 95% of coordinates extreme samples and 5% middle samples from coordinate-wise Gaussian quantiles.The resulting vectors are combined with Z and Z′ for training the bias-corrector models.
- Seen-class data preparation: Algorithm 5 optionally balances seen-class training data by adding sampled feature differences to underrepresented classes.It uses data not used to train M^(k) together with additional synthetic data.
- Class prediction: Algorithm 6 predicts among seen classes with a random forest when available, logistic regression otherwise, and nearest distance as the final fallback.The input includes a feature vector and the relevant seen-class vocabulary W_s^(k).
B.1 Why Use Em over a Consensus Voting?
The paper contrasts strict consensus voting with a thresholded ensemble rule. Consensus can remove seen-class bias but may be too stringent for correctly identifying seen classes, motivating E_m.
- Consensus voting: The consensus rule E_all declares an example seen only when all binary predictors classify it as seen.This rule is designed to remove bias toward seen classes.
- Consensus voting: Under independence and a bound β_k ≤ β < 1 on Type II errors, consensus voting can suppress seen-class bias for unseen examples.The guarantee depends on the predictors’ error behavior.
- Consensus limitation: Consensus is too stringent for seen classes, so its accuracy in identifying seen classes can be substantially below one.The number of predictors trained with a word in their seen vocabulary affects this trade-off.
- Thresholded ensemble: The analysis therefore motivates the less stringent threshold rule E_m with an appropriate choice of m.E_m relaxes the requirement that every predictor vote seen.
B.2 Proofs of Various Results
These proofs analyze an ensemble classifier that labels a point as seen when at least m component classifiers do so, deriving error bounds under Bernoulli-style assumptions and class-allocation conditions.
- Under allocation assumptions, the proof applies concentration bounds to sums of independent Bernoulli variables and controls the resulting probabilities with a union bound.
- For a seen word, the relevant error event is that the ensemble classifies it as unseen, represented as P(E_m(x) = ‘unseen’ | w ∈ W_s) = P(X < m | w ∈ W_s).
B.3 A Result under Exchangeability of the Predictors
This section extends the ensemble analysis from independent predictors to exchangeable ones. It retains exponentially shrinking prediction errors under the stated exchangeability framework, supporting ensemble-based seen/unseen classification.
- The analysis allows predictors to share training samples by replacing independence with the weaker assumption of exchangeability.
- Theorem 3 assumes that the predictor error-rate pairs are exchangeable and have a common mean.
- The exchangeability argument applies a result for exchangeable variables to the case where each X_i is a Rademacher random variable.
- The prediction errors shrink exponentially in L, making the ensemble predictor a viable approach.