PaperlayerView exact revision on arXiv

Source-linked AI summary

Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, Donald Metzler

arXiv:2106.12672v3cs.CLcs.AIcs.LG

TL;DR

Rigid subword tokenization limits adaptation and robustness, while direct character modeling is computationally expensive. The paper introduces GBST and Charformer, which learn latent subwords end-to-end from bytes; across English, noisy-text, and multilingual tasks, Charformer generally matches or exceeds subword models and improves efficiency. Its contiguous-subword formulation may struggle with discontinuous morphology.

  • Problem

    Rigid tokenization limits generalization and adaptation, while direct character-level modeling increases computational and memory complexity.

  • Method

    GBST learns latent subword representations by softly scoring candidate character blocks, and Charformer integrates GBST into a byte-level Transformer.

  • Results

    Across English, noisy-text, and multilingual tasks, Charformer outperforms byte-level baselines, generally matches or exceeds subword models, and improves efficiency by 2× in memory and 10–93% in speed against a comparable ByT5 setup.

  • Takeaways & Limitations

    The results support token-free, end-to-end models that retain competitive quality while combining character-level representations with subword-like efficiency.

  • Takeaways & Limitations

    Because GBST assumes contiguous character sequences, discontinuous morphology may be harder to model; skipgram extensions would increase computational costs.

Abstract

from arXiv · show

State-of-the-art models in natural language processing rely on separate rigid subword tokenization algorithms, which limit their generalization ability and adaptation to new settings. In this paper, we propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model. To this end, we introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters in a data-driven fashion. Concretely, GBST enumerates candidate subword blocks and learns to score them in a position-wise fashion using a block scoring network. We additionally introduce Charformer, a deep Transformer model that integrates GBST and operates on the byte level. Via extensive experiments on English GLUE, multilingual, and noisy text datasets, we show that Charformer outperforms a series of competitive byte-level baselines while generally performing on par and sometimes outperforming subword-based models. Additionally, Charformer is fast, improving the speed of both vanilla byte-level and subword-level Transformers by 28%-100% while maintaining competitive quality. We believe this work paves the way for highly performant token-free models that are trained completely end-to-end.

1 INTRODUCTION

Charformer replaces rigid, separate tokenization with an end-to-end gradient-based approach, combining character-level compositionality with subword efficiency. Across English, noisy, multilingual, and efficiency evaluations, it matches or exceeds strong baselines while improving computational efficiency.

  • Rigid frequency-based tokenization can make models brittle to rare words and perturbations, disadvantage low-resource languages, and create pre-training–downstream mismatches.
  • Character-level modeling avoids separate tokenization but incurs substantially higher sequence length, computational complexity, and memory use.
  • GBST learns latent subword representations from characters through position-wise soft scoring of candidate blocks, enabling end-to-end tokenization.
  • Charformer integrates GBST into a byte-level Transformer, with a re-scaled variant allocating more capacity to the encoder.
  • Charformer performs on par with or better than subword-based models across English, noisy-text, and multilingual tasks, while outperforming byte-level baselines.
  • 2× memory efficiency and 10–93% faster training or inference distinguish Charformer from comparable byte-level baselines, while it is also 28% faster than mT5 with 3× fewer parameters.

2 CHARFORMER

Charformer is an efficient character-level architecture that processes bytes through GBST before applying deep Transformer layers. GBST downsamples the input sequence to construct latent subwords, using a shared 256-value byte vocabulary across settings.

  • Charformer consists of a GBST module followed by deep Transformer layers, operating directly on character or byte sequences.
  • GBST receives character embeddings and downsamples the sequence to construct latent subwords.
  • Bytes provide a vocabulary of 256 possible values across settings, with non-Latin characters generally represented by 2–3 bytes.

2.1 GRADIENT-BASED SUBWORD TOKENIZATION (GBST)

GBST learns soft, position-wise subword selections from character blocks, then forms shorter latent-subword sequences for the Transformer.

  • Constructing Candidate Latent Subword Blocks: GBST enumerates candidate contiguous blocks up to maximum size M and pools each block into a single representation.Strided pooling creates candidate representations for multiple block sizes and shortens sequences through downsampling.
  • Constructing Candidate Latent Subword Blocks: Offsets and convolution address limitations of strided blocks, while intra-block position information preserves character ordering.Without offsets, some sliding windows are omitted; positional embeddings or convolution can distinguish different within-block orders.
  • Block Scoring Network: A block-scoring network assigns each candidate block a score at every original character position.The scoring function is a parameterized linear transformation from each block representation to a scalar.
  • Block Scoring Network: GBST applies a position-wise softmax over block sizes, producing relative selection probabilities for the realigned candidates.Each block representation is replicated across its block length before candidate scores are normalized.
  • Forming Latent Subwords: The model combines candidate block representations using their learned probabilities to produce a latent subword representation at each position.This soft mixture allows different possible segmentations to remain available during learning.
  • Forming Latent Subwords: The method generally assumes contiguous subwords, making discontinuous morphology harder to model.The paper suggests skipgrams as a possible extension, at increased computational cost.
  • Forming Latent Subwords: A subsequent fixed mean-pooling downsampler reduces the latent sequence length by a factor of d_s before Transformer processing.Mean pooling can remove redundancy when adjacent positions select similar blocks.

2.2 TRANSFORMER STACK

The Transformer stack operates on GBST’s downsampled latent subwords, with a scaling variant reallocating capacity toward a deeper encoder.

  • Transformer Stack: CHARFORMER keeps a regular Transformer encoder-decoder stack but feeds it downsampled latent subwords instead of subword embeddings.GBST precedes the Transformer layers and operates directly on character or byte inputs.
  • Re-scaling of the Transformer Stack: Character-level models devote less capacity to embeddings than subword models, motivating deeper encoder stacks to compensate for smaller embedding capacity.The paper reports that contemporary multilingual subword models may allocate up to 71% of parameters to subword embeddings.
  • Transformer Stack: Future work could model discontinuous morphology with skipgrams, but this would increase computational costs.This scope boundary follows the method’s reliance on contiguous character sequences.
  • Re-scaling of the Transformer Stack: The scaling variant reallocates parameters from the decoder to a deep, narrow encoder.The Base configuration is adjusted toward the T5 Small parameter scale with an expanded 24-layer encoder.

A Note on Comparing Character-level and Subword-based Methods

Comparisons between character-level and subword-based models must account for how parameters are allocated and how tokenization affects computation.

  • A Note on Comparing Character-level and Subword-based Methods: Matching parameter counts alone may not fairly compare character-level and subword-based models.Subword embedding lookup is O(1), whereas reallocating embedding parameters into character-level dense layers incurs higher computational cost.
  • A Note on Comparing Character-level and Subword-based Methods: ByT5’s redistribution of subword embedding parameters to dense layers incurs a 25% training-speed penalty.The paper uses this cost to motivate fairer re-scaling of character-level models.
  • A Note on Comparing Character-level and Subword-based Methods: The pre-training scheme masks N contiguous characters and predicts them with a sequence-to-sequence model using teacher forcing.Training uses the cross-entropy loss and follows T5 closely.

3 EXPERIMENTS

The experiments evaluate CHARFORMER across standard English, noisy English, multilingual tasks, and compute efficiency against subword, byte-level, and character-level baselines. CHARFORMER is competitive with subword models while improving performance or efficiency in several settings.

  • Experimental setup: CHARFORMER is compared with subword, byte-level, and character-level baselines across English, noisy-text, multilingual, and efficiency benchmarks.The evaluation includes GLUE, long-document classification, toxicity detection, multilingual question answering and classification, and compute measurements.
  • Standard English: CHARFORMERSBase outperforms same-condition character-level baselines across GLUE tasks and performs on par with or better than subword models on some standard-English tasks.The rescaled SBase configuration also performs better despite having fewer parameters than the Base configuration.
  • Long documents: CHARFORMERSBase is the only byte-level model to outperform T5Base,Subword on IMDb, while both CHARFORMER models outperform byte- and subword-level baselines on AGNews.These results concern long-document text classification.
  • Noisy English: Character-level models outperform the subword-based T5 model on both toxicity-detection datasets, while CHARFORMER is on par with or better than other character-level methods.The datasets contain noisy, user-generated text with spelling variation and non-standard language.
  • Multilingual experiments: CHARFORMERSBase is competitive with multilingual subword models, while the longer-pretrained variant outperforms them on TyDiQA-GoldP and reaches parity or near-parity on several translate-train tasks.CHARFORMER also outperforms other character-level models in the cross-lingual zero-shot setting, though token-free models remain challenged there.
  • Efficiency: CHARFORMERSBase is 28% faster than mT5Base while using 33% of the FLOPS, and CHARFORMER is generally more efficient than other character-level models.Efficiency varies with input length, downsampling rate, and model size.

4 ANALYSES

CHARFORMER combines learned latent subwords with efficient character-level processing, achieving competitive quality while reducing computational costs. Analyses show interpretable segmentation behavior and task-dependent effects from architectural choices.

  • Speed, memory and parameters: CHARFORMER is twice as fast as T5Base with similar performance, while using a deep thin encoder and more FLOPS than smaller alternatives.On English C4 pre-training at 1K input length, speed, parameters, and FLOPS were measured on 16 TPU-v3 chips.
  • Interpretability: CHARFORMER learns interpretable block scores that allocate single-character subwords mainly to English vowels and whitespace.The visualization examines multilingual CHARFORMER’s softmax weights for blocks of different sizes on “on subword tokenization.”
  • Interpretability: The learned segmentation suggests dynamic mixing between byte-level and subword-level features, with consonants contributing to larger subword units.The authors relate this behavior to the relative importance of consonants for word identification in reading studies.
  • Ablations: Pre-GBST convolutions are preferred to offset-block enumeration because they provide similar or better quality with a more efficient implementation.Block-score calibration helps English performance but has little multilingual effect; downsampling impacts vary by task and model size.

5 RELATED WORK

Related work frames CHARFORMER against rigid subword tokenization, direct character modeling, multilingual tokenization, and efficient Transformer architectures. These approaches address parts of the problem but leave trade-offs in robustness, efficiency, or meaningful composition.

  • Subword tokenization: BPE, WordPiece, and SentencePiece are standard deterministic subword tokenizers, but rigid tokenization handles language variation poorly.Prior work reports limitations in these algorithms and weaker coping with morphological and compositional variation.
  • Character-level models: Pure character-level recurrent models were largely superseded by character-aware methods because direct character modeling often performed poorly.Character-aware methods typically construct token-level representations using convolutions over characters.
  • Multilingual models: Multilingual BERT and XLM-R apply shared subword tokenization to multilingual corpora, exacerbating over-segmentation in low-resource languages.The passage describes multilingual models as analogues of successful monolingual Transformer models.
  • Efficient Transformers: Character inputs lengthen sequences and make Transformer self-attention costly because its complexity is quadratic in sequence length.Efficient attention can address computational costs, but does not guarantee locally meaningful compositions.

6 CONCLUSION

The paper concludes that CHARFORMER integrates learned latent subwords into an efficient token-free architecture. Across English, noisy social-media, and multilingual settings, it combines competitive quality with improved efficiency.

  • Conclusion: CHARFORMER integrates gradient-based subword tokenization into a re-scaled Transformer that learns latent subwords directly from characters.The authors describe GBST as a lightweight method enabling efficient end-to-end learning.
  • Conclusion: CHARFORMER outperforms character-level baselines, matches subword models on standard English and multilingual data, and exceeds them on noisy social-media data.The conclusion reports this pattern across various datasets without narrowing it to a single benchmark.
  • Conclusion: CHARFORMER is faster than both byte-level and subword-level baselines while retaining comparable quality on multilingual benchmarks.The paper also provides a method for inspecting GBST’s learned behavior.

ETHICS STATEMENT

Standard subword tokenization can represent languages unevenly, disadvantaging under-represented languages in multilingual models. Tokenization-free approaches may ameliorate this imbalance to some extent.

  • Ethics statement: Standard subword tokenizers are biased toward well-resourced languages, causing poorer multilingual-model performance on under-represented languages.The passage attributes this to unequal representation of words and phrases across languages.
  • Ethics statement: Tokenization-free approaches may help reduce the effects of resource-driven segmentation bias in multilingual models.The passage presents this as a qualified possibility rather than a guaranteed outcome.

7 APPENDIX

The appendix documents model configurations, multilingual training settings, large-scale evaluation scope, and a simplified GBST implementation. It also notes that the example implementation omits block score calibration and block-by-block attention.

  • Large-scale experiments: The appendix includes preliminary scaling experiments matching mT5Large and ByT5Large at 1.23B parameters.The scaling configuration follows ByT5 and uses the main multilingual hyperparameter settings.
  • Evaluation breakdowns: Per-language appendix tables break down TyDiQA, XQuAD, MLQA, XNLI, and PAWS-X results, including cross-lingual zero-shot evaluations.Additional tables examine the effect of freezing the GBST layer for XNLI and PAWS-X.
  • Example implementation: The simplified GBST code enumerates block widths, forms and scores candidate blocks, realigns them, applies softmax weighting, aggregates candidates, and optionally downsamples by mean pooling.The implementation accepts character-scale tensors and returns a sequence reduced by the configured downsample rate.
  • Example implementation: The example implementation omits block score calibration and block-by-block attention.The omission is explicitly marked in the code comments.
Loading 2106.12672v3…