PaperlayerView exact revision on arXiv

Source-linked AI summary

Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models

Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, Yulia Tsvetkov

arXiv:2305.13707v1cs.CL

TL;DR

Commercial language-model APIs charge by tokens, but tokenization varies across languages, raising questions about pricing fairness. The paper analyzes tokenizer fragmentation, API cost, and model utility across multilingual settings, finding that many over-fragmented languages face higher costs and poorer results, often alongside lower affordability.

  • Problem

    Token counts are model- and training-data-dependent, so equivalent information can require different numbers of tokens and create unequal API pricing across languages.

  • Method

    The study analyzes OpenAI and multilingual model APIs using multilingual benchmarks, tokenization measurements, cost comparisons, and utility evaluations.

  • Results

    Over-fragmented languages are often charged more and perform worse, with the disparity rooted partly in language or script properties and associated with lower API affordability.

  • Takeaways & Limitations

    API vendors should improve transparency and reconsider token-based processing and pricing strategies to make access more equitable.

  • Takeaways & Limitations

    The study cannot measure how translationese differences affect whether parallel texts convey exactly the same information.

Abstract

from arXiv · show

Language models have graduated from being research prototypes to commercialized products offered as web APIs, and recent works have highlighted the multilingual capabilities of these products. The API vendors charge their users based on usage, more specifically on the number of ``tokens'' processed or generated by the underlying language models. What constitutes a token, however, is training data and model dependent with a large variance in the number of tokens required to convey the same information in different languages. In this work, we analyze the effect of this non-uniformity on the fairness of an API's pricing policy across languages. We conduct a systematic analysis of the cost and utility of OpenAI's language model API on multilingual benchmarks in 22 typologically diverse languages. We show evidence that speakers of a large number of the supported languages are overcharged while obtaining poorer results. These speakers tend to also come from regions where the APIs are less affordable to begin with. Through these analyses, we aim to increase transparency around language model APIs' pricing policies and encourage the vendors to make them more equitable.

1 Introduction

Commercial language-model APIs charge by tokens, but tokenization varies across languages, creating disparities in cost and utility. The analyses link these disparities to poorer performance and greater economic burden for speakers of many underrepresented languages.

  • Motivation: Commercial language-model APIs charge users according to processed or generated tokens, while token definitions depend on model and training data.Subword tokenizers split text into chunks learned from character or byte frequency patterns.
  • Motivation: Languages with greater subword fragmentation require more tokens to convey the same information, directly increasing API usage costs.The cost difference can arise even when speakers convey equivalent information.
  • Approach: The study systematically analyzes tokenizer fragmentation, API cost disparity, and model utility across languages and scripts.It examines whether tokenization creates unequal pricing and poorer in-context learning outcomes.
  • Findings: Languages with longer token sequences from greater fragmentation derive less model utility with in-context learning.The paper attributes the disparity to inherent language or Unicode-representation properties, not only data imbalance.
  • Findings: Languages that cost more and perform worse are often associated with populations for whom the APIs are less affordable on average.This pattern exacerbates the economic divide in access to NLP technology.
  • Implications: The authors encourage API vendors to revisit processing and pricing strategies and urge NLP researchers to pay greater attention to tokenizers.They characterize tokenizers as an often neglected part of the LLM pipeline.

2 Do All Languages Cost the Same?

The paper asks whether tokenization imposes unequal token counts, API costs, and model utility across languages. It studies subword-tokenizer behavior and multilingual benchmarks, finding substantial fragmentation differences, higher costs, and poorer performance for some languages.

  • Background: API providers commonly support many languages and charge based on total input and generated tokens, although token is a developer design choice.Users may need a separate tokenizer interface to determine token counts.
  • Background: Subword tokenization balances vocabulary size and sequence length by recursively merging frequent characters, bytes, or candidate subwords.Byte-level approaches use a small script-agnostic base vocabulary, while multilingual character vocabularies can exceed 130K types.
  • RQ1 (number of tokens): Some languages require 5 times as many tokens as others to convey the same information, with fragmentation depending partly on linguistic or script properties.The analysis uses parallel sequences and compares average tokens per sentence across tokenizers.
  • RQ2 (cost): Non-uniform fragmentation can lead to significantly higher API usage costs for less represented languages.The paper frames this disparity as a reason to pursue more equitable API pricing.
  • RQ3 (model utility): High fragmentation can negatively affect in-context learning performance, reducing model utility in that language.Token limits also constrain how much information and how many demonstrations fit into a request.
  • RQ4 (socio-economic aspects): Users who likely cannot afford high API costs may receive more expensive and less effective service, hindering uniform accessibility.The analysis correlates crosslingual cost and performance measurements with speakers’ socio-economic indicators.

3 Experimental Setup

The experiments compare ChatGPT and BLOOMZ across multilingual tokenization, generation, classification, and span-prediction benchmarks. They use parallel data to measure token efficiency and evaluate both models with k-shot in-context learning under API token limits.

  • Models: The study focuses on ChatGPT and BLOOMZ, both general-purpose instruction-following models.ChatGPT is accessed through OpenAI’s API, while BLOOMZ is an open-source multilingual model evaluated through Hugging Face’s API.
  • Tokenization analysis: FLORES-200 parallel sentences are tokenized with ChatGPT’s tokenizer to compare average tokens per sentence across languages conveying the same information.A language is considered more efficiently tokenized when it uses fewer tokens per sentence on average.
  • Benchmarks: The evaluation covers summarization, classification, and span prediction using Cross Sum, XLSUM, XNLI, XFACT, and XQUAD.The benchmarks span multilingual summarization, natural-language inference, fact verification, and cross-lingual question answering.
  • Prompting formulation: Both models are evaluated with k-shot in-context learning and task instructions, with the maximum number of examples determined by API input-token limits and language fragmentation.The setup varies k from zero to the maximum number of examples that fit within the API’s input-token constraint.
  • Prompting formulation: Instructions are provided in English, and each task uses at most 500 randomly sampled test examples.The prompting choice follows prior evidence that English instructions outperform in-language prompts on several multilingual benchmarks.
  • Tokenization analysis: Figure 2 summarizes average tokens by script after tokenizing the FLORES dataset, with language-group sizes shown above each bar.The figure compares tokenization fragmentation across script groups rather than individual benchmark outcomes.

4 Results and Analysis

Tokenization varies substantially across languages and scripts, producing unequal API costs and constraining how much context some languages can use. Controlled tokenizer experiments and multilingual evaluations link these disparities to script properties, higher costs, and reduced utility.

  • RQ1: Number of tokens: Up to 5× more tokens are required by Telugu and Georgian than Latin-script languages to convey the same information.Latin-script languages use substantially fewer tokens, while Cyrillic and Japanese are closer to Latin-script efficiency.
  • RQ1: Number of tokens: Tokenizer uniformity depends on both pretraining-data proportion and inherent language or writing-script properties.Controlled BBPE experiments use parallel corpora with matched content and data size across scripts.
  • RQ2: Cost: Close to 5× higher API costs occur for most mid-resourced Indic languages using non-Latin scripts compared with English.Costs are estimated from average sequence lengths and are generally cheaper for Indo-European and Latin-script languages.
  • RQ2: Cost: Up to 4× more is spent on prompting and generation for Telugu and Amharic than for English in XLSUM evaluations.Similar cost patterns are reported for XFACT and CROSSUM.
  • RQ3: Model utility: ChatGPT fits more few-shot examples in Latin-script languages, while Telugu and Amharic often cannot fit even one example.Non-uniform fragmentation reduces the information that fits within the model’s token context limit.
  • RQ3: Model utility: Adding in-context examples usually improves ChatGPT performance, but BLOOMZ gains no benefit and sometimes drops sharply because it copies spans from demonstrations.BLOOMZ evaluations also require truncated articles for examples exceeding its 1000-token context limit, with expected performance loss.

5 What is the Way Forward?

The paper proposes greater transparency and more equitable pricing for multilingual APIs, alongside technical and organizational alternatives. It also identifies open-source models and tokenizer improvements as possible ways to improve accessibility, while noting practical barriers.

  • Transparency in API limitations: API providers should disclose model flaws and biases, especially when advertising multilingual capabilities.Users may otherwise face higher charges without understanding how the models work.
  • Rethinking the API pricing models: Pricing could account for languages, regions, and language-specific benchmark performance instead of charging uniformly by tokens.The paper also mentions character-based and hourly pricing as alternatives requiring further fairness analysis.
  • Technological improvements in language models: Figure 10 relates each language’s fragmentation rate to the Human Development Index of its top country of use.The supplied figure caption specifies the variables but does not state the comparison’s outcome.
  • Open-source models vs. paid APIs: BLOOMZ performs better in zero-shot evaluations than ChatGPT does in few-shot evaluations in most cases.Open-source models may nevertheless involve academic-license restrictions and substantial deployment costs.
  • Technological improvements in language models: Data balancing, optimal transport, fuzzy subwords, and longer context lengths are proposed avenues for addressing tokenization and utility problems.BLOOMZ uses data balancing to improve fragmentation rates, while longer contexts allow more in-context examples.

6 Related Work

Related work examines tokenization’s effects on multilingual model performance, efficiency, and fairness. It also documents broader sources of language and social disparity across language-model development and generation.

  • Analyzing Tokenization methods: Prior studies analyze tokenization effects on model performance, inference speed, and memory usage.This work builds on research spanning multilingual and practical deployment settings.
  • Analyzing Tokenization methods: mBERT’s vocabulary favors Indo-European languages, while over-fragmentation helps explain why monolingual models can outperform mBERT.Other work finds sentence-level machine translation models are not especially sensitive to tokenizer-training data imbalance.
  • Analyzing Tokenization methods: Research also shows that unfairness can arise at multiple development stages and includes social, cultural, and language disparities.These studies extend beyond tokenizer behavior to biases in generated text and multilingual model coverage.

7 Conclusion

The conclusion reports that commercial multilingual APIs disproportionately tokenize some languages, charge their speakers more, and perform worse for them. It calls for greater transparency and revised pricing to improve accessibility.

  • 7 Conclusion: API tokenizers favor Latin-script languages and over-fragment less represented languages and scripts.The conclusion identifies tokenizer behavior as a source of cross-language disparity.
  • 7 Conclusion: Token-based pricing is unfair toward speakers of over-fragmented languages, while API performance is poorer for them than for less-fragmented counterparts.The conclusion links these disparities to reduced accessibility for marginalized communities.

Ethics Statement

The paper frames equitable access to commercial language technologies as an ethical concern for speakers of less-represented languages and scripts. It also acknowledges that incomplete knowledge of training data limits conclusions about optimization across languages.

  • Ethics Statement: Unfair tokenization can impose consequences on API users whose languages use scripts less represented in pretraining data.The paper connects widespread commercial-language-model use with the goal of accessibility across languages.
  • Ethics Statement: The analysis cannot make solid claims about all languages optimized by ChatGPT because OpenAI’s training data are unavailable.The authors note that non-uniform tokenization may also reflect factors beyond data size and writing systems.

Limitations

The analysis uses translated parallel text to control information content, but cannot measure translationese differences. It also estimates ChatGPT’s language statistics with CC100 because OpenAI has not released its training details.

  • Translationese: FLORES-200 provides professionally translated parallel examples to control for the information conveyed across languages.The authors acknowledge that translated text may differ from original writing, but expect these differences not to meaningfully affect fragmentation-rate trends.
  • Translationese: The authors cannot measure whether translationese differences alter the information conveyed across languages.They expect such differences to be insufficiently large to meaningfully affect fragmentation-rate trends.
  • Language statistics of ChatGPT training data: OpenAI has not released ChatGPT’s training details or the language statistics needed to determine its actual training-data distribution.The analysis therefore uses CC100 as a multilingual corpus to estimate those statistics.
  • Language statistics of ChatGPT training data: CC100 serves as an estimate of ChatGPT’s language statistics rather than a direct measurement of its training data.ChatGPT is a closed model, so the actual statistics of all supported languages cannot be ascertained.

A Appendix

The appendix documents prompt templates and visualizes token costs, fragmentation, language-resource relationships, affordability, and the limits of achieving uniform fragmentation across scripts.

  • A Appendix: Table 2 presents the prompt template used for each dataset.
  • A Appendix: Figures 11–12 compare relative evaluation cost and GPT-3.5 token counts across language families.Figure 11 covers XFACT and CROSS-SUM; Figure 12 reports Flores token counts and notes lower fragmentation for Latin-script languages than other scripts.
  • A Appendix: Figures 13–15 show tokenizer fragmentation by script and language, ordered by pretraining-corpus size for GPT-3.5 and BLOOM.Figure 13 notes higher average BLOOM fragmentation for Latin-script languages because many low-resourced languages use Latin script.
  • A Appendix: Figure 16 estimates API access cost relative to English, while Figure 17 relates language fragmentation to Human Development Index.
  • A Appendix: Figure 18 shows that equal pretraining-data sizes across 30 scripts cannot produce uniform fragmentation rates with BBPE tokenization.The figure varies vocabulary sizes while illustrating the persistence of non-uniform fragmentation across scripts.
Loading 2305.13707v1…