PaperlayerView exact revision on arXiv

Source-linked AI summary

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization

Dixuan Wang, Yanda Li, Junyuan Jiang, Zepeng Ding, Ziqin Luo, Guochao Jiang, Jiaqing Liang, Deqing Yang

arXiv:2405.17067v2cs.CLcs.AI

TL;DR

The paper investigates whether tokenization errors underlie inaccurate LLM responses to specific queries, with particular attention to Chinese scenarios. It constructs ADT to challenge token segmentation and reports that it challenges studied open- and closed-source LLMs, resulting in incorrect answers.

  • Problem

    The paper identifies unsatisfactory LLM token segmentation, particularly in Chinese, as an under-studied vulnerability associated with inaccurate responses to specific queries.

    Source: §1 · p. 1

  • Method

    The authors construct ADT from vocabularies of mainstream LLMs, combining manually built ADT-Human with automatically generated ADT-Auto.

    Source: §1 · p. 2

  • Results

    ADT challenges studied open- and closed-source LLM token segmentation and leads to incorrect answers; ADT-Human TN cases average 80.91% for Chinese data and 79.78% for English data.

    Source: §4.2 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    The study draws attention to tokenization’s impact on LLM performance, while strategies for improvement remain under exploration.

    Source: Limitations · p. 9

    The study focuses primarily on Chinese, and whether other languages are similarly affected remains to be investigated.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have shown remarkable capabilities in language understanding and generation. Nonetheless, it was also witnessed that LLMs tend to produce inaccurate responses to specific queries. This deficiency can be traced to the tokenization step LLMs must undergo, which is an inevitable limitation inherent to all LLMs. In fact, incorrect tokenization is the critical point that hinders LLMs in understanding the input precisely, thus leading to unsatisfactory output. This defect is more obvious in Chinese scenarios. To demonstrate this flaw of LLMs, we construct an adversarial dataset, named as $\textbf{ADT (Adversarial Dataset for Tokenizer)}$, which draws upon the vocabularies of various open-source LLMs to challenge LLMs' tokenization. ADT consists of two subsets: the manually constructed ADT-Human and the automatically generated ADT-Auto. Our empirical results reveal that our ADT is highly effective on challenging the tokenization of leading LLMs, including GPT-4o, Llama-3, Deepseek-R1 and so on, thus degrading these LLMs' capabilities. Moreover, our method of automatic data generation has been proven efficient and robust, which can be applied to any open-source LLMs. In this paper, we substantially investigate LLMs' vulnerability in terms of challenging their token segmentation, which will shed light on the subsequent research of improving LLMs' capabilities through optimizing their tokenization process and algorithms.

1 Introduction

The paper frames inaccurate responses to specific queries as a tokenization vulnerability observed across mainstream LLMs. It introduces ADT, with human- and automatically constructed subsets, to challenge token segmentation and examine its relation to response accuracy.

1 Introduction

  • The authors report that mainstream open- and closed-source LLMs can produce incorrect or nonsensical responses when their input tokenization is wrong.They argue that tokenization errors hinder accurate input understanding and lead to incorrect responses.Source: §1 · pp. 1–2
  • The introduction situates tokenization flaws in subword vocabularies that cannot cover every input expression, and says later LLM optimization cannot completely solve this underlying problem.It notes a lack of research specifically focused on unsatisfactory LLM token segmentation, particularly in Chinese scenarios.Source: §1 · p. 2
  • The paper introduces ADT, comprising manually constructed ADT-Human and automatically generated ADT-Auto, to challenge LLM tokenization.The dataset is intended to investigate the relationship between unsatisfactory tokenization and inaccurate responses to specific queries.Source: §1 · p. 2
  • The authors export vocabularies from multiple mainstream LLMs to construct ADT-Human and use GPT-4-generated instances, manually inspected for quality, to build ADT-Auto.ADT-Auto contains 231 instances and was constructed with minimal human effort.Source: §1 · p. 2

2 Related Work

Related work covers subword tokenization algorithms and prior work on tokenization and LLM vulnerabilities. The paper distinguishes its focus on challenging LLM capabilities through tokenization from these existing lines of research.

Algorithms for tokenization

  • BPE merges frequent adjacent token pairs, WordPiece uses mutual information for merging, and Unigram trims a large vocabulary according to subword-removal loss.The section also notes that SentencePiece integrates tokenization algorithms and that several LLM families use BPE, WordPiece, or Unigram.Source: §2 · pp. 2–3
  • Prior tokenization work includes improving segmentation of derivationally complex English words, while the authors report no work on shared LLM risks from inaccurate tokenization.They position their study as addressing this issue.Source: §2 · p. 3

Attack techniques in LLMs

  • Prior LLM attack research covers areas including prompt injection, model theft, data reconstruction, data poisoning, and member inference; this paper proposes challenging capabilities through tokenization.The paper describes this angle as diverging from existing work.Source: §2 · p. 3

3 Methodology of Dataset Construction

ADT uses language-model vocabularies to build manually constructed ADT-Human and automatically generated ADT-Auto datasets for challenging tokenization. The construction turns vocabulary-based token traps into sentence–question instances and filters generated examples for tokenization problems.

3.1 Vocabulary Export

  • ADT exports vocabularies from five Chinese-focused and three English-focused open-source LLMs, noting that two Chinese and two English model pairs share vocabularies.The same Chinese vocabulary is used by Qwen-7B-Chat and Qwen1.5-72B-Chat; Llama-3-8B-Instruct and Llama-3-70B-Instruct also share one.Source: §3.1 · p. 3

3.2 ADT-Human Construction

  • For ADT-Human, the authors select eleven tokens per Chinese vocabulary and seven per English vocabulary, then create challenging spans by inserting character sequences before, after, or on both sides of a token.They compose a sentence and a question whose answer comes from the challenging span; English examples also omit some spaces to test tokenization without regular delimiters.Source: §3.2 · pp. 4–5, Table 2 · p. 4

3.3 ADT-Auto Generation

  • ADT-Auto focuses primarily on Chinese data because the authors consider English tokenization challenges less severe and manual dataset construction inefficient.The automatic framework is based on exported vocabularies.Source: §3.3 · p. 5

3.3.1 Word-pair Matching

  • A word pair qualifies for ADT-Auto when a part of one word joined to a part of the other forms a vocabulary Trap Word, but the full word pair is not itself a vocabulary token.The remaining parts must also be vocabulary tokens, and the Trap Word cannot begin or end with a Chinese stop character.Source: §3.3.1 · pp. 5–6

3.3.2 Instance Generalization

  • GPT-4 generates each ADT-Auto instance with a sentence containing the concatenated word pair and a related question whose answer comes from either word, the Trap Word, or their combination.The sentence must convey both words’ meanings.Source: §3.3.2 · p. 6

3.3.3 Filtering Based on Trap Word

  • An automatically generated instance is retained only if its Trap Word appears in the model’s tokenization lists for both the sentence and the answer.This criterion is intended to select cases indicating tokenization errors in understanding the sentence and response.Source: §3.3.3 · p. 6

3.3.4 Manual Filtering

  • Manual filtering checks that retained instances have reasonable expressions and LLM responses, while allowing examples that one model answers correctly if other LLMs may still tokenize them inaccurately.In the Qwen-7B example, 894 instances remained after automatic filtering and 231 after manual filtering.Source: §3.3.4 · p. 6

4 Experiments

The experiments test open- and closed-source LLMs on ADT-Human and ADT-Auto, with Chinese and English ADT-Human data and Chinese-only ADT-Auto. ADT-Human produces high error rates and frequent cases where both tokenization and responses are incorrect.

4.1 Experiment Setup

  • The evaluation covers open-source and closed-source LLMs, using Chinese and English ADT-Human instances and Chinese-only ADT-Auto instances.The model set includes locally deployed and API versions, with exceptions described for Chatglm3-6B and Deepseek-R1.Source: §4.1 · p. 6

4.2 ADT-Human’s Challenges to LLMs

  • For ADT-Human, response correctness is assessed by human reviewers; the paper also examines tokenization–response relationships using open-source models because closed-source tokenization lists are inaccessible.For API models, the corresponding locally deployed versions’ tokenization results are used; the analysis appears in appendices.Source: §4.2 · p. 7
  • ADT-Human’s TN cases average 80.91% for Chinese data and 79.78% for English data, while the reported error rates are high across open- and closed-source LLMs.The authors also report that GPT-4o cannot outperform GPT-4 on ADT-Human.Source: §4.2 · pp. 7–8

4.3 ADT-Auto’s Challenges to LLMs

  • ADT-Auto instances are drawn from Qwen-7B’s vocabulary, and their LLM inaccurate-response rates are reported in Table 5.The supplied passage does not state the table’s cell values or comparisons.Source: §4.3 · p. 8, Table 5 · p. 8

1. Compared with closed-source LLMs, open-

ADT-Auto challenges LLM tokenization, with TN cases averaging 46.11%. The results also indicate that challenge difficulty and response accuracy vary with dataset construction and model scale.

1. Compared with closed-source LLMs, open-

  • Compared with ADT-Human, ADT-Auto is less challenging because its GPT-4-generated sentences use more formal, regular, or simple syntax and expressions.The authors describe those sentences as relatively easy for LLMs to understand.Source: §1 · p. 8
  • Qwen1.5-72B-Chat has lower error rates than Qwen-7B-Chat despite matching tokenization on checked instances, suggesting larger models are more robust to incorrect tokenization.The authors attribute the larger model’s robustness to stronger capabilities from its larger scale.Source: §1 · p. 8
  • Closed-source models suffer from ADT-Auto challenges more apparently than open-source models.Source: §1 · p. 8
  • 46.11% average TN cases on ADT-Auto further indicates a relationship between tokenization errors and response accuracy.The authors report that TN cases make up a high proportion of results on ADT-Auto.Source: §1 · p. 8

5 Conclusion

The conclusion presents ADT as a two-subset adversarial dataset for investigating tokenization vulnerability and inaccurate responses. Experiments show that it challenges studied open- and closed-source LLMs’ token segmentation, resulting in incorrect answers.

5 Conclusion

  • ADT comprises manually constructed ADT-Human and automatically generated ADT-Auto, with each instance containing a sentence and a corresponding question.Source: §5 · p. 9
  • The paper investigates how LLM tokenization vulnerability relates to unsatisfactory responses on some tasks.Source: §5 · p. 9
  • Experiments show that ADT challenges studied open- and closed-source LLMs’ token segmentation, resulting in incorrect answers.Source: §5 · p. 9
  • The authors hope the work and dataset will inform subsequent research on improving LLM performance.Source: §5 · p. 9

Limitations

The study draws attention to tokenization’s impact on LLM performance and provides two data-generation frameworks, but improvement strategies remain under exploration. Its focus is primarily Chinese, and whether other languages are similarly affected remains to be investigated.

Limitations

  • The study focuses primarily on Chinese; whether other languages are similarly affected remains to be investigated.Source: Limitations · p. 9
  • Improvement strategies based on the observed phenomenon remain under exploration.Source: Limitations · p. 9

Ethics Statement

The ethics statement describes annotation safeguards and notes residual risks from synthetic data. The appendices provide examples of token-boundary challenges for Yi, Llama-3, and Mixtral.

Ethics Statement

  • The authors report that annotators consented, their privacy was protected, and they were paid according to local standards.Source: Ethics Statement · p. 9
  • Proofreading was used to refine offensive or harmful GPT-4-generated data, but some unsatisfactory data may remain in the final dataset.Source: Ethics Statement · p. 9

A.3 Yi

  • Yi examples use Chinese token-boundary markers in sentences paired with questions about occupations, clothing, and other content.Source: §A.3 · p. 13

A.5 Llama-3

  • Llama-3 examples split English words with a vertical bar, including “envelope” and “domestically.”Source: §A.5 · p. 14

A.6 Mixtral

  • Mixtral examples likewise split words with vertical bars, including “live|return” and “im|ports.”Source: §A.6 · p. 14

B.1 ADT-Human (Chinese)

The ADT-Human Chinese results are presented as counts crossing right/wrong tokenization with right/wrong responses. The panels identify local and API variants of Baichuan2-13B-Chat and Qwen1.5-72B-Chat.

B.1 ADT-Human (Chinese)

  • The displayed records distinguish right and wrong tokenization from right and wrong responses.For example, one record lists 2, 1, 1, and 40 across those labels.Source: §B.1 · p. 15
  • The Baichuan2-13B-Chat and Qwen1.5-72B-Chat panels include local and API variants.The records also show counts for the four right/wrong tokenization and response combinations.Source: §B.1 · p. 15
  • Additional Chinese-result records display the same four-way count format, including values 3, 0, 23, and 18.The supplied passages do not identify which model panel these values belong to.Source: §B.1 · p. 15

B.2 ADT-Human (English)

The supplied block combines ADT-Human English and ADT-Auto count records with panels for Chinese and English model variants. It also defines four tokenization-response outcome categories.

B.2 ADT-Human (English)

  • The ADT-Human English records display right/wrong tokenization and response counts, including 1, 0, 0, and 13 in one record.Source: §B.2 · p. 15

B.3 ADT-Auto

  • ADT-Auto records include displayed count sets such as 28, 32, 56, and 115, and 20, 6, 114, and 91.The records label the outcomes by right/wrong tokenization and right/wrong response.Source: §B.3 · p. 16

C Proportion of four situations between tokenization and response

  • The four defined outcomes are TP for correct tokenization and response, FP for incorrect tokenization but correct response, FN for correct tokenization but incorrect response, and TN for both incorrect.Source: C Proportion of four situations… · p. 16

C.1 ADT-Human (Chinese)

  • The cited panel labels identify Baichuan2-13B-Chat and Qwen1.5-72B-Chat, with each model shown in both Local and API variants.Source: §C.1 · pp. 16–17

C.2 ADT-Human (English)

  • The panel labels identify Llama-3-8B-Instruct, Llama-3-70B-Instruct, and Mixtral-8x7B-Instructv0.1, each shown in both Local and API variants.Source: §C.2 · p. 17

C.3 ADT-Auto

The ADT-Auto panel labels include FP, FN, and TN, alongside a displayed 2.2% value. The supplied passages also identify Baichuan2 and Qwen local/API panels.

C.3 ADT-Auto

  • The ADT-Auto passage displays FP, FN, and TN labels, but does not establish their proportions.Source: §C.3 · p. 17
  • Among the supplied panel labels, Baichuan2-13B-Chat appears in Local and API variants, as does Qwen1.5-72B-Chat; the labels distinguish each variant.Source: §C.3 · p. 17
  • A displayed 2.2% value appears alongside FP and FN labels in the ADT-Auto passage.Source: §C.3 · p. 17

Related papers

Loading 2405.17067v2…