PaperlayerView exact revision on arXiv

Source-linked AI summary

GlitchPatch: Repairing Glitch Tokens in Frozen Language Models via Local Retokenization

Kunsheng Tang, Peigui Qi, Yide Song, Peijun Huang, Weiming Zhang, Nenghai Yu

arXiv:2610.04399v1cs.CLcs.AI

TL;DR

Glitch tokens can make language-model outputs inconsistent with their inputs, while existing repairs require access to model internals that frozen deployments may not permit. GLITCHPATCH optimizes equivalent tokenizations offline and substitutes matched token IDs online; across ten models, it reports an 85.10% mean fix rate and a reduction in mean glitch rate from 14.88% to 2.27%.

  • Problem

    Existing glitch-token repair methods require access to model internals, leaving external repair for frozen checkpoints unexplored.

    Source: §1 · pp. 1–2

  • Method

    GLITCHPATCH uses offline Behavioral Path Optimization to compile validated replacement sequences, then substitutes matched glitch-token IDs online without modifying model internals.

    Source: §1 · p. 2

  • Results

    85.10% mean fix rate outperforms the strongest baseline by 14.37 percentage points, while mean glitch rate falls from 14.88% to 2.27% across ten models.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The results support GLITCHPATCH as a practical option for improving tokenizer reliability, with evaluated impacts on time cost, language understanding, and capability.

    Source: §1 · p. 2, §7 · p. 9

    Patches must preserve decoded bytes: D(R(T(x))) = D(T(x)).

    Source: §4.1.1 · p. 5

Abstract

from arXiv · show

Glitch tokens are anomalous vocabulary entries that can cause large language models (LLMs) to produce outputs inconsistent with their inputs. Existing repair methods require access to model internals, making them impractical for frozen checkpoints. We investigate whether glitch tokens can be repaired outside the model by optimizing the input tokenization. An empirical study on BPE merge-rule deletion reveals that (1)deleting a glitch token's merge rule can fix a substantial fraction of failures, yet disrupting normal tokens sharing intermediate merge nodes causes the overall glitch rate to rise, and (2)different decomposition granularities yield non-monotonic fix rates while collateral damage on normal tokens grows monotonically. Motivated by these findings, we propose GlitchPatch, a repair framework for frozen language models based on local retokenization, consisting of two stages: the offline stage uses Behavioral Path Optimization (BPO) to find the behaviorally optimal replacement token sequence for each glitch token and compiles validated replacements into a rule table; the online stage substitutes only the IDs of matched glitch tokens in the canonical token sequence, with no modification to model parameters or internal states. Experiments on ten models spanning six tokenizer families show that GlitchPatch achieves an 85.10% mean fix rate, outperforming the strongest baseline by 14.37 percentage points, and reduces the average glitch rate from 14.88% to 2.27%. GlitchPatch achieves a 0.00% RR in full-vocabulary evaluation and leaves rule-unmatched inputs unchanged by design. We further evaluate the practical impact of repair from the perspectives of time cost, language understanding, and capability, supporting its deployment feasibility. We hope this work provides a practical option for improving tokenizer reliability.

1 INTRODUCTION

Glitch tokens can make model outputs inconsistent with inputs, while existing repairs require model-internal access that is impractical for frozen deployments. GLITCHPATCH instead optimizes tokenization locally, and the paper reports high fix rates with no full-vocabulary regression.

1 INTRODUCTION

  • Glitch tokens can induce outputs inconsistent with inputs, and existing repair methods require model-internal access that creates risks for frozen production pipelines.Providers need fast, reversible fixes that leave the model untouched.Source: §1 · pp. 1–2
  • Deleting a glitch token’s merge rule can fix failures but disrupts normal tokens sharing intermediate nodes, while deeper decompositions have non-monotonic fix rates and increasing collateral damage.These findings motivate optimizing a path for each glitch token without affecting normal tokens.Source: §1 · p. 2
  • 85.10% mean fix rate surpasses the strongest baseline by 14.37 percentage points, while mean glitch rate falls from 14.88% to 2.27% and full-vocabulary RR is 0.00%.Requests with no applicable rule remain unchanged by design.Source: §1 · p. 2
  • GLITCHPATCH uses offline Behavioral Path Optimization to compile validated replacements, then substitutes matched glitch-token IDs online without changing the tokenizer or model internals.The online stage uses rule matching and ID substitution only.Source: §1 · p. 2
  • The paper reports practical feasibility across time cost, language understanding, and general capability evaluations.The reported average token-level loss drops by 13.5% on natural texts containing glitch tokens, with negligible online latency overhead.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

This background reviews tokenization, glitch-token detection, and repair, situating the paper’s focus on externally repairing frozen models.

2 BACKGROUND AND RELATED WORK

  • The background connects tokenizer behavior with prior work on glitch-token detection and repair.It frames the gap between existing methods and frozen deployment needs.Source: §2 · p. 2

2.1 TOKENIZER AND TOKENIZATION

  • BPE repeatedly merges adjacent units by learned rule priority; alternative segmentations can decode to the same bytes and serve as equivalent tokenizations.The paper treats tokenization as a manipulable variable because some frozen models tolerate or benefit from non-canonical segmentations.Source: §2.1 · p. 3

2.2 GLITCH-TOKEN DETECTION AND REPAIR IN LLMS

  • Glitch tokens can cause anomalous behavior; detection uses repetition, spelling, and length tasks, while existing repair methods intervene in activations, embeddings, or model parameters.Those repair methods require internal access, which the paper’s frozen-checkpoint setting avoids.Source: §2.2 · p. 3

3 EMPIRICAL STUDY ON TOKENIZATION REPAIR

The empirical study finds that global BPE merge deletion can repair some glitch tokens, but also damages normal tokens; deeper deletion does not provide a universally better path.

3 EMPIRICAL STUDY ON TOKENIZATION REPAIR

  • The setup evaluates Llama-3.1-8B-Instruct and Qwen3.8-27B with repetition, spelling, and length tasks across every vocabulary entry.The models represent the Llama and Qwen BPE tokenizer families.Source: §3 · p. 3
  • Across deletion depths, FR is non-monotonic and model-specific, while RR rises monotonically and overall GR worsens.Llama FR changes from 36.82% to 33.14% to 37.84%; Qwen FR peaks at 44.23% at depth-2 before falling to 38.61% at depth-3.Source: Figure 3 · p. 4, §3 · p. 4
  • The study applies globally unioned deletion sets at three backtracking depths, then re-encodes the vocabulary and reruns detection using GR, FR, and RR.GR measures tokens still failing, FR measures original glitches made normal, and RR measures original normal tokens made glitchy.Source: §3 · pp. 3–4
  • Global deletion has no universally optimal depth, and its collateral damage to normal tokens intensifies with depth.The study concludes that effective external repair must find each token’s best replacement path while avoiding effects on normal tokens.Source: §3 · p. 4

4 GLITCHPATCH

GLITCHPATCH repairs known glitch tokens through offline behavioral path optimization and online local ID substitution, leaving unmatched normal-token IDs unchanged. Its offline search selects validated equivalent token sequences under behavioral and length constraints; the online stage applies eligible rules to canonical token IDs.

4 GLITCHPATCH

  • GLITCHPATCH combines offline Behavioral Path Optimization (BPO), which compiles validated replacement sequences into a rule table, with online local ID substitution.BPO searches for an optimal replacement for each glitch token; online substitution is budget-constrained.Source: §4 · p. 5, Figure 5 · p. 5

4.1.1 PROBLEM FORMULATION

  • BPO selects a replacement path with positive net utility, subject to decoding equivalence and a token-length budget.When no candidate qualifies, the original representation is retained.Source: §4.1 · p. 5, §4.1.1 · p. 5
  • A glitch token’s candidate space comprises paths of normal tokens that cover its byte sequence from start to end.Byte positions are graph nodes, and a normal token spanning a substring forms an edge.Source: §4.1.1 · p. 5

4.1.2 BEHAVIORAL PATH OPTIMIZATION (BPO)

  • BPO constructs candidate paths, ranks them with a proxy score, fully evaluates the top-K, and confirms accepted rules on independent calibration probes.Candidates must pass the full-sequence behavioral checks and the utility threshold; a rule failing calibration is rejected without trying the next-best candidate.Source: §4.1.2 · p. 6

4.2 ONLINE LOCAL RETOKENIZATION

  • The online pass replaces a token only when it is a registered glitch token with a validated rule and the length budget permits substitution.Unmatched normal IDs and known glitch-token IDs lacking a usable rule are retained; matching is against the initial canonical IDs and is not recursive.Source: §4.2 · p. 6

5 EXPERIMENTAL SETUP

The evaluation tests ten models across six BPE tokenizer families using full-vocabulary glitch detection and token-level repair metrics. It compares GLITCHPATCH with activation- and parameter-based methods and assesses time cost, language understanding, general capability, and additional contextual and downstream outcomes.

Models and Execution

  • Ten models spanning six BPE tokenizer families, with vocabularies from 32K to 262K, are evaluated using BF16, official chat templates, and greedy decoding.Deepseek-vl2 uses text-only input.Source: §5 · pp. 6–7

Detection and Evaluation

  • Every vocabulary token is tested on repetition, spelling, and length tasks; tokens passing all three are labeled normal, and the rest glitch.Full-vocabulary evaluation tracks each token’s status change without selection bias.Source: §5 · p. 7

Baselines

  • Baselines are GlitchProber, GlitchEdit, and GlitchCleaner, which retain their required model-internal access while GLITCHPATCH is input-only.Methods share the same vocabulary detection results and classification criteria.Source: §5 · p. 7

Repair Effectiveness

  • Repair effectiveness is measured by Glitch Rate (GR), Fix Rate (FR), and Regression Rate (RR) over vocabulary tokens.GR measures remaining glitches, FR measures originally glitchy tokens repaired, and RR measures originally normal tokens that become glitches.Source: §5 · p. 7

Practicality Assessment

  • Practicality is assessed through offline build time and online latency, token-level loss reduction, and four-benchmark General Score.The study also evaluates contextual regression and out-of-distribution downstream performance.Source: §5 · p. 7

6 EXPERIMENTAL RESULTS

Across ten models, GLITCHPATCH improves full-vocabulary repair results over the baselines, with low regression and reduced glitch rate. The evaluation also reports practical costs and capability-related outcomes, examines proxy-score quality and repairability, and tests the effect of the candidate-count parameter.

6.1 REPAIR EFFECTIVENESS

  • 85.10% mean FR accompanies 0.00% RR and a mean GR decrease from 14.88% to 2.27% across ten models.Table 1 reports full-vocabulary, token-level repair results.Source: §6.1 · p. 7, Table 1 · p. 7
  • 85.10% mean FR exceeds GlitchCleaner’s 70.73% by 14.37 pp; GLITCHPATCH has the highest FR on every model.Table 2 compares Fix Rate across four methods.Source: §6.1 · p. 7, Table 2 · p. 8

6.2 PRACTICALITY ASSESSMENT

  • Mean online latency increases by less than 0.1%, with offline builds taking 0.36–2.08 h.The practicality comparison measures build time and latency on 8×A100.Source: §6.2 · p. 8, Figure 6 · p. 8
  • Relative token-level loss reductions are 13.2%, 13.9%, and 11.9% on the three tested models, exceeding all baselines.The ten-model mean relative loss reduction is 13.47%.Source: §6.2 · p. 8
  • General Score changes range from 0.000 to +0.025 pp across three models, while contextual QA accuracy changes by at most ±0.2 pp.The contextual evaluation measures short-answer QA on adjacent normal-token content; GLITCHPATCH also achieves the highest accuracy in the reported out-of-distribution evaluation.Source: §6.2 · p. 8

6.3 PROXY SCORE QUALITY AND REPAIRABILITY

  • Spearman’s ρ(−J, U) is 0.72, 0.68, and 0.71 for Llama-8B, Qwen3.8, and Gemma-31B, respectively.Increasing K from 8 to 16 improves FR by 0.43 pp while nearly doubling cost.Source: §6.3 · p. 9

6.4 REPAIRABILITY ANALYSIS

  • 5.00% of Qwen3.8-27B’s 29,451 glitch tokens have no legal equivalent path; FR rises from 85.4% for 1–3 paths to 98.7% for >50 paths.The 4–10-path group is the largest, at 39.87%, with 93.8% FR.Source: §6.4 · p. 9, Table 3 · p. 9

6.5 ABLATION STUDIES

  • FR rises from 71.83% at K = 1 to 87.50% at K = 8, then gains only 0.43 pp at K = 16 as build cost nearly doubles.The authors adopt K = 8 to balance repair effectiveness and evaluation budget; Table 4 reports candidate count on Qwen3.8-27B.Source: §6.5 · p. 9, Table 4 · p. 9

7 CONCLUSION

GLITCHPATCH repairs glitch-token behavior in publicly released language models without modifying model weights. The paper reports full-vocabulary improvements across ten models and frames the approach as a practical option for improving tokenizer reliability.

7 CONCLUSION

  • GLITCHPATCH reduces mean glitch rate from 14.88% to 2.27% across ten models, with 0.00% regression on normal tokens and negligible overhead.The conclusion presents this as input-level local ID substitution that requires no access to model internals.Source: §7 · p. 9
  • The experiments use open-weight models and publicly available benchmarks, and the method introduces no new training data.The paper states that the patch does not alter model capabilities beyond restoring intended tokenization.Source: ETHICS STATEMENT · p. 10

A DATA CONSTRUCTION, EVALUATION PROTOCOL, AND SCORING

The appendix specifies binary glitch-token classification across three behavioral tasks, model-wide evaluation settings, and a supplemental probe pool for ranking candidate paths.

A.1 GLITCH TOKEN DETECTION PROTOCOL

  • Tokens are labeled glitch if they fail any repetition, spelling, or length test; otherwise they are normal, with no partial-glitch category.Table A1 gives the task scoring rules.Source: §A.1 · p. 12, Table A1 · p. 12

A.2 BYTE-LEVEL AND CONTROL-TOKEN BOUNDARIES

  • When bytes cannot decode as valid Unicode, spelling and length are evaluated at byte level; partial UTF-8 tokens use their decodable portion as the target.Byte-level spelling requires byte-by-byte enumeration, and length uses the correct byte count.Source: §A.2 · p. 12

A.3 VOCABULARY SIZES AND INITIAL GLITCH RATES

  • 19.50% is Mistral-7B’s glitch ratio, versus 11.86% for Qwen3.8-27B; vocabulary size and glitch ratio are not simply proportional.Gemma-4-E4B has the largest vocabulary; Table A2 summarizes vocabulary sizes and initial glitch counts.Source: §A.3 · p. 12, Table A2 · p. 12

A.4 DECODING AND EXECUTION SETTINGS

  • Greedy decoding, a 64-token generation limit, BF16 precision, and seed 2027 make outputs deterministic for identical inputs on each model.Model-specific chat templates and text-only settings are also specified.Source: §A.4 · p. 12

A.5 SHARED EVIDENCE POOL.

  • Three supplemental probes per component token and adjacent pair estimate failure fractions using prompt wordings disjoint from selection, calibration, and final evaluation.Queried estimates are cached across targets; unqueried components and pairs need no estimate.Source: §A.5 · p. 13

B.1 AGGREGATE RESULTS ACROSS METHODS

This section reports the structural guarantee for untriggered inputs and specifies the propositions, graph search, adapters, and BPO evaluation procedure underlying the aggregate and per-model results.

B.1 AGGREGATE RESULTS ACROSS METHODS

  • 0.00% RR across all models follows because GLITCHPATCH substitutes only matched glitch-token IDs and leaves normal-token encodings unchanged.Table B1 covers aggregate results across all four methods, but its cell values are not supplied here.Source: §B.1 · p. 13

B.2 PER-MODEL COMPLETE COUNTS

  • Table B2 reports per-model complete counts, with FR, RR, and GR expressed as percentages.The supplied Gemma-31B entry identifies 36,045 glitch tokens and 226,099 normal tokens.Source: Table B2 · p. 13, §B.2 · pp. 13–14

C.1 LOCAL EQUIVALENCE (PROPOSITION 1)

  • Non-overlapping replacements preserve global bytes, special-token IDs, and decoder exit state when entry states match and local decoding transitions are equivalent.Adjacent spans that cannot be validated independently require joint validation; absent a joint rule, the original is retained, and the guarantee is limited to the adapter contract.Source: §C.1 · p. 15

C.2 PRESERVATION OF UNTRIGGERED INPUTS (PROPOSITION 2)

  • When repair leaves effective inputs unchanged under identical model settings, the conditional distribution—and deterministic greedy output—remains unchanged.For rule-unmatched inputs, the rule table contains only glitch-token IDs, so normal inputs retain their original IDs.Source: §C.2 · p. 15

C.3 GRAPH SEARCH GUARANTEE

  • The candidate graph is acyclic, and augmenting nodes with prior-token and length state enables budgeted top-K search with an O(BK|E′| log K) bound.The graph objective is only a proposal cost, not the model’s true conditional negative log-probability, so shortlisted paths need full-context behavioral evaluation.Source: §C.3 · p. 15

C.4 TOKENIZER ADAPTER CONTRACT; C.5 BPO PSEUDOCODE AND CANDIDATE PATH GRAPH

  • Adapter-specific byte extraction, decoder-state management, and leading-space handling support shared candidate-graph construction across tokenizer implementations.BPO’s offline procedure forms normal-token candidate paths, ranks and evaluates shortlisted replacements, then independently calibrates passing candidates before storing rules; Figure 7 depicts proxy ranking, full-sequence evaluation, and calibration.Source: §C.4 · pp. 15–16, §C.5 · p. 17

D.1 SELECTION PROBE STRUCTURE

  • Twelve selection probes use four prompt variants per GlitchQuiz dimension, followed by four independent calibration probes that do not participate in ranking or selection.Both stages use greedy decoding with a 64-token generation limit.Source: §D.1 · p. 17, Table D1 · p. 17

D.2 DEFAULT PARAMETERS; D.3 BASELINE ADAPTATION CONFIGURATIONS

  • Global parameters were selected on a Qwen3.8-27B development set and fixed for other models; baseline settings are specified separately, while GLITCHPATCH changes only input IDs.The selection objective maximizes net repair gain under limited sequence growth; the baseline configurations include GlitchCleaner training and GlitchEdit’s handling of tied embeddings.Source: §D.2 · pp. 17–18, Table D2 · p. 18, §D.3 · p. 18

D.4 PARAMETER AND COMPONENT ABLATIONS

The ablations identify balanced settings for path selection and token budget; proxy-score analysis and cost comparisons assess ranking quality and deployment overhead.

D.4 PARAMETER AND COMPONENT ABLATIONS

  • At γ = 0.05, coverage is 89.2% and FR is 87.50%; at γ = 0, FR rises to 90.23%, while at γ = 0.10 it falls to 80.86%.Moderate adjacency weighting also matters: FR is 81.73% at η = 0, 87.50% at η = 0.50, and 86.47% at η = 1.00.Source: §D.4 · pp. 18–19, Table D4 · p. 18, Table D5 · p. 19
  • At B = 4, FR is 82.48%; increasing B from 8 to 16 adds only 1.02 pp while mean token increase rises from 2.6% to 4.1%.At κ = 4, the stronger regression penalty slightly reduces FR by over-excluding candidates with positive net benefit.Source: §D.4 · p. 19, Table D6 · p. 19
  • Across three models, BPE-history-only search yields the lowest FR, removing the gate yields the highest classification FR but weaker behavioral guarantees, and the full method balances FR with reliability.The ablations use each model’s full glitch set.Source: §D.4 · pp. 18–19, Table D7 · p. 19

D.5 PROXY SCORE QUALITY ANALYSIS

  • Mean per-target Spearman correlation ρ(−J, U) ranges from 0.68 to 0.72, while increasing K from 8 to 16 improves FR by only 0.43 pp at nearly double candidate-evaluation cost.The correlation is computed for targets with at least two evaluated candidates.Source: §D.5 · p. 19, Table D8 · p. 19

E.1 PER-MODEL SERVING LATENCY

  • Mean end-to-end latency increase is 0.065%; added tokens affect prefill only, and offline build times range from 0.36 h to 2.08 h.The online overhead includes host-side rule matching, sequence reconstruction, and additional prefill computation on triggered requests.Source: §E.1 · pp. 19–20, Table E1 · p. 20

E.2 BUILD TIME COMPARISON

  • GLITCHPATCH’s average build time is 0.89 h, approximately 41% of GlitchProber’s 2.19 h, 21% of GlitchEdit’s 4.28 h, and 9% of GlitchCleaner’s 10.41 h.GlitchCleaner reaches 22.50 h on Llama-70B.Source: §E.2 · p. 20, Table E2 · p. 20

E.3 COMBINED COST SUMMARY

  • GLITCHPATCH has the shortest average offline build time (0.89 h) and keeps mean online latency increase below 0.1%, with no additional model parameters or calls.Across methods, its average build time is 0.89 h, while GlitchProber has the highest online latency increase (∼9.20%).Source: §E.3 · p. 20, Table E3 · p. 21

F.1 GENERAL BENCHMARKS

General-benchmark results show little aggregate change after repair, while failure analysis identifies unrepaired cases and contextual tests measure effects on adjacent normal-token content.

F.1 GENERAL BENCHMARKS

  • Across ten models, General Score ∆ ranges from −0.025 to +0.025 pp, with a mean change of only +0.008 pp and no systematic degradation.Correct-answer counts remain nearly unchanged across four general benchmarks.Source: §F.1 · p. 21, Table F2 · p. 22

F.2 FAILURE AND RETENTION CASES

  • For Qwen3.8-27B, 87.50% of 29,451 initial glitch tokens are repaired, 5.00% are structurally unrepairable, and 7.50% remain unrepaired despite feasible paths.Among the remaining cases, approximately 5.6% receive no stored rule because the best candidate does not exceed the default utility threshold.Source: §F.2 · p. 22, Table F3 · p. 22

H.1 CONTEXTUAL REGRESSION ANALYSIS

  • On repaired-token contextual tests, GLITCHPATCH’s accuracy changes stay within ±0.2 pp per model (mean −0.1 pp), and population-level changes remain within ±0.1 pp of zero.The study tests questions answered from normal-token content adjacent to a repaired glitch token.Source: §H.1 · pp. 23–24, Table H1 · p. 23

H.2 OUT-OF-DISTRIBUTION DOWNSTREAM EVALUATION

GLITCHPATCH is evaluated on three downstream tasks using uniformly sampled glitch tokens. It has the highest overall mean accuracy among the compared methods.

H.2 OUT-OF-DISTRIBUTION DOWNSTREAM EVALUATION

  • Each method is evaluated on the same 500 uniformly sampled glitch tokens per model, without conditioning on repair success.On sampled sets, GLITCHPATCH applies rules to 84–87% of tokens, versus 70–74% for GlitchCleaner, 50–55% for GlitchEdit, and 35–40% for GlitchProber.Source: §H.2 · p. 24
  • GLITCHPATCH’s overall mean accuracy is 70.0%, compared with 59.5% for GlitchCleaner, 53.5% for GlitchEdit, and 48.3% for GlitchProber.The results cover Entity QA, verbatim copy, and code-identifier completion.Source: §H.2 · p. 24, Table H2 · p. 24
  • Entity QA and verbatim copy show mean gains of +26.6 and +27.5 pp over the original, while code-identifier completion gains +16.3 pp.Verbatim copy tests exact reproduction of a full sentence containing a glitch token; code completion tests whether the model correctly references the identifier.Source: §H.2 · p. 24
Loading 2610.04399v1…