PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code

Nadezhda Chirkova, Sergey Troshin

arXiv:2308.00683v1cs.LGcs.CLcs.SE

TL;DR

Subtokenization is often overlooked in source-code language-model pretraining, even though code’s complex identifiers make its design consequential. This paper studies code-specific subtokenization choices and finds that punctuation grouping shortens sequences without a downstream performance drop, while other choices can improve performance.

  • Problem

    Subtokenization is usually not paid much attention in code-LM pretraining, despite complex identifiers and the need to balance frequent subtokens with reasonable sequence lengths.

    Source: §1 · p. 1

  • Method

    The paper empirically studies code-specific subtokenization options for large language-model pretraining, using PLBART as a testing ground.

    Source: §1 · p. 2

  • Results

    17% shorter sequences without downstream performance drop; the most effective subtokenization improves performance by 0.5–2%, significantly in three of eight tasks and by one standard deviation in two others.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors recommend punctuation-combination CodeBPE or CodeUnigramLM and suggest considering smaller vocabularies when releasing models.

    Source: §8 · p. 9

    The study considers only PLBART, focuses on source code, and compares only BPE and UnigramLM.

    Source: §8 · p. 9

Abstract

from arXiv · show

Recent works have widely adopted large language model pretraining for source code, suggested source code-specific pretraining objectives and investigated the applicability of various Transformer-based language model architectures for source code. This work investigates another important aspect of such models, namely the effect of different subtokenization options, and aims at identifying most effective and length-efficient subtokenizations, taking into account code specifics. We propose subtokenziation that reduces average length by 17% without downstream performance drop, and show that a carefully chosen subtokenization may improve quality by 0.5-2%, possibly with some length increase.

1 INTRODUCTION

Subtokenization is an underexamined choice in source-code LMs, where complex identifiers make the balance between frequent subtokens and reasonable sequence length especially relevant. The paper studies code-specific options and reports shorter sequences without downstream quality loss, alongside performance gains for some choices.

1 INTRODUCTION

  • Subtokenization splits long tokens into smaller units, balancing frequent subtokens against reasonable sequence length while accommodating complex code identifiers.Identifiers such as variable and function names can have unrestricted complexity.Source: §1 · p. 1
  • Grouping punctuation chars into single tokens reduces average length by 17% without downstream performance drop; more complex composite tokens can reduce length by 40%, sometimes with quality loss.The punctuation-grouping approach is called CodeBPE or CodeUnigramLM, depending on the subtokenization algorithm.Source: §1 · p. 2
  • The study also finds UnigramLM generally preferable to BPE, smaller vocabularies may improve quality with 3–19% length increases, and subtokenizers transfer well between programming languages.These findings are among the paper’s stated contributions.Source: §1 · p. 2
  • 17% shorter sequences without downstream performance drop, and 0.5–2% performance improvements from the most effective subtokenization, are the headline findings.The improvements were significant in three of eight tasks and reached one standard deviation in two others.Source: §1 · p. 2

2 METHODOLOGY AND EXPERIMENTAL SETUP

The study modifies a commonly used code-subtokenization baseline along five dimensions and evaluates the resulting PLBART models on code tasks. The setup specifies the baseline, training data, task suite, and sequence clipping procedure.

2 METHODOLOGY AND EXPERIMENTAL SETUP

  • The experiments vary composite-token complexity, subtokenization algorithm, vocabulary size, training-language set, and stochastic subtokenization, changing one dimension at a time from the baseline.Each modified subtokenization is used to pretrain PLBART.Source: §2 · p. 3
  • PLBART uses a 6-layer encoder and decoder, 768-dimensional model, 12 heads, and 140M parameters, pretrained for 100k updates on Python, Java, and StackOverflow sequences.The corpus comprises 230M Python functions, 470M Java functions, and 47M natural-language descriptions.Source: §2 · p. 3
  • Evaluation covers code generation, code summarization, clone detection, and code translation, including translation scored with CodeBLEU or computational accuracy.The tasks include both generative and discriminative evaluations on Python or Java data.Source: §2 · p. 3
  • Sequences are clipped to 510 subtokens, or 250 for summarization, while reported average lengths are measured before clipping.The paper reports that clipping leaves most sequences unclipped, with rates varying by data split and task.Source: §2 · p. 3
  • The baseline uses SentencePiece UnigramLM with a 50K vocabulary, 99.99% character coverage, and a strategy that prohibits composite tokens.Its training sample contains 10M functions and natural-language descriptions; the setup also uses PLBART preprocessing and permits underscores inside tokens.Source: §2 · p. 3
  • Table 1 defines levels of allowed composite-token complexity, while Figure 2 reports results across granularity levels averaged over four finetuning runs.Level 0 is the baseline; Figure 2’s numerical plot data is in the appendix.Source: Table 1 · p. 4, Figure 2 · p. 4

3 SUBTOKENIZATION GRANULARITY

The section tests how composite-token complexity trades sequence length against downstream performance. Punctuation-only merges offer 17% shorter sequences without performance loss, while more complex merges yield larger savings with task-dependent quality changes.

3 SUBTOKENIZATION GRANULARITY

  • Unrestricted Level 4 cut average length 40%, but performed significantly worse on translation and generation; code-understanding outcomes ranged from similar to significantly better.Because Level 4 lost quality on several tasks, the authors considered intermediate levels.Source: §3 · p. 4
  • 17% shorter average sequences (97 to 80.7) with no performance drop in all tasks came from punctuation-only composites (Level 1).The approach is called CodeBPE or CodeUnigramLM.Source: §3 · pp. 4–5
  • Level 2 dot-text merges cut average length 23% versus Level 0, with similar-or-better results in six tasks but losses in Java generation and Python summarization.The summarization loss was marginal; the Java code-generation loss was significant.Source: §3 · p. 5
  • Level 3 generally performed better than Level 4 but worse than Level 2 in six tasks, and similarly in two.The two similar-performing tasks were generation and clone detection.Source: §3 · p. 5
  • Manual analysis found some inaccurate Level 3 outputs repeated array elements or changed code structure, yet often preserved code semantics.Examples include repeated traversal and more concise equivalent code forms.Source: §3 · p. 5
  • Level 1 produced 1,239 syntactically correct translation snippets versus 1,226 for Level 0; Level 3 produced 1,163, and higher granularity lowered input-output subtoken overlap.The authors suggest lower overlap may help explain why lower-granularity subtokenizations perform better.Source: §3 · p. 5
  • In Java-Python translation-1, generated sequences were 15% (Level 1) and 40% (Level 3) shorter than Level 0, versus ground-truth reductions of 13.5% and 50%.Autoregressive decoding is described as the slowest part of the encoder-decoder pipeline.Source: §3 · p. 5
  • Sequence clipping can explain rare cases where lower-granularity subtokenizations perform worse; fairer cropping made Level 1 and Level 0 similar in code summarization.The authors note that clipping lower-granularity sequences more strongly does not devalue their overall findings.Source: §3 · p. 6

4 SUBTOKENIZATION ALGORITHM

The section compares UnigramLM with BPE for source-code pretraining and examines how their identifier splits differ. UnigramLM performs similarly or better across tasks and aligns more closely with native identifier boundaries.

4 SUBTOKENIZATION ALGORITHM

  • UnigramLM outperformed BPE by one standard deviation in three tasks and had very similar performance in the remaining tasks; their average sequence lengths were similar.The authors recommend UnigramLM for source code.Source: §4 · p. 6
  • For identifiers tokenized differently by the two methods, UnigramLM more closely matched CamelCase or snake_case splits, with higher Jaccard similarity than BPE.On 150K Python identifiers, scores were 26.6% versus 15.2%; after splitting tokens at native boundaries, they were 55.2% versus 47.9%.Source: §4 · pp. 6–7
  • BPE more often detached a single uppercase letter or merged multiple native subtokens than UnigramLM in the sampled identifiers.BPE had at least one single uppercase letter in 14.6% of tokenizations versus 11.8% for UnigramLM; at least one _X subtoken occurred in 4.4% versus 1.4%.Source: §4 · p. 7

5 VOCABULARY SIZE

The section tests whether smaller vocabularies improve downstream quality despite longer sequences. Smaller vocabularies sometimes improve task performance, but the effects vary by task and come with sequence-length increases.

5 VOCABULARY SIZE

  • UnigramLM 10K significantly outperformed BPE 50K in three tasks and by one standard deviation in two others, while smaller vocabularies had task-dependent effects versus 50K.Against 50K, 10K and 2K were similar in code translation; smaller vocabularies outperformed in code summarization, while decreasing vocabulary size deteriorated clone detection quality.Source: §5 · p. 7
  • With 50K versus 10K and 2K vocabularies, sequences were shorter by 9.5% and 33%, while model sizes were 139M, 108M, and 102M, respectively.The authors conclude that smaller vocabularies may improve quality but elongate sequences.Source: §5 · p. 7
  • With 10K versus 50K, sequence length increased 6% in code generation and 19% in summarization, but only 3.5% in code-translation-1.The authors relate smaller-granularity identifiers to composing API names from subtokens in identifier-centered generation and summarization data.Source: §5 · p. 7

6 TRANSFERABILITY BETWEEN PROGRAMMING LANGUAGES

The baseline subtokenizer appears broadly transferable between Python and Java: most subtokens are frequent in both languages, and using a Python-only subtokenizer causes modest length and quality changes on Java.

6 TRANSFERABILITY BETWEEN PROGRAMMING LANGUAGES

  • 12.6% of baseline Level 0 subtokens are frequent in one language and rare in the other, versus 20.1% for Level 4 vocabulary.Although composite tokens occupy almost half of Level 4 vocabulary, the remaining 30% are common to both languages.Source: §6 · p. 8
  • 6.5% longer Java sequences result from a Python-only subtokenizer, while its Python sequences shorten marginally versus the all-data baseline.At Level 4, Java sequences are 13% longer, but average length is still 79 versus 83 for Level 1 Python-only sequences.Source: §6 · p. 8, Figure 5 · p. 8
  • Using the Python-only subtokenizer changes Java quality by up to one standard deviation in either direction; the baseline is judged usable across languages.PLBART was pretrained on all languages in this comparison; the authors conclude the baseline can be used on other programming languages with small length increase and slight quality change.Source: §6 · p. 8

7 RELATED WORK

Prior work explores subtokenization algorithms, vocabulary granularity, and stochastic approaches in NLP, while code models largely use BPE-like methods with large vocabularies. The authors identify limited experimental analysis of code subtokenization choices, especially composite-token complexity.

Subtokenization studies for NLP

  • NLP studies compare open-vocabulary methods, granularity, stochastic subtokenization, and shared models across languages, and this work investigates most of these directions for code.UnigramLM was reported to outperform BPE for language-model pretraining; small vocabularies outperformed large ones by up to 4 BLEU in Transformer machine translation.Source: §7 · pp. 8–9

Subtokenization practices in neural source code processing

  • Most source-code language models use BPE-like subtokenization with large vocabularies, but prior work lacks in-depth comparisons of code subtokenization options and composite-token complexity.The concurrent work uses unrestricted composite tokens without comparing them to other subtokenizations; this paper relies on commonly used subtokenization rather than separately mined idioms.Source: §7 · p. 9

8 CONCLUSION

The study offers reference experiments on how subtokenization choices affect source-code pretraining and recommends punctuation grouping, UnigramLM, and considering smaller vocabularies. Its main scope limitation is evaluation only with PLBART.

8 CONCLUSION

  • The study's value is reference experiments showing which subtokenization choices merit attention and which may bring slight improvements.The authors emphasize composite tokens as a direction for closer practical study, with vocabulary size and algorithm less important to experiment with.Source: §8 · p. 9
  • 17% shorter sequences without quality drop are reported for punctuation grouping, while UnigramLM-10K is 0.5–2% more effective than BPE 50K in 5/8 experiments with 3.5–19% length increase.The authors recommend UnigramLM when changing algorithms is easy and suggest considering models with smaller vocabularies.Source: §8 · p. 9
  • The study considers only PLBART because computational resources were limited.It also focuses on source code and compares only BPE and UnigramLM, leaving code-and-natural-language subtokenization and other algorithms for future study.Source: §8 · p. 9

BROADER IMPACT

The authors anticipate no direct negative social impact, but note possible environmental impact from computation and potential use in developing source-code language models.

BROADER IMPACT

  • The experiments used approximately 5K Tesla A-100 and 4K Tesla V-100 GPU hours, which the authors identify as a possible environmental impact.They anticipate no direct negative social impact, while noting potential downstream use in pretrained source-code models and referring to concerns such as over-reliance on generated code or vulnerable code.Source: BROADER IMPACT · p. 10

A AVERAGE LENGTHS ON THE FINETUNING DATA

On finetuning data, programming-language sequence lengths follow the same broad patterns as pretraining data, while natural-language lengths respond differently to subtokenizer design.

A AVERAGE LENGTHS ON THE FINETUNING DATA

B ADDITIONAL EXPERIMENTS WITH BPE

The BPE experiments compare Level 0 and Level 1 at 50K vocabulary with BPE at 10K. Level 0 and Level 1 perform similarly, while the 10K model has task-specific advantages over 50K.

B ADDITIONAL EXPERIMENTS WITH BPE

C ADDITIONAL ANALYSIS

The additional analyses test cropping that equalizes the character spans represented by different subtokenizations, examine input–output subtoken overlap, and compare subtoken frequency profiles. Results show that cropping can alter performance comparisons, while overlap and frequency patterns vary across subtokenization choices and tasks.

C ADDITIONAL ANALYSIS

  • Fair cropping may amplify performance differences: compact subtokenizations can worsen, while less compact ones may be unaffected.The main experiments use equal subtoken limits, matching common practice; most examples are not clipped, but long examples can lose unequal amounts of information.Source: C ADDITIONAL ANALYSIS · pp. 14–16
  • After cropping, Level 1 and Level 0 perform similarly in Java summarization; the authors attribute Level 1’s main-text advantage to less clipping.Source: C ADDITIONAL ANALYSIS · p. 16
  • With character-span-matched cropping in three downstream tasks, higher-granularity subtokenizations often lose or retain performance, while some improve slightly.Level 4 improves slightly in all three tasks; Level 0 improves slightly in clone detection.Source: Figure 7 · p. 16, C ADDITIONAL ANALYSIS · p. 16
  • The input–output textual-subtoken Jaccard analysis uses fair cropping; overlap generally falls as granularity rises, including across 2K, 10K, and 50K vocabularies.The authors relate smaller vocabularies’ greater overlap to more repeated word parts and note that this correlates with their generally slightly better performance.Source: C ADDITIONAL ANALYSIS · pp. 16–17, Table 5 · p. 17
  • Level 1 has slightly higher input–output intersection than Level 0; punctuation combinations use 3.4% of the vocabulary, leaving less capacity for textual tokens.The authors suggest spending a length budget on identifier splitting rather than separate punctuation tokens.Source: C ADDITIONAL ANALYSIS · p. 17
  • For Levels 2–4, overlap reflects competing effects of repeated identifier pieces and rarer composite tokens, with Level 2 patterns varying by task.Level 2 overlap is higher than Levels 0 and 1 in translation and summarization, but lower in generation; Levels 3 and 4 show lower overlap overall.Source: C ADDITIONAL ANALYSIS · p. 17
  • UnigramLM’s input–output intersection is slightly higher than BPE’s in summarization and generation, and equal in translation.The authors connect this to UnigramLM’s closer alignment with native subtokenizations and suggest that repeated subtokens may be easier to predict.Source: C ADDITIONAL ANALYSIS · p. 17
  • UnigramLM has a slightly heavier frequency tail than BPE, while Level 0 and Level 1 frequency profiles are close; Level 4 has lower left-range and higher right-range frequencies.The authors hypothesize that high frequencies among basic subtokens support performance; they report that Level 4’s right-range pattern does not correspond to a performance advantage.Source: C ADDITIONAL ANALYSIS · p. 17, Figure 8 · p. 18
Loading 2308.00683v1…