PaperlayerView exact revision on arXiv

Source-linked AI summary

Hot or Cold? Adaptive Temperature Sampling for Code Generation with Large Language Models

Yuqi Zhu, Jia Li, Ge Li, YunFei Zhao, Jia Li, Zhi Jin, Hong Mei

arXiv:2309.02772v3cs.SEcs.CL

TL;DR

Temperature sampling was designed for natural-language generation, leaving its suitability for code’s high-accuracy demands an open question. The paper analyzes code-token loss patterns and proposes Adaptive Temperature sampling, which varies temperature by token type; it reports improved pass@k on HumanEval and MBPP across model sizes, with up to 13.6% higher HumanEval pass@15 than the state-of-the-art method.

  • Problem

    Standard temperature sampling is less effective for code generation, a high-accuracy task, than for natural language with flexible syntax.

    Source: Introduction · p. 1

  • Method

    AdapT sampling analyzes token loss categories and dynamically uses higher temperature for challenging tokens and lower temperature for confident tokens.

    Source: Introduction · p. 2

  • Results

    AdapT improves pass@k on HumanEval and MBPP across LLM sizes from 2B to 13B, exceeding the state-of-the-art method by up to 13.6% on HumanEval pass@15.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    Token-level loss patterns and benchmark results provide a basis for further work on decoding strategies specialized to code generation.

    Source: Conclusion · p. 7

    Incorrect code logic remains AdapT’s most common error type, leaving room for improvement.

    Source: Code Quality Evaluation · p. 7

Abstract

from arXiv · show

Recently, Large Language Models (LLMs) have shown impressive abilities in code generation. However, existing LLMs' decoding strategies are designed for Natural Language (NL) generation, overlooking the differences between NL and programming languages (PL). Due to this oversight, a better decoding strategy for code generation remains an open question. In this paper, we conduct the first systematic study to explore a decoding strategy specialized in code generation. With an analysis of loss distributions of code tokens, we find that code tokens can be divided into two categories: challenging tokens that are difficult to predict and confident tokens that can be easily inferred. Among them, the challenging tokens mainly appear at the beginning of a code block. Inspired by the above findings, we propose a simple yet effective method: Adaptive Temperature (AdapT) sampling, which dynamically adjusts the temperature coefficient when decoding different tokens. We apply a larger temperature when sampling for challenging tokens, allowing LLMs to explore diverse choices. We employ a smaller temperature for confident tokens avoiding the influence of tail randomness noises. We apply AdapT sampling to LLMs with different sizes and conduct evaluations on two popular datasets. Results show that AdapT sampling significantly outperforms state-of-the-art decoding strategy.

Introduction

The paper studies decoding for code generation, where standard temperature sampling is limited by the difference between flexible natural-language syntax and code’s high accuracy demands. It identifies token-level loss patterns, proposes AdapT sampling, and reports improved pass@k results across two datasets and multiple model sizes.

Introduction

  • Standard temperature sampling is less effective for code’s high-accuracy demands than for natural language’s flexible syntax.The introduction gives CodeGeeX’s 36.0% Pass@15 on HumanEval as an example.Source: Introduction · p. 1
  • The authors present the work as the first systematic study of decoding strategies for code generation with LLMs.Source: Introduction · p. 1
  • Code tokens divide into high-loss challenging tokens and low-loss confident tokens; challenging tokens frequently appear at code-block beginnings.The analysis finds less variation in source-code loss values than in natural-language text.Source: Introduction · p. 2
  • AdapT dynamically adjusts temperature, using a higher value for challenging tokens and a lower value for confident tokens.The higher temperature encourages diverse choices; the lower temperature minimizes randomness noise.Source: Introduction · p. 2
  • AdapT exceeds the state-of-the-art temperature-sampling method by up to 13.6% on HumanEval pass@15.The reported pass@k improvements span HumanEval and MBPP and LLM sizes from 2B to 13B.Source: Introduction · p. 2
  • The introduction also identifies current challenges and proposes future research directions for code-generation decoding strategies.Source: Introduction · p. 2

Background

The background describes LLM-based code generation and surveys decoding strategies. It contrasts search and sampling approaches, then outlines temperature and Top-p sampling and their trade-offs.

Code Generation with LLMs

Decoding Strategy

  • Search methods select high-probability tokens but can produce conservative, repetitive, or empty outputs; sampling selects tokens randomly by predicted probability.Greedy search selects the most probable next token, while beam search retains the top B; sampling can alleviate empty sequences and repetitions.Source: Decoding Strategy · p. 2
  • Temperature sampling rescales logits before Top-p sampling; increasing temperature can improve exploration of correct answers while introducing more errors.Top-p sampling selects from the smallest token set whose cumulative probability reaches threshold p.Source: Decoding Strategy · p. 2

Analysis of the code generation process

The analysis finds that code tokens differ in prediction difficulty: challenging tokens are harder to predict, while confident tokens are easier to infer. Challenging tokens cluster at line beginnings, especially at code-block starts.

Loss Distribution Comparison

  • CodeGen-2B has higher mean loss, greater loss variation, more high-loss tokens, and higher perplexity on NL descriptions than on source code.The comparison used loss distributions for aligned NL descriptions and code snippets.Source: Loss Distribution Comparison · p. 3

Analysis of the code generation process

In-depth Study of Code Tokens

  • 69.0% (HumanEval), 73.1% (MBPP), and 71.9% (APPS): first-position tokens have the highest prediction difficulty across code-line positions.The study averages token difficulty by position within lines; Figure 3 plots position against prediction difficulty.Source: In-depth Study of Code Tokens · pp. 3–4, Figure 3 · p. 4
  • With H=0.9, 24.8% (HumanEval), 22.6% (MBPP), and 28.1% (APPS) of challenging tokens occur at line beginnings.The results support the assumption that challenging tokens tend to appear at the first position of each code line.Source: In-depth Study of Code Tokens · p. 4
  • 79.8% (HumanEval), 83.4% (MBPP), and 82.5% (APPS): code-block initial tokens have higher difficulty than other line-initial tokens.A Python code block begins at its first indented statement and continues until indentation returns to a previous level.Source: In-depth Study of Code Tokens · p. 4

AdapT sampling

AdapT sampling varies temperature by token type: it uses higher temperature for challenging tokens and lower temperature for confident tokens. Its parameters control sampling randomness.

AdapT sampling

  • AdapT uses high temperature for challenging tokens to increase diversity and low temperature for confident tokens to minimize randomness noise.The method adjusts the temperature coefficient for different next tokens.Source: AdapT sampling · p. 4
  • Temperature a is used for a code-block initial token and b otherwise, with a>b.The displayed rule routes temperature according to whether the sampled token is the code-block initial token.Source: AdapT sampling · p. 4
  • The temperature coefficient T and timestep t are used with hyperparameters a and b, which control sampling randomness.The passage specifies a,b ∈ [0, 1] and a>b.Source: AdapT sampling · p. 4

Experiments

Experiments compare AdapT with standard temperature sampling across three open-source models and two code-generation benchmarks. AdapT improves pass@k results, while analyses examine pass@1, temperature settings, hyperparameters, and generated-code errors.

Benchmarks

  • The evaluation uses HumanEval’s 164 Python problems and MBPP’s 500 problems, with generated function bodies checked against their respective unit tests.HumanEval prompts provide a natural-language requirement and function signature; MBPP problems include an English requirement, signature, and three test cases.Source: Benchmarks · p. 4

Main Results

  • On HumanEval, CodeGeeX-13B AdapT raises pass@15 from 36.0% to 40.9% (13.6% improvement), and AdapT outperforms SP on pass@5, pass@10, and pass@15 across HumanEval and MBPP.For CodeGeeX-13B, solved problems on HumanEval rise from 60 to 67; AdapT solves 14 previously unsolved HumanEval problems and 36 on MBPP.Source: Table 1 · p. 5, Table 2 · p. 5, Main Results · p. 6

Base Models

  • The experiments use CodeGen-2B, InCoder-6B, and CodeGeeX-13B, compare AdapT with SP, and run zero-shot with a 500-token generation limit.OpenAI models are excluded because their probability distributions are unavailable through API calls; pass@k measures functional correctness using test execution.Source: Base Models · p. 5, Baselines · p. 5, Implementation Details · p. 5

pass@1

  • On HumanEval, AdapT beats SP-best in 83.3% of pass@1 cases and reaches pass@1 comparable to greedy search.Greedy search samples one answer per question, while AdapT can sample n answers and increase the number of solved questions.Source: pass@1 · p. 6, Table 3 · p. 6

Compare with Different T Settings

  • AdapT significantly outperforms temperature sampling at all tested settings on pass@5, pass@10, and pass@15; it solves 67 questions versus greedy search’s 31.The SP comparison tested temperatures from 0 to 1 in 0.1 increments; results for T ≥ 1 are omitted because performance drops significantly.Source: Figure 4 · p. 5, Compare with Different T Settings · p. 6

Hyperparameters Analysis

  • With a fixed b = 0.3, increasing a from 0.2 to 0.9 raises pass@15 from 32.9% to 37.8%, a 14.9% improvement; a > b improves pass@5, pass@10, and pass@15.At fixed a = 0.8, b values 0.3, 0.5, 0.6, and 0.7 outperform SP (T = 0.8) across pass@1, pass@5, pass@10, and pass@15.Source: Hyperparameters Analysis · p. 6, Figure 5 · p. 7

Code Quality Evaluation

  • On HumanEval, AdapT yields 517, 333, and 382 correct samples for CodeGen-2B, InCoder-6B, and CodeGeeX-13B, respectively, exceeding SP by up to 17.2%, 8.8%, and 14.0%.Figure 6 compares generated-code quality across the three models.Source: Code Quality Evaluation · p. 6, Figure 6 · p. 7
  • AdapT produces fewer TypeError and SyntaxError instances; NameError falls by 44.3% for CodeGen and 23.4% for CodeGeeX, while incorrect logic remains the most common error.The authors attribute fewer syntax errors to the smaller T used inside code blocks, which improves sampled-code coherence.Source: Code Quality Evaluation · pp. 6–7

Future Work

The section identifies unresolved decoding challenges and proposes several directions for future work, including learning-based temperature adjustment, domain knowledge, and staged generation.

Future Work

  • Challenging tokens within statements lack a clear distributional pattern, and tested linear and exponential temperature schedules yielded no substantial improvement.The authors propose learning-based adjustment of temperature coefficients as future work.Source: Future Work · p. 7
  • Future decoding could incorporate domain knowledge, such as private code libraries and code specifications, to improve usability in real-world scenarios.Source: Future Work · p. 7
  • A multi-stage strategy could first generate a natural-language plan and then produce an executable program from it.The authors connect this direction to the difficulty of generating code from scratch and incorrect code logic.Source: Future Work · p. 7

Conclusion

The paper presents an initial systematic study of LLM decoding for code generation, analyzes code-token loss patterns, and proposes AdapT sampling. It reports effectiveness on code-generation datasets and identifies directions for further research.

Conclusion

  • The paper presents what it describes as the first attempt to study LLM decoding strategies specifically for code generation.Source: Conclusion · p. 7
  • The analysis categorizes source-code tokens as challenging or confident and finds that challenging tokens often occur at the beginning of a code block.Source: Conclusion · p. 7
  • AdapT dynamically adjusts the temperature coefficient during sampling and is reported effective on code-generation datasets.Source: Conclusion · p. 7
  • The paper presents challenges and insights for developing more advanced code-generation decoding strategies and identifies further exploration as future work.Source: Conclusion · p. 7

Hyperparameter Settings

The captions identify separate hyperparameter tables for the Main Result and Pass@1 sections.

Hyperparameter Settings

  • Table 4 lists hyperparameters used in the Main Result Section.Source: Table 4
  • The captions distinguish hyperparameter settings for the main results from those for Pass@1.Source: Table 5 · p. 10, Table 4
  • Table 5 lists hyperparameters used in the Pass@1 Section.Source: Table 5 · p. 10

Analysis of challenging tokens

The analysis finds that challenging tokens are more likely to occur at the beginnings of code lines as H increases, examining settings from 0.5 to 0.9 across three datasets.

Analysis of challenging tokens

Comparison with different T

Across all tested temperature settings, AdapT outperforms the SOTA method on every model. In the case study, only AdapT generates code that passes all test cases with correct logic.

Comparison with different T

  • Temperature comparisons report CodeGeeX-13B on MBPP, CodeGen-2B on HumanEval and MBPP, and InCoder-6B on HumanEval and MBPP, each evaluated with different T.Source: Comparison with different T · p. 10
  • AdapT outperforms the SOTA method in all T settings on all tested models.The paper says this exceeds the SOTA method’s upper bound.Source: Comparison with different T · p. 10

Case Study

  • In the Figure 9 case study, only AdapT generates code that passes all test cases and has correct code logic.The SOTA method fails to generate the correct answer under different T settings.Source: Case Study · p. 10, Figure 9 · p. 11
Loading 2309.02772v3…