PaperlayerView exact revision on arXiv

Source-linked AI summary

Astraios: Parameter-Efficient Instruction Tuning Code Large Language Models

Terry Yue Zhuo, Armel Zebaze, Nitchakarn Suppattarachai, Leandro von Werra, Harm de Vries, Qian Liu, Niklas Muennighoff

arXiv:2401.00788v1cs.CLcs.AIcs.SE

TL;DR

Evidence was limited on PEFT for instruction-tuned code LLMs across methods, scales, and evaluation dimensions. Astraios compares seven tuning methods across four model sizes and multiple code tasks. FFT generally leads downstream performance, while PEFT results vary by scale and LoRA usually offers a favorable cost–performance trade-off.

  • Problem

    Evidence on PEFT for instruction-tuned code LLMs is limited across tuning methods, domains, evaluation dimensions, and model scales.

    Source: §1 · p. 2

  • Method

    Astraios evaluates 28 instruction-tuned StarCoder models across seven tuning methods and four model sizes, on code tasks, robustness, and security.

    Source: §1 · p. 2, §2.1 · p. 4

  • Results

    FFT generally leads downstream performance across scales, PEFT efficacy varies with model size, and LoRA usually offers the most favorable cost–performance trade-off.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Relative loss performance across methods in small models can help predict larger-model performance, and lower test loss generally aligns with better downstream performance.

    Source: §6 · pp. 10–11

    The study does not vary Code LLM architecture, so its findings may not generalize to other architectures.

    Source: J Limitations and Future Work · p. 23

Abstract

from arXiv · show

The high cost of full-parameter fine-tuning (FFT) of Large Language Models (LLMs) has led to a series of parameter-efficient fine-tuning (PEFT) methods. However, it remains unclear which methods provide the best cost-performance trade-off at different model scales. We introduce Astraios, a suite of 28 instruction-tuned OctoCoder models using 7 tuning methods and 4 model sizes up to 16 billion parameters. Through investigations across 5 tasks and 8 different datasets encompassing both code comprehension and code generation tasks, we find that FFT generally leads to the best downstream performance across all scales, and PEFT methods differ significantly in their efficacy based on the model scale. LoRA usually offers the most favorable trade-off between cost and performance. Further investigation into the effects of these methods on both model robustness and code security reveals that larger models tend to demonstrate reduced robustness and less security. At last, we explore the relationships among updated parameters, cross-entropy loss, and task performance. We find that the tuning effectiveness observed in small models generalizes well to larger models, and the validation loss in instruction tuning can be a reliable indicator of overall downstream performance.

1 Introduction

Evidence was limited on which PEFT methods best serve instruction-tuned code LLMs across scales and evaluation dimensions. Astraios compares tuning methods and finds that FFT generally leads downstream performance, while PEFT effectiveness depends on scale.

1 Introduction

  • Prior PEFT studies leave instruction-tuned code LLMs underexplored across methods, domains, evaluation dimensions, and model scales.Earlier work often focused on task-specific models and limited downstream tasks, leaving transfer to instruction tuning unclear.Source: §1 · p. 2
  • Astraios comprises 28 instruction-tuned Code LLMs, tuned with seven methods on StarCoder models from 1B to 16B parameters.The evaluation covers five code tasks as well as robustness and output security.Source: §1 · p. 2
  • FFT generally leads downstream performance across scales, but the most competitive PEFT methods vary with model size; LoRA usually offers the best cost–performance trade-off.At 16B, Parallel Adapter and LoRA are most competitive with FFT; at 1B, P-Tuning and (IA)3 slightly outperform them.Source: §1 · p. 2
  • Larger PEFT models perform better on code generation, but this pattern does not appear for clone detection and defect detection.Increasing model size also accompanies vulnerabilities to adversarial examples and biases toward insecure code.Source: §1 · p. 3
  • Relative loss performance across tuning methods is consistent across model sizes, and lower final loss generally aligns with better downstream task performance.The authors suggest general instruction data may serve as a proxy for downstream performance.Source: §1 · p. 3

2 The ASTRAIOS Suite and Benchmark

Astraios uses StarCoder models, multiple tuning methods, and a shared instruction-tuning dataset to compare code models across task performance, robustness, and security. Its evaluation spans code comprehension, generation, perturbed inputs, and security scenarios.

2.1 Model

  • The suite uses StarCoder base models at 1B, 3B, 7B, and 16B parameter scales.The authors chose StarCoder to maximize transparency because some alternative models do not fully disclose training data.Source: §2.1 · p. 3
  • Six PEFT methods cover four adapter-based methods, one prompt-based method, and one intrinsic-rank method, alongside FFT at each model size.LoRA represents intrinsic-rank tuning, which assumes low-rank weight changes suffice for adaptation.Source: §2.1 · pp. 3–4

2.2 Instruction Tuning

  • Instruction tuning uses CommitPackFT+OASST; all models train for 200 steps, with different learning rates for PEFT and FFT.PEFT uses a 1 × 10−4 learning rate and 8-bit quantized models; FFT uses 1 × 10−6.Source: §2.2 · p. 4

2.3 Evaluation

  • Code comprehension is tested on BigCloneBench for clone detection and Devign for defect detection, using F1 and Accuracy, respectively.The evaluation samples 2,000 examples from each dataset.Source: §2.3 · p. 4
  • HumanEvalPack evaluates code synthesis, repair, and explanation on Python and Java splits using Pass@1.The setup samples 20 outputs per test example.Source: §2.3 · p. 4
  • Robustness is assessed with ReCode perturbations of HumanEval inputs, while security is evaluated with AATK scenarios focused on diversity of weakness.The security analysis examines 40 automatically testable scenarios across C and Python, covering 18 CWEs.Source: §2.3 · pp. 4–5

3 Preliminary Study: Cross-Entropy Loss

This preliminary study examines how model size and training time relate to instruction-tuning loss in PEFT Code LLMs. Loss generally falls as models grow and training continues, with method-specific differences in stability and scaling.

3 Preliminary Study: Cross-Entropy Loss

  • Prior loss-scaling studies mainly examined pretraining or FFT; this study varies model size and training time for PEFT, but not training-data quantity.Training time is measured in global steps, with a maximum of 200 steps.Source: §3 · p. 5
  • Train and test loss both strictly decrease as model size grows from 1B to 16B; FFT has slightly better loss than LoRA and Parallel Adapter.Train and test loss are well aligned, which the authors interpret as evidence that the models are not overfitted.Source: Figure 3 · p. 5, §3 · p. 6
  • Test loss generally continues to fall with longer training, but P-Tuning is non-monotonic, (IA)3 changes insignificantly, and FFT tends to decline beyond 200 steps.PEFT methods do not show a clear decreasing trend after 200 steps.Source: §3 · p. 6, Figure 4 · p. 6

4 Main Results: Task Performance

Across code comprehension and generation tasks, the results compare tuning methods and model scales, with FFT generally leading overall. Performance scaling differs between comprehension and generation, while parameter efficiency does not always track performance.

Evaluation

  • The evaluation examines five representative downstream tasks because cross-entropy loss may not reliably proxy performance across source-code tasks.The tasks are defect detection, code clone, code synthesis, code repair, and code explanation.Source: §4 · pp. 6–7

Code Comprehension

  • Both comprehension tasks show inverse scaling across tuning methods, unlike the patterns observed for code generation.The authors hypothesize that limited task-specific training data may constrain generalization to unseen defects and cloned code.Source: §4 · p. 7, Table 2 · p. 7, Figure 5 · p. 7

Code Generation

  • FFT gains consistently as model size increases, and PEFT methods also show reasonable scalability across the six HumanEvalPack benchmarks.At 1B and 3B, models typically underperform in repair versus synthesis; at 7B and 16B, performance across those tasks becomes more comparable.Source: §4 · p. 7, Table 3 · p. 8

Overall Performance

  • FFT ranks best overall, while LoRA and Parallel Adapter are comparable to it; most PEFT methods still show a large performance gap.The overall comparison uses mean cumulative task scores for each tuning method and model size.Source: §4 · p. 8
  • (IA)3 performs reasonably while updating far fewer parameters, and AdapterP outperforms AdapterH despite updating fewer parameters.The authors conclude that AdapterH may not be worth deploying in practice.Source: §4 · p. 8

5 Further Analysis

The further analysis examines robustness and generated-code security across tuning methods and model sizes. It reports scale-related robustness and security concerns alongside differences among tuning methods.

5 Further Analysis

  • The analysis treats robustness and generated-code security as measures of real-world practicality beyond downstream task performance.The security experiment examines how tuning methods affect secure-code generation on the AATK benchmark.Source: §5 · p. 8, §5.2 · p. 10

5.1 Model Robustness

  • RC@1 rises as models get larger, indicating reduced robustness; all models perform worst under syntax transformation.FFT may not lead RP@1 on 1B and 3B models, but has the best RP@1 at 16B.Source: Table 4 · p. 8, §5.1 · p. 9, Figure 7 · p. 9
  • Adapter-based tuning appears more robust than FFT and LoRA while maintaining performance comparable to FFT.This comparison uses mean RC@1; lower RC@1 indicates better robustness.Source: §5.1 · p. 9, Figure 7 · p. 9

5.2 Code Security

  • Larger models are more likely to generate insecure code, while FFT and LoRA have better security levels than other tuning methods.P-Tuning may generate less secure programs, a concern for security-sensitive deployment.Source: Table 5 · p. 9, §5.2 · p. 10

6 Discussion

The discussion examines whether updated parameters and instruction-tuning loss can help predict model behavior. Relative loss differences across tuning methods persist across model sizes, and lower loss generally accompanies better downstream performance.

Discussion

  • The analysis asks whether updated parameters relate to cross-entropy loss and whether loss can predict downstream task performance.Source: §6 · p. 10

Updated Parameters and Loss

  • Relative loss performance across tuning methods is consistent across model sizes, making small-model loss a useful predictor for larger models.The analysis finds no consistent pattern between the number of updated parameters and final loss across model sizes.Source: §6 · p. 10, Figure 8 · p. 10

Loss and Downstream Performance

  • Lower instruction-tuning loss generally accompanies better downstream task performance, with a stronger pattern for test loss than train loss.The authors suggest general instruction data can serve as a proxy for downstream tasks, although the instruction data is not directly aligned with them.Source: §6 · pp. 10–11, Figure 9 · p. 11

7 Related Work

Related work covers code language models, fine-tuning, and model scaling, and identifies parameter-efficient instruction tuning as an underexplored area.

Code Large Language Models

  • Prior work includes base models mostly targeting code completion, task-specific fine-tuning, and instruction tuning for generalization across tasks.Because full-parameter tuning is costly, many instruction-tuned models have used LoRA; this paper asks how it compares with other methods.Source: §7 · p. 11

Model Analysis Across Scales

  • Earlier scaling analyses focused on pre-training or task-specific full-parameter tuning, leaving parameter-efficient instruction tuning unexamined.The paper studies this gap in Code LLMs.Source: §7 · p. 11

8 Conclusion

ASTRAIOS studies parameter-efficient instruction tuning of Code LLMs through a multi-scale model suite, evaluating tuning methods and their downstream behavior.

8 Conclusion

  • The study characterizes tuning methods on downstream tasks, robustness, output security, and relationships among updated parameters, cross-entropy loss, and task performance.Source: §8 · p. 11
  • The authors hope these analyses inspire follow-up work on understanding tuning mechanisms and developing new approaches.Source: §8 · p. 12

A What is ASTRAIOS?

  • ASTRAIOS comprises 28 instruction-tuned StarCoder models spanning seven PEFT methods and four sizes, up to 16B parameters.Source: A What is ASTRAIOS? · p. 18

B Artifacts

  • Table 6 catalogs artifacts used and produced by the work.Source: Table 6 · p. 18

C Contributions

  • The contribution statement assigns model training, evaluation, analysis, advising, and editing roles across the authors.Source: C Contributions · p. 19

D Instruction Tuning

  • All instruction-tuning experiments use A100 80G GPUs; PEFT strategies train with 8-bit quantized bases, while FFT uses unquantized base models.Source: D Instruction Tuning · p. 19

E Evaluation Setup

The evaluation settings vary by benchmark, including distinct generation limits and decoding setups.

E Evaluation Setup

F Failure of Scaling

The scaling analysis finds that some PEFT methods initially exhibit rising final loss with model size, and that learning-rate changes alter this pattern. One-shot demonstrations flatten defect-detection trends but do not flatten all clone-detection trends.

F Failure of Scaling

  • For Prompt Tuning and AdaLoRA, final loss initially rose with model size; higher learning rates aligned their results with the other methods’ findings.Prompt Tuning worked well at 3 × 10^-2; AdaLoRA worked better at 3 × 10^-2 and 1 × 10^-3.Source: Figure 11 · p. 20, F Failure of Scaling · p. 20

G Visualization on HumanEvalPack

  • Figure 12 presents ASTRAIOS models’ Pass@1 results on HumanEvalPack.Source: Figure 12 · p. 21

H Mitigating Inverse Scaling

  • With one-shot examples, all PEFT strategies’ defect-detection patterns became flatter, but LoRA and FFT clone-detection performance fell at 15B after rising through 7B.The authors hypothesize that scaling only to 15B may be insufficient to show an increasing pattern for LoRA and FFT on clone detection.Source: Figure 13 · p. 22, Figure 14 · p. 22, H Mitigating Inverse Scaling · p. 22

I Further Discussion

The discussion measures correlations among trainable parameters, final loss, and downstream performance. It reports moderate parameter-loss correlations and stronger aggregate loss-performance correlations than size-level alignment.

I Further Discussion

  • Trainable parameters show moderate correlations with final loss, stronger for train loss than test loss, and correlations decrease slightly when statistics are aggregated across sizes.Table 7 reports correlations between trainable parameters and final loss; p-values are shown in gray.Source: Table 7 · p. 22, I Further Discussion · p. 22
  • For downstream performance, 1B models align poorly with final loss, while larger models correlate more strongly with train and test loss; aggregated correlations are stronger than size-level predictions.The authors suggest 1B models may lack sufficient capability to learn instructions and identify general instruction-data loss as a useful proxy for downstream task performance.Source: Table 8 · p. 23, I Further Discussion · p. 23
  • Correlations among updated parameters and cumulative task scores are 0.3016 for Kendall, 0.4128 for Pearson, and 0.4138 for Spearman.Source: I Further Discussion · p. 23

J Limitations and Future Work

The appendix identifies limits on the study’s evaluation and scope, describes its prompt design, and records the project timeline.

J Limitations and Future Work

  • Each task was evaluated in a single run because of budget constraints, leaving results susceptible to experiment noise; the authors call for multiple runs.They say repeated runs are needed to establish more robust conclusions and assess variance and reliability.Source: J Limitations and Future Work · p. 23
  • Shared training configurations may disadvantage hyperparameter-sensitive methods such as Prompt Tuning, while optimizing each method separately may cost more than FFT.The authors adopt library and framework defaults and note that other practical benchmarking approaches warrant investigation.Source: J Limitations and Future Work · p. 23
  • Limited computation left newer PEFT strategies out of ASTRAIOS, so its comparisons cover selected methods rather than the full strategy space.The authors made their source code, data, and models publicly available.Source: J Limitations and Future Work · p. 23
  • The study does not verify data scaling for PEFT, despite observing monotonically increasing performance with model size and training time.The authors recommend further verification of data scaling.Source: J Limitations and Future Work · p. 23
  • The study varies model size but not Code LLM architecture, so findings are not tested on encoder-decoder models.The authors suggest observations may transfer to other GPT-based models, while noting the possibility of limited generalization to encoder-decoder models such as CodeT5 and CodeT5+.Source: J Limitations and Future Work · pp. 23–24
  • A scaling law generally requires more than 100 models, and downstream task performance is harder to predict because it is noisier than final loss.The authors describe predicting overall task performance as very challenging.Source: J Limitations and Future Work · p. 24

K Prompts

  • For tasks not designed for instruction-tuned Code LLMs, the authors define instructions and use unoptimized prompts from model authors or intuitive formats.The instructions aim to elicit correct behavior and parseable outputs; prompt examples cover Devign, BigCloneBench, HumanEvalPack, ReCode, and Asleep At The Keyboard.Source: K Prompts · p. 24, Figure 15 · p. 24, Figure 17 · p. 24

L Timeline

  • The project progressed from experiment design, training, and evaluation in September 2023 to paper finalization and codebase construction in December.October included training and evaluation discussion; November included results discussion and paper writing.Source: L Timeline · p. 25
Loading 2401.00788v1…