Source-linked AI summary
Less is More: DocString Compression in Code Generation
Guang Yang, Yu Zhou, Wei Cheng, Xiangyu Zhang, Xiang Chen, Terry Yue Zhuo, Ke Liu, Xin Zhou, David Lo, Taolue Chen
TL;DR
Redundant DocStrings can increase code-generation inference and API costs, while the effectiveness of prompt compression for code generation remains a concern. The paper introduces ShortenDoc, which adapts compression to token importance, and reports better preservation of essential information and lower inference costs than baselines across its evaluations.
Problem
Redundant DocStrings can increase inference costs, creating a need to improve code-generation efficiency and reduce resource and API costs.
Source: §1 · pp. 2–3Method
ShortenDoc dynamically adjusts DocString compression according to token importance, aiming to preserve essential information while removing redundant content.
Source: §1 · p. 3Results
ShortenDoc showed stronger preservation of essential information and reduced inference costs than baseline methods across six datasets and six LLMs, with additional evaluations across four languages and in a human study.
Source: §1 · p. 4Takeaways & Limitations
Reducing input tokens can lower third-party API costs, with end-to-end savings depending on the relative prices of input and output tokens.
Source: §1 · p. 3The authors identify dataset and model selection as an external-validity threat to the study.
Source: §6.4 · p. 25
Abstract
from arXiv · showhide
The widespread use of Large Language Models (LLMs) in software engineering has intensified the need for improved model and resource efficiency. In particular, for neural code generation, LLMs are used to translate function/method signature and DocString to executable code. DocStrings which capture user re quirements for the code and used as the prompt for LLMs, often contains redundant information. Recent advancements in prompt compression have shown promising results in Natural Language Processing (NLP), but their applicability to code generation remains uncertain. Our empirical study show that the state-of-the-art prompt compression methods achieve only about 10% reduction, as further reductions would cause significant performance degradation. In our study, we propose a novel compression method, ShortenDoc, dedicated to DocString compression for code generation. Our extensive experiments on six code generation datasets, five open-source LLMs (1B to 10B parameters), and one closed-source LLM GPT-4o confirm that ShortenDoc achieves 25-40% compression while preserving the quality of generated code, outperforming other baseline methods at similar compression levels. The benefit of this research is to improve efficiency and reduce the cost while maintaining the quality of the generated code, especially when calling third-party APIs, and is able to reduce the token processing cost by 25-40%.
1 INTRODUCTION
The introduction motivates DocString compression as a way to improve code-generation efficiency without losing essential requirements. It presents ShortenDoc as an adaptive approach and reports evaluations across datasets, models, languages, and a human study.
1 INTRODUCTION
- Long, redundant DocStrings increase inference costs with prompt length, while shorter API inputs can reduce financial costs at scale.The paper notes that input-token pricing varies across providers and relative to output-token pricing.Source: §1 · pp. 2–3
- The study also presents DocString compression as complementary to other efficiency techniques and as a foundation for exploring more complex retrieval-augmented code generation.The authors summarize contributions as empirical analysis of feasibility and limitations, ShortenDoc, and insights into compression techniques.Source: §1 · pp. 3–4
- Existing prompt-compression methods risk significant code-quality loss beyond 10% compression and require manual ratios that are difficult to adapt across scenarios.The authors attribute information loss in part to failure to extract code-related semantics.Source: §1 · p. 3
- ShortenDoc dynamically adjusts compression using token importance, preserving essential information while removing redundancy rather than applying a fixed ratio.Its process preprocesses the DocString, scores and ranks tokens, constrains the candidate search space, then iteratively compresses eligible tokens.Source: §1 · p. 3
- ShortenDoc was evaluated across six datasets and six LLMs, with additional tests across four programming languages and a human study of informativeness and comprehensibility.The authors report stronger preservation of essential information and lower inference costs than baseline methods.Source: §1 · p. 4
2 BACKGROUND
The background frames code generation as mapping signatures and DocStrings to executable code. It describes decoder-based generation as sequential token prediction and prompt compression as shortening inputs while retaining critical information.
2 BACKGROUND
- Code generation maps signatures and natural-language DocStrings to executable code, formalized as a function from T × D to Y.Signatures specify function or method details such as parameters and return type; DocStrings describe purpose, behavior, parameters, and returns.Source: §2 · p. 4
- A decoder model tokenizes the signature and DocString, transforms their embeddings through Transformer layers, and maps the final hidden state to next-token probabilities.The model generates code sequentially, conditioning each token on the inputs and previously generated tokens.Source: §2 · pp. 4–5
- Prompt compression seeks a shorter prompt that retains critical information, reducing input length for code generation.Source: §2 · p. 5
3 EMPIRICAL STUDY
The study evaluates existing prompt compressors for code generation and finds that modest compression can preserve or improve Pass@1, while stronger compression often degrades it. Results vary by dataset, and the study examines which tokens the methods remove.
3.1 Experiment Setup
- The study evaluates six diverse datasets with five language models and measures code-generation Pass@1 alongside prompt Compression Ratio.The datasets include HumanEval and two EvoEval variants, MBPP, CodeHarmony, and BigCodeBench; the models include four coding-specialized models and Llama3.1.Source: §3.1 · pp. 5–7
3.2 Empirical Findings
- At 10% compression, CodeGeeX4 on HumanEval reaches 62.20% Pass@1 with Selective_Context and 62.80% with LLMLingua2, versus 60.37% originally.Most models maintain comparable or improved Pass@1 at this compression rate.Source: Table 1 · p. 8, §3.2 · pp. 8–9
- As compression increases, Pass@1 degradation becomes evident; on BigCodeBench, most models decline, and outcomes vary with dataset characteristics.At 40% compression on HumanEval, CodeGeeX4 falls to 49.39% with Selective_Context and 37.20% with LLMLingua2.Source: §3.2 · p. 9
- At 10% compression, the methods remove different frequent tokens: Selective_Context favors semantic coherence, while LLMLingua2 is sensitive to common articles and prepositions.The authors connect this divergence to each method’s distinct criteria for identifying important tokens and emphasize balancing redundancy removal with preservation of code-critical information.Source: §3.2 · pp. 9–10
4 APPROACH
ShortenDoc compresses DocStrings while preserving their semantics and usefulness for code generation. It ranks tokens, explores removal candidates, and constrains compression to limit changes in model output.
4 APPROACH
- ShortenDoc frames DocString compression as minimizing compressed length while preserving semantics, with the compressed DocString constrained to be a subsequence of the original.The approach has four stages: preprocessing, token-importance sorting, search-space construction, and constraint definition.Source: §4 · p. 10
4.1 Pre-processing
- Removing line breaks and tabs is assumed not to significantly affect Pass@1; stop words are compressed only when semantic similarity exceeds 0.999.The preprocessing identifies frequent low-impact tokens such as “the” and “a” as stop words.Source: §4.1 · p. 11
4.2 Token Importance Sort
- Token importance is measured by the change in DocString perplexity when a token is omitted; tokens are sorted from least to most important for compression.Tokens whose removal changes conditional entropy little are considered candidates, while those whose removal raises it substantially are prioritized for retention.Source: §4.2 · pp. 11–12
4.3 Search Space Construction
- The search space includes consecutive k-grams drawn from the Top-N least-important tokens, allowing both individual tokens and token sequences to be considered for removal.The number of candidate combinations is bounded by N(N+1)/2.Source: §4.3 · p. 12
4.4 Constraint Definition
- Compression aims to keep the original and compressed code-output distributions close, approximating this requirement with cosine similarity between their next-token logits.The cosine-similarity threshold τ is set to 0.999.Source: §4.4 · p. 13
5 EVALUATION
The evaluation tests whether ShortenDoc can reduce DocString length while retaining code-generation quality, then examines efficiency, cross-language transfer, and human judgments.
5.1 RQ1: What is the effectiveness of our proposed ShortenDoc compared to existing prompt compression methods? (Pass@1 Comparison)
- At a 30% reduction in DocString length, ShortenDoc achieves the highest HumanEval Pass@1 across all models, consistently outperforming alternative methods across datasets and models, particularly at higher compression rates.On HumanEval, DeepSeekCoder-6.7B, CodeQwen and CodeGeeX4 can achieve higher Pass@1 with ShortenDoc than with the uncompressed baseline.Source: §5.1 · p. 14
- Compressed-model FLOPs are lower than Raw across datasets; HumanEval falls from 0.34 to 0.28 and CodeHarmony from 0.15 to 0.12.ShortenDoc’s compression adds about 2 seconds per DocString, while the combined compression-and-inference pipeline is described as efficient.Source: §5.1 · pp. 15–16, Table 4 · p. 16
5.2 RQ2: What is the effectiveness of our proposed ShortenDoc in other program languages? (Generalization Capability)
- Across four non-Python languages, ShortenDoc beats compression baselines after Python-compressed DocStrings are migrated, though Pass@1 can remain below the uncompressed baseline.The authors attribute the slight degradation to CodeGPT’s Python-specific training and suggest language-specific models may yield the best results.Source: Table 5 · p. 17, §5.2 · pp. 17–18
5.3 RQ3: What is the impact of compressed prompts on developers? (Human Study)
- In human ratings, ShortenDoc outperforms Selective_Context and LLMLingua2 on informativeness and comprehensibility; agreement is 0.5–0.9 for informativeness and 0.1–0.3 for comprehensibility.The study assesses whether compressed DocStrings preserve key information and remain understandable.Source: §5.3 · pp. 19–21, Table 6 · p. 20
6 DISCUSSION
The discussion relates compression outcomes to method-name quality and DocString style, then examines hyperparameter and base-model choices alongside threats to validity.
6.1 The Impact of Method Name Quality
- Pass@1 fell when informative method names became ‘foo’ and rose when BigCodeBench’s generic names were made descriptive, suggesting overlap with information in compressed DocStrings.The comparison is reported in Table 7.Source: §6.1 · pp. 21–22, Table 7 · p. 22
6.2 The Impact of DocString Style
- 11.36% to 7.32%: BigCodeBench Pass@1 declined after newline/tab removal, while most datasets stayed stable; tokenization and dataset complexity were proposed explanations.The authors attribute the decline partly to how models process the DocString’s triple-quote/newline prefix and to BigCodeBench’s complexity.Source: §6.2 · pp. 22, 24, Table 8 · p. 23
- Stop-word removal had nuanced, slight Pass@1 effects, whereas removing MBPP’s generic “write a function to” instruction modestly improved scores.The experiments compare DocStrings with and without these elements.Source: §6.2 · pp. 23–24, Table 9 · p. 23
6.3 Hyper-Parameter Analysis
- τ=0.999 remained competitive across datasets, but detailed tuning was conducted only on CodeHarmony and may not be optimal everywhere.The authors leave broader dataset-specific analysis of τ for future work.Source: §6.3 · pp. 24–25
- Base models produce different token-importance rankings and performance; CodeGPT-py-adapted performed best, while ShortenDoc remained robust across CodeHarmony models.The discussion links baseline differences to pretraining strategies and reports ranking variation across models.Source: §6.3 · p. 25
6.4 Threats to Validity
- Cross-language generalizability may be limited because evaluation used translated HumanEval-X; proprietary models were omitted amid API-cost and version-stability concerns.For cross-language evaluation, the study relied on translations of the original HumanEval dataset.Source: §6.4 · pp. 25–26
- Pass@1 and compression ratio measure generation performance and prompt reduction, but Informativeness and Comprehensibility may miss other DocString-quality aspects.The human-study evaluators had strong Python backgrounds and received evaluation criteria.Source: §6.4 · p. 26
7 RELATED WORK
Related work spans code-generation models and prompt-compression approaches. The paper positions ShortenDoc as an adaptive compression method specifically for code-generation DocStrings.
7.1 Code Generation
7.2 Prompt Compression
- Prompt compression includes token pruning or merging, soft prompts, and entropy-based filtering; task-aware methods may limit generalizability, while entropy methods are task-agnostic [16].Selective Context [22] and LLMLingua [15] are examples of entropy-based approaches.Source: §7.2 · p. 27
- ShortenDoc targets DocStrings for code generation, adapting compression to token importance and using Top-N to remove redundancy while preserving semantic integrity.The paper motivates this focus by observing that models can sometimes generate correct code after redundant DocString information is removed.Source: §7.2 · p. 27, §7.1 · p. 27
8 CONCLUSION AND FUTURE WORK
ShortenDoc dynamically adjusts DocString compression to retain informativeness and comprehensibility while aiming to reduce computational resource costs. The authors outline extensions to broader code-generation settings.
8 CONCLUSION AND FUTURE WORK
- ShortenDoc dynamically adjusts compression while seeking to retain essential DocString information, informativeness, and comprehensibility.The stated goal is to improve model efficiency and reduce computational resource costs.Source: §8 · p. 28
- Future work will extend the study from function-level to class-level code generation, involving more complex logic and larger code structures.The authors expect to examine whether ShortenDoc can accommodate those structures while maintaining compression efficiency and output quality.Source: §8 · p. 28
- The authors plan to explore multilingual code generation by training a multilingual base language model.They identify wider language support as a potential source of greater utility for DocString compression tools.Source: §8 · p. 28
- The authors also plan to extend the work to general code-completion tasks.They cite code completion’s higher frequency in real-world software engineering compared with natural-language-to-code generation as motivation.Source: §8 · p. 28