Source-linked AI summary
Talking to Itself While Coding: What Makes Comments Help Code Generation?
Dangfeng Pan, Zhensu Sun, Cenyuan Zhang, David Lo, Xiaoning Du
TL;DR
The paper asks which properties of self-emitted comments affect code-generation performance and studies the question with observational analyses and controlled interventions. It finds that correct solution content improves recipient pass@1, while prompting alone cannot reliably reproduce the external-comment benefit.
Problem
The central gap is determining whether comment properties such as form, intent, topic, or solution correctness affect an LLM’s own code generation.
Method
The study profiles natural comments, then prefills weaker recipients with stronger sources’ comment blocks while varying content, form, topic, and order.
Results
Correct source comments raise recipient pass@1 by 17.2% on average, whereas wrong-topic comments reduce it by 20.8%; failed-solution comments provide no reliable gain.
Takeaways & Limitations
Comments help when they convey correct solution content, but self-prompting cannot reliably elicit that content or recover the external-comment lift.
Takeaways & Limitations
The tested prompting methods did not reliably reproduce the external-comment lift, so closing this gap likely requires training-time methods.
Abstract
from arXiv · showhide
Large Language Models (LLMs) often generate natural-language comments while writing code, and these comments become part of the context used to generate the code that follows. However, it remains unclear which properties of comments affect code-generation performance. We study this question through observational analyses and controlled interventions. On LiveCodeBench, neither comment frequency nor broad comment intent reliably predicts pass@1. We then prefill weaker recipient models with comment blocks written by stronger source models, allowing us to separate comment surface form from the solution content they convey. Comments from source solutions that pass the tests raise recipient pass@1 by 17.2% on average. In contrast, comments describing failed solutions provide no reliable gain, while comments written for a different problem reduce pass@1 by 20.8%. Finally, across a wide range of models and prompt variants, most recipient models show no significant recovery of the external-comment gain, and the best case recovers only 24%. These results show that comments help code generation not merely because they are comments, but because they can provide correct solution content that prompting cannot reliably elicit.
1 Introduction
The study asks which properties of self-emitted comments affect code-generation performance. Observational patterns are insufficient, but controlled interventions show that correct solution content—not comment form alone—improves pass@1, while self-prompting cannot reliably reproduce the gain.
- Motivation: Self-emitted comments become context for the same model’s subsequent code, making their functional effect on pass@1 unclear.The comments also consume output tokens, so suppressing them depends on whether they causally help.
- Observational analyses: Neither comment volume or form nor coarse intent reliably explains pass@1 in the observational analyses.Stronger models explain choices somewhat more often, but the trend does not survive Bonferroni correction.
- Controlled intervention: 17.2% average pass@1 gain follows when weaker recipients receive comments describing source solutions that pass the tests.The intervention varies comment content while holding the recipient, prompt, and problem fixed.
- Controlled intervention: Wrong-content comments leave pass@1 unchanged when surface form matches, whereas comments for a different problem reduce it by 20.8%.These comparisons identify correctness of the described solution as the relevant property rather than comment appearance.
- Self-prompting: Self-prompting recovers at most about 24% of the external-comment lift across models and prompt variants.Prompting changes comment style but cannot reliably make the content describe a correct solution.
2 Related Work on Code Comments in Code Generation
Prior work treats comments as text for human readers or as model inputs, but it does not isolate which comment property drives code-generation effects. This study adapts content-versus-form intervention logic to the comment interface.
- Prior research: Traditional comment research focuses on human comprehension, maintenance activities, and communication of rationale or intent.Informative comments have been reported to improve human program comprehension.
- Open question: Existing code-generation studies examine comment inputs and related effects but do not isolate the responsible preamble property.Candidate properties include surface form, line order, topic, intent, and correctness of described solution content.
- Study organization: Figure 1 organizes the study into observational analyses of form and intent, external-comment intervention, and self-prompting.The observational components profile 88 models for form and 13 models for intent.
- Study approach: The intervention varies one comment property at a time while holding other factors fixed, identifying content correctness as dominant.This extends the ICL content-versus-form question through the comment interface.
3 Observing Self-Emitted Comments
The observational study profiles comment form across 88 LiveCodeBench models and comment intent across 13 models. Neither dimension provides a robust signal of pass@1, although the intent analysis is limited by heterogeneous distributions and a small pool.
- 3.1 Comment form: The study measures comment form using tok-cmt%, block%, inline%, and doc% under a shared tokenizer.Form is evaluated both across models at matched pass@1 and within models using Spearman correlations.
- 3.1 Comment form: 700-fold tok-cmt% variation occurs among 16 top-tier models with pass@1 ≥80.0%, showing that high-performing models use radically different comment volumes.The range is 0.1% to 71.4%.
- 3.1 Comment form: At most 13.0% of models exceed |ρ|=0.20 on the three PEP-8 form axes, while 22.0% exceed it for tok-cmt%.The maximum correlation is 0.32 for block%, inline%, or doc%, and 0.53 for tok-cmt%, still insufficient to change the conclusion.
- 3.2 Comment intent: Intent correlations are positive for WHY (ρ = +0.61) and negative for WHAT (ρ = −0.20), but neither survives Bonferroni correction.The raw p values are 0.027 and 0.52, against corrected threshold α=0.0167.
- 3.2 Comment intent: The 13-model intent pool cannot establish absence of an effect because models with similar pass@1 show widely different intent distributions.At around 60% pass@1, WHY-fraction ranges from 18.1% to 40.7%; label reliability is also mixed.
4 Intervening on Comment Content
Controlled prefilling shows that comment content, especially whether it conveys a correct solution, drives changes in recipient pass@1 more than comment form, order, or topic-independent text. External correct comments improve performance, while wrong-topic or corrupted content harms it, and prompting recipients to generate comparable comments recovers little of the benefit.
- Intervention design: The intervention attaches source-model comment blocks to recipient outputs while holding the recipient, prompt, and problem fixed, isolating content correctness, line order, topic, and surface form.Source-written conditions include source-correct and source-wrong comments; shuffled, wrong-topic, and random-text conditions test additional properties.
- Content correctness: 17.2% average pass@1 lift comes from same-form comments describing correct solutions, whereas same-form comments describing failed solutions produce no reliable gain.All 12 source-correct cells survive Holm–Bonferroni correction, while none of the 12 source-wrong cells reaches significance.
- Topic and coherence: 20.8% pass@1 damage from wrong-topic comments and 17.9% damage from length-matched random text show that extra tokens or comment-shaped text alone do not explain the effect.Wrong-topic comments are coherent but describe another problem; random text removes coherent problem content.
- Line order: 68.0% of the source-correct lift survives line shuffling, indicating that comment lines act mainly as local hints with a smaller order-sensitive component.Shuffled comments yield +11.7% versus +17.2% for unshuffled source-correct comments, and 10 of 12 shuffled cells survive Holm correction.
- Per-problem effects: 89 problems flip from fail to pass under correct prefill versus 3 in the opposite direction, spanning easy, medium, and hard difficulty bands.The flips include 56 medium, 18 easy, and 15 hard problems, so the lift is not confined to the hardest tasks.
- Prompting alone: Prompting recovers at most 24% of the external-comment lift, while prompt variants can improve some receiving models and damage others.Every non-baseline variant lifts at least one model and damages at least one model; the direction depends on the receiving model rather than comment-form compliance alone.
5 Conclusion
Across LiveCodeBench, natural commenting habits do not track pass@1, while controlled interventions show that correct solution content in comments improves generation and wrong-topic content harms it. Self-prompting rarely recreates this benefit, so comment use should depend on whether comments help a given model.
- Commenting frequency and content do not track pass@1 across 88 models on LiveCodeBench.
- 17.2% average pass@1 improvement follows when weaker recipients receive comments describing source solutions that pass.Same-form comments describing wrong solutions leave pass@1 unchanged, whereas different-topic comments reduce it by 20.8%.
- At most about 24% of the external-comment lift is recovered when recipients generate comments from their own prompts.The benefit is absent for three of four recipients, and prompt engineering changes comment appearance more reliably than content correctness.
- Suppressing comments costs up to 14.5% on comment-dependent models but helps others, motivating model-specific comment policies.The paper proposes checking this profile with the none-versus-base contrast.
Limitations
The experimental scope is limited because tested prompting strategies did not reliably reproduce the external-comment lift through self-elicitation.
- Despite testing ten prompt variants, explicit self-planning, and two-stage pipelines, no prompting method reliably reproduces the external-comment lift.The authors leave training-time methods for closing this gap to future work.
Ethics Statement
The study uses publicly available language models and a public benchmark under its public release terms, without human subjects research.
- The experiments analyze publicly available LLMs on LiveCodeBench-v6 under the benchmark’s public release terms.No human subjects research was conducted.
- The authors used commercial LLM assistants for sentence-level revision and analysis-script drafting, while retaining experimental and manuscript responsibility.The authors state that experimental design, statistical claims, and final manuscript content remained their own.
A.1 LLM-judge calibration on 100-snippet stratified subset
The study calibrates an LLM judge against human labels for comment intent, then aggregates seven classes into WHY, WHAT, and OTHER buckets for analysis. Reliability supports WHY, while OTHER is excluded from quantitative claims.
- Codebook and aggregation: The seven-class codebook labels comments as rationale, algorithm, step, mechanism, definition, decoration, or misleading.The main analysis maps rationale to WHY, algorithm plus step, mechanism, and definition to WHAT, and decoration plus misleading to OTHER.
- Calibration: Human and LLM-judge annotations are compared on a stratified 100-snippet calibration set using per-label binary Cohen’s κ.The judge is Gemini-3.1-Flash-Lite.
- Per-label agreement: The two anchor labels, rationale and algorithm, show κ ≥0.7, while the remaining labels are absorbed into aggregated buckets.
- Bucket-level agreement: WHY reaches reliable bucket-level agreement with κ = AC1 = 0.74, while WHAT’s low κ is interpreted alongside high prevalence and high raw agreement and AC1.
- Limitations: OTHER does not reach usable reliability and is therefore excluded from quantitative statements.
- Multiple comparisons: Three intent–pass@1 correlations are tested in parallel with Bonferroni correction at α = 0.0167 per test, and none passes the corrected threshold.The correction bounds the family-wise false-positive rate at 0.05.
- Reported analysis: Intent fractions are reported for 13 models, with WHY, WHAT, and OTHER groupings shown as supercolumns and rows summing to 100%.
C.3 Compute statement
The study documents its compute and reproducibility resources, standardizes comment measurement with a shared tokenizer, and evaluates comment form against pass@1 across models.
- Compute and reproducibility: Approximately 200 A100-hours supported prefix-recipient inference and self-elicit generations, while source-prefix generation used about $500 in OpenRouter credit.Re-analysis of existing generations requires no GPU or API spend; end-to-end reruns require the stated A100 budget and listed model access.
- Measurement: All 88 model outputs use the shared cl100k_base tokenizer so comment volumes are comparable across vendors.The measured form axes include tok-cmt%, block%, inline%, and doc%.
- Analysis: The form analysis reports matched-pass@1 variation across 16 top-tier models and within-model Spearman correlations across all 88 models.The study evaluates both cross-model spread and per-problem associations between comment form and pass@1.
- Findings: 22.0% of models exceed |ρ|=0.20 for comment volume, while the maximum reaches |ρ|=0.53 and remains below 30.0% explained variance.For block%, inline%, and doc%, no model exceeds |ρ|=0.32 and at most 13.0% exceed |ρ|=0.20.
D.5 Length-matched random-text control
The length-matched random-text control tests whether performance changes arise from comment content or merely from adding a similarly sized prefix. Random filler harms pass@1, but less than wrong-topic comments, while source-correct gains concentrate on medium problems.
- Random-text control: Random text damages pass@1 on all four recipients, with a mean ∆=−17.9% across 12 cells.The control uses #-prefixed filler matched in length to source-written blocks and compares against the same per-cell matched-prompt baseline.
- Comparison: Wrong-topic comments damage pass@1 more than random text, with mean ∆=−20.8% versus −17.9% across the 12 cells.The comparison is made on common problems using paired tests with Holm correction.
- Difficulty bands: Source-correct comment blocks lift pass@1 most on medium problems for most recipient-source pairs.Wrong-topic and random-text damage concentrates on easy problems.
- Problem-level transfer: 89 of 870 problems are source-transferable, 3 are source-toxic, and 778 are mixed or neutral across four recipients.Source-transferable problems require at least three recipients to flip from baseline-fail to source-correct-pass with no opposing loss.
- Difficulty distribution: Among 89 source-transferable problems, 56 are medium, 18 easy, and 15 hard.
D.8 Per-problem flip atlas: qualitative anchors
The flip-atlas examples show that useful prefixes can transmit a decisive algorithmic choice, redirecting recipients from plausible incorrect strategies to correct solutions.
- Qualitative pattern: On 119 problems with unanimous improvement among at least 6 of 12 voters, source comments typically reroute recipients from plausible-but-wrong strategies to correct ones.
- LeetCode 3617 example: In LeetCode 3617, the source comment identifies summing repeated-character group contributions rather than multiplying them.The example uses Gemini-3.1-Flash-Lite as source and Qwen3-8B as recipient.
- Baseline error: Without the prefix, Qwen3-8B uses a plausible-looking but incorrect product of group counts.
- Transferred solution content: With the prefix, Qwen3-8B implements the source formula and passes the problem.The transferred decision is “a sum, not a product.”
- Statistical audit: The audit reports per-model dramatic cells and Holm-significant effects across the self-elicit grid, alongside mean output-token and comment-fraction analyses.These audits distinguish surface-form compliance from performance effects.
D.12 Per-cell mean code tokens and tok-cmt%
The per-cell analysis shows that models generally follow instructions about comment form and length, but these output changes do not predict pass@1 effects.
- Form compliance: Form policies are followed in aggregate: none usually lowers tok-cmt%, while dense, WHY, and mix generally raise it above 40%.The detailed grid reports these patterns across 14 models.
- Token effects: Under none, mean output code tokens shift by an average of −21.8% across 14 models, with model-dependent variation.Verbose variants can raise mean token counts to 2.6× the base value, while inline can reduce them to 0.7×.
- Model variation: 13 of 14 models contract under none, with the largest drop at −59%, while MiMo-V2-Flash expands by +46%.
- Evaluation: The grid’s pass@1 effects are evaluated with per-model dramatic-cell audits, global Holm correction, and baseline comparisons.
- Performance relationship: Surface-form compliance does not track the corresponding pass@1 effect across the self-elicit grid.The tables organize per-cell token counts and tok-cmt% for the 14×10 model-variant grid.