What Survives the Next Model? Benchmarking LLM-Based Techniques Against Single-Prompts

Motivation

• ICSE Research Track abstracts mentioning “LLM” increased from 2% (5 of 207) in 2023 to 45% (144 of 321) in 2026; bespoke workarounds for current LLM limitations may become obsolete as future models improve.Source: §I · pp. 1–2
• On a new Rust-verification benchmark, AutoVerus solved 20% of problems, while Sonnet 4.5 solved 81% with a single straightforward prompt two months later.The authors present this case as a cautionary illustration, not proof of a broader trend.Source: §I · pp. 1–2

Study design

• The analysis covers 35 ICSE 2026 papers and compares their core techniques with an automatically generated few-shot prompt on a newer model, without refinement, execution loops, or external tools.The study reproduces the papers’ key results before evaluating the baseline.Source: §I · p. 2

Findings

• Between 37% and 63% of heavily engineered LLM-based software-engineering techniques published less than six months earlier were replaceable by a single-prompt baseline; constructive tasks were most susceptible.For most of these techniques, the half-life was less than a calendar year.Source: §I · p. 2
• A single prompt was insufficient for tasks needing semantic understanding, complex feedback validation, or project-wide knowledge, though execution loops or tool access could bridge much of the gap.The paper presents these additions as projections, not as evaluated results.Source: §I · p. 2
• The study identifies resilient techniques that model advances amplify: domain-specific processing, symbolic integration, and inputs structured to provide richer context.It argues that research should focus on enduring challenges that scale with model capabilities rather than being erased by them.Source: §I · p. 2

II. BACKGROUND

• Surveys document LLM adoption across software engineering, from coding, testing, requirements, repair, refactoring, and documentation to code generation and agents,.The study likewise spans tasks and strategies across 35 ICSE papers.Source: §II · p. 2
• As prompting strategies and autonomous agents evolve,,, [20],,, this study tests the simplest replacement: one prompt to one newer LLM, without tools or iterative interaction.Prompting surveys catalog strategies such as zero- to few-shot and chain-of-thought to tree-of-thought, while agents add planning, reasoning, memory, and tools.Source: §II · p. 2
• The study asks whether newer models will make current LLM-based software engineering techniques obsolete, a question that defines their operational half-life.Unlike benchmarks that measure a model on a task in isolation, it substitutes a single-prompt technique while keeping tasks and artifacts fixed; conducting this systematically across papers is non-trivial.Source: §II · p. 2

III. RESEARCH QUESTIONS

• RQ1 asks how effectively single-prompt inference on state-of-the-art LLMs competes with a recent software engineering tool for a given task.Source: §III · p. 3
• RQ2 asks which factors make a technique more amenable to substitution by a single-call LLM.Source: §III · p. 3
• RQ3 asks how much embedding the original paper’s methodology as reasoning guidance in Pw improves performance compared with Pb.Source: §III · p. 3

IV. METHODOLOGY

• The methodology compares techniques from selected ICSE 2026 papers with a single-prompt newer-LLM alternative, requiring evaluable candidate papers and a consistent prompt-generation mechanism.The candidate papers must provide artifacts needed to evaluate the prompt-based alternative.Source: §IV · p. 3
• Generated prompts run on the selected LLM with the candidate papers’ inputs and conditions, and outputs are evaluated against each paper’s tool on the same artifacts and process.Source: §IV · p. 3

A. Pool of Candidate Papers

• Another 19 papers were excluded because their inputs were too large to fit the study’s budget and posed challenges for LLM context handling.For, processing 25 repositories containing roughly 71K C/C header files was estimated to cost about $500 for input alone and several thousand dollars including generated Rust output; the passage also notes that LLMs struggle with large contexts.Source: A. Pool of Candidate Papers · p. 3
• The researchers randomly selected 35 of the 47 remaining papers for analysis.These papers are cited with the prefix EP and listed under “Evaluated Papers” in the references.Source: A. Pool of Candidate Papers · p. 3

B. Black-box and White-box Prompt Design and Generation

• Both prompt types shared fields for role, task, input, output, example input-output pairs, and instructions.The study generated prompts for each paper to specify the task and corresponding inputs and outputs.Source: B. Black-box and White-box Prompt… · p. 3
• Black-box prompts asked the LLM to infer task functionality from inputs and outputs, while white-box prompts added the paper’s methodology as internal reasoning guidance.The white-box prompt added a Steps field to the common prompt structure.Source: B. Black-box and White-box Prompt… · p. 3
• The researchers provided three examples whose combined outputs illustrated the full schema.Source: B. Black-box and White-box Prompt… · p. 4
• Prompt refinement evaluated generated prompt structure, not target-task performance, and the accepted meta-prompt became the single template for all experiments.The template is shown in Figure 4.Source: B. Black-box and White-box Prompt… · p. 4

C. Prompt Executor

• Before execution, datasets had to provide inputs and expected outputs, and the team performed preprocessing or retrieval needed to match each paper’s technique inputs.For example, they removed function arguments with Python’s ast module to recreate incomplete-source inputs for [EP1], and excluded two [EP14] datasets without ground-truth vulnerability labels.Source: C. Prompt Executor · p. 4
• Scripts made a single LLM call with each prompt and input, stored raw and structured responses, and evaluated formatted outputs against the original expected outputs using each paper’s protocol.The structured outputs included code, labels, test cases, constraints, and predictions.Source: C. Prompt Executor · p. 4
• Claude Sonnet 4.6, released in February 2026, was selected as the trailing-edge model to preserve the study’s baseline, control costs, and establish a capability floor for future frontier models.Anthropic’s Opus models were approximately 1.67x more expensive than Sonnet for input and output tokens; the authors reasoned that if Sonnet succeeds, future frontier models should achieve at least the same performance.Source: C. Prompt Executor · pp. 4–5
• A $15-per-paper budget enabled full-dataset analysis for 15 papers, 10%–70% coverage for 16 papers, and 5%–7% coverage for four papers.The authors randomly sampled inputs under the fixed budget; they did not pursue the four lowest-coverage papers further because their results were mostly negative.Source: C. Prompt Executor · p. 4, Figure 5 · p. 5

D. Evaluator

• Evaluators used each paper’s end-to-end research question and metrics to make single-prompt results directly comparable to the technique’s reported performance.Because papers used distinct protocols, constructing evaluators was the most effort-intensive phase.Source: D. Evaluator · p. 5
• 18 papers supplied explicit ground truth, while 17 required execution or artifact checks before the researchers could compute the paper’s metrics.Checks included tests, coverage tools, compilers, semantic-equivalence checkers, or benchmark-specific harnesses.Source: D. Evaluator · p. 5
• The researchers reproduced original evaluation conditions, including running [EP26] tests against both patches and comparing [EP29] PostgreSQL and MySQL query results.These examples required constructing Docker-based environments and running the corresponding harnesses or database queries.Source: D. Evaluator · pp. 5–6
• Each of the 35 techniques received a paper-specific evaluator encoding its metrics, ground truth, and any required execution environment or external checks.Preparing, revising, and packaging each evaluator took several hours; scripts, generated outputs, and results are available in the replication repository.Source: D. Evaluator · p. 6

E. Artifact Quality Control and Paper Categorization

• Reviewers independently cross-checked artifacts against a common checklist and discussed, corrected, and reran identified issues before final analysis.The checklist covered paper and dataset selection, research-question and metric alignment, prompt examples and task formulation, and evaluator implementation.Source: E. Artifact Quality Control and Paper… · p. 6
• Reviewers incrementally coded each paper’s domain and technical contribution, then refined categories until two reviewers reached consensus on granular, mutually exclusive groupings.Several reviewers first characterized papers independently; a primary encoder synthesized those characterizations with the papers.Source: E. Artifact Quality Control and Paper… · p. 6
• The resulting task labels cover eight software-engineering tasks, and papers may receive multiple task labels when they address more than one.The labels are code generation, bug finding, repair, impact analysis, requirement formalization, log analysis, test generation, and verification.Source: E. Artifact Quality Control and Paper… · p. 6
• Strategy labels capture seven approaches: Knowledge Grounding, Feedback and Validation, Structured Reasoning, Search and Selection, Input/Output Control, Domain-Specific Processing, and Model Adaptation.Papers may receive multiple strategy labels when they combine techniques.Source: E. Artifact Quality Control and Paper… · p. 6

• Results are reported for each research question.Source: §V · p. 6

A. RQ1: Single-Prompt LLM vs. Engineered SE Approaches

• Table II classifies comparisons as positive for full single-prompt wins, mixed for partial wins, and negative for consistent underperformance.Its Diff(nl) Performance compares the best single-prompt result against the original technique.Source: A. RQ1: Single-Prompt LLM vs.… · p. 6, Table II · p. 7
• Among 35 studied papers, single-prompt inference consistently outperformed the original technique in 13 cases and had mixed results in 9 additional cases.Mixed results mean the single prompt outperformed the original on only a portion of datasets or metrics.Source: A. RQ1: Single-Prompt LLM vs.… · pp. 6, 8
• Between 37% and 63% of recent SE techniques can be effectively replaced by an automatically generated single prompt to a general-purpose LLM.Table II compares paper-reported results with single-prompt results.Source: Table II · p. 7, A. RQ1: Single-Prompt LLM vs.… · p. 8

B. RQ2: Amenability to single-prompt LLM substitution

• For code generation, single-prompts beat techniques in six of nine papers, with one mixed result and two losses; all three Search and Selection cases were positive.In [EP4], the single-prompt scored 78.85% on pass@1 versus 61.9% for the original approach; the findings suggest newer models synthesize functional code well when given simple constraints.Source: B. RQ2: Amenability to single-prompt… · p. 8
• For code repair, single-prompts outperformed three of six techniques and produced two mixed results; Structured Reasoning cases included two wins and one mixed result.The repair results also included no negative outcomes among the two papers using Feedback and Validation: one was positive and one mixed. A single prompt beat [EP17]’s three-agent workflow for translating buggy code, generating repairs, and validating or revising the fix.Source: B. RQ2: Amenability to single-prompt… · p. 8
• For Bug Finding, prompts struggled with Domain-Specific Processing [EP16], [EP31], [EP22], Knowledge Grounding [EP23], [EP27], and Feedback and Validation [EP35]; Impact Analysis had no positive or mixed results [EP32], [EP20].The remaining task categories had too few papers to show consistent patterns.Source: B. RQ2: Amenability to single-prompt… · p. 8
• Four of five Domain-Specific Processing papers resisted replacement, while Knowledge Grounding yielded one positive, three mixed, and four negative results.These techniques can add domain-specific procedures or intermediate context, such as method dependencies, static-analysis results, and API relationships, unavailable to prompts given only raw task input. Model and dataset variation obscures definitive patterns; the authors note that prompts beat every technique on HumanEval, which may involve benchmark leakage but does not establish its extent.Source: B. RQ2: Amenability to single-prompt… · pp. 8–9
• Single-prompts most often replace Code Generation and Repair, especially with Search and Selection or Structured Reasoning, but struggle with Bug Finding, Program Analysis, and Requirement Formalization.Source: Figure 6 · p. 9, B. RQ2: Amenability to single-prompt… · p. 9
• Across strategies, Search and Selection was most replaceable (3/3 positive), while Structured Reasoning had five positive, three mixed, and one negative result among nine papers.These results suggest newer models can often internalize selection-based techniques and structured reasoning in a single prompt.Source: B. RQ2: Amenability to single-prompt… · p. 8, Figure 6 · p. 9

• Techniques depending on static analysis, slicing, fuzzing, and runtime oracles can require tools Pw cannot mimic; this study withheld tool use and feedback, leaving agentic implementations as future work.Examples include smart-contract replay-vulnerability analysis [EP27] and vulnerability detection using state dependencies, branch conditions, function-call sequences, and fuzzing [EP14].Source: C. RQ3: The Value of White-Box Prompting · pp. 9–10
• Pw required an average of 80% more prompt tokens than Pb; the authors conjecture that guidance a model cannot use may distract it and weaken results.Source: C. RQ3: The Value of White-Box Prompting · p. 10
• For Knowledge Grounding, Pb outperformed Pw in 5 of 8 papers, while Feedback and Validation was the only strategy where Pw slightly outperformed Pb, in 4 of 9.Pb also slightly outperformed Pw for Domain-Specific Processing (3 of 5, with none mixed) and Structured Reasoning (3 of 9, with 4 mixed); validation criteria and feedback descriptions may help models anticipate evaluation and revise internally.Source: C. RQ3: The Value of White-Box Prompting · p. 10
• Pw cost $7.5 per paper on average versus $6.5 for Pb, making Pw approximately 15% more expensive.Both cost estimates had standard deviations of slightly over $4.Source: C. RQ3: The Value of White-Box Prompting · p. 10
• In 40% to 65% of studied SE tasks, Pb matched or outperformed Pw; Pb beat Pw in 14 of 35 papers, Pw beat Pb in 12, and 9 were mixed.Source: C. RQ3: The Value of White-Box Prompting · pp. 9–10

a) External Validity

• The ICSE-based sample and changing model availability, costs, and access restrictions limit how broadly the findings generalize.The study documents its selection criteria and uses contemporary, widely accessible models to mitigate these concerns.Source: D. Threats to Validity · p. 10

b) Internal Validity

• Best-effort replication across 35 studies, technique categorization, budget-limited sampling, and possible benchmark exposure constrain internal validity.Multiple-researcher cross-checking refined evaluations, but could not eliminate these limitations.Source: D. Threats to Validity · p. 10

c) Construct Validity

• Coarse performance categories enable comparisons across heterogeneous studies but may obscure smaller patterns and technique-specific qualitative differences.The authors comment on observed patterns and provide data for further scrutiny.Source: D. Threats to Validity · p. 10

d) Conclusion Validity

• Heterogeneous papers and evaluations, together with the small sample, prevent significance tests, so conclusions are presented as indicative trends rather than causal findings.Source: D. Threats to Validity · p. 10

VI. CONCLUSION

• The study is the first systematic investigation of the longevity and strategic value of LLM-based software engineering techniques, analyzing 35 ICSE 2026 papers with an automated, reproducible framework.The authors note limitations from fixed execution budgets, dataset sampling constraints, and categorization schemas, and describe the work as a first step.Source: §VI · p. 10
• Between 37% and 63% of SE techniques published less than six months ago can be replaced by a single automatically generated prompt on a newer model, without iteration or other tools.The findings warn of the short half-life of many heavily engineered workarounds.Source: §VI · pp. 10–11
• Constructive code generation and repair techniques are easiest to replace, while domain-specific processing and project-wide knowledge grounding are least likely; deeper semantic or feedback-loop techniques still fare better.Some performance gains may be matched by placing single prompts in a basic loop, delegating them to agents, or granting access to other tools.Source: §VI · p. 11
• The findings urge researchers to shift from transient LLM scaffolds toward enduring work on where symbolic approaches can add complementary value and techniques that scale alongside model capabilities.The paper characterizes heuristics that patch temporary model deficits as rapidly cleared away by computational scaling.Source: §VI · p. 11

EVALUATED PAPERS

• The bibliography excerpts list titles on API argument completion with knowledge-augmented prompts, layerwise behavioral-model generation, and cross-language LLM repair with multi-agent refinement, each in ICSE 2026 proceedings.Source: EVALUATED PAPERS · pp. 12–13