Source-linked AI summary
What Survives the Next Model? Benchmarking LLM-Based Techniques Against Single-Prompts
Nahian Salsabil, Joy Saha, Simantika Bhattacharjee Dristi, Nicholas Phair, Nusrat Jahan Mozumder, Matthew B. Dwyer, Sebastian Elbaum
TL;DR
The paper asks whether complex LLM-based software engineering techniques remain strategically valuable as newer models emerge. It evaluates 35 ICSE 2026 techniques against automatically generated single prompts on a newer model and finds that many are replaceable, while techniques adding structured domain or project context are more resilient.
Problem
The study addresses limited evidence about whether complex LLM-based software engineering techniques survive newer model generations and what determines their operational half-life.
Method
The authors reproduce 35 ICSE 2026 techniques and compare them with automatically generated single few-shot prompts executed on a newer model without iterative refinement or external tools.
Results
37% to 63% of heavily engineered techniques published less than six months earlier can be effectively replaced by a single prompt, especially for constructive tasks such as code generation and repair.
Takeaways & Limitations
Research should prioritize enduring challenges and techniques that add domain-specific processing, symbolic integration, or richer structured context rather than temporary model workarounds.
Takeaways & Limitations
The study is constrained by best-effort replication, coarse categorization, fixed execution budgets, and dataset sampling constraints.
Abstract
from arXiv · showhide
The software engineering research community has enthusiastically embraced the integration of Large Language Models (LLMs) into complex techniques to solve a wide variety of tasks. However, the extent to which this investment is strategic remains unclear, as the native capabilities of successive frontier model generations can rapidly render existing techniques obsolete. To assess this research investment, we analyze 35 LLM-based technique papers from ICSE 2026. We evaluate whether their complex tools can be outperformed by the simplest possible alternative: a single, automatically generated prompt executed on a newer generation model, without any iterative refinement. We find that for between 37% and 63% papers, a newer model with a single prompt natively outperforms the heavily engineered tooling proposed just a year prior. We identify that constructive techniques like code generation or repair are more amenable to substitution by a single-prompt. We also identify a surviving set of papers relying on strategies that provide additional insights to the model where newer LLMs will amplify the proposed technique. Our findings raise questions about the cost-benefit proposition of techniques designed as workarounds to temporary model deficits and the need to focus on enduring challenges that scale synergistically with future model generations. Our source codes and results are made publicly available at https://github.com/less-lab-uva/What-Survives-the-Next-Model.
I. INTRODUCTION
LLM adoption in software engineering research has accelerated rapidly, raising concern that complex techniques may be overtaken by newer models. This study tests that concern across 35 ICSE 2026 papers using automatically generated single prompts.
- LLM mentions in ICSE Research Track abstracts increased from 2% (5 of 207) in 2023 to 45% (144 of 321) in 2026.
- Specialized human engineering that patches temporary AI deficits can become obsolete as model capabilities scale.
- AutoVerus resolved 20% of Rust verification problems, while Sonnet 4.5 solved 81% with a single straightforward prompt.The newer model achieved a four-fold performance improvement in the cited benchmark comparison.
- The study analyzes 35 ICSE 2026 LLM-based techniques against a single few-shot prompt automatically generated from each paper’s text.The baseline uses no iterative refinement, execution loops, external tools, or infrastructure.
- 37% to 63% of heavily engineered techniques published less than six months earlier could be replaced by the single-prompt baseline.Constructive tasks were most susceptible, whereas semantic understanding, feedback validation, and project-wide grounding remained more resistant.
- Resilient techniques enrich model inputs through domain-specific processing, symbolic integration, or structured context rather than temporary workarounds.
II. BACKGROUND
Prior work documents broad LLM adoption and increasingly sophisticated prompting, but this paper asks whether existing software engineering techniques survive newer models. It frames the comparison through three research questions and a standardized single-prompt methodology.
- II. BACKGROUND: LLMs are being applied across software engineering tasks including coding, testing, requirements engineering, repair, refactoring, and documentation.
- II. BACKGROUND: Prompting has evolved from zero- and few-shot strategies toward structured approaches such as chain-of-thought and tree-of-thought prompting.
- Research Questions: The paper asks whether current LLM-based techniques become obsolete as newer models emerge, because the answer determines their operational half-life.
- Research Questions: RQ1 compares single-prompt inference on state-of-the-art LLMs with recent software engineering tools for the same tasks.
- Research Questions: RQ2 examines which factors make techniques more amenable to single-call substitution, while RQ3 evaluates methodological guidance added to the prompt.
- Methodology: The methodology requires candidate papers, automated prompt generation, consistent execution, and evaluation against the original tools under matching conditions.
A. Pool of Candidate Papers
The candidate pool was narrowed from ICSE 2026 Research Track papers to studies with LLM-based automated techniques and evaluable public artifacts. The resulting prompts separate task inference from explicit methodological guidance.
- A. Pool of Candidate Papers: The selection process began with 321 ICSE 2026 Research Track papers.
- A. Pool of Candidate Papers: 85 papers without automated task-solving techniques were removed, including empirical studies, surveys, benchmarks, and human studies.
- A. Pool of Candidate Papers: Of the remaining 236 papers, 103 without an LLM were removed; 105 of 133 LLM papers had publicly available artifacts.
- Prompt Designs: Black-box prompts ask the LLM to infer task functionality from inputs and outputs, while white-box prompts additionally provide the original methodology.
- Prompt Designs: A meta-prompt generated paper-specific black-box and white-box prompts from each paper and its input/output examples.
- Prompt Designs: The meta-prompt was refined on five calibration papers for structural completeness and separation of methodological details, without testing target-task performance.
C. Prompt Executor
The executor replaces each candidate technique with one generated prompt and the paper’s inputs, while controlling cost through a fixed budget and random dataset sampling. The study also compares model-generation and prompt-structure choices.
- C. Prompt Executor: The executor invokes the LLM with each generated prompt and the candidate paper’s dataset.
- C. Prompt Executor: The meta-prompt receives a candidate paper and generates white-box and black-box prompts for the single-call approach.
- C. Prompt Executor: The executor uses the incomplete-source inputs supplied to the original technique.
- C. Prompt Executor: A $15-per-paper budget required random sampling because not every evaluation dataset could be fully analyzed.The budget covered the full dataset for 15 papers and 10% to 70% for 16 papers.
- C. Prompt Executor: The study selected Claude Sonnet 4.6 as a trailing-edge frontier model, partly because Opus models cost approximately 1.67x more per input and output token.
D. Evaluator
The study builds paper-specific evaluators to reproduce each technique’s metrics, ground truth, and execution conditions, then validates artifacts and categorizes papers by task and strategy.
- Each evaluator identified the paper’s end-to-end research question and reused its metrics for direct comparison.
- Ground truth came either from explicit dataset labels or reference outputs, or from executing and checking generated artifacts.
- The evaluation replicated each paper’s setup so generated outputs were assessed under the original conditions and metrics.
- The study created 35 paper-specific evaluators encoding metrics, ground-truth formats, execution environments, or external checking procedures.
- Artifacts were independently cross-reviewed, and papers were incrementally coded by mutually exclusive task and strategy categories.
V. RESULTS
The results section introduces the paper’s research-question findings.
- The authors introduce the results for each research question.
- The section is organized around research questions.
- The passage marks the transition into the reported findings.
A. RQ1: Single-Prompt LLM vs. Engineered SE Approaches
Across 35 studied papers, newer-LLM single prompts were competitive with engineered techniques and consistently outperformed them in 13 cases, with mixed results in 9 more.
- Table II compares paper-reported results with single-prompt replacements using each paper’s LLM, dataset, metric, and original result.
- 13 of 35 studied papers were consistently outperformed by single-prompt inference, while 9 additional papers produced mixed results.
- Between 37% and 63% of recent SE techniques could be effectively replaced by an automatically generated single prompt to a general-purpose LLM.
B. RQ2: Amenability to single-prompt LLM substitution
Single-prompt substitution varies by task and strategy: constructive code tasks are most replaceable, whereas approaches supplying domain knowledge or contextual information are harder to replace.
- Figure 6: Figure 6 scores each task-strategy cell from −1.0 to +1.0, where 0.0 represents an equal balance and n counts papers.
- Code Generation: Code generation single prompts outperformed techniques in six of nine papers, with every Search and Selection instance favoring the single prompt.
- Repair: Code repair single prompts outperformed three of six techniques and produced mixed results for two others, especially under Structured Reasoning.
- Task Categories: Single prompts struggled on Bug Finding, Impact Analysis, and other tasks requiring deeper semantic understanding or domain-specific information.
- Strategies: Search and Selection was the most replaceable strategy, with all three papers outperforming the original technique; Structured Reasoning had five positive, three mixed, and one negative result.
- Strategies: Domain-Specific Processing and Knowledge Grounding were hardest to replace because they encode or provide task-specific knowledge and additional context.
C. RQ3: The Value of White-Box Prompting
White-box prompts rarely improved on black-box prompts: performance was mixed, and added methodological guidance often increased complexity and cost without reliable benefit.
- Performance comparison: Pb outperformed Pw in 14 papers, Pw outperformed Pb in 12, and 9 produced mixed results across datasets or metrics.The comparison therefore showed no consistent advantage for including the original methodology.
- Cost and complexity: Pw used 80% more prompt tokens on average than Pb, and its average cost was $7.5 versus $6.5 for Pb.The resulting cost difference was approximately 15%.
- Strategy-level patterns: Pb outperformed Pw for Knowledge Grounding, Domain-Specific Processing, and Structured Reasoning, while Pw slightly led only for Feedback and Validation.The strategy-level comparison found Pb ahead in 5 of 8 Knowledge Grounding papers, 3 of 5 Domain-Specific Processing papers, and 3 of 9 Structured Reasoning papers; Pw led 4 of 9 Feedback and Validation papers.
- Performance comparison: 40% to 65% of studied SE tasks saw Pb match or outperform Pw, often because single prompts could not exploit technical insights without external tools.Descriptions of a procedure were insufficient when the model could not mimic or invoke components such as static analysis.
D. Threats to Validity
The study’s conclusions are constrained by limited representativeness, best-effort replication, coarse cross-paper classifications, and a small heterogeneous sample that prevents significance testing.
- External validity: The ICSE-based sample may not generalize to the breadth of software engineering or broader community practices, and model access changes may further affect generalizability.The authors document selection criteria and use contemporary, widely accessible models to mitigate these concerns.
- Internal validity: Replication across 35 diverse research groups is necessarily best-effort despite standardized prompts, execution setups, evaluation pipelines, and researcher cross-checking.Some evaluations were also bounded by budget constraints and sampling, while benchmark exposure during model training could inflate performance.
- Construct validity: The outperforming, mixed, and underperforming categories are coarse constructs that may miss smaller patterns and technique-specific qualitative differences.The paper provides core data and repository materials for further scrutiny.
- Conclusion validity: Paper and evaluation heterogeneity plus the small sample prevent statistical significance testing, so findings are presented as indicative trends rather than definitive causal relationships.The authors explicitly temper their conclusions accordingly.
VI. CONCLUSION
The study systematically examines the longevity of LLM-based software engineering techniques and finds that many recent complex techniques can be replaced by a single prompt on a newer model. Constructive tasks are especially vulnerable, while domain-specific processing and project-wide knowledge grounding are more likely to survive.
- Study scope: 35 ICSE 2026 software engineering papers are analyzed to investigate the longevity and strategic value of LLM-based techniques.The framework is described as automated, systematic, and reproducible.
- Limitations: The study acknowledges fixed execution budgets, dataset sampling constraints, and categorization schemas as limitations, and presents itself as a first step.These limitations bound interpretation of the reported longevity comparisons.
- Replacement findings: 37%–63% of recently published SE techniques can be replaced by a single automatically generated prompt executed on a newer model without iterative refinement or other tools.The baseline represents the simplest possible LLM invocation.
- Implications: The findings warn that heavily engineered LLM workarounds can have an ephemeral nature and short half-life.The authors frame this as a strategic concern for software engineering research investment.
- Replacement findings: Constructive techniques for generating or repairing code are easiest for increasingly powerful frontier models to replace.Techniques requiring deeper semantic understanding or feedback loops still fare better under the evaluated baseline.
- Surviving strategies: Domain-specific processing and project-wide knowledge grounding are least likely to be replaced by the single-prompt baseline.These strategies provide information or context beyond the bare invocation.
EVALUATED PAPERS
The evaluated corpus comprises ICSE 2026 papers proposing LLM-based techniques across code generation, testing, debugging, verification, analysis, security, and related software engineering tasks.
- Analysis and verification: The set covers verification, requirements formalization, program dependence analysis, correctness reasoning, and log-related software engineering tasks.Named examples include Auto-related verification work, distinguishing traces, partial dependence analysis, structural correctness reasoning, log parsing, and anomaly detection.
- Construction and modeling: The corpus includes papers on LLM-based code generation, repair, completion, and behavioral model generation.Examples include ensemble-based generation, hierarchical debugging, cross-language repair, API argument completion, and behavioral model generation.
- Testing and fuzzing: Several evaluated papers address testing and fuzzing through oracle mining, regression-test generation, smart-contract fuzzing, and hybrid program analysis.The titles include RBCTest, Testweaver, EchoFuzz, and LLM test generation via iterative hybrid program analysis.
- Security and diagnosis: Security-focused papers in the corpus study secure code review, vulnerability detection, signature replay, SQL dialect translation, and root-cause analysis.The listed techniques use fine-tuning, agent-based mock courts, vulnerability analysis, query reduction, and structured deep thinking.