PaperlayerView exact revision on arXiv

Source-linked AI summary

Aging of Prompt Engineering Techniques Across LLM Versions

Anastasiia Rudyk, Julian Oertel, Regina Hebig

arXiv:2608.24641v1cs.SE

TL;DR

It remains unclear how prompt-engineering effectiveness changes across successive generations of different LLM families. This study compares five techniques across three model-version pairs for function-level Python code generation and finds family-specific aging, with prompting benefits diminishing for newer GPT models but persisting for Qwen and some Mistral techniques.

  • Problem

    Evidence remains limited on how prompt-engineering effectiveness evolves across successive versions of different LLMs, despite its importance for software-engineering code quality.

  • Method

    The study evaluates five prompting techniques across six instruction-tuned models in three version pairs using 218 cleaned CodePromptEval Python tasks and pass@k functional correctness.

  • Results

    Prompt engineering ages by model family: newer GPT models show marginal or negative structured-prompting gains, Qwen benefits from Few-Shot and CCoT, and Mistral shows mixed effects.

  • Takeaways & Limitations

    Prompting strategies should be re-evaluated for each model family and generation rather than transferred unchanged.

  • Takeaways & Limitations

    Mistral-Large has approximately 123B parameters versus 7B for Mistral-7B-Instruct, confounding comparisons between training or architecture improvements and parameter count.

Abstract

from arXiv · show

Prompt engineering and prompt engineering techniques (PETs) have become an integral part of software engineering for AI systems. However, new LLMs are released frequently and it remains unclear how the effectiveness of prompt engineering techniques changes across successive generations of Large Language Models (LLMs). To this end, we conduct a partial replication of the study by Khojah et al. (2025). We evaluate five techniques - Zero-Shot, Few-Shot, Chain-of-Thought (CoT), Contrastive Chain-of-Thought (CCoT), and an adapted version of Program-of-Thought (PoT) - on six instruction-tuned models grouped into three version pairs: GPT-3.5-Turbo/GPT-4o, Qwen2 7B Instruct/Qwen2.5 7B Instruct, and Mistral-7B-Instruct/Mistral-Large. We use a cleaned subset of the CodePromptEval dataset with 218 context-rich Python functions and 19,620 total generations assessed via pass@k-based functional correctness to evaluate model pairs on function-level code generation tasks. We show that prompt engineering "ages" in a model-family-specific way: Newer GPT models exhibit diminishing or even negative marginal gains from structured prompting, suggesting that instruction-following and reasoning scaffolds are increasingly internalized, whereas Qwen models continue to benefit substantially from Few-Shot and CCoT. Mistral models show mixed behavior with persistent gains from CCoT but attenuated benefits from CoT and PoT. Our results imply that effective prompting strategies must be adapted per model family and generation rather than transferred unchanged. This motivates future work on adaptive, model-aware prompting and broader, multi-dimensional code quality evaluation.

I. INTRODUCTION … 1) Established Benchmarks and Pass@k Metrics:

The paper examines how prompt-engineering effectiveness changes across successive LLM generations, situating this problem within evolving code-generation benchmarks and pass@k-based evaluation. It builds on prior prompting research and established functional-correctness benchmarks while comparing newer prompting strategies across model families.

  • I. INTRODUCTION: Prompt engineering directs LLM behavior through strategic textual inputs without changing model parameters, supporting higher-quality automated code generation.Its importance follows from the increasing use of LLMs in software engineering, particularly automated code generation.
  • I. INTRODUCTION: Frequent releases of improved LLM generations have intensified comparisons across architectures and tasks, while leaving cross-version prompt-effectiveness changes insufficiently analyzed.The paper identifies systematic analysis across successive versions and different LLMs as a notable research gap.
  • I. INTRODUCTION: Prior work studied prompting effects on code quality and performance, but the present study focuses on how techniques evolve across successive LLM generations.The motivation includes instability and sensitivity to input format in software-engineering tasks.
  • I. INTRODUCTION: Khojah et al.’s CodePromptEval contains 7,072 prompts testing five PETs across project-level code-generation tasks and measuring correctness, similarity, and code quality.The examined models included GPT-4o, Llama3-70B-Instruct, and Mistral-22B-Instruct, with GPT-3.5-Turbo and Llama2-7B-Instruct not detailed.
  • I. INTRODUCTION: This study modifies CodePromptEval to compare prompting across GPT-3.5-Turbo, GPT-4o, Qwen2/Qwen2.5 7B Instruct, and Mistral-7B-Instruct/Large.It replaces signature and persona with CCoT and adapted PoT, focuses on individual effects, and uses different non-GPT models because of cost constraints and comparison goals.
  • II. RELATED WORK: The related-work review follows a systematic funnel from foundational LLM evolution through evaluation methodologies and benchmarking practices.This positions the study within the broader academic landscape.
  • A. Benchmarking and Evaluation Methodologies: Rigorous evaluation frameworks are necessary for assessing LLM code-generation capabilities, motivating benchmarks that emphasize different synthesis aspects and address earlier limitations.Multiple benchmarks have been developed for these complementary evaluation goals.
  • 1) Established Benchmarks and Pass@k Metrics:: HumanEval includes 164 hand-crafted programming problems with function signatures, docstrings, and unit tests for functional-correctness assessment.Its evaluation uses pass@k with a novel unbiased estimator designed to reduce sampling variance because multiple samples can outperform single attempts.

2) Context-Dependent Function Generation: … 3) Causality-Based Analysis and Linguistic Features:

Prior research shows that code-generation prompting is context-, model-, version-, and task-dependent, with advanced reasoning models often needing less structured guidance. Adaptive prompt selection can improve accuracy while reducing token use, supporting model- and complexity-aware prompting.

  • 2) Context-Dependent Function Generation:: CoderEval introduced 460 context-dependent Python and Java tasks, reflecting realistic development scenarios where over 70% of functions depend on external context.The benchmark emphasizes non-standalone functions invoking project-specific APIs.
  • 1) Model-Dependent Effectiveness:: Prompt-engineering outcomes fluctuate markedly across language models and prompting methods, with more capable models requiring less explicit guidance.Murr et al. tested four models across over 100 problems and various prompt structures.
  • 1) Model-Dependent Effectiveness:: ChatGPT produced competent functional code but showed deficiencies in efficiency, security, and standalone usability, often requiring human intervention.These findings qualify the promise of generating code with minimal user effort.
  • 1) Model-Dependent Effectiveness:: Across 7,583 Dev-GPT code files, Zero-Shot, Few-Shot, CoT, and Persona prompts produced few overall quality problems and no notable metric variations.The analysis used Kruskal-Wallis tests and evaluated maintainability, security, and reliability.
  • 2) Diminishing Returns in Advanced Models:: Reasoning-native models often match or exceed structured-prompt performance with zero-shot instructions, unlike non-reasoning models that still benefit substantially from structured prompts.Wang et al. observed this pattern across code generation, translation, and repair tasks when comparing GPT-4o with o1-mini.
  • 2) Diminishing Returns in Advanced Models:: 16.67% was o1-mini’s advantage over GPT-4o on problems with chains of thought exceeding five steps, versus 2.89% on simpler problems.The results indicate that reasoning-native advantages increase with programming-task logical complexity.
  • 3) Causality-Based Analysis and Linguistic Features:: Causal analysis found that linguistic prompt features, including formal versus fluent phrasing and specific keywords, create trade-offs in generated code quality.The analysis covered GPT-Neo, GPT-3.5, and GPT-4 and showed version-dependent sensitivity to prompt design.
  • 3) Causality-Based Analysis and Linguistic Features:: PET-Select improved pass@1 accuracy by 1.9% while reducing token usage by 74.8% through complexity-based automatic prompt selection.The study reports that adaptive selection outperformed static approaches.

III. DESCRIPTION OF THE ORIGINAL STUDY … D. Analysis Procedure

The original study introduced CodePromptEval to evaluate how LLMs and prompting techniques affect Python code generation. It measured correctness, similarity, and quality using generated solutions, established metrics, and descriptive and inferential analyses.

  • A. Data Set: CodePromptEval: CodePromptEval adapts CoderEval’s repository-derived tasks, each containing a docstring or comment, function signature, source code, and associated tests.CoderEval includes 230 Python and 230 Java functions; task difficulty reflects code dependencies.
  • A. Data Set: CodePromptEval: 221 Python tasks with nonfailing ground truth were selected, and 32 prompts were created for each PET combination alongside a zero-shot baseline.The zero-shot prompt served as the evaluation baseline.
  • A. Data Set: CodePromptEval: Five LLMs generated 3 solutions per prompt at temperature 0.2: GPT-4o, Llama3-70B-Instruct, Mistral-Small-Instruct-2409, GPT-3.5-turbo, and Llama2-7B-Instruct.Mistral-Small-Instruct-2409 is identified as a 22B model.
  • B. Research Questions: The study asked how different LLMs perform on CodePromptEval and how prompting techniques and their combinations affect code generation.The second research question was divided into code correctness, similarity to human-written code, and code quality.
  • C. Prompting Techniques: The six evaluated PETs were zero-shot, few-shot, persona, chain-of-thought, signature, and packages, with zero-shot used as the uncombined baseline.Few-shot used two input-output examples, while every prompt included a shared extraction constraint for automation.
  • D. Analysis Procedure: Correctness was measured with pass@k on the original CoderEval tests, similarity with CrystalBLEU, and quality with Pylint, McCabe complexity, and cognitive complexity.These metrics were extracted from the generated functions and compared against the human-written baseline where applicable.
  • D. Analysis Procedure: The analysis used descriptive metrics for RQ1 and added regression models and statistical tests for RQ2 and its sub-research questions.The inferential analyses addressed the effects of prompting techniques and combinations across the specified outcomes.

E. Results … 2) Prompt Engineering Techniques:

The replicated study retained five prompt engineering techniques while cleaning the dataset to 218 tasks and expanding the prompt set to 1,090. Prior findings indicate that Few-Shot and signature had the clearest correctness effects, whereas additional information could worsen correctness and alter code smells, similarity, and complexity.

  • E. Results: GPT-4o slightly outperformed the other studied LLMs, while all models solved between 31% and 90% of tasks depending on difficulty.
  • E. Results: Signature and Few-Shot had the clearest impact on correctness, while the other prompt techniques showed no statistically significant impact.
  • E. Results: CoT, persona, and package decreased code smells but also reduced general correctness, while signature and persona increased similarity to human-written code.
  • IV. REPLICATION STUDY DESIGN: The replication study outlines its procedure and identifies differences from Khojah et al., with the replication scope summarized in Table I.
  • A. Prompt Data Set: The study’s data-acquisition procedure is summarized in Figure 2.
  • 1) Cleaning of Original Data Set:: Removing three CodePromptEval tasks that violated few-shot input-output formatting reduced the dataset from 221 to 218 tasks.
  • 2) Prompt Engineering Techniques:: The study used Zero-Shot, Few-Shot, CoT, CCoT, and an adapted PoT, with the first three retained from the original study.
  • 2) Prompt Engineering Techniques:: CCoT combined Few-Shot and CoT positive examples with 2–3 incorrect demonstrations created by shuffling core reasoning objects, while adapted PoT instructed models to use code for all reasoning steps.The shuffled objects included key variables, numbers, function names, and the constants None, True, and False. The study extracted 654 prompts and created 436 additional prompts, totaling 1,090.

B. Generated Data Set · 1) LLM Selection:

The study’s generated-data procedure used systematically selected LLMs for code generation and evaluation. Models were chosen for cutoff-date eligibility, coding suitability, and traceable successive versions across three model families.

  • B. Generated Data Set: The generated-data section presents the LLMs and procedure used to generate and evaluate code from the prompts.Figure 3 outlines the general procedure.
  • 1) LLM Selection:: Models were selected using training cut-off dates, code-generation suitability, and version traceability.These criteria support transparent comparisons across successive model versions.
  • 1) LLM Selection:: Training data cutoffs had to precede the CodePromptEval source release in February 2024, although GitHub-data leakage could not be ruled out completely.The dataset derives from CoderEval, and prior construction work attempted to mitigate leakage risks.
  • 1) LLM Selection:: The GPT models use September 2021 and October 2023 cutoffs, respectively, while Qwen2/Qwen2.5 and Mistral models were associated with October 2023 and December 2023 cutoffs.The Qwen cutoff was adopted from Qwen3-related evidence, whereas the Mistral 7B date came from community sources.
  • 1) LLM Selection:: The selected models comprise GPT-3.5 Turbo, GPT-4o, Qwen2 7B Instruct, Qwen2.5 7B Instruct, Mistral-7B-Instruct, and Mistral-Large.Their documented or reported coding, reasoning, instruction-following, and code-synthesis capabilities supported inclusion.
  • 1) LLM Selection:: Successive versions were required to share core architecture and task orientation while incorporating incremental updates in data, hyperparameters, or scale.GPT-3.5-Turbo/GPT-4o, Qwen2/Qwen2.5, and Mistral-7B-Instruct/Mistral-Large were treated as coherent within-family comparisons.

2) Generation: · 3) Code Extraction:

The study generated responses through model-specific APIs under standardized sampling and token limits, then extracted code automatically where possible. Generation failures affecting 23 prompts led to excluding those prompts and reducing the evaluation set to 17,550 generations across 195 tasks.

  • 2) Generation:: GPT-3.5-Turbo, GPT-4o, Qwen2.5 7B Instruct, Mistral-7B-Instruct, and Mistral-Large used OpenRouter, while Qwen2 7B Instruct used Replicate.Mistral-7B-Instruct prompts were wrapped with the official [INST] and [/INST] tokens.
  • 2) Generation:: All models used temperature 0.2 and a maximum limit of 1024 tokens per generation attempt.The temperature matched the original study, while the token limit was additionally applied.
  • 2) Generation:: 19,620 responses were produced from 1090 prompts over three runs.This total reflects the initial generation procedure before exclusions during code extraction.
  • 3) Code Extraction:: Code was automatically extracted from markdown formatting, with manual extraction attempted when formatting-based extraction failed.The procedure targeted relevant code within each response.
  • 3) Code Extraction:: 23 prompts experienced problems in at least one generation attempt, mainly because responses lacked code.Other causes included token-limit cutoffs and endlessly repeated output.
  • 3) Code Extraction:: All generation attempts for the 23 problematic prompts were excluded from evaluation.This exclusion addressed failures caused by missing code, cutoffs, or repeated output.
  • 3) Code Extraction:: 17,550 generations remained in the final dataset, corresponding to 195 effective evaluation tasks.The exclusions reduced both the number of generations and the effective number of tasks.

4) Code Evaluation: … B. Individual Output Pass Rates

The study evaluates five prompt engineering techniques across LLM versions using pass@k-focused code correctness analysis and compares selected replication results with prior findings. Individual-output results show stronger baselines for newer models, with GPT-4o achieving the highest pass@1 rate and Qwen2 7B the lowest.

  • 4) Code Evaluation:: The evaluation preserves the original CoderEval Docker environments and applies the CodePromptEval methodology to generated code.These choices maintain consistency with the original evaluation setup.
  • C. Analysis Procedure: The analysis focuses on descriptive statistics for pass@k because code correctness is the primary evaluation target.The procedure follows the original approach outlined in Section III-D.
  • V. RESULTS: The results report empirical findings from evaluating five prompt engineering techniques across LLM versions.The study examines how these techniques perform across successive model generations.
  • A. Comparison to Original Results: The replication modifies components of the original study while retaining GPT-4o and some techniques, enabling direct comparison with Khojah et al.The comparison uses data revisited from the original replication package and presents original results after task exclusions.
  • A. Comparison to Original Results: 0.07 percentage points is the average difference between the original and reduced original datasets, with a maximum zero-shot difference of 0.2 percentage points.The reduced dataset comparison therefore produces only small changes for the zero-shot technique.
  • B. Individual Output Pass Rates: Table IV reports cumulative task pass counts and percentages for each individual output run across all models, without ensemble effects.These rates measure direct functional correctness for single generated outputs.
  • B. Individual Output Pass Rates: 43.38% is GPT-4o’s best pass@1 result, while Qwen2 7B records 35.59%, creating a 7.79 percentage point gap.All GPT variants exceed 42% pass rates across the three output runs, and within-model variation is typically within ±0.5–2%.

C. Pass@k Rates · D. Per-Run Technique Effectiveness · E. Pass@k Progression and Per-Run Technique Deltas

Pass@k results show that stronger models have higher pass@1 performance but less benefit from repeated sampling, while Qwen2.5 and Mistral-7B gain more across attempts. Per-run and successor-model analyses define technique deltas relative to Zero-Shot and compare progression across model families and pass@k levels.

  • C. Pass@k Rates: Qwen2.5 gains +5.95% from pass@1 to pass@3, the largest absolute improvement across the pass@k progression.The analysis identifies Qwen2.5 as benefiting most from multi-attempt sampling.
  • C. Pass@k Rates: Pass@1 baselines range from 42.36% to 43.38% for GPT-3.5-Turbo, GPT-4o, and Mistral-Large, versus 35.59% to 36.92% for Qwen2 and Mistral-7B.The 6–8 percentage point gap is consistent with instruction-tuning and scale advantages.
  • C. Pass@k Rates: GPT-4o improves only +1.13% from pass@1 (43.38%) to pass@3 (44.51%), indicating ensemble-performance saturation.The small gain is attributed to advanced models solving reachable tasks successfully on the first attempt.
  • C. Pass@k Rates: Mistral-7B improves by 3.90 percentage points from pass@1 to pass@3, showing more noticeable gains than advanced models.The result contrasts with GPT-4o’s limited multi-attempt improvement.
  • D. Per-Run Technique Effectiveness: Per-run deltas are reported relative to Zero-Shot prompting, with absolute pass@1 rates for Zero-Shot, Few-Shot, CoT, CCoT, and PoT.The rates represent passed functions across 195 generated functions per run, and Zero-Shot is abbreviated ZS.
  • D. Per-Run Technique Effectiveness: Bold values in the per-run effectiveness analysis indicate performance changes with |∆| > 10%, while the run column identifies the output number.The analysis averages results across three independent runs.
  • D. Per-Run Technique Effectiveness: Table VII aggregates average performance deltas from Table VI to characterize pairwise model progression within each model family.These deltas summarize successor-versus-baseline effects derived from the per-run results.
  • E. Pass@k Progression and Per-Run Technique Deltas: Table VIII compares successor-model improvements over their respective base models across pass@1, pass@2, and pass@3.This extends successor-model analysis beyond a single pass@k level.

F. Prompt Engineering Effectiveness Across Model Generations … G. Answering RQ1

Prompt-engineering effectiveness changes systematically across generations in a model-family-dependent manner. GPT shows saturation or degradation, Qwen retains substantial gains, and Mistral combines declining generic reasoning benefits with persistent CCoT and few-shot gains.

  • 1) Few-Shot Learning: Pass@1 Gains Saturate at Higher Sampling:: Qwen2.5 and Mistral-Large improve over predecessors on few-shot averaged deltas (+7.9 and +0.8 percentage points), while GPT-4o declines (-7.4).Few-shot has positive absolute performance in five of six baseline cases, except Qwen2 (-5.5).
  • 1) Few-Shot Learning: Pass@1 Gains Saturate at Higher Sampling:: +11.4% few-shot improvement for Qwen2.5 at pass@1 declines to +7.9% at pass@3, while Mistral-Large falls from +2.8% to −2.1%.GPT-4o degrades consistently across k levels, from −4.9% to −6.3%, indicating few-shot examples may interfere with multiple sampled attempts.
  • 2) CoT: Structural Degradation Across Pass@k:: Standard CoT yields averaged deltas of -13.8% for Mistral-Large, -0.8% for GPT-4o, and +1.9% for Qwen2.5.Mistral-Large worsens to −12.9% at pass@3, whereas GPT-4o remains marginal and stable at −1.2% for pass@1/2 and 0.0% at pass@3.
  • 3) CCoT: Robust Consistency Across Averaging and Pass@k:: CCoT improves all three model pairs: Qwen2.5 (+11.5%), Mistral-Large (+5.6%), and GPT-4o (+1.1%).Qwen and Mistral improvements remain stable across pass@k, ranging from +5.2% to +13.4%, while GPT-4o declines −1.3% to −3.8% at higher k.
  • 4) Program-of-Thought: Architecture-Dependent Instability:: PoT produces +4.2% for GPT-4o, -2.3% for Qwen2.5, and -12.1% for Mistral-Large in averaged deltas.Despite improving relative to GPT-3.5, GPT-4o remains negative in absolute PoT performance across all pass@k levels.
  • G. Answering RQ1: For GPT, newer models show only modest gains or degradation from CoT, PoT, CCoT, and few-shot prompting.The findings attribute this reduced marginal utility to increasingly internalized instruction-following and reasoning capabilities.
  • G. Answering RQ1: Qwen models remain substantially responsive to explicit prompting across generations, with CoT, PoT, CCoT, and few-shot techniques frequently producing stable or increasing baseline gains.Model evolution does not fully subsume the benefits of explicit prompting for Qwen.
  • G. Answering RQ1: Mistral shows mixed aging: CoT and PoT become neutral or negative, while CCoT and few-shot prompts retain measurable gains, especially with concrete examples.These differential patterns indicate that prompt-engineering aging reflects distinct training regimes and architectural choices across model families.

VI. DISCUSSION … 2) Comparison to Prompt Technique Studies:

The discussion finds close alignment with prior CodePromptEval results while showing lower aggregate performance and model-specific prompt-technique effects. Few-Shot remains robust, CCoT is consistently beneficial, whereas CoT and PoT can degrade performance depending on the model and task.

  • VI. DISCUSSION: The discussion frames the section as an analysis of results and threats to validity.
  • A. Comparison with Prior Work: The comparison with prior prompt-technique studies confirms that prompt-engineering effects are strongly technique- and model-dependent.Few-Shot generally provides the most robust improvements, especially for smaller or less capable models, while advanced techniques have mixed effects.
  • 1) Alignment with CodePromptEval Results:: Overall pass@1 levels and model rankings resemble Khojah et al.’s full CodePromptEval results despite the reduced task set and different model mix.Using the same 195 replication tasks also produced very similar pass@1 rates, although GPT-4o was lower by an average of 6.5 percentage points.
  • 1) Alignment with CodePromptEval Results:: Aggregated pass rates were below Khojah et al.’s approximately 47–52% for GPT-4o and Mistral, partly because different Mistral variants were evaluated.The study used Mistral-Large-2407 (123B), whereas Khojah et al. used Mistral-Small-Instruct-2409 (22B), and omitted the successful Signature technique.
  • 1) Alignment with CodePromptEval Results:: Qwen2 and Qwen2.5 achieved 35.59% and 36.41%, respectively, aligning with Mistral-7B’s 36.92% and remaining below the models used by Khojah et al.
  • 2) Comparison to Prompt Technique Studies:: Few-Shot produced double-digit average gains for Mistral-7B and GPT-3.5, whereas standard CoT often degraded performance, particularly for Mistral-Large and GPT-4o.Prior work reports that structured CoT variants such as SCoT can improve pass@1 by up to 13.79% on code benchmarks.
  • 2) Comparison to Prompt Technique Studies:: CCoT had a consistently positive effect across all three model families, consistent with using explicit positive and negative reasoning examples as discriminative signal.
  • 2) Comparison to Prompt Technique Studies:: PoT showed architecture-dependent instability, yielding moderate gains for some models but substantial degradation for Mistral-Large and negative or near-zero absolute performance in many pass@k settings.This contrasts partly with findings that PoT can outperform CoT on numerical reasoning tasks, which differ from context-dependent code generation.

3) Relationship to Advanced-Model Prompt Engineering Studies: · B. Possible Reasons for Divergent Results

The study aligns with evidence that advanced proprietary models internalize many traditional prompt-engineering benefits, while Qwen retains substantial gains from explicit prompting. Divergent results may reflect differences in models, tasks, technique constraints, model-size pairings, and decoding configurations.

  • 3) Relationship to Advanced-Model Prompt Engineering Studies:: GPT-4o and Mistral-Large show diminishing or negative gains from Few-Shot and CoT, with modest improvements from CCoT.This pattern mirrors evidence that advanced instruction-tuned and reasoning models internalize traditional prompt-engineering benefits.
  • 3) Relationship to Advanced-Model Prompt Engineering Studies:: GPT-4o’s pass@k gains from Few-Shot and CoT are small or negative, while contrastive explanations provide incremental improvements.Techniques that add genuinely new information remain beneficial in function-level code generation.
  • 3) Relationship to Advanced-Model Prompt Engineering Studies:: Qwen2.5 shows large positive deltas for Few-Shot and CCoT across pass@k levels, indicating that newer models do not share one prompt-saturation regime.Traditional prompt engineering may remain crucial when instruction tuning and training data are less extensive than proprietary counterparts.
  • B. Possible Reasons for Divergent Results: The six-model set spans three families and different training cut-off dates, unlike prior evaluations often centered on one or two proprietary models and older open-source baselines.Training-data recency, instruction-tuning regimes, and internal architecture may amplify family-specific prompt sensitivity.
  • B. Possible Reasons for Divergent Results: Context-dependent CodePromptEval functions require integration with project APIs and existing code, unlike many HumanEval, MBPP, and synthetic reasoning tasks.Techniques effective on small, self-contained problems may be less effective or harmful when tasks require fitting into partially specified systems.
  • B. Possible Reasons for Divergent Results: All PETs use single-pass prompts without iterative feedback, although iterative or critique-based prompting can improve correctness and reduce vulnerabilities.Iterative CoT or refinement prompts could recover performance lost by static CoT and PoT.
  • B. Possible Reasons for Divergent Results: The Mistral comparison pairs a 7B instruct model with the much larger Mistral-Large because no comparable intermediate open-access model was available.This prevents cleanly separating architectural evolution, training changes, and parameter-count effects.
  • B. Possible Reasons for Divergent Results: Together, these factors explain broad agreement with prior work on diminishing returns for advanced proprietary models while exposing less-visible Qwen and Mistral family-specific behavior.The study also confirms the importance of Few-Shot and specialized CoT variants.

C. Threats to Validity … VII. CONCLUSION

The study’s validity is constrained by dataset scope, limited sampling, Mistral comparison confounds, and pass@k’s narrow view of code quality. Across the evaluated models, prompting effectiveness varies by family and maturity, so strategies require re-evaluation across generations.

  • 1) External Validity:: Results may not generalize beyond CodePromptEval’s 218 Python function-synthesis tasks to other languages, full programs, or production development.Model training cutoff dates were also not always transparently reported.
  • 2) Internal Validity:: Only 3 independent outputs per prompt-model combination enabled pass@1-3 but limited statistical power for characterizing nondeterminism.Larger samples, such as 10 runs, would provide more robust confidence intervals.
  • 2) Internal Validity:: Mistral-7B-Instruct required [INST] brackets whereas Mistral-Large processed raw prompts, introducing methodological inconsistency that could influence relative gains.The adaptation was necessary for fair evaluation but makes the comparison less uniform.
  • 2) Internal Validity:: Mistral Large’s approximately 123B parameters versus Mistral-7B-Instruct confounds version effects with sheer parameter count.The mismatch complicates separating improved training or architecture from scale-related performance differences.
  • 4) Conclusion Validity:: Conclusion validity was assessed through functional correctness and relative improvement over the zero-shot baseline, with every technique compared against that baseline.Additional dimensions such as readability, efficiency, and security remain possible extensions.
  • VII. CONCLUSION: The study evaluated five techniques across six instruction-tuned models in three version pairs using 218 Python tasks and 19,620 generations.Evaluation used pass@k functional correctness on a cleaned CodePromptEval subset.
  • VII. CONCLUSION: No single prompting strategy transfers unchanged across generations; PET utility depends on model family and maturity, requiring re-evaluation when adopting new models.Some techniques lose relevance as models advance, while others remain crucial.
Loading 2608.24641v1…