PaperlayerView exact revision on arXiv

Source-linked AI summary

Compound Prompt Constraints in LLM Code Generation: A Factorial Study of Format, Persona, and Urgency

Shrenik Jadhav, Nickalsa LaPlaca, Caleb Stone, Ashok Raja, Omar Ochoa, Vidhyashree Nagaraju

arXiv:2609.03156v1cs.SE

TL;DR

Prompt constraints are often studied one at a time even though deployed code-generation prompts combine them, leaving their joint reliability effects uncertain. This paper tests format, persona, and urgency factorially across multiple model families and finds architecture-dependent outcomes, from GPT-4o degradation to GPT-4.1 resistance and o3-mini improvement under structured constraints.

  • Problem

    Prompt factors have mostly been evaluated individually, leaving their simultaneous effects and architecture dependence uncharacterized despite their routine combination in production prompts.

    Source: §I · p. 2

  • Method

    A 3×3×3 full-factorial study evaluates format, persona, and urgency across 164 HumanEval+ problems and five OpenAI models, decomposing compound conditions into additive predictions and residual interactions.

    Source: §I · p. 2, D. Factorial Design and Interaction… · p. 3, §V · p. 8

  • Results

    Across architectures, GPT-4o shows 3–12 percentage-point super-additive pass@1 losses, while GPT-4.1 resists the effect and o3-mini improves under structured output constraints.

    Source: §I · p. 2, §V · p. 8

  • Takeaways & Limitations

    Single-factor evaluations cannot reliably predict compound-prompt behavior, so reliability assessment should consider compound constraints and model selection.

    Source: §I · p. 2, §V · p. 8

    The study covers five OpenAI models on one benchmark of function-level Python problems under greedy decoding, with prompt token length confounded with constraint intensity.

    Source: §V · p. 8

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used in software engineering pipelines for code generation, where production prompts often combine multiple constraints. This paper presents a full-factorial empirical study of how output formatting, persona assignment, and urgency framing jointly affect LLM code-generation reliability. We evaluate all 27 combinations in a controlled 3x3x3 design and decompose each compound condition into an additive prediction and a residual interaction term that captures super-additive degradation. The study uses all 164 HumanEval+ problems across five OpenAI models from the GPT-4o family, GPT-4.1 family, and o3-mini, yielding 22,140 greedy-decoding evaluations. A format-aware extraction pipeline separates formatting failures from reasoning failures, and significance is assessed with McNemar's test, odds ratios, and 95% confidence intervals. Results show that compound constraints can produce architecture-dependent degradation not predictable from single-factor experiments. The GPT-4o family exhibits consistent super-additive effects, with pass@1 reductions 3-12 percentage points beyond additive predictions; the largest interaction is -12.2 pp on GPT-4o-mini for JSON + expert persona + moderate urgency. JSON combinations generally produce larger interactions than XML. In contrast, the GPT-4.1 family is largely resistant, while o3-mini shows a qualitatively different pattern in which structured output constraints can improve performance. These findings show that vulnerability is architecture-dependent rather than size-dependent, that individually neutral or beneficial constraints can combine to cause substantial degradation, and that compound-prompt testing should be standard in reliability assessment for LLM-assisted engineering pipelines.

I. INTRODUCTION

The introduction motivates testing prompt constraints jointly because production code-generation prompts combine them, while existing single-factor studies cannot establish their compound effects. It presents a factorial evaluation and reports architecture-dependent outcomes.

I. INTRODUCTION

  • The literature studies formatting, persona, and urgency mostly in isolation, although production code-generation prompts routinely combine these constraints.Prompt phrasing itself can measurably shift correctness, and the architecture sensitivity of combined constraints is also unknown.Source: §I · pp. 1–2
  • A 3×3×3 factorial study tests output format, persona, and urgency across 164 HumanEval+ problems and five OpenAI models, totaling 22,140 evaluations.The models span the GPT-4o family, GPT-4.1 family, and o3-mini.Source: §I · p. 2
  • Each triple-constraint condition is decomposed into an additive prediction and residual interaction, with format-aware extraction and paired McNemar tests, odds ratios, and 95% confidence intervals.The extraction pipeline separates formatting failures from reasoning failures.Source: §I · p. 2
  • GPT-4o shows 3–12 pp super-additive pass@1 losses, peaking at −12.2 pp on GPT-4o-mini, while GPT-4.1 resists and o3-mini improves under structured constraints.JSON combinations produce larger interaction effects than XML-based combinations.Source: §I · p. 2
  • Individually neutral or beneficial constraints can combine into degradation invisible to single-factor tests, supporting compound-prompt evaluation and model selection in deployment reliability.The analysis also contrasts JSON with XML and tracks prompt token length as a confound.Source: §I · pp. 2–3

II. BACKGROUND

This background introduces the framework’s core techniques and treats code-capable LLMs as prompt-conditioned generators whose reliability is evaluated through their outputs.

II. BACKGROUND

  • The background section introduces concepts used to elicit code from LLMs and quantify the reliability of generated outputs.Source: §II · p. 3

A. Large Language Models for Code Generation

  • Code-capable LLMs generate programs autoregressively, predicting one token at a time from prompts containing a task description, function signature, and optional instructions.Source: A. Large Language Models for Code… · p. 3
  • The framework treats LLMs as black-box code generators, with system messages, user messages, and decoding parameters as the controllable inputs available to practitioners.Source: A. Large Language Models for Code… · p. 3

B. Prompt Constraints

The section defines the prompt constraints studied and the HumanEval-based benchmark and metric used to assess code-generation reliability.

B. Prompt Constraints

  • The study’s three prompt-constraint categories are output format, persona, and urgency framing.Format specifies response structure, persona assigns a role, and urgency adds motivational language.Source: B. Prompt Constraints · p. 3
  • JSON or XML instructions constrain output structure for downstream parsing, and prior evidence reports possible reasoning-quality degradation from prompt-enforced formats.Source: B. Prompt Constraints · p. 3
  • Persona instructions assign an identity such as developer or domain expert, while urgency prefixes add high-stakes or time-pressure language, [8], [9].The cited evidence reports mixed persona effects and measurable output changes from urgency framing.Source: B. Prompt Constraints · p. 3

C. HumanEval+ and the pass@k Metric

D. Factorial Design and Interaction Effects

The factorial design estimates three-way interactions as departures from additive predictions, while paired outcomes support McNemar testing. The methodology applies this analysis within a five-stage evaluation framework.

D. Factorial Design and Interaction Effects

  • A three-way interaction is the residual between the actual compound-condition outcome and the baseline plus individual main effects.A negative interaction indicates super-additive degradation: the compound constraint performs worse than the additive prediction.Source: D. Factorial Design and Interaction… · p. 3

E. McNemar’s Test for Paired Binary Outcomes

  • McNemar’s test compares paired pass/fail outcomes across conditions, with 95% confidence intervals on odds ratios used to assess degradation.The same 164 problems appear in every condition, so comparisons use paired rather than independent outcomes.Source: E. McNemar’s Test for Paired Binary… · p. 3

III. METHODOLOGY

  • The five-stage framework processes 22,140 evaluations across 27 conditions, 164 problems, and five models.It runs from factorial design and prompt assembly through model dispatch, code extraction, and EvalPlus evaluation with interaction analysis.Source: D. Factorial Design and Interaction… · p. 3, §III · p. 4

A. Factorial Design

The design crosses format, persona, and urgency levels, assembles each combination into an API prompt, and evaluates five OpenAI models with model-specific decoding settings.

A. Factorial Design

  • The 27 conditions cross free-form, JSON, and XML; no, generic-developer, and senior-Python-developer personas; and no, moderate, and extreme urgency.Format compliance is enforced through prompt instructions rather than an API response-format mechanism.Source: A. Factorial Design · p. 4

B. Prompt Construction

  • Each API call places persona content in the system message and concatenates the urgency prefix, format instructions, and benchmark function signature in the user message.Constraint complexity increases with factor intensity while task semantics remain consistent across conditions.Source: B. Prompt Construction · p. 4

C. Models

  • Five models span GPT-4o and GPT-4.1 mini/full variants plus o3-mini, which uses merged messages, a 4096-token budget, and temperature 1.Standard models use greedy decoding, temperature 0, and a 512-token output limit; o3-mini’s settings are noted when interpreting its results.Source: C. Models · p. 4

D. Benchmark

The study evaluates code on HumanEval+ and extracts executable code from format-wrapped or free-form responses using format-specific fallback strategies.

D. Benchmark

  • HumanEval+ contains 164 Python function problems, each augmented with approximately 764 test cases to reduce false positives and detect subtle correctness failures.It is an enhanced version of HumanEval [11].Source: D. Benchmark · p. 4

E. Code Extraction Pipeline

  • Extraction routes JSON through parsing, key matching, repair, and plain-text fallback; XML through tag search and entity decoding; and free-form responses through code-fence or indentation checks.JSON extraction also corrects double-escaped newlines in code strings.Source: E. Code Extraction Pipeline · p. 4
  • Recording the extraction method enables post-hoc separation of formatting failures from reasoning failures.This separation follows recommendations for structured-output evaluation.Source: E. Code Extraction Pipeline · p. 4

F. Evaluation Metrics and Statistical Analysis

Evaluation uses HumanEval+ pass@1 and a three-way interaction measure, with paired tests for condition comparisons. Prompt length rises with constraint intensity and is treated as a limitation.

F. Evaluation Metrics and Statistical Analysis

  • Pass@1 is the fraction of problems whose extracted code passes all augmented HumanEval+ test cases in one attempt.The framework’s five stages culminate in EvalPlus evaluation and computation of three-way interaction effects.Source: F. Evaluation Metrics and Statistical… · p. 4, Figure 1 · p. 5
  • A negative three-way interaction indicates that a compound condition reduces performance beyond the sum of its individual effects.The interaction term quantifies the difference between actual performance and the additive prediction.Source: F. Evaluation Metrics and Statistical… · p. 4
  • Because the same 164 problems recur across conditions, McNemar’s test compares paired outcomes, with odds ratios and 95% confidence intervals reported alongside p-values.These paired outcomes avoid treating repeated problem evaluations as independent samples.Source: F. Evaluation Metrics and Statistical… · pp. 4–5
  • Prompt length necessarily increases with constraint intensity as format instructions and urgency framing add tokens.The paper tracks token counts and describes this confound as inherent to real-world compound prompts, which are longer because they contain more constraints.Source: F. Evaluation Metrics and Statistical… · p. 5

IV. RESULTS AND DISCUSSION

Performance under compound constraints differs across model families: GPT-4.1 remains comparatively stable, while GPT-4o shows greater variability and super-additive degradation. In GPT-4o-mini, all eight triple-constraint combinations underperform additive predictions, with larger interaction effects for JSON than XML.

IV. RESULTS AND DISCUSSION

  • The evaluation covered 164 HumanEval+ problems across five models and all 27 prompt conditions, with one greedy-decoded response per problem and condition.HumanEval+ pass@1 was the primary metric.Source: §IV · p. 5, Table I · p. 5

A. Performance Under Compound Constraints

  • GPT-4.1 models varied by only 6.1–6.7 pp across 27 conditions, while GPT-4o-mini ranged 14.6 pp and GPT-4o ranged 12.8 pp.The GPT-4o-mini range was 70.1% to 84.8%; GPT-4o ranged from 75.0% to 87.8%.Source: A. Performance Under Compound… · pp. 5–6, Figure 2 · p. 6
  • For GPT-4o-mini, urgency raised average pass@1 from 76.6% (none) to 80.2% (extreme), while XML raised it from 78.1% (none) to 82.0%; persona differences stayed below 1.7 pp across four standard models.Source: A. Performance Under Compound… · p. 6

B. Super-Additive Degradation in Compound Constraints

  • GPT-4o-mini averaged −7.6 pp across triple-constraint interactions, with all 8/8 combinations super-additive; GPT-4o averaged −4.2 pp, versus +1.7 pp for GPT-4.1-mini and +1.3 pp for GPT-4.1.Fig. 3 summarizes the average interaction across the eight combinations for each model.Source: B. Super-Additive Degradation in… · p. 6, Figure 3
  • On GPT-4o-mini, interactions ranged from −3.0 pp for XML+generic+extreme to −12.2 pp for JSON+expert+moderate; JSON averaged −10.7 pp versus −4.6 pp for XML.All eight triple combinations underperformed their additive predictions.Source: B. Super-Additive Degradation in… · pp. 6–7, Figure 4 · p. 7

C. Architecture-Dependent Resilience

The observed resilience to compound constraints tracks model family more closely than model size: both GPT-4o models show degradation, while both GPT-4.1 models resist it. The results also show why evaluating constraints only in isolation can miss substantial losses when they are combined.

C. Architecture-Dependent Resilience

  • Both GPT-4o models showed super-additive degradation, while both GPT-4.1 models resisted it, indicating that the pattern is architecture-dependent rather than size-dependent.GPT-4o-mini averaged −7.6 pp and GPT-4o −4.2 pp; both GPT-4.1 models averaged approximately +1.5 pp.Source: C. Architecture-Dependent Resilience · p. 7
  • GPT-4o-mini’s individual constraints were neutral or beneficial, yet their combination degraded pass@1 by 12.2 pp beyond the additive prediction.This interaction is a failure mode that single-factor prompt evaluation would not detect.Source: C. Architecture-Dependent Resilience · p. 7

D. The o3-mini Case

Unlike the standard models, o3-mini improved markedly with structured output constraints, and its interaction term could not be interpreted under the study’s additive framework. The paper also connects compound-prompt findings to engineering reliability assessment, including model choice and format selection.

D. The o3-mini Case

  • o3-mini rose from 59.8% unconstrained pass@1 to 88.6% with JSON and 88.1% with XML, while its interaction term was uninterpretable because large factor effects created ceiling effects.All 27 conditions with at least one active constraint clustered between 77% and 92%; the proposed internal-reasoning explanation is tentative.Source: D. The o3-mini Case · p. 7

E. Implications for LLM-Assisted Engineering Workflows

  • For GPT-4o-class models, compound constraints degraded code-generation reliability by 3–12 pp beyond what individual constraint testing would predict.The paper frames this as an interaction-driven failure mode in common production prompts.Source: E. Implications for LLM-Assisted… · p. 7
  • A 12.2 pp pass@1 loss corresponds to approximately 20 additional failing solutions per 164 problems, making model selection a reliability consideration alongside prompt design.The paper recommends evaluating models under compound constraints representative of deployment prompts.Source: E. Implications for LLM-Assisted… · p. 8
  • When structured output is combined with persona and urgency framing, XML-style wrapping may impose lower compound overhead than JSON encoding.This is presented as a possible mitigation based on the observed asymmetry between JSON and XML interaction effects.Source: E. Implications for LLM-Assisted… · p. 8
  • Compound-prompt evaluation is intended to complement existing system safety practices by informing tool assessment, hazard analysis, and verification under operationally representative prompts.The paper presents it as an added source of empirical evidence, not a replacement for existing safety practices.Source: E. Implications for LLM-Assisted… · p. 8

V. CONCLUSION

Across 164 HumanEval+ problems and five OpenAI models, compound prompt constraints interact in architecture-dependent ways: GPT-4o degrades beyond additive predictions, while GPT-4.1 resists and structured constraints can improve o3-mini performance. The findings caution against extrapolating compound-prompt reliability from single-factor tests and identify model choice and format as practical considerations.

V. CONCLUSION

  • The study covered 164 HumanEval+ problems and five OpenAI models in a 3 × 3 × 3 design, producing 22,140 evaluations and decomposing compound conditions into additive predictions and residual interactions.Source: §V · p. 8
  • GPT-4o compound constraints reduced pass@1 by 3–12 percentage points beyond additive predictions, with the largest interaction reaching −12.2 pp on GPT-4o-mini.JSON-based combinations produced substantially larger interaction effects than XML-based combinations.Source: §V · p. 8
  • Effects depended on architecture rather than parameter count: GPT-4.1 resisted compound interference, while structured output constraints improved performance on o3-mini.Source: §V · p. 8
  • Single-factor results cannot reliably predict compound-prompt reliability; XML-style wrapping may impose lower compound overhead than JSON when formatting is combined with persona and urgency framing.Source: §V · p. 8
  • Evidence is limited to five OpenAI models, one function-level Python benchmark, and greedy decoding; prompt token length is confounded with constraint intensity, and additive decomposition was inapplicable to o3-mini due to ceiling effects.Source: §V · p. 8
  • Future work will extend evaluation to other model families and benchmarks, examine why architectures differ, and test mitigation through format enforcement, prompt decomposition, or constraint-aware fine-tuning.Source: §V · p. 8
Loading 2609.03156v1…