What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts

• Application prompts must generalize across many usage scenarios, unlike end-user prompts that can succeed with one satisfactory response.Source: §1 · p. 1
• Underspecification omits essential requirements, leaving multiple valid but inconsistent behaviors possible and undermining stable user expectations.Source: §1 · p. 1
• The study evaluates requirement satisfaction across three representative tasks and collects 8.4k data points of aggregated LLM-plus-prompt behavior.Source: §1 · pp. 1–2
• 41.1%: LLMs often fill underspecification gaps, yet a later model version can degrade by more than 20%.Source: §1 · p. 2
• Two requirements-aware optimizers target how to specify requirements and which requirements to specify, and together improve accuracy by 4.8%.Source: §1 · p. 2
• The paper advocates proactive requirement discovery, reliable evaluators, and ongoing evaluation and monitoring to manage prompt underspecification in practice.Source: §1 · p. 2

2 Underspecification is Amplified in Prompting LLMs

• Compared with traditional software and ML pipelines, prompting is more prone to underspecification, less stable, and more frequently changed.Source: §2 · p. 2, Table 1 · p. 3
• Ad-hoc trial-and-error prompt revision exposes only a narrow slice of behaviors, while conditional, rare-but-critical, or hard-to-recognize requirements remain easy to overlook.Source: §2 · p. 3
• Rapid prompt and model changes introduce continual behavioral drift, so previously validated behaviors may fail and requirements need rediscovery.Source: §2 · p. 3

3.1 Experiment Setups

• The study measured satisfaction of specified and unspecified requirements across 60 plausible requirements from three tasks, using human-validated automated validators.Requirements were elicited from existing prompts, brainstorming, and error analysis, then selected by annotators.Source: §3 · p. 3, §3.1 · p. 4

3.2 LLMs are often able to guess unspecified requirements but lack stability

• When requirements were unspecified, LLM+Prompts’ accuracy fell by an average of 22.6% versus explicit specification, yet exceeded 98% accuracy in 41.1% of cases.For 65% of all requirements, at least one LLM+prompt combination could guess the requirement without explicit specification.Source: §3.2 · pp. 4–5, Figure 2 · p. 5
• o3-mini guessed 44.7% of unspecified requirements, a 20.2% increase over Llama-3.3-70B-Instruct.The authors suggest extra test-time compute may contribute to this difference.Source: §3.2 · p. 5
• Guessing rates varied by requirement type: format requirements reached 70.7%, while conditional requirements reached 22.9%.The authors suggest universal format conventions may be built into models, whereas corner cases are harder to predict.Source: §3.2 · p. 5
• Across prompts, unspecified requirements had 8.9% accuracy standard deviation, more than twice the variation for specified requirements.The difference was stronger when controlling for requirement conflicts: 7.9% for unspecified versus 0.8% for specified requirements.Source: §3.2 · p. 5

3.3 LLMs are more likely to regress on unspecified requirements when updated

• 5.9% of unspecified requirements regressed by more than 20% across model updates, almost twice the rate for specified requirements.Figure 3 summarizes accuracy drops across updates; even small hidden updates could reduce satisfaction of particular requirements.Source: Figure 3 · p. 5, §3.3 · p. 5

3.4 LLMs struggle with following many requirements at the same time

• 19 specified requirements together yielded 85.0% average accuracy for gpt-4o and 79.7% for Llama-3.3-70B-Instruct.Average accuracy declined as more requirements were specified; individual requirements could also drop substantially.Source: Figure 4 · p. 6, §3.4 · p. 6
• Intentional underspecification may focus models on selected requirements, but developers lack support for choosing what to leave implicit.The authors call for automated optimization of prompt specification as requirements grow.Source: §3.4 · p. 6

4 Requirements-Aware Prompt Optimization

• Both requirement-aware optimizers consistently improved prompt performance by +4.8% on average; Bayesian optimization reduced token usage by 41–45%.The reported trend also held for real-world prompts in an agentic coding context.Source: Table 2 · p. 7, §4.2 · p. 7

4.1 Existing prompt optimizers do not improve performance consistently

• Off-the-shelf optimizers produced small gains on two tasks (+2.8% on average) but reduced trip-advisory performance by 1.1% on average.The authors found no consistent overall improvement.Source: §4.1 · p. 7

4.2 Designing requirement-aware prompt optimizers

• COPRO-R adds requirement-specific validators and optimizes average accuracy across all requirements, whether specified or not.The validators provide requirement-specific feedback to guide prompt optimization.Source: §4.2 · p. 7
• The Bayesian optimizer represents each requirement as specified or unspecified and searches for the configuration that maximizes performance.It uses average requirement accuracy and a Tree-structured Parzen Estimator to find a good configuration in few trials.Source: §4.2 · p. 7
• Bayesian optimization improved prompt performance by 3.8% while reducing token usage by 41–45%.It more often dropped global, format, and developer-written requirements than the average requirement.Source: §4.2 · p. 7
• COPRO-R improved performance by 5.8%, with generated prompts reordering, merging, and sometimes dropping requirements.The paper contrasts this restructuring with Bayesian optimization’s selection of which requirements to specify.Source: §4.2 · p. 7
• On the website-generation coding task, both optimizers significantly improved the Cursor community baseline, and Bayesian optimization reduced token use by 71.9%.The authors describe this as generalization to a real-world context.Source: §4.2 · pp. 7–8

5 Towards Managing Prompt Underspecification

• A structured process should discover key task requirements, curate them to reflect real needs, and monitor for drift.Requirement discovery may draw on synthetic data, brainstorming, data-driven analysis, or safety engineering approaches.Source: §5 · p. 8
• Because the best specification depends on the model, developers should validate the full tracked requirement set during model migrations and reselect what to specify.The paper recommends background validation and alerts for significant changes.Source: §5 · p. 8
• Unstable requirement validators can signal ambiguity or misalignment, motivating comparison with human annotations and refinement of requirements.The authors propose meta-evaluation of LLM judges and strategic developer review of validation results.Source: §5 · p. 8

6 Related Work

• Instruction-following research studies datasets, model training, and task-specific optimization; this paper focuses on reliability when requirements are unspecified.The authors distinguish reliability under underspecification from usability through instruction following.Source: §6 · p. 8
• Prior empirical work examines prompt sensitivity, differences across models, and performance regressions after LLM updates; this paper studies those changes for unspecified requirements.Its focus is robustness on unspecified requirements rather than specified tasks.Source: §6 · p. 8
• Ambiguity research includes clarifying questions and methods to improve ambiguity handling, while studies report that LLMs can struggle to detect ambiguity.The paper treats ambiguity resolution as closely related to underspecification.Source: §6 · pp. 8–9
• Prompt optimizers adapt prompts using a clear metric; this work emphasizes discovering and operationalizing task requirements to construct a multifaceted optimization objective.Source: §6 · p. 9

• Prompt underspecification is common and challenging: models can infer missing requirements but behave inconsistently, while fully specified requirements can overwhelm them.Source: §7 · p. 9
• Requirements-aware optimizers make prompts easier to follow.Source: §7 · p. 9
• Systematic discovery, evaluation, and monitoring can help manage underspecification.Source: §7 · p. 9

Limitations

• Experiments cover 60 requirements and 240 synthetic prompts, limiting breadth; scaling requires more realistic requirements and validators and more efficient fine-grained evaluation.The experiments involved over 1.5 million LLM calls.Source: Limitations · p. 9
• LLM validators were manually checked and had a small error rate, but may favor same-family models; the authors say this does not affect their main results, which do not compare model families.Source: Limitations · p. 9
• Without a user study or direct developer-workflow integration, the observed patterns may not transfer cleanly to real-world processes, constraints, and decisions.Source: Limitations · p. 9

A.5 Complete list of curated requirements

• The curated requirements cover medical advice and lesson planning, including safety guidance, clear structure, standards alignment, and differentiation.The listed medical requirements include professional-care guidance and possible explanations of symptoms; lesson-planning requirements include detailed outlines and strategies tailored to student needs.Source: §A.5 · pp. 20–21

A.6 Human validation of requirement validators

• 95.6% (SD=0.08) human–LLM agreement was observed for requirement evaluators validated on 1,095 stratified evaluation results.A planner drafted an evaluation plan or Python script for each requirement; validators were iterated on the training split to improve human–LLM agreement.Source: §A.6 · p. 22

A.7 Prompt templates and construction

• A cyclic design varied which requirements were specified while keeping prompt complexity roughly constant, and the experiments used a simple task-description template.Each prompt covered the same number of consecutive requirements; each requirement was specified equally often, with requirements randomized in order.Source: §A.7 · p. 22, Figure 11 · p. 22

A.8 LLM judge setups for generic prompt optimizers

• Generic prompt optimizers used a requirement-agnostic gpt-4.1-mini judge to rate outputs from 1 to 10 at temperature 0.The judge received task-level information, the prompt instruction, and task inputs and outputs; its judgments were checked for consistency and reasonableness.Source: §A.8 · p. 22

A.9 Compute resources used in the experiments

• The first experiment used 6k inferences with each of 7 models and 840k evaluations with gpt-4.1-mini.Source: §A.9 · p. 22
• The second experiment used 6k inferences with each of two models across 3 prompt configurations, followed by 720k evaluations with gpt-4.1-mini.Source: §A.9 · p. 22
• Prompt-optimization experiments used 16.2k inferences per optimizer to produce prompts, 324k evaluations, and 480k evaluations across four optimization results.Source: §A.9 · p. 22
• LLM validators ran at temperature 0, while other calls used recommended temperatures of 1.0 for gpt-4o and 0.6 for Llama-3.Source: §A.9 · p. 22
• With a 95% discovery target, developers need to inspect more examples when a requirement is rarer, violated less often, or harder to notice.Source: Figure 12 · p. 23

B Detailed Analysis on Requirements Discovery

• Conditional requirements can be missed by ad-hoc inspection, even though LLMs struggle more with them; infrequent violations and hard-to-notice failures add to discovery difficulty.Source: B Detailed Analysis on Requirements… · p. 23
• More than 29 examples are needed to detect a requirement with 95% probability when it applies to 20% of inputs, is violated half the time, and is perfectly noticeable.The estimate assumes independent Bernoulli trials; discovery becomes harder when requirements are rarer, violated less often, or harder to notice.Source: B Detailed Analysis on Requirements… · p. 23

C Additional experiment results

• 37.5% of requirements drop significantly by more than 5% as prompts include more requirements, while unspecified requirements perform worse and diverge more.Source: Figure 14 · p. 23, Figure 13 · p. 23

C.2 ANOVA results

• The analysis uses ANOVA to test effects of requirement scope, source, category, and model, then Tukey’s HSD for significant factors.The supplied results passage identifies Table 4 as the report location but gives no specific test outcomes.Source: §C.2 · pp. 23–24

C.1 Requirement guess rates breakdown

• LLMs more often guess global, format-related, and prompt-sourced requirements, but struggle with conditional, style-related, and bottom-up requirements.Source: §C.1 · p. 23, Table 3 · p. 24

C.3 Requirements Conflict Analysis

• About 11.4% of requirement pairs showed clear conflicts, with accuracy drops reaching 41.1% in an example pair.Conflicts were defined as cases where specifying one requirement reduced another requirement’s accuracy by more than 5%.Source: §C.3 · p. 24
• After conflicting requirements were removed, prompt variance was 0.8% for specified requirements versus 7.1% for unspecified ones.The authors conclude that conflict does not explain the remaining instability in how models fill in unspecified constraints.Source: §C.3 · p. 24

C.4 Budget-controlled comparison between CORPO and CORPO-R

• COPRO improved with more iterations (+3.6%) and fell only slightly short of COPRO-R (-0.2%), but took 2.8x longer end-to-end.The authors attribute the time difference to sequential prompt exploration in COPRO versus more-parallelizable evaluations for COPRO-R.Source: §C.4 · p. 24

C.5 Optimizer robustness to training set

• The selected constraints were robust to training-set composition, with mean Jaccard similarity of 0.754 across matched prompts.Step-by-step execution order, error handling, and dependency detection consistently ranked among the most-selected constraints.Source: §C.5 · pp. 24–25
• Whether a requirement was specified had the largest impact on average accuracy (+0.2) and significantly affected SD (-0.037); conditional requirements were harder for models (-0.09 accuracy, +0.017 SD).Models did better on requirements found in existing prompts and on format-related requirements.Source: Table 4 · p. 25

C.6 Prompt Optimization Examples

• 86.7% for COPRO-optimized prompts versus 75.4% for unoptimized prompts; COPRO reordered and merged requirements, and sometimes dropped them.The figures show the optimized and unoptimized prompt examples, respectively.Source: Figure 15 · p. 26, Figure 16 · p. 27

D Additional tasks setups and results

• The additional experiments covered health consulting and lesson planning, using sampled HealthBench and lesson_plan examples.The HealthBench subset was filtered to medical consulting requests in English; duplicate lesson_plan queries were removed.Source: D Additional tasks setups and results · p. 28
• In additional tasks, specified versus unspecified requirement performance was 83.6% versus 59.5%, and models guessed unspecified requirements 25.1% of the time.The authors report that format-related, global, and developer-written requirements were easier to guess.Source: D Additional tasks setups and results · p. 28
• Across model updates, the reported regression figures were 4.3% and 2.3%; separately, performance dropped from 95.0% to 78.5% when requirements increased from 1 to 19.The results also held for gemini-2.5-flash.Source: D Additional tasks setups and results · p. 28

E Details on Agentic Coding Case Study

• The coding case study evaluated a 25-requirement Cursor Community prompt using OpenHands-generated repositories and gemini-2.5-flash assessment against all requirements.Each proposed prompt was run three times per user query to reduce variance in generated repositories.Source: E Details on Agentic Coding Case Study · p. 28
• 47.4% for Bayesian-optimized prompts versus 44.1% for unoptimized prompts; the optimized prompts used fewer tokens while achieving similar accuracies.The figures identify the unoptimized and Bayesian-optimized prompt examples, respectively.Source: Figure 17 · p. 29, Figure 18 · p. 30