IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

• Reliable research automation requires detecting and resolving omitted, ambiguous, or inconsistent method-defining choices before codification, because unsupported assumptions can produce code for a different method.Existing evaluations largely focus on idea quality or downstream plans and artifacts while assuming the underlying method is sufficiently specified.Source: §1 · pp. 1–2
• Codification readiness asks whether a competent implementer can construct a faithful initial implementation without unsupported assumptions about the core method.A specification is not ready when competent implementers could make materially different method-defining choices without evidence indicating the intended choice.Source: §1 · pp. 1–2
• IdeaAMBIG evaluates three successive capabilities: readiness assessment, defect localization from the specification, and clarification action generation given the annotated defect.This separates discovering an implementation blocker from requesting the information needed to resolve it.Source: Figure 1 · p. 1, §1 · p. 2
• IdeaAMBIG contains 660 single-defect instances: 163 real-world gaps and 497 controlled synthetic gaps, each paired with a supported resolution and category labels.The synthetic instances alter exactly one implementation-critical detail in a codification-ready reference, while real-world instances derive from GitHub issues and reproducibility reports.Source: §1 · p. 2
• 9.6% Macro Defect Recovery Rate versus 80.6% Macro Clarification Action Success Rate shows that models struggle to find blockers independently but can often clarify them once identified.In an oracle study, supplying the missing information raises the downstream codification-ready rate from 14% to 98%.Source: §1 · p. 2

Scientific Ideation and Automated Research

• Existing ideation evaluations measure properties of proposed ideas or resulting artifacts, whereas IdeaAMBIG examines the specification connecting an idea to faithful implementation.Research agents integrate ideation with review, experimentation, coding, and writing, yet stronger ideas before execution need not produce equally strong projects after execution.Source: §2 · p. 2

Scientific Design and Artifact Verification

• Scientific design, code-generation, consistency, and critique benchmarks evaluate reasoning, artifacts, discrepancies, or feedback rather than unresolved method-defining choices blocking implementation.Unlike SciConvBench, which elicits missing information through multi-turn task formulation, IdeaAMBIG evaluates pre-implementation readiness, localization, and targeted clarification using evidence from papers, code, issues, and reproducibility reports.Source: §2 · p. 2

Specification Defects and Clarification

• Requirements and LLM benchmarks study ambiguity, incompleteness, inconsistency, and clarification, but IdeaAMBIG targets research-specific defects that can change objectives, model structures, training procedures, or evaluation protocols.The benchmark distinguishes method-defining blockers from routine engineering choices and requires evidence-supported resolutions distributed across research artifacts.Source: §2 · p. 3
• IdeaAMBIG is presented as the first benchmark to jointly assess codification readiness, method-defect localization, and targeted clarification while separating blocker discovery from response generation.It uses Level-2 research-specific categories and evidence-grounded instances containing a single defect.Source: §2 · p. 3

3 IdeaAMBIG

• IdeaAMBIG evaluates readiness assessment, defect localization, and clarification-action generation over evidence-grounded, single-defect research-method specifications.Codification readiness means a competent implementer can construct a faithful initial implementation without unsupported assumptions about core method decisions; routine hyperparameters and explicitly open choices need not be fixed.Source: §3 · p. 3, §3.1 · pp. 3–4

3.1 Task Formulation

• The single-target design isolates identification and resolution of one implementation-critical decision, while audits and exploratory analyses support it as a controlled but realistic evaluation unit.Instances with multiple independent blockers are split when possible or discarded, and defect granularity is analyzed as a covariate for localization difficulty.Source: §3.1 · p. 4

3.2 Data Collection

• The benchmark constructs retrospectively grounded specifications from papers, codebases, reproducibility reports, issue discussions, and execution artifacts, excluding environmental or resource-related problems.Each NOTREADY specification is paired with an evidence-grounded READY counterpart resolving only the target defect, and evaluated models do not receive downstream artifacts or resolution evidence.Source: §3.2 · p. 4

3.3 Dataset Statistics

• IdeaAMBIG contains 163 real-world and 497 controlled synthetic instances spanning ten domains, with incompleteness most frequent overall and method-related defects dominating both subsets.Real-world cases emphasize ambiguity, while synthetic cases more often omit method procedures; the subset distributions remain broadly similar across methodological components.Source: §3.3 · pp. 5–6

3.4 Benchmark Validation

• Human studies support benchmark validity and annotation reliability, with gap-validity, criticality, sufficiency, and realism judgments exceeding 90% for both subsets.Readiness agreement reaches κ = 0.84/0.92, Level-1 agreement κ = 0.89/0.89, Level-2 agreement κ = 0.92/0.85, and target-defect agreement is 87.7/84.0% for real/synthetic instances.Source: §3.4 · p. 6

4.1 Experimental Setup

• The experiments evaluate 13 proprietary and open-weight LLMs using fixed task-specific prompts and structured outputs, with semantic and rubric-based judgments validated against adjudicated human annotations.Metrics include Macro-F1, Macro DRR, Macro-CAS, and RGS; source-clustered analyses account for shared repositories, papers, or executed projects.Source: §4.1 · pp. 6–7

4.2 Main Results

• Readiness assessment remains unreliable: GPT-5.6-Sol achieves 67.5 Macro-F1 on real-world instances and 86.4 on synthetic instances, with highest RGS values of 0.36 and 0.49.On real-world instances, it accepts 31% of underspecified specifications and rejects 34% of codification-ready ones.Source: §4.2 · p. 7
• GPT-5.6-Sol reaches 9.6 Macro DRR on real-world instances, compared with 16.0 Loc-Acc, while clarification reaches 80.6 Macro-CAS when the blocker is supplied.The controlled comparison raises Macro-CAS from 13.6 to 80.6 and identifies blocker discovery as the primary bottleneck.Source: §4.2 · p. 7

4.3 Error Analysis and Representative Cases

• Error analysis finds that only 1.5% of real-world Task 1 predictions are both correct and fully grounded, while 66.3% of Task 2 predictions identify a different blocker.Once the blocker is provided, insufficient clarification occurs in 19.6% of instances versus 4.3% imposing unsupported assumptions; coarse defects are easier to recover than non-coarse defects.Source: §4.3 · pp. 7–8

4.4 Oracle Clarification Utility

• Oracle clarification raises READY rate from 14% to 98% and increases implementations passing all predefined tests from 45% to 85%.Faithful implementation of the target method detail rises from 30% to 90%, while the READY-rate improvement is measured on 50 paired real-world instances.Source: §4.4 · p. 8, Table 2 · p. 8

• IdeaAMBIG contains 660 evidence-grounded single-defect instances for assessing implementation readiness, defect localization, and clarification-action generation.The benchmark evaluates 13 LLMs on these capabilities.Source: §5 · p. 8
• 9.6 Macro DRR on real-world localization contrasts with 80.6 Macro-CAS when the annotated defect is provided to the strongest model.This gap identifies defect localization as the main bottleneck, with a consistent pattern across models.Source: §5 · p. 8
• Readiness judgments remain frequently incorrect or weakly grounded, and taxonomy prediction adds difficulty beyond identifying unresolved method-defining decisions.For the strongest model, identifying the unresolved method-defining decision remains the primary challenge.Source: §5 · p. 8
• Oracle clarification raises the codification-ready rate from 14% to 98%, supporting readiness gates that identify blockers, seek grounded clarification, and prevent unsupported methodological choices.The result suggests that reliable research agents should separate specification-readiness assessment from clarification after defect identification.Source: §5 · p. 8

Limitations

• IdeaAMBIG’s evidence-resolved instances are concentrated in AI, NLP, and machine learning, limiting established transfer to other computational sciences and early-stage ideas.The paper proposes broader collection to test whether codification readiness and its taxonomy transfer.Source: Limitations · p. 8
• A 50-pair exploratory ablation found no evidence that combining two known defects hindered localization, but response-count asymmetry limits the conclusion.A larger, response-budget-matched study of specifications with multiple interacting defects remains open.Source: Limitations · p. 8
• Clarification is evaluated as a single action, whereas future systems could use multi-turn interaction, artifact inspection, conflicting-evidence reconciliation, and stopping decisions.The oracle utility study currently supplies the gold resolution rather than requiring these richer clarification behaviors.Source: Limitations · p. 8
• Codification readiness is measured before execution rather than through full implementation, leaving implementation divergence, unsupported assumptions, correction cost, and scientific outcomes unmeasured.Integrating the diagnostic as a gate in research agents would enable direct evaluation of these downstream effects.Source: Limitations · p. 8

A.1 Publication-Year Distribution

• Most identifiable source-paper instances date from 2020–2025, with the publication-year distribution peaking in 2021.Figure 4 covers instances with identifiable source-paper years.Source: §A.1 · p. 12
• The publication-year concentration in recent machine-learning literature is consistent with IdeaAMBIG’s focus on recent reproducibility and specification issues.This distribution characterizes the benchmark’s temporal scope rather than establishing broader literature coverage.Source: §A.1 · p. 12

Construct distinction

• IdeaAMBIG targets implementation-critical choices needed to faithfully instantiate an already proposed method, not open-ended refinement of objectives, hypotheses, motivation, or scientific direction.This focuses on a later failure mode in automated research pipelines, where plausible ideas lack sufficient specification for faithful codification.Source: §A.2 · pp. 12–13

Annotation Protocol

• Two experienced annotators independently reviewed 100 instances—50 real-world and 50 synthetic—using a binary test of whether resolving each gap changes the research idea itself.They examined the original specification, target defect, and supported clarification.Source: §A.2 · p. 12
• Instances were labeled IDEA-LEVEL CLARIFICATION when resolving a gap required modifying the research idea, and SPECIFICATION-LEVEL CLARIFICATION otherwise.Specification-level cases preserve the research objective while resolving missing, ambiguous, or inconsistent implementation-critical choices.Source: §A.2 · p. 12

Results

• Across both subsets, the vast majority of instances were SPECIFICATION-LEVEL CLARIFICATION, confirming that IdeaAMBIG primarily evaluates implementation-gap identification and resolution rather than general-purpose idea refinement.Only a small fraction required changes to the underlying research idea.Source: §A.2 · p. 12

A.3 Diagnostic Error Analysis

• The analysis classifies readiness errors by decision correctness and rationale grounding, blocker-localization errors by target and taxonomy accuracy, and clarification errors by insufficiency and assumption imposition.The clarification error rates are evaluated independently and may overlap; the three localization categories are mutually exclusive.Source: §A.3 · p. 13
• 67.5% of real-world readiness decisions are correct, but only 1.5% are both correct and fully grounded; false-READY and false-NOTREADY errors remain substantial.On controlled-synthetic instances, accuracy rises to 86.5%, although most correct predictions remain weakly grounded.Source: §A.3 · p. 13
• Only 16.0% of real-world blockers are recovered, while 66.3% identify a different blocker and 17.8% select a neighboring implementation decision.Almost no outputs are vague or lack a diagnosis, indicating that failures mainly involve choosing the wrong unresolved decision rather than failing to produce a concrete concern.Source: §A.3 · p. 13
• 80.6 Macro-CAS on real-world instances shows that clarification is effective once the blocker is supplied, with insufficient clarification at 19.6% and assumption-imposing actions at 4.3%.Controlled-synthetic instances reach 96.2 Macro-CAS, with corresponding error rates of 2.0% and 1.0%.Source: §A.3 · p. 14
• Overall, weak grounding and incorrect blocker selection constitute the main end-to-end bottleneck, whereas clarification generally succeeds when the unresolved decision is explicitly provided.The analysis reuses original outputs and evaluation sets, covering balanced readiness samples, all 163 real-world instances, and 200 controlled-synthetic instances.Source: §A.3 · pp. 13–14

A.4 Qualitative Error Cases

• GPT-5.6-Sol fails at three clarification stages: it overlooks a required operation, localizes the wrong blocker, or requests an incomplete resolution.The cases concern a missing LeakyReLU before softmax, a missing 1×1 classifier producing 65 logits, and unresolved TimeMixer routing or channel-independence settings.Source: §A.4 · pp. 14–15

B.1 Taxonomy Construction

• IdeaAMBIG’s taxonomy begins with Incompleteness, Ambiguity, and Inconsistency, then derives implementation-critical Level-2 categories from reproducibility reports and paper-associated GitHub issues.Candidate defects are filtered, mined with LLM-assisted signals, clustered, and prepared for human open coding.Source: §B.1 · p. 15
• Human open coding consolidates categories by specification component, with codebook criteria and boundary cases designed to separate recurring implementation blockers from isolated complaints.The resulting taxonomy covers procedures, model structures, data, evaluation, configuration, and formal definitions under the Level-1 types.Source: §B.1 · p. 15
• The taxonomy is coverage-tested through pre-adjudication agreement and saturation, but is scoped to AI, NLP, and machine-learning research specifications rather than all scientific domains.Additional sampled instances must introduce no stable new Level-2 categories to count as complete for IdeaAMBIG’s scope.Source: §B.1 · p. 15, Table 7 · p. 16

B.2 Annotation instructions.

• Annotators retain cases only when gaps are actionable, affect faithful core-method implementation, and have evidence sufficient for a codification-ready clarification.Environment setup, resources, efficiency, formatting, and other non-methodological issues are excluded.Source: §B.2 · p. 16
• Each retained case receives exactly one Level-1 and one Level-2 label, with adjudication prioritizing evidence support, implementation relevance, and category boundaries.Cases spanning categories are assigned according to the primary implementation blocker; unmatched cases receive Level-2 “other” with an explanation.Source: §B.2 · p. 16

B.3.1 Ambiguity

• Ambiguous Definition covers formal elements whose meaning, scope, or operation permits materially different computations, excluding obvious typographical errors or definitions already supplied elsewhere.Examples include unclear norm semantics, marginalization conventions, variable scope, operator behavior, and function domains; the boundary is whether the mathematical object or operation changes.Source: §B.3.1 · p. 16
• Ambiguous Procedure covers unclear method operations or execution rules, whereas missing structural details, undefined formal elements, and conflicting sources receive different categories.Examples include fixed features versus fine-tuning and unresolved update ordering; the distinction depends on whether uncertainty concerns operation rather than a formal object or absent step.Source: §B.3.1 · p. 17

B.3.2 Incompleteness

• Missing Method Procedure covers omitted operational rules that prevent faithful implementation, while configuration-selection gaps and structural choices belong to separate categories.Examples include routing rules, termination conditions, bookkeeping, special-token construction, detokenization, weight sharing, convolution parameters, quantization, and execution procedures; the primary blocker determines the label when procedure and configuration details overlap.Source: §B.3.2 · p. 17
• The incompleteness taxonomy separately identifies omitted model structure, data preparation, configuration protocols, and evaluation procedures according to whether the gap changes architecture, inputs or supervision, result-sensitive selection, or reported performance.Model structure includes components such as pooling and dimensional mappings; data specification covers construction and preprocessing; configuration protocol covers tuning procedures; evaluation specification covers scoring, aggregation, and interpretation.Source: §B.3.2 · pp. 18–19

B.4 Inconsistency

• Inconsistency categories apply when sources explicitly conflict over optimized objectives, model or processing designs, or formal mathematical definitions, rather than when information is merely missing or ambiguous.The boundaries distinguish Conflicting Objective, Conflicting Model Design, and Conflicting Formal Definition from Missing Method Procedure, Missing Model Structure, and ambiguous categories.Source: §B.4 · pp. 20–21

C Construction of Codification-Ready Counterparts

• Each benchmark record pairs an underspecified NOTREADY specification with a READY counterpart that resolves the annotated defect using available evidence without unsupported assumptions or changes to non-target content.Real-world resolutions come from reproducing-author clarifications or implementation choices, GitHub resolutions come from author or maintainer responses, and synthetic READY counterparts are the pre-injection references.Source: C Construction of Codification-Ready… · p. 22
• Paired specifications preserve the same task, inputs, outputs, and contextual information except where clarification necessarily changes them, isolating codification-readiness effects from surface-form differences.The construction retains method, training, and evaluation contexts across each pair.Source: C Construction of Codification-Ready… · p. 22

D.1 Inference and Deployment Details

• The study evaluates 13 proprietary and open-weight models using identical benchmark inputs, task definitions, and output requirements across model families.Proprietary models use provider-hosted APIs, while publicly released open-weight checkpoints run in the authors’ inference environment.Source: §D.1 · p. 22
• All records use fixed task-specific prompts and isolated inputs, with models receiving only task-defined information and producing task-specific readiness, defect-localization, or clarification outputs.Models never access the downstream papers, codebases, issue threads, reproducibility evidence, or source artifacts used to construct or validate instances.Source: §D.1 · p. 22
• Inference uses deterministic decoding when supported, fixed JSON Schema-constrained outputs when available, and separate generation and evaluation stages with shared criteria across model families.Generation excludes evaluator judgments, gold resolutions, and feedback.Source: §D.1 · p. 22

D.2 Information Bottleneck Analysis for Clarification

• The information-bottleneck experiment compares end-to-end and defect-guided clarification on identical Task 3 instances using GPT-5.6-Sol, prompts, and decoding configurations.The end-to-end condition must discover the missing information, whereas the defect-guided condition additionally receives the annotated target defect.Source: §D.2 · p. 23
• 13.6 to 80.6 Macro-CAS and 8.6 to 80.4 Sufficiency show that defect-guided clarification greatly outperforms end-to-end clarification.No-Assumption remains nearly unchanged at 96.3 versus 95.7, indicating the gap mainly reflects defect discovery rather than unsupported implementation choices.Source: §D.2 · p. 23

D.3 Taxonomy-Free Defect Localization

• Taxonomy-free localization is evaluated on 50 randomly sampled real-world instances from the 163-instance Task 2 split, with two blinded machine-learning annotators and adjudicated disagreements.The same-sample comparison also computes L2-aware accuracy for the original taxonomy-constrained outputs.Source: §D.3 · p. 23
• Removing taxonomy prediction improves direct blocker identification, but residual taxonomy-free failures show that discovering the unresolved implementation decision remains the primary localization challenge.The ablation predicts only a natural-language defect description, and human annotators judge whether it identifies the same implementation-critical blocker as the gold defect.Source: §D.3 · pp. 23–24

D.4 Small-Scale Executable Validation

• The executable study samples 20 controlled instances with verifiable bounded components and excludes large-scale training, private data, unavailable services, and complete-pipeline reproduction.It compares direct generation from underspecified ideas with clarification-assisted generation using the annotated defect, clarification action, and oracle resolution.Source: §D.4 · p. 24
• Executable fidelity is assessed against paper-and-reference-code evidence using output equivalence, targeted unit tests, or blinded inspection, with identical checks across conditions.The study distinguishes successful execution from faithful instantiation of the target method detail.Source: §D.4 · p. 24
• 95.0% versus 70.0% of generated implementations are runnable, while target fidelity rises from 30.0% to 90.0% with clarification assistance; all-tests success also increases from 45.0% to 85.0%.Further-clarification cases decrease from 35.0% to 5.0%; these results provide component-level evidence rather than full paper-reproduction results.Source: §D.4 · p. 24

D.5 Source-Level Robustness and Uncertainty

• Source-clustered bootstrap resamples complete sources with 10,000 replicates, recomputes task-specific primary metrics, and reports percentile 95% confidence intervals.Source-balanced estimates additionally assign equal total weight to represented sources within each readiness class or Level-2 category, while paired comparisons use identical sampled cluster multiplicities across models.Source: §D.5 · p. 25
• A 67.0-point increase in real-world Macro-CAS from providing the annotated blocker remains robust under paired source-clustered resampling (95% CI = [59.0, 75.2], p < 0.001).Real-world Macro DRR remains 9.6 (95% CI = [3.1, 18.7]), while defect-guided Macro-CAS reaches 80.6 (95% CI = [70.1, 89.0]); source-balanced estimates differ by at most 0.6 points across all six task–subset combinations.Source: §D.5 · p. 26

E.1 Source-Specific Candidate Verification

• Candidate verification applies source-specific two-stage review and retains only implementation-critical, atomic, self-contained, evidence-supported gaps or controlled defects.Reviewers exclude non-methodological engineering concerns, unresolved evidence conflicts, unsupported interpretations, and synthetic transformations that introduce multiple interacting defects or alter non-target content.Source: §E.1 · p. 26

E.2 Target-Uniqueness and Alternative-Blocker Audit

• 91.4% of real-world and 95.6% of controlled-synthetic instances had targets judged unique primary blockers; adjudication retained 660 final single-target instances.Co-primary alternatives occurred in 4.9% and 2.4% of the respective subsets; 624 instances were unchanged, 21 revised, 7 split into 15, and 8 removed.Source: §E.2 · p. 28

E.3 Defect Granularity as a Difficulty Covariate

• Granularity labels defects as coarse, medium, or fine by the scope of the underdetermined decision, but Fine is merged with Medium because only 5 real-world and 12 synthetic cases receive it.Granularity-stratified Task 2 conclusions are reported only if model–human agreement reaches substantial agreement (κ ≥0.6).Source: §E.3 · pp. 28–29
• On Task 2, GPT-5.6-Sol performs better on Coarse than Non-coarse defects: real-world Loc-Acc is 34.0 versus 9.0 and Macro DRR is 21.0 versus 7.0, while synthetic Loc-Acc is 47.0 versus 15.0 and Macro DRR is 26.0 versus 9.0.All differences remain significant after Holm adjustment; category-level bootstrap intervals are [7.8, 22.1] for real-world and [9.8, 24.6] for synthetic comparisons, both p < 0.001. The pattern supports a needle-in-a-haystack account because most real-world defects are Non-coarse and require finding a missing or incorrect operation within an otherwise specified component.Source: §E.3 · p. 29

E.4 Multi-Defect Readiness and Localization Ablation

• The ablation combined two independently validated defects in 50 controlled-synthetic specifications, evenly split between same-slot and different-slot pairs, with all preservation checks passing.The construction preserved both target defects, introduced no third defect, and retained non-target content.Source: §E.4 · p. 29
• The multi-defect evaluation removed the one-defect response cap and scored recovery independently using ANY-OF-2 and ALL-OF-2 against a paired single-defect baseline.Readiness used AND aggregation, whereas Any-of-2 used OR aggregation across the same defect pairs.Source: §E.4 · pp. 29–30
• 64.0% vs. 36.0% Any-of-2 recovery favored combined-defect specifications, while All-of-2 was 8.0% vs. 2.0% and readiness was 88.0% vs. 80.0%.The Any-of-2 difference was significant (∆= +28.0, p = 0.0013); readiness and All-of-2 differences were not significant (p = 0.289 and p = 0.250).Source: §E.4 · p. 30
• The multi-defect ablation is exploratory and limited to controlled-synthetic instances because real-world sibling specifications lack a shared reference; 94.5% of instances have a unique primary blocker.Thus, the result does not estimate the frequency of naturally co-occurring defects.Source: §E.4 · p. 30

E.5 Benchmark Annotation Guidelines

• Annotators label a specification READY when a faithful initial implementation requires no unsupported core-method assumption, and NOT READY when an implementation-critical choice remains underdetermined.UNSURE is reserved for cases requiring expert adjudication, not merely technical unfamiliarity; exact numerical reproduction is not required.Source: §E.5.2 · p. 30, Exact Reproduction Is Not Required · p. 33

Material-Divergence Test

• The guidelines apply a material-divergence test across task and I/O, algorithms, architecture, objectives, training, data, inference, evaluation, and internal consistency, while exempting ordinary engineering choices.Two materially different core-method implementations imply NOT READY, whereas differences in seeds, hardware, conventional batch sizes, optimizer settings, or minor details do not.Source: §E.5.3 · p. 30, Ordinary Engineering Choices · p. 31, Material-Divergence Test · p. 31

E.5.6 Annotation Procedure

• For NOT READY cases, annotators identify one primary blocker and affected slot, highlight the smallest revealing span, characterize blocker atomicity, and state the concise implementation-critical clarification required.Independent unresolved blockers receive MULTIPLE INDEPENDENT BLOCKERS, while unresolved blocker boundaries receive UNCLEAR; READY cases use NONE and leave defect fields empty.Source: §E.5.4 · p. 31, §E.5.6 · pp. 31–32, EXACTLY ONE PRIMARY BLOCKER · p. 32

Annotation Summary for a NOT READY Decision

• For NOT READY decisions, annotation records the affected specification slot, required implementation-critical clarification, and whether the case has one or multiple independent blockers.The protocol also uses independent annotation followed by senior adjudication to establish final labels and primary target defects when applicable.Source: Annotation Summary for a NOT READY… · p. 34, §E.6 · p. 34

E.6 Annotation Reliability

• 92.0% and 96.0% readiness agreement on real-world and synthetic instances, respectively, with κ = 0.84 and 0.92; Level-1 agreement exceeds 93%.Level-2 categories also show substantial agreement, with κ = 0.92 on real-world and κ = 0.85 on synthetic instances; adjudicated decisions form the final gold labels.Source: §E.6 · p. 34

F.1 Task 1 Evaluation Metrics

• Task 1 reports Macro-F1, UnsafePass, OverFlag, and Reason Grounding Score separately for real-world and controlled synthetic subsets.Macro-F1 equally weights READY and NOTREADY; UnsafePass measures missed underspecification, OverFlag measures false rejection, and RGS evaluates rationale consistency with the gold readiness rationale.Source: §F.1 · p. 34
• Reason Grounding Score combines label support, target-defect match, and faithfulness, with each sub-score in {0, 0.5, 1} and higher overall values indicating stronger grounding.Label support is prioritized, while target-defect match rewards identifying the annotated gap and faithfulness penalizes unsupported assumptions, contradictions, or hallucinated gaps.Source: §F.1 · p. 35

F.2 Task 2 Evaluation Metrics

• Macro DRR requires recovering the annotated target defect with correct Level-1 and Level-2 labels, macro-averaged equally across observed Level-2 categories.The evaluation unit is an atomic target defect, and plausible neighboring defects do not receive credit.Source: §F.2 · pp. 35–36
• GPT-5.6-Sol achieves the strongest L1-aware recovery on both subsets, yet L1-aware recovery remains below Loc-Acc for every model.For Qwen3-8B on the synthetic subset, Loc-Acc is 10.0 while L1-aware recovery is 2.0, indicating target-label alignment errors.Source: §F.2 · p. 37

F.3 Task 3 Evaluation Metrics

• Macro-CAS counts clarification actions as successful only when they are target-relevant, resolution-sufficient, and free of unsupported implementation assumptions.Sufficiency requires obtaining the missing implementation-critical detail, while No-Assumption rejects asserted or presupposed unsupported choices.Source: §F.3 · pp. 37–38

Successful Task 3 Clarification Action

• A successful clarification action asks a targeted question covering the needed implementation decisions without presupposing the gold resolution.The example concerns an ambiguous parameter-matched dense baseline whose activated parameters are undefined.Source: §F.3 · p. 38, Successful Task 3 Clarification Action · p. 40, Figure 6 · p. 40
• INTERNAL CONSISTENCY covers conflicts among specification components, including objectives and losses, architectures, input-output definitions, procedures, or implementation requirements.Source: Successful Task 3 Clarification Action · p. 47
• Missing or ambiguous information should be assigned to the substantive slot affected by the gap rather than INTERNAL CONSISTENCY.Source: Successful Task 3 Clarification Action · p. 47
• NONE is selected only when a specification is labeled READY and contains no implementation-critical blocker.Source: Successful Task 3 Clarification Action · p. 47
• The annotation uses defined specification slots for codification-readiness assessment.Source: Table 23

G.1 Human Validation of LLM-Based Evaluation

• Human validation used 300 randomly sampled outputs stratified across real-world and synthetic instances, defect categories, and model groups, with two blinded annotators and third-annotator adjudication.Annotators received only the information available to the automatic evaluator, and agreement was measured using exact agreement, Cohen’s κ, weighted κ, and MAE as appropriate.Source: §G.1 · p. 48
• Claude Opus 4.8 agrees with adjudicated human judgments on 86.0% of Task 2 decisions, 91.0% of Task 3 target-relevance judgments, and 85.0% of resolution-sufficiency judgments.Task 1 exact agreement ranges from 76.0% to 84.0%, with weighted κ between 0.68 and 0.78 and MAE below 0.14; No-Unsupported-Assumption agreement is 93.0%.Source: §G.1 · p. 48

G.2 Synthetic-Controlled Defect Validation

• The synthetic-defect study evaluated validity, implementation-criticality, clarification resolution, and realistic specification failure under source-blinded annotation.Each criterion tests whether a gap affects faithful interpretation, can change implementation outcomes, is resolved by the clarification, and could arise in real implementation or reproduction workflows.Source: §G.2 · p. 49
• Synthetic-controlled instances achieve 96.0% validity and 94.0% implementation-criticality, indicating that controlled defect injection preserves the benchmark’s core construct.These judgments come from blind human evaluation of 50 real-world and 50 synthetic-controlled instances using four binary criteria.Source: §G.2 · p. 49
• Synthetic-controlled instances receive 91.0% realism and 98.0% clarification-resolution ratings, compared with 93.0% realism for real-world instances.The results support synthetic defects as plausible research-workflow failures while providing precise, scalable defect targets.Source: §G.2 · p. 49

G.3 Clarification Improves Downstream Codification

• The clarification experiment compares direct and clarification-assisted generation with identical GPT-5.6-Sol decoding, evaluating readiness, completeness, hidden-detail recovery, and unsupported assumptions.The clarification-assisted pipeline receives the target defect, clarification action, and oracle answer, so the study measures the upper-bound utility of resolving missing information rather than end-to-end clarification-agent performance.Source: §G.3 · p. 50
• Clarification-assisted specifications achieve a 98% READY Rate and recover target missing details in 98% of cases, versus 14% for direct generation, while eliminating unsupported assumptions.Direct generation has 30% completeness and introduces unsupported assumptions in 6% of specifications; the comparison is an oracle analysis measuring the upper-bound utility of clarification.Source: §G.3 · p. 50

Evaluation Protocol

• The baseline samples 100 defect-centered records—50 real-world and 50 synthetic—while preserving the benchmark’s Level-1 defect distribution.The records come from the same evaluation pool as the LLM experiments, enabling matched comparisons.Source: §G.4 · p. 50
• Two machine-learning researchers independently perform each task, with disagreements resolved by a third reviewer through adjudication.Final human predictions use the adjudicated responses and the same metrics as the LLM evaluation.Source: §G.4 · p. 51
• Participants receive the same inputs and instructions as LLMs: readiness classification, three-part defect localization, or structured clarification-action generation.They receive no additional evidence beyond each benchmark input.Source: §G.4 · p. 51

Results

• Humans achieve higher Macro-DRR than LLMs on both real-world and synthetic subsets, although defect localization remains far from perfect.Table 27 reports human performance on the same records used for model comparisons.Source: §G.4 · p. 51, Table 27 · p. 51
• Humans achieve high Macro-CAS when the target defect is provided, making clarification easier than discovering the blocker.This supports the conclusion that accurate defect localization, rather than clarification-language generation, is IdeaAMBIG’s main challenge.Source: §G.4 · p. 51

Scope of evaluation

• IdeaAMBIG assesses the codification readiness of a proposed methodological mechanism rather than the research idea’s novelty, scientific value, motivation, or broader completeness.The benchmark targets whether the mechanism is sufficiently specified for faithful implementation.Source: H Scope of Research-Idea Specifications · p. 51

Evidence sources

• Benchmark specifications reconstruct implementation-facing information from papers, codebases, reproducibility reports, issue discussions, and executed projects because original pre-implementation exchanges are rarely preserved.These artifacts support identifying implementation-critical gaps and evidence-supported resolutions, but are not treated as original idea-handoff records or shown to evaluated models.Source: H Scope of Research-Idea Specifications · p. 51

Interpretation of inputs

• The benchmark inputs represent the methodological information that would need to be available when a proposed method is handed to a competent implementer or coding agent.Accordingly, claims concern the codification readiness of the methodological component, not the readiness of the research idea in every scientific or project-level respect.Source: H Scope of Research-Idea Specifications · p. 51

I.2 GitHub Issue Candidate Cleanup

• Candidate cleanup repairs only gold clarified details or clearly incorrect taxonomy labels, preserves evidence quotes, and routes candidates to retention, review, or rejection.Deterministic quote and taxonomy validation precedes this cleanup step.Source: §I.2 · p. 51, Figure 8 · p. 56

I.3 GitHub Issue Benchmark Instance Construction

• GitHub-based instances pair a paper-derived underspecified specification with an issue-informed codification-ready reference, one atomic defect, taxonomy labels, and an expected clarification action.The issue determines the gold clarification, while fixed labels are inserted rather than re-derived during generation.Source: §I.3 · p. 52, Figure 9 · p. 58

I.4 Reproducibility Paper Gap Extraction

• Reproducibility reports are routed into resolved real-gap, synthetic-controlled, or unusable paths based on method-core defects, resolution evidence, and exclusion of ordinary reproducibility problems.The routing rejects reports limited to compute, resources, data access, software setup, performance mismatch, unresolved gaps, or insufficient evidence.Source: §I.4 · p. 52, Prompt for Reproducibility Report… · p. 59, Figure 10 · p. 60

I.5 Reproducibility Paper Defect Injection

• Synthetic instances remove, abstract, or lightly corrupt exactly one implementation-critical detail from a codification-ready reference while preserving all other information.The generated specification must conceal the gold detail and read naturally without diagnostic defect terminology.Source: §I.5 · p. 52, Prompt for Reproducibility Paper… · p. 61

I.6 Defect Granularity Labeling

• Each defect receives a coarse, medium, or fine granularity label from its taxonomy, codification slot, underspecified form, and gold resolution.The labeling prompt also requires a short rationale and follows the Appendix E.3 rubric.Source: §I.6 · p. 52

I.7 Task 1 Reason Grounding Evaluation

• Task 1 evaluates readiness justifications with support, blocker-match, and faithfulness subscores that are deterministically aggregated into Reason Grounding Score.The evaluator receives the gold readiness label, annotated defects, blocking specifications, rationale, prediction, and model reason.Source: §I.7 · p. 52

I.8 Task 2 LLM-as-a-Judge Evaluation; I.9 Multi-Defect Task 2 Prompt

• Task 2 judges whether a predicted defect identifies the same concrete implementation decision or specification slot as the gold target, independently of taxonomy labels.A multi-defect ablation removes the single-defect cap while retaining taxonomy, boundary, and selectivity criteria.Source: §I.8 · p. 52, §I.9 · p. 52

I.10 Task 3 LLM-as-a-Judge Evaluation

• Task 3 evaluates clarification actions for target relevance, resolution sufficiency, unsupported content, and assumptions, then macro-averages deterministic success across Level-2 defect categories.The evaluator receives the underspecified specification, target defect, private gold resolution, supporting evidence, and predicted action.Source: §I.10 · p. 52

Prompt for GitHub Issue Real-Gap Annotation

• GitHub issues qualify only when they contain an atomic, resolved method-core specification gap with concrete evidence and an implementation-ready clarification.The annotation excludes resource, environment, runtime, usage, unresolved, composite, and code-only issues, while assigning defect taxonomy labels.Source: §I.1 · p. 51, Prompt for GitHub Issue Real-Gap… · p. 53

Prompt for GitHub Issue Benchmark Instance Construction

• The underspecified specification must read like a normal paper method description, avoid diagnostic or GitHub-specific language, and expose only the paper-side surface form without revealing the defect.It must not copy issue questions, invent repository workflows or APIs, or use repo-only terminology as the defect surface form.Source: Prompt for GitHub Issue Benchmark… · p. 57
• The gold reference must be a standalone method-level specification containing the clarified detail, while paper-derived fields remain neutral and unknown paper details are not fabricated.Both specifications cover the full research idea, including goals, inputs and outputs, core method structure, and the affected component.Source: Prompt for GitHub Issue Benchmark… · p. 57

Prompt for Reproducibility Paper Defect Injection

• Synthetic-controlled instances alter exactly one implementation-critical detail while preserving the remaining specification and recording its resolution, evidence, taxonomy label, and preservation checks.The injection prompt prioritizes missing algorithm steps, architecture, objectives, training, preprocessing, input construction, evaluation computation, inference, or prompt construction over weak details such as batch size or hardware.Source: Figure 11 · p. 61, Prompt for Reproducibility Paper… · p. 61

Prompt for Defect Granularity Labeling

• Defect granularity is labeled as coarse, medium, or fine, with coarse defects representing an absent method component that leaves multiple downstream implementation steps underdetermined.The granularity prompt receives the defect labels, codification slot, surface form, and gold resolution before returning the label and rationale.Source: Prompt for Defect Granularity Labeling · p. 62, Figure 12 · p. 62

Task 1 Reason Grounding Judge Prompt

• Task 1 readiness assessment uses a zero-shot specification prompt with counterbalanced READY/NOTREADY option letters, while its reason-grounding judge scores label support, blocker alignment, and faithfulness.The deterministic Reason Grounding Score is computed as 0.7 slabel + 0.2 sblocker + 0.1 sfaithful.Source: Figure 13 · p. 63, Figure 14 · p. 65

Task 2 Zero-shot Evaluation Prompt

• Task 2 instructs models to localize a single controlled defect under the IdeaAMBIG taxonomy without repairing the idea, inventing details, or listing secondary concerns.The output is constrained to strict JSON.Source: Task 2 Zero-shot Evaluation Prompt · p. 66

Task 3 LLM-as-a-Judge Evaluation Prompt

• Task 3 evaluates whether a proposed clarification action would obtain information needed to resolve the annotated defect, rewarding semantic correctness rather than lexical similarity to a reference action.The judge accepts differently worded questions or different action types when they are substantively correct and treats the defect as an adjudicated input.Source: Task 3 LLM-as-a-Judge Evaluation Prompt · p. 73