PaperlayerView exact revision on arXiv

Source-linked AI summary

IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

Yiling Ma, Yilun Zhao, Sihong Wu, Manasi Patwardhan, Arman Cohan

arXiv:2609.10539v1cs.CL

TL;DR

Research ideas can be scientifically plausible yet underspecified for faithful implementation, creating a gap between ideation and execution. The paper introduces IdeaAMBIG, an evidence-grounded benchmark of readiness, localization, and clarification, and finds that models clarify well once the blocker is known but struggle to find it.

  • Problem

    Implementation readiness asks whether a research-method specification lets a competent implementer construct the intended method without unsupported assumptions about core choices.

  • Method

    IdeaAMBIG constructs 660 evidence-grounded single-defect instances from real-world gaps and controlled synthetic defects, evaluating readiness assessment, defect localization, and clarification action generation.

  • Results

    Across 13 LLMs, the strongest model reaches 9.6% Macro Defect Recovery Rate on real-world localization versus 80.6% Macro Clarification Action Success Rate when given the defect.

  • Takeaways & Limitations

    Defect localization is the main bottleneck, while supplying the gold resolution raises downstream codification-ready rate from 14% to 98%.

  • Takeaways & Limitations

    The benchmark is concentrated in AI, NLP, and machine learning, leaving transfer to other computational sciences and naturally occurring early-stage ideas open.

Abstract

from arXiv · show

A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation. We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions. We construct evidence-grounded specifications and their supported resolutions from papers, codebases, issue threads, and reproduction artifacts. We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances: 163 real-world gaps from reproducibility reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references. IdeaAMBIG evaluates three capabilities: codification-readiness assessment, defect localization, and clarification action generation. Defect localization receives only the specification, whereas clarification additionally receives the annotated defect. Across 13 LLMs, the best model achieves 9.6% Macro Defect Recovery Rate on real-world instances but 80.6% Macro Clarification Action Success Rate when given the defect. In an oracle study, supplying the gold resolution raises the downstream codification-ready rate from 14% to 98%. Across all evaluated models, defect localization is the main bottleneck, with stronger clarification given the defect.

1 Introduction

IdeaAMBIG treats implementation readiness as whether a competent implementer can faithfully construct a method without unsupported assumptions, rather than judging novelty or scientific value. It benchmarks readiness assessment, defect localization, and clarification, finding localization weak but clarification strong when the defect is known.

  • Problem definition: Implementation readiness asks whether a proposed methodological mechanism can be faithfully implemented without unsupported assumptions about core method choices.The benchmark distinguishes this from novelty, scientific value, or likely success.
  • Research gap: Existing evaluations leave unresolved whether models can assess specification readiness, locate implementation blockers, and elicit missing information before codification.They typically target idea quality or downstream plans and artifacts while assuming the method is sufficiently specified.
  • Benchmark: IdeaAMBIG contains 660 evidence-grounded single-defect instances: 163 real-world gaps and 497 controlled synthetic defects paired with supported resolutions.The benchmark draws real-world cases from GitHub issues and reproducibility reports, while synthetic cases alter one implementation-critical detail in codification-ready references.
  • Evaluation: The benchmark evaluates readiness assessment, defect localization, and clarification action generation as successive capabilities.Clarification is separated from blocker discovery by providing the annotated defect for the action-generation task.
  • Results: 9.6% Macro Defect Recovery Rate on real-world instances contrasts with 80.6% Macro Clarification Action Success Rate when the annotated defect is given.Across 13 LLMs, models often request useful information once directed to the blocker but struggle to identify it independently.
  • Results: The oracle study raises downstream codification-ready rate from 14% to 98% when the missing information is supplied.This result motivates detecting and resolving specification gaps before codification.

2 Related Work

Related work covers research ideation, scientific design and artifact verification, and specification clarification. IdeaAMBIG extends these lines by focusing on methodological fidelity: a specification is blocking when it underdetermines a method-defining decision, whose resolution must be evidence-supported.

  • Scientific ideation and agents: Research-ideation evaluations measure novelty, feasibility, diversity, alignment, and distributional differences, while research agents connect ideation with review, experimentation, coding, and writing.These approaches motivate evaluating the transition from promising ideas to executable research methods.
  • Scientific design and verification: Scientific benchmarks study inspiration-based reasoning, experiment design, scientific code generation, artifact consistency, limitation identification, and actionable review feedback.These targets differ from assessing whether an incomplete method specification can be faithfully codified.
  • Specification defects and clarification: Requirements and recent LLM benchmarks study ambiguity, incompleteness, inconsistency, and clarification in software or dialogue specifications.IdeaAMBIG shares these defect classes but adds the requirement of methodological fidelity.
  • Specification defects and clarification: A research-specification gap blocks implementation only when it underdetermines a method-defining decision rather than a routine engineering choice.Its intended resolution may be distributed across papers, codebases, issue threads, and reproducibility artifacts, so the resolution must be evidence-supported.

3 IdeaAMBIG

IdeaAMBIG evaluates whether research-method specifications contain enough information for faithful codification, using evidence-grounded real-world and controlled synthetic defects. Its dataset construction isolates atomic implementation-critical gaps and documents their supported resolutions across multiple research sources.

  • Task formulation: IdeaAMBIG evaluates implementation readiness before codification through three diagnostic tasks over evidence-grounded, single-defect research-method specifications.The benchmark focuses on whether a competent implementer can construct the intended method without unsupported assumptions.
  • Task formulation: A specification defect is an omission, ambiguity, or internal inconsistency that leaves a method-defining decision underdetermined.Routine hyperparameters, engineering details, and explicitly open design choices need not be fixed.
  • Task formulation: The benchmark separates readiness assessment, defect localization, and clarification action generation, with localization predicting taxonomy labels and a natural-language blocker description.Clarification generation receives the gold target-defect description and produces an action specifying how to obtain the missing information.
  • Data construction: The dataset combines resolved real-world gaps from reproducibility reports and GitHub issues with controlled synthetic defects injected into codification-ready references.Real-world instances retain atomic, implementation-relevant, evidence-supported defects; synthetic instances alter one implementation-critical detail.
  • Dataset statistics: IdeaAMBIG contains 163 real-world and 497 controlled synthetic instances spanning ten research domains and six methodological components.Incompleteness is the most frequent Level-1 type overall, while real-world instances emphasize ambiguous procedures and synthetic instances emphasize omitted procedures.
  • Dataset validation: Human verification supports the validity and realism of both subsets, with positive judgments ranging from 91% to 98% across evaluated criteria.Reviewers independently checked gap validity, implementation criticality, clarification sufficiency, and realism.

4 Experiments

Experiments evaluate readiness assessment, blocker localization, and clarification generation across 13 LLMs using task-specific metrics and source-aware analyses. Models perform substantially better when the implementation blocker is supplied than when they must discover it from the specification.

  • Readiness assessment: 67.5 Macro-F1 on real-world instances and 86.4 on controlled synthetic instances are achieved by GPT-5.6-Sol for readiness assessment.On real-world instances, it accepts 31% of underspecified specifications and rejects 34% of codification-ready ones.
  • Defect localization: 9.6 Macro DRR on real-world instances is achieved by GPT-5.6-Sol for blocker localization, despite 60.1 Level-1 accuracy and 25.2 Level-2 accuracy.A taxonomy-free ablation improves from 10.0% to 40.0% on same-blocker identification, showing that taxonomy prediction adds difficulty while localization remains challenging.
  • Clarification generation: 80.6 Macro-CAS on real-world instances and 96.2 on controlled synthetic instances are achieved when GPT-5.6-Sol receives the annotated blocker.Its real-world No-Assumption score is 95.7, compared with 80.4 Sufficiency, indicating that remaining failures mainly involve incomplete clarification.
  • Robustness: Source-clustered analyses preserve the main findings, with source-balanced GPT-5.6-Sol estimates differing from instance-level results by at most 0.6 points.The DEFECT-GUIDED advantage remains large and reliable at ∆= 67.0, with 95% CI = [59.0, 75.2] and p < 0.001.
  • Oracle clarification utility: Oracle clarification raises READY rate from 14% to 98% and completeness from 30% to 98% on 50 paired real-world instances.In the complementary executable study, all-tests-passing implementations increase from 45% to 85%, while faithful target-method implementation increases from 30% to 90%.

5 Conclusion

IdeaAMBIG finds that identifying unresolved method-defining decisions is the main bottleneck for current LLMs, while clarification is substantially stronger once the blocker is known. Oracle clarification markedly improves codification readiness.

  • Conclusion: IdeaAMBIG benchmarks readiness assessment, defect localization, and clarification action generation across 660 evidence-grounded single-defect instances.The benchmark includes 163 real-world and 497 controlled synthetic instances.
  • Conclusion: 9.6 Macro DRR on real-world localization contrasts with 80.6 Macro-CAS when the annotated defect is provided.Readiness judgments also remain frequently incorrect or weakly grounded.
  • Conclusion: Oracle clarification raises the codification-ready rate from 14% to 98%, highlighting blocker identification as the main remaining challenge.The conclusion frames a specification-readiness gate as necessary for identifying blockers and seeking grounded clarification.

Limitations

IdeaAMBIG’s scope is limited to pre-implementation assessment, with evidence-resolved single-defect instances concentrated in AI, NLP, and machine learning. Its multi-defect and execution-validity questions remain open.

  • IdeaAMBIG focuses on assessing specifications before implementation rather than evaluating full implementations.
  • Its evidence-resolved, single-defect instances are concentrated in AI, NLP, and machine learning.
  • The benchmark leaves transfer to other computational sciences and naturally occurring early-stage ideas for future study.
  • A 50-pair exploratory ablation found no evidence that combining two known defects made localization harder, but larger response-budget-matched studies remain open.
  • The benchmark evaluates codification readiness before execution rather than through full implementation of every specification.

A.2 Construct Validity Analysis: Specification Clarification vs. Idea Refinement

The construct-validity analysis distinguishes implementation-specification clarification from changes to an underlying research idea. Human annotation indicates that IdeaAMBIG primarily measures the former.

  • Construct distinction: Idea refinement changes objectives, hypotheses, motivations, or scientific directions, whereas specification clarification resolves implementation-critical details without changing the research objective.
  • Annotation protocol: The annotation study sampled 100 instances, split evenly between real-world and controlled-synthetic examples, for independent review by two machine-learning researchers.
  • Annotation protocol: Instances were labeled IDEA-LEVEL CLARIFICATION when resolving the gap required modifying the research idea, and SPECIFICATION-LEVEL CLARIFICATION otherwise.
  • Results: The vast majority of instances across both subsets were categorized as SPECIFICATION-LEVEL CLARIFICATION, with only a small fraction requiring changes to the underlying idea.
  • Results: The analysis confirms that IdeaAMBIG targets implementation-critical specification gaps rather than general-purpose refinement of research ideas.

A.3 Diagnostic Error Analysis

Diagnostic analysis shows that the main difficulty occurs before clarification: models often make weakly grounded readiness judgments and select the wrong blocker. Clarification is substantially more effective once the blocker is supplied.

  • Analysis setup: GPT-5.6-Sol’s diagnostic analysis evaluates readiness, localization, and clarification using task-specific category decompositions and error rates.
  • Readiness decisions: 67.5% of real-world readiness predictions are correct, but only 1.5% are both correct and fully grounded.
  • Defect localization: 16.0% of real-world instances recover the annotated blocker, while 17.8% identify a neighboring decision and 66.3% identify a different blocker.
  • Clarification: 80.6 Macro-CAS on real-world instances and 96.2 on controlled-synthetic instances show stronger clarification after the blocker is supplied.
  • Overall error profile: Overall, plausible but incorrect blocker selection is the main end-to-end difficulty, while clarification is usually effective once the unresolved decision is explicit.

A.4 Qualitative Error Cases

The qualitative cases illustrate failures in readiness assessment, blocker localization, and clarification completeness, while the taxonomy is built from literature-informed categories and evidence-grounded human coding.

  • Qualitative cases: Table 6 presents representative real-world errors covering false readiness, neighboring-blocker localization, and insufficient clarification.
  • Case A: A missing LeakyReLU operation before softmax changes attention coefficients, yet the model treats the graph-attention specification as READY.
  • Case B: In the keypoint head, the annotated defect is a missing 1×1 classifier producing 65 logits per cell, but the model questions the 8 × 8 grid convention instead.
  • Case C: For TimeMixer, the model asks about routing historical variables through x enc or x mark enc but omits whether channel independence must be disabled.
  • Taxonomy construction: The taxonomy begins with Incompleteness, Ambiguity, and Inconsistency, then derives Level-2 categories through corpus mining, open coding, consolidation, and agreement checks.
  • Taxonomy example: Ambiguous Definition covers formal elements whose imprecise meaning, scope, or operational interpretation permits materially different implementations.

B.3.2 Incompleteness

The section distinguishes implementation-critical incompleteness from ordinary configuration questions and inconsistent or ambiguous specifications. It defines missing procedures, data specifications, and model structure through inclusion, exclusion, and boundary rules.

  • Missing Method Procedure: Missing Method Procedure covers omitted operational steps, algorithmic rules, update mechanisms, decision criteria, or execution procedures that prevent faithful reproduction.It excludes configuration-selection gaps and conflicting architectures, which receive separate categories.
  • Missing Method Procedure: Routing representations to losses is implementation-critical because the loss computation and gradient flow cannot be reproduced without the routing rule.The example resolves routing by assigning per-token representations to MLM, a [CLS] representation to NSP/SOP, and no direct loss to the binary representation.
  • Missing Model Structure: Missing Model Structure covers omitted architectural choices that materially affect capacity, information flow, representation shape, or component connectivity.Examples include pooling composition, normalization, activation functions, initialization, fixation-point generation, and dimensional mappings.
  • Missing Data Specification: Missing Data Specification covers omitted rules for constructing, filtering, labeling, normalizing, tokenizing, segmenting, augmenting, or splitting data before use.The boundary is whether the gap changes examples, inputs, labels, or supervision signals rather than how predictions are scored.
  • Exclusion rules: Standard settings or design justifications are rejected when they do not omit an implementation-critical procedure or structural choice.Examples include a reported batch size, an already specified preprocessing pipeline, and a rationale for choosing U-Net.

D.2 Information Bottleneck Analysis for Clarification

The information-bottleneck analysis tests whether clarification failures arise from discovering the defect or formulating an action after the defect is known. Providing the target defect sharply improves clarification quality, while taxonomy-free results indicate discovery remains difficult even without label prediction.

  • Analysis design: The analysis controls defect information to isolate difficulty in discovering specification defects from difficulty in formulating clarification actions.It compares clarification with and without access to the annotated target defect while keeping the original specification unchanged.
  • Clarification results: Macro-CAS increases from 13.6 to 80.6 and Sufficiency from 8.6 to 80.4 when the target defect is provided.Both settings use the same Task 3 instances, prompts, and decoding configurations.
  • Clarification results: No-Assumption remains nearly unchanged at 96.3 versus 95.7, indicating that unsupported implementation choices do not primarily explain the performance gap.Without the defect, models more often produce cautious but insufficient or misdirected clarification requests.
  • Interpretation: Defect discovery is the primary bottleneck: models can request missing information effectively once directed to the correct implementation-critical gap.They remain substantially less reliable at identifying which unresolved decision requires clarification.
  • Taxonomy-free localization: Taxonomy-free blocker identification removes Level-1 and Level-2 label prediction, yet remaining failures show that discovering the unresolved decision itself remains difficult.The ablation evaluates natural-language defect descriptions on the same underspecified specifications.
  • Scope: Executable validation is limited to bounded components because the study excludes large-scale training, private data, unavailable services, and complete pipeline reproduction.The setup uses fixed instance-specific tests or verification criteria derived from papers and reference code.

D.5 Source-Level Robustness and Uncertainty

The evaluation accounts for dependence among instances sharing repositories, source papers, or projects through source-clustered uncertainty and balanced estimates. Results show that the main cross-task bottleneck remains blocker identification rather than clarification once the blocker is supplied.

  • Uncertainty estimation: Source-clustered bootstrap sampling includes all instances from each sampled source cluster and recomputes the complete primary metric across 10,000 replicates.The procedure retains category representation requirements for Tasks 2 and 3 and reports percentile 95% confidence intervals.
  • Source balancing: Source-balanced estimates assign equal total weight to each source cluster within each readiness class before computing class-specific F1 and their unweighted mean.This tests whether sources contributing multiple inputs disproportionately influence results.
  • Results: Source-level clustering is nontrivial: 200 Task 1 real-world and controlled-synthetic inputs originate from only 34 and 31 source clusters, respectively.Despite this clustering, source-balanced GPT-5.6-Sol estimates differ from instance-level estimates by at most 0.6 points across six task–subset combinations.
  • Results: 9.6% Macro DRR persists on real-world instances, while defect-guided Macro-CAS reaches 80.6%; most GPT-5.6-Sol versus Claude Sonnet 5 differences are not stable rankings.Five of six exploratory model comparisons have confidence intervals containing zero; only the Task 1 controlled-synthetic difference remains reliable after Holm adjustment.
  • Blocker-availability effect: 67.0 points of real-world Macro-CAS improvement follows from providing the annotated blocker under paired source-clustered resampling.The 95% CI is [59.0, 75.2] with p < 0.001, supporting blocker identification as the primary bottleneck after accounting for source dependence.
  • Audit and validity: The benchmark audit retains 660 single-target instances after adjudication, with 624 unchanged, 21 revised, 7 split into 15, and 8 removed.Only instances whose annotated target remained the unique primary blocker were retained.

E.4 Multi-Defect Readiness and Localization Ablation

The multi-defect ablation combines two independently validated defects in controlled-synthetic specifications and compares them with paired single-defect evaluations. Combining defects does not make strict localization harder in this exploratory setting, although the analysis is limited to controlled-synthetic instances.

  • Design: The ablation combines two known, independently validated defects into one specification to test whether the single-target setting understates difficulty.The benchmark’s standard construction isolates one implementation-critical decision, while the ablation probes specifications containing more than one defect.
  • Evaluation: The comparison evaluates GPT-5.6-Sol on readiness and multi-defect localization, using the same taxonomy, boundary rules, and selectivity criteria as the single-target prompt.Predictions are scored independently against both gold targets and compared with paired single-defect siblings covering the identical defect pair.
  • Results: 88.0% versus 80.0% readiness assessment is directionally higher for combined defects, with ∆= +8.0 and McNemar p = 0.289.This result is consistent with the intuition that specifications containing more independent problems are at least as likely to be flagged NOTREADY.
  • Results: 64.0% versus 36.0% Any-of-2 localization is significantly higher for combined defects, with ∆= +28.0 and p = 0.0013.All-of-2 is also directionally higher at 8.0% versus 2.0%, ∆= +6.0, p = 0.250, providing no evidence that combination makes strict localization harder.
  • Scope: The ablation uses only controlled-synthetic instances because real-world sibling instances lack a shared reference specification for combination without additional content synthesis.It is a secondary exploratory check, not a claim about naturally co-occurring defects; 94.5% of instances overall have a unique primary blocker under the single-target construction.
  • Annotation criteria: Readiness distinguishes READY, NOT READY, and UNSURE according to whether core-method choices can be implemented without unsupported assumptions.Ordinary engineering choices such as seeds, hardware, software patch versions, and conventional optimizer settings generally do not determine NOT READY.

F.2 Task 2 Evaluation Metrics

Task 2 evaluates whether models recover a single annotated implementation-critical defect and classify it at two taxonomy levels. Its primary metric requires both correct target localization and correct Level-1 and Level-2 labels, while diagnostics separate localization from taxonomy assignment.

  • Evaluation setup: Task 2 uses atomic NOTREADY specifications containing exactly one annotated target defect and one model diagnosis.Evaluation is centered on the implementation-critical defect associated with each instance, not any plausible issue in the specification.
  • Localization: Loc-Acc measures whether the predicted defect description refers to the same atomic defect as the gold annotation.An LLM-based semantic matcher allows paraphrases and differences in specificity when they identify the same target decision; in this single-target setting, Loc-Acc equals Recall@1.
  • Primary metric: Macro DRR counts a hit only when the model localizes the target defect and predicts both taxonomy levels correctly.Macro-averaging gives equal weight to observed Level-2 categories, preventing frequent defect types from dominating the aggregate.
  • Diagnostics: L1-aware recovery requires target localization and the correct Level-1 category but does not require a correct Level-2 label.Unlike Macro DRR, it is micro-averaged across instances.
  • Diagnostics: L1-aware recovery remains below Loc-Acc across models, indicating mismatches between relevant target descriptions and coarse taxonomy labels.For Qwen3-8B on the synthetic subset, Loc-Acc is 10.0 while L1-aware recovery is 2.0.

G.1 Human Validation of LLM-Based Evaluation

The study validates Claude Opus 4.8’s rubric-based judgments against blinded human annotations and examines whether synthetic defects preserve the benchmark’s intended construct. Agreement is generally strong, while synthetic instances receive high validity, implementation-criticality, realism, and resolution ratings.

  • Validation design: The validation samples 300 outputs across tasks, balanced between real-world and synthetic instances and stratified by defect category and model group.The same candidate output is reused across all judgment dimensions for its task.
  • Human annotation: Two machine-learning researchers independently annotate blinded outputs using the automatic evaluator’s definitions, with disagreements adjudicated by a third annotator.Annotators receive only the task-specific information available to Claude Opus 4.8.
  • Judgment criteria: For Task 2, human judgments assess semantic equivalence of the candidate diagnosis to the same atomic implementation-critical defect rather than lexical overlap.Task 3 judgments separately assess target relevance, resolution sufficiency, and unsupported assumptions.
  • Agreement results: Task 2 evaluator agreement with adjudicated human judgments reaches 86.0%, while Task 1 exact agreement ranges from 76.0% to 84.0%.Task 1 weighted κ ranges from 0.68 to 0.78, with MAE below 0.14.
  • Synthetic validation: Synthetic instances are judged valid specification gaps by 96.0% of annotators and implementation-critical by 94.0%.These rates are reported as evidence that controlled defect injection preserves the benchmark construct.
  • Synthetic validation: 91.0% of synthetic instances are judged plausible workflow failures, and 98.0% are judged correctly resolved by the provided clarification.The synthetic realism rate is close to the 93.0% rate for real-world instances.

G.4 Human Baseline Evaluation

The human baseline uses experienced machine-learning researchers evaluated under the same task inputs and metrics as the LLM study. Humans outperform models on defect localization but still find the task imperfect, while clarification is easier once the target defect is known.

  • Evaluation protocol: The human baseline evaluates 100 records, split evenly between real-world and controlled synthetic instances.The sample preserves the benchmark’s overall Level-1 defect distribution.
  • Evaluation protocol: Two researchers independently complete each task, and a third reviewer adjudicates disagreements.Participants receive the same task-specific inputs and instructions as the evaluated LLMs without additional evidence.
  • Metrics: Human predictions use Macro-F1 for Task 1, Macro DRR for Task 2, and Macro-CAS for Task 3.These metrics follow the main LLM evaluation protocol.
  • Results: Humans substantially outperform LLMs on defect localization across both benchmark subsets, although their performance remains imperfect.The result indicates that identifying implementation-critical gaps requires careful methodological reasoning even for experienced researchers.
  • Results: Humans achieve high Macro-CAS once the target defect is identified, supporting the finding that blocker resolution is easier than blocker discovery.The benchmark’s main challenge is accurately locating the unresolved implementation decision.

H Scope of Research-Idea Specifications

IdeaAMBIG evaluates whether a research idea’s proposed methodological mechanism is specified for faithful codification, not whether the broader idea is scientifically complete. Its benchmark inputs are reconstructed implementation-facing specifications based on retrospective evidence.

  • Scope: The benchmark focuses on the proposed methodological mechanism within a research idea and its readiness for faithful codification.It does not assess novelty, scientific value, motivation, or broader research-idea completeness.
  • Evidence basis: Authentic pre-implementation information exchanges are rarely preserved, so specifications are reconstructed from papers, codebases, reports, issues, and executed projects.These artifacts support identifying implementation-critical gaps and their resolutions.
  • Scope boundary: The benchmark’s claims concern readiness of the methodological component, not readiness of a research idea in every scientific or project-level respect.The inputs represent information that would need to be available to a competent implementer or coding agent.

I Prompts

The prompts operationalize IdeaAMBIG’s construction and evaluation by extracting atomic, resolved method-core gaps, labeling their defect types, and measuring readiness, localization, and clarification. They distinguish implementation specification problems from unrelated reproducibility or software issues and use evidence-grounded judgments throughout.

  • I.1 GitHub Issue Extraction: GitHub issue prompts identify resolved, method-core specification gaps and convert them into atomic implementation-ready clarifications with IdeaAMBIG taxonomy labels.
  • I.2 GitHub Issue Candidate Cleanup: GitHub candidates undergo validation and cleanup that can repair gold clarified details or taxonomy labels without rewriting the underlying evidence quotes.
  • I.4 Reproducibility Paper Gap Extraction: Reproducibility reports are routed into resolved real-gap, synthetic-controlled, or unusable paths, excluding issues without identifiable method-core specification defects.
  • I.5 Reproducibility Paper Defect Injection: Synthetic-controlled instances modify exactly one implementation-critical detail while preserving all non-target information from a codification-ready reference.
  • I.6 Defect Granularity Labeling: Defect granularity is labeled as coarse, medium, or fine using taxonomy labels, the codification slot, the underspecified surface form, and the gold resolution.
  • I.7 Task 1 Reason Grounding Evaluation: Task 1 evaluates readiness decisions and justifications through label support, blocker match, and faithfulness against annotated defects and rationales.
  • I.8 Task 2 LLM-as-a-Judge Evaluation: Task 2 semantically matches predicted defects to the same concrete implementation decision or specification slot as the annotated target.
  • I.10 Task 3 LLM-as-a-Judge Evaluation: Task 3 evaluates clarification actions for target relevance, resolution sufficiency, and unsupported assumptions, then deterministically macro-averages success across Level-2 defect categories.
Loading 2609.10539v1…