PaperlayerView exact revision on arXiv

Source-linked AI summary

HARDEN: Constrained Evolutionary Search for Harder, Answer-Preserving Evaluation Cases

Aditya Kumaran, Rahul Singhal, Karime Maamari, Amine Mhedhbi, Pradyumna Tambwekar

arXiv:2609.30571v1cs.AI

TL;DR

Curated benchmarks can underrepresent the complexity of real-world deployments, and making cases harder can risk changing their intended answers. HARDEN evolves existing evaluation inputs along domain-specific complexity axes while enforcing feasibility constraints and keeping expected outputs fixed. Across three benchmarks and three Qwen3.5 model scales, it reduces accuracy relative to single-pass baselines using the same feasibility checks.

  • Problem

    Curated benchmarks often abstract away real-world ambiguity and contextual complexity, while harder modifications can invalidate the evaluation target by changing the correct answer.

    Source: §1 · p. 1

  • Method

    HARDEN uses constrained evolutionary search to modify existing evaluation inputs along domain-specific complexity axes while keeping expected outputs fixed and checking correctness, realism, and validity.

    Source: §1 · pp. 1–2, §3 · p. 2

  • Results

    22.7% average accuracy reduction and up to 49.9% reduction were reported across FinQA, PubMedQA, and ContractNLI relative to single-pass baselines using the same feasibility checks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across the studied domains and Qwen3.5 model scales, HARDEN constructs harder, answer-preserving evaluations, with effects remaining substantial as task-model capability increases.

    Source: §6 · p. 6

    The evaluation covers three professional-domain benchmarks and one task-model family, so trends across Qwen3.5 scales do not establish transfer across architectures.

    Source: §5 · p. 6

Abstract

from arXiv · show

Language models are often evaluated on curated benchmarks that underrepresent the complexity of enterprise deployments. We introduce HARDEN, a constrained evolutionary search method to adapt the input of existing evaluation cases into more challenging variants while keeping their expected outputs fixed. HARDEN searches along generated domain-specific complexity axes while enforcing feasibility constraints such as preserving task semantics, realism, and execution validity. Across FinQA, PubMedQA, and ContractNLI and three Qwen3.5 model scales (35B-A3B, 122B-A10B, and 397B-A17B), HARDEN reduces task-model accuracy by 22.7% on average and by up to 49.9% relative to single-pass baselines using the same feasibility checks. These results show that evolutionary search can produce substantially harder valid evaluation cases.

1 Introduction

Curated benchmarks can abstract away real-world ambiguity and contextual complexity, while making cases harder can also invalidate their original evaluation targets. HARDEN adapts existing cases under constraints and reports harder valid cases across professional domains and model sizes.

1 Introduction

  • Benchmark inputs can omit ambiguity, grounding requirements, and contextual complexity, weakening their signal of reliability for deployment.The introduction notes that high benchmark effectiveness may not carry over to deployments.Source: §1 · p. 1
  • Increasing difficulty alone can change the correct answer by introducing contradictions or removing necessary evidence, so the evaluation target must be preserved.The motivating example changes a referenced fiscal period in a financial reasoning case.Source: §1 · p. 1
  • HARDEN modifies existing benchmark cases to increase task complexity and model uncertainty while enforcing correctness, realism, and execution validity.Correctness preserves evidence needed to derive the expected output; realism requires deployment-domain plausibility, and validity requires well-formed executable cases.Source: §1 · pp. 1–2
  • The framework adapts cases along domain-specific complexity axes and uses language-model feasibility checks for correctness and realism, evaluated against human judgments.Variants that fail either check are discarded.Source: §1 · p. 2
  • 22.7% average accuracy reduction and up to 49.9% reduction, with normalized output semantic entropy up 112% on average, are reported against single-pass baselines using the same feasibility checks.The evaluation spans professional domains and model sizes.Source: §1 · p. 2

2 Related Work

Prior work has pursued answer-preserving adversarial cases and constrained generation for specific tasks. HARDEN distinguishes itself by treating correctness, realism, and validity as separate constraints rather than relying only on semantic similarity or predefined transformations.

2 Related Work

  • Answer-preserving adversarial testing adds distractors to SQuAD, while SEARs derives semantically equivalent replacement rules across NLP tasks.Population-based search has also been used for semantically and syntactically similar adversarial text, and TextAttack formalized attacks through transformations, constraints, objectives, and search.Source: §2 · p. 2
  • More recent constrained methods include semantic-equivalence-preserving prompt search, semantics-preserving code and Text-to-SQL transformations, validated reasoning-graph edits, and instruction-preserving adversarial suffixes.The cited methods are SECA, CETBench, SQLMorph, DARG, and AdvPrompter.Source: §2 · p. 2
  • HARDEN treats correctness, realism, and validity as separate constraints rather than relying only on semantic similarity or predefined transformations.Prior approaches include adversarial text, code, Text-to-SQL, reasoning-graph, and prompt methods.Source: §2 · p. 2

3 Our Approach: HARDEN

HARDEN searches for harder candidate inputs while holding each case’s evaluation target fixed and enforcing feasibility constraints. It uses domain analysis to set complexity axes and a realism rubric, then evolves variants and returns an improved feasible candidate when one is found.

3 Our Approach: HARDEN

  • The search minimizes a task-model fitness score over candidate inputs while requiring correctness, realism, validity, and a positive score that excludes cases collapsing the task.Correctness preserves evidence needed for the expected output; realism requires domain plausibility; validity requires well-formed execution by the original benchmark.Source: §3 · p. 2
  • Domain analysis identifies eight to eleven domain-specific complexity axes with initial priorities and constructs a realism rubric using benchmark details and external web sources.The rubric is calibrated on a held-out set before advancement.Source: §3 · pp. 2–3
  • For each selected case, the original input initializes an independent search across generations, with a mutation model generating variants and a separate task model evaluating them.The search uses the task model to estimate fitness.Source: §3 · pp. 2–3
  • Each variant selects a domain-analysis complexity axis to guide mutation, and axis priority adapts using initial ranking, prior feasibility and fitness outcomes, and prior use.The selection favors high-ranked axes while encouraging diversity across variants; the mutation model determines how the selected complexity is realized.Source: §3 · p. 3
  • Candidates failing any correctness, realism, or validity check are discarded; feasible candidates are retained by fitness, with semantic entropy favoring greater uncertainty in ties.A remaining tie favors a newly generated variant.Source: §3 · p. 3
  • After the final generation, HARDEN returns the best candidate only if its fitness improves on the original and feasibility checks pass; otherwise it retains the original input.Source: §3 · p. 3

4 Experiments

Experiments compare HARDEN with single-pass mutations across three professional-domain benchmarks and three Qwen3.5 scales, and assess feasibility checks against human judgments. HARDEN produces more challenging cases than the baselines, while the check-validation evidence is mixed.

4.1 Setup

  • The study samples 200 cases each from FinQA, PubMedQA, and ContractNLI, testing three Qwen3.5 scales against tool-free and web-search few-shot mutations.It measures execution accuracy for FinQA, exact label match for PubMedQA and ContractNLI, and semantic entropy for predictive uncertainty.Source: §4.1 · pp. 3–4

4.2 Results

  • Across benchmark-model combinations, HARDEN reduces accuracy for 52.4% of original cases, versus 12.7% for Few-Shot and 13.3% for Few-Shot (web-search).Lower accuracy indicates more effective hardening.Source: Figure 1 · p. 4, §4.2 · pp. 4–5
  • At 35B, 122B, and 397B, accuracy falls 21.5%, 25.2%, and 21.2%; entropy averages 112% above constraint-filtered baselines.At the largest scale, HARDEN successfully hardens an average of 81.7 out of 200 cases per benchmark, compared with 33.0 for the strongest baseline at the smallest scale.Source: Figure 1 · p. 4, Figure 2 · p. 4, Figure 3 · p. 5
  • Realism-check agreement was 51.7% on FinQA and 85.0% on PubMedQA; correctness validation remained limited because participants accepted 39 of 40 mutations.The separate correctness study achieved 85.7% agreement on a FinQA set designed to test detection of incorrect examples.Source: §4.2 · p. 5

5 Limitations

The evaluation spans three professional-domain benchmarks but is limited to one task-model family and one mutation model. Validation studies are small and mixed, and evolutionary search costs substantially more than single-pass mutation.

5 Limitations

  • The experiments cover three professional-domain benchmarks, but use only one task-model family and one mutation model.The tested Qwen3.5 scales do not establish transfer across architectures.Source: §5 · p. 6
  • Reasoning was disabled for all task models, leaving its effect on robustness to mutated examples open.Cross-model transfer of difficulty also remains unmeasured.Source: §5 · p. 6
  • Feasibility-check validation is small and mixed, while evolutionary expansion makes HARDEN substantially more expensive than single-pass mutation.The secondary adjudicated correctness study covers only 21 FinQA pairs.Source: §5 · p. 6

6 Conclusion

HARDEN adapts curated benchmark inputs with constrained evolutionary search while preserving evaluation targets. Across the studied domains and Qwen3.5 scales, it increases task difficulty relative to single-pass agentic baselines.

6 Conclusion

  • HARDEN injects domain-realistic complexity into benchmark inputs while preserving their evaluation targets.The method uses constrained evolutionary search.Source: §6 · p. 6
  • Across FinQA, PubMedQA, and ContractNLI, HARDEN increases task difficulty as measured by accuracy and uncertainty relative to single-pass agentic baselines.Source: §6 · p. 6
  • The effect remains substantial as Qwen3.5 capability increases from 35B to 122B to 397B.The conclusion is scoped to the studied domains and model scales.Source: §6 · p. 6

A HARDEN artifacts and check validation

The appendix records experimental artifacts to make information boundaries, calibration decisions, and check behavior auditable.

A HARDEN artifacts and check validation

A.1 Example target-blind realism rubric

The FinQA realism rubric uses explicit pass thresholds and blocking flags. Applied diagnostically to synthetic cases, it rejected cases including those flagged for degenerate tables.

A.1 Example target-blind realism rubric

  • A case passes only if its mean and every dimension score at least 4 out of 5, with no blocking red flag.The blocking flags are an absent or degenerate financial table and numeric cells lacking an authentic accounting convention.Source: §A.1 · p. 8
  • The rubric also lowers relevant dimension scores for generic placeholder vocabulary or questions ungrounded in the supplied report.Source: §A.1 · p. 8
  • In the diagnostic check, 19 of 20 rejected GPT-3.5 Turbo cases triggered the degenerate-table flag.The frozen rubric was applied to original and synthetic FinQA cases; per-source acceptance rates are reported in Table 10.Source: §A.1 · p. 8

A.2 Example complexity-axis catalog

The FinQA domain-research stage produced ten complexity axes, each describing realistic target-preserving mutation families rather than a single fixed transformation.

A.2 Example complexity-axis catalog

  • The domain-research stage produced a catalog of ten FinQA complexity axes.Source: §A.2 · p. 8
  • Each axis specifies a family of realistic mutations that preserve the evaluation target.Source: §A.2 · p. 8
  • The catalog’s axes describe mutation families, not one fixed transformation per axis.Source: §A.2 · p. 8

A.3 Example original and HARDEN-generated case

HARDEN adds ceremonial legal material while retaining the evaluation target and operative agreement clauses. In the example, the mean Qwen 3.5 397B score falls from 1.0 to 0.3 over ten trials.

A.3 Example original and HARDEN-generated case

  • The hardened case adds legal boilerplate and recital inflation without changing the operative agreement clauses.Source: §A.3 · p. 8
  • The mean Qwen 3.5 397B score falls from 1.0 to 0.3 over ten trials on the hardened ContractNLI case.The added material is ceremonial and non-determinative, while operative clauses are preserved verbatim.Source: §A.3 · p. 8
  • The mutation leaves the hypothesis’s gold label NotMentioned unchanged.The hypothesis concerns whether the Receiving Party must destroy or return some Confidential Information upon termination.Source: §A.3 · p. 8

A.4 Search and task model details

The section describes how rubric research is separated from complexity-axis discovery, then details model evaluation, uncertainty measures, and reproducibility disclosure.

A.4 Search and task model details

  • Rubric research receives the use-case description, schemas, execution plan, and system prompt, but not target cases, labels, mutations, evaluator results, or task-model scores.Complexity-axis discovery is separate and may inspect target inputs, immutable outputs, and the evaluator to identify label-preserving difficulty mechanisms.Source: §A.4 · pp. 8–9
  • Each benchmark is evaluated with Qwen 3.5 35B, 122B, and 397B models served through OpenRouter with reasoning disabled.Ten sampled completions support primary performance and uncertainty; a temperature-zero completion and token log-probabilities are retained for post-hoc sequence NLL and mean-token NLL.Source: §A.4 · p. 9
  • Enabling reasoning showed mixed accuracy effects in the preliminary evaluation and substantially increased token use and latency.Accuracy decreased by 3.3%, from 76.7% to 73.3%, on FinQA with the 35B model, and increased by 3.3%, from 46.7% to 50.0%, on PubMedQA with the 122B model; token usage rose approximately 3.7× on FinQA and 8.3× on PubMedQA.Source: §A.4 · pp. 9–10
  • Appendix A.5 discloses prompts, schemas, search configuration, model settings, and tool boundaries, but releases no code, data, or exact run outputs.The disclosure is intended to support an independent implementation and reproduction of the experimental procedure.Source: §A.4 · p. 10
  • Post-hoc likelihood-weighted semantic entropy groups samples into task-level semantic classes and weights them by length-normalized sequence likelihood, while online selection estimates class probabilities from sample frequencies.FinQA classes are distinct executed numerical results with normalization log T, where T = 10; PubMedQA and ContractNLI use three answer labels and normalization log 3.Source: §A.4 · p. 10

A.5 Reproducibility prompt and configuration disclosure

Appendix A.5 discloses the method-critical prompts, schemas, and configuration used to generate and evaluate candidate inputs, including checks for correctness, answer preservation, and realism and the single-pass baseline template.

A.5.1 Search, model, and tool configuration

  • The search samples 200 cases without replacement using seed 42, preserves source order, and partitions them into ten deterministic worker runs sharing one mutation-axis catalog.Source: §A.5.1 · p. 10
  • Task-model profiles use one greedy and ten sampled completions, temperature 0 for greedy calls, disabled reasoning, and up to 16,384 output tokens; the three Qwen profiles vary provider and sampling parameters.Rubric research, axis research, and mutation use Claude Opus 4.8; direct correctness and realism checks use temperature 0, JSON responses, and majority voting for three-member ensembles.Source: §A.5.1 · pp. 10–11
  • Deterministic selection keeps the five highest-ranked cases, rejects zero scores, and promotes a final case only after strict improvement over the immutable baseline.Before scoring, code checks require valid mutation metadata, explicit correctness and realism passes, and edits limited to declared inputs; finalization rebuilds cases from immutable snapshots and checks structural completion.Source: §A.5.1 · p. 11

A.5.2 Rubric research templates

  • Rubric research is target-blind and requires independent benchmark and primary real-world sources, evidence-backed authenticity dimensions, and calibration on disjoint probe and held-out samples.The rubric excludes expected answers and evaluator outputs as realism evidence, and its pass rule requires a minimum mean score of 4.0, all dimensions meeting threshold, and no blocking red flags.Source: §A.5.2 · p. 11

A.5.3 Axis research templates

  • Axis research treats every case in the corpus as in scope and derives a reusable, evidence-backed catalog of plausible domain-specific transformations while preserving immutable gold outputs.Each axis operates only on input artifacts; exactly one must be named `domain_specific_realism`, and each requires at least three subtypes and five prompt-guidance strings.Source: §A.5.3 · pp. 13–14

A.5.4 Mutation template and context schema

  • Mutation-axis priorities combine prior application success with research priority, penalties for repeated failure or lineage reuse, and—in FinQA—bonuses for table-label compression and table-lookup axes.The mutation agent receives scores and counts, then selects one axis per offspring while preferring distinct, previously unused axes.Source: §A.5.4 · pp. 14–15
  • Mutation generation produces exactly the requested number of candidates, applying one supplied axis per candidate only to writable input files and leaving acceptance decisions to independent gates.The generator must preserve load-bearing information and avoid planted traps or deletion of necessary information; it writes a manifest and does not run or predict executor, evaluator, or gate outcomes.Source: §A.5.4 · p. 15
  • Mutation calls receive a deterministic context and must generate exactly the requested number of candidates, write replacements and a manifest, and avoid judging correctness or fitness.The context includes candidate directories and writable files, case and population state, candidate axes, axis feedback, realism criteria, and gold-output information.Source: §A.5.4 · pp. 15–16

A.5.5 Correctness extraction and audit prompts

  • The correctness check makes one extraction call for the parent case and one audit call for each candidate.The appendix presents the complete user requests after fixed instructions and dynamic inputs are combined.Source: §A.5.5 · p. 16
  • Extraction identifies only input information necessary for the expected output and requires quoting exact output text with a direct causal link for each necessity claim.It excludes supplementary data and context-only information, and asks for the requested task semantics as a necessary item.Source: §A.5.5 · pp. 16, 18
  • The requested task semantics must capture the target, operation, direction, scope, units, and answer form; paraphrases are invalid if they change output semantics.The audit is instructed to reject changed operations, comparison directions, target entities, periods, or unsupported requests, even when the relevant facts remain present.Source: §A.5.5 · pp. 16, 19
  • Extraction records each necessary fact’s input source, semantic label, value, decision criteria, description, and exact-output reason.Decision criteria specify valid and invalid values that would preserve or change the output.Source: §A.5.5 · pp. 16–18
  • When necessary facts interact, extraction records their relationships and how changes affect output preservation, including OR, AND, conditional, compensatory, and hierarchical dependencies.The instructions also require coverage of every output component and a relationship between task semantics and facts needed to answer it.Source: §A.5.5 · pp. 17–18
  • The audit searches version 2’s input for every necessary fact and checks its value against the extraction’s decision criteria; semantically equivalent paraphrases are acceptable.It must also compare the explicit version-2 request with the extracted task semantics rather than infer the request from surrounding facts.Source: §A.5.5 · p. 19
  • The correctness audit requires checking exact semantic meaning, values, qualifiers, context, and whether the candidate still requests the identical task and answer form.Any failed check must be marked missing rather than preserved.Source: §A.5.5 · p. 20
  • A correctness pass requires all necessary information and relationships to be preserved; any decision-critical change or broken relationship means fail.The prompt calls for validating entity relationships as well as individual entities.Source: §A.5.5 · pp. 20–21
  • The correctness audit treats task instructions, output format, and normalization rules as preserved when they reside in the unchanged baseline prompt, and audits only case-specific information in the candidate.The output includes a pass verdict, preserved and missing information, a rationale, and validation reasoning.Source: §A.5.5 · pp. 21–22

A.5.6 Realism audit-call template

  • The realism audit is independent of the mutation generator and correctness auditor, and judges domain authenticity against an immutable rubric and reference profile—not answer correctness or difficulty.Deterministic code checks rubric scores, thresholds, blocking red flags, and ensemble voting; only correctness-approved candidates are audited.Source: §A.5.6 · p. 22

A.5.7 Single-pass baseline template

  • The single-pass baseline asks for one realistic, self-contained, more challenging input while keeping the expected output exact and preserving the schema and all decision-relevant facts and constraints.It prohibits contradictions, false facts, corruption, answer leaks, ambiguity, and instructions directed at the solver.Source: §A.5.7 · pp. 22–23
  • The baseline may use reference examples only as descriptive guidance and read-only web tools only for domain conventions, terminology, and document structures—not target records or answers.The prompt requests a complete schema-conforming replacement input and checks for realism and correctness retention.Source: §A.5.7 · p. 23

A.6 Benchmark details

The evaluation uses three professional-domain benchmarks with distinct task formats, scoring rules, and data licenses.

A.6 Benchmark details

  • FinQA, PubMedQA, and ContractNLI represent financial, biomedical, and legal tasks, respectively, with 2,091 ContractNLI pairs formed from 123 agreements and 17 hypotheses.FinQA is scored on executed answers and program equivalence; PubMedQA and ContractNLI use exact-match labels.Source: §A.6 · p. 23
  • FinQA evaluates executed numerical programs, while PubMedQA and ContractNLI use fixed three-way labels; ContractNLI mirrors first-pass NDA review.PubMedQA is described as plateaued for models at the evaluated scales.Source: §A.6 · p. 23
  • FinQA and ContractNLI data use CC BY 4.0, PubMedQA uses MIT, and the Qwen3.5 model weights use Apache 2.0.The FinQA official code and transcribed evaluation code use MIT; Claude Opus 4.8 and GPT authoring services are proprietary, and OpenRouter access is subject to its terms and applicable model-provider terms.Source: §A.6 · pp. 23–24

A.7 Task model prompts

This section describes benchmark-specific prompts and deterministic answer extraction, alongside a secondary FinQA validation of the correctness check. The validation tests selected output-invalidating mutations and controls, while its limited, class-imbalanced examples constrain what acceptance establishes.

A.7 Task model prompts

  • Each prompt combines the official task definition, delimited input fields, and a machine-parsable final line for consistent scoring of ten sampled generations.Evaluations send one user message without a separate system message; regex extraction handles the reason-then-answer format.Source: §A.7 · p. 24
  • FinQA uses the official executable program grammar, PubMedQA follows its reasoning-required yes/no/maybe setting, and ContractNLI retains conservative three-way labels rather than binarizing them.ContractNLI elicits labels only because evidence identification is a separable subtask and evidence spans are unstable under input mutation.Source: §A.7 · pp. 24–25
  • FinQA prompts score 71.6–73.8%, PubMedQA 76.1–77.7%, and ContractNLI 70.5–72.9%, serving as plausibility checks rather than exact calibration.The reported comparisons are respectively zero-shot DSL prompting and the fine-tuned FinQANet baseline; other LLM and human-performance ranges; and the fine-tuned Span NLI ceiling and three-way chance floor.Source: §A.7 · p. 26

A.8 Baseline configuration

  • Both single-request baselines use five scored reference examples from a shared bank of 20 lowest-scoring cases per benchmark, sampled without replacement with case-specific seeds.A disjoint 100-case pool is scored using the reasoning-off largest task model; references provide difficulty inspiration only and must not supply content.Source: §A.8 · p. 26

A.9 Why the single-pass baselines inject complexity weakly

  • Single-pass edits mainly add context length, paraphrase, domain-authentic formatting, or boilerplate while leaving answer-bearing evidence and operative quantities or clauses unchanged.The baselines cannot select rare valid, realistic, difficult edits based on task-model performance, whereas evolutionary search can compound valid changes.Source: §A.9 · pp. 26–27

A.10 Accuracy and uncertainty results

  • HARDEN raises all three post-hoc uncertainty measures in every benchmark-model combination; on successfully hardened cases, accuracy falls by 25–64 points and normalized semantic entropy rises by 0.28–0.43.The successful-case comparison excludes cases that reverted to their originals, and immutable evaluation targets remain unchanged.Source: §A.10 · pp. 27–28

A.11 Human-participant check-validation study

  • Human-check validation found 51.7% direct realism agreement on FinQA, 18.2% agreement on unrealistic PubMedQA cases, and participant correctness specificity of 10.0% on FinQA and 0.0% on PubMedQA.PubMedQA realism agreement was 85.0% overall, while feasibility-check provenance accuracy was 55% (22/40); the results motivate threshold recalibration and further correctness validation.Source: §A.11 · p. 33

A.12 Secondary correctness-check validation

  • A secondary correctness-check evaluation uses agreed co-author labels for 21 FinQA pairs after excluding four split judgments, with corresponding PubMedQA evaluation left to future work.The submitted FinQA bank contained 25 pairs spanning manual mutations and two single-model mutation operations.Source: §A.12 · p. 33
  • It rejects all ten tested single-model counterfactual and label-flip examples and accepts all three agreed manually degraded correct controls.These results support distinguishing the tested explicit output-invalidating mutations from the tested valid controls, not indiscriminate rejection.Source: Table 13 · p. 34, §A.12 · p. 34
  • The check matches 18 of 21 agreed FinQA labels, with 81.6% balanced accuracy, Cohen’s κ = 0.577, and ϕ/MCC = 0.583.Table 14 reports secondary correctness performance on the co-author-agreement set; its confidence intervals are Wilson 95% intervals for accuracy.Source: Table 14 · p. 34, §A.12 · p. 34
  • The check rejects one pair both co-authors judge correct, treating a changed table header as decision-critical despite compensating narrative context.All three feasibility-check decisions are unanimous.Source: §A.12 · p. 34
  • It accepts two pairs both co-authors judge incorrect, overlooking a location mismatch with unchanged numeric operands and a shifted temporal bucket in a financial table.All three feasibility-check decisions are unanimous.Source: §A.12 · p. 34
  • The evaluation is deliberately small and class-imbalanced, with four of 25 submitted FinQA pairs excluded after split judgments.Its model-generated negatives were prompted with the original output, while all three agreed correct controls were manually authored.Source: §A.12 · p. 34
  • The check is a useful but incomplete validity filter: it can over-reject benign inconsistencies and over-accept subtler entity-alignment or temporal-semantics errors.The tested model-generated changes were detected and the agreed controls preserved, but their construction may differ from naturally occurring invalid mutations.Source: §A.12 · p. 34
  • Larger adjudicated studies, including a corresponding PubMedQA evaluation, are needed before check acceptance can be treated as conclusive proof of validity.The small secondary validation is not sufficient to establish that acceptance guarantees validity.Source: §A.12 · p. 34

Related papers

Loading 2609.30571v1…