PaperlayerView exact revision on arXiv

Source-linked AI summary

When Scientific Contradictions Are Lost in Translation

Tal Zeevi, Trey W. Jensen, Maxwell Strome

arXiv:2609.38621v1cs.AIcs.CL

TL;DR

Determining whether scientific findings contradict each other requires deciding whether they describe comparable observations. The paper tests this decision with controlled XOR problems translated into scientific reports. Claude Opus 5’s answers shift with biological framing and paired-design guidance, while GPT-5.6 Sol is less sensitive.

  • Problem

    Whether two scientific findings conflict depends on whether they describe comparable measurements, a decision not isolated by existing scientific-verification benchmarks.

    Source: §1 · pp. 1–2

  • Method

    The authors hold a controlled XOR constraint system fixed while varying its biological framing and the information available for recognizing paired findings.

    Source: §1 · p. 2, §1.1 · p. 2

  • Results

    Claude Opus 5’s 23/24 recovery rises from 27% to 79% with Biology-neutral framing and from 44% to 92% with paired-design guidance; GPT-5.6 Sol’s corresponding changes are not significant.

    Source: §2 · p. 4

  • Takeaways & Limitations

    Reliable scientific verification may need to expose which findings were connected and what relations they implied, in addition to reporting a final judgment.

    Source: §3 · p. 4

    Because only final answers are observed, the study cannot determine exactly how either model translates reports into formal relations.

    Source: §3 · p. 4

Abstract

from arXiv · show

Two scientific findings can disagree without contradicting each other. Determining whether they conflict requires knowing whether they describe comparable measurements. We study how language models behave at this decision point. In a controlled task, we generate an unsatisfiable XOR constraint system and translate its constraints into scientific reports from different laboratories. One assignment satisfies more constraints, while another satisfies fewer but better matches expected biology. This creates a simple dilemma: does the model choose the assignment that best fits the constraints, or the one that better matches biological expectations? When the constraints are stated directly, GPT-5.6 Sol and Claude Opus 5 recover the best-supported assignment in 90% and 96% of cases, respectively. In scientific prose, however, the models behave differently. Claude Opus 5 often prefers the biologically expected assignment. Removing that biological preference increases recovery of the better-supported assignment from 27% to 79% (p<.001); recovery reaches 92% when the same Biology-favored record is accompanied by a formalization request and an explicit paired-design cue (p<.001). GPT-5.6 Sol is less sensitive, with neither corresponding change reaching statistical significance. These results suggest that reliable scientific verification depends not only on formal reasoning, but also on how models decide which findings should be compared and what relations they imply.

1 Resolving a contradiction takes two steps

Scientific contradiction checking requires deciding which observations are comparable before reasoning over their agreements and disagreements. The paper isolates this earlier decision with a controlled task that keeps formal constraints fixed while changing scientific framing and structural cues.

1 Resolving a contradiction takes two steps

  • Scientific review must first establish which observations are comparable, then determine which conclusion best fits their agreements and disagreements.Existing scientific-verification benchmarks test evidence retrieval or evaluation but do not isolate which experimental findings can be compared.Source: §1 · p. 1

1.1 The task

  • The task models laboratories’ records as unchanged or flipped and expresses same-or-different experimental directions as XOR constraints.Because the constraints can be inconsistent, the goal is to find the assignment that satisfies the most constraints.Source: §1.1 · p. 2

1.2 Creating a controlled dilemma

  • Each case has one assignment satisfying 23/24 constraints and a close alternative satisfying 22/24, with all others satisfying at most 21/24.This creates a controlled choice between assignments whose formal support differs by one constraint.Source: §1.2 · p. 2

1.3 Two controlled manipulations

  • The scientific-framing versions preserve the XOR constraints while changing whether expected biology favors the 22/24 alternative.Selected paired reports are reversed together, changing their biological interpretation without changing the implied relation.Source: §1.3 · p. 2, Figure 1 · p. 3

1.4 Evaluation setup

  • Evaluation uses five conditions, with the direct-constraint Formal condition as a reference and the remaining conditions implementing the two controlled comparisons.The study covers 48 cases across three graph structures and two prose formats; recovery requires exact agreement with all 12 labels of the 23/24 assignment.Source: §1.3 · pp. 2–3, §1.4 · p. 3

2 Results

The formal problem is largely recoverable, but scientific framing and paired-design guidance affect Claude Opus 5 much more than GPT-5.6 Sol. Additional tested variants show a similar framing difference, while paired-design effects vary more across models.

Formal baseline

  • 43/48 (90%) for GPT-5.6 Sol and 46/48 (96%) for Claude Opus 5 recover the best-supported 23/24 assignment when constraints are stated directly.The Formal condition serves as the direct-constraint reference.Source: §2 · p. 3, Figure 2 · p. 4

Scientific framing

  • Claude Opus 5’s 23/24 recovery rises from 13/48 (27%) to 38/48 (79%) with Biology-neutral framing; GPT-5.6 Sol’s change is not significant.For Claude, selection of the biologically favored 22/24 alternative falls from 34/48 (71%) to 1/48 (2%).Source: §2 · p. 4, Figure 2 · p. 4

Structural guidance

  • Claude Opus 5’s recovery rises from 21/48 (44%) to 44/48 (92%) with paired-design guidance, while GPT-5.6 Sol changes from 42/48 (88%) to 41/48 (85%).Claude’s selection of the 22/24 alternative falls from 27/48 (56%) to 4/48 (8%).Source: §2 · p. 4, Figure 2 · p. 4

Additional model variants

  • Across four additional variants, scientific-framing effects remain larger for tested Claude variants than tested GPT variants; paired-design effects are larger on average for Claude but vary more.Source: §2 · p. 4

3 Conclusion

The study separates deciding what relations scientific evidence implies from reasoning over those relations. Its conclusion is that verification may need to expose how observations became comparable, not only the final judgment.

3 Conclusion

  • The formal problem is largely within reach, yet the same constraints can produce different answers in scientific prose, especially for Claude Opus 5.The authors cannot determine exactly how either model translates reports into formal relations and do not equate greater constraint adherence with greater scientific correctness.Source: §3 · p. 4
  • Reliable verification may need to show which observations were connected, what relations they implied, and what was ultimately verified.Source: §3 · p. 4

A Prompt examples

  • The prompt examples encode archive orientations as NORMAL or SWAPPED, with XOR values representing shared or differing orientations.The example lists three relations and notes that the full prompt contains 19 additional relations.Source: A Prompt examples · p. 6

Scientific-framing manipulation

  • Biology-favored and Biology-neutral records share the same archive structure and XOR system, but differ in selected report directions.Reversing paired reports removes the systematic biological preference between the leading assignments while preserving their formal relations.Source: Scientific-framing manipulation · p. 6

Structural guidance manipulation

  • The paired-design condition adds that each experiment was independently performed by two laboratories, without stating the pairs, constraints, or answer.The comparison retains the Biology-favored record and formalization request.Source: Structural guidance manipulation · p. 7

B Task construction and validation

Cases are generated to produce a fixed, unsatisfiable XOR problem with two leading assignments, then rendered as paired-laboratory reports.

Case generation

Case validation

Scientific-report construction

  • Each of 22 edges becomes an experiment reported by its two endpoint laboratories, yielding 44 reports; reversing selected pairs preserves the XOR relations and leading assignments.Source: B Task construction and validation · p. 7

Scientific prose generation

The authors built a fixed catalog of synthetic biological protocols and used it to generate paired laboratory reports under two prose formats. Case reports were generated deterministically from finalized templates and checked before evaluation.

Scientific prose generation

  • The 22-protocol catalog used ordinary directional findings with matched experiments but nonparallel wording, and its conventional directions were treated as synthetic rather than validated biological priors.The catalog was developed interactively with GPT-5.6 Sol using Codex CLI.Source: Scientific prose generation · p. 7
  • A seeded permutation assigned protocols to graph edges, and endpoint laboratories received separate descriptions whose directions were determined by archive orientations and framing conditions.The reports were assembled in either Consortium report or Recovery chronicle format.Source: Scientific prose generation · pp. 7–8
  • Draft language was revised to remove transparent logical cues and ensure paired reports described the same experiment; finalized cases were then generated without further manual or model revision.Cases were retained only after prespecified automated checks.Source: Scientific prose generation · p. 8
  • Catalogs, cases, reports, hidden pair maps, prompts, validation outputs, and hashes were frozen before evaluation, which used fresh, tool-free sessions.GPT-5.6 Sol participated in protocol development but was evaluated later using only the frozen prompts.Source: Scientific prose generation · p. 8

Biological-fit validation

The framing manipulation changed which leading assignment better matched the synthetic biological prior while keeping formal support unchanged. A toy example illustrates the constraint-scoring task, while the full experiment creates a unique winner and runner-up.

Biological-fit validation

  • In Biology-favored, the 22/24 assignment matched the synthetic biological expectation for 15–21 more reports than the 23/24 assignment; in Biology-neutral, the difference was at most one.Biological fit counts reconstructed reports matching each protocol's model-proposed conventional direction, using the hidden pair map.Source: Biological-fit validation · p. 8
  • Table 2 presents pre-call construction and manipulation checks for the biological-fit validation.The checks establish that framing shifted biological fit without changing formal support.Source: Table 2 · p. 8, Biological-fit validation · p. 8

C Toy example: exhaustive check

  • The three-laboratory toy example illustrates maximizing satisfied XOR constraints, but its assignments tie; the full task adds known orientations and relations to yield a unique best-supported assignment and runner-up.Table 3 lists all eight toy assignments, with 0 marking unchanged records and 1 marking flipped records.Source: C Toy example: exhaustive check · p. 8, Table 3 · p. 8

D Case families and framing robustness

The robustness analysis balanced cases across graph topology and prose organization while holding the laboratory and comparison structure fixed. Subgroup patterns differed by model and are treated as descriptive.

D Case families and framing robustness

  • Across the three graph topologies and two prose families, the 48 cases included eight cases per topology–prose combination.Both prose families used 12 laboratories, 22 pairwise comparisons, and two known archive orientations.Source: D Case families and framing robustness · p. 9
  • The coordinated-report format combines findings into a multi-lab account, whereas the recovery chronicle presents archived experiment notes sequentially without identifying repeated experiments.Figure 3 schematically depicts graph-family structures and the two narrative organizations.Source: D Case families and framing robustness · p. 9, Figure 3 · p. 9
  • Table 4 reports exact 23/24 recovery by graph topology and prose family; Claude favored Biology-neutral across prespecified subgroups, while GPT's framing effect varied by topology and was modest within prose families.The subgroup patterns are descriptive, not separate inferential tests, because each subgroup contains relatively few cases.Source: Table 4 · p. 9, D Case families and framing robustness · p. 9

E Full results for primary models

The full-results tables cover exact outcomes for both primary models across conditions and paired comparisons of exact 23/24 recovery.

E Full results for primary models

  • Table 5 reports exact outcome counts for both primary models in every condition, with 48 scheduled calls per condition.“Other” groups different assignments, invalid outputs, and timeouts.Source: Table 5 · p. 10
  • Table 5 reports exact outcome counts for the two primary models in every condition, with 48 scheduled calls per condition.“Other” includes any different assignment, invalid output, or timeout.Source: Table 5 · p. 10
  • Table 6 compares exact 23/24 recovery between conditions across 48 matched cases, using case-bootstrap 95% intervals and exact two-sided McNemar tests.Changes are calculated as the second condition minus the first.Source: Table 6 · p. 10

F Additional model replications

Four additional variants extend the evaluation to three tested models per family, with each variant evaluated on the same 48 cases across five conditions.

GPT-5.6 Terra

  • 73% of GPT-5.6 Terra’s Formal cases recovered the 23/24 assignment; recovery rose from 27% in Biology-favored to 60% in Biology-neutral and from 50% under Formalize only to 60% with paired design.Because its formal baseline was below 75% and lower than GPT-5.6 Sol’s, the authors interpret these manipulation effects descriptively.Source: F Additional model replications · p. 10

GPT-5.5

  • GPT-5.5 recovered the target in 73% of Formal cases; recovery rose from 13% in Biology-favored to 31% in Biology-neutral and from 31% under generic formalization to 48% with paired design.The authors caution that it did not pass the formal-baseline gate, so the changes do not establish robust preservation of the constraint system in prose.Source: F Additional model replications · p. 10

Claude Opus 4.8

  • Claude Opus 4.8 recovered the 23/24 assignment in 100% of Formal cases; recovery rose from 31% in Biology-favored to 79% in Biology-neutral and from 56% to 69% with paired design.The paired-design comparison is Formalize only versus Formalize + paired design.Source: F Additional model replications · p. 10

Claude Sonnet 5

  • Claude Sonnet 5 recovered the target in 92% of Formal cases; Biology-neutral recovery was 69% versus 0% in Biology-favored, while paired-design recovery was 63% versus 8% under generic formalization.In Biology-favored, 88% selected the exact 22/24 alternative; under generic formalization, 77% selected it, and both paired changes were positive across every prespecified graph topology and prose family.Source: F Additional model replications · p. 12

Interpretation

  • Across the three tested variants per family, Claude variants showed larger scientific-framing effects than GPT variants, while their paired-design effect was larger on average but varied substantially.The authors treat these as descriptive patterns for selected models, not broader family estimates; family is confounded with training, inference procedure, and capability.Source: Figure 4 · p. 11, F Additional model replications · p. 12

G Efficiency analysis

For the two primary models, adding a paired-design sentence reduces median reasoning tokens, latency, and cost, but the comparison is descriptive and accuracy differs between conditions.

Efficiency comparison

  • Adding the paired-design sentence reduces median reasoning tokens, latency, and cost for both primary models.Figure 5 compares Formalize only with Formalize + paired design; GPT cost is API-equivalent, while Claude cost is provider-reported.Source: Figure 5 · p. 12, G Efficiency analysis · p. 12

Figure 5

  • Figure 5 uses arrows for the condition change and reports medians for tokens and latency.Reasoning-token scales are model-specific and should not be compared across panels.Source: Figure 5 · p. 12

Interpretation

  • These descriptive efficiency comparisons do not establish how either model used the added sentence, and accuracy differs between conditions, especially for Claude.Source: G Efficiency analysis · p. 12

H Limitations and scope

The evaluation uses synthetic scientific records and a deliberately restricted formal task, limiting how directly its findings extend to real scientific literature.

Scope

  • The scientific records are synthetic and do not capture the full ambiguity of real scientific literature; expected biological directions are model-proposed priors, not independently validated facts.Source: H Limitations and scope · p. 12
  • The formal problem is limited to binary XOR with one intentionally inconsistent relation, and the primary evaluation covers two models, 48 cases, and one response per model-condition-case combination.Appendix F adds four variants, for three tested variants per model family.Source: H Limitations and scope · p. 12

Measurement limitation

  • Measurements cover final answers only, not inferred pairings, constructed constraint systems, or internal representations; therefore, the representation responsible for sensitivity cannot be identified.Source: H Limitations and scope · p. 12

Related papers

Loading 2609.38621v1…