Jev for Scientific Decisions: Evaluating Semantic Choices and Their Consequences

1 Introduction

• Whether observations share a culture, treatment, or reference standard determines what a program should count and which claims the evidence supports.The answer space may be finite even when choosing the appropriate relation requires interpreting scientific language.Source: §1 · p. 1
• Jev is designed for typed questions about supplied state, with its Choice interface selecting among explicit candidates and code handling numerical operations.The paper asks whether those semantic judgments remain correct at useful request cost and latency, and whether evaluation captures their downstream consequences.Source: §1 · p. 1
• The study contributes source-grounded decision materials, a shared harness following the documented code–model division, and a comparison of Jev with eleven other configurations.Prior evaluations address scientific question answering and claim verification; this paper examines executable relations whose selection controls a specified calculation.Source: §1 · p. 1

2.1 Cases and the shared harness

• The development collection includes twenty model-routed Choices in ten cases, plus twenty scientific Choices resolved by case-specific code rules and an engineering control.Models receive the same prepared evidence and structured records, while reference answers and assessed claims remain outside their inputs.Source: §2.1 · pp. 1–2
• The harness asks atomic questions grounded in supplied evidence, then uses shared deterministic code for case computations and claim assessments.The model selects relations from evidence and questions; the program applies the case rules to produce scientific outputs and a final label.Source: §2.1 · p. 2, §2.2 · p. 2

2.2 Three levels of correctness

• Semantic correctness checks selected relations, downstream exactness checks declared scientific outputs, and label correctness checks the canonical final label.Distinct relations can yield the same quantities or label, while executing a wrong relation exactly can produce a wrong scientific result; because all ten reference labels are inconsistent, an always-inconsistent baseline solves the label task.Source: §2.2 · p. 2

2.3 Experimental comparison

• Twelve configurations were compared on ten cases, each repeated five times, giving each configuration fifty planned requests and one hundred semantic answers.Missing responses receive no correctness credit, and latency and cost are measured among successful responses under each configuration’s service conditions.Source: §2.3 · p. 2

3.1 Decision quality

• Jev answered all one hundred planned semantic questions correctly and obtained all fifty downstream results, matching five other configurations; seven errors by three configurations concerned one culture-history question.Terra and DeepSeek configurations answered every received question correctly, but missing responses reduced their planned-denominator scores.Source: §3.1 · p. 2

3.2 Correct labels can conceal wrong quantities

• Across 588 received results, all final labels were correct while 581 downstream outputs were exact, exposing seven incorrect outputs on one scientific question despite expected verdicts.The seven wrong selections produced growth-history counts of (10, 1) four times, (1, 1) twice, and (10, 10) once; the final label remained inconsistent.Source: §3.2 · p. 3

3.3 Request latency and cost

• Jev had the lowest observed successful-request median latency, 0.335 s, and a p95 of 0.442 s.Its known mean cost was approximately $0.000060 per successful response; among configurations with complete semantic correctness and response coverage, Sol’s median was 1.385 s and cost was $0.003047 per successful response.Source: §3.3 · p. 3

4 Implications for Scientific Workflows

• Jev is a candidate component when evidence is prepared, candidate relations are explicit, and downstream operations are specified.Its observed request economy is described as useful at this interface between scientific text and executable rules.Source: §4 · p. 3
• A final verdict can conceal an error elsewhere, so fixed-choice evaluations should retain relation-level references and declared downstream outputs.Joint scoring checks whether the component supplies correct relations as well as the expected result.Source: §4 · pp. 3–4
• Table 1 covers twenty Choices in ten cases repeated five times, with known mean successful-response costs excluding preparation and unmeasured charges.The caption describes the comparison scope and cost accounting, not specific configuration outcomes.Source: Table 1 · p. 4

5 Conclusion

• The harness evaluated prepared scientific decisions using shared deterministic computation and a twelve-configuration comparison.The conclusion frames Jev as a candidate for this evaluated role.Source: §5 · p. 4
• Jev combined complete observed semantic correctness with the lowest recorded successful-request median latency and known mean response cost.The comparison used twelve configurations and shared deterministic computation on prepared scientific decisions.Source: §5 · p. 4
• Separating semantic selections, downstream outputs, and final labels revealed errors hidden by the final verdict.This yields a practical evaluation contract for scientific decision components.Source: §5 · p. 4

• Six configurations achieved complete correctness, but all observed semantic errors concerned one Choice in a 20-Choice, ten-case development collection.Source: Limitations · p. 4
• Because curated English packets recur across repetitions, transfer requires new source-separated cases and broader relation coverage to be evaluated.Source: Limitations · p. 4
• Every model-routed reference label is inconsistent, with no not_stated, other, or conflicting Choices, leaving label discrimination and abstention unassessed.Source: Limitations · p. 4
• Different relations can yield identical downstream outputs, so declared outputs do not replace semantic scoring.Source: Limitations · p. 4
• One implemented configuration per model leaves component effects unresolved; assessing them requires a separate ablation study.Source: Limitations · p. 4
• Latency and cost vary with services, execution windows, reasoning settings, and cache states; successful-request-only accounting omits missing responses, preparation, and local computation costs, leaving total workflow expenditure undetermined.Source: Limitations · p. 4

A Scientific questions and case construction

• The development collection has 40 scientific Choices in 20 groups, split between 20 deterministic-rule decisions and 20 model-routed decisions.It also contains 25 claim outcomes; 21 source records belong to 18 documented source families, with related documents from one experiment kept together.Source: A Scientific questions and case… · p. 6

A.1 Relations that determine scientific quantities

• Cases ask which observations share a unit, history, dependence, protocol, or evidence-supported scope, because these relations determine supported quantities.Examples include repeated scans of one wear event, shared culture growth, and unresolved exact parent-lot counts.Source: §A.1 · p. 6

A.2 Evidence and reference construction

• Each input supplies a target, explicit scope, selected evidence, and, when needed, structured records; reference choices, outputs, and constructed claim labels are withheld.Evidence and records were prepared from articles, supplements, project documentation, a methods handbook, and an industrial case report.Source: §A.2 · p. 6

E004: powder reuse histories

• E004 interprets build-8-and-later top-ups as mixed reuse histories with partial provenance, leaving assignments incomplete and the exact parent-lot count unknown.A descriptive history label alone does not identify an original lot.Source: §A.3 · p. 7

E005: staircase fatigue testing

• E005’s regular staircase starts at 325 MPa and steps 30 MPa, yielding 295 MPa after failure and 355 MPa after runout; fixed-stress replicate counts remain unknown.Each next test uses a new specimen, and stress selection depends on the preceding outcome.Source: §A.3 · p. 7

E006: paired-fixed cosmological simulations

• E006 pairs phase-reversed initial fields and uses pair averages; its N1000 subset has 100 paired realizations, 200 executions, and 100 ensemble values.These counts do not specify an effective sample size for every evolved observable.Source: §A.3 · p. 7

E007: climate-ensemble protocol groups

• E007’s two protocol groups each contain 50 members, but bundled changes, including code corrections, prevent isolating smoothing alone.The comparison includes original-forcing and SMBB groups.Source: §A.3 · p. 7

B001: administration and developmental exposure

• B001 treats F3 as beyond the defined exposure window, with 162 pups in 14 litters and no newly injected F3 animals; distinct F0 ancestors remain unknown.The exposure alternatives distinguish the full window from maternal-only exposure, mother-plus-embryo exposure, and exposure of later descendant generations.Source: §A.3 · p. 8

B002: cultures and mutation origins

• B002 distinguishes shared from separate growth histories: Experiments 11a and 11 have one and ten histories, while mutation-origin counts remain unknown.The selected columns contain 514 and 624 colonies; clonal descendants prevent these counts from establishing mutation origins.Source: §A.3 · p. 8

B003: pooling and biological attribution

• B003 identifies physical RNA pooling before library preparation; four pooled libraries represent 16 contributors and 32 appearances, without separately observed contributor profiles.The individual-library arm remains outside the selected pooled-arm claim.Source: §A.3 · pp. 8–9

B004: mosaic founders and transmission

• Mosaicism and a pre-existing allele limit attribution of alleles and transmission from founder counts and phenotype; 23 sequenced hybrid founders yielded 57 mutant alleles, while validated new germline-line count remains unknown.For founder 30, manuscript prose reports five alleles whereas publisher Figure 3C lists six entries.Source: §A.3 · p. 9

A.4 Deterministic derivation and claim composition

• Deterministic rules resolve ten groups from prepared evidence; model selections feed shared operations for arithmetic, filtering, deduplication, covariance substitution, and claim composition.Missing answers and invalid options remain response failures, while unknown scientific quantities remain explicitly unknown rather than becoming zero.Source: §A.4 · p. 9
• Unresolved contradictions take precedence over outside-candidate answers, which take precedence over missing evidence; these alternatives stop label composition.A substantive partial-identification answer can still produce an output with unknown fields.Source: §A.4 · p. 9
• For biological compound claims, one rejected clause makes the final label inconsistent even if another clause lacks sufficient evidence.The code-only condition leaves residual questions unresolved, while the reference-oracle condition checks shared derivations using reference Choices.Source: §A.4 · p. 9

B Metric Definitions and Output Coverage

• Semantic, downstream, and label indicators separately record exact Choice agreement, declared-field agreement, and canonical final-label agreement; missing responses score zero.The metrics use planned denominators.Source: B Metric Definitions and Output Coverage · p. 9
• Downstream exactness requires every declared field to match in both value and type, including explicitly unknown quantities; Table 4 lists assessed outputs.Omitted values differ from explicitly unknown values, and Boolean values differ from numbers.Source: B Metric Definitions and Output Coverage · p. 9, Table 4 · p. 10
• A rejected claim clause yields an inconsistent label; otherwise an unresolved clause yields insufficient evidence, and support for every clause yields consistent.All reference labels in this comparison are inconsistent.Source: B Metric Definitions and Output Coverage · p. 9
• Downstream and joint scores coincide empirically because each observed semantic error changes a growth-history count.Different selections can nevertheless yield the same declared output when a relation does not enter the recorded calculation.Source: B Metric Definitions and Output Coverage · p. 9

C Correctness Conditional on a Received Response

• Table 5 assesses only complete received responses; availability is reported separately, so selective response coverage can affect the conditional comparison.Absent answers are not treated as semantic mistakes or insufficient-evidence selections.Source: C Correctness Conditional on a… · p. 10
• All 588 received labels matched references, versus 581 exact downstream results and 1,169 correct Choices out of 1,176.The seven semantic errors each altered a growth-history count while preserving the final label.Source: C Correctness Conditional on a… · p. 10
• The seven errors on one scientific question changed growth-history counts but preserved final labels, separating correctness at the selection, output, and label levels.A label-only comparison would miss the incorrect quantities passed onward.Source: C Correctness Conditional on a… · p. 10