PaperlayerView exact revision on arXiv

Source-linked AI summary

Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding

Fan Zhang, Yankai Chen, Zhuohan Xie, Yixi Zhou, Sijia Peng, Lei Fan, Xinhua Ji, Cunyuan Zheng, Huangyong Shan, Philip S. Yu, Xue Liu, Yu Chen, Preslav Nakov, Songwei He

arXiv:2609.27678v1cs.CL

TL;DR

Contract review requires multiple judgments tied to one document, yet aggregate accuracy can hide changed decisions and repeated agreement can preserve errors. We compare Jev with nine language models on ContractNLI using controlled request conditions and measures of cost, time, and correctness. Jev has the lowest cost and median baseline response time, while hosted models achieve higher baseline accuracy.

  • Problem

    Contract review requires multiple judgments tied to one document, but aggregate accuracy can hide changed decisions and repeated agreement can preserve incorrect labels.

    Source: §1 · p. 1

  • Method

    The study compares Jev with nine language models on ContractNLI using fixed target judgments, controlled request conditions, and measures of cost, time, average correctness, and persistent correctness.

    Source: §1 · p. 2

  • Results

    Jev had the lowest cost and median baseline response time, while hosted models achieved higher baseline accuracy; baseline-accuracy rankings differed from all-twelve-correct rankings.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Reporting correctness, direction of change, and validity alongside expenditure clarifies trade-offs that aggregate accuracy or agreement alone can obscure.

    Source: §7 · p. 9

    The small stability panel makes rankings sensitive to a few targets, and its intervals and paired comparisons are descriptive and unadjusted for multiple testing.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Contract inference requires multiple judgments about a shared document, but aggregate accuracy can conceal changes in the individual decisions. Repeated agreement is also insufficient: a model may consistently return the wrong answer. In this paper, we compare Jev with nine language models on ContractNLI, evaluating inference cost, response time, average correctness, and correctness across repeated request conditions. Controlled comparisons vary hypothesis visibility, requested outputs, and output order while keeping the contract and target judgment fixed. Jev has the lowest cost and median response time among the evaluated configurations, while hosted language models achieve higher baseline accuracy. Rankings by baseline accuracy differ from rankings by correctness across every condition and repeat, although small differences in the latter do not establish a general stability advantage. Development diagnostics further reveal compensating corrections and regressions, as well as persistent errors. These findings motivate evaluating cost and response time alongside whether individual judgments remain correct as the request configuration changes. Code: https://github.com/ZF-Utokyo/Jev-Benchmark

1 Introduction

Contract review requires multiple judgments tied to one document, but aggregate accuracy can hide changed decisions and repeated agreement can preserve wrong answers. This study compares Jev and language models across cost, response time, correctness, and controlled request conditions.

Motivation

  • Contract review requires several judgments about one shared document, requested jointly or separately, with intended answers tied to the contract.Practical value therefore depends on answer cost, time, and correctness across request configurations, not only one configuration's accuracy.Source: §1 · p. 1
  • Corrections and regressions can cancel in aggregate accuracy, while repeated agreement can preserve an incorrect label.Figure 1 illustrates a fixed judgment changing when additional non-target hypotheses become visible.Source: §1 · p. 1, Figure 1 · p. 2

Research questions

  • The study asks how cost, response time, and accuracy trade off; whether higher accuracy predicts persistent correctness; and which decisions change across request configurations.The controlled comparisons vary visible hypotheses, requested outputs, and output order while preserving target judgments and gold labels.Source: §1 · p. 2

Study design

  • The evaluation retains invalid responses in accuracy denominators and estimates uncertainty at the contract level.The main test evaluation and stability subset are complemented by separate development diagnostics.Source: §1 · p. 2

Contributions

  • The paper combines paired cost–accuracy comparisons, average and persistent correctness rankings, and behavioral diagnostics of corrections, regressions, and stable errors.The diagnostics also examine request sensitivity alongside repetition under unchanged conditions.Source: §1 · p. 2

2 Task and Evaluation Design

The evaluation scores ContractNLI labels against existing annotations and compares average accuracy with correctness maintained across controlled request conditions. It separates visibility, output workload, and output order while keeping each anchor judgment fixed.

Task and scope

  • ContractNLI evaluation covers its classification labels against existing annotations; evidence-span extraction is outside the study's scope.Predictions may be labels or no valid prediction, and no new human labeling or model-based judging is used.Source: §2 · p. 3

Accuracy and validity

  • Baseline accuracy requests all 17 labels jointly, counts invalid responses as incorrect, and also reports Macro-F1 and valid-response coverage.A valid response must contain exactly the requested IDs and one allowed label for each.Source: §2 · p. 3

Controlled request conditions

  • The four stability conditions vary visible hypotheses, requested output workload, and output order while preserving the contract and anchor label.A–B changes visibility, B–C changes output workload and structure, and C–D changes requested output order and structure; each condition is repeated three times.Source: §2 · p. 3

Correctness across repeats

  • An anchor is consistently correct only if all 12 responses across four conditions and three repeats give the correct label; stable errors are valid, identical, and wrong.The remaining categories distinguish changing valid labels from outcomes containing an invalid response, while retaining every anchor in the denominator.Source: §2 · p. 3

Paired changes

  • Average and all-twelve correctness are compared on identical targets, but the mean alone cannot determine whether every response for a target succeeds.Paired valid-response changes are separated into correct-to-wrong, wrong-to-correct, and wrong-to-different-wrong; with complete coverage, accuracy change reflects corrections minus regressions.Source: §2 · p. 3

3 Data and Experimental Protocol

The protocol evaluates ten model configurations on official ContractNLI test judgments and a selected stability panel, with contract-level uncertainty estimates. It measures cost and client response time alongside baseline accuracy and correctness across repeated conditions.

3.1 Data and uncertainty

  • The official test comprises 123 contracts and 2,091 judgments, with a 30-contract stability panel of one anchor per contract; repeated requests are not independent contracts.Uncertainty uses 5,000 whole-contract bootstrap resamples, and three comparators were added after earlier results were inspected, so the model comparison is not wholly prospective.Source: §3.1 · pp. 3–4

3.2 Models and inference

  • Jev receives shared contract state and native Choice questions, while generative models receive aligned classification instructions; their provider-specific operating points do not equalize computation or reasoning length.The Qwen test configurations use enabled thinking, temperature 1, top-p 0.95, and a shared 32,768-token allowance; Jev's output-order intervention is not equivalent to autoregressive output position.Source: §3.2 · p. 4

3.3 Cost and timing

  • API costs use provider charges or token-price estimates, local costs value observed inference time, and response time measures end-to-end client elapsed time.The local $2.00-per-GPU-hour valuation is not a measured bill; elapsed time includes preparation, network transfer, service processing, and response validation.Source: §3.3 · pp. 4–5
  • Jev 1.13.0 occupies the lowest-cost end of the comparison, while Gemini 3.1 Pro Preview attains the highest baseline accuracy.Figure 2 summarizes 123 official-test contracts and 2,091 judgments; bubble area represents the fraction of 30 anchors answered correctly in all four conditions and three repeats.Source: Figure 2 · p. 5

4 Results on the Official Test Split

The official-test comparison finds Jev lowest in cost and median baseline response time, while hosted models have higher baseline accuracy. Across repeated conditions, average correctness and persistent correctness differ, and label changes can occur even when aggregate accuracy is unchanged.

4.1 RQ1: Cost, response time, and accuracy

  • 77.38% accuracy at $0.000228 per contract for Jev; Flash-Lite is 5.16 percentage points higher at approximately six times the cost.Flash-Lite’s paired 95% interval versus Jev is 3.49–6.98 points, while Luna’s is −0.05–3.11 points; higher observed scores do not establish a reliable improvement for every comparator.Source: §4.1 · pp. 5–6, Table 1 · p. 6
  • 1.24 seconds is Jev’s median baseline response time, versus 1.60 for Flash-Lite and 1.74 for Luna.Paired B–C median increases are 0.19 seconds for Jev, 0.49 for Flash-Lite, and 0.67 for Luna; speed rankings depend on the requested workload.Source: §4.1 · p. 6

4.2 RQ2: Correctness across repeated conditions

  • 24 of 30 anchors remain correct across all twelve responses for Sonnet, versus 23 for Jev; this panel does not establish a general stability advantage over Sonnet.On the same 30 anchors, Jev answers 286 of 360 responses correctly (79.44%) and Qwen9B answers 285 (79.17%), yet their all-twelve-correct counts are 23 and 18.Source: Table 2 · p. 7, §4.2 · p. 7
  • Five of Jev’s anchors have the same wrong label throughout, while Qwen9B has eight anchors with changing valid labels and two with invalid responses.Jev and Astra repeat the same label on 28 of 30 targets, but five of Jev’s and six of Astra’s consistent labels are wrong; Sonnet repeats labels on 25 targets, with one consistently wrong.Source: §4.2 · p. 7

4.3 RQ3: Changes hidden by average accuracy

  • Nine of 90 Qwen4B responses change between A and B despite identical accuracy: four regressions, four corrections, and one switch between wrong labels.Terra likewise has equal accuracy in B and C despite seven changes: three regressions, three corrections, and one wrong-label switch.Source: §4.3 · p. 7
  • 14 of 90 Haiku responses change from C to D, including twelve regressions and two corrections; its accuracy falls by 11.11 percentage points.Jev has no C–D changes in this panel, but this finite sample does not establish invariance.Source: §4.3 · pp. 7–8
  • Qwen9B changes 15 of 90 valid pairs between A and B, while repeats within A change 12 of 90 pairs, showing variation without a changed request.These are different dependent comparisons, so subtracting their counts would not estimate an intervention’s causal contribution.Source: §4.3 · p. 8, Figure 3 · p. 8

4.4 Can incomplete answers explain the differences?

  • Even optimistically correcting every missing baseline label leaves Qwen4B at most 79.10% and Qwen9B at most 79.96%, both below Flash-Lite’s 82.54%.Correcting missing responses could raise Qwen4B’s all-twelve-correct count only to at most 19, still below Jev’s 23; the difference is not attributable solely to truncated responses.Source: §4.4 · pp. 8–9

5 Development Diagnostics

Development diagnostics show that substantial label changes can coexist with small changes in correct counts, and that higher mean correctness need not yield a clear gain in persistent correctness.

5 Development Diagnostics

  • 87 of 510 Qwen4B labels change between joint and single-hypothesis requests, while its correct count changes by only four.In a disjoint anchor panel, A and B have equal accuracy despite 24 of 90 responses changing, evenly split between corrections and regressions.Source: §5 · p. 9
  • A paired Qwen configuration comparison finds higher mean correctness but no clear gain in all-twelve correctness despite much greater token use.Reasoning, sampling, and output allowance change together, so the diagnostics motivate separating visible hypotheses, requested outputs, and inference settings.Source: §5 · p. 9

6 Related Work

The related work situates the study across legal-task evaluation, request sensitivity, and joint quality–cost assessment. This paper uses controlled contract judgments to distinguish changes, persistent errors, and repeat variation alongside cost and correctness.

6 Related Work

  • ContractNLI evaluates contract inference alongside CUAD evidence extraction and LegalBench’s broader legal reasoning tasks.The study uses ContractNLI’s existing labels to examine request configuration effects on multiple judgments about one document.Source: §6 · p. 9
  • Prior prompt and behavioral studies examine order, formatting, batching, evidence location, and answer-option order; this study varies hypotheses, requested outputs, and output order while holding contract and target fixed.It distinguishes corrections, regressions, persistent errors, and variation across unchanged repeats.Source: §6 · p. 9
  • Joint quality, robustness, and efficiency evaluations and accuracy–cost routing studies provide context for comparing correctness with inference expenditure.Here, average correctness is related to whether fixed judgments remain correct across repeated conditions, with explicit costs and invalid-response accounting.Source: §6 · p. 9

7 Discussion and Conclusion

The discussion treats cost, response time, average accuracy, and persistent correctness as complementary measures. It concludes that decision-level reporting clarifies trade-offs that averages and agreement alone can obscure.

7 Discussion and Conclusion

  • Jev has the lowest cost and median baseline response time among evaluated configurations, while hosted language models have higher baseline accuracy.Jev is competitive on repeated-condition correctness, but the small anchor panel does not establish a general stability advantage; the cost frontier is specific to evaluated configurations and prices.Source: §7 · p. 9
  • Corrections and regressions can cancel in aggregate accuracy, while agreement can preserve errors; even optimistic completion of truncated answers leaves Qwen below Flash-Lite in baseline accuracy and below Jev in all-twelve correctness.Reporting correctness, direction of change, and validity alongside expenditure clarifies the observed trade-offs.Source: §7 · p. 9

Limitations

The evaluation’s scope, model configurations, and cost and latency measurements constrain how its comparisons should be interpreted. The samples and selection process also limit generalization to other legal questions, domains, or professional use.

Limitations

  • The 123-contract test and 30-target stability panel cover only 17 hypothesis types; there is no human comparison, new adjudication, or external-domain validation.An official held-out split also does not establish that the data were absent from model training.Source: Limitations · p. 10
  • Baseline accuracy and all-twelve correctness use different target sets, while three repeats and the small stability panel make rankings sensitive to a few targets.Intervals are descriptive and unadjusted for multiple testing; bootstrap intervals may degenerate when no changes are observed.Source: Limitations · p. 10
  • Model comparisons vary in inference settings, interfaces, hardware, providers, and collection times, and hosted models may change behind fixed identifiers.The Qwen development comparison changes reasoning, temperature, and generation allowance together, so it cannot isolate reasoning alone; common token limits also do not equalize realized computation.Source: Limitations · p. 10
  • Local GPU cost estimates exclude loading, idle allocation, and other non-inference expenditure, while latency includes infrastructure and network effects.The observed frontier may change with other prices, serving methods, or reasoning budgets.Source: Limitations · p. 10
  • Latency comparisons cannot isolate architecture-level efficiency, separate computation from network and queueing delays, or establish throughput under concurrent load.They also involve different collection times, providers, routes, and inference settings, with uncontrolled provider caching.Source: Limitations · p. 10
  • Sonnet, Haiku, and Terra were added after earlier results were inspected, so the ten-model comparison is not wholly prospective.Broader evidence is needed to assess suitability for legal practice or financial decision-making.Source: Limitations · p. 10

Ethical considerations

The study uses existing ContractNLI data and annotations, with no new personal-data collection or human participation. Supplementary official-test and development analyses examine class-specific errors, target-level response patterns, and controlled request comparisons, with development results kept separate from the test evaluation.

A Supplementary Official-Test Comparisons

  • Class-recall leaders differ: Jev leads on contradiction recall, Astra on entailment recall, and Sonnet on not-mentioned recall.These descriptive comparisons involve classes with unequal support and do not establish significant model differences.Source: A Supplementary Official-Test… · p. 14
  • 170 of 220 contradictions are correctly classified by Jev, versus 135 by Gemini Pro; Gemini Pro gains 47 entailments and 110 not-mentioned labels but loses 35 contradictions, netting 122.These counts do not establish an application-specific preference without an independently justified valuation of the errors.Source: A Supplementary Official-Test… · p. 14
  • 71 or 72 of 90 Qwen9B judgments are correct per condition, despite target-level changes across its twelve responses.Eight targets receive different valid labels, two encounter generation-budget exhaustion, two repeat the same wrong label, and eighteen are correct throughout.Source: A Supplementary Official-Test… · p. 14

B Development Diagnostics

  • On 30 development contracts, Qwen4B changes 87 of 510 labels between joint and single-hypothesis requests, but the number correct changes by only four.The changes include 38 regressions, 34 corrections, and 15 switches between wrong labels, showing how aggregate accuracy can understate affected decisions.Source: B Development Diagnostics · p. 14

C.1 Data and sampling

  • The development sample contains 510 targets across 30 contracts, with 280 entailments, 48 contradictions, and 182 not-mentioned labels; scores use existing annotations that may remain ambiguous.Documents were sampled without replacement, retaining the full contract text, 17 hypotheses, and original class annotations.Source: §C.1 · pp. 15–16

C.2 Models and inference settings

  • Jev receives each hypothesis as a native Choice question, while generative models receive classification instructions and return JSON label maps.Instructions are semantically aligned across model-specific interfaces, but reasoning settings and budgets vary; later added model controls use separate routes where checkpoints are not assumed identical.Source: §C.2 · p. 16

C.3 Experimental conditions

The development analyses compare cost and accuracy with changes in individual judgments, using repeated and controlled request conditions. They show that aggregate scores can mask decision changes and that the observed cost–accuracy frontier differs from the accuracy ranking.

C.3 Experimental conditions

  • The initial baseline compares seven models on 30 contracts, then evaluates grouping, permutation, and repetition under shared or expanded development conditions.Later model extensions reuse the development sample, and some comparisons use only five contracts or separate collections.Source: §C.3 · pp. 16–17
  • Latency is median document completion time, while costs use reported API charges or token rates and a rental-equivalent GPU estimate for local models.The local estimate uses $2 per GPU-hour and measured time; it excludes other costs and does not imply equal cloud throughput.Source: §C.3 · p. 18

D.1 Correctness and sensitivity are distinct

  • At the stated prices, Jev, Qwen4B, Flash-Lite, and Gemini Pro form the observed cost–accuracy frontier; Terra, Sonnet, and Haiku do not extend it.Jev records 77.06% accuracy; Gemini Pro has the highest observed accuracy at 82.94% and Macro-F1 at 77.19%.Source: §D.1 · pp. 18–19, Figure 4 · p. 19

D Development Grouping and Repetition Results

  • Qwen4B changes 87/510 labels from joint K = 17 to K = 1, although correct predictions shift only from 399 to 395.The changes include 38 regressions, 34 corrections, and 15 switches between wrong labels, illustrating how corrections and regressions can partly cancel in accuracy.Source: §D.1 · p. 19

D.2 Repetition and class effects matter

Request changes and unchanged repeats produce distinct patterns of judgment variation. Controlled comparisons further show that visible hypotheses can alter a target decision even when its requested answer and aggregate accuracy remain fixed.

D.2 Repetition and class effects matter

  • Unchanged-repeat variability differs by model: Qwen4B and Qwen9B show no repeat changes, while Flash-Lite and Luna show 16 and 14, respectively.Jev changes four labels on repetition, and extended controls find only 4/510 changes for GPT-6 (Convert) under both repetition and permutation.Source: §D.2 · pp. 19–20
  • For Qwen4B, K = 1 raises contradiction recall from 16/48 to 28/48 while entailment and not-mentioned recall fall; accuracy dips as Macro-F1 rises.Luna shows the complementary pattern: entailment recall falls while contradiction and not-mentioned recall rise, for a net gain of only one correct decision.Source: §D.2 · p. 20

D.3 Separating request factors

  • The pilot’s grouping comparison changes several request factors together, motivating follow-up contrasts that vary hypothesis visibility and requested output separately.The fixed K = 1 concurrency comparison reduces median completion time to 27.5–37.6% of its serial value across five core models.Source: §D.3 · pp. 20–21

E.1 Controlled comparison

  • The controlled follow-up samples 30 anchors from ten development contracts and compares four arms that hold the contract and target label fixed.Arms vary visible hypotheses, requested output workload, and output order; each anchor–arm pair is evaluated three times.Source: §E.1 · p. 21

E.2 Fixed accuracy, changed targets and class balance

  • With the requested target and answer set fixed, Qwen changes 24/90 labels when the visible catalog expands, while accuracy remains 69/90 in both arms.The changes split evenly between 12 correct-to-wrong and 12 wrong-to-correct labels; Macro-F1 falls from 71.90% to 55.14%.Source: §E.2 · pp. 22–23
  • Jev changes 15/90 labels between the anchor-only and full-catalog arms, while Flash-Lite changes 4/90, all regressions.The relative sensitivity differs from the pilot grouping comparison, indicating that observed patterns depend on which request feature changes.Source: §E.2 · p. 23

E.3 Output composition and stable errors

The anchor analyses distinguish correct persistence from mere answer consistency and show that request changes can alter decisions. Additional model comparisons and development diagnostics remain bounded by their sampled contracts and exploratory settings.

E.3 Output composition and stable errors

  • Jev and Flash-Lite each keep 21/30 anchors correct across all twelve responses, while their higher strict-stable counts include stable errors.Jev has three stable errors and Flash-Lite four, showing why consistency alone does not establish correctness.Source: §E.3 · pp. 23–24, Table 11 · p. 24

E.4 Exploratory Qwen9B comparison

  • Qwen9B changes 6/90, 24/90, and 21/90 predictions across A–B, B–C, and C–D, despite agreement across repeats within each arm.Sixteen of 30 anchors remain correct across all twelve observations, while 20 remain label-consistent, including four stable errors.Source: §E.4 · p. 24

E.5 Additional API comparators

  • Sonnet keeps 23/30 anchors correct throughout, Haiku 22/30, Terra 19/30, and Jev and Flash-Lite 21/30 each.Terra has slightly higher baseline accuracy than Sonnet but fewer anchors remain correct throughout; the different baseline and anchor targets do not isolate request sensitivity.Source: §E.5 · p. 24

F Development Sampling and Classification Prompts

H Comparison with Related Evaluation Settings

  • The paper distinguishes inference-time variability under fixed configurations from variability across training runs, and limits its ContractNLI evidence from generalizing to LexGLUE or MAUD.It motivates joint quality–cost reporting with request times and provider charges interpreted within the configurations that produced them.Source: H Comparison with Related Evaluation… · pp. 25–26

I.1 Partial development evaluation of Astra

  • The anchor study uses ten contracts, with Astra’s evaluation stopped at 293 valid responses and 67 requests unissued under a prespecified spending rule.Astra’s unequal coverage precludes direct comparison with complete-sample results; Qwen9B’s later anchor evaluation is exploratory and uses ten disjoint development contracts.Source: §I.1 · p. 27, J Supplementary Qwen9B Development… · p. 27

K Qwen Inference-Configuration Comparison

In the development comparison, higher average Qwen correctness need not increase the number of targets correct across all twelve responses, and reasoning-enabled configurations use substantially more completion tokens. The generation-budget and response-time analyses retain failed attempts and describe the evaluated configurations.

K Qwen Inference-Configuration Comparison

  • Qwen4B average anchor accuracy: 76.67%→81.11%, all-twelve-correct 17/30 unchanged; Qwen9B: 74.17%→82.50%, all-twelve-correct 16/30→18/30.These comparisons use the same 30 development anchors and are separate from the official test split.Source: K Qwen Inference-Configuration… · p. 28, Table 22 · p. 32
  • Completion-token use increases 68.30 times for Qwen4B and 63.51 times for Qwen9B under reasoning-enabled configurations.Source: K Qwen Inference-Configuration… · p. 28
  • Thinking, temperature, and token allowance changed together; all-twelve correctness change intervals include zero for Qwen4B and Qwen9B.The descriptive 95% intervals are −16.67 to 20.00 points for Qwen4B and −16.67 to 30.00 points for Qwen9B.Source: K Qwen Inference-Configuration… · pp. 28–29

L Generation-Budget Exhaustion

  • The 17 truncated reasoning-enabled responses exhausted a shared 32,768-token limit without final answers; this establishes noncompletion, not why it occurred or whether more budget helps.Source: L Generation-Budget Exhaustion · p. 29
  • Unfinished responses remain in accuracy, stability, and cost analyses; replacing them would condition on completion and alter the twelve-response correctness criterion.Source: L Generation-Budget Exhaustion · p. 30

M Response Time and Output Workload

  • With all 17 hypotheses visible, conditions B and C request one versus 17 labels; paired timing differences match contract and repeat.Source: M Response Time and Output Workload · p. 31
  • Figure 6 reports Jev's lowest baseline median latency among tested configurations and a small delay when more judgments are requested from the same visible catalog.Source: Figure 6 · p. 33
Loading 2609.27678v1…