PaperlayerView exact revision on arXiv

Source-linked AI summary

Verdict: Encode Once, Decide Many

Avdhesh Singh Chouhan, Jack Perry

arXiv:2610.20778v1cs.CLcs.AIcs.LG

TL;DR

Organizational automation requires calibrated probabilities for many typed questions about one state, while per-question encoders repeatedly process that state. Verdict-MQ packs a case’s questions into one encoding with structured attention; it improves soft-label agreement on typed decisions, while standard-benchmark results and controlled comparisons reveal task-dependent trade-offs.

  • Problem

    Organizational automation needs calibrated probabilities for multiple typed decisions about one state, while per-question models encode the state separately for each question.

    Source: §1 · pp. 1–2

  • Method

    Verdict-MQ packs a case’s state and questions into one encoder pass, with structured attention preserving question-independent state encoding and option-order invariance.

    Source: §1 · pp. 1–2

  • Results

    On typed decisions, Verdict-MQ matches soft labels significantly better than Laya on every fresh seed; it beats the matched baseline on MultiRC but loses on RACE and BoolQ.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    Giving each question its own question-aware state copy costs 0.9–3.5 points on RACE and MultiRC at 1.4–2.5× latency, with no measurable cost on typed decisions.

    Source: §8 · p. 9

    BoolQ training remains unstable across seeds, and the authors say Verdict-MQ is not a reliable choice for BoolQ-like tasks until that instability is understood.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Automating organizational work means answering many small typed questions about one piece of state (does this email need a reply, how urgent is it, who owns it) with probabilities good enough to act on without a human. The open typed-decision model we compare against, like a standard cross-encoder, answers each question with a separate pass over the input. We present Verdict-MQ, an open encoder that reads a case once and answers all of its questions: given identical retained state tokens after truncation (verified on every reported case), a structured attention mask makes the state encoding independent of the questions, keeps questions from influencing each other, and makes option order irrelevant by construction. On the typed-decisions benchmark, a 395M Verdict-MQ matches the soft gold labels significantly better than the released 421M Laya specialist on all five fresh seeds (KL 0.080 vs 0.117, Brier 0.045 vs 0.061), an advantage that survives per-system temperature scaling; its higher accuracy (0.801 vs 0.767) lies past the benchmark's own saturation point, so we read that margin as closer agreement with the benchmark's labelling teacher. On RACE, BoolQ and MultiRC, against a matched fine-tuned baseline it wins on MultiRC and loses on RACE and on BoolQ, where its training is unstable; we measure zero-shot references ourselves, including a closed general system (Jev) that beats every fine-tuned encoder. A controlled comparison measures what the shared state costs: giving each question its own question-aware copy of the state gains 0.9-3.5 points on RACE and MultiRC at 1.4-2.5x the per-case latency, and nothing measurable on typed decisions, with the invariance guarantees unchanged.

1 Introduction

Verdict-MQ packs a case’s state and questions into one encoding while preserving question independence and option-order invariance. It improves soft-label agreement on typed decisions, while results on standard benchmarks and controlled comparisons show trade-offs across tasks.

1 Introduction

  • Organizational automation needs calibrated probability distributions for typed decisions, and must answer several questions about each state efficiently.The open Laya model and similar cross-encoders read the state once per question; the authors use “pass” to mean a full state encoding, noting Laya’s internals were not confirmed.Source: §1 · p. 1
  • Verdict-MQ uses one ModernBERT pass to encode a state and its questions; structured attention keeps state encoding question-independent, isolates questions, and makes option order irrelevant by construction.These properties are verified numerically given identical retained state tokens.Source: §1 · pp. 1–2
  • On typed decisions, the 395M Verdict-MQ matches soft labels significantly better than the 421M Laya specialist on every fresh seed, even after temperature scaling.Both systems’ accuracy is at or past the benchmark’s saturation point, so the accuracy margin mainly reflects agreement with its labelling teacher.Source: §1 · p. 2
  • Giving each question its own question-aware state copy costs 0.9–3.5 points on RACE and MultiRC at 1.4–2.5× latency, with no measurable cost on typed decisions.The comparison holds the invariance guarantees constant.Source: §1 · p. 2
  • Verdict-MQ beats the matched fine-tuned baseline on MultiRC but loses on RACE and BoolQ; the measured closed general system Jev beats every fine-tuned encoder.Local inference is one to two orders of magnitude faster per case than Jev’s API.Source: §1 · p. 2

2 Related Work

Verdict-MQ builds on work in typed decisions, set processing, and shared input encodings, while distinguishing its contribution as packing multiple independent questions about one state into one pass. The section also situates its measured references and training signals.

2 Related Work

  • Laya, Jev, and meraGPT Decider expose typed-question interfaces; the authors measured Laya and Jev themselves, while Decider’s figure is reported from the benchmark card.The benchmark distinguishes specialists from zero-shot generalists and warns that labels come from a small teacher model.Source: §2 · p. 2
  • Cross-encoders re-encode context per option; Set-LLM and UniMC use related option-isolation ideas, but neither packs multiple questions into one pass.Verdict-MQ’s stated contribution is packing independent questions and measuring the shared encoding’s costs and gains; direct typed-decision comparisons with Set-LLM or UniMC remain future work.Source: §2 · pp. 2–3
  • DuplexJev packs questions about one audio clip into a frozen causal LLM; Verdict-MQ applies question-independent state encoding and option-order invariance in a bidirectional encoder.The authors do not compare against four concurrent preprints, including DuplexJev.Source: §2 · p. 3
  • Verdict-MQ brings shared and precomputed encoding ideas to typed decisions, keeping state encoding question-independent while several questions share a pass.The section relates this design to late interaction, packed sequences, batch prompting, shared-prefix caching, and permutation-equivariant set processing [Lee et al., 2019].Source: §2 · p. 3
  • The evaluation uses ECE, Brier score, distance to soft gold, risk–coverage curves, temperature scaling, soft-label training, and an earth-mover loss for ordinal targets.The section cites prior work on calibration, selective prediction, soft labels, ordinal loss, fine-tuning instability, and intermediate-task training.Source: §2 · p. 3
  • Figure 1 shades row-token attention to column tokens and shows shared option positions that make question grouping and option order invisible.The mask implements rules 1–4.Source: Figure 1 · p. 3

3 Model

Verdict-MQ packs a state with typed questions and options into one bidirectional encoder pass, using structured attention to preserve question-independent state representations and separate question blocks. Option-marker readouts produce distributions over declared options.

3 Model

  • A case comprises one text or JSON state and K typed questions, each with instructions and described options; Verdict-MQ returns a distribution over each question’s declared options.The packed sequence places the state before question blocks and option markers.Source: §3 · p. 3
  • Structured attention makes state tokens attend only to state tokens, while each question block attends to the state and itself, preventing questions from influencing one another.Question tokens attend to every option; each option attends only to the state, its question, and its own option.Source: §3 · pp. 3–4
  • All options in a block share position ids, so permuting options permutes outputs without changing anything else.The sliding window is defined on position ids rather than sequence index.Source: §3 · p. 4
  • Reusing an existing special token as the option marker raises large-model typed-decision accuracy from 0.790 ± 0.007 to 0.801 ± 0.005 over five fresh seeds.All headline models use [CLS] as the marker.Source: §3 · p. 4
  • The final hidden state at each option marker, together with a question-type embedding, passes through an MLP to logits and a softmax distribution.Training uses cross-entropy on soft labels plus a squared earth-mover term for score questions; an optional position-free option mixer allows options to be compared while preserving permutation equivariance.Source: §3 · p. 4
  • In fp32, tested permutations, unpacked answers, and padding change output probabilities by at most 5.4 × 10^-7; the exact-arithmetic guarantee loosens to 3.9 × 10^-3 in bf16.On the trained checkpoint, permutation never changes the chosen option or crosses the autonomy threshold, while answering alone crosses it in 0.75% of cases.Source: §3 · p. 4
  • Allowing options to attend to one another collapses identical option markers to one representation, yielding uniform distributions and no learning; rule 3 prevents this failure.Verdict-CE is the per-question cross-encoder baseline, running one pass per question over the state, question, and options.Source: §3 · p. 4

4 Experimental Setup

The experiments use a small synthetic typed-decisions benchmark and standard reading-comprehension benchmarks, comparing Verdict-MQ with measured or card-reported references. Development-split choices, multiple seeds, and case-cluster bootstrapping structure the evaluation.

4 Experimental Setup

  • The typed-decisions benchmark contains four synthetic workflows, 1,200 training cases, and 400 test cases, with five questions per case and soft labels averaged from three teacher samples.Its scores measure agreement with the teacher; the benchmark describes about 0.75 as saturation.Source: §4 · p. 4
  • Reported metrics include 10-bin top-label ECE and other card metrics; the implementation reproduces the benchmark’s Uniform-row KL, TV, and Brier values exactly.Because the card omits ECE binning and prior construction, ECE and the prior are compared only among measured systems.Source: §4 · p. 5
  • Laya and Jev are measured by the authors, while several other references are taken from the benchmark card; Verdict-CE is their per-question cross-encoder.Jev is scored zero-shot because no specialist mode was available to the authors.Source: §4 · p. 5
  • Hyperparameters are selected on a development split held out by state hash, and comparisons with Laya use a 10,000-sample case-cluster bootstrap as the primary significance test.The last epoch is reported, but the research process as a whole was adaptive rather than test-blind; item-level McNemar is also reported and described as anti-conservative.Source: §4 · p. 5
  • The experiments use ModernBERT-base and large, their multi-task NLI initializations, and Laya’s encoder; runs use one RTX 4090 Laptop GPU.Full training hyperparameters are in Appendix E.Source: §4 · p. 5

5 Results on Typed Decisions

Verdict-MQ’s soft-label fit remains better than Laya’s after temperature matching, while its higher accuracy is interpreted as closer agreement with the benchmark’s teacher. Held-out confidence results show selective automation, but whole-case coverage is much lower than per-decision coverage.

Large model and training stability

  • 0.801 ± 0.005 accuracy over five fresh seeds puts the 395M model above the benchmark’s approximately 0.75 saturation point, so the margin is read as closer agreement with the teacher.Laya’s accuracy is 0.767; the paper cautions that accuracy margins above saturation reflect teacher agreement, not better decisions.Source: §5 · p. 5, Table 1 · p. 6
  • Two of six seeds stalled with 6% warmup; the selected 20% warmup reached 0.790 ± 0.007, and replacing the marker yielded the headline 0.801 ± 0.005.The 395M model’s 0.804 ± 0.003 at k=22 was statistically indistinguishable, so the authors retained shared k=0.Source: §5 · p. 5

Temperature-matched comparison

  • 0.074–0.077 vs 0.105 held-out KL: Verdict-MQ remains below Laya after soft-gold temperature matching.The Brier difference also remains significant on every seed.Source: Table 2 · p. 6, §5 · p. 6

The autonomy dial

  • At a 90% target, Verdict-MQ auto-handles 62–71% of decisions against Laya’s 56%, but requiring all five decisions to clear the threshold leaves only 19–27% of cases.The per-decision comparison has realized accuracy of 89.4–92.2% vs 86.5%; whole-case coverage for Laya is 2.5%.Source: §5 · p. 6

Latency

  • 2.7×–2.8× faster per case at batch size 8, packing five questions beats one-pass-per-question mode; at batch size 1, the gain is 1.3–1.8×.Both modes use the same weights.Source: §5 · p. 6

6 What the Architecture Contributes

A controlled k=0 versus k=22 comparison isolates the cost of sharing a question-independent state: accuracy falls on RACE and MultiRC, but invariance guarantees remain unchanged. The paper attributes the typed-decision distributional advantage more to NLI initialization than to architecture.

Held-out autonomy evaluation

  • The held-out autonomy table selects each target-accuracy threshold on one test-case half and evaluates it on the other, reporting coverage and realized accuracy per cell.The split keeps a case’s five decisions together.Source: Table 3 · p. 7

Latency comparison

  • The latency table compares all five questions packed against one pass per question, using the same weights on typed-decision test cases.Measurements use bf16 on an RTX 4090 Laptop GPU.Source: Table 4 · p. 7

Controlled architecture comparison

  • 0.9–3.5 points of accuracy are lost on RACE and MultiRC when sharing one question-independent state instead of giving each question its own question-aware copy; invariance guarantees are identical.The controlled comparison holds initialization, loss, objective, epochs, and learning rate fixed, differing only in whether the state can read its question.Source: §6 · pp. 6–7

Initialization and distributional gains

  • Raw-ModernBERT-initialized models have worse typed-decision KL than Laya (0.191 at 150M; 0.153 at 395M vs 0.117), making NLI initialization the more consistent explanation for the distributional advantage.The passage says this advantage is most consistent with NLI initialization, not architecture.Source: §6 · p. 7

7 Standard Benchmarks

On the standard benchmarks, Verdict-MQ beats the matched fine-tuned baseline on MultiRC but loses on RACE and BoolQ; the controlled comparison measures the accuracy and latency cost of sharing the state.

7 Standard Benchmarks

  • Both systems are fine-tuned on each benchmark’s training split; raw ModernBERT is the primary initialization for BoolQ and MultiRC because NLI initialization used those datasets.Laya’s general checkpoint and Jev are evaluated as zero-shot references on the same test items and case grouping.Source: §7 · p. 7
  • Verdict-MQ gains +9 points over the baseline’s mean on MultiRC (F1a 77.5 vs 67.8), trails RACE by 1.5 points, and loses on BoolQ across five seeds.The zero-shot Jev reference outperforms every fine-tuned encoder, while Laya’s general checkpoint is a zero-shot reference rather than a fine-tuned comparison.Source: §7 · pp. 7–8, Table 5 · p. 8
  • At k=22, question-aware state copies score 77.4±0.4% on RACE versus 76.2±0.2% at k=0 (+1.0–1.5 points), and 80.6±0.5% on MultiRC versus 78.4±0.8% (+0.9–3.5).The gain costs 1.4× per-case latency on RACE and 2.5× on MultiRC at batch size 1; invariance guarantees are identical in both configurations.Source: §7 · p. 8
  • BoolQ training remains unstable: five test seeds range from 72.3 to 79.4%, and a single run’s accuracy is uncertain by several points.Roughly half of runs converged and half stalled near the 62% majority-class rate.Source: §7 · p. 8
  • Verdict-MQ ties standard classifiers on CLINC150 and Banking77 intent classification; its Enron, synthetic-corpus, and evaluation-pitfall studies are supplementary.The paper says these supplementary studies do not bear on the central shared-state tradeoff.Source: §7 · p. 8

8 Conclusion

Verdict-MQ encodes each case once for its typed questions, with structural guarantees on question independence and option order. Its results vary across benchmark settings, and the controlled comparison quantifies the cost of question-aware state copies.

8 Conclusion

  • Verdict-MQ reads a case once for all typed questions, with option-order and question-grouping invariance given identical retained state tokens.On typed decisions, it matches soft labels better than Laya’s specialist on every seed, robust to temperature scaling; accuracy is at or past saturation.Source: §8 · p. 8
  • On standard benchmarks, Verdict-MQ beats the matched fine-tuned baseline on MultiRC but loses on RACE and BoolQ; a zero-shot closed system beats every fine-tuned encoder.The zero-shot system is one to two orders of magnitude slower.Source: §8 · p. 9
  • Sharing the state costs 0.9–3.5 points on RACE and MultiRC for 1.4–2.5× latency, with nothing measurable on typed decisions and invariance guarantees unchanged.Source: §8 · p. 9

Limitations

The paper qualifies its findings by the limits of the typed-decision benchmark, unstable BoolQ training, non-like-for-like comparisons, adaptive research processes, uncontrolled training differences, and measurement scope.

Limitations

  • Typed-decisions has 400 synthetic test cases, labels from a small teacher, and no peer review, so high scores partly measure agreement with that teacher.The headline results are specialist results; the trained generalists fell below the input-ignoring prior.Source: Limitations · p. 9
  • BoolQ remains unstable across seeds, and Verdict-MQ stays 1.5 points below the RACE baseline even when each question has its own state copy.The RACE gap’s cause is open, while the instability is not specific to shared-state training; the authors say the model is not yet reliable for BoolQ-like tasks.Source: Limitations · p. 9
  • RACE, BoolQ, and MultiRC comparisons use zero-shot Laya and Jev references; no fine-tuned Laya comparison was run, and Jev outperforms every model but is likely much larger.The MultiRC baseline varies widely across seeds, without a stability search comparable to Verdict-MQ’s BoolQ search.Source: Limitations · p. 9
  • Test results informed parts of the adaptive research process, and reported p-values are not adjusted for the seeds, benchmarks, or configurations tried.Test examples did not enter training; one k=0 standard-benchmark seed reuses a previously reported seed.Source: Limitations · p. 9
  • Warmup length is confounded with a lower realized peak learning rate, and packed case-averaged loss gives questions in larger cases less gradient than the baseline.The authors did not test per-decision-weighted loss; at matched initialization, Verdict-MQ ties Laya’s architecture on accuracy.Source: Limitations · pp. 9–10
  • Jev uses an unpinned vendor alias, local timings come from one laptop GPU, email labels come from LLM teachers, and live organizational-corpus validation is not reported.Per-decision autonomy coverage overstates whole-case automation; fully automated cases can fall below the target accuracy.Source: Limitations · p. 10

Ethical Considerations

The paper reports that email labels are teacher-derived and describes intended routing and human-review safeguards, alongside disclosure of AI assistance in the research process.

Ethical Considerations

  • Organizational-email labels use public Enron messages and LLM teachers, not human annotators; labels are released by message hash rather than message body.Teacher agreement and limitations are reported in the appendix.Source: Ethical Considerations · p. 10
  • Verdict-MQ is intended to route organizational decisions to automation or a human, with deployment thresholds validated on local data and human review retained for consequential decisions.Per-decision coverage overstates whole-case automation.Source: Ethical Considerations · p. 10
  • AI assistance supported experiment execution, debugging, statistical analysis, and drafting under human direction and review; authors retain responsibility for design choices and claims.Source: Ethical Considerations · p. 10

A Controlled-attribution methodology detail

The attribution checks do not establish an accuracy advantage from Verdict-MQ’s architecture over Laya’s. Component tests find few accuracy gains beyond seed noise, and the schedule effect on raw ModernBERT does not carry over to NLI initialization.

A Controlled-attribution methodology detail

  • Matched-initialization runs tie within about a point, but different training recipes prevent attributing the small gap to architecture alone.Laya’s architecture averages 0.771 and Verdict-MQ 0.763 over two seeds each.Source: A Controlled-attribution methodology… · p. 12, Table 6 · p. 12
  • The option mixer’s development gain did not carry to test: 0.764 ± 0.005 with it versus 0.766 ± 0.008 without it.The large model’s mixer was neutral on development and was not carried to test; on typed decisions the mixer is neutral.Source: A Controlled-attribution methodology… · p. 12
  • Most component removals stayed within seed noise; option-subset augmentation may add 1.5 points, while shared option positions remain for invariance, not accuracy.The question-type embedding, ordinal loss, and long context window did not move accuracy beyond seed noise.Source: A Controlled-attribution methodology… · p. 12, Table 7 · p. 13
  • With raw ModernBERT, a constant 2 × 10−5 learning rate scored 0.733 versus 0.681 for warmup and decay, but this effect did not carry to NLI initialization.After rechecking on development data, the headline recipes remained unchanged.Source: A Controlled-attribution methodology… · pp. 12–13

B Calibration and autonomy methodology detail

This appendix supplies detailed calibration, autonomy, benchmark-method, and supplementary-experiment checks. It qualifies the headline comparisons with temperature-fit results, case-level automation outcomes, measurement details, and implementation conditions.

B Calibration and autonomy methodology detail

  • After soft-gold temperature fitting, Verdict-MQ’s held-out KL was 0.074–0.077 versus Laya’s 0.105, Brier was 0.041–0.042 versus 0.056, and TV was 0.128–0.131 versus 0.158.Every difference was at the bootstrap resolution floor; hard-label temperatures must not be reused for these soft-gold metrics.Source: B Calibration and autonomy methodology… · p. 13
  • At the 90% per-decision target, only 19–27% of Verdict-MQ cases passed all five thresholds, and their realized case accuracy was 69.8–81.6%.Laya had 2.5% of cases qualify; individually meeting a 90% target does not yield 90% joint case accuracy.Source: B Calibration and autonomy methodology… · p. 14

C Standard-benchmark methodology detail

  • Jev’s transient-error exclusions changed no reported ranking or gap on the common item subset, while local inference was one to two orders of magnitude faster per case.The authors did not isolate Jev inference time; the latency comparison includes gateway overhead and retry backoff.Source: C Standard-benchmark methodology detail · p. 14
  • The k=0 comparison was one of four complete eligible candidates; its test results reused seed 0 and added only two previously untrained seeds.Against k=22, k=0 did worse on RACE but had more uneven MultiRC gains over the matched baseline.Source: C Standard-benchmark methodology detail · p. 15

D Full typed-decisions search history

  • The full typed-decision search tables extend the headline configurations, and paired comparisons use case-cluster resampling rather than treating five decisions per case as independent.The search history includes every configuration’s paired test against Laya on the same 2,000 decisions.Source: D Full typed-decisions search history · p. 15, Table 9 · p. 17

E Hyper-parameters

  • No reported test state was truncated, but pack() could return oversized question-option sequences unless callers chunked first; the implementation added a check.The unchunked MultiRC example reached 6,378 tokens against a 2,048-token limit; reported paths chunked cases first.Source: E Hyper-parameters · p. 15

G Split-depth Verdict-MQ

  • The split-depth sweep found a +1.5-point RACE gain from k=0 to k=22 without closing the matched-baseline gap; MultiRC already beat baseline at k=0.Depth and marker interacted, and results at k > 0 were not stable across seeds.Source: G Split-depth Verdict-MQ · pp. 15–16

I.1 Organizational email decisions

  • On Enron email decisions, the strong teachers agreed 86.0% on needs-reply and 60.9% on priority; escalation agreement was 92.5%, though κ was only 0.16 because both marked almost no email as an escalation.The evaluation set consists of validation records on which the two teachers agree.Source: §I.1 · p. 16

I.2 Negative result: synthetic-corpus generalists

The synthetic multi-workflow corpus did not produce a zero-shot typed-decision generalist, and the paper identifies evaluation pitfalls and reproducibility boundaries relevant to interpreting its comparisons.

I.2 Negative result: synthetic-corpus generalists

  • Despite 9,277 synthetic decisions, Verdict-MQ scored 0.25–0.34 zero-shot on typed-decisions, below the input-ignoring prior, and synthetic-workflow performance was 42–53%.The authors conclude a working generalist would require substantially more workflow diversity and realistic case-distribution priors than this corpus provided.Source: §I.2 · p. 18, Table 14 · p. 18

I.3 Evaluation pitfalls

  • Evaluation pitfalls include removing option descriptions, selecting on test data, fitting thresholds in-sample, truncating options, and fixed-order position shortcuts.The passage also warns that uniform outputs can pass invariance tests and that mixed specialist/generalist modes or unsourced baselines distort comparisons.Source: §I.3 · pp. 18–19

J Licences and contamination

  • The NLI models list BoolQ and MultiRC among their training tasks, so the primary runs on those benchmarks start from raw ModernBERT.Enron-derived labels are released keyed to message hashes rather than message bodies.Source: J Licences and contamination · p. 19

K Reproducibility

  • Released scripts generate results tables and document run commands, while per-decision predictions enable recomputation of paired comparisons.A separate script checks the architecture guarantees.Source: K Reproducibility · p. 19

L Jev measurement record

  • Jev’s results depend on a live vendor-operated service, so later attempts may differ after an unobserved service update.The paper’s measurements were made 2026-09-25 through 2026-09-27.Source: L Jev measurement record · p. 19

Related papers

Loading 2610.20778v1…