PaperlayerView exact revision on arXiv

Source-linked AI summary

PACT: Pairwise-Anchored Calibrated Tuning for Single-Token Typed Decisions

Yida Lin

arXiv:2609.35865v1cs.CLcs.AIcs.CVcs.GT

TL;DR

Single-token typed decisions efficiently return probabilities over schema-defined answers, but cross-entropy ignores the pairwise and certificate structure in their curated training data. PACT turns that existing structure into training terms and contextual calibration. It matches the published recipe in accuracy while improving position bias, ordinal error, and stability relative to the cross-entropy-only control.

  • Problem

    The released recipe trains on contrastive examples independently, leaving pair, certificate, and answer-code structure unused by its cross-entropy objective.

    Source: §I · p. 1

  • Method

    PACT derives four training terms from certified contrastive pairs and fits a three-parameter contextual temperature using training-side calibration families.

    Source: B. Contrastive curation with… · p. 2, G. Objective, batching and cost · p. 4

  • Results

    PACT matches the published recipe in accuracy while reducing position bias and ordinal error, and it is more stable across seeds than the cross-entropy-only control.

    Source: §VII · p. 8, §IX · p. 10

  • Takeaways & Limitations

    The supported benefits are robustness to answer-code order, better ordinal-field performance, and seed stability rather than higher accuracy over a well-reproduced baseline.

    Source: §VII · p. 8, §IX · p. 10

    The 324-item synthetic holdout uses labels from the same model family as training, so shared labeling errors may be undetectable; per-term ablations use one seed and loss weights were fixed a priori.

    Source: §VIII · pp. 9–10

Abstract

from arXiv · show

Single-token typed-decision models answer a schema question by reading the logits of a few one-letter answer codes at a single position: they are fast and return a probability for every allowed answer, but they are trained with plain cross-entropy that ignores most of the structure in their training data. We study such a model whose data is curated as contrastive pairs---two contexts that differ in one edited fact that flips the answer---each carrying a machine-checked certificate that deleting the decisive sentence makes the fact unknown. We propose PACT, which turns this structure into four training terms that need no new annotation: a difference-in-differences margin over each pair that is invariant to any shared logit offset, a permutation-consistency term against answer-code position bias, an evidence-necessity term on certificate-verified ablated contexts, and an ordinal transport cost for rubric fields, plus a three-parameter contextual temperature. On a frozen 324-item holdout with three seeds, PACT matches the published recipe in accuracy ($84.6\%$ vs. $85.2\%$; McNemar $p \ge 0.50$ at every seed) while giving the lowest position bias of all runs (answer flips under relabelling $9.8\%$ vs. $13.8\%$) and the lowest ordinal error on rubric fields (MAE $0.232$ vs. $0.311$). Against a control with the same optimiser and schedule but cross-entropy only, PACT is significantly more accurate at two of three seeds, halves the seed-to-seed spread and lowers NLL by $26\%$. Seed-matched ablations and pre-specified falsification tests locate these gains precisely: no single term raises raw accuracy, and the method's value lies in robustness and stability rather than headline accuracy. Code, data splits, trained adapters, and all run records are available at https://github.com/BennyLinntu/PACT-Pairwise-Anchored-Calibrated-Tuning-for-Single-Token-Typed-Decisions.

I. INTRODUCTION

Typed decisions efficiently score allowed answer codes at one position, but the released training objective discards the pair, certificate, and answer-code structure in curated data. PACT uses that structure in its training objective while retaining the serving interface.

I. INTRODUCTION

  • Typed decisions score one-token answer codes at a single position, avoiding generated JSON parsing or sampling while assigning probabilities to every allowed answer.The schema supplies the label space at inference time.Source: §I · p. 1
  • Each curated pair differs in an edited evidence sentence that flips the answer, and its certificate verifies that removing either focus sentence leaves the fact unknown.The released objective treats pair members independently, ignoring the pairing, certificate, and arbitrary answer-code assignment.Source: §I · p. 1
  • PACT adds four structure-derived terms and a three-parameter contextual temperature without changing the serving prompt, answer codes, or adapter format.Its evaluation finds no accuracy gain over the published recipe, but lower position bias and ordinal error, greater stability, and cheap post-hoc calibration.Source: §I · p. 1

II. RELATED WORK

  • Prior work studies single-token serving and data,, while this paper keeps that interface fixed and changes the objective.Counterfactual and contrast-set examples commonly retain an unchanged objective; PACT instead uses pair output geometry and certified evidence removals as training constraints.Source: §II · p. 2
  • PACT trains against answer-code permutation bias with bounded Jensen–Shannon consistency, makes temperature depend on prompt statistics, and applies squared earth-mover loss to ordered rubric fields.These choices build on prior work on option-order bias –, temperature scaling, and ordinal losses, [22].Source: §II · p. 2

III. PROBLEM SETTING

The task uses one-position logits over schema-defined answer codes, with curated pairs that flip a certified focus fact. PACT's pair margin supervises the change in answer log-odds across each edited pair.

A. Single-token typed decisions

  • A typed decision renders a schema's allowed answers under one-token codes and serves their normalized logits from the model's final-position hidden state.The answer-to-letter mapping is arbitrary, so dependence on the code letter is error; candidate-logit differences are the natural object for supervision.Source: A. Single-token typed decisions · p. 2

B. Contrastive curation with certificates

  • The training file contains 1,338 two-record families whose contexts differ in one edited evidence sentence, flip the reference answer, and carry certificates verifying that removing either focus sentence makes the fact unknown.The released recipe trains on the records independently and does not construct removed-evidence contexts, while the certificate makes indifference a logit-geometry constraint.Source: B. Contrastive curation with… · p. 2

A. Canonical choice space

B. Counterfactual difference-in-differences margin

  • The pair margin is unchanged by any shared logit offset, and is positive when both pair members are correct under the binary restriction to their two reference answers.The pair members must be evaluated together for the coupled margin.Source: B. Counterfactual… · p. 3

C. Permutation consistency

PACT adds consistency across answer-code permutations, constrains the two answers linked to a removed decisive fact, and uses an order-sensitive loss for rubric levels.

C. Permutation consistency

  • The permutation-consistency term uses Jensen–Shannon divergence between distributions from two code permutations, a bounded, symmetric, label-free penalty with no privileged teacher view.At inference, canonical log-probabilities can also be averaged across K views; the K=1 to K=4 gain measures residual bias not removed by training.Source: C. Permutation consistency · p. 3

D. Evidence-necessity regularisation

  • On a certificate-verified ablated context, evidence necessity encodes indifference between the pair's two reference answers because the remaining evidence cannot discriminate them.The term makes no constraint on other answers, the ablated distribution's entropy, or which of the pair's answers is correct.Source: D. Evidence-necessity regularisation · p. 3

E. Ordinal transport for rubric fields

  • Squared earth-mover distance between predicted and reference cumulative distributions makes ordinal training favor probability mass near the target level, not merely on it.Cross-entropy does not distinguish near misses from far ones because it is invariant to level ordering.Source: E. Ordinal transport for rubric fields · p. 3

F. Contextual temperature

A global temperature cannot reflect differing miscalibration across choice counts and prompt lengths. PACT fits a contextual temperature from those prompt statistics while preserving argmax decisions.

F. Contextual temperature

  • A global temperature cannot express the differing miscalibration of prompts with different choice counts and lengths.The contextual temperature is parameterized as a + b log C + c log(L/1000), with L denoting prompt length in tokens.Source: F. Contextual temperature · p. 3
  • The contextual temperature is fit by minimizing NLL with L-BFGS on a calibration split of training-side families.The temperature fit does not use those families to fit the adapter.Source: F. Contextual temperature · p. 3
  • Because the temperature positively rescales each item's logits, calibration leaves argmax and accuracy unchanged but can re-rank confidence across items and affect selective risk.The illustrated training pair changes one focus-sentence token from 6 to 9 days, flipping the answer; deleting that sentence makes the fact unknown.Source: F. Contextual temperature · p. 4, Figure 2 · p. 4

G. Objective, batching and cost

PACT combines four structure-derived loss terms, with group-based batching for paired and permuted views. The experiments use family-disjoint splits and a training setup aligned with the published recipe.

G. Objective, batching and cost

  • The objective sums cross-entropy with four regularizers, and each pair contributes two primary, two permuted, and one ablated view.The regularizer weights are (0.5, 0.5, 0.2, 0.3), chosen for scale comparability rather than tuned on evaluation data.Source: G. Objective, batching and cost · p. 4

A. Data and splits

  • The 324-record holdout contains 162 pairs from six source families and is never used for selection, calibration, or early stopping.Splits are disjoint by source family; the training file contains 2,676 records, 1,338 pairs, 34 families, and 10 domains.Source: A. Data and splits · p. 4, Table I · p. 4

B. Model and training

  • Training uses the published recipe’s Qwen3.5-9B base model and LoRA adaptation, with two epochs, cosine scheduling, and 584 steps.The runs use the pinned model revision, BF16, AdamW, and a 2,048-token limit; Table II distinguishes the published one-epoch linear-schedule setup from the two-epoch cosine, rsLoRA, and LoRA+ setup.Source: B. Model and training · p. 4, Table II · p. 5

C. Compared runs

The comparisons separate the objective from optimization changes, score performance across accuracy, calibration, ordinal error, and code-order sensitivity, and use pre-specified tests for the proposed effects.

C. Compared runs

  • PACT, CE-only, and the reproduced published recipe form three main arms, with seed-matched leave-one-out ablations using seed 17.PACT versus CE-only isolates the objective; the reproduced recipe versus CE-only isolates optimization changes.Source: C. Compared runs · p. 5

D. Metrics and statistics

  • The evaluation reports accuracy, pair accuracy, NLL, calibration, risk-coverage, Score-field MAE, and code-order sensitivity, with paired tests for accuracy.Accuracy comparisons use exact McNemar tests and 10,000-resample paired bootstrap tests per seed.Source: D. Metrics and statistics · p. 5

E. Pre-specified falsification criteria

  • Five pre-specified criteria test pair-margin, permutation-consistency, evidence-necessity, ordinal-transport, and calibration effects.They predict changes in pair accuracy, code-order TV and flips, confidence on ablated contexts, Score MAE, and ECE, respectively.Source: E. Pre-specified falsification criteria · p. 5

A. Main comparison

PACT matches rather than beats the published recipe in accuracy, while improving stability relative to CE-only and showing its strongest advantages in position bias and rubric-score error. Contextual temperature lowers holdout ECE, although its fitted shape is unstable across runs.

A. Main comparison

  • PACT raises mean accuracy by 2.4 pp over CE-only, halves seed-to-seed standard deviation from 6.0 to 2.8 pp, and lowers NLL by 26%.The accuracy difference is significant at seeds 18 and 19, but seed 17 favours CE-only.Source: A. Main comparison · p. 5
  • PACT has the highest mean pair accuracy at 73.3%, the lowest Score MAE among multi-seed arms at 0.232 versus 0.311 and 0.322, and clears the base model by 18–24 pp.The pair-accuracy lead is not significant given the 6 pp seed spread.Source: A. Main comparison · p. 5

C. Calibration

  • Contextual temperature lowers holdout ECE in all nine multi-seed runs, from 10.8% to 7.9% for PACT, while leaving accuracy unchanged.The corresponding changes are 9.5% to 5.4% for Nimble recipe and 14.3% to 10.3% for CE-only.Source: C. Calibration · p. 5

A. Main comparison

  • 84.6±2.8% accuracy for PACT versus 85.2±2.1% for Nimble recipe; McNemar p ≥0.50 at every seed.The published recipe is also the best-calibrated run, with ECET 5.4% versus 7.9% for PACT.Source: A. Main comparison · p. 5, Table III · p. 6

B. Position bias and permutation ensembling

  • At K=4, PACT has the lowest TV at 5.6% and flip rate at 9.8%, and gains nothing from ensembling (∆K = −0.7 pp).The published recipe has a 13.8% flip rate; PACT’s single pass already captures what ensembling would recover.Source: B. Position bias and permutation… · p. 5, Figure 3 · p. 6
  • In a local sweep over K=1–4, PACT’s residual TV stays at 7.3–7.6%, 2.2–2.4× below the base model, while accuracy peaks at K=2 for both.The BF16 multi-seed results show the same non-monotone pattern: accuracy is 84.6% at K=1 and 83.8% at K=4.Source: B. Position bias and permutation… · p. 5, Figure 4 · p. 6

C. Calibration

  • For the shipped model, negative fitted slopes on choice count (b = −0.42) and prompt length (c = −0.48) indicate greater under-confidence on many-choice fields and long prompts.A single scalar temperature cannot express this pattern.Source: C. Calibration · p. 6
  • Across nine runs, contextual-temperature slopes vary widely, and with 226 calibration records they are weakly identified; the authors call scalar temperature a reasonable default.They recommend regularising the contextual form or fitting it on a larger split.Source: C. Calibration · p. 6

D. Seed-matched ablations

Seed-matched removals show that PACT’s terms mainly support robustness and ordinal behavior, while holdout differences vary by field kind rather than yielding uniform accuracy gains.

D. Seed-matched ablations

  • At seed 17, removing LCF or LEMD leaves accuracy at 84.3% vs. 84.0%, while removing LPC or all terms raises it to 87.7% or 87.3%.Each ablation is a single run against a relatively weak seed-17 PACT reference, so these directions are not significance claims.Source: D. Seed-matched ablations · p. 6, Table VI · p. 8
  • In leave-one-out ablations, removing terms raises the flip rate from 8.0% to 13.0–17.6% and TV; removing LCF, LNR, LEMD, or permuted views raises Score MAE from 0.229 to 0.282–0.356.Table VI uses seed 17 for every row, with single-pass results except the last two columns, which use K=4. Its colours indicate directions, not significance.Source: D. Seed-matched ablations · p. 7, Table VI · p. 8, E. Where the differences are · p. 8
  • Removing LNR degrades accuracy by 3.1 pp, pair accuracy by 8.0 pp, AURC from 4.44 to 6.63, and position bias from TV 5.0% to 11.7%.The removal also worsens Score MAE.Source: D. Seed-matched ablations · p. 7, Table VI · p. 8

E. Where the differences are

  • Across all three seeds, PACT is better than Nimble recipe on Score fields by 6.2–15.6 pp, but worse on Choice by 3.4–4.8 pp and Boolean by 0.9–4.4 pp.By domain, PACT is better on public services and worse on education and media at every seed; the authors report this as an observation, not a pre-specified hypothesis.Source: E. Where the differences are · p. 7
  • For Score fields, PACT’s ECE is 10.8% vs. 21.2% before T(x), and 10.1% vs. 16.6% after T(x); for Choice fields it is 16.0% vs. 10.3%.These are averages over seeds; per-kind ECE uses 64–146 items and is biased upward relative to the pooled value.Source: E. Where the differences are · p. 7
  • In 39 holdout pairs (24%; 33% for Choice), the model gives both members the same answer, ignoring the edited fact; this is 74% of imperfect pairs.The behavior remains the dominant error mode despite the mean training-pair margin ending well above m.Source: E. Where the differences are · pp. 7–8

F. Training dynamics

The discussion finds that PACT’s evidence supports robustness and ordinal-error benefits, not accuracy gains over a well-reproduced cross-entropy baseline. Training objectives behave as intended, but transfer, training cost, and checkpoint selection remain important constraints.

F. Training dynamics

  • The pair margin rises from −0.3 nats to 10–13 nats after exceeding m = 2 at step 20, while the ablated-context logit gap falls from 3.4 to below 0.1 nats.This indicates the intended training behavior: separating pair members and becoming indifferent between their answers on certificate-verified ablations; LNR is the noisiest term.Source: F. Training dynamics · p. 8, Figure 9 · p. 9

G. Hypothesis scorecard

  • The pre-specified scorecard supports two hypotheses, partly supports two, and leaves one inconclusive; none is contradicted outright.H1, the paper’s motivating idea, lacks clean support from a single-seed ablation.Source: G. Hypothesis scorecard · p. 8

VII. DISCUSSION

  • The evidence does not support higher accuracy over a well-reproduced cross-entropy baseline, but supports reduced position bias and ordinal error, greater stability, and lower ECE after post-hoc calibration in every run.Only the overall temperature level, not its contextual shape, is reliably estimated; PACT is about five times more expensive to train than the published recipe, which suffices for pooled accuracy.Source: §VII · p. 8
  • In about a quarter of holdout pairs, both members receive the same answer; meeting the mean margin on training pairs does not fully transfer to unseen source families.The authors point to margin schedules, harder negative edits, or pair-level data augmentation as promising next steps.Source: §VII · pp. 8–9
  • PACT takes 1.37 h per run versus 0.50 h for CE-only and 0.26 h for the Nimble recipe, with peak GPU memory of 28.8 GiB versus 21.8 GiB.Cost scales with the number of views: dropping permuted views saves 45% of time, while serving cost is unchanged.Source: §VII · p. 9, Table VIII · p. 9
  • Checkpoint selection is a variance source comparable to the effects studied: the lowest-selection-NLL run has the lowest holdout accuracy among fine-tuned runs, at 75.6%.The selection split has only two source families and 116 records, and can reward fitting those families rather than transfer.Source: §VII · p. 9

VIII. LIMITATIONS

PACT improves robustness and stability rather than headline accuracy, while the paper’s conclusions remain bounded by a narrow model-labelled holdout, directional ablations, and one model scale.

VIII. LIMITATIONS

  • The 324-item holdout spans six synthetic source families, but model-generated labels may share errors with training labels that the benchmark cannot detect.The authors describe the holdout as narrow.Source: §VIII · p. 9
  • Per-term ablations use one seed each, making their claims directional; significance tests cover only the three multi-seed arms.Source: §VIII · pp. 9–10
  • CE-only reaches the best selection scores, but those scores do not carry over to the holdout.Source: Figure 10 · p. 10
  • The loss weights were fixed a priori, so negative accuracy findings apply to this weighting setting rather than every possible weighting.Source: §VIII · p. 10

IX. CONCLUSION

  • PACT does not make the model more accurate than a well-reproduced baseline, but reduces answer-code-order sensitivity, improves ordinal rubric performance, and stabilizes results across seeds.The authors characterize its value as robustness and stability rather than headline accuracy.Source: §IX · p. 10

APPENDIX A ADDITIONAL TABLES

The appendices provide additional accuracy and calibration tables, implementation records, and a Tetris demonstration of the single-token interface; that demonstration is qualitative and does not support the holdout claims.

APPENDIX A ADDITIONAL TABLES

  • Tables IX and X report field-kind accuracy for every run and seed, plus fitted temperatures and ECE before and after calibration on both splits.Source: Appendix A · p. 11

APPENDIX B IMPLEMENTATION AND REPRODUCIBILITY

  • The release includes configurations, data splits, adapters, run records, and regeneration scripts; checkpoints also verify the scoring-prompt hash.The appendix states that hyperparameters were not tuned on the holdout.Source: Appendix B · p. 11

APPENDIX C APPLIED DEMONSTRATION: SEQUENTIAL DECISIONS IN

  • The Tetris demonstration turns legal placements and their two-ply consequences into schema choices, then further tunes the adapter with REINFORCE using placement scores as rewards.The engine enumerates options; the model selects the highest-logit choice.Source: Appendix C · pp. 11–12
  • The sampled top-pick rate rose from 19% at the first update to 87.5% by step 16, then averaged 88%–99% from step 200 onward.Fig. 11a shows per-update agreement and its 50-step mean.Source: Figure 11 · p. 12, Appendix C · p. 12
  • Greedy evaluation stayed near its 400-piece ceiling at 17,000–18,000 mean points and about 158 lines; one uncapped game reached 1,500 pieces and 598 lines.The capped protocol cannot show improvement; live app play cleared more than 130 lines and reached level 13 in six minutes.Source: Appendix C · p. 12
  • The Tetris result is a single run without held-out repetitions or a greedy pre-training baseline, and the policy follows rankings already computed by the engine.The authors limit its evidence to the serving interface’s use in a new decision task.Source: Appendix C · p. 12

Related papers

Loading 2609.35865v1…