Source-linked AI summary
Scored vs. Generated Readouts in Behavioral Language Models: An Empirical Study of Elicitation Format
Touchapon Kraisingkorn, Krittin Pachtrachai, Wachiravit Modecrua
TL;DR
Behavioral language models are often treated as interchangeable predictors and explainers, but the paper tests whether their readout format changes ranking quality. Holding checkpoints and prompt content fixed across retail tasks, it compares scored answer-token probabilities with rationale-mediated predictions and finds a consistent advantage for scoring, with deployment and scope conditions.
Problem
Behavioral language models can predict outcomes and generate explanations, but evidence is limited on whether these readouts are interchangeable for ranking real customer outcomes.
Method
The study holds checkpoint and prompt content fixed, varies elicitation format, and compares scored probabilities with generated rationale-then-answer predictions across retail behavioral tasks.
Results
Across 13 model-domain cells, the scored readout ranks outcomes better in 12 of 13 cells, with differences of up to 14.5 AUC points.
Takeaways & Limitations
Generated rationales should be retained for auditability and usability, while ranking should be sourced from the scored head rather than trusted implicitly from rationale-mediated answers.
Takeaways & Limitations
The evidence covers four retail binary-prediction domains and does not claim generalization to other task families; ranking is only one property of a deployed simulator.
Abstract
from arXiv · showhide
Language models fine-tuned on customer behavior can predict outcomes and generate explanations, but these readouts are often treated as interchangeable. Holding model checkpoint and prompt content fixed, we compare probabilities obtained by scoring answer tokens with predictions generated after a written rationale. Across 13 model-domain cells covering four retail tasks in three markets, including two using fully public data and checkpoints, the scored readout ranks outcomes more accurately in 12 of 13 cells (two-sided sign test, p approximately 0.003), by 1.5 to 14.5 points in area under the receiver operating characteristic curve (AUC). Paired bootstrap confidence intervals exclude zero in every newly measured cell. The gap varies with task-specific supervision and mismatch between training and serving formats, ranging from -2.2 points for an untuned base model to +13.7 for rationale-format supervision. Analysis of approximately 9,000 rationales identifies two correlates: reduced reliance on the dominant predictive feature and convergence on stock formulations. Probability saturation does not track the gap. A third readout, eliciting a probability before any verdict, improves calibration (Brier score from 0.47 to 0.15) while ranking within noise of scoring, but only for outcome rates represented in training; it is worse than scoring when the scored head is already calibrated. We interpret these differences through the objectives matched by each readout, identify training choices that narrow the gap, and propose retaining generated rationales while sourcing ranking from the scored head.
1 INTRODUCTION
Behavioral language models are often expected to predict and explain interchangeably, but this paper directly tests that assumption by holding the checkpoint and prompt content fixed while varying elicitation. Across retail domains, scored probabilities generally rank outcomes better than generated rationale-then-answer readouts, while rationales retain distinct auditability and interface value.
- Motivation: A behavioral simulator can predict customer actions and generate explanations from the same model, encouraging practitioners to treat both readouts as interchangeable.The paper frames this issue in retail decisions such as coupon targeting, post-delisting prioritization, and incentive sizing.
- Research question: The paper tests readout interchangeability directly by holding checkpoint and prompt content fixed while varying only how the model is asked.The comparison contrasts scored decision probabilities with generated rationales followed by an answer.
- Main finding: The scored readout ranks real outcomes more accurately than the generated readout across four behavioral domains, with the largest difference in models trained most heavily on the task.This measures ranking accuracy rather than the overall value of generated rationales.
- Scope: Generated rationales remain valuable for auditability, interpretability, transfer to unseen question forms, and user-facing interaction despite weaker ranking in this measurement.The paper quantifies a ranking cost rather than arguing against rationale-producing behavioral models.
- Interpretation: The scored readout matches the binary cross-entropy objective and AUC ranking target, whereas generation routes evidence through a text bottleneck that can omit or distort information.The downstream probability cannot recover evidence that the rationale fails to carry.
- Contributions: The paper reports four contributions: measuring the gap, identifying supervision and format mismatch as moderators, analyzing rationale correlates, and evaluating an additional probability elicitation.The proposed deployment guidance keeps generated rationales while sourcing ranking from the scored head.
2 RELATED WORK
Prior work shows that elicitation format affects token probabilities, generated answers, reasoning confidence, and verbalized probabilities, but it does not directly test readout choice for real behavioral outcomes. This paper extends that literature with a controlled deployment comparison and supervision-dependent analysis.
- Elicitation and evaluation: Earlier studies found disagreement between first-token probabilities and generated answers, establishing elicitation protocol as a source of evaluation divergence.Those studies primarily addressed multiple-choice evaluation validity rather than behavioral deployment.
- Reasoning and calibration: Research on reasoning reports overconfidence and non-monotone accuracy with longer chains, while this paper studies ranking effects that can differ from calibration.Some generated arms can be better calibrated yet rank outcomes worse than scored arms.
- Verbalized probabilities: Verbalized confidence can rival logprob-based confidence, but prior results are sensitive to elicitation protocol and often use instruction-following or survey-style tasks.This paper extends verbalized-probability analysis to real behavioral outcomes.
- Behavioral simulators: Behavioral-simulator research established training recipes and in-domain evaluation, but generally did not vary elicitation format under a fixed checkpoint.The present contribution treats serving readout as a determinant of delivered ranking quality and relates robustness to supervision format.
3 MODELS, TRAINING, AND DATA
The study evaluates behavioral language models built from a shared base through domain adaptation and response-format supervision, using four binary retail prediction tasks with controlled splits. All readouts share prompt content and differ only in elicitation, while tabular baselines provide a separate reference where available.
- Training pipeline: Open-weights checkpoints derive from Qwen3.5-27B through continued pretraining, supervised fine-tuning, and, for some lines, preference or RL post-training.The study treats checkpoints as fixed and varies the readout rather than retraining models for each comparison.
- Training pipeline: The training pipeline injects domain dynamics through continued pretraining and activates response formats through supervised fine-tuning.The SEA and Western lines use consumer-voice data and transaction-related material in their continued-pretraining construction.
- Evaluation data: The evaluation covers four binary prediction tasks with temporal or customer-disjoint splits and leak audits, including proprietary SEA data and public voucher and grocery datasets.The domains include coupon redemption, voucher redemption, assortment-change response, and coupon-campaign redemption.
- Readouts: All arms use the same system prompt and factual user content; SCORED reads first-token P(YES) over {YES,NO}, while GENERATED first produces a rationale before scoring the answer.Thus the experimental contrast is elicitation format rather than prompt information.
- Baselines: The study reports tabular baselines where fitting is possible, and those baselines outperform language-model arms on the cited assortment-change and voucher settings.The paper therefore evaluates which readout recovers language-model information, not whether language models beat tabular prediction.
- Statistical comparison: Newly measured contrasts are paired on identical rows with 104-resample bootstrap confidence intervals.Historical cells use their original pinned protocols for decoding, shards, and seed counts.
4 EXPERIMENTS AND RESULTS
Across the experiments, scored readouts generally rank behavioral outcomes better than generated rationales, with the gap shaped by supervision format and accompanied by rationale-specific feature abandonment and template convergence. The analyses also test alternative explanations and identify a practical constraint for API models without token logprobs.
- Headline measurement: 12 of 13 cells favor scored over generated readouts, with the two-sided sign test rejecting chance agreement (p ≈.003).The untuned Base on SEA coupon is the lone exception, at −2.2 points.
- Supervision dose and format mismatch: +9.3 to +13.7 points is the gap for Rationale-SFT, while InDomain-SFT has the smallest tuned-model gap (+3.0) and best scored AUC (.808).The comparison links readout sensitivity to supervision dose and mismatch between training and serving formats.
- Order, isolated: −8.7 points separates decision-first from reasoning-first ordering on the matched voucher ablation (.736 versus .649).On the full voucher test set, the strongest supervised checkpoint falls from .841 to .775 when prose precedes the verdict.
- Degenerate case: .766 leaves-AUC is recovered by rescoring assortment-change predictions that generated a single class across all 380 rows.The generated arm assigned stated try-probabilities of .34–.43 despite a 5.3% base rate, leaving the information present but unsurfaced.
- Cross-cell correlates: Figure 1 compares scored-minus-generated AUC with anchor abandonment and probability saturation across 11 open-weights cells.Saturation does not order the cells: the untuned Base saturates most while showing one of the smaller gaps.
- Alternative explanations: The analyses treat rationale features as observational correlates, with the ordering ablation as the sole causal manipulation.Neutral or empty thinking blocks preserve scored-level AUC, and self-consistency recovers only 28% of the gap.
- Cross-cell correlates: Anchor abandonment tracks the gap: generated rationales reduce reliance on dominant predictors, including SEA correlation from .716 to .305 and voucher usage from .52 to .19.InDomain-SFT preserves the track-record correlation (.739 → .720), and anchor abandonment plus template share order gaps whereas saturation does not.
- Cross-cell correlates: Top-three opening phrasings cover 19% of SEA and 33% of US generations, while near-duplicate rationales create tie blocks that reduce ranking resolution.The pattern is described as convergence on stock formulations, often in a price-sensitivity register applied across personas.
5 DISCUSSION
The paper argues that readout choice has practical consequences: scoring can recover substantial ranking gains at no retraining cost, while alternative readouts and training choices have specific scope conditions. Generated rationales remain useful, but ranking should generally come from the scored head and deployments can be audited with rationale-based diagnostics.
- Practical implications: 3–14 AUC points can be recovered by changing the inference readout without retraining, new data, or checkpoint changes.The correction changes only which token is read from weights already in production.
- Practical implications: Two rationale-dump diagnostics let operators estimate the ranking cost of an existing deployment without reproducing the original training pipeline.The diagnostics are anchor rank correlation and top-3 template share.
- Alternative readout: Requesting a probability before any verdict improves calibration while ranking within noise of scoring on a single MultiTask-SFT checkpoint.The Brier score changes from .469 to .149, matching the frontier model’s .143, while the ranking remains close to the scored readout.
- Alternative readout: Verbalized probabilities help only for outcome rates represented in training, because unseen aggregate quantities produce nearly flat tuned-checkpoint outputs.For unseen delisting-cost rates, event-level spread is approximately .001, whereas the frontier model ranks events well with Pearson .894 and constant +17-point optimism.
- Alternative readout: The verbal-probability readout costs 4.6 AUC points for InDomain-SFT, whose scored readout is already calibrated with Brier .104.The paper therefore treats verbalized probability as a remedy for miscalibrated scored heads rather than a default upgrade.
- Training guidance: In-domain decision-format supervision, faithfulness filtering, and scored probes are the main training and serving choices proposed for retaining rationales while narrowing ranking losses.About half of draft negative-label rationales contained factually false disqualifying conditions before faithfulness filtering.
- Limitations: The evidence is limited to four retail binary-outcome domains, and the authors do not claim generalization to other task families.Mechanism analyses are suggestive rather than causal except for the ordering ablation, and tabular baselines outperform language-model arms where they are fittable.
6 CONCLUSION
The paper shows that scored and generated readouts from the same behavioral model are not interchangeable for ranking real outcomes. It recommends retaining rationales for auditability while sourcing ranking from the scored head, with the remaining question being whether generation’s ranking cost can be trained away.
- 6 CONCLUSION: 12 of 13 model-domain cells favor scored readouts, with differences reaching up to 14.5 AUC points and a sign-test p value of approximately .003.The pattern spans four retail prediction tasks and is presented as too consistent to attribute to chance.
- 6 CONCLUSION: The ranking gap varies with training and serving format mismatch and correlates with abandoning dominant predictive features and converging on stock phrasing.The paper reports that verdict saturation does not explain the observed gap.
- 6 CONCLUSION: The recommended deployment preserves generated rationales for auditability and usability while using scored probabilities for ranking.The serving-time change yields 3–14 additional AUC points and can be paired with diagnostics that operators run on their own deployments.
- 6 CONCLUSION: Whether faithfulness objectives or evidence-grounded rewards can eliminate generation’s ranking cost remains an open question.The paper recommends routing around the cost at serving time rather than claiming that training has already removed it.
7 DEPLOYMENT GUIDANCE
The paper recommends sourcing operational ranking from the scored readout while retaining generated rationales for interpretability. It also advises validating verbal probabilities, preferring in-domain supervision, and using rationale-based diagnostics in deployment.
- Deployment recommendations: Use the scored readout for targeting, ordering, and budget allocation, which yielded 3–14 AUC points over generating first.Generated rationales can still be retained after the scored decision without measured ranking cost.
- Deployment recommendations: Use verbal-probability readouts for rate forecasts when the scored head is miscalibrated and an anchor-style feature is present, after validating the trained register.The recommendation does not extend automatically to novel aggregates.
- Deployment recommendations: Prefer in-domain decision-format supervision when the target task’s training split is available because it improved ranking and readout robustness.
- Deployment recommendations: Diagnose deployments with approximately 2k rationales using prediction–dominant-feature rank correlation and top-3 template share.These diagnostics correctly ordered the readout gap in the measured cells, whereas saturation did not.
- Reproducibility and governance: The public US grocery domain includes 2,318 test rows with a 14.0% positive rate, public checkpoints, evaluation and analysis code, and per-row dumps.The voucher domain uses the public DMBGN benchmark; proprietary-domain protocols are released without row-level data.
- Reproducibility and governance: All customer data is pseudonymized or public, and no row-level proprietary data leaves the training environment.The paper also notes that calibration-preserving readouts can make overconfident targeting more visible to operators and support auditing.
A PROMPTS AND READOUT TEMPLATES
The supplement provides verbatim system and user templates for each domain and readout, together with prompt substitution mappings and parser specifications.
- Prompt templates: Templates cover the <think>, YES/NO, and probability cues used by the readouts.
- Prompt templates: The materials include the substitution mapping between prompts, parser regular expressions, and parser failure counts.
B TRAINING DETAILS
The training-details materials document parameter-efficient fine-tuning, optimization, data composition, and targeted ablations across checkpoints and task families.
- Optimization and adaptation: Training records include per-checkpoint LoRA ranks and target modules, optimizer settings, schedules, and sequence-packing choices.
- Data and ablations: The documentation reports CPT source mixes, token counts, SFT lane counts by task family, replay-fraction ablations, and a retrieval-in-CPT negative result.
C PER-CELL PROTOCOL LEDGER
The per-cell protocol ledger records inference configuration and quantifies possible serving-stack drift across evaluation cells.
- Per-cell configuration: Each cell records decoding mode, shard count, seed count, inference-engine version, and attention backend.
- Per-cell configuration: The ledger also includes a cross-stack drift measurement.
D INSTRUMENT CONSTRUCTION
The instrument-construction section covers detection rules, defect handling, temporal leakage auditing, and customer-disjoint construction across the study’s natural experiments and datasets.
- Detection rules define the assortment-change natural experiments.
- The defect ledger documents issues, including the withdrawn introduction family.
- The construction includes a US grocery time-split and leak audit and a customer-disjoint SEA coupon instrument.