PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Language Models To Be Coherent Decision-Makers

Khurram Yamin, Xavier Fernandes, Paul Koch, Bryan Wilder, Eric Horvitz

arXiv:2610.09164v1cs.AI

TL;DR

Language models can predict outcomes yet make incoherent decisions when payoffs shift their beliefs, they misapply utilities, or required information is missing. This paper fine-tunes models to preserve elicited beliefs and act on supplied utilities, then tests transfer and missing-information recognition. Targeted fine-tuning improves coherent decision-making, and routing separates completeness recognition from decision execution.

  • Problem

    Reliable decisions require models to preserve outcome beliefs, apply supplied utilities, and recognize when information needed to justify action is missing.

    Source: §1 · p. 1

  • Method

    The study fine-tunes models to pair their independently elicited beliefs with utility-maximizing actions, tests held-out domains, framings, and payoff classes, and trains a gate for missing utilities.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    Targeted SFT improves coherence and transfers across domains and framings; routing gives the best end-to-end accuracy for five of six models with almost no loss in belief stability.

    Source: §1 · p. 2, §4.2 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    Models can learn to identify missing decision-relevant utilities, while routing separates that recognition from utility-sensitive execution on complete cases.

    Source: §1 · p. 2

    The study is scoped to binary decisions with asymmetric error costs; richer action and outcome spaces and larger or closed models remain future tests.

    Source: §3.1 · p. 3, §5 · p. 10

Abstract

from arXiv · show

Reliable decision-making requires more than accurate prediction: a model must preserve its beliefs, apply the relevant utilities, and recognize when the information needed to justify an action is missing. We study whether language models can learn this decision procedure from supervised fine-tuning and generalize it across domains and differing natural-language expressions of the decision challenge. Across 20 datasets, we explore challenges of belief instability and decision-making errors by first eliciting probabilities of outcomes and then varying only the utilities and the framing of the decision problems, while holding the evidence fixed. We train models to preserve elicited beliefs while selecting the action that maximizes expected utility, and evaluate transfer to unseen application domains, held-out framings, and different classes of payoff structures. We further introduce incomplete-information settings in which required utilities are withheld and replaced with irrelevant text, testing whether models can distinguish missing decision-relevant information from merely additional context. We find that targeted fine-tuning substantially improves coherent decision-making and that in many situations, learning transfers across domains and framings to situations unobserved during training. Further, models trained for decidability learn to identify when action cannot be justified based on missing information. Finally, we show the value of a routed system that considers separately the recognition of decision completeness and utility-sensitive decision execution.

1 Introduction

The section asks whether small language models can follow a coherent decision procedure beyond seen domains and framings. It motivates preserving beliefs while applying utilities, and recognizing when information needed to justify an action is missing.

1 Introduction

  • Good decisions require combining beliefs about uncertain outcomes with utilities over consequences; the same risk can justify different actions under different utilities.When utilities are unstated, the passage says no action is justified.Source: §1 · p. 1
  • The section asks whether small models can learn a coherent decision procedure that generalizes beyond observed domains and framings.The motivation notes that some decision-support settings require on-premises data, low-cost and low-latency calls, or local hardware.Source: §1 · p. 2
  • Across 20 prose-rendered domains, the study holds evidence fixed while varying utilities and framing to separate belief shifts from actions inconsistent with stated beliefs.Base models act against their own beliefs and shift those beliefs when payoffs appear.Source: §1 · p. 2
  • Supervised fine-tuning targets each model’s independently elicited belief and the expected-utility-maximizing action, using either a structured answer or a chain of thought.The targets teach the mapping from belief and consequences to action without imposing external probability labels.Source: §1 · p. 2
  • Targeted fine-tuning improves coherence and transfers across domains and framings, while a routed system gives the best end-to-end accuracy for five of six models with almost no loss in belief stability.Models trained for decidability identify missing information but decide less well on complete cases, motivating separate recognition and execution specialists.Source: §1 · p. 2

2 Related Work

The related work distinguishes this paper’s goal of implementing supplied utilities without changing beliefs from work on calibration, preferences, and downstream optimization. It also situates the paper’s tests of generalization and decision completeness alongside work on procedure learning and routing.

Beliefs, utilities, and decisions

  • Unlike decision-focused learning, this work preserves the model’s belief and trains the mapping from belief and supplied consequences to action.The related work contrasts this objective with methods that modify predictors for downstream optimization.Source: §2 · p. 2

Learning procedures that generalize

  • Because fine-tuning gains can reflect surface associations or memorization, the paper holds out whole domains, framings, and payoff classes to test generalization.The related work connects this design to concerns raised by Ren & Sutherland (2025) and Kang et al. (2024).Source: §2 · p. 2

Incomplete information and specialization

  • The paper treats a missing utility as making an action undecidable and routes only complete cases to a specialist, separating completeness recognition from decision execution.Figure 1 depicts complete-information training followed by incomplete-utility cases, where the model withholds action or routes a complete problem to the specialist.Source: §2 · p. 3, Figure 1 · p. 3

3 Methods

The methods separate complete-information decisions from cases where missing utilities make an action undecidable, then train and evaluate models on both settings.

3 Methods

  • The study first evaluates complete-information decisions, then withholds utility information to test whether models identify when a decision is undecidable.For incomplete cases, models must identify what is missing rather than choose an action.Source: §3 · p. 3

3.1 Decision setting

  • Binary actions minimize expected cost by comparing the model’s belief with a threshold determined by false-positive and false-negative costs.The study elicits an evidence-only reference belief, then tests whether the model preserves it when payoffs are introduced.Source: §3.1 · pp. 3–4

3.2 Complete-information benchmark

  • The benchmark renders 200 cases from 20 tabular datasets across five cost pairs and eight framings, preserving the same implied threshold across renderings.Crossing cases, utilities, framings, and two repetitions yields 16,000 decision prompts per prompting condition.Source: §3.2 · p. 4

3.3 Complete decision models and generalization

  • Complete-information SFT uses each model’s own elicited belief and the action it implies, with domains and framings held out for transfer tests.Training splits hold out five domains entirely and test framing transfer with both cross-class and full-class holdouts.Source: §3.3 · pp. 4–5

3.4 Decision decidability under incomplete information

  • The decidability task withholds one or both error costs and substitutes length-matched irrelevant text; a routed system sends predicted complete cases to a CoT specialist.For incomplete conditions, targets identify the missing cost and omit the action; complete cases receive an action target.Source: §3.4 · pp. 5–6, Figure 2 · p. 5

3.5 Evaluation measures

  • Evaluation separates belief distortion from decisions inconsistent with the reported belief, while decidability measures assess condition labels, complete-case actions, and belief stability.Belief-conditional decision error isolates utility application; belief distortion compares evidence-only and decision-time beliefs, with a sampling-noise floor.Source: §3.5 · p. 6, Figure 3 · p. 7

4 Results

Fine-tuning improved complete-information decisions and belief stability, with transfer varying by whether the challenge changed the domain or the framing. Under incomplete information, separating missing-cost recognition from decision execution produced the strongest end-to-end accuracy for most models.

4.1 Setup

  • The evaluation covered six instruction-tuned checkpoints and pooled eight framing holdouts, with 25,600 test prompts per model.Results were split by framing and domain status.Source: §4.1 · p. 7

4.2 Complete-information decisions

  • Direct SFT lowered utility-misapplication error for every model on every slice, with held-out-domain error within 0.03 of seen-domain error for five models and lower for the sixth.Error rose across seen, cross-class, and full-class framings for every model; full-class error remained 0.16 to 0.43.Source: §4.2 · p. 8
  • CoT SFT produced the most coherent complete-information decisions, though its advantage over prompting shrank with scale and full-class framing remained its weakest transfer setting.For Gemma 12B and the two larger Qwen models, CoT SFT misapplied utilities on fewer than 2% of seen-framing items; some models performed worse than their base CoT prompts on held-out full-class framings.Source: §4.2 · p. 8
  • Direct SFT reduced belief distortion, while held-out framings raised distortion by at most 0.02 and held-out domains raised it two- to threefold.CoT SFT preserved beliefs better for Gemma, while Direct SFT did so better for Qwen.Source: §4.2 · p. 8

4.3 Decision decidability

  • Direct SFT recognized missing costs on at least 93% of items for every model except Gemma E2B, which reached 85%; CoT SFT was never better and often worse.Direct SFT reached 99.7% for Gemma 12B; CoT SFT ranged from 59–93%.Source: §4.3 · p. 8
  • On complete items, Direct SFT errs 12–27%, four-condition CoT SFT 9–50%, and the complete-information CoT specialist 2–18% on items routed to it.The results motivate routing: a Direct SFT gate recognizes missing information, while the specialist handles complete-item decisions.Source: §4.3 · p. 8
  • 88–99% end-to-end joint accuracy made the routed system most accurate for five of six models; it was within two points of Direct SFT on Gemma E2B.The routed system shares Direct SFT’s gate and hands complete items to a specialist.Source: §4.3 · pp. 8–9, Figure 4 · p. 9
  • Under incomplete information, Direct SFT kept excess belief distortion to 0.04–0.09; the routed system stayed within 0.04 of Direct SFT for every model.CoT SFT had lower distortion for Gemma but was unstable for Qwen, where the 4B and 9B models distorted beliefs more than base prompts.Source: §4.3 · p. 9

5 Discussion

The study finds that supervised fine-tuning improves coherent decisions and often transfers across domains and framings, while its measures certify coherence rather than belief accuracy. The authors identify richer decisions and learning to acquire missing information as future work.

5 Discussion

  • Across six small models, targeted fine-tuning repaired belief shifts and utility-misapplication errors, with gains transferring to unseen domains and framings.Direct SFT improved every model on every slice; CoT SFT generalized least well to payoff structures absent from training.Source: §5 · pp. 9–10
  • The benchmark covers binary decisions with two error costs, and its model-relative reference belief measures coherence rather than accuracy.The evaluation holds evidence fixed while varying utilities and their framing, separating belief distortion from utility misapplication.Source: §5 · p. 9
  • The authors propose extending the procedure to richer action and outcome spaces, teaching models to acquire identified missing information, and testing routing on larger and closed models.Source: §5 · p. 10
  • The reproducibility statement specifies prompts, datasets, training and decoding settings, and says code, case lists, model outputs, and reported metrics will be released with the paper.Source: Reproducibility statement · p. 10
  • The authors report using generative AI for experimental-pipeline coding and light text editing, while stating they did not use it to generate research ideas or substantive paper content.They say AI-assisted work was reviewed and that code outputs were verified.Source: AI use statement · p. 10

A Training and inference details

The appendix specifies the models, adapter training, inference, reproducibility, and noise-floor calibration used in the experiments. It also reports the total training runs and the shared hardware and software setup.

A Training and inference details

  • The experiments use six instruction-tuned checkpoints from Gemma and Qwen families, with the same data construction and training protocol across models.Source: A Training and inference details · p. 12
  • SFT uses QLoRA with frozen 4-bit NF4 base weights and rank-16 LoRA adapters on language-model attention and feedforward projections.The adapters are trained with AdamW for two epochs at a constant learning rate of 2 × 10−4.Source: A Training and inference details · p. 13
  • Training rows use the model’s chat template and causal language-modeling loss; examples are never truncated, and validation checkpoints are evaluated and exported for reported results.Source: A Training and inference details · p. 13
  • The study trains 192 adapters in total, while the routed system reuses existing decidability Direct and complete-information CoT adapters without additional training.Source: A Training and inference details · p. 13
  • Training and inference run on a single 80 GB A100, with model adapters served through vLLM’s LoRA path.Source: A Training and inference details · pp. 13–14
  • Generation uses temperature 0.7 and top-p 0.95, with identifier-derived seeds to keep repeated samples independent while making evaluation reproducible.Qwen3.5 runs with thinking mode disabled so scored reasoning comes from the prompted or supervised CoT.Source: A Training and inference details · pp. 13–14
  • The sampling-noise floor is estimated by repeating prompts with a different seed and pairing beliefs by prompt identifier; decidability prompts use a separate floor.The reported conversion for direct-style beliefs is approximate because those beliefs fall on a coarse grid.Source: A Training and inference details · p. 14

B Prompts

The prompts keep case evidence and event descriptions fixed while varying payoff presentation and response instructions. They separately elicit beliefs, test decisions under complete information, and assess whether missing costs make a decision undecidable.

B Prompts

  • Across elicitation and decision prompts, the case evidence and event strings remain identical; only payoff presentation changes across the eight framings.Source: B Prompts · p. 14

B.1 Response-style instructions

  • Direct requires a structured final answer, CoT invites reasoning, and In-context adds a framing-neutral worked example before step-by-step reasoning.For decision prompts, CoT asks the model to carry its probability into an expected-value comparison.Source: §B.1 · pp. 14–15

B.2 Belief elicitation

  • Beliefs are elicited five times from evidence and the event alone, then averaged into the reference belief.The elicitation prompt contains no actions or payoffs.Source: §B.2 · p. 15

B.3 Complete-information decision framings

  • Complete-information framings derive their quantities from a shared decision threshold and base scale, expressing the same decision through varied payoff descriptions.Source: §B.3 · p. 16
  • The framings include abstract loss-ratio and utility-ratio versions, which compare expected costs or rewards without giving specific amounts.Source: §B.3 · p. 18

B.4 Decision decidability prompts

  • Decidability prompts retain the evidence and framing but replace a withheld error cost with matched-length irrelevant administrative text.The illustrated incomplete items each state one error cost and omit the other.Source: §B.4 · p. 18, Figure 5 · p. 19
  • The four-way response distinguishes a missing false-positive cost, missing false-negative cost, both missing, or enough information to decide.A decision is requested only when both costs are available.Source: §B.4 · pp. 19–20
  • The worked example shows that with both costs supplied, the model compares expected error costs and chooses the lower-cost action.The decision-specific CoT instruction asks the model to identify the applicable information condition before reasoning about evidence.Source: §B.4 · pp. 19–20

C Domains

The benchmark draws on 20 public tabular domains with binary outcomes, rendering selected row covariates as prose and defining an event from the outcome. Five domains are reserved for test-time transfer to unseen subject matter.

C Domains

  • Each case renders selected covariates from a dataset row as prose and defines its event from the binary outcome.Source: C Domains · p. 22
  • Fifteen domains supply training, validation, and in-domain test splits, while five appear only at test time to measure transfer to unseen subject matter.Source: C Domains · p. 22
  • Table 1 lists 20 explored domains, with ten cases sampled per domain for 200 evidence contexts.Source: Table 1 · p. 23
  • Outcome labels construct and audit cases but are not used as probability targets for supervised fine-tuning.Source: C Domains · p. 24
Loading 2610.09164v1…