KTO: Model Alignment as Prospect Theoretic Optimization

• Alignment methods such as RLHF and DPO outperform supervised finetuning alone, but preference feedback is scarce and expensive to collect.Source: §1 · p. 1
• Prospect theory explains alignment methods through human biases such as loss aversion, motivating the human-aware loss (HALO) family.Source: §1 · p. 1
• KTO directly maximizes generation utility using a binary desirable-or-undesirable signal instead of maximizing preference log-likelihood.Source: §1 · p. 2
• KTO matches or exceeds DPO from 1B to 30B parameters, including mathematical reasoning and human or GPT-4 evaluations.Source: §1 · p. 2
• KTO matches DPO while using up to 90% fewer desirable examples, showing robustness to extreme data imbalance.Source: §1 · p. 2
• With a sufficiently capable pretrained model, KTO can skip SFT without reducing generation quality, whereas DPO still benefits from SFT.Source: §1 · p. 2
• The authors conclude that no HALO is universally superior and that loss choice should reflect the inductive biases appropriate to each setting.Source: §1 · p. 2

• LLMs are traditionally trained through pretraining, supervised finetuning, and reinforcement learning from human feedback.Source: §2 · p. 2
• Pretraining maximizes next-token likelihood on a large corpus, while SFT adapts the model to downstream instruction-response data.Source: §2 · p. 2
• RLHF learns a proxy reward model from preferred and dispreferred outputs using a Bradley-Terry preference model.Source: §2 · p. 2
• A KL penalty constrains policy drift from the reference model, and PPO commonly optimizes the resulting RLHF objective.Source: §2 · p. 2
• DPO provides a closed-form preference loss that can recover the same optimal policy as RLHF under certain conditions, avoiding RLHF's slow and unstable sampling process.Source: §2 · p. 2

3. A Prospect Theoretic View of Alignment

• Prospect theory models human decisions under uncertainty using value and probability-weighting functions, including loss aversion relative to a reference point.Source: §3 · p. 3, §3.1 · p. 3

3.1. Prospect Theory

• Because LLMs do not expose complete outcome probabilities, the analysis focuses on value functions and their reference point, curvature, and loss sensitivity.Source: §3.1 · p. 3

3.2. HALOs

• HALOs define human value by applying a non-decreasing value function to a reference-adjusted log-ratio reward between the aligned and reference policies.Source: §3.2 · p. 3

3.3. Does being a HALO matter?

• HALOs match or outperform non-HALOs at every tested scale, with statistically significant differences only at 13B+; only HALO-aligned Llama-13B and Llama-30B reach at least 50% winrate.Source: Figure 2 · p. 4, §3.3 · p. 5

3.2. HALOs

• DPO and PPO-Clip are formally shown to be human-aware losses, and their HALO-defined rewards remain equivalent to the optimal reward up to an input-specific shift.Source: §3.2 · pp. 3–4

3.3. Does being a HALO matter?

• The evaluation compares HALO and non-HALO methods across Pythia and Llama families using GPT-4 judgments against SFT targets.Source: §3.3 · p. 5
• Offline PPO with dummy +1/-1 rewards performs as well as DPO except on Llama-30B, suggesting suitable inductive bias can make simple rewards effective.Source: §3.3 · p. 5
• The offline PPO baseline remains sensitive to hyperparameters and training instability, limiting its practical usability despite its strong results.Source: §3.3 · p. 5

4. Kahneman-Tversky Optimization

• KTO directly maximizes prospect-theoretic utility for generations using only binary desirable or undesirable feedback instead of preference likelihood.The loss uses a Kahneman-Tversky value model with a reference point, risk-aversion parameter β, and separate desirable/undesirable loss-aversion parameters.Source: §4 · p. 5, §4.1 · p. 5

4.1. Derivation

• KTO estimates a shared reference point from mismatched outputs within each microbatch, using a biased but convenient KL surrogate with clamping to reduce variance.The mismatched-output construction avoids using examples deliberately chosen as canonically good or bad, which can have unrepresentatively large rewards.Source: §4.1 · p. 6

4.3. Experiments

• KTO matches or exceeds DPO from 1B to 30B parameters, with KTO alone outperforming DPO alone for Llama-{7B, 13B, 30B} and improving GSM8K by 13.5 points in one comparison.The Llama gaps are significant at 7B and 30B after multiple-comparison correction, while Pythia shows no significant difference.Source: §4.3 · p. 8
• At sufficient scale, KTO works without SFT, whereas DPO without SFT tends to produce rambling and hallucinated conversations.KTO-aligned Llama-{13B, 30B} remains competitive with SFT+KTO, and the authors associate this with KTO maintaining average response length.Source: §4.3 · p. 8, Figure 4 · p. 8
• KTO tolerates extreme imbalance and unpaired data: it can outperform DPO after discarding up to 90% of desirable data and still wins with one output per input after 72% less training data.The one-y-per-x result on Mistral-7B uses OpenAssistant data without paired preferences.Source: §4.3 · pp. 8–9, Figure 5 · p. 9, Table 3 · p. 9

4.2. Hyperparameters

• Changing KTO’s loss structure causes substantial degradation, including 3.6- and 4.0-point drops after removing the reference point and 9.4- and 11.0-point drops after removing value-function symmetry.A risk-neutral identity value function causes BBH performance to collapse, while changing risk and loss aversion can help or hurt depending on the task.Source: Table 2 · p. 8, §4.3 · p. 9

4.4. Theoretical Analysis

• KTO can ignore examples whose implied rewards are too difficult or easy to learn from, potentially filtering noisy labels but also underfitting complex distributions.The analysis states that lower β and more epochs may mitigate this underfitting risk.Source: §4.4 · p. 9

4.5. KTO vs. DPO – when to use which?

• DPO and KTO are favored under different feedback conditions: DPO may work better with sufficiently clean, transitive preferences, whereas KTO offers better worst-case behavior under noise and intransitivity.The authors recommend KTO especially for binary or imbalanced feedback and attribute its empirical success partly to noisy, contradictory human preferences.Source: §4.5 · p. 10, §4.4 · p. 10

• HALOs should incorporate granular scores, especially when optimizing for multiple desiderata.Source: §5 · p. 10
• Future HALOs should support other modalities and model classes, including diffusion models without explicit output distributions.Source: §5 · p. 10
• Future work should address contradictory feedback under different fairness definitions and support online alignment data.Source: §5 · p. 10
• Ecologically valid evaluation in deployed real-world settings is needed to judge different HALOs.Source: §5 · p. 10

• KTO was as good or better than preference-based methods despite learning only whether outputs were desirable or undesirable.Source: §6 · p. 10
• The paper frames alignment objectives as human-aware losses whose inductive biases have analogs in prospect theory.Source: §6 · p. 10
• The best HALO depends on the inductive biases appropriate for each setting, so no single loss function is universally superior.Source: §6 · p. 10

• Improving helpfulness for one person may reduce utility for broader society, while unrepresentative feedback may limit benefits for others.Source: Impact Statement · p. 11
• Alignment datasets can contain preferences from groups that are not representative of the broader population.Source: Impact Statement · p. 11
• KTO may propagate dataset biases, favor majority-preferred outcomes when feedback conflicts, and homogenize preferences when deployed at scale.Source: Impact Statement · p. 11
• Binary feedback is more abundant, cheaper, and faster to collect, potentially enabling feedback from under-represented groups and user-specific models.Source: Impact Statement · p. 11

LLM Alignment

• DPO and other closed-form losses operate directly on offline preferences, unlike conventional preference-based RL, which learns rewards before fitting a policy.Source: A. Related Work · p. 14

Binary Feedback

• Binary-feedback alignment includes unlikelihood training, control-token methods, sparse-feedback learning in retrieval, and contrastive grounding for retrieval-augmented generation.Source: A. Related Work · p. 14

Online Alignment

• Online self-training and self-play continually align policies using filtered data sampled from the policies themselves.Source: A. Related Work · p. 14

Prospect Theory

• Prospect theory has had limited influence in machine learning, with prior work concentrated in human-robot interaction.Source: A. Related Work · p. 14

B. Proofs

• DPO and PPO-Clip are human-aware loss functions.Source: B. Proofs · pp. 14–15
• DPO is a HALO because its expected preference loss satisfies the required invariance condition.Source: B. Proofs · p. 15
• PPO-Clip is a HALO when token-level advantages and probability ratios are treated as fixed feedback data.Source: B. Proofs · p. 15
• KTO pushes probabilities up for desirable outputs and down for undesirable outputs, while its parameter gradient vanishes as the reference-adjusted reward diverges.Source: B. Proofs · p. 16
• Reward-equivalent functions can induce the same optimal policy and Bradley–Terry preferences but different human-value distributions when the reference point is fixed.Source: B. Proofs · p. 16
• With contradictory preferences, DPO can favor the minority-preferred output under a reference-dependent condition, whereas loss-neutral KTO increases the majority-preferred output’s relative likelihood.Source: B. Proofs · pp. 16–17

C. Implementations

• DPO uses the original implementation with β = 0.1, while SLiC applies cross-entropy to SFT data instead of sampling from the reference model.Source: C. Implementations · p. 18
• CSFT distinguishes desirable and undesirable outputs with separate good and bad control tokens.Source: C. Implementations · p. 18
• KTO uses β = 0.1, typically sets λD = λU = 1, and adjusts λ values for dataset imbalance.Source: C. Implementations · p. 18
• Offline PPO-Clip training is stabilized by keeping the reference fixed, using wider asymmetric clipping, adding a token-level KL estimate, and detaching the value head.Source: C. Implementations · p. 18

D. Human Evaluation

• The evaluation used 214 human comparisons per method after skipping questions requiring specific domain experience.Source: D. Human Evaluation · p. 18
• 72.9% ± 5.3 KTO versus 62.1% ± 5.7 DPO winrates over SFT targets were observed in human evaluation.Source: D. Human Evaluation · p. 18
• The KTO–DPO difference was significant with human evaluation at p < 0.05 but not with GPT-4-based evaluation.Source: D. Human Evaluation · p. 18

E. Additional Experiments

• KTO trained only on desirable examples yields results similar to the comparison reported in Table 4.Source: Table 4 · p. 19
• KTO outperforms SFT- and DPO-aligned models at all temperatures under the reported LM-judge evaluation.Source: Figure 6 · p. 19, E. Additional Experiments · p. 19
• UltraFeedback experiments compare KTO variants differing in output sampling, reference point, value-function shape, reference model, and risk neutrality.Source: Table 5 · p. 20
• In a show-recommendation example, SFT+KTO is succinct and provides multiple options, unlike the less helpful or hallucinated comparison outputs.Source: Table 6 · p. 21