Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment

• Noisy and inconsistent preference labels can degrade the quality and robustness of aligned models.Source: §1 · p. 1
• Existing uncertainty-aware methods weight preference-level uncertainty but overlook the intrinsic reliability of the individual answers being compared.Source: §1 · p. 1
• CFA constructs conformal prediction sets with controllable coverage, aggregates answer-set memberships into a preference weight, and applies it to PPO- and DPO-style alignment.Higher-coverage sets are considered more reliable.Source: §1 · pp. 1–2
• Experiments report that CFA improves alignment performance and data efficiency, outperforming standard baselines and offering comparable performance to methods modeling preference-level uncertainty.Source: §1 · p. 2

2.1 LLM Alignment

• PPO-style RLHF trains a reward model from pairwise comparisons and then optimizes a policy using that model's reward.DPO instead directly optimizes model parameters from pairwise preferences without reward-model training; extensions target stability, adaptiveness, or data efficiency.Source: §2.1 · p. 2
• AI-generated preference data can reduce labeling costs, but its noise has prompted methods that weight preference pairs without modeling response reliability.Source: §2.1 · p. 2

2.2 Uncertainty Quantification

• Conformal Prediction offers a non-parametric, distribution-free way to construct confidence sets with formal guarantees, including extensions for white-box and black-box LLM settings.Source: §2.2 · p. 2
• Despite work on token- and output-level uncertainty and conformal prediction for LLMs, uncertainty quantification is rarely integrated into preference learning or alignment.Source: §2.2 · p. 2

3.1 Preference-Based Alignment

• Preference-based alignment compares preferred and dispreferred outputs for a prompt; PPO-style methods train a reward model from these pairs and use it to optimize a policy.Source: §3.1 · pp. 2–3
• In the PPO-style reward-model loss, σ is the sigmoid function.Source: §3.1 · p. 3
• DPO directly optimizes the policy without a reward model, using a temperature parameter and a frozen reference policy, typically the supervised fine-tuned model.Source: §3.1 · p. 3

3.2 Conformal Prediction

• Conformal Prediction scores each output's atypicality; calibration examples determine a quantile threshold for the desired coverage, and the resulting set has a stated coverage guarantee.Lower nonconformity scores indicate more typical or confident outputs.Source: §3.2 · p. 3
• In white-box settings, token-level log probabilities support a negative-log-likelihood nonconformity score, where a higher-probability output receives a lower score.Source: §3.2 · p. 3
• In black-box settings, the nonconformity score uses sample frequency, normalized entropy, and similarity to the most frequent response; frequent, similar outputs receive lower scores.Source: §3.2 · pp. 3–4

4 Method

• CFA aims to improve alignment robustness by using answer reliability estimated through conformal prediction.Its overview describes answer-side uncertainty as the basis for weighting alignment signals.Source: §4 · p. 4, §4.1 · p. 4

4.1 Overview

• For each preference pair, CFA derives a score from the conformal-set memberships of its two responses and uses it to weight policy optimization.The score is intended to prioritize feedback based on reliable answers and reduce the effect of less reliable comparisons.Source: §4.1 · p. 4

4.2 Set-wise Uncertainty Estimation via CP

• When both responses belong to set A, CFA uses q_a; when both belong to set B, it uses q_b; for different sets, it averages the quantiles.The paper describes lower confidence for uncertain sets or mismatched memberships.Source: §4.2 · p. 4

4.3 Uncertainty-Aware Alignment

• CFA scales preference contributions by estimated reliability, applying weights during PPO-style reward-model training and to the DPO-style preference loss.The two methods differ in where the weights enter: reward-model training for PPO and the log-sigmoid term for DPO.Source: §4.3 · p. 4

4.4 Algorithm Description

• Each iteration collects preference pairs, estimates their uncertainty with CP, forms an uncertainty-augmented dataset, and updates the policy using PPO- or DPO-style training.The algorithm begins by initializing the policy and reference policy and calibrating nonconformity-score thresholds.Source: §4.4 · p. 5, §4.3 · p. 5

5.1 Experimental Setup

• Experiments cover Llama2-7B, Llama3.1-8B, and Qwen models across WebGPT, Pairwise, and Summarize, using GPT-4o to average four evaluation scores.The scores assess accuracy, relevance, completeness, and expression; the study also reports augmented dataset sizes by model.Source: §5.1 · pp. 5–6, Table 2 · p. 6

5.2 Comparison with Baselines (RQ1)

• CFA consistently outperforms standard alignment baselines; on Llama2-7B with Summarize, its gain is +1.62 versus +1.77 for base alignment over SFT.The comparison covers three language models and three benchmark datasets; DPO-style CFA yields greater gains than PPO-style CFA.Source: Table 1 · p. 6, §5.2 · p. 7

5.3 Scalability and Efficiency Analysis (RQ2)

• On the Summarize dataset, CFA with DPO works stably across different model sizes, and performance gains are larger for smaller models.Source: §5.3 · p. 7
• On Llama2-7B and Summarize, CFA outperforms base DPO at every training-data proportion, including 5%, and its gain grows with more samples.The tested proportions are 5%, 10%, 20%, 35%, and 50%.Source: §5.3 · p. 7, Figure 3 · p. 7

5.4 Ablation Study and Sensitivity Analysis (RQ3)

• On Llama2-7B and Summarize, CFA outperforms WPO and IW-DPO with different uncertainty signals; Direct Ask weighting performs worse than base DPO.The compared signals include Direct Ask and p(true).Source: §5.4 · pp. 7–8, Figure 4 · p. 7
• White-box CP could improve performance over black-box CP, and CFA shows an advantage over base DPO in win-rate comparisons.The white-box comparison uses Llama2-7B on Summarize; win rates are judged by GPT-4o.Source: §5.4 · p. 8, Figure 7 · p. 8, Table 3 · p. 8
• Coverage α2 of 0.7–0.8 gives the best overall performance, while CFA consistently outperforms base DPO across tested α2 values.At α2 = 0.6, performance is worst because α2 is close to α1 = 0.5, reducing the effect of CFA weights.Source: §5.4 · pp. 8–9, Figure 8 · p. 9

5.5 Case Analysis

• In a summarization case, CFA corrects Base DPO’s misreading of advertisers and earns accuracy and relevance scores of 8 and 9.The SFT output is repetitive and uninformative, scoring 5.5 out of 10 overall.Source: Figure 6 · p. 8, §5.5 · p. 9

6 Conclusion

• CFA improves LLM alignment by using the reliability of responses.The conclusion describes this as the framework’s central contribution.Source: §6 · p. 9
• CFA provides reward-model-based and reward-free training paradigms under white-box and black-box settings.Source: §6 · p. 9
• Empirical results indicate that answer confidence and reliability enhance alignment quality and scale well with limited data and smaller models.Source: §6 · p. 9
• The authors identify multimodal feedback and human-in-the-loop calibration under real-world constraints as future research directions.Source: §6 · p. 9

Limitation

• Small calibration sets may produce inaccurate reliability estimates.Source: Limitation · p. 9
• The reported experiments cover text-based models and datasets, not multimodal large language models.Source: Limitation · p. 9
• CFA does not combine answer reliability with preference uncertainty.Source: Limitation · p. 9