Source-linked AI summary
Configurable Preference Tuning with Rubric-Guided Synthetic Data
Víctor Gallego
TL;DR
Static preference models limit how easily language models can reflect diverse, context-dependent human preferences. CPT uses rubric- and score-conditioned synthetic preference data with system prompts, then applies DPO so models can change behavior at inference time without retraining. Experiments report stronger configuration adherence than baseline models, while deployment faces scalability, teacher-bias, and misuse concerns.
Problem
Existing alignment methods often encode a singular, static preference profile, limiting adaptation to diverse and context-dependent human preferences.
Source: §1 · p. 1Method
CPT generates rubric- and score-conditioned preference data with concise system prompts and fine-tunes a student LLM using a DPO objective.
Source: §1 · p. 1, §2 · p. 2, §2.1 · p. 3Results
CPT-distilled models showed substantially higher target-bin accuracy and stronger rank correlations than baseline counterparts across various base models.
Source: §4 · p. 5Takeaways & Limitations
CPT provides fine-grained, human-interpretable control over LLM behavior through explicit system-prompted configurations without retraining for each directive.
Source: §1 · p. 1, §4 · p. 5Scaling CPT may be resource-intensive, and capable teacher models can introduce biases that propagate through the synthetic dataset into the student model.
Source: Impact Statement · p. 5
Abstract
from arXiv · showhide
Models of human feedback for AI alignment, such as those underpinning Direct Preference Optimization (DPO), often bake in a singular, static set of preferences, limiting adaptability. This paper challenges the assumption of monolithic preferences by introducing Configurable Preference Tuning (CPT), a novel framework for endowing language models with the ability to dynamically adjust their behavior based on explicit, human-interpretable directives. CPT leverages synthetically generated preference data, conditioned on system prompts derived from structured, fine-grained rubrics that define desired attributes like writing style. By fine-tuning with these rubric-guided preferences, the LLM learns to modulate its outputs at inference time in response to the system prompt, without retraining. This approach not only offers fine-grained control but also provides a mechanism for modeling more nuanced and context-dependent human feedback. Several experimental artifacts, such as training code, generated datasets and fine-tuned models are released at https://github.com/vicgalle/configurable-preference-tuning
1. Introduction
Existing alignment methods often encode a singular, static preference profile, despite human preferences being dynamic, contextual, and multifaceted. CPT addresses this gap by using explicit rubric-guided system prompts to make preference control interpretable and configurable.
1. Introduction
- Human preferences vary across users, cultures, ethical contexts, and tasks, but existing feedback models often bake in an averaged, monolithic profile.Source: §1 · p. 1
- Because preferences are fixed during fine-tuning, changing style, safety settings, or user targeting typically requires resource-intensive retraining.Source: §1 · p. 1
- CPT conditions synthetically generated preference pairs on rubric-derived system prompts, enabling inference-time behavioral changes without retraining.Source: §1 · p. 1
- CPT’s stated contribution is more granular, transparent, and controllable alignment that can represent diverse behavioral configurations.Source: §1 · p. 1
1.1. Related Work
- Unlike methods that infer latent user representations or aggregate preferences, CPT uses explicit rubrics and target scores to specify stylistic dimensions and configurations.Source: §1.1 · p. 2
2. Configurable Preference Tuning
CPT constructs configurable preference data by pairing rubric- and score-conditioned teacher outputs with concise system prompts, then trains a student with DPO. Reversing the preferred response under different prompts teaches the model to switch its preference according to the requested configuration.
2. Configurable Preference Tuning
- CPT learns a preference model p(yw ≻ yl|x, s), conditioning winner–loser preferences on a system prompt rather than only the user prompt.Source: §2 · p. 2
2.1. Synthetic Preference Data Generation
- Rubrics define axes such as formality, creativity, safety, or persona adherence, while teacher models generate responses at different target score levels.Source: §2.1 · p. 2
- For each rubric and score, the teacher summarizes the desired behavior into a concise two- to three-sentence system prompt.Source: §2.1 · p. 3
- The same response pair is labeled in opposite directions under two system prompts, teaching the student to switch preferences according to the requested configuration.Source: §2.1 · p. 3
2.2. Illustrative Example: Stylistic Control
- In the stylistic example, one prompt favors fragmented, absurd writing while another favors clear, conventional prose, producing reversed preference pairs.Source: §2.2 · p. 3
2.3. Training with DPO
- DPO fine-tuning increases the likelihood of the prompt-appropriate preferred response and decreases that of the rejected response, making rubric-guided behavior controllable via the system prompt.Source: §2.3 · p. 3
3. Experiments
Experiments test whether rubric-conditioned synthetic preferences produce controllable model behavior and improve downstream generation. Results show strong rubric alignment after CPT fine-tuning and better Best-of-N efficiency.
3. Experiments
- The experiments used four fine-grained rubrics with three score targets, producing a synthetic preference dataset of 900 samples.The evaluation prompted models with customized system prompts and used an LLM judge to score outputs against the rubric.Source: §3 · p. 3, §3.2 · p. 4
3.1. Rubric-Conditioned Generation Quality.
- Teacher models generated rubric-conforming responses near target levels, including average scores of 96.3 and 97.9 for extremely high targets and 14.1 and 23.1 for low targets.Unconditioned prompting produced moderately high but less sharply targeted scores.Source: §3.1 · p. 4, Table 2 · p. 4
3.2. Fine-tuning experiments with DPO
- Accuracy measures whether judge scores fall into the system prompt’s target qualifier bin, while Kendall’s τ and Spearman’s ρ measure ordinal agreement with low, moderate, and high targets.These metrics compare continuous judge scores with the intended qualifier categories.Source: §3.2 · p. 4
- CPT fine-tuning improved Mistral-Nemo-12B from 0.60 to 0.83 accuracy, 0.62 to 0.81 Kendall’s τ, and 0.74 to 0.93 Spearman’s ρ.Similar substantial gains were observed for Rocinante-12B, Qwen3-4B, Mistral-Small-24B, and Phi-4-14B.Source: §3.2 · p. 4, Table 3 · p. 5
3.3. Comparison to Best-of-N sampling
- CPT-tuned models consistently achieved higher Best-of-N scores than the baseline for each sample count and reached target quality with significantly fewer samples.The comparison used baseline and CPT-tuned Mistral-Nemo-12B models, with an LLM judge selecting the highest-scoring response.Source: §3.3 · pp. 4–5, Figure 1 · p. 5
4. Conclusions and Further Work
CPT uses rubric-guided synthetic preferences and DPO fine-tuning to let student LLMs adjust behavior from system prompts without retraining. Experiments show improved configuration adherence and better Best-of-N sampling efficiency, while future work targets compositional control and broader scalability.
4. Conclusions and Further Work
- CPT fine-tunes student LLMs with DPO on rubric-guided preferences so system prompts can change output configurations without retraining.The prompts encode desired attributes and target score levels through structured rubrics.Source: §4 · p. 5
- CPT-distilled models matched target quality bins more accurately and achieved stronger rank correlations than baseline models across varied base models.These results indicate improved adherence to diverse system-prompted configurations.Source: §4 · p. 5
- CPT also improved Best-of-N sampling by providing a better initial response distribution, producing higher-quality outputs with fewer samples.Source: §4 · p. 5
- Future work could support compositional control over multiple attributes and automate rubric or system-prompt generation to improve scalability.The paper also identifies extension to other domains and modalities as an open direction.Source: §4 · p. 5
Impact Statement
Configurable control may improve personalization and context-specific responses, but it raises risks around misuse, scalability, teacher-model bias, and transfer across domains or cultural contexts. Responsible deployment therefore requires safeguards.
Impact Statement
- Dynamic control over safety and style attributes creates potential societal-impact and misuse risks that require careful consideration.Source: Impact Statement · p. 5
- Creating detailed rubrics and validating synthetic data may become resource-intensive when deploying CPT at real-world scale.Source: Impact Statement · p. 5
- Teacher-model biases can propagate through synthetic preference data and influence the final student model’s behavior.Source: Impact Statement · p. 5
- Synthetic preference quality and diversity depend on teacher models’ ability to follow rubric instructions, potentially limiting effectiveness in unforeseen domains or cultural contexts.Source: Impact Statement · pp. 5–6
- Robust safeguards and responsible development and deployment practices are presented as necessary for mitigating risks from configurable AI systems.Source: Impact Statement · p. 6
A. Sample data: rubric tables and system prompts
The sample-data appendix presents rubric examples for unconventional and ornate styles alongside generated system prompts tied to rubric scores.
A. Sample data: rubric tables and system prompts
- Table 4 exemplifies a rubric targeting an unconventional and absurdist writing style.Source: Table 4 · p. 7
- Table 5 exemplifies a rubric targeting an ornate and baroque writing style.Source: Table 5 · p. 8
- Table 6 provides examples of generated system prompts for rubric R and a specified score.Source: Table 6 · p. 9