QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA

Motivation

• Scalar RLHF rewards entangle factuality, tone, safety, and other criteria, obscuring why responses fail and preventing targeted improvement.Composite scores can also encourage excessive refusal of benign requests by prioritizing harmlessness over helpfulness.Source: §1 · p. 1

Contribution

• Unlike RLHF and RLAIF, which optimize monolithic human or AI judgments, QA-LIGN preserves independent alignment objectives as transparent, fine-grained signals.The decomposed feedback integrates into standard reinforcement-learning pipelines without additional optimization stages.Source: §1 · pp. 1–2, Figure 2 · p. 2

Approach

• QA-LIGN replaces black-box rewards with hierarchical natural-language programs that independently evaluate Helpful­ness, Honesty, and Harmlessness through fine-grained checks.The framework’s evaluation questions are positively framed, so True indicates ideal behavior for a specific query.Source: §1 · p. 1, Figure 1 · p. 1
• A strong LLM expands constitutional principles into evaluation programs, and the same rubric scores drafts and revisions during supervised self-critique training and GRPO.After supervised fine-tuning on 500 examples, vectorized multi-axis rewards drive policy updates along identifiable dimensions.Source: §1 · pp. 1–2

Results

• 68.7% lower attack success rates with 0.7% false refusals establish QA-LIGN’s Pareto-optimal safety-helpfulness result against DPO and GRPO with state-of-the-art reward models.The comparison uses equivalent training steps, while reasoning-task accuracy on GSM8K, CSQA, and ARC-Challenge is preserved.Source: §1 · p. 2

Preference-Based and Constitutional Alignment

• QA-LIGN preserves constitutional principles as distinct reward dimensions, unlike Constitutional AI’s single judgment that combines all constitutional rules.This aims to retain constitutional interpretability while supporting preference-based training effectiveness.Source: §2 · p. 2

LLM-as-Judge and Critique-Augmented Reward Modeling

• LLM-as-judge methods improve alignment feedback with critiques, while QA-LIGN adds symbolic yes/no or graded sub-questions for clearer specifications and finer policy-optimization signals.The structured interface is described as reducing reward hacking by decomposing each principle into concrete checks.Source: §2 · pp. 2–3

Rule-Based and Verifiable Rewards

• Rule-based and verifiable-reward methods eliminate human judgment when outcomes are programmatically checkable, but ethical alignment requires structured oversight for open-ended dialogue.QA-LIGN bridges learned and rule-based approaches with LLM-answered Q&A probes as principle-specific verifiers.Source: §2 · p. 3
• Rubric-based reward systems and checklist feedback make subjective criteria interpretable, whereas QA-LIGN specifically targets constitutional safety and helpfulness through hierarchical decomposition.The approach complements concurrent rubric methods by focusing on safety-helpfulness alignment.Source: §2 · p. 3

Inference-time Scaling via Self-Reflection

• Inference-time methods such as Self-Refine and Reflexion improve outputs through repeated critique or memory, while QA-LIGN embeds principle-specific probes in both the critique loop and training reward.This yields transparent, decomposed feedback during optimization rather than only at generation time.Source: §2 · p. 3

Framework overview

• QA-LIGN replaces conventional scalar reward models with a unified pipeline whose symbolic programs guide reflection targets, evaluation, and GRPO updates.The same declarative rubric supplies feedback during supervised priming and reinforcement learning, avoiding black-box reward models.Source: §3 · p. 3

Stage 1: Program Generation

• The framework decomposes helpfulness, honesty, and harmlessness into dimensions containing natural-language binary checks and categorical grades.Questions target fine-grained properties such as harmful activity, factual correctness, uncertainty, and relevance, with irrelevant checks gated out.Source: §3 · p. 4
• The evaluation program spans 3 principles, 40 dimensions, and 167 questions, including 42 binary gates that control downstream A–F grading.Strong language models draft the questions, which are lightly edited or deduplicated before remaining fixed during training.Source: Figure 4 · p. 4, §3 · p. 4

Stage 2: Think SFT Priming

• Stage-2 SFT uses 500 automatically generated draft–reflection–revision examples to teach a two-pass workflow before reinforcement learning.The policy learns to produce structured reflections and revisions even though the resulting model remains largely unsafe at this stage.Source: §3 · pp. 4–5
• The draft–reflect–revise pipeline approximately doubles response length across 520 AdvBench samples, providing more computation for safety reasoning.The paper connects this increased inference-time computation with iterative decomposition and refinement.Source: Table 1 · p. 5, §3 · p. 5

Stage 3: Symbolic-Reward RL

• During GRPO, QA-LIGN scores each draft and revision across all questions, then converts hierarchical principle evaluations into rewards.GRPO samples candidate answers and normalizes their raw rewards into group-relative advantages.Source: §3 · p. 5

Hierarchical Reward Aggregation

• Within each principle, failed binary gates assign a dimension score of −1; otherwise, the dimension receives its minimum grade, and dimension scores are averaged per principle.The three principle scores cover Helpfulness, Honesty, and Harmlessness, while the safety-first constraint keeps harmful responses negative regardless of other qualities.Source: §3 · p. 5

Self-Correction Incentive

• The final reward combines absolute draft and revision quality with an improvement bonus, rewarding beneficial revisions and penalizing degraded responses.This objective encourages genuine self-correction rather than superficial editing.Source: §3 · pp. 5–6

4.1 Training Setup

• QA-LIGN is trained from an uncensored Llama-3.1-8B model with a fixed uncensored judge and symbolic programs, using 100 GRPO updates.Training uses 1,600 harmful prompts, with a disjoint 500-prompt set for supervised self-reflection examples.Source: §4.1 · p. 6

4.2 Baselines

• DPO baselines use matched WildJailbreak prompt pairs, while GRPO baselines use URM and Skywork rewards with identical training budgets and hyperparameters.The comparison includes DPO variants trained for 100 and 800 steps.Source: §4.2 · p. 6
• QA-LIGN’s ablation without draft-reflect-revise reaches 23% weighted FRR, while the main method maintains 0.67% FRR in the reported comparison.Table 2 compares attack success rates among models trained on 1,600 harmful prompts with equivalent 100-step budgets.Source: §4.2 · p. 6, Table 2 · p. 7

4.3 Evaluation Benchmarks

• Safety evaluation covers six static suites and HARMBENCH attacks, with ASR scored by LlamaGuard after the model’s reasoning or revision output.The HARMBENCH evaluation uses seven attack methods against Llama-3.1-8B-Instruct.Source: §4.3 · pp. 6–7
• False-refusal evaluation measures declines on safe SGX and OR-Bench prompts, where lower FRR indicates better helpfulness without relaxing safety constraints.SGX is evaluated in full, while 100 OR-Bench prompts are sampled across the original category distribution.Source: §4.3 · p. 7
• General-capability retention is measured on GSM8K, CSQA, and ARC-Challenge against uncensored and aligned baselines.The alignment-tax evaluation targets possible losses in reasoning, question answering, and related capabilities after safety fine-tuning.Source: §4.3 · pp. 7–8

5.1 Robust Safety

• 26.3% ASR on Generic Safety benchmarks beats DPO at 61.4%, GRPO-Skywork at 41.9%, and GRPO-URM at 42.8% under equivalent compute.On HarmBench, QA-LIGN reaches 50.9% ASR versus 66.6%, 54.5%, and 55.1% for those baselines, respectively.Source: §5.1 · p. 8

5.2 False Refusals

• 0.67% FRR for QA-LIGN and URM compares with 1.00% for Skywork, 4.8% for DPO-100, and 6.5% for DPO-800.The reported improvement is associated with the shared draft-reflect pipeline rather than conservative default refusals.Source: §5.2 · p. 8
• QA-LIGN records 26.3% Generic Safety ASR versus 42.8% for GRPO-URM and 41.9% for GRPO-Skywork while maintaining similarly low false refusals.The authors attribute this combination to independently evaluating Helpfulness, Honesty, and Harmlessness before aggregation.Source: §5.2 · p. 9

5.3 Mitigated Alignment Tax

• QA-LIGN matches or exceeds the uncensored and DPO-aligned models on GSM8K, ARC-Challenge, and CSQA.These results are presented as preserving problem-solving and knowledge-retrieval abilities during safety alignment.Source: §5.3 · p. 9
• GSM8K accuracy rises 4.09% over the baseline, nearly 2% above DPO-100, and remains within 0.15% of DPO-800.Comparable gains are reported on CSQA and ARC-Challenge.Source: §5.3 · p. 9

5.4 Human Evaluation

• 98.2% of 50 self-corrections were judged genuine improvements, while harmful outputs fell from 98% of drafts to at most 2% of continuations.Inter-rater agreement was 96–98%.Source: §5.4 · p. 9

6 Conclusion

• QA-LIGN decomposes feedback into principle-specific checks for Helpfulness, Honesty, and Harmlessness, making reward signals transparent and modular.The same structured evaluation is embedded in the training process through a draft→reflect→revise pipeline.Source: §6 · p. 9
• 68.7% lower attack success rates with low false refusal rates establish QA-LIGN’s Pareto-optimal safety-helpfulness result and outperforming alignment baselines.The reported comparison includes DPO and GRPO with state-of-the-art reward models.Source: §6 · p. 9
• Transparent, question-level reward decomposition provides traceable credit assignment and reduces objective entanglement across multiple alignment criteria.Interpretability is built into the objective rather than delegated to a single opaque training signal.Source: §6 · p. 9

7 Limitations

• P × G × B LLM calls per update create substantial computational overhead that slows training and hinders scaling to larger datasets or longer dialogues.The approach may require batching, caching, smaller judge models, and dedicated GPUs for reward-script execution.Source: §7 · p. 10
• LLM-as-Judge consistency, alignment, and susceptibility to reward hacking constrain the reliability of the learned reward signal.Decomposing rewards into P = 167 independent checks substantially mitigates, but does not eliminate, successful exploitation risk.Source: §7 · p. 10
• Harmful or misaligned draft content persisted after 100 training steps, so longer runs are needed to improve the initial drafts that revisions depend on.The pipeline trains both drafts and revisions through QA-LIGN rewards, making draft quality relevant to later revision quality.Source: §7 · p. 10
• Fixed symbolic QA programs can miss problematic behaviors that fall outside their predefined questions.The rubric’s interpretability therefore comes with rigidity and depends on the breadth of the questions selected.Source: §7 · p. 10

Ethics Statement

• Proprietary LLM use in the study adhered to the providers’ usage policies.Source: Ethics Statement · p. 10
• AI assistants were used to correct grammatical errors and rephrase sentences.Source: Ethics Statement · p. 10
• The ethics statement reports language-editing assistance rather than assistance with the study’s substantive research claims.Source: Ethics Statement · p. 10

8 Human Subjects and Annotators

• Two trained lab raters evaluated 50 AdvBench draft→reflect pairs using public prompts and model-generated text without collecting personal or sensitive data.They received written instructions, could skip unsettling items without penalty, and could provide optional rationales.Source: §8 · p. 10
• 98.2% of self-corrections were judged genuine improvements, harmfulness fell from 98% in drafts to at most 2% in continuations, and agreement reached 96–98%.The raters were graduate ML/NLP researchers in the United States, and no crowdsourcing or extra compensation was used.Source: §8 · p. 10

Prompts

• The general evaluator prompt requests a one-sentence reasoning field and a score of TRUE, FALSE, or NA, or a categorical letter grade when the criterion is not binary.Source: Prompts · p. 15
• The judge-LLM injection supplies the assistant’s conversation, evaluation results, principle-score vector, and lowest-scoring dimensions as inputs for self-reflection.Source: Prompts · p. 15
• The self-reflection prompt restricts responses to one or two first-person sentences that identify a specific weakness and propose a concrete improvement.It also prohibits mentioning dimension names, scores, or evaluation metrics.Source: Prompts · p. 15
• The refusal-versus-engagement judge must always classify a response as either refusal or engagement and return a JSON judgment with a brief reason.Engagement includes partial, hedged, or warned responses.Source: Prompts · p. 15

• Table 3 provides the sample counts for each safety evaluation dataset.Source: Table 3 · p. 15

• All models use a maximum sequence length of 1,536 tokens and BFloat16 mixed precision, with training conducted on 2–8 H200 GPUs.Source: Training and Inference Details · p. 15
• Table 4 compares base instruction-tuned models with DPO-aligned variants across general, safety, truthfulness, reasoning, and coding benchmarks.Reported metrics include accuracy, strict match, exact match, and pass@1, with 95% confidence intervals.Source: Table 4 · p. 16
• Table 5 reports wall-clock training time and GPU utilization for each method.Source: Table 5 · p. 16
• QA-LIGN GRPO uses a 1×10^-6 learning rate, 0.001 KL coefficient, five rollouts per update, and batch size 16 for 100 steps.Training takes around 8–12 hours on eight H200 GPUs, with most time spent on program execution and question-level checks.Source: Training and Inference Details · p. 16
• The judge runs asynchronously at temperature 0 while concurrent HTTP requests are issued during GRPO rollouts.This setup is intended to minimize idle GPU time.Source: Training and Inference Details · p. 16
• The constitutional program batches questions and applies the rubric structure during post-processing to avoid sequential VLLM calls.Source: Training and Inference Details · p. 16

HHH Offline Evaluation

• 94.57% strict accuracy and 96.38% non-strict accuracy are achieved on HHH preference pairs.Strict accuracy requires the chosen completion to score higher, whereas non-strict accuracy also counts ties.Source: HHH Offline Evaluation · p. 16
• QA-LIGN produces continuous ratings that can serve as scalar rewards for RL or be converted into labels for preference-based algorithms.Source: HHH Offline Evaluation · p. 16

QA-LIGN integrated into DPO

• Offset DPO incorporates the magnitude of chosen–rejected label differences from QA-LIGN preferences when training safety-aligned base models.Source: QA-LIGN integrated into DPO · p. 17

B Algorithms

• The generated program contains three principles, 40 dimensions, and 167 questions, combining binary hard-violation gates with A–F graded quality questions.The principles are harmlessness, honesty, and helpfulness.Source: B Algorithms · p. 18
• Symbolic-reward RL samples groups of five full responses containing a draft, a Think reflection, and a revision before evaluating the revision.Source: B Algorithms · p. 19

Evaluation Dimensions by Principle

• The rubric spans dimensions such as physical-harm avoidance, truthfulness, transparency, relevance, clarity, and safer alternatives.Its sample questions use binary checks and graded assessments for harmful details, uncertainty, ambiguity, and helpful redirection.Source: Evaluation Dimensions by Principle · p. 20, Sample Evaluation Questions (QA-LIGN… · p. 21

Sample Evaluation Questions (QA-LIGN blocks)

• A subliminal-messaging example shows the draft–reflection–revision trace turning an unsafe draft into a refusal with legitimate alternatives.Source: Figure 8 · p. 22