PaperlayerView exact revision on arXiv

Source-linked AI summary

Consistency Is Not Validity: Self-Rewarded Moral Argumentation Makes Language Models Decisive but Pliable

Marius De Kuthy Meurers, Aaron Parisi, Bernd Bohnet

arXiv:2610.30349v1cs.CLcs.AIcs.LG

TL;DR

The paper asks whether rewarding better arguments can improve moral judgment, and trains a policy to judge its own arguments on verified literary dilemmas. The policy becomes more decisive and less sensitive to option order, but these consistency gains do not improve moral validity.

  • Problem

    LLMs are increasingly used for moral judgment, yet machine-ethics research predominantly treats moral reasoning as discrete label prediction.

    Source: §1 · p. 1

  • Method

    The authors train a Gemma-4 4B policy with masked thinking on literature dilemmas verified by an SMT solver, using the policy’s own pairwise judgments of argument quality as rewards.

    Source: §3.2 · p. 2, §8 · p. 8

  • Results

    The policy adopts a prescriptive register and halves presentation flips on ambiguous MoralChoice scenarios, while sycophancy remains flat, framing acceptance rises, and MORABLES accuracy falls.

    Source: §8 · p. 8

  • Takeaways & Limitations

    Consistency under reordering can be optimized without improving the moral judgment it is meant to signal, so consistency and validity should be evaluated separately.

    Source: §8 · p. 8

    The judge is validated only on human-written arguments, and neither judge scoring the remaining ELEPHANT metrics is validated against human annotations.

    Source: Limitations · p. 8

Abstract

from arXiv · show

LLMs are increasingly consulted for moral judgment. We aim to improve performance on such tasks by rewarding good arguments, on the premise that arguing well increases the likelihood of judging well. We train a reinforcement learning policy initialized from a state-of-the-art language model in a self-judged tournament with masked thinking: the generated arguments are judged pairwise on argument quality by the policy itself. As an empirical basis, we extract authentic dilemmas from classic novels, using a logic solver to verify that each constitutes a conflict. We show that our judge rates argument quality like human annotators, and the policy then adopts the language register that it rewards. Testing on the external benchmarks MoralChoice, ELEPHANT, and MORABLES, we find that the policy becomes more decisive and invariant to the order in which options are presented, but sycophancy and reframing remain a challenge to validity. Two ablation studies with synthetic dilemmas and unmasked thinking match the reordering gains, but show a weaker register shift and higher sycophancy. We conclude that consistency is not validity: consistency under reordering can be optimized without improving the moral judgment it is meant to signal.

1. Introduction

The paper asks whether rewarding argument quality can improve moral judgment, addressing a field that often treats moral reasoning as discrete label prediction. It trains a policy to argue and judge dilemmas, then finds that decisiveness and order consistency improve without corresponding gains in moral validity.

1. Introduction

  • The paper tests whether rewarding argument quality improves moral judgment, given that discrete-label evaluation can miss the weighing of conflicting duties and trade-offs.The authors identify subjective justifications, limited resolution in scalar rewards, and distributional-tail loss from model-generated data as obstacles.Source: §1 · p. 1
  • The policy generates competing arguments under assigned ethical frameworks and judges them pairwise across a multi-dimensional quality taxonomy to produce reinforcement-learning rewards.Masked thinking restricts policy gradients to visible answer tokens; literature dilemmas are checked with an SMT solver for conflicting duties.Source: §1 · p. 1
  • The evaluation tracks whether the reward selects a linguistic register and whether that register transfers to held-out moral judgment and decision consistency.Ablations examine how dilemma source and masked thinking shape those effects.Source: §1 · pp. 1–2
  • The judge rates argument quality like human annotators, while training selects a decisive, prescriptive register that trades caveats and backward justifications for forward obligations.On held-out benchmarks, presentation flips halve, but sycophancy remains flat, framing acceptance rises, and accuracy falls; both ablations replicate the reordering gains.Source: §1 · p. 2

2. Related Work

The related work contrasts discrete moral classification with richer deliberative approaches, and motivates evaluating argument quality while addressing limitations of synthetic data and learned rewards.

2. Related Work

  • Most machine-ethics benchmarks use discrete labels over short vignettes, while narrative ethics treats judgment as weighing competing duties and virtues.Critics argue that closed classification flattens value pluralism and non-monotonic deliberation; moral evaluation has also been framed as requiring audits of argumentative traces.Source: §2 · p. 2
  • Newer benchmarks test contextual ambiguity, adversarial fables, and sycophancy, while models still disagree with humans and one another on everyday dilemmas.The literature contrasts synthetic data’s scalability with recursive-generation risks and describes canonical literature as a source of authentic interpersonal stakes.Source: §2 · p. 2
  • Argument-quality assessment spans logical, rhetorical, and dialectical dimensions, and pairwise tournament ranking avoids point-scale rating compression.Normative reasoning lacks rule-based verifiers, limiting earlier reinforcement learning to keyword heuristics and discrete rules.Source: §2 · p. 2
  • Self-rewarding and self-play offer alternatives for reward learning, but optimizing a learned reward invites reward hacking.Source: §2 · p. 2

3. Method

The method trains a policy on verified literary conflicts using a self-judged argument tournament, then converts tie-aware comparisons into policy advantages while restricting updates on thinking tokens.

3.1. Literature-Grounded Dilemmas

  • The authors formalize each dilemma’s premise and two duties as SMT constraints, retaining cases where either duty is individually satisfiable but both cannot be satisfied together.156 of 185 raw candidates pass the three verification criteria; each verified dilemma is paired with five ethical frameworks, yielding 780 prompts.Source: §3.1 · p. 2

3.2. The Self-Judged Tournament

  • For each prompt, the policy generates eight completions and judges them pairwise using its own shared, updating weights and a 15-dimension argument-quality taxonomy.The overall preference supplies the reward; three dimension preferences serve as diagnostics.Source: §3.2 · pp. 2–3
  • A full round-robin compares all completion pairs in both presentation orders, and reversed-order preference changes count as ties during training.This setup targets position bias while preserving gradient signal across evenly matched completions.Source: §3.2 · p. 3

3.3. Tie-Aware Policy Advantage

  • The tie-aware Rao–Kupper model estimates latent argument strengths, while a length penalty and RLOO advantages address verbosity and group-reward scaling.RLOO subtracts the leave-one-out group mean without variance scaling.Source: §3.3 · p. 3
  • Policy gradients are masked on thinking tokens, concentrating optimization on visible answers, and the full sequence is regularized against the base checkpoint with token-level KL divergence.Source: §3.3 · p. 3

4. Experimental Setup

The experiments compare the initial and trained policy against baselines, test two motivated ablations, evaluate three held-out benchmarks, and measure stylistic changes across training.

4.1. Checkpoints and Baselines

  • The study compares the starting Gemma-4 4B checkpoint with a two-epoch GRPO policy trained on 156 literature dilemmas, using Gemma-4 31B only as a reference judge.Leak-free four-shot in-context demonstrations test whether post-training acquires capabilities or activates latent behaviors; evaluations use uniform decoding with thinking enabled.Source: §4.1 · p. 3

4.2. Motivated Ablations

  • The Unmasked ablation applies gradients and length penalties to thinking tokens as well as answers, while Synthetic trains on dilemmas without literary grounding.The ablations test chain-of-thought monitorability and whether literary narratives drive the learned register.Source: §4.2 · p. 3

4.3. Held-Out Benchmarks

  • Held-out tests use MoralChoice for order-reversal robustness, ELEPHANT for indirectness, framing acceptance, and sycophancy, and MORABLES for allegorical accuracy.Presentation flips, moral sycophancy, and MORABLES accuracy need no judge; the evaluated model scores other ELEPHANT metrics, which are also rescored by the 31B judge.Source: §4.3 · p. 3

4.4. Measuring Register Change

  • Register analysis covers 101 features across layout, syntax, discourse relations, and stance, with replicated changes required across all three trained runs under q < .05 control.The study uses 753 paired epoch comparisons per group and estimates reward attribution as the regression slope of completion advantage on standardized feature value within each prompt group.Source: §4.4 · pp. 3–4
  • Figure 1 plots resolved accuracy and presentation flips, explicit ties, and pair density across the human preference gap for 4B Base, post-GRPO 4B, and 31B judges.Source: Figure 1 · p. 4

5. Judge Validation

The judges’ resolved choices track human preferences, especially when those preferences are strong. Model scale primarily improves consistency under option reordering, while training the 4B judge increases abstention.

5. Judge Validation

  • 55.0% (4B) and 57.1% (31B) resolved accuracy on near-ties rose to 93.3% and 95.0% when human preference was strongest.The 31B judge’s agreement with three experts fell within the inter-expert range.Source: §5 · p. 4
  • 78.5% (4B) versus 80.4% (31B) resolved accuracy accompanied a lower abstention rate at 31B, 10.7% versus 33.5%.Presentation flips were more common than explicit ties at both scales.Source: §5 · p. 4
  • After GRPO, the 4B judge’s human-winner selection rate remained similar, 82.2% versus 82.4%, while unresolved pairs rose to 47.3%.Unresolved pairs included 38.5% presentation flips and 8.8% explicit ties.Source: §5 · p. 4
  • The 4B judge abstained most on logic and least on rhetoric, while the 31B judge reversed that ordering.At both scales, the dimension with the highest abstention also had the highest resolved accuracy.Source: §5 · p. 4

6. Policy Results

The policy adopts a prescriptive, more decisive register, but held-out results do not show moral judgment improving in parallel. Both ablations retain reordering consistency while increasing sycophancy relative to GRPO.

6.1. Reward Selects a Prescriptive Register

  • Training shifted completions from exploratory analysis toward direct, categorical pronouncements, with necessity modals surging by 1.01 SD in GRPO.Possibility modals also rose, suggesting the more prescriptive register did not become dogmatic.Source: §6.1 · p. 5
  • Answers simplify structurally: bullets and bold replace hierarchical outlines, and maximum dependency depth falls by 0.18 SD net of sentence length.Across prompts, backward justifications and concessive connectives give way to forward obligations and inferential conjuncts.Source: §6.1 · p. 5
  • Reward alignment correlates with observed register change (ρ=0.61), but the shift is not an ethical doctrine and varies little in direction across frameworks.Feature identity accounts for 85% of variance across frameworks; 86% of features keep their sign under every framework.Source: §6.1 · p. 5

6.2. Consistent, yet Pliable under Reframing

  • 14.3% of high-ambiguity MoralChoice verdicts flipped under reversed presentation for Base, compared with 6.6% for GRPO; ELEPHANT indirectness fell from 25.5% to 6.2%.GRPO recognized ambiguity slightly more often than Base, 19.1% versus 16.0%.Source: §6.2 · pp. 5–6
  • 45.3% (Base) versus 44.0% (GRPO) moral sycophancy stayed nearly flat, framing acceptance rose from 5.2% to 7.5%, and MORABLES accuracy fell from 56.5% to 53.4%.A frozen 31B judge confirmed the framing-acceptance increase on the same completions.Source: §6.2 · p. 6, Figure 5 · p. 6

6.3. The Divergence Survives Both Ablations

  • Both ablations retain GRPO-like reordering consistency, with presentation-flip changes of only +0.7 and +1.0 points, but raise sycophancy by +3.7 and +4.6 points.The divergence recurred in all three runs; the ablation comparison used one seed per run.Source: §6.3 · p. 6, Figure 6 · p. 7
  • Masking reallocates words from thinking to visible answers: GRPO thinking fell from 392 to 265 words while answers grew from 780 to 1,078.Unmasked thinking grew 47%, while answer growth was 17%; masking reduced lexical overlap between thinking and answers by 35%.Source: §6.3 · p. 6
  • Unmasked recovered 2.1 MORABLES accuracy points over GRPO but remained below Base; Synthetic retained 69% of GRPO’s significant feature shifts at nearly full strength.Unmasked replicated 71% of those features at 0.71 of GRPO’s magnitude; on ELEPHANT it accepted framing more often than GRPO.Source: §6.3 · pp. 6–7

7. Discussion

The policy learns a prescriptive register that transfers as decisiveness, but its consistency does not establish better judgment. The discussion connects this result to judge preferences, masking’s effects on monitorability, and the limited role of literary grounding.

7. Discussion

  • The 4B policy’s prescriptive register transfers as decisiveness, not better judgment.The stated goal for machine ethics is normative competence, not persuasiveness.Source: §7 · p. 7

7.1. The Judge Selects the Register

  • On human-written arguments, the 4B judge’s overall preference agrees most often with rhetoric preference, its weakest predictor of human preference.On the policy’s own completions, reward tracks dialectic (mean Spearman ρ=0.94), then logic (0.68), then rhetoric (0.53); rhetoric favors shallower dependency trees (−0.09 SD per SD of maximum depth).Source: §7.1 · p. 7
  • Answers lengthen despite the leave-one-out length penalty because within-group normalization cannot penalize drift shared by all completions.The rewarded stance markers co-occur with longer answers, and masking leaves visible answers as the only channel for optimization pressure; response-level loss normalization may also favor length.Source: §7.1 · p. 7

7.2. Decisive but Pliable

  • The learned prescriptive register transfers as decisiveness rather than better judgment, while consistency under reordering and framing acceptance rise and sycophancy falls in no run.The same policy that halves presentation flips on MoralChoice flips more often as an IBM-ArgQ pairwise judge after training.Source: §7.2 · p. 7, §7 · p. 7

7.3. Masking Is a Lever with a Price

  • Without masking, gradients spread across the full sequence; under per-token loss normalization, this may weaken register acquisition and reduce degradation on MORABLES fables.The authors describe masking as a lever that concentrates optimization on the answer, with a possible cost to monitorability.Source: §7.3 · p. 8
  • Masking sends no policy gradient to thinking tokens, yet answers repeat less of the thinking, consistent with feedback spillover through shared weights.Lexical overlap is not faithfulness, so masking risks separating the stated justification from the verdict.Source: §7.3 · p. 8
  • Unmasked thinks longer than GRPO on ELEPHANT and accepts user framing more often, but longer traces correlate with framing acceptance only within GRPO.Without an external verifier, longer deliberation does not make the policy judge user premises independently.Source: §7.3 · p. 8

7.4. What Narrative Grounding Changes

  • Synthetic training preserves most of the prescriptive register, indicating that the reward rather than literary style accounts for it; narrative grounding reliably affects only directness.Synthetic training makes answers +5.6 points more indirect (95% CI [+3.7, +7.6]); verified literary conflicts ensure moral tension, but pairwise reinforcement learning distills a way of arguing rather than sound moral deliberation.Source: §7.4 · p. 8

8. Conclusion

The self-judged tournament produces a prescriptive register and more decisive responses, but these gains do not translate into more valid moral judgments. The authors conclude that consistency and validity should be evaluated separately.

8. Conclusion

  • The policy’s prescriptive register makes it decisive, halving presentation flips on ambiguous MoralChoice scenarios and making ELEPHANT replies more direct.The judge rates argument quality like human annotators, and the register appears across assigned ethical frameworks.Source: §8 · p. 8
  • On ELEPHANT posts, moral sycophancy remains flat while framing acceptance rises; on MORABLES fables, allegorical accuracy falls despite greater decisiveness.Both ablations match reordering gains but show weaker register shifts and higher moral sycophancy; consistency under reordering can be optimized without improving the judgment it is meant to signal.Source: §8 · p. 8

Limitations

The study’s limitations concern attribution of the register shift, validation of policy-argument judgments, experimental scope, and what formal verification can establish. In particular, the verification checks whether duties conflict, not how they should be prioritized.

Limitations

  • The shared-weight, self-judging design cannot separate register effects from self-judgment versus pairwise argument-quality rewards, and may favor familiar completions.The authors also have not tested whether framing acceptance rises with decisiveness during training.Source: Limitations · p. 8
  • Judges are validated on human-written arguments, not policy completions; ELEPHANT metric judges lack human validation, so the authors interpret direction rather than level.The evaluated model and frozen Gemma-4 31B judge agree weakly on individual items.Source: Limitations · p. 8
  • Evidence comes from one Gemma-4 4B policy scale, thinking-enabled training, one run per configuration, English dilemmas, and four nineteenth-century Western novels.Confidence intervals cover dilemma and benchmark-item variation, not variation across training runs.Source: Limitations · p. 8
  • Z3 verifies the Boolean encoding’s conflict, with every duty a hard constraint, but cannot represent which duty outweighs another.Resolution is left to the judge; weighted encodings such as MaxSMT soft constraints could represent duty ordering.Source: Limitations · pp. 8–9

Ethical Considerations

The paper flags risks from confident, pliable moral advice and limits its normative scope; its appendices document the dilemma pool, verification, training, judge validation, and framework-specific register shifts.

Ethical Considerations

  • Consistency checks may miss confident, pliable advice; the five Western frameworks and benchmark moral assumptions constrain what the verdicts represent.The authors also note that source novels may carry period views on class, gender, and religion.Source: Ethical Considerations · p. 9
  • No new human-subject data were collected, and public-domain training texts and licensed materials were used under their terms.Source: Ethical Considerations · p. 9

A.1. Literature Pool

  • 185 dilemmas were extracted from four PG19 novels, and 156 passed the Z3 gate for training; Les Misérables supplied the largest share.Compassion was the most frequent competing value-concept.Source: §A.1 · p. 13, Table 1 · p. 14

A.2. Z3 Gate Failures

  • 77.7% of first-batch synthetic dilemmas passed the Z3 gate; 15.4% were co-satisfiable, and training used gate passes from two batches.The table distinguishes genuine dilemmas, co-satisfiable duties, and degenerate encodings.Source: §A.2 · p. 14, Table 2 · p. 14

A.3. Literature vs. Synthetic Dilemmas

  • Literature dilemmas had higher effective rank on all three embedders, but the two pools’ jackknife confidence intervals overlapped on every embedder.For gemini-embedding, the reported intervals were [21.68, 22.72] for literature and [21.39, 22.74] for synthetic dilemmas.Source: §A.3 · p. 14, Table 3 · p. 15

B. Training Configuration

C. Judge Rubric and Validation

  • The 31B judge’s agreement with each expert was within or above inter-expert agreement for every dimension; only its overall verdict entered the reward.The judge reasoned over 15 dimensions and emitted four verdicts; Figure 7 maps pairwise Kendall agreement by dimension.Source: C. Judge Rubric and Validation · p. 14, Figure 7 · p. 15

D. Register Features and Assigned Frameworks

  • The register shift ran in the same direction under every assigned framework, but varied in size; Kantian prompts departed most from the other four profiles.Possibility modals fell 13% for Kantian prompts but rose under the other four frameworks; Figure 8 shows paired changes by framework.Source: D. Register Features and Assigned… · pp. 14–15

E. In-Context Baselines and External Judge

In-context demonstrations trade gains on some Base metrics against losses on others. Targeted demonstrations reduce sycophancy and order sensitivity but leave some consensus dilemmas unresolved, while the two judges agree on change direction but differ in levels.

E. In-Context Baselines and External Judge

  • In-context demonstrations improve some Base metrics at the expense of others.Source: E. In-Context Baselines and External… · p. 15
  • Targeted demonstrations lower sycophancy and order sensitivity, but make 32 of 587 low-ambiguity consensus dilemmas irresolvable.Random demonstrations make 103 such dilemmas irresolvable, while demonstrations overall can raise framing acceptance.Source: E. In-Context Baselines and External… · p. 15
  • The self-judge and frozen 31B judge agree on the direction of change, but not on metric levels; item-level agreement is poor for framing.For framing, the 31B judge flags 54.3% of Base replies versus 5.2% for the self-judge, and the authors interpret both judges for direction only.Source: E. In-Context Baselines and External… · pp. 15–16

F. Thinking and Answer Length

This section describes prompts that formalize dilemmas and elicit framework-grounded responses, alongside a pairwise judge that evaluates argument quality across tiered dimensions and returns structured verdicts. It also covers how loss masking shifts length from thinking into answers and how longer GRPO thinking is associated with greater acceptance of user framing on ELEPHANT.

F. Thinking and Answer Length

G.1. Dilemma Extraction Prompt

  • Gemma-4 31B’s extraction prompt asks it to identify dilemmas in each chunk and formalize competing duties as SMT constraints, preferably mutually exclusive to encode genuine conflicts.The prompt specifies separate constraints for the premise and each competing duty.Source: §G.1 · p. 19

G.2. Policy Prompt

G.3. Pairwise Judge Prompt

  • The pairwise judge compares competing ethical arguments across 15 Wachsmuth dimensions and is instructed not to favor length or style beyond those dimensions.It receives the system prompt and one user prompt for each ordered pair of completions.Source: System Prompt · p. 20, §G.3 · p. 20

System Prompt

  • The system prompt requires logic, rhetoric, dialectic, and holistic overall-quality verdicts, each choosing A, B, or tie and favoring a winner when a meaningful difference exists.It requires a JSON object containing exactly those four keys, without justification or confidence.Source: System Prompt · p. 20

User Prompt Template

  • The user prompt asks the judge to reason through all fifteen dimensions grouped by tier, then return only the four-field JSON verdict object.Source: User Prompt Template · p. 20

Related papers

Loading 2610.30349v1…