Source-linked AI summary
Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation
Minsoo Song, Chanwoo Kim, Sugyeong Eo, Chanjun Park
TL;DR
The paper examines whether consensus-based MAD improves human alignment for subjective rubric-based scoring, where inter-agent agreement may not match human judgments. It compares single-judge and consensus protocols with ablations for role prompting, interaction, and score sharing, finding that single judges align best and asymmetric roles primarily drive MAD degradation.
Problem
The paper addresses whether multi-agent consensus improves alignment with human judgments in criterion-level subjective scoring.
Method
The study compares a single-judge baseline with consensus-based MAD and ablations removing role asymmetry, interaction, or explicit score sharing across subjective evaluation tasks.
Results
The single-judge baseline achieves the strongest average human alignment, while Consensus-based MAD degrades RMSE and Spearman correlation on both tasks; asymmetric role prompting is the primary driver.
Takeaways & Limitations
Consensus-style, role-specialized MAD can enforce agreement while moving subjective scores away from human references, making role design central to alignment.
Takeaways & Limitations
The study evaluates one strict–lenient configuration, fixes five total rounds, uses unweighted averaging, and may not generalize to other MAD designs.
Abstract
from arXiv · showhide
Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus. However, for subjective rubric-based scoring, inter-agent agreement does not guarantee alignment with human judgments. In this paper, we compare a single-judge baseline against a consensus-based MAD protocol on subjective evaluation tasks and design three ablations to isolate the impact of role prompting, multi-round interaction, and explicit score sharing. Evaluations across six LLMs show that the single-judge baseline achieves the strongest human alignment on average across six judge models, whereas MAD shows degradation in human alignment on both tasks. Our ablations demonstrate that this performance drop stems primarily from asymmetric role prompting rather than the interaction itself. Specifically, assigning a strict judge role introduces a systematic downward bias that the consensus process fails to correct. The central finding is that this bias reflects strict-stance dominance beyond averaging: the consensus score falls well beyond the arithmetic midpoint of the standalone strict and lenient conditions, rather than averaging them out. Removing role asymmetry (Symmetric MAD) largely recovers baseline performance, while masking peer scores widens inter-agent disagreement on average and worsens average human alignment. These findings demonstrate that multi-agent consensus can enforce artificial agreement at the expense of true human alignment, revealing a structural limitation in consensus-style, role-specialized MAD protocols for subjective scoring.
1 Introduction
This paper asks whether consensus-based MAD improves human alignment in subjective rubric-based scoring and investigates mechanisms behind any degradation. It compares single-judge and consensus protocols while testing role asymmetry and numerical anchoring as competing explanations.
- The study compares single-judge and consensus-based MAD protocols using RMSE and Spearman correlation against human reference scores.
- LLM judges offer scalable, rubric-aware evaluation but can exhibit systematic biases.
- Prior MAD studies have not systematically tested human alignment in criterion-level subjective scoring.
- The experiments treat numerical anchoring and asymmetric role prompting as competing hypotheses and design ablations to test them directly.
2 Methodology
The study uses a compact MAD protocol with role-specialized agents, iterative exchange, and final aggregation, then removes individual components through targeted ablations. This isolates role prompting, interaction, and score sharing while holding core task inputs and models fixed.
- Single Judge: The single-judge condition independently assigns criterion-level scores without role specialization, interaction, or aggregation.
- Consensus-based MAD: The main protocol uses Strict and Lenient agents that independently score, exchange scores and rationales, revise across rounds, and average final scores.The process runs from Round 0 through Round 4.
- Ablations: Strict Role Judge removes interaction to isolate the effect of strict role framing.
- Ablations: Symmetric MAD preserves multi-round exchange but gives both agents the same neutral prompt, testing whether role asymmetry is necessary for alignment changes.
- Ablations: Score-Masked MAD preserves roles and rationale exchange while removing numerical scores to test whether score exposure acts as an anchoring signal.
3 Experimental Setup
The experiments evaluate standardized judge protocols across two substantially different subjective scoring tasks, six LLM judges, and human-reference alignment metrics. The setup also documents a prompt asymmetry that conservatively affects the single-judge comparison.
- Tasks: The study uses Korean Essay Scoring and SummEval, both with criterion-level human reference scores and a 1–5 scoring scale.
- Tasks: Korean Essay Scoring contains 600 essays assessed on content, organization, and expression using averaged human evaluations.
- Tasks: SummEval includes 700 system-summary instances rated for coherence, consistency, fluency, and relevance.
- Experimental Design: The SummEval single-judge prompt includes a strictness cue absent from the Korean Essay prompt, and the authors do not quantify its magnitude.
- Models and Metrics: Six LLMs spanning three families and 4B–27B capacity levels serve as judge models, with RMSE and Spearman correlation measuring alignment.
- Experimental Design: Figure 2 tracks protocol-level RMSE after each Consensus-based MAD exchange round on Korean Essay Scoring.
4 Results
Across both tasks, single-judge evaluation aligns best with human references, while Consensus-based MAD degrades RMSE and Spearman performance. The ablations indicate that asymmetric role prompting, especially strict framing, is the main source of this loss rather than interaction alone.
- Human Alignment: The single-judge baseline achieves the best average RMSE and Spearman alignment across six judge models on both tasks.Consensus-based MAD reaches RMSE 0.935 and 0.813, with Spearman 0.424 and 0.409.
- Role Asymmetry: 2.900 versus 3.345 and 3.486: Consensus-based MAD predicts lower Korean Essay scores than Single Judge and human reference.On SummEval, the corresponding means are 3.831, 3.935, and 4.285.
- Role Asymmetry: 0.973 versus 0.935: Strict Role Judge has higher Korean Essay RMSE than Consensus-based MAD; on SummEval, the values are 0.884 versus 0.813.The strict condition's mean scores are 2.733 and 3.672 across the two tasks.
- Role Asymmetry: 0.733 versus 0.644: Symmetric MAD remains closer to the Single Judge on Korean Essay RMSE; on SummEval, the values are 0.687 versus 0.682.Its mean scores also remain close to the single-judge means.
- Role Asymmetry: Cross-model role pairing changes RMSE magnitude but does not consistently recover the strongest single-judge alignment.
5 Conclusion
The study finds that asymmetric role prompts bias consensus judgments away from human references, while role design is central to the performance of consensus-style MAD protocols.
- Asymmetric role prompts bias consensus scores away from human reference judgments.
- The strict–lenient design produces degradation in human alignment, whereas removing role asymmetry improves alignment.
Limitations
The study evaluates one controlled MAD design with fixed roles, five rounds, unweighted averaging, two domains, and six judge models, limiting generalization beyond these settings.
- The protocol is a controlled operationalization rather than a direct replication of one prior MAD system.Results may differ with other role prompts, interaction structures, or aggregation strategies.
- The study tests one strict–lenient role configuration and fixes the protocol to five total rounds.The effect of varying the round count is not evaluated.
- Final scores use unweighted averaging, while weighted ensembles and majority voting are not explored.
- The evaluation covers Korean essay scoring and English summarization with six judge models.Generalization to other tasks, languages, or model families requires further investigation.
Ethical Statement
The study uses public datasets and structured, rubric-based prompts with standardized JSON outputs for research evaluation without collecting new human-subject data.
- The study uses two public datasets and collects no new human-subject data or personally identifiable information.LLM outputs are used solely for research analysis.
- System prompts combine role definitions, dataset-specific criteria, scoring rubrics, and structured JSON output formats.Reusable prompt components are defined once and inserted through placeholders.
- The essay rubric scores content, organization, and expression on a 1–5 scale.
- SummEval evaluates coherence, consistency, fluency, and relevance using rubric-based criteria.
- Prompt rules require one JSON object, integer scores from 1 to 5, and Korean text fields for the essay task.
B Dataset Implementation Details
The experiments use sampled Korean essay and SummEval datasets, standardized judge outputs, and score-masking controls for consistent analysis across conditions.
- 600 essays are sampled from six Korean writing prompts, with paragraph fields concatenated into plain-text essay inputs.Each essay has rubric-based scores from two trained human annotators.
- SummEval contributes 700 article–summary pairs from 100 articles and seven summarization systems.Human annotations cover coherence, consistency, fluency, and relevance on a 1–5 scale.
- Table 4 lists the judge models and their OpenRouter identifiers.
- All judge outputs follow the same JSON schema to support consistent parsing across models and protocols.Structured-output settings are used only to ensure parseable outputs.
- Score-Masked MAD removes numeric scores from shared responses while retaining rationale exchange.Numeric tokens in rationales and adjustment notes are replaced with [NUM].
D Score-Masked MAD analysis
Score masking reduces coordination between agents and worsens average human alignment, indicating that shared numerical scores function primarily as deliberation signals rather than merely anchors.
- 0.935 → 1.040 on Essay and 0.813 → 0.994 on SummEval: score masking worsens average RMSE across both domains.A minority of model–task pairs improve locally, but the aggregate effect is negative.
- 0.176 → 0.385 on Essay and 0.076 → 0.224 on SummEval: masking substantially widens the final inter-agent score gap.The agents reach less coordinated final positions when numerical peer scores are removed.
- Removing numerical scores reduces inter-agent convergence without restoring alignment with human ratings.This pattern suggests that score exposure provides mutual calibration rather than causing a pure anchoring bias.
E Model Specific Analysis
Model-level results show a dominant advantage for Single Judge, but sensitivity to role specialization and score exposure varies across model families and capacities. Round-wise trajectories indicate that alignment depends more on the convergence point’s location than on agreement alone.
- Model-level performance: For most models, Single Judge achieves the strongest human alignment, while the aggregate ordering remains a dominant rather than universal trend.The reported ordering is Single Judge > Symmetric MAD > Consensus-based MAD > Score-Masked MAD.
- Model-level performance: Qwen3.5-27B favors Symmetric MAD over Single Judge on both tasks: Essay RMSE 0.864 vs. 0.938 and SummEval RMSE 0.734 vs. 0.799.This suggests neutral multi-round interaction can occasionally improve alignment for larger models.
- Model-level performance: GPT-4o-mini shows the largest Consensus-based MAD degradation, with Spearman dropping from 0.516 to 0.315 on Korean Essay and from 0.481 to 0.296 on SummEval.Score-Masked MAD gives this model the weakest SummEval ranking alignment, with ρ = 0.224.
- Round-wise trajectories: Gemma3-27B and Gemma3-4B maintain mean predicted scores near the gold mean, whereas Qwen3.5-27B drifts from 2.76 to 2.32 on Korean Essay.Mean-score stability corresponds to lower alignment loss, while downward drift corresponds to larger RMSE increases.
- Round-wise trajectories: Convergence does not guarantee mean-score stability: Gemma3-12B’s gap trajectory resembles Gemma3-27B’s, but its mean declines from 3.06 to 2.79.The convergence point must be close to the gold reference for agreement to preserve alignment.
- Statistical analysis: Strict prompting significantly shifts score levels but has statistically uncertain effects on rankings: p < 0.0001 for score levels on both tasks, versus p = 0.54 and p = 0.21 for rankings.The paired-bootstrap analysis uses 10,000 resamples with judge model treated as a fixed effect.
G.1 Aggregate Results
The lenient role is not a symmetric counterweight to the strict role, and consensus exhibits strict-stance dominance beyond simple averaging. Agreement trajectories therefore need to be interpreted alongside score level and human-reference alignment.
- Aggregate role effects: +0.086 and −0.057: Lenient-only score shifts are far smaller than Strict-only shifts of −0.754 and −0.613 on Korean Essay and SummEval, respectively.The lenient effect is only about one tenth as large as the strict effect.
- Consensus score trajectory: Rapid convergence can still produce poor alignment when agents converge below the human reference, whereas stable proximity to the reference corresponds to lower RMSE.Figure 3 tracks round-wise mean predicted score against the human reference; Figure 4 tracks the absolute inter-agent gap.
- Aggregate role effects: 0.644 → 0.582 and 0.682 → 0.498: Lenient-only reduces RMSE on Korean Essay and SummEval, respectively.However, Spearman correlation decreases from 0.504 to 0.478 and from 0.458 to 0.417, so no protocol dominates both metrics.
- Dominance analysis: The Consensus-based MAD mean falls below the arithmetic midpoint of the Strict-only and Lenient-only score shifts, establishing strict-stance dominance beyond averaging.The midpoint is the outcome expected under simple averaging of the standalone conditions.