Source-linked AI summary
Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives
Yinuo Xu, Veronica Derricks, Allison Earl, David Jurgens
TL;DR
Subjective NLP disagreement can reflect social and demographic perspectives, while many existing systems lack structured group-level modeling. DEM-MoE routes judgments to demographic-aware experts and evaluates synthetic annotations and blending; across three datasets it outperforms other models for nearly all demographic groups, while blending strategies depend on dataset structure.
Problem
Method
DEM-MoE routes text, annotator, and demographic inputs to expert subnetworks, while the paper evaluates zero-shot persona-generated annotations and strategies for blending real and synthetic data.
Source: §1 · p. 1Results
Across three datasets, DEM-MoE outperforms other models in predicting ratings for nearly all demographic groups; synthetic annotations align moderately with human ratings, and blending strategies vary by dataset.
Source: §8 · p. 8Takeaways & Limitations
The findings support treating annotator disagreement as meaningful signal and show that dataset-specific blending strategies can enhance demographic alignment.
Source: §8 · p. 8The model does not explicitly represent intersectional identities, and this lack of supervision may unintentionally essentialize identity.
Source: §9 · p. 9
Abstract
from arXiv · showhide
We present an approach to modeling annotator disagreement in subjective NLP tasks through both architectural and data-centric innovations. Our model, DEM-MoE (Demographic-Aware Mixture of Experts), routes inputs to expert subnetworks based on annotator demographics, enabling it to better represent structured, group-level variation compared to prior models. DEM-MoE consistently performs competitively across demographic groups, and shows especially strong results on datasets with high annotator disagreement. To address sparse demographic coverage, we test whether LLM-generated synthetic annotations via zero-shot persona prompting can be used for data imputation. We show these synthetic judgments align moderately well with human annotations on our data and offer a scalable way to potentially enrich training data. We then propose and evaluate approaches for blending real and synthetic data using strategies tailored to dataset structure. We find that the optimal strategies depend on dataset structure. Together, these contributions improve the representation of diverse perspectives.
1 Introduction
Subjective annotation disagreement often reflects socially grounded differences, yet conventional aggregation can erase minoritized perspectives. The paper introduces DEM-MOE and synthetic-data strategies to model this variation, with blending choices tied to dataset structure.
1 Introduction
- Disagreement in subjective tasks can reflect meaningful social, cultural, or demographic differences rather than random noise.Treating it as noise and collapsing annotations into one label can marginalize minoritized perspectives.Source: §1 · p. 1
- DEM-MOE routes inputs to expert subnetworks based on annotator demographics to represent structured variation in subjective judgments.The paper also evaluates synthetic annotations and data-blending strategies for sparse demographic coverage.Source: §1 · p. 1
- The paper presents synthetic-annotation alignment and a framework for incorporating synthetic data, whose effectiveness depends on dataset structure.The contribution statement identifies zero-shot LLM-generated annotations as the synthetic-data approach.Source: §1 · p. 1
2 Related Work
Prior NLP approaches range from aggregating disagreement to modeling annotator-specific judgments, but often lack architectural bias for shared group-level patterns. Related recommender-system work models group preferences, with different objectives from predicting individual subjective judgments.
2 Related Work
- Recommendation systems use MoE for multifaceted interests, interactions, and personalization, but target unified item rankings rather than preserving distinct judgments.The related-work discussion contrasts their weak demographic patterns and unified-ranking objective with this paper’s focus on socio-demographic regularities in singular judgments.Source: §2 · p. 2
- Traditional majority voting or averaging can obscure meaningful disagreement from underrepresented or minoritized groups.Other approaches measure or filter disagreement, or learn from it directly through uncertainty weighting, distributions, and annotator-specific models.Source: §2 · p. 2
- Many NLP models use demographics as shared-encoder inputs, limiting explicit specialization for group-specific reasoning.Prior approaches include demographic prediction and jury learning, but most struggle to capture group-level differences.Source: §2 · p. 2
3 Data
The five datasets vary in task, demographic coverage, annotation density, agreement, and demographic signal, providing different forms of annotator variation for evaluation.
3 Data
- Toxicity has 2,535 demographic combinations and α=0.27; Safety has 48 combinations, 123 annotations per instance, and α=0.24.Ridge regression finds strong demographic signal for Toxicity, especially age (2.06); Safety’s signal is weak, likely reflecting complex harmfulness judgments.Source: §3 · p. 2
- Politeness has the highest agreement (α=0.44), low entropy, and strong race effects; Offensiveness shows notable age and occupation signal.Source: §3 · p. 2
- PCC has sparse annotations (3.3 per instance), high uncertainty, and the strongest demographic signal, especially for age.Source: §3 · p. 2
- The dataset selection spans diverse tasks, demographic coverage, and disagreement levels.Source: §3 · p. 2
4 DEM-MOE for Modeling Disagreement
DEM-MOE addresses limited group-level modeling in prior systems with demographic-aware routing to expert subnetworks. Its embeddings, sparse expert selection, and specialization-oriented losses encode a structured bias toward shared demographic judgment patterns.
4 DEM-MOE for Modeling Disagreement
- DEM-MOE uses demographic-aware routing because prior systems commonly place demographics in a shared network and cannot explicitly specialize for distinct group judgments.The related-work discussion also notes limits in learning intersectional patterns with separate independent components.Source: §4 · p. 3
- The architecture assumes annotators from similar demographic groups may share systematic ways of judging texts.The paper presents this as an inductive bias intended to make subgroup differentiation more robust and interpretable under imbalance or sparse subgroup representation.Source: §4 · p. 3
- Text, annotator, and demographic embeddings form the input; a gate selects the top k experts, whose outputs are sparsely mixed to predict a rating.The model uses Bayesian embeddings, hard selection with k in {2, 3}, and a linear regression head.Source: §4 · p. 3
- Load, orthogonality, variance, and within-group specialization losses encourage expert diversity and distinct routing paths.Load regularization targets roughly even use rather than strict uniformity, allowing demographic specialization.Source: §4 · p. 3
5 Experiment 1: Modeling Perspectives
Experiment 1 tests whether demographic structure helps models represent diverse annotator perspectives. DEM-MoE performs especially well when disagreement and demographic signal are strong, while its advantages vary with dataset properties.
5.1 Experiment Setup
- Models predict individual annotator ratings from item text and, where applicable, annotator or demographic information; instance-level splits test generalization to new content and unfamiliar annotators.Annotators can overlap between train and test sets, while snippets do not.Source: §5.1 · p. 4
5.2 Results
- No competing system performs consistently across datasets: PMF generally performs worst but beats LLaMA on PCC, while Jury Learning performs much worse on Toxicity.The authors attribute these differing patterns to variation sources involving content, identity, and demographics, and recommend evaluation across multiple datasets.Source: §5.2 · p. 4
- Increasing structure generally benefits models using annotator identity and demographics, while DEM-MoE is more robust across labeling conditions than text-dominant alternatives.LLaMA performs best on Safety, where demographic influence is minimal; the reported MoE advantage is not explained by parameter count alone.Source: §5.2 · pp. 4–5
- DEM-MoE outperforms every model on all Toxicity demographic groups, nearly all Offensiveness and PCC groups, and ties the best models on specified Politeness groups.It loses only for race on Offensiveness and gender on PCC, where it is statistically equivalent to the best model.Source: §5.2 · p. 4, Figure 1 · p. 5
- Expert usage specializes within demographic groups, with the relevant groups varying by dataset and the strongest Politeness specialization appearing for race.The analysis measures mean pairwise KL divergence across subgroup expert-usage distributions, normalized by log(K), where K is the number of experts.Source: §5.2 · p. 5, Figure 2 · p. 5
- The model exhibits both fine-grained subgroup distinctions and broader demographic-aligned expert roles, especially on Politeness, Offensiveness, and PCC.Ridge regression predicts expert usage from demographic attributes, and heatmaps show experts specializing in subgroup perspectives.Source: §5.2 · p. 6
- Low agreement, predictive demographic attributes, and annotation density favor DEM-MoE; its demographic error correlates with group annotation count (r=0.40).Toxicity has 213 annotations per demographic combination, compared with 44 for Politeness.Source: §5.2 · p. 6
6 Experiment 2: Synthetic Annotations
Experiment 2 tests whether zero-shot LLM annotations prompted with demographic personas approximate human ratings, as a step toward combining synthetic and real annotations. LLaMA aligns best among the tested models, though simulation is harder for low-agreement conversation tasks.
6.1 Experiment Setup
- The evaluation prompts four instruction-tuned LLMs to adopt demographic personas, give brief reasoning, and output ratings; Pearson’s r and MAE assess alignment and accuracy.The models are LLaMA-3.3-70B-Instruct, QwQ-32B, OLMo-2-13B, and Mistral-Nemo-Instruct-2407.Source: §6.1 · p. 6
6.2 Results
- Zero-shot LLaMA outperforms the other tested models in alignment with human judgments, although alignment varies by task.The authors also report directional alignment and reasonably calibrated magnitude; the Table 2 caption identifies MAE across models and datasets.Source: §6.2 · p. 6, Table 2 · p. 7
- LLMs struggle to simulate ratings for Safety and PCC conversations, where human annotator agreement is low; Safety’s lowest MAE reflects its narrower 1–3 scale.The other datasets use a 1–5 scale, and demographic-group performance generally shows no dominant-versus-minoritized gap, with noted exceptions.Source: §6.2 · p. 6
7 Experiment 3: Model Training with Real and Synthetic Annotations
Experiment 3 tests how synthetic annotations are generated and blended with real ratings, aiming to improve DEM-MOE’s perspective diversity. It finds that synthetic-data effects vary with dataset structure.
7 Experiment 3: Model Training with Real and Synthetic Annotations
- LLM-generated annotations offer moderately aligned training ratings and potentially broader, scalable coverage, but poorly aligned synthetic data can introduce bias and harm generalization.The paper frames careful integration as necessary because synthetic data may misrepresent minority-group perspectives.Source: §7 · p. 7
7.1 Experiment Setup
- Experiment 3 varies both which synthetic annotations are generated and how they are incorporated during training.The goal is to enhance DEM-MOE by increasing the diversity of perspectives.Source: §7.1 · p. 7, §7 · p. 7
7.1.1 Generating Synthetic Annotations
- Generation compares random 0.5x, 1x, and Fill quantity strategies with Cluster, which samples representative and distant annotators to add view diversity.Cluster uses k-means on demographic profiles and rating behavior to select representative annotators and induce disagreement.Source: §7.1.1 · p. 7
7.1.2 Blending Real and Synthetic Data
- Training compares PT+FT, unweighted mixing, and weighting synthetic ratings by human alignment, cluster trustworthiness, and demographic-cluster rarity.Real ratings receive weight 1, while synthetic weights scale the training loss.Source: §7.1.2 · pp. 7–8
7.2 Results
- Synthetic-data effects are dataset-specific: Offensiveness and Safety benefit most from blended training, while Politeness and PCC see limited or no benefit.For Politeness, weighted cluster-generated data achieves the lowest MAE; for PCC, cluster generation performs worst.Source: Figure 3 · p. 8, §7.2 · p. 8
- Randomly selected 0.5x and 1x synthetic annotations provide more gain than demographically curated Cluster annotations.The reported comparison concerns Offensiveness and Safety, the datasets that benefit most from unweighted and weighted blending.Source: §7.2 · p. 8
8 Conclusion
The conclusion presents DEM-MOE as a way to model structured demographic variation in annotator disagreement and reports that synthetic-annotation benefits depend on the dataset.
8 Conclusion
- DEM-MOE routes inputs to experts by annotator demographics to capture structured variation in group-level reasoning patterns.Across three datasets, it outperforms other models in predicting ratings for nearly all demographic groups.Source: §8 · p. 8
- Zero-shot LLM persona annotations align moderately with human ratings, and dataset-specific blending strategies enhance demographic alignment.Source: §8 · p. 8
9 Limitations
DEM-MoE is most effective when disagreement, demographic signal, and annotation density are sufficient, but its evaluation and modeling assumptions constrain the scope of its findings. The authors also note limitations in identity representation, data sampling, task wording, synthetic annotations, and hyperparameter tuning.
9 Limitations
- DEM-MoE is most effective with high annotator disagreement, strong demographic signal, and sufficient annotation density; only three of five evaluated datasets had sufficient demographic signals for its experts.The authors identify PCC, Politeness, and Offensiveness as those three datasets.Source: §9 · p. 9
- The evaluation covers norm-violation tasks, and the authors leave effectiveness in domains such as moral reasoning or humor detection unresolved.They also note that disagreement may reflect noise or inconsistency and recommend assessing whether it is substantive.Source: §9 · p. 9
- The model does not explicitly represent intersectional identities, although experts can specialize in perspectives associated with particular subgroups.The authors note this may unintentionally essentialize identity.Source: §9 · p. 9
- DEM-MoE adds hyperparameters beyond standard settings, and missed configurations could improve model performance.Source: §9 · p. 9
- PCC’s Prolific sample may not represent the broader population, despite recruitment designed to balance key U.S. demographic groups.The authors note that participants tend to be younger, more educated, and more technologically literate than the general population.Source: §9 · p. 9
- Annotation wording may shape human and LLM judgments, while synthetic-data effectiveness depends on data quality and task complexity.The authors note that alternative prompt configurations, including more elaborate instructions, may improve synthetic annotation fidelity and downstream modeling.Source: §9 · p. 9
10 Ethics
The ethics discussion cautions against treating synthetic annotations or model outputs as substitutes for diverse human perspectives. It emphasizes risks of stereotyping and overgeneralization, while the appendix briefly describes the datasets and demographic signals.
10 Ethics
- Synthetic annotations based on sociodemographic inputs may essentialize identities, stereotype groups, or misrepresent minoritized perspectives, so the authors recommend sparing use and never substituting them for diverse human judgments.They also recommend clear labeling and minimizing synthetic data’s influence through weighting based on assessed trustworthiness.Source: §10 · pp. 9–10
- DEM-MoE outputs may reflect aggregate group patterns rather than individual preferences, especially for intersectional or underrepresented identities, and should be treated as perspective-informed estimates alongside human input.The authors warn that strong group-level performance does not necessarily establish faithful or equitable representation of lived experience.Source: §10 · p. 10
A.1.1 Data Overview
- The appendix provides an overview of the five datasets in Table 3.Source: §A.1.1 · p. 10
A.1.2 Demographic Signals
- Ridge regression coefficients using demographic group features serve as a proxy for demographic signal, with Safety showing the weakest signal.Source: §A.1.2 · p. 10
A.1.3 Patient Centered Communication Data
This section describes a patient-centered communication dataset from prostate cancer conversations, its annotation protocol, and DEM-MoE’s architecture and training choices. It also reports comparisons across datasets and annotator groups, along with experiments on LoRA and model capacity.
A.1.3 Patient Centered Communication Data
- The dataset contains 71 doctor-patient transcripts and 2,232 twelve-turn snippets, sampled with context and sufficient doctor speech.The window advances six turns; snippets with doctor utterances below the 25th-percentile word count were removed, excluding backchannels.Source: §A.1.3 · pp. 10–11
- Annotators rated nine doctor qualities, including partnership, support, informativeness, warmth, empathy, respect, perceived views of patient communication, agency, and competence.The measures were drawn from the PAACT study and prior measures of patient perceptions of doctor qualities.Source: §A.1.3 · pp. 10–11
- The study recruited 594 U.S. Prolific annotators, targeting demographic balance; each rated 15 snippets, with four annotators per snippet.Annotators imagined themselves as the patient; the nine measures were summed into patient-centered communication, perceived patient communication, and relational communication.Source: §A.1.3 · pp. 11–12
- Krippendorff’s α ranged from 0.244 to 0.338 across doctor-quality dimensions, while Cronbach’s α was 0.958.Doctor informativeness had the lowest agreement and warmth the highest; the nine measures were aggregated into three doctor-quality measures.Source: §A.1.3 · p. 12
2. What is your current age?
- The survey produced 7,553 annotations, and some demographic categories were aggregated for modeling.The paper reports the ten most common profiles in Table 12 and subgroup distributions in Table 28.Source: §2 · p. 14
B DEM-MOE Model Details and Training
- DEM-MoE combines text, annotator, and demographic embeddings to select top-k experts, whose normalized weights combine their outputs into one rating.The selector produces logits over a shared expert pool, and softmax normalizes the selected experts’ logits.Source: B DEM-MOE Model Details and Training · p. 14
B.1 Experiment 1 Training Details
- Training used early stopping, phased loss-weight schedules for MoE expert routing and specialization, and a randomly initialized shared default embedding for unseen test annotators.Jury learning models and ModernBERT trained for 10 epochs, while MoE models trained for 50 epochs; phase transitions used expert-load standard-deviation thresholds.Source: §B.1 · p. 15
B.1.1–B.1.6 Training Details
- Optuna hyperparameter searches were conducted for Offensiveness, Politeness, Safety, PCC, and Toxicity.Other models used grid search for Jury Learning and Ea + En, while Llama parameters followed Orlikowski et al. (2025).Source: §B.1.1 · p. 15, §B.1.2 · p. 15, §B.1.3 · p. 15
B.1.7 Computational Budget
- MoE runs take 20–30 minutes on PCC, Offensiveness, Politeness, and Safety, 1–2 hours on Toxicity, and about half as long as Jury Learning runs.Ea + En runs take about 2 hours on non-Toxicity datasets and 4 hours on Toxicity, using one NVIDIA RTX A6000 with 48GB memory.Source: §B.1.7 · p. 16
C.1 Performance on Seen vs. Unseen Annotators
- Seen-versus-unseen annotator performance is evaluated with Pearson correlation, MAE, and EMD; MAE and correlation compare snippet-level predictions with actual ratings, while EMD measures preservation of rating distributions.Annotator-level predictions are averaged for DEM-MOE, Jury Learning, LLaMA, and En + Ea before computing MAE and correlation.Source: §C.1 · pp. 16–17
- MoE trains roughly twice as fast as Jury Learning.The passage describes its efficiency and representativeness as well-suited to large-scale, heterogeneous annotation data.Source: §C.1 · p. 17
C.2 Best zero-shot performance
- Zero-shot LLaMA performs worst across datasets, while LoRA fine-tuning substantially improves its performance toward the other baselines.The passage attributes zero-shot LLaMA’s weakness to not being optimized for annotator judgments and its lack of learned annotator behaviors or dataset-specific text-to-rating mappings.Source: §C.2 · p. 17
C.1 Performance on Seen vs. Unseen Annotators
- DEM-MOE matches or surpasses Jury Learning, MBERT, and PMF across datasets and annotator groups, with strongest gains on low-agreement Safety and improved EMD on PCC and Offensiveness.On Safety, it significantly outperforms Jury Learning in correlation and MAE; on Politeness and Toxicity, it performs similarly to other SOTA models.Source: Figure 6 · p. 17, §C.1 · p. 17
C.3 Capacity vs. Inductive Bias
- Despite having fewer parameters, MoE outperforms larger models in most demographic categories on Toxicity, Offensiveness, and PCC, supporting architecture rather than parameter capacity as the beneficial inductive bias.The passage identifies LoRA-LLaMA, MBERT, and En + Ea as having much larger parameter sizes than MoE.Source: §C.3 · p. 19, Table 25 · p. 19
C.4 Experiment 1 within-group expert specialization
This section examines demographic-conditioned expert specialization within and across groups, along with demographic-tailored synthetic-perspective evaluations. It compares zero-shot and few-shot persona prompting, selects zero-shot Llama-70B for Experiment 2, and reports Pearson-correlation trends across models and datasets on raw data.
C.4 Experiment 1 within-group expert specialization
- Within-group specialization appeared in Politeness and Offensiveness, with different experts associated with distinct demographic categories in each task.For Politeness, experts 1 and 4 had distinct category associations; for Offensiveness, experts 2 and 3 did. The figures aggregate subgroup-level expert usage, so multiple experts can appear active despite discrete per-instance routing.Source: Figure 7 · p. 19, §C.4 · p. 19, Figure 8 · p. 20
- Within-group specialization differed by task: it was inadequate for Safety, where experts 0 and 3 dominated, but better for Toxicity, where most perspectives mapped to experts 3 and 4.The authors suggest Safety’s weak specialization may reflect low predictive power of demographics for annotation ratings; expert 2 specialized in Toxicity perspectives from African Americans, people with a Master’s degree, and conservatives.Source: Figure 9 · p. 20, Figure 10 · p. 20, §C.4 · p. 20
- PAACT showed sufficient within-group specialization: expert 1 mapped to several young-to-middle-aged, Asian and White, and lower-trust profiles, while expert 2 mapped to moderate healthcare visitors and several younger, Black, and higher-trust profiles.Both expert profiles also included annotators with less than high school education or particular forms of trust toward the medical system.Source: §C.4 · p. 20
C.5 Experiment 1 Cross-group expert specialization
- Ridge-regression coefficients linked demographic attributes to expert usage: Politeness experts specialized in race, occupation, education, and age, while Offensiveness experts specialized in age, occupation, and education.In Politeness, expert 1 specialized in race and occupation, expert 4 in race and education, and experts 2 and 3 in race and age; Offensiveness experts 3 and 4 specialized in age and occupation, and expert 2 in occupation and education.Source: §C.5 · pp. 20, 22, Figure 12 · p. 21, Figure 13 · p. 21
- Cross-group specialization also varied across PCC, Toxicity, and Safety: experts were associated with gender, age, race, education, and LGBTQ status in task-specific patterns.For PCC, experts 1, 2, 3, and 6 specialized in gender; in Toxicity, experts 2 and 4 specialized in education, age, gender, and race; in Safety, experts 0 and 3 specialized in race, and to a lesser extent education and gender.Source: Figure 14 · p. 21, Figure 15 · p. 21, Figure 16 · p. 22
D.1 Experiment 2 Group MAE Results
- Group MAE generally showed no performance gap between dominant and minoritized groups, except for PCC annotators with “Other” race and Politeness annotators with less than a high school diploma.The analysis compared demographic groups relative to their frequency in the data.Source: Figure 17 · p. 22, §D.1 · p. 22, Figure 18 · p. 23
D.2 Experiment 2 zero-shot prompts
- The zero-shot prompts represented annotator demographics as personas and tailored the evaluation task to Safety, Politeness, Offensiveness, Toxicity, or PCC.The authors tested prompt variations in length and instruction granularity and report that the selected versions performed best.Source: §D.2 · pp. 25–28
- Synthetic responses were required to use the exact format “[Quality Name]: [Explanation]:::[Rating]”.Source: §D.2 · p. 28
D.4 Experiment 2: Raw dataset performance
- Pearson correlation was reported across models and datasets in the raw-data results, following the same trend as the previously reported MAE.The supplied passage does not specify the correlation values or which models lead.Source: Table 26 · p. 29, §D.4 · p. 29
D.3 Zero-shot vs. Few-shot prompting
- Zero-shot Llama-70B achieved the highest correlation and lowest MAE on PCC, so it was selected for Experiment 2 for performance and scalability.The comparison covered 3-shot Llama-8B, Mistral-7B, and Llama-70B, with examples selected by demographic match, race and doctor-trust match, random choice, rating diversity, or mixed demographic match.Source: §D.3 · pp. 28–29, Figure 22 · p. 29
D.5 Experiment 2: cross-dataset rank consistency
The analysis compares reasoning-model rankings across datasets, finding that rankings align across Safety, Toxicity, and Offensiveness but Politeness is less aligned with other tasks.
D.5 Experiment 2: cross-dataset rank consistency
- The analysis ranks four reasoning models by within-dataset Pearson’s r, then computes Spearman correlation between their rankings.Source: §D.5 · p. 29
- Spearman’s ρ ranges from 0.6 to 0.8 across Safety, Toxicity, and Offensiveness model rankings.The table reports correlations between model rankings across tasks.Source: §D.5 · p. 29, Table 27 · p. 29
- Politeness rankings correlate most with Offensiveness (ρ = 0.4) and have low correlations with other datasets.Source: §D.5 · p. 29
- The authors suggest shared norm-violation reasoning may explain alignment among Safety, Toxicity, and Offensiveness, while Politeness may need dedicated pragmatic supervision.PCC also correlates highly with Safety, Toxicity, and Offensiveness despite assessing prosocial qualities; the authors propose joint training for those three tasks.Source: §D.5 · p. 29