PaperlayerView exact revision on arXiv

Source-linked AI summary

Controversy and Group Certainty Jointly Shape Everyday Moral Judgments

Ziyu Chen, Minjeong Shin, Tuan Dung Nguyen, Colin Klein, Chenhao Tan, Nick Schuster, Nicholas George Carroll, Alasdair Tran, Lexing Xie

arXiv:2609.04750v1cs.CY

TL;DR

The paper asks how anonymous aggregate signals about disagreement and group certainty shape judgments of everyday moral dilemmas. In a randomized experiment, it finds that social information increases both weakening and strengthening, while disagreement is the strongest signal and certainty effects become asymmetric when both signals appear.

  • Problem

    The paper examines how judgments about everyday moral dilemmas change when people learn about peers’ disagreement and certainty without receiving new arguments or first-order evidence.

  • Method

    In a randomized experiment, 2,159 participants evaluated dilemmas before and after viewing controversy, group certainty, both signals, or an active control task.

  • Results

    All three treatments increased weakening and strengthening relative to Control; disagreement showed the strongest association with updating, while combined displays left only in-group certainty clearly associated with less weakening.

  • Takeaways & Limitations

    Representing both controversy and certainty may give platforms and AI systems a fuller account of collective opinion without treating popularity or certainty as moral correctness.

  • Takeaways & Limitations

    The findings establish that social signals prompt updating, not that resulting moral judgments are more accurate or better justified.

Abstract

from arXiv · show

Everyday moral life rarely resembles a trolley problem. It involves disputes about families, relationships, work, money, and care, situations in which people often encounter the judgments of others. We examined how judgments about nuanced interpersonal dilemmas respond to social information that conveys collective opinion without revealing the arguments behind it. Specifically, we studied two signals: controversy, the extent to which community judgments are divided between two opposing verdicts; and group certainty, the confidence expressed by each side. We derived these signals from 54,827 judgments on 135 dilemmas posted to Reddit's r/AmItheAsshole and presented them separately or together in a preregistered randomized experiment (N = 2,159). Relative to the control condition, all three treatments increased both weakening, a changed verdict or reduced confidence, and strengthening, increased confidence without a verdict change. Thus, aggregated social information can reshape moral judgments even without arguments. Controversy alone tripled the weakening rate, from 5.3% to 15.9%. Shown alone, certainty on either side was associated with movement toward that side. Shown alongside controversy, only like-minded certainty remained clearly associated with less weakening. This asymmetry of influence is double-edged: the confidence of like-minded others may help minorities resist majority pressure, but it may also insulate mistaken judgments from correction.

Main

The study asks how controversy and group certainty shape judgments about everyday interpersonal moral dilemmas, using aggregate social signals without underlying arguments. A preregistered randomized experiment tested these signals and found that they can shift judgments in either direction.

  • Everyday moral judgments may change under subtle social influence, despite their ties to identity, emotion, and deeply held values.
  • Controversy describes how judgments are distributed across opposing verdicts, whereas group certainty describes how confidently each side expresses its judgment.
  • The study drew 135 interpersonal dilemmas across 32 topics from Reddit’s r/AmItheAsshole and derived controversy and group-certainty signals from 54,827 community judgments.
  • Participants were randomly assigned to controversy, certainty, both signals, or an active control task, then reported verdicts and confidence before and after exposure.
  • All three social-information conditions increased both weakening and strengthening relative to Control, indicating that brief argument-free summaries shifted moral judgments in either direction.

Experimental setup

MORALMOMENTS combines real-life AITA dilemmas with aggregated verdict and certainty signals, then tests their effects through a three-step randomized judgment task. The analysis defines outcomes relative to each participant’s initial verdict and confidence.

  • Dataset construction: The dataset contains 135 real-life dilemmas from AITA posts, each supported by at least 50 verdicts and 54,827 judgments across 32 topics.
  • Dataset construction: Group controversy is the minority verdict proportion from 0 to 0.5, while certainty difference is majority certainty minus minority certainty.
  • Study design: Participants first reported a verdict and confidence, then viewed controversy, certainty, both, or an active control task before responding again.
  • Study design: Controversy displayed verdict shares, while certainty displayed certain and uncertain proportions separately for agreeing and disagreeing groups.
  • Outcome coding: Responses were classified as maintained, weakened, or strengthened by jointly comparing post-treatment verdicts and confidence with initial responses.
  • Derived variables: The analysis treated majority or minority status and relative in-group versus out-group certainty as positions defined by each dilemma’s verdict distribution.

Results

All three social-information treatments increased weakening and strengthening relative to Control. Controversy was associated with more weakening and less strengthening, while certainty effects depended on whether certainty aligned with the participant’s initial verdict.

  • Overall treatment effect: Weakening rose from 5.3% in Control to 10.6% under Certainty, 15.9% under Controversy, and 12.1% under Controversy+Certainty.
  • Overall treatment effect: Strengthening rose from 8.6% in Control to 25.6%, 25.9%, and 20.9% under Certainty, Controversy, and Controversy+Certainty, respectively, among 1,924 eligible observations.
  • Signal associations: Greater disagreement was associated with more weakening and less strengthening, whereas higher in-group certainty showed the opposite pattern when signals were displayed separately.
  • Combined signals: In the combined condition, higher in-group certainty remained associated with less weakening, while out-group certainty showed no clear independent association with weakening or strengthening.
  • Social position: Minority participants weakened more often than majority participants under Controversy alone, at 27.3% versus 7.9%.
  • Written explanations: Controversy was mentioned most often in written explanations, and in-group certainty was mentioned more often than out-group certainty.
  • Exploratory analyses: Exploratory dilemma analyses found no consistent relationship between judgment change and severity, affected party, or vulnerable-person involvement, with cautious interpretation warranted for two unadjusted differences.

Discussion

The study shows that people update everyday moral judgments in response to aggregate disagreement and group certainty, even when social information supplies no arguments. However, the effects establish updating rather than improved accuracy or justification, and several mechanisms and generalizability questions remain unresolved.

  • Core findings: Random assignment to each social-information condition increased both weakening and strengthening relative to Control across diverse interpersonal dilemmas.The result indicates that brief, argument-free summaries can shift judgments in either direction.
  • Core findings: Disagreement was the strongest updating signal, particularly when participants learned that their initial verdict placed them in the minority.The combined display did not simply add the effects of controversy and certainty.
  • Contribution: Aggregate social information extends research on numerical opinion proportions, interpersonal confidence, and social influence in moral judgment to 135 varied interpersonal dilemmas from online discussions.The study connects previously separate lines of research using everyday rather than primarily stylized moral scenarios.
  • Interpretation and limits: The findings do not establish that updated moral judgments are more accurate or better justified, because the dilemmas generally lack independently verifiable answers and the treatments supplied no new arguments or facts.Aggregate judgments may reflect reliable insight or shared error.
  • Open questions: The mechanism behind the asymmetric certainty pattern remains unresolved because retrospective, self-selected explanations could not distinguish attention, conformity, resistance, biased assimilation, or other processes.Future work should directly measure attention and information processing, manipulate candidate mechanisms, or elicit reasoning before and after exposure.
  • Open questions: The design establishes immediate responses to controlled summaries in this sample, not generalization across populations and platforms or persistence after repeated exposure.The study used English-language Reddit dilemmas and a predominantly English-speaking sample, with concise summaries rather than richer platform information.
  • Interpretation and limits: The outcomes capture judgment revision direction rather than whether revision improved the judgment.Future research should test effects on consideration of affected parties and, where answers are verifiable, accuracy.
  • Implications: Platforms and AI systems may represent collective opinion more fully by showing both popularity and certainty without treating either as a marker of moral correctness.The proposed representation preserves disagreement and side-specific confidence while avoiding an unsupported correctness inference.

Methods

The study combined a preregistered four-arm randomized experiment with aggregate signals derived from real-world Reddit moral dilemmas. Participants repeatedly reported verdicts and confidence before and after seeing controversy, group certainty, both, or no social information.

  • Design overview: The study described dataset construction, recruitment, experimental design, outcome definitions, statistical models, and exploratory analyses of written explanations and feedback.These components structured the paper’s methods and analyses.
  • Materials: The researchers selected 135 dilemmas from AITA posts spanning 32 common topics and oversampled highly controversial posts for coverage across the controversy distribution.The source corpus began with 102,998 posts, of which 9,998 met the minimum verdict-comment criterion.
  • Materials: Controversy was computed from verdict distributions, while group certainty was inferred from commenter language and aggregated separately within the YA and NA groups.The signals were derived after extracting verdict labels and excluding INFO comments.
  • Participants and allocation: Participants were adults recruited through Prolific, assigned with equal probability to Controversy, Certainty, Controversy + Certainty, or Control conditions.Eligibility required age 18 or older, English fluency, and no prior participation in the study or pilot.
  • Participants and allocation: The final analytic sample comprised 2,159 participants contributing 4,318 participant–dilemma observations after preregistered attention-check screening.There were 2,171 completed submissions before 12 records failed the attention criterion.
  • Procedure: Each participant evaluated two randomly selected dilemmas, recording an initial verdict and confidence, viewing the assigned display or Control task, and recording them again.The survey used three phases: onboarding and training, the repeated moral-judgment task, and post-study surveys and debriefing.
  • Measures: Certainty referred to commenters’ apparent confidence within each verdict group, whereas confidence referred to participants’ self-reported certainty before and after treatment.Keeping the terms distinct identifies group certainty as the treatment information and participant confidence as an outcome.
  • Measures: Displays presented percentages as human-icon arrays, numerical labels, and short descriptions to support comprehension across differences in numeracy and attention.The icon array represented proportions of people while the numerical label preserved exact values.

Extended figures

The extended tables document the regression models for judgment weakening and strengthening, alongside a table categorizing participants’ written explanations by experimental variables and outcomes.

  • Regression tables: Table 2 reports weakening-model coefficients, odds ratios, and 95% confidence intervals relative to the Control baseline.The model includes sex and nationality as controls, with covariate selection by cross-validated greedy forward selection.
  • Regression tables: Table 3 reports strengthening-model coefficients, odds ratios, and 95% confidence intervals relative to the Control baseline.The model uses GDMS and intellectual humility as covariates.
  • Written explanations: Table 4 categorizes selected written explanations by mentions of disagreement rate, in-group certainty, or out-group certainty.It further separates reasoning for changing or maintaining a stance by condition and outcome.

Competing Interests.

The study situates everyday moral judgment within online collective evaluation and distinguishes controversy from group certainty as social signals. It measures certainty as an aggregate property of verdict groups rather than as private confidence or correctness.

  • Background: Online moral discussions provide a setting for studying rich interpersonal dilemmas and collective evaluations at scale.
  • Contribution: The study examines responses to community judgments rather than analyzing, classifying, or generating normative content.
  • Caveat: Participants’ verdicts and confidence should not be interpreted as direct measures of underlying moral conviction, moralization, or moral status.
  • Contribution: The experiment independently manipulates controversy and group certainty, separating certainty among agreeing and disagreeing verdict groups.
  • Certainty as social information: Group certainty represents how confidently people on each verdict side appear to hold their collective view, not whether that view is objectively correct.
  • Data: The dataset comprises everyday dilemmas categorized across topics, supporting broad coverage of interpersonal moral situations.

B.3.1 Individual certainty

Individual certainty was annotated using human experts and language models, with performance improving under few-shot and chain-of-thought prompting. The best reported setup achieved a test-set F1 score of 0.92 and AUC of 0.88.

  • Human annotation: Four human experts showed overall Krippendorff’s alpha of 0.69 when labeling certainty.Annotator A had the highest agreement with the majority, with Spearman’s correlation of 0.87 and Krippendorff’s alpha of 0.85.
  • Model prompting: GPT-4 zero-shot prompting reached test-set Krippendorff’s alpha of 0.75 and Spearman’s correlation of 0.76 for certainty labeling.GPT-3.5 Turbo performed substantially worse without examples, with test-set alpha of 0.14 and correlation of 0.38.
  • Model prompting: Zero-shot chain-of-thought increased GPT-3.5 Turbo’s Spearman’s correlation from 0.38 to 0.76 and Krippendorff’s alpha from 0.14 to 0.75.For GPT-4 and GPT-4o, zero-shot chain-of-thought slightly decreased performance.
  • Model prompting: Few-shot prompting supplied 14 annotated verdict comments and produced strong certainty-labeling results for GPT-4 and GPT-4o.GPT-4 and GPT-4o each reached Spearman’s correlation of 0.79, with Krippendorff’s alpha of 0.75 and 0.76, respectively.
  • Model prompting: GPT-3.5 Turbo with few-shot chain-of-thought prompting achieved the best performance, with a test-set F1 score of 0.92 and AUC of 0.88.The reported aggregation did not significantly affect model-performance rankings.

C.2.1.2 Ablation results

Model-selection analyses identified a compact covariate set for weakening, while treatment information showed distinct associations with judgment change across social positions. The selected model included GDMS, IH, Sex, and Nationality, with treatment-specific coefficients linking disagreement and certainty to weakening.

  • Single-block addition: GDMS was the strongest single predictor across all four weakening-model specifications, with Δℓ̄≈0.007 and rank 1 in every model.IH ranked second or fourth depending on specification, while Topic and raw nationality variables reduced out-of-sample fit.
  • Greedy forward selection: GDMS entered first in every greedy forward-selection path, followed by different combinations of IH, Sex, and Nationality across specifications.The GLM without dilemma factors reached its best model at step 4, whereas the GLM with dilemma factors reached its best model at step 3.
  • Selected model: The best model was the GLM without dilemma as a factor, combining GDMS, IH, Sex, and Nationality with ℓ̄=-0.3062 and SE=0.0091.The glmmTMB model was nearly identical, with ℓ̄=-0.3063 and SE=0.0092; its dilemma-level random-effect variance was effectively zero.
  • Best-model coefficients: Disagreement increased weakening, whereas in-group certainty reduced it and out-group certainty increased it in the relevant treatment conditions.In the combined condition, disagreement remained positively associated with weakening and in-group certainty remained negatively associated with it.
  • Social-position comparisons: 27.3% of minority-position participants weakened under Controversy, compared with 7.9% in the majority position.Under Certainty, weakening was 18.5% when the in-group was less certain than the out-group, versus 5.3% in the reverse certainty contrast.
  • Cross-treatment comparisons: Controversy alone produced higher weakening than Controversy+Certainty in three of four matched subcells, and adding certainty lowered weakening across the cells that differed.The first two contrasts were significant; the third fell just short after correction.

C.2.3 Regression robustness: continuous treatment variables versus binary social-position variables

The robustness analysis compared continuous treatment signals with binary social-position indicators in the weakening model. Both representations improved prediction, but continuous treatment information retained more predictive variation and was therefore selected.

  • Representation comparison: Both binary social-position and continuous treatment-information blocks improved held-out prediction over the same base model.The binary block increased mean log-likelihood by 0.0182, while the continuous block increased it by 0.0227.
  • Representation comparison: The continuous treatment-information block produced a larger gain of +0.0045 and was retained for the primary model.It preserves variation in the displayed percentages that binary variables discard.
  • Robustness check: The continuous block reproduced the primary covariate selection and final held-out score, with ℓ̄=-0.3062.The robustness check therefore supported retaining continuous disagreement and certainty measures rather than only binary social positions.
  • Scope of strengthening analysis: The strengthening analysis used a separate logistic model restricted to observations with initial confidence below the ceiling, yielding N=1,924 observations.Among these observations, 391 (20.3%) were strengthened.

C.3.1.1 Ablation results

The analyses compared covariate contributions and treatment associations for strengthening and verdict change. Social-information variables were central: disagreement increased verdict change but reduced strengthening, while like-minded certainty showed the opposite pattern.

  • Strengthening: GDMS was the strongest single predictor of strengthening across specifications, with a marginal gain of approximately 0.003–0.005 in mean log-likelihood.No other block produced a positive gain in the GLM or glmmTMB specifications.
  • Strengthening: GDMS and IH were the only blocks selected in the same order across all strengthening specifications.The best models were reached at step 2, with no further block improving fit.
  • Strengthening: Disagreement reduced strengthening, whereas in-group certainty increased it and out-group certainty decreased it in the Certainty condition.For strengthening, disagreement had β = −4.08 in Controversy; in-group certainty had β = 1.82, and out-group certainty had β = −1.21.
  • Strengthening: 35.1% versus 14.4% strengthened under Controversy when participants held the majority versus minority position.Under Certainty, strengthening was 32.4% versus 16.6% when in-group certainty was at least as high as out-group certainty.
  • Model comparison: Continuous treatment information improved prediction more than binary social-position variables, with a held-out mean-log-likelihood gain of +0.0042.The continuous block improved fit by 0.0223 versus 0.0181 for the binary block.
  • Verdict change: Disagreement increased verdict-change odds in Controversy and the combined condition, while in-group certainty reduced verdict changes and out-group certainty increased them when shown alone.The combined condition retained the negative in-group-certainty association but not the out-group-certainty association.

C.4.2 Confidence-change regression

The confidence-change analysis modeled signed confidence shifts among participants who retained their initial verdict. Disagreement lowered confidence, whereas certainty aligned confidence with the participant’s side, and GDMS plus Personality provided the selected covariates.

  • Model specification: The model analyzed signed confidence change only among responses where participants kept their initial verdict.Negative values indicated reduced confidence and positive values indicated increased confidence.
  • Ablation: GDMS was the strongest decision-relevant block, while Personality consistently improved out-of-sample prediction for confidence change.GDMS contributed approximately 0.003–0.004 in mean log-likelihood, and Personality contributed approximately 0.0015–0.0017.
  • Model selection: GDMS entered first and Personality second in every forward-selection specification; no further block improved fit.The best model was base + GDMS + Personality, reaching mean log-likelihood 0.6086 in the GLM without dilemma factors.
  • Treatment effects: Disagreement reduced confidence, whereas in-group certainty raised it and out-group certainty lowered it in the Certainty condition.The estimated effects were β = −0.186 for disagreement, β = 0.108 for in-group certainty, and β = −0.050 for out-group certainty.
  • Covariates: Intuitive decision style raised confidence, while Dependent style lowered it; Emotional Stability had a small positive association.The corresponding coefficients were 0.038, −0.046, and 0.023.

C.5.2.1 Model Specification

The explanation-willingness analysis used multilevel logistic regression with a dilemma-level random intercept and cross-validated forward selection. Age, GDMS, grouped nationality, Personality, intellectual humility, Sex, and post-treatment confidence were selected, but treatment assignment mostly did not predict explanations.

  • Ablation: Age produced the largest single-block prediction gain, followed by GDMS and grouped Nationality in both tested specifications.Age improved mean log-likelihood by +0.0077 in the GLM and +0.0072 in glmmTMB.
  • Model selection: The two specifications selected the same seven blocks in the same order: Age, GDMS, grouped Nationality, Personality, IH, Sex, and post-treatment confidence.Pre-treatment confidence and change type were the first rejected blocks.
  • Model specification: The willingness model was a multilevel logistic regression with a dilemma-level random intercept.Treatment terms and the random intercept formed the base model, while covariates were selected by cross-validated forward selection.
  • Covariates: Older participants and participants with Intuitive style or intellectual humility provided explanations more often, whereas Rational style reduced explanation likelihood.The reported coefficients were 0.273 for Age, 0.164 for Intuitive style, 0.174 for intellectual humility, and −0.390 for Rational style.
  • Treatment effects: Treatment indicators were mostly unrelated to explanation provision, although higher disagreement under Controversy increased explanation likelihood.The treatment-indicator p values were 0.484, 0.114, and 0.137; disagreement had β = 0.846, p = 0.013, under Controversy.

Task 2: Treatment reference and stance.

The treatment-reference task classified whether written explanations mentioned disagreement, in-group certainty, or out-group certainty and whether participants accepted or rejected each as relevant evidence. Model annotations were evaluated against expert labels and then applied to all explanations.

  • Stance coding: Each treatment variable received one of three stance labels: reject, none, or accept.A zero indicated no reference or an unshown variable; positive and negative labels represented acceptance and rejection of relevance.
  • Stance coding: The task distinguished stance toward information from whether the participant’s verdict or confidence changed.Labels were based on how participants treated the information in their deliberations, not on outcome movement.
  • Annotation procedure: The annotation prompt required classification of all three variables and output labels for disagreement rate, in-group certainty, and out-group certainty.Unshown variables were assigned zero.
  • Evaluation: The model recovered three-class stances well enough to annotate the full set of 2,744 explanations.The evaluation compared model labels with expert labels on a random sample of 150 explanations.
  • Evaluation: The annotation model detected references with 0.91–0.95 accuracy across disagreement, in-group certainty, and out-group certainty.Referenced-class F1 scores were 0.89, 0.83, and 0.73, respectively.

C.5.3.2 Results: General reasoning for change

Treatment information reshaped participants’ reasoning about moral judgments, especially by activating conformity and resistance while reducing situation-based reasoning. Conformity was the clearest pathway among participants who changed their judgments.

  • Written explanations: Treatment responses without explanations increased among weakeners and strengtheners but decreased among maintainers relative to Control.The corresponding comparisons were significant after Benjamini–Hochberg correction.
  • Weakened: Conformity was absent in Control but appeared in 38.4% of Controversy, 30.9% of Certainty, and 50.0% of Controversy + Certainty explanations among weakened responses.All treatment-versus-control comparisons remained significant after correction.
  • Maintained: Among maintained responses, conformity and resistance increased across all treatments, while fairness, contextual, and procedural reasoning decreased.Conformity rose to 6.7–9.4% from 0.5%, and resistance rose to 12.1–14.3% from 1.4%.
  • Strengthened: Among strengthened responses, conformity increased from 4.0% in Control to 45.9–63.6% across treatments.Moral conviction, fairness, and contextual reasoning also declined in several treatment conditions.
  • Overall pattern: Across change types, treatment information shifted reasoning toward social reference and conviction-based justifications and away from situational engagement.The overall pattern suggests that social signals changed the composition of reasoning rather than simply adding another consideration.

C.5.3.4 Participant-clustered regression robustness check.

Participant-clustered regressions confirmed the main attention and acceptance comparisons while accounting for dependence among signal codes and repeated responses. In-group certainty remained more accepted than out-group certainty in the certainty conditions.

  • Robustness check: All four reference-rate contrasts remained significant after participant-clustered regression adjustment.The models used robust standard errors clustered by participant and Benjamini–Hochberg correction.
  • Acceptance rates: In-group certainty was accepted more often than out-group certainty in both the Certainty and combined conditions.In the combined condition, in-group certainty was accepted more often than disagreement, while disagreement and out-group certainty did not differ.
  • Acceptance rates: Most comments that engaged with a signal accepted it, whereas comments rejecting a signal almost always maintained the judgment.Across treatments, acceptance among engagers ranged from 60% to 81%.

C.5.4.2 Results: the consequence profile of dilemmas

The MORALMOMENTS corpus spans everyday friction to serious and lasting interpersonal harms, with dilemmas concentrated at medium to high severity. Family involvement and vulnerability are common features of the dilemmas.

  • Severity: Severity covered the full 1–7 scale, with median 4.0 and interquartile range 3.0–5.0.Nineteen dilemmas were rated 2 or below, while 48 were rated 5 or above.
  • Affected parties: Family was named in 73% of dilemmas, strangers in 22%, ongoing contacts in 15%, and friends in 14%.The author was implicated in 99% of dilemmas by design of the source material.
  • Vulnerability: Forty-six percent of dilemmas affected someone identifiably vulnerable, most often a child at 39%.Vulnerability was recorded separately from the multi-select affected-party categories.

C.5.4.3 Results: consequences and judgment revision

Dilemma severity and most consequence features showed little association with judgment revision. Two exploratory contrasts suggested more weakening for family dilemmas and less strengthening for child-involved dilemmas, but both require caution.

  • Overall associations: Severity and most affected-party and vulnerability categories had little association with dilemma-level weakening or strengthening rates.The analyses collapsed responses to one pair of rates per dilemma to handle response nesting.
  • Exploratory contrasts: Family dilemmas weakened more often than friend dilemmas, with r = +0.61 and p = .003.This contrast was treated as exploratory because the friends group contained only nine dilemmas.
  • Exploratory contrasts: Child-involving dilemmas strengthened less often than dilemmas involving nobody vulnerable, with r = −0.23 and p = .031.The contrast was based on only 6–10 responses per condition per dilemma and was therefore treated cautiously.
  • Interpretation: Future tests should include more dilemmas balanced across consequence categories and more responses per dilemma.The authors identify these design changes as necessary to assess whether the exploratory differences are reliable.
Loading 2609.04750v1…