PaperlayerView exact revision on arXiv

Source-linked AI summary

A Quantitative Approach to Asch Conformity in Frontier Large Language Models

Senthil Rajasekaran, Stylianos Loukas Vasileiou

arXiv:2610.36668v1cs.CLcs.AI

TL;DR

Prior LLM conformity studies measured consensus adoption across many questions, while the human Asch task’s social-pressure interpretation does not directly carry over to LLMs. This paper repeatedly samples LLM-natural fill-in-the-blank questions to compare answer distributions alone and under manufactured consensus. All six tested frontier models conform, with the degree depending on the consensus answer and participant count.

  • Problem

    Prior LLM Asch-style studies measured consensus adoption across many questions, while wrong answers in the human line task could be attributed to social pressure, which has no meaningful LLM analog.

    Source: §1 · p. 1

  • Method

    The study repeatedly samples fill-in-the-blank questions and compares answer distributions when the model answers alone and after seeing manufactured participants’ answers.

    Source: §1 · p. 1, §2.1 · p. 2, §2.2 · p. 3

  • Results

    49 points at n = 10 for (w1, wp), the largest pooled shift; all six tested frontier LLMs conform, with little or no shift toward far words.

    Source: §1 · p. 2, Results · pp. 4–5

  • Takeaways & Limitations

    The results suggest a plausible route to manufacturing conformal behavior through a series of small, plausible steps, each suggested by fewer agents.

    Source: §4 · p. 6

    The frontier models’ answer distributions are approximated with 100 repeated samples, adding measurement uncertainty, and the narrow SAT-style question format limits scope.

    Source: Limitations · p. 6

Abstract

from arXiv · show

The Asch conformity experiment has recently gained considerable attention as a way of measuring conformity in large language models (LLMs). While previous studies have focused on demonstrating conformity in LLMs, our work adds a further quantitative layer to this approach. Instead of sampling a frontier LLM on a suite of questions, our approach samples a single question repeatedly, leveraging the probabilistic nature of LLMs to produce a distribution of answers. In doing so, we can compare the distributions of answers before and after the model is presented with the manufactured consensus. Our experiments confirm that frontier LLMs do conform, reinforcing prior literature, and, through our distribution-to-distribution comparisons, provide additional quantitative data on the conditions that lead to conformity and the degree to which it occurs.

1 Introduction

Prior Asch-style LLM studies measured adoption of a manufactured consensus across many questions, but the human line task’s social-pressure interpretation has no meaningful LLM analog. This paper instead uses repeated sampling of an LLM-natural task to measure answer-distribution changes and finds that conformity varies with the consensus answer.

1 Introduction

  • Earlier LLM Asch studies measured how often a model adopted manufactured consensus answers across many different questions.The original human line-judgment task made consensus-matching wrong answers attributable to social pressure, which has no meaningful analog in the LLM setting.Source: §1 · p. 1
  • The study uses SAT-style fill-in-the-blank sentences as an LLM-natural task and repeated sampling to compare answer distributions before and after manufactured consensus.Each question is sampled hundreds of times rather than testing hundreds of questions only once.Source: §1 · pp. 1–2
  • All six tested frontier LLMs conform, with conformity strongest toward preferred words, substantial toward semantically close unseen words, and nearly absent toward semantically distant words.Conformity grows with the number of reported participants up to five and changes little between five and ten.Source: §1 · p. 2

2 Methodology

The methodology adapts Asch’s comparison task to fill-in-the-blank word choices, samples repeated answers alone and under manufactured consensus, and measures changes in both consensus choice and the full answer distribution.

2 Methodology

  • Repeated independent trials compare the same question’s answer distribution alone with its distribution after the model sees other participants’ answers.This replaces the line judgment with a sentence-completion task natural to LLMs; candidate words vary in model preference or meaning-distance from the most frequent answer.Source: §2 · p. 2

2.1 Building the Questions

  • Preferred words are selected from T open-answer trials using a frequency threshold, while non-preferred words were never produced and are generated at near, medium, or far meaning-distance from the top word.Questions pair the top preferred word with other preferred words or non-preferred words, and also pair the non-preferred words with one another.Source: §2.1 · p. 2

2.2 Testing and Measuring Conformity

  • Alone and group phases use T trials with the same randomized option orders; in group prompts, n manufactured participants endorse the pair word chosen less often alone.Conformity shift is the change in distractor share from alone to group trials; total variation distance and Jensen–Shannon divergence capture changes across the full answer distribution, with sampling variation tested.Source: §2.2 · p. 3

3 Experiments

Across ten SAT-style sentences and six frontier LLMs, the experiments quantify answer shifts under manufactured consensus. The shifts vary by candidate relationship and model, and often plateau as the number of participants increases.

3 Experiments

  • The experiments apply the methodology to ten English SAT-style fill-in-the-blank sentences and six frontier LLMs from three providers.Each sentence has one blank, narrowed by a contrast clause, with an adjective or noun as the answer.Source: §3 · p. 3, Experimental Setup · pp. 3–4

Results

  • Pooled over models and sentences, (w1, wp) has the largest mean conformity shift, rising from 22 points at n=1 to 49 at n=10.At n=10, shifts are 30, 24, and 0 points for (w1, unear), (w1, umedium), and (w1, ufar), respectively; (unear, umedium) reaches 43 points, while pairs with ufar stay below 1. Significant increases occur in 31/36 (w1, wp), 38/60 (w1, unear), 28/60 (w1, umedium), 47/60 (unear, umedium), and 3/60 (umedium, ufar) questions, with none for (w1, ufar) or (unear, ufar).Source: Results · pp. 4–5

Experimental Setup

  • Models use temperature 1; each blank receives 100 open-answer trials, and words appearing in at least 5% of trials are preferred-model-words.Questions use preferred words up to rank 4; alone and group phases each have 100 trials, with group sizes {1, 2, 5, 10}.Source: Experimental Setup · p. 4
  • Non-preferred-model-words were generated by GPT-5.6-TERRA at temperature 0 with reasoning off; for each of three closeness levels, the generator received the sentence, its w1-blank version, and the level definitions.Source: Experimental Setup · p. 4

Results

  • All six models shift toward distractors in four question types, but (w1, wp) shifts at n=10 range from 33 points for claude-opus-5-5 to 73 for gemini-3.8-flash.No model shifts by more than 2 points in pairs involving ufar; gemini-3.8-flash also has the largest shifts for (w1, unear), (w1, umedium), and (unear, umedium).Source: Figure 3 · p. 5, Results · p. 5
  • For GPT-6-SOL, shifts decrease from n=5 to n=10 across all four question types where models generally shift toward distractors.Individual-model shifts do not always increase with participant count.Source: Results · p. 5
  • For (w1, wp), the shift exceeds (w1, unear) by 14–19 points across n, a significant paired contrast (26 model-sentence pairs; Wilcoxon p ≤.016).The (w1, unear) shift exceeds (w1, umedium) by 3–7 points, but not significantly (p ≥.052); (w1, umedium) exceeds (w1, ufar) by 5–24 points.Source: Results · p. 5
  • In 1,564 of 1,584 question-and-n combinations, total variation distance is within 0.02 of |∆n|, so conformity shift accounts for nearly all answer-distribution change.Models chose a filler in 583 of 197,976 counted trials.Source: Results · p. 5

4 Conclusion

All tested frontier models conform, but the amount depends on the answer presented and how many participants report it. The findings suggest conformity can be elicited through smaller, plausible steps rather than distant answers backed by many agents.

  • Conformity depends strongly on the target: it is immediate and significant toward answers close to the model’s output, but negligible toward semantically far answers.Source: §4 · p. 5
  • Conformity is more pronounced in the informational setting using preferred-model-words than in the normative setting using nonpreferred-model-words.Source: §4 · p. 5
  • Conformity often changes little between five and ten participants, and some distractors are chosen more often with five; the authors suggest small, plausible steps from fewer agents as a route to manufacturing conformity.Source: §4 · pp. 5–6

Limitations

The paper's distribution-to-distribution measurements rely on repeated sampling because the frontier models do not expose their answer distributions, and the study uses a deliberately narrow question style.

Limitations

  • The study approximates inaccessible model distributions by running each identical question 100 times, which introduces additional measurement uncertainty.Its scope is also limited by the deliberately specific 2015-era SAT-style fill-in-the-blank format.Source: Limitations · p. 6

A Sentences

This section provides fill-in-the-blank sentence stimuli, including examples about reaching a summit and the view from the top.

A Sentences

  • A sentence about an old bridge contrasts its solid appearance from afar with badly damaged supports found during inspection.Source: A Sentences · p. 7
  • One stimulus contrasts a grueling summit ascent with hikers motivated by stories of the view at the top.Source: A Sentences · p. 7
  • Another stimulus describes a gallery that was advertised across the city but was almost empty on Saturday afternoon.Source: A Sentences · p. 7

B Prompts

The prompts ask models to complete sentence blanks and generate words at specified levels of semantic closeness. The conformity prompt presents answer options and other participants' answers.

B Prompts

  • The prompt defines “very close,” “somewhat similar,” and “very different” by how much the replacement word's meaning overlaps with the original.Source: B Prompts · p. 7
  • The non-preferred-word generator requests five grammatical, sensible completions at one of three specified levels of meaning closeness.Source: B Prompts · p. 7
  • In the conformity test, models choose among four options after seeing the other participants' answers, with the answer options presented in trial order.Source: B Prompts · p. 7

C Detailed Results

The detailed results compare alone-phase and group-phase answer distributions using conformity shifts and Jensen–Shannon divergence. Tables report question-level results by model, while a separate table identifies cases where total variation distance differs notably from the absolute shift.

C Detailed Results

  • Both distribution-comparison measures are zero for identical distributions and at most 1.The measures are total variation distance and Jensen–Shannon divergence in bits.Source: C Detailed Results · p. 7
  • Tables 1–6 give every question’s results for each model, listing the more-frequently chosen alone-phase word, distractor, alone distractor share, and, for each n, conformity shift in percentage points and Jensen–Shannon divergence in bits.Table 7 lists question-and-n combinations where total variation distance differs from |∆n| by more than 0.02.Source: C Detailed Results · pp. 7–8, Table 7 · p. 15
Loading 2610.36668v1…