PaperlayerView exact revision on arXiv

Source-linked AI summary

Reach Into The CHOIR: Free-List Elicitation Uncovers Distinct Model Voices in LLM Ensembles

Ben Wigler, Maria Tsfasman

arXiv:2609.38448v1cs.AIcs.CL

TL;DR

Single-pass similarity cannot distinguish narrow prompts, lexical cueing, and shared defaults from broad prompts with recoverable alternatives. CHOIR repeatedly elicits and compares concept lists; across its evaluations, it separates these cases while retaining model signatures and persona shifts.

  • Problem

    Single-pass answers leave unclear whether similar outputs reflect narrow prompts, prompt-vocabulary echo, shared defaults, or broad answer spaces with stable alternatives.

    Source: §1 · p. 1

  • Method

    CHOIR repeatedly elicits ranked lists, clusters entries into prompt-level concepts, and compares concept salience across models and elicitation conditions.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    CHOIR found above-chance surface agreement on 93/100 Infinity-Chat prompts while distinguishing narrow prompts from broader prompts with recoverable alternatives; base-model identity remained dominant, with persona shifts within model signatures.

    Source: §5 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    CHOIR distinguishes measurable forms of open-ended homogeneity before conclusions are drawn about model sameness, ensemble value, or non-default answers.

    Source: §5 · p. 10

    The external run covers English-language Infinity-Chat 100 prompts, and persona effects may differ with short descriptors or human-demographic role prompts.

    Source: §5 · p. 10

Abstract

from arXiv · show

Open-ended LLM homogeneity can create false plurality when several systems appear to offer independent perspectives while returning the same familiar default. Single-pass answers obscure the distinction between agreement produced by a tightly constrained answer space, prompt-vocabulary echo, and broader answer spaces with stable alternatives beneath the surface. We introduce CHOIR (Collective Hierarchically-Ordered Inquiry Responses), a framework that adapts free-list elicitation from cognitive anthropology to LLM ensembles. CHOIR repeatedly elicits ranked lists, clusters items into prompt-level concepts, and measures concept salience across models, prompt variants, and persona conditions. We evaluate CHOIR on Infinity-Chat 100, an external prompt bank from recent work on open-ended model homogeneity, and on a 27-question targeted diagnostic bank designed to isolate mechanism-level contrasts. On Infinity-Chat 100, CHOIR reproduces high surface agreement (93/100 prompts above chance) while separating narrow prompts from broad prompts with recoverable depth. Across targeted probes and the external prompt bank, base-model identity remains the strongest recoverable signature, and persona prompts shift surfaced concepts within base-model signatures. A source-blind ranking module prioritises rare-but-stable candidates for later inspection. CHOIR turns open-ended homogeneity into a diagnostic measurement problem by asking where models converge, why they converge, and what remains reachable under structured depth probing.

1 Introduction

Open-ended agreement can reflect independent convergence or a narrow shared slice of possible answers, making surface similarity insufficient to identify diversity. CHOIR uses repeated ranked-list elicitation and concept-level salience analysis across prompts and model conditions to examine what lies beneath first answers.

1 Introduction

  • Several models can appear to offer independent perspectives while returning the same narrow slice of possible answers, creating false plurality.This matters because people use LLMs for advice, judgement, explanation, and synthetic perspective.Source: §1 · p. 1
  • Single-pass answers obscure whether convergence reflects narrow prompts or stable alternatives that remain reachable under sustained probing.The paper also identifies prompt vocabulary and trained defaults as possible sources of convergence.Source: §1 · p. 1
  • CHOIR repeatedly elicits ranked lists, maps entries into prompt-level concepts, and compares their salience across models and elicitation conditions.The framework distinguishes frequent defaults, stable rarities, model-specific signatures, persona-sensitive shifts, and prompt-vocabulary echo.Source: §1 · p. 2
  • The evaluation pairs Infinity-Chat 100 with a 27-question diagnostic bank designed to isolate mechanisms such as prompt echo and persona-conditioned shifts.The diagnostic bank targets contrasts that the external prompt bank cannot guarantee.Source: §1 · p. 2

2 Related work

Prior work documents convergent model outputs as well as persistent model-specific differences, and studies ways to elicit broader perspectives. CHOIR addresses the lack of a common concept-level framework for separating prompt structure and lexical cueing from model- and persona-sensitive differences.

Open-ended homogeneity and diversity

  • Research reports convergence in model opinions, normative answers, and open-ended prompts, alongside evidence of persistent model-specific idiosyncrasies.The cited work also connects generated-text homogenisation with human expression and thought.Source: §2 · p. 2
  • Other approaches elicit diversity through criteria-based, recall, or multilingual prompting, while CHOIR tracks stable defaults and rare repeatable concepts.CHOIR tests whether cross-model agreement is associated with prompt wording or independent convergence.Source: §2 · p. 2

Free-list elicitation and cultural-domain methods

  • CHOIR adapts free-list elicitation and salience logic to repeated LLM conditions, using concept salience and overlap to characterise cross-model agreement.Unlike token-local log probabilities, its concept-level measure is available across the closed- and open-weight providers studied.Source: §2 · p. 2

Research questions

  • Existing work leaves no common concept-level framework for distinguishing narrow prompts and lexical cueing from model- and persona-sensitive differences beneath similar first answers.The paper frames four questions about prompt width, model signatures and persona shifts, mechanism checks, and candidate triage.Source: §2 · p. 3

3 Method

CHOIR treats models as respondents in repeated free-list elicitation, grouping responses into shared prompt-level concepts and comparing salience profiles. The method combines external and targeted prompt banks with extraction, consensus, diagnostic, and candidate-ranking procedures.

3.1 The CHOIR algorithm

  • CHOIR elicits ranked lists across templates and temperatures, groups near-paraphrases in a shared per-prompt codebook, and compares salience profiles against shuffle baselines.Consensus is stable concept-level agreement after elicitation and codebook construction; apparent convergence may reflect wording, narrow task structure, or generic defaults.Source: §3.1 · p. 3

3.2 Prompt banks, study roles, and models

  • The study pairs the 100-prompt Infinity-Chat 100 external inventory with a targeted diagnostic bank for controlled mechanism contrasts.The targeted bank is a transparent study instrument, not a canonical benchmark; Table 1 assigns the banks and ranking check complementary validity roles.Source: §3.2 · p. 4, Table 1 · p. 4
  • Infinity-Chat 100 receives full elicitation depth, while the nine-model ensemble uses ranked-list templates, supported temperature settings, and five generations per condition.Persona-conditioned generation uses Template A with five synthetic identity profiles; targeted-probe cell coverage varies, with seven or nine models in the reported persona-shift contrasts.Source: §3.2 · p. 4

3.3 Extraction, codebook construction, and salience

  • Claude Haiku 4.5 extracts numbered entries, which are embedded and clustered with HDBSCAN to form a semantically deduplicated shared codebook.The extraction unit is one numbered response entry; clustering uses a minimum cluster size of 5 and ε=0.3.Source: §3.3 · pp. 4–5
  • Smith’s S scores concept salience by recurrence and rank, while cross-model consensus uses Rank-Biased Overlap with p=0.9.Signal-to-chance compares observed mean pairwise RBO with a 1,000-permutation shuffle baseline; echo rate measures concepts near question-vocabulary terms.Source: §3.3 · p. 5
  • Diagnostic modules test model-to-cluster associations, measure persona shifts within base-model signatures, and use source-blind evaluator cohorts to shortlist candidates.Independent conditioned and unconditioned cohorts address simple persona-evaluator self-preference, but do not eliminate judge bias; an initial human calibration appears in Appendix D.1.Source: §3.3 · p. 5

4 Results

CHOIR separates surface agreement by prompt width, recovers model-specific signatures alongside persona-conditioned shifts, and tests mechanisms behind convergence. Targeted probes distinguish vocabulary echo from other agreement patterns, while source-blind ranking prioritises candidates for later validation.

4.1 RQ1: Infinity-Chat 100 separates narrow prompts from depth-sensitive prompts

  • 93/100 Infinity-Chat 100 prompts showed above-chance unconditioned agreement, while width analysis separated narrow, high-baseline prompts from broader prompts with lower first-pass agreement.Broader prompts had larger relative persona lifts, but absolute same-persona RBO remained higher for narrow prompts; relative lift is interpreted alongside absolute RBO.Source: §4.1 · pp. 5–6, Table 2 · p. 6

4.2 RQ2: Model signatures dominate while persona prompts shift salience

  • Across 4,500 conditioned cells, model identity was more recoverable than persona identity: accuracy was 0.877 [0.867, 0.886] versus 0.101 [0.093, 0.110].Within-model persona accuracy was 0.323 [0.309, 0.337] against five-way chance of 0.20, and profile-stripped variants remained above chance.Source: §4.2 · p. 6, Figure 3 · p. 7
  • Persona conditioning increased relative same-persona cross-model agreement on 82/100 prompts, with mean lift 3.55x in the highest-width bins and 0.95x in low-width bins.Profile-embedding leakage correlated with conditioned multiplier (Spearman ρ = .311) and persona recoverability (ρ = .800).Source: §4.2 · p. 7

4.3 RQ3: Targeted probes distinguish prompt echo from stable convergence

  • The matched self-description prompts contrasted 78.3% prompt-vocabulary echo and 1.10 signal-to-chance for the cued version with 4.4% echo and 2.27 for the open version.The two prompts therefore produced different kinds of agreement: high echo with low signal-to-chance versus low echo with higher signal-to-chance.Source: §4.3 · pp. 7–8, Figure 4 · p. 8
  • Same-persona cross-model RBO rose by +0.154 on emotional pain, +0.138 on retirement, +0.079 on tea-party behavior, and +0.018 on open self-description.On the targeted corpus, model accuracy was 0.836 [0.797, 0.872], compared with 0.110 [0.078, 0.146] for persona.Source: §4.3 · p. 8
  • Cue stripping lowered item-level and codebook-cluster echo in all six Infinity-Chat pairs, while cross-model RBO fell in three pairs and rose in three.The depth-pressure template produced more than 50% novel concepts in 65/100 prompts, with novelty highest in wider prompts.Source: §4.3 · p. 8

4.4 RQ4: Source-blind ranking triages candidates for later validation

  • Source-blind evaluator cohorts advanced conditioned-source candidate pools in 11/12 and 8/12 prompts, respectively, and selected the same top candidate in 7/12.In a 25-participant calibration, participants matched the conditioned AI cohort’s top-ranked item more often than the unconditioned cohort’s (37.3% versus 28.3%, paired participant-level sign-flip p < .05).Source: §4.4 · p. 9

5 Discussion

CHOIR finds that high surface agreement can reflect narrow prompts, lexical cueing, or recoverable alternatives, while model identity remains the strongest signature and persona prompts shift concept salience within it. Mechanism checks and candidate ranking help distinguish these patterns, with scope and validation limits.

RQ1: prompt width and surface homogeneity

  • 93/100 prompts had above-chance surface agreement, while prompt-width categories separated high-baseline narrow prompts from broader, lower-baseline prompts.Persona conditioning produced larger relative multipliers in broader prompts, although absolute same-persona RBO remained higher in narrow prompts.Source: §5 · p. 9

RQ2: model signatures and persona-conditioned salience

  • Model identity was recovered at 0.877 accuracy across 4,500 conditioned Infinity-Chat cells, versus 0.101 for global persona identity.Holding model identity fixed, persona recovery reached 0.323 against five-way chance of 0.20; persona prompts shift concept salience within base-model signatures.Source: §5 · p. 9

RQ3: mechanism checks

  • The matched self-description prompt produced 78.3% echo and 1.10 signal-to-chance when vocabulary-cued, versus 4.4% echo and 2.27 when open.Cue stripping reduced lexical echo in all six Infinity-Chat pairs and changed RBO in both directions.Source: §5 · p. 9

RQ4: candidate triage

  • On 12 candidate-rich prompts, conditioned evaluators advanced conditioned-source pools in 11/12 prompts and unconditioned evaluators did so in 8/12.Both cohorts chose the same top-ranked candidate in 7/12 prompts; the module prioritises rare, repeatable candidates for later validation.Source: §5 · p. 9

Discussion

  • Open-ended homogeneity can reflect narrow questions, lexical cueing, shared defaults, or broad prompts with alternatives recoverable through structured probing.The external run covers English-language Infinity-Chat 100; personas are long synthetic identity profiles, and the width score is a composite descriptive heuristic.Source: §5 · p. 10

6 Conclusion

CHOIR moves evaluation from first-pass answers to concepts recurring across repeated elicitation, separating different sources of apparent homogeneity. Its evidence supports claims about elicited concept coverage and salience, while downstream decision benefits remain to be validated.

6 Conclusion

  • Across Infinity-Chat 100 and targeted probes, CHOIR separated narrow prompts, lexical cueing, shared defaults, and broad prompts with recoverable alternatives.Base-model identity was the dominant recoverable structure in conditioned external-prompt analysis, while persona effects appeared as salience shifts within those signatures.Source: §6 · p. 10
  • CHOIR makes complementary concept coverage auditable across models and elicitation conditions, but whether that coverage improves decisions requires task-specific human or expert validation.Future work should test practical consequences across languages and domains.Source: §6 · p. 10

Ethics Statement

The appendix documents participant safeguards, elicitation and extraction procedures, diagnostic prompts, and the external prompt bank. It also describes the synthetic persona profiles used for conditioning.

Ethics Statement

  • The source-blind calibration recruited 25 adults, obtained informed consent, pseudonymised analysis records, and is reported only as a directional ranking-module check.The calibration was not submitted for institutional review; persona conditioning may amplify cultural bias or misrepresent identities, and the five profiles are synthetic.Source: Ethics Statement · p. 10

A Prompt Materials and Question Banks

  • The appendix reproduces the 27-question diagnostic bank and its prompt stems verbatim, including original spelling, with legacy IDs retained for reproducibility.The persona-impact question appears in bare and anchored forms, yielding 28 prompt stems.Source: A Prompt Materials and Question Banks · p. 13

A.1 Elicitation templates

  • Templates elicit ranked lists, with baseline, first-principles depth, and non-anthropomorphic self-description framings used for distinct prompt conditions.Template C results should be interpreted within its explicitly non-adversarial, non-anthropomorphic frame.Source: §A.1 · p. 13

A.2 Concept extraction unit and worked example

  • Each numbered response becomes one concept item; comma-separated phrases stay together, and justifications are stored separately from concept text.Codebook clusters group extracted items, and Smith’s S, RBO, signal-to-chance, and permutation tests operate over cluster-level items.Source: §A.2 · pp. 13–14

A.3 Targeted diagnostic probe bank with legacy IDs

  • The targeted diagnostic bank spans social, memory, self-description, meta-inquiry, control, and personality-impact prompts for mechanism-focused contrasts.Its included questions cover varied tasks such as welfare, functional cognition analogies, medication safety, and personality-prompting effects.Source: §A.3 · pp. 15–17

A.4 Infinity-Chat 100 external prompt set

  • Infinity-Chat 100 is the external 100-prompt representative seed set used by Jiang et al. for intra- and inter-model homogeneity analysis.The appendix reproduces its prompts and compact consensus taxonomy labels, spanning creative, bounded, constrained, and named-answer tasks.Source: §A.4 · pp. 17–19

B Persona Profiles

  • Persona conditioning used five synthetic identity profiles designed to span HEXACO personality space.The full 10,000–30,000 character profiles are in supplementary materials; the main text treats conditioning as an output-level salience perturbation within base-model signatures.Source: B Persona Profiles · p. 21

C Additional Recoverability and Leakage Checks

Additional checks find persona information within model-specific output profiles, while profile-proximate language contributes to the external-bank persona signal. Persona recovery remains above chance after concept stripping.

Additional Recoverability and Leakage Checks

  • 0.323 persona accuracy (bootstrap 95% CI [0.309, 0.337]) exceeded five-way chance of 0.20 in a leave-one-question-out classifier holding model identity fixed.Per-model accuracies ranged from 0.260 for GPT-4.1 to 0.398 for Gemini 3 Flash.Source: C Additional Recoverability and… · p. 21
  • Question-level embedding leakage correlated with conditioned multiplier (Spearman ρ = .311), and per-model leakage correlated with persona recoverability (Spearman ρ = .800).After stripping, within-model persona accuracy was 0.318, 0.303, and 0.284 at thresholds 0.5, 0.4, and 0.3, respectively, above five-way chance in all runs.Source: C Additional Recoverability and… · p. 21

D Ranking Protocol and Human Calibration

The source-blind ranking module filters CHOIR-derived candidate pools through LLM evaluators, narrowing candidates for later inspection and validation.

D.1 Initial human calibration

Human participants matched the conditioned AI cohort’s top-ranked item more often than the unconditioned cohort’s, providing directional calibration of the ranking signal.

D.1 Initial human calibration

  • 112/300 (37.3%) participant choices matched the conditioned AI cohort’s top item, versus 85/300 (28.3%) for the unconditioned cohort (paired exact sign-flip p < .05).Source: §D.1 · p. 21
  • On seven questions where AI cohorts disagreed, participants matched the conditioned cohort’s top item 57/175 (32.6%) versus 30/175 (17.1%; p < .05).Source: §D.1 · p. 21
  • Conditioned-source item selection was near within-question random expectation: 129/300 observed versus 132.9/300 expected.Source: §D.1 · p. 21
  • Matched cue stripping reduced item-level echo in every prompt pair, while signal-to-chance rose in some pairs and fell in others.Source: Figure E.1 · p. 22
  • The ranking figure separates conditioned-source and unconditioned-source advanced pools for LLM-based filtering of elicited concepts.Source: Figure E.2 · p. 22

E Additional Diagnostic and Ranking Figures

The supplementary archive provides machine-readable materials for inspecting reported results and partially reproducing the analysis pipeline.

AI Disclosure

The disclosure describes LLMs as both experimental subjects and research instruments, while assigning design decisions, interpretations, and conclusions to the human authors.

  • Nine base models were queried unconditioned and under five persona-conditioned profiles.Source: AI Disclosure · p. 22
  • LLMs extracted concepts, ranked source-blind candidate pools, and drafted persona profiles; Claude Haiku 4.5 performed concept extraction.Source: AI Disclosure · p. 22
  • A long-context generative agent supported literature search, analysis development, and manuscript preparation, while human authors retained sole responsibility for decisions and conclusions.Source: AI Disclosure · pp. 22–23
Loading 2609.38448v1…