Source-linked AI summary
Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration
Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, Yulia Tsvetkov
TL;DR
Human preferences vary across communities, but alignment methods may not represent this pluralism and some existing approaches do not apply to black-box LLMs. Modular Pluralism adds specialized community models to a base LLM and uses them in different ways for pluralistic alignment. Experiments report advances across pluralism objectives, and analysis indicates that adding community models can patch underrepresented groups.
Problem
Human values and preferences vary across communities, while existing alignment approaches may harm pluralism and are not directly applicable to black-box LLMs.
Source: §1 · p. 1Method
Modular Pluralism connects a black-box base LLM with specialized community LMs whose comments support Overton, steerable, or distributional collaboration.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Adding a community LM representative of an underrepresented culture and its values can patch representation gaps in LLMs.
Source: §7 · p. 9The approach relies on community-representative training corpora, whose large-scale collection may be challenging, while intersectional communities may bring additional challenges.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
While existing alignment paradigms have been integral in developing large language models (LLMs), LLMs often learn an averaged human preference and struggle to model diverse preferences across cultures, demographics, and communities. We propose Modular Pluralism, a modular framework based on multi-LLM collaboration for pluralistic alignment: it "plugs into" a base LLM a pool of smaller but specialized community LMs, where models collaborate in distinct modes to flexibility support three modes of pluralism: Overton, steerable, and distributional. Modular Pluralism is uniquely compatible with black-box LLMs and offers the modular control of adding new community LMs for previously underrepresented communities. We evaluate Modular Pluralism with six tasks and four datasets featuring questions/instructions with value-laden and perspective-informed responses. Extensive experiments demonstrate that Modular Pluralism advances the three pluralism objectives across six black-box and open-source LLMs. Further analysis reveals that LLMs are generally faithful to the inputs from smaller community LLMs, allowing seamless patching by adding a new community LM to better cover previously underrepresented communities.
1 Introduction
Human values and preferences vary across communities, while many alignment methods require retraining that is unsuitable for black-box LLMs and costly for patching representation gaps. Modular Pluralism combines a base LLM with specialized community models to support three forms of pluralism, with reported improvements across their objectives.
1 Introduction
- Human preferences vary by community, culture, demographics, and ideology, making a single averaged preference an incomplete target for alignment.The introduction also notes that existing alignment procedures might harm pluralism.Source: §1 · p. 1
- Existing approaches may require retraining or realignment, which is not directly applicable to black-box LLMs and is expensive for patching newly identified representation gaps.Some widely used LLM services are proprietary and black-box.Source: §1 · p. 1
- The framework connects a black-box LLM with specialized community LMs, using their comments to support Overton, steerable, and distributional pluralism.The LLM summarizes viewpoints, selects a community comment for an attribute, or aggregates distributions conditioned on community comments.Source: §1 · pp. 1–2, Figure 1 · p. 2
- Modular Pluralism improves value coverage by 68.5% on average, steerability by 26.6% and 10.4% of cases, and distributional alignment by at least 10.9%.The paper also reports that a new community LM can be added to better cover underrepresented communities.Source: §1 · p. 2
2 Methodology
Modular Pluralism trains community-specific language models and uses their comments as inputs to a base LLM. The LLM’s role varies by objective: summarize perspectives, select a fitting community, or aggregate community-conditioned answer distributions.
Modular Framework
- A pool of community LMs is fine-tuned on community corpora, generates comments for a query, and supplies them to an LLM that can be accessed as a black box.The framework assumes access to the LLM’s outputs and token probabilities.Source: §2 · pp. 2–3
Overton Pluralism
- For Overton pluralism, all community LMs comment on the query and the LLM synthesizes their viewpoints into a coherent response.The prompt encourages faithful representation of the diverse comments.Source: §2 · p. 3
Steerable Pluralism
- For steerable pluralism, the LLM selects the community comment best reflecting a requested attribute, then generates a response conditioned on that comment and attribute.The selection prompt asks which comment best reflects the specified attribute.Source: §2 · p. 3
Distributional Pluralism
- For distributional pluralism, the LLM produces a separate answer distribution for each community comment and aggregates them using community priors that sum to 1.The priors can represent population proportions, such as the shares of registered Democrats, Republicans, and independents.Source: §2 · p. 3
3 Experimental Settings
The experiments evaluate pluralistic alignment across six tasks and four English-language datasets, covering value coverage, attribute steering, and population-level response distributions. Results include higher value coverage, human- and GPT-4-preferred responses, and stronger steerable classification performance in the reported comparisons.
Models and Baselines
- The study uses six open and proprietary LLMs, fine-tunes community models on perspective-laden corpora, and compares against vanilla, prompting, and mixture-of-experts baselines.The main paper focuses on Llama2-13B and ChatGPT; the other model results are in the appendix.Source: §3 · p. 3
Tasks and Datasets
- Six tasks across four English datasets evaluate three pluralism modes, including attribute-conditioned OpinionQA responses and MoralChoice answer distributions for low- and high-ambiguity scenarios.GlobalOpinionQA compares model distributions with country survey responses using Jensen–Shannon distance.Source: §3 · pp. 3–5
- The aligned Modular Pluralism model consistently achieves the best Value Kaleidoscope steerable performance, with up to 23.8% higher balanced accuracy and 21.8% higher Macro-F1 than the second-best model.The table distinguishes binary classification from three-way classification, which adds the “either” cases.Source: Table 1 · p. 4
- In Overton NLI evaluation, Modular Pluralism improves value coverage over the strongest baseline by 27.8% and 50.3% for the two LLMs.The results are presented in Figure 2.Source: Figure 2 · p. 4
- With ChatGPT, Modular Pluralism has 16.5% and 45.8% higher win rates than the strongest baseline in human and GPT-4 evaluations.The results are presented in Figure 3.Source: Figure 3 · p. 5
4 Results
MODULAR PLURALISM improves results across Overton, steerable, and distributional tasks. The experiments also show that community-model comments are incorporated alongside new content from the LLM.
4 Results
- In Overton NLI evaluation, MODULAR PLURALISM achieved the highest value coverage on both LLMs, improving by up to 50.3%.Source: §4 · p. 5
- MODULAR PLURALISM achieved higher win rates than all three baselines in human and GPT-4 evaluations, with rates 45.8% and 16.5% higher, respectively.Together with the NLI evaluation, the authors characterize its responses as more Overton-pluralistic and better-rounded.Source: §4 · p. 5
- In Value Kaleidoscope, MODULAR PLURALISM improved over baselines by up to 23.8% in balanced accuracy and 21.8% in Macro-F1 for three-way classification, and by 15.1% on average in binary classification.Source: §4 · p. 5
- On steerable OpinionQA, aligned MODULAR PLURALISM improved overall accuracy by 8.9% over the strongest baseline, with a 12.8% improvement for political-party attributes.Source: §4 · p. 5, Table 2
- 16.1% lower overall distance than the strongest baseline on average: MODULAR PLURALISM moved aligned and unaligned LLMs toward the center of their contrasting MoralChoice patterns.Source: Figure 4 · p. 6, §4 · p. 6
- With default community LMs based on US news media and subreddits with mainly US- and West-centric content, MODULAR PLURALISM’s performance gains were largest for the United States (25.8%) and smallest for Nigeria (9.3%) (Sorensen et al., 2024b).Source: §4 · p. 7
- 14.9% average reduction in J-S distance on GlobalOpinionQA; gains ranged from 25.8% for the United States to 9.3% for Nigeria.Source: §4 · pp. 6–7, Table 3 · p. 7
- Community comments received 40%–60% coverage in final responses, while 20%–40% of response sentences were new content added by the LLM.Source: Figure 5 · p. 7
5 Analysis
The analysis finds that adding cultural community models can improve alignment for underrepresented communities while preserving alignment elsewhere. It also examines how community comments and LLM-generated content contribute to responses.
5 Analysis
- The analysis tests whether LLM responses reflect community LM comments and whether comments from different community LMs may conflict with the LLM’s own viewpoints.Source: §5 · p. 7
- Community LM comments had 51.2% average coverage in final responses, while 33.8% of sentences contained new LLM-added content.Reddit-based community LMs had 57.7% average coverage versus 44.7% for news-based LMs; stronger LLMs showed higher comment coverage and new-content rates.Source: §5 · pp. 7–8
- Cultural community LMs hurt alignment on US-centric OpinionQA, but a mixture of perspective and cultural LMs worked best on other tasks such as Distributional-GlobalOpinionQA.Source: Table 4 · p. 8, §5 · p. 8
- Adding an Asian or African cultural community LM reduced J-S distance by an average of 5.2% or 6.7%, respectively, for the corresponding countries, while preserving alignment for unrelated communities.Source: Figure 6 · p. 8, §5 · p. 8
6 Related Work
Related work frames pluralistic alignment as a response to averaged human preferences and describes three pluralism objectives. The paper positions its modular collaboration framework as a black-box-compatible alternative to approaches requiring white-box access.
6 Related Work
- Earlier alignment approaches include reward models trained with human feedback and preference optimization, alongside self-alignment, iterative alignment, and self-play methods.Source: §6 · pp. 8–9
- Prior work notes that alignment objectives can average human preferences despite differences across cultures, demographics, and perspectives, and identifies Overton, steerable, and distributional pluralism objectives.Source: §6 · p. 9
- MODULAR PLURALISM operationalizes the three objectives by integrating smaller community-specialized LMs with black-box LLMs, unlike prior proposals requiring white-box access for RLHF tuning or parameter merging.Source: §6 · p. 9
7 Conclusion
The paper proposes Modular Pluralism, which augments general-purpose LLMs with specialized community LMs that collaborate in distinct modes. Experiments report advances in pluralistic alignment, and modularity allows underrepresented communities to be added through new community LMs.
7 Conclusion
- Modular Pluralism augments general-purpose LLMs with smaller, specialized community LMs that collaborate in distinct modes.The framework is presented as a multi-LLM collaboration approach to pluralistic alignment.Source: §7 · p. 9
- Extensive experiments show that Modular Pluralism advances pluralistic alignment across numerous models and evaluation datasets.Source: §7 · p. 9
- Adding a representative community LM can patch coverage of previously underrepresented communities.The conclusion describes this as a benefit of the framework’s modularity.Source: §7 · p. 9
Limitations
The paper’s instantiation focuses on perspective-informed and culture-informed communities, and its evaluations model pluralism across four datasets and six schemes. The framework depends on community-representative training corpora and adds inference-time computation.
Limitations
- The study mainly instantiates community pluralism through perspective-informed and culture-informed groups, though other community definitions may also matter.Source: Limitations · p. 9
- Modular Pluralism incurs extra inference-time computation because community LMs are also prompted.The authors envision using smaller community LMs to reduce this cost.Source: Limitations · p. 9
- The evaluation uses four datasets and six schemes focused on value, culture, and perspective plurality, with future work proposed on real-world evaluations with human participants.Source: Limitations · p. 9
- Training community LMs relies on community-representative corpora, whose large-scale collection may be challenging; intersectional communities may bring additional challenges and opportunities.Source: Limitations · p. 9
Ethics Statement
The authors identify dual-use risks from community LMs and warn that an imbalanced or poorly designed pool could reinforce stereotypes or introduce bias.
Ethics Statement
- Hateful fringe communities could seek better representation, including through community LMs trained on hateful social media content.Source: Ethics Statement · p. 10
- The authors argue that deployed community LMs should not be specially engineered for malicious purposes.Source: Ethics Statement · p. 10
- An imbalanced or ill-designed community-LM pool might reinforce stereotypes or introduce bias, motivating broader community representation.Source: Ethics Statement · p. 10
A Analysis (cont.)
The analysis reports value-coverage gains with an additional entailment model, illustrates how community comments contribute distinct perspectives, and examines entropy, model size, and compute. It also reports results for other LLMs and estimates the added compute under its default settings.
Another NLI Model
- Table 5 results with WANLI reaffirm improved value coverage and Overton pluralism on Value Kaleidoscope.WANLI is added alongside VitaminC, which focuses on fact-based entailment.Source: A Analysis (cont.) · p. 14
Qualitative Analysis
- In two examples, community comments add perspectives on animal-care decision-making, animal-welfare laws, and possible backfire from taking down 4chan; the LLM combines these with its own points.Source: A Analysis (cont.) · p. 14
Entropy and Distributional Pluralism
- For aligned LLMs, Modular Pluralism yields higher entropy through aggregation of community-specific distributions; for unaligned LLMs, entropy remains similar to baselines.The authors characterize this as curbing over-confidence for aligned models while reflecting steerability rather than an entropy shortcut for unaligned models.Source: A Analysis (cont.) · p. 14, Table 6 · p. 14
Model Sizes
- 17.2% average improvement is reported for the smallest LLAMA2-7B model, while Modular Pluralism often improves most with the largest 70B model.The table caption describes performance across LLAMA2 model sizes.Source: A Analysis (cont.) · pp. 14–15, Table 7 · p. 15
Other LLMs
- Other LLM results are reported for Overton and steerable pluralism on Value Kaleidoscope and distributional pluralism on MoralChoice.The supplied table caption identifies Table 8 as reporting Overton value coverage for other LLMs.Source: A Analysis (cont.) · p. 15, Table 8 · p. 15
Computational Costs
- 10.4% extra compute is estimated for GPT-4 with six 7B community LMs, using LLaMA3-405B as an approximation; steerable mode adds 1.7% when only one 7B model is called.The authors note that the exact size of GPT-4 is unknown and that community LMs need not be called every time.Source: A Analysis (cont.) · p. 15
B Experiment Details
The experiments evaluate three forms of pluralistic alignment across six tasks and four datasets, using automated and human judgments, task-specific baselines, and multiple open and black-box LLMs.
B Experiment Details
- For Overton evaluation, 3,132 VK situations are scored for whether responses reflect associated values, with NLI entailment judgments aggregated across values and situations.Human and GPT-4 pairwise evaluations provide additional Overton assessments.Source: B Experiment Details · p. 15
- Human annotators assess 100 response pairs, while GPT-4 assesses 600 pairs, judging which response better reflects pluralistic values or whether they tie.Five annotators participate in the human evaluation.Source: B Experiment Details · p. 15
- Steerable evaluation uses 21,840 Value Kaleidoscope pairs and 22,378 OpinionQA questions; distributional evaluation uses MoralChoice and 28,763 GlobalOpinionQA questions.MoralChoice compares answer distributions with low-ambiguity consensus or high-ambiguity uncertainty targets using Jensen–Shannon distance; GlobalOpinionQA prompts specify a country.Source: B Experiment Details · pp. 15–16, Table 10 · p. 16, Table 9 · p. 16
- For LLAMA2-70B, evaluation uses a random 20% data sample because of computing constraints.Source: B Experiment Details · p. 16
- The comparisons include vanilla prompting, a pluralism prompt, and a mixture-of-experts baseline across six LLM configurations, with an uncertain assumption that gpt-3.5-turbo is davinci-002’s aligned version.The six configurations include LLAMA2-13B, CHATGPT, LLAMA2-7B, LLAMA2-70B, LLAMA3-8B, and GEMMA-7B.Source: B Experiment Details · pp. 16–17
- Community LMs are LoRA-finetuned on community-representative corpora; the distributional demographic prior is uniform by default, and generation is capped at 512 new tokens.The implementation uses one training epoch, 16-bit precision, and a learning rate of 1e-6.Source: B Experiment Details · p. 17