Source-linked AI summary
Collective Constitutional AI: Aligning a Language Model with Public Input
Saffron Huang, Divya Siddarth, Liane Lovitt, Thomas I. Liao, Esin Durmus, Alex Tamkin, Deep Ganguli
TL;DR
Language-model developers are not the only stakeholders whose preferences may shape model behavior, but the field lacks a clear process for incorporating collective public input. The paper proposes CCAI, a multi-stage public-deliberation and Constitutional AI process, and finds that its Public model reduces social biases while retaining comparable evaluated capabilities to a Standard baseline.
Problem
The research community lacks a well-defined process for eliciting collective public input and incorporating it into language-model training, despite growing calls to account for broader public preferences and values.
Method
CCAI uses Polis deliberation to source public preferences, converts them into a constitution, and applies Constitutional AI to fine-tune a language model.
Results
The Public model reduces social biases across all tested categories while remaining equivalent to the Standard model in language understanding, helpfulness, and harmlessness.
Takeaways & Limitations
The results support the feasibility and potential benefit of participatory methods for incorporating public input into the normative principles used to fine-tune language models.
Takeaways & Limitations
The study does not perform potentially useful analyses of Pass/Unsure responses or demographic differences because of scope, privacy, and ethical constraints.
Abstract
from arXiv · showhide
There is growing consensus that language model (LM) developers should not be the sole deciders of LM behavior, creating a need for methods that enable the broader public to collectively shape the behavior of LM systems that affect them. To address this need, we present Collective Constitutional AI (CCAI): a multi-stage process for sourcing and integrating public input into LMs-from identifying a target population to sourcing principles to training and evaluating a model. We demonstrate the real-world practicality of this approach by creating what is, to our knowledge, the first LM fine-tuned with collectively sourced public input and evaluating this model against a baseline model trained with established principles from a LM developer. Our quantitative evaluations demonstrate several benefits of our approach: the CCAI-trained model shows lower bias across nine social dimensions compared to the baseline model, while maintaining equivalent performance on language, math, and helpful-harmless evaluations. Qualitative comparisons of the models suggest that the models differ on the basis of their respective constitutions, e.g., when prompted with contentious topics, the CCAI-trained model tends to generate responses that reframe the matter positively instead of a refusal. These results demonstrate a promising, tractable pathway toward publicly informed development of language models.
1 INTRODUCTION
The paper introduces Collective Constitutional AI (CCAI), a multi-stage process for incorporating collectively sourced public preferences into language-model training. It applies this process to compare a Public-constitution model with a Standard-constitution baseline while documenting key design choices and limitations.
- Motivation: The research community lacks a well-defined process for eliciting collective public input and incorporating it into language-model training.
- Approach: CCAI combines Polis-based online deliberation for sourcing public preferences with Constitutional AI for fine-tuning models to follow those preferences.The resulting constitution expresses desirable behavior as natural-language principles.
- Design considerations: The framework explicitly defines the relevant public because preferences, values, and harms are diverse, subjective, contextual, and potentially contested.
- Experiment: The experiment created a Public constitution from a representative sample of U.S. adults and compared its model with a Standard constitution used by Anthropic for Claude models.
- Analyses: The paper reports qualitative analyses of constitution and output differences alongside quantitative analyses of similarities and differences between the two models.
- Limitations: A key limitation is the absence of a direct metric for measuring a model’s adherence to constitutional principles.The authors provide anonymized public-input data and a notebook to facilitate critique and further work.
2 RELATED WORK
The paper builds on Constitutional AI, value-alignment research, participatory AI, and democratic theories of public input. It situates CCAI within efforts to increase affected communities’ influence over language-model development and to move beyond purely mechanistic preference aggregation.
- Constitutional AI: Constitutional AI fine-tunes instruction-following language models to adhere to high-level ethical principles written as a constitution.
- Value alignment: Value-alignment research examines whether artificial-intelligence systems operate consistently with human values, ethics, and preferences.CCAI extends this framing by recognizing that alignment with public input depends on context.
- Participatory AI: Participatory AI research seeks to increase the involvement and influence of communities affected by or contributing to AI systems.Relevant communities include end users, marginalized voices, data workers, and people harmed by model biases.
- CCAI framework: The paper presents Figure 1 as a flowchart of CCAI stages and significant design decisions intended to support adaptation across contexts.
- Democratic theories: Research on public-input processes draws on social-choice theory’s quantitative preference aggregation and deliberative theory’s emphasis on more participatory approaches.
3 METHODS
The framework moves from representative public participation through deliberative input processing and consensus-based statement selection to constitution-based model training. It explicitly exposes subjective choices that affect how well the resulting principles represent public preferences.
- Framework: The framework guides the process from selecting a representative population and eliciting input to preparing principles and training and evaluating a model.The process includes participant selection, public input elicitation, input collation, and model training.
- Participant selection: The study sampled 1,002 U.S. adults across age, gender, income, and geography, screening for familiarity with generative AI.The screening was introduced after pilot data issues without the filter.
- Public input: Participants used a modified Polis process to vote on statements and submit new ones after casting at least 30 votes.The modification was intended to reduce duplicate submissions.
- Public input: Researchers supplied seed statements and moderated duplicates, nonsense, hateful or offensive, irrelevant, and incomprehensible submissions, sometimes rewriting them.The seed statements helped participants produce in-scope, appropriately formatted contributions, while moderation involved judgment.
- Input transformation: Statements were selected using group-aware consensus, which favors agreement across opinion groups rather than support from only a majority.Opinion groups were identified with principal components analysis and k-means clustering; the process produced two groups.
- Input transformation: The Public constitution matched the Standard constitution's 95 unique ideas, while deduplication, aggregation, threshold choice, and principle rewriting introduced researcher discretion.These choices were recognized as affecting the validity of the constitution's representation of public preferences.
4 RESULTS
The study finds that collectively sourced principles produced model behavior aligned with constitutional differences, with lower bias and preserved core capabilities relative to the Standard model. Public input also yielded principles emphasizing objectivity, accessibility, and positive behavior, while political-opinion representativeness decreased.
- Quantitative Analysis of the Public Statements: Participants submitted 275 statements, with average group-aware consensus of 0.64, median 0.70, minimum 0.04, and maximum 0.96.The top statements were selected until they covered 95 different ideas.
- Qualitative Analysis of the Constitutions: The Public constitution differed from the Standard constitution through greater emphasis on objectivity, impartiality, accessibility, and promoting desired behavior.Roughly 50% of concepts overlapped between the constitutions.
- Quantitative Model Evaluations: The Public and Standard models performed essentially equivalently on MMLU and GSM8K, indicating that differing normative principles did not measurably change these capabilities.Both models also performed roughly equivalently to Claude Instant 1.2.
- Quantitative Model Evaluations: The Public model showed lower bias across all nine social dimensions than the Standard model, and lower bias than Claude Instant 1.2 in six dimensions.The authors associate the larger disability-bias decrease with the Public constitution’s emphasis on accessibility.
- Quantitative Model Evaluations: Public and Standard models did not significantly differ in relative representation of U.S. political ideologies, but Public-model responses were consistently 2 to 3 percentage points less representative across groups.The authors attribute this pattern to more frequent refusals to provide subjective political opinions.
- Qualitative Model Evaluations: Qualitative comparisons suggest Public-model behavior aligned with constitutional differences, including more substantive answers to controversial evidence-based questions and more positive reframing of contentious matters.The authors note that definitive measurement of adherence to the constitutions remains unavailable.
5 LIMITATIONS AND FUTURE WORK
The study identifies limitations in participant scope, principle elicitation, model training, and evaluation, while outlining concrete directions for future work.
- The participant sample is small and not globally representative, so international testing could yield different principles and model behaviors.
- Conflicting principles were included independently, leaving trade-offs to the model rather than resolving them through human input.
- Using the same harmful-prompt dataset for both models may have produced less relevant response pairs for the Public constitution.
- The evaluation relies on automated metrics and human ratings, which may not capture intended harms or how end users perceive model differences.
- The evaluations do not directly assess adherence to constitutional principles, motivating new metrics, qualitative scenarios, and statistical measures.
- Future work could improve public input through behavioral examples, participant deliberation, structured principles, and more precise elicitation methods.
- The study made subjective choices when formatting and selecting free-text statements, without weighting principles by their importance to participants.
- Additional analyses of Pass / Unsure responses and demographic differences were not conducted because of scope, privacy, and ethical concerns.
6 DISCUSSION AND CONCLUSION
The authors conclude that participatory input can shape a language model’s normative principles and behavior. The Public model reduced tested social biases while maintaining equivalent language understanding, helpfulness, and harmlessness.
- The participatory method produced a model that seems to reflect preferences and values from a representative sample of the U.S. public.
- The resulting public input showed relatively low polarization and high consensus, with greater emphasis on objectivity and accessibility than the Standard constitution.
- The Public model reduced social biases across all tested categories while remaining equivalent in language understanding, helpfulness, and harmlessness.
- The work may be one of the first instances of a group using an online public-input process to direct a language model’s behavior.
7 ETHICAL CONSIDERATION STATEMENT
The authors discuss the ethical responsibility of shaping public-facing language models and acknowledge limits in representation, privacy, and the broader impacts of their process.
- Shaping public-facing language models involves normative choices with significant implications for people’s lives.
- Because the study focused on the U.S. public, its sample cannot represent everyone affected across geographic and cultural contexts.
- The researchers avoided demographic disaggregation because they judged it nonessential to this work and potentially risky for privacy and ethics.
- The process does not address other AI-development decisions, and more realistic testing is needed to understand impacts across contexts.
A.1 Author Contributions
The work was jointly led and designed, with authors dividing responsibilities across writing, public-input elicitation, project management, model training, evaluation, and experimental framing.
- Saffron Huang, Divya Siddarth, Liane Lovitt, and Deep Ganguli jointly led and designed the work.
- Huang led writing and framing, while Huang and Siddarth ran input elicitation, Lovitt managed the project, and Lovitt analyzed constitutions qualitatively.
- Thomas I. Liao trained the models and ran quantitative and qualitative evaluations, while Esin Durmus conducted political-ideology evaluation and helped design the experiments.
- Alex Tamkin contributed to paper framing and provided guidance on experimental design and analysis.
A.2 Screening Questions
Participants were screened using recent discussions and news consumption about generative AI or ChatGPT. Those meeting both criteria were invited because pilots found this reduced spammy submissions.
- Screening questions: Screening asked about topics discussed with friends or family during the previous month.Generative AI/ChatGPT was one listed response option.
- Screening questions: A second question asked which news articles participants had read during the previous four months.Generative AI/ChatGPT was again one listed response option.
- Eligibility: Participants selecting Generative AI/ChatGPT in both questions were invited to the public input process.The two criteria were based on topic discussion and news consumption.
- Eligibility: Pilot experiments found that omitting these screening criteria produced more spammy submissions.The screening criteria were therefore used to improve submission quality.
A.3.1 Instructions.
Participants saw instructions and the Polis voting mechanism used in the public input process.
- Instructions: Figure 6 shows the instructions and Polis voting mechanism presented to participants.The figure is a screenshot of the participant-facing interface.
A.3.2 Frequently Asked Questions.
The participant interface included collapsible Frequently Asked Questions information boxes and a contact form for questions or feedback.
- Frequently Asked Questions: Figure 7 shows collapsible Frequently Asked Questions information boxes included in the interface.The boxes provided expandable information for participants.
- Contact Form: Figure 8 shows a contact form for participants to ask questions or give feedback.The form provided a channel for participant communication.
A.3.3 Contact Form.
The appendix documents how public statements were seeded, deduplicated, aggregated, converted into constitutional principles, and contrasted with model responses under the Public and Standard constitutions.
- A.4 Seed Statements: The process began with 21 diverse seed statements covering behavioral, rights-related, LM-specific, existential-risk, and controversial themes.Seven seed statements were inspired by the Standard constitution.
- A.4 Seed Statements: Examples of seed principles addressed helpfulness, controversial political issues, discrimination, aggression, harm, equality, and human rights.The statements were tagged by intended behavior or topic category.
- A.5 Deduplicating and Aggregating Statements: Duplicate or closely related public statements were grouped and rewritten into aggregated principles, including accessibility, clarification, and lawfulness.Examples include combining three disability-related statements and two clarification statements.
- A.5 Deduplicating and Aggregating Statements: Other aggregations consolidated helpfulness, kindness, honesty, objectivity, factuality, reliability, and accessibility themes.The resulting statements included objective information, truthful answers, and disability access.
- A.6 Public Constitution: The Public constitution was reproduced as a set of response-selection instructions, including minimizing danger or hate and maximizing reliability, truth, clarity, good qualities, fairness, and impartiality.These principles operationalized the public input as constitutional guidance.
- A.10 Example Prompt-Response Pairs That Differentiate Public and Standard Constitution Models: For contentious matters, the Public model reframed responses positively, while the Standard model more often declined to make a negative value judgment.The example contrasts positive reframing with caution and refusal.
- A.10 Example Prompt-Response Pairs That Differentiate Public and Standard Constitution Models: The Public model answered some evidence-based controversial questions more often, favoring impartiality or scientific grounding where the Standard model refrained.The climate-change example contrasts reporting scientific consensus with declining to make a judgment.
- A.10 Example Prompt-Response Pairs That Differentiate Public and Standard Constitution Models: On harmful topics, the Public model more often attempted to prevent harm, whereas the Standard model more often disengaged.The same contrast is reported across the repeated table passages.