Training Socially Aligned Language Models on Simulated Social Interactions

1 INTRODUCTION

• Language models can generate harmful content, reinforce bias, and disseminate disinformation, motivating efforts to promote desirable societal behavior and inhibit undesirable behavior.Source: §1 · p. 1
• SFT can leave models vulnerable to jailbreaks, while optimizing against an imperfect reward model can invite reward gaming or tampering [Subhash, 2023; Krakovna et al., 2020; Lehman et al., 2018; Pan et al., 2022].Source: §1 · pp. 1–2
• SANDBOX simulates social interaction among language-model agents and records demonstrations, collective ratings, detailed feedback, and iteratively revised responses.Source: §1 · p. 2
• SANDBOX is introduced as an open-source platform for simulating society and studying AI behavioral patterns, while Stable Alignment learns from simulated interactions in three stages.Source: §1 · p. 3
• Stable Alignment outperforms existing methods in six alignment benchmarks.The introduction also reports that feedback and revision significantly improve robustness against jailbreaking prompts, and ablations support specialized data preparation for efficient, stable alignment learning.Source: §1 · p. 3
• Feedback and revision significantly boost robustness against jailbreaking prompts, while ablations support specialized data preparation for efficient, stable alignment learning.Source: §1 · p. 3

2 RELATED WORK

• Prior social-simulation work includes generative agents and language-model agents used as human substitutes; this work extends alignment simulation to one hundred LM-based agents.Source: §2 · p. 3
• Alignment research has explored human and AI-generated feedback as alternatives to reward-model supervision, and prior work reports improved safety and alignment from synthetic supervision by larger models.The cited work notes that smaller models may not gain equivalent factuality and reasoning capabilities.Source: §2 · p. 3
• SANDBOX models social interactions with Back-Scatter and constructs three alignment datasets—Imitation, Self-Critic, and Realignment—from 169k samples.Source: Figure 2 · p. 4

3.1 SIMULATING SOCIAL INTERACTIONS IN SANDBOX

• Back-Scatter shares an initial response with nearby agents, uses their ratings and explanations to guide revisions, and retrieves relevant response history through agent memory.Observer agents without memory rate responses for alignment and engagement.Source: §3.1 · pp. 4–5
• Alignment training achieved higher optimality with fewer interactions, while increasing GPT-3 size from 6.8B to 175B yielded no significant improvement.The figure describes the alignment and engagement ratings used to measure optimality.Source: §3.1 · p. 5, Figure 3 · p. 5

3.2 STABLE ALIGNMENT: LEARNING ALIGNMENT FROM SOCIAL INTERACTIONS

• Stable Alignment has three stages—Imitation, Self-Critic, and Realignment—with CPO in Stages 1 and 3 and SFT in Stage 2.Feedback generation acquired in Self-Critic is a prerequisite for Realignment.Source: §3.2 · p. 5, Table 1 · p. 6
• CPO learns from higher-rated responses and unlearns lower-rated ones, with a contrastive margin influenced by the rating difference.The contrastive loss is discounted by λ as alignment progresses.Source: §3.2 · p. 6
• Stable Alignment runs SANDBOX simulation offline and needs no online reward model in memory during training, supporting deployment in resource-constrained settings.The collected data can be used to train directly and asynchronously.Source: §3.2 · p. 6

4 EXPERIMENTS

• The experiments use three 100-agent virtual societies and evaluate six tasks, including 200-sample HH and HH-A, which appends a misaligned response to each instruction.HH simulation questions are excluded from the HH test set.Source: §4 · pp. 6–7, §4.1 · p. 8

4.1 MAIN RESULTS ON ALIGNMENT BENCHMARKS

• Human evaluations find Stable Alignment generally beats existing methods, especially on adversarial HHH-A; it is at least as good as ChatGPT 66% and 69% of the time on HHH and HHH-A, respectively.Figure 4 compares Stable Alignment with six baselines, using ChatGPT as a reference.Source: Figure 4 · p. 8, §4.1 · p. 8
• Stable Alignment achieves the best overall performance across six alignment tasks, including in-domain HH/HH-A and out-of-domain tasks.HH and HH-A use questions sourced from the HH training set; the remaining tasks are out-of-domain.Source: Table 2 · p. 8, §4.1 · p. 8

4.2 STABILITY, EFFICIENCY, AND HYPERPARAMETER OPTIMIZATION OF TRAINING

• Stable Alignment training is as stable as SFT, less noisy than RRHF, and reaches higher reward in fewer steps than TRLX under the same reward model.The authors suggest incremental improvements in interaction-data mini-batches may accelerate alignment learning.Source: Figure 5 · p. 9, §4.2 · p. 9
• The authors recommend λ = 0.2 and N = 3 low-rating responses per mini-batch; excessively large values lower alignment ratings and increase perplexity.Figure 5 examines penalty intensity and the number of low-rating responses in each mini-batch.Source: Figure 5 · p. 9, §4.2 · p. 9

4.1 MAIN RESULTS ON ALIGNMENT BENCHMARKS

• Removing Realignment lowers Stable Alignment’s HH-A score from 8.23 to 6.59, while Self-Critic training generally improves on Imitation Learning.The authors report the Realignment omission’s impact is significant in adversarial settings.Source: §4.1 · p. 9

4.3 LIMITATION

• The approach is limited to text-based interactions, assumes static societal norms, and is evaluated primarily in English.The authors note that non-verbal cues are not interpreted and that further research is needed to validate extension to other languages.Source: §4.3 · pp. 9–10

5 CONCLUSION

• Stable Alignment uses interaction data from simulation to train language models for social alignment.The authors suggest the approach could extend to other domains or modalities and may mitigate privacy concerns associated with data collection in certain sectors.Source: §5 · p. 10

• SANDBOX simulation data may inherit language-model-agent biases, and its learned knowledge may not reflect current societal norms and practices.The authors note that knowledge demonstrations could partially mitigate bias, while real-time web access is one potential response to temporal limitations.Source: ETHICS AND REPRODUCIBILITY STATEMENT · p. 10
• The experiments and analyses are conducted in English, so the authors do not assert that findings apply universally across languages.They state that the framework could potentially be adapted to other languages with appropriate modifications.Source: ETHICS AND REPRODUCIBILITY STATEMENT · p. 10
• Evaluations use publicly available datasets and codebases, compare against published papers and public leaderboards, and include reproducibility scripts in the supplementary materials.Source: ETHICS AND REPRODUCIBILITY STATEMENT · p. 10

• SANDBOX consists of Social Agents, a grid-based Simulated Society, and Back-Scatter social interactions.Most experiments use a 10×10 grid, with each cell representing a Social Agent.Source: §A.1 · p. 16
• Each Social Agent has an internal cache of prior question-answer pairs and an external dictionary of other agents’ feedback and observation scores.The external scores cover engagement and moral value alignment for draft and revised answers.Source: §A.1 · p. 16
• Agents retrieve a historical answer when its question is sufficiently similar to a new query, using the retrieved answer to inform the draft response.The internal memory cache is pre-embedded to obtain semantic similarity scores.Source: §A.1 · p. 16
• Updating internal memory is described as promoting increasingly consistent opinions and contributing to Pareto Optimality in alignment and engagement.The authors state that without memory, agents cannot accumulate interaction experience and societal progress under the SANDBOX rule would be impeded.Source: §A.1 · p. 16
• Observer Social Agents lack memory and rate alignment and engagement before and after revision, while standard agents draft responses, give peer feedback, and revise them.The ratings contribute to the final alignment dataset.Source: §A.1 · p. 16

• A dropout rate activates only a subset of agents within interaction range, modeling selective engagement with nearby people.Source: §A.2 · p. 16
• The center agent revises a draft using its internal memory and aggregated feedback, then stores the revision as a constraint for subsequent interactions.The draft is shared with activated agents, whose feedback is aggregated and returned to the center agent.Source: §A.2 · pp. 16–17
• SANDBOX interaction data is described as more diverse than Alpaca instruction-tuning data and HHH-RLHF binary comparison data.The figure encodes instruction root verbs in its inner circle and direct objects in its outer circle.Source: Figure A2 · p. 17
• Stable Alignment is named for storing final answers that reflect group consensus rather than individual opinion, approximating social value formation through feedback and common ground.The authors connect this process to empathy and to understanding which words and behaviors are appreciated in daily interactions.Source: §A.2 · p. 17
• Recorded interactions yield imitation data for desired and undesired behaviors, self-critic data for value-judgment rationales, and realignment data that previews misaligned behavior before a realigned response.The figure caption describes the pipeline as constructing these three data types from SANDBOX interactions; the release contains approximately 42k samples in version 1.0 and 93.8k in version 2.0.Source: §A.2 · pp. 17–18, Figure A1 · p. 17

• Stable Alignment rewards highest-rated responses and penalizes lower-rated responses, using lower-rated examples from the same mini-batch as its contrastive signal.The figure caption describes supervised fine-tuning loss from highest-rated responses and a rating-modulated penalty loss, with responses for the same question grouped in each mini-batch.Source: §A.3 · p. 18, Figure A3 · p. 18
• The contrastive signal comes from low-rated responses rather than a twin network or shifted embeddings.Source: §A.3 · p. 18
• The margin is dynamically adjusted in proportion to rating differences, unlike the fixed margin of hinge or triplet loss.This is intended to make the model work harder to unlearn lower-rated responses while learning more from the highest-rated ones.Source: §A.3 · p. 18

• The guidance describes honesty and helpfulness as avoiding false information and giving step-by-step explanations.Source: §A.4 · pp. 18–19
• Annotators compare two candidate answers and choose which is more socially aligned.Source: §A.4 · p. 18
• The prompt frames social alignment through harmlessness and lack of gender, ideology, or racial bias.Source: §A.4 · pp. 18–19
• Annotators choose A, B, or neither; the paper reports these choices as win/loss/tie ratios and separately uses 1-to-10 alignment ratings for Table 2.Source: §A.4 · p. 19

A.5 ADDITIONAL HUMAN EVALUATION

• Stable Alignment (M = 5.52, SD = 1.57) and ChatGPT (M = 5.69, SD = 1.54) rated significantly above Alpaca + SFT (M = 4.25, SD = 1.93) for harmlessness (p < .001).Stable Alignment was similar to ChatGPT in all evaluated aspects except engagement, where it scored higher (M = 4.68, SD = 1.78 vs. M = 4.15, SD = 1.86; p < .001).Source: §A.5 · p. 19, Figure A4 · p. 19

A.6 SAMPLE GENERATION

• In the fingerprint-removal example, Alpaca and Alpaca + SFT give actionable guidance, while Stable Alignment and ChatGPT refuse; Stable Alignment adds a more detailed safety explanation.Source: §A.6 · p. 19, Table A1 · p. 20
• For the bank-robbery jailbreak prompt, Alpaca gives an actionable plan, while Stable Alignment and ChatGPT refuse; their robustness is described as similar.Source: §A.6 · p. 20, Table A2 · p. 21