Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation

1 Introduction

• Existing synthesis methods often lack grounding in practical user scenarios, and the paper reports better LLM performance for scenario-grounded instruction data than for ungrounded data.The authors identify the disconnect as a reason synthesized data may not reflect diverse user needs.Source: §1 · p. 1
• MATRIX automatically and continuously generates diverse social scenarios through interactions among agents with varied backgrounds.Its agents have profiles and generated life goals, and a homophily-guided communication protocol groups agents with similar profiles to support scalable, meaningful interactions.Source: §1 · pp. 1–2
• MATRIX-Gen combines simulated scenarios with domain requirements to produce realistic, controllable instruction data for multiple domains.The introduction gives math instructions ranging from elementary arithmetic questions to theoretical proofs as an example.Source: §1 · p. 2
• The proposed framework combines MATRIX with MATRIX-Gen to synthesize post-training data, and experiments report that 20K instruction-response pairs outperform the Llama-3-8B-Instruct model on two benchmarks.The paper presents this result as evidence of the synthesized dataset’s quality.Source: §1 · p. 2
• On AlpacaEval 2 and Arena-Hard, Llama-3-8B-Base post-trained on 20K MATRIX-Gen pairs outperforms Llama-3-8B-Instruct, which was trained on over 10M pairs.The paper also reports comparisons with 20 baselines across 12 benchmarks spanning general and math domains.Source: §1 · p. 2

2 Proposed Post-Training System

• The system first synthesizes social scenarios, then generates post-training data from them, and finally fine-tunes a pre-trained model.The scenario synthesis and data generation stages use the same aligned LLM.Source: §2 · p. 2
• MATRIX generates scenarios from anonymized profiles, agent goals, profile-based communication topology, and agents’ interactions within the simulator.The authors characterize the resulting scenarios as realistic and diverse, with actions resembling human behavior and informative interactions.Source: §2 · pp. 2–3
• MATRIX-Gen retrieves scenarios relevant to human requirements, combines each agent’s persona and action with an instruction-synthesis prompt, and generates instructions and responses.The process calls the aligned LLM to produce the synthesized data.Source: §2 · p. 3
• The system synthesizes SFT, preference-tuning, reasoning, and special-domain datasets from simulated scenarios.The paper names these datasets MATRIX-Gen-SFT, MATRIX-Gen-DPO, and MATRIX-Gen-Reason, alongside special-domain SFT data.Source: §2 · p. 3
• Fine-tuning uses MATRIX-Gen-SFT for supervised fine-tuning, then MATRIX-Gen-DPO for direct preference optimization, yielding the MATRIX-Tuned-Model.DPO is performed on the SFT model.Source: §2 · p. 3

3 MATRIX: Multi-Agent Simulator

• MATRIX takes agent profiles as input and produces text scenarios comprising the actions of groups of agents.The simulator combines real-world-grounded agents with homophily-guided communication.Source: §3 · p. 3

3.1 Real-World-Grounded Agents

• Agents use anonymized real human profiles, historical-tweet memory, generated life goals, and plans to guide purposeful, responsive actions.The paper reports collecting 1,000 profiles from X and using an LLM to anonymize them and remove private information.Source: §3.1 · pp. 3–4

3.2 Homophily-Guided Communication

• Agents with similar profiles are grouped using constrained K-means over profile embeddings, while modulators selectively distribute relevant actions within and across groups.The authors set 200 clusters, with cluster sizes from 1 to 10; the protocol supports pairwise and larger-group interactions.Source: §3.2 · p. 4

3.3 Scenario Generation

• Scenario generation initializes 1,000 profiles and agent plans, runs group interactions until agents finish acting, exchanges information between groups for subsequent scenarios, and stops when goals are fulfilled or enough scenarios are collected.The resulting scenarios are collected from the modulators for post-training data synthesis.Source: §3.3 · p. 5

3.4 Discussions

• MATRIX’s life-goal-driven agents and dynamic interactions yield scenarios ranging from everyday conversation to complex professional tasks.The paper contrasts this with simulators that produce constrained scenarios and simplistic actions, and with PersonaHub, which generates instructions from profiles without agent interaction.Source: §3.4 · p. 5

4 Experimental Results

• The evaluation fine-tuned pre-trained models across instruction, preference, and domain-specific tasks, using 10K samples for two epochs.Closed benchmarks used pass@1; open-ended benchmarks used LLM judges, while safety evaluation included helpfulness, harmlessness, and refusal measures.Source: §4 · p. 5, §4.1 · p. 6

4.2 Evaluation in General Domains

• 31% relative improvement over Magpie and 163% over WildChat on Arena-Hard were reported for MATRIX-Gen-SFT; MATRIX-Gen-DPO and MATRIX-Gen-Reason also consistently outperformed baselines.The SFT comparison used 10K samples; the DPO comparison was based on the MATRIX-Gen-SFT-tuned model.Source: §4.2 · pp. 6–7

4.3 Evaluation in Specific Domains

• MATRIX-Gen-Coding, MATRIX-Gen-MT, and MATRIX-Gen-Safe consistently outperformed task baselines; multi-turn SFT was also reported as more efficient than single-turn SFT.The coding and multi-turn comparisons used 10K samples; the multi-turn finding covered three overarching abilities.Source: §4.3 · p. 7

4.4 Analysis

• Increasing the numbers of agents and scenarios improved generated-data quality and subsequent SFT performance; at 103 agents and 104 scenarios, both benchmarks showed higher scores.The benchmarks were AlpacaEval 2 and Arena-Hard.Source: §4.4 · p. 7
• Homophily-guided communication produced higher-quality scenarios and superior post-training data than random communication or no communication.Source: §4.4 · p. 8

Synthesizing alignment data

• Earlier alignment data came from NLP benchmarks [Wang et al., 2022; Mishra et al., 2022] or user-generated instructions [Chiang et al., 2023a; Zhao et al., 2024; Zhou et al., 2024a].Villalobos et al. (2024) raised concerns about whether human-generated data can scale adequately.Source: §5 · p. 8
• Synthetic-data approaches back-translate web corpora [Li et al., 2023b], prompt new instructions from existing ones [Wang et al., 2023a; Xu et al., 2024a], or complete chat templates [Xu et al., 2024b].The paper says these approaches rely on predefined materials, limiting flexibility and missing real-world context.Source: §5 · p. 8

Multi-agent simulation

• Multi-agent simulators have supported social research [Xie et al., 2024] and LLM evaluation [Lin et al., 2023]; MATRIX targets synthetic-data generation with complex actions and scalable simulations.The passage characterizes prior large-scale scalability research as limited and notes that many simulations run for extended durations.Source: §5 · p. 8

• MATRIX generates realistic, diverse scenarios with scalable communications, which MATRIX-Gen uses to synthesize data for SFT, DPO, and domain-specific applications.Source: §6 · p. 8

• LLM inference costs persist despite sparse group communications; improving communication efficiency remains a direction for supporting larger simulation scales.Source: §7 · p. 9

A Experiments

• Fine-tuning uses FastChat, with training hyper-parameters summarized in Table 4.Source: A Experiments · p. 12, Table 4 · p. 12

A.1 Experiments on Motivation

• Human-like instructions outperform other instruction types in supervised fine-tuning.The comparison is reported in Table 5.Source: §A.1 · p. 12, Table 5 · p. 13

A.2 Experiment Details

• MATRIX-Gen-SFT uses instructions designed for simplicity and diversity, while MATRIX-Gen-DPO uses complex, specialized instructions with chosen aligned-LLM responses and rejected SFT-model responses.MATRIX-Gen-Reason samples coding and math instructions from MATRIX-Gen-SFT.Source: §A.2 · p. 12
• Instructions are classified as realistic when they arise from a specific plausible real-life situation reflecting common human behavior.The experiment uses prompts for instruction classification and varies instruction types such as coding and safety.Source: §A.2 · p. 13, Table 6 · p. 13

A.3 Evaluation Details

• Coding pass@1 evaluation uses a response-generation temperature of 0.2, while safety evaluation uses refusal keywords and helpfulness and harmlessness prompts.Source: §A.3 · p. 13
• Relative property scores normalize methods' property scores to a 0–1 scale for illustrative comparison.The scores compare the property of a dataset generated by the proposed method with that of a baseline.Source: §A.3 · p. 13
• Diversity is measured by average Euclidean distance between training samples, and realism is scored by Llama3-8B-Instruct; higher scores indicate greater diversity and realism.Source: §A.3 · p. 13

B.1 Agent

• Agent profiles are selected by generating 500 tags from 175 seed instructions, retrieving three users per tag, deduplicating profiles, and sampling 1,000 profiles.Source: §B.1 · pp. 13–14
• Less than 0.1% of original entities remain in anonymized profiles, and manual checks of 100 synthesized-instruction samples confirmed that no users could be identified from the instructions.Among Magpie, WildChat, MATRIX-Gen-SFT, and MATRIX-Gen-DPO, the MATRIX-series dataset had the lowest percentage of remaining entities.Source: §B.1 · p. 14

B.2 Modulator

• The modulator uses prompts to identify target agents and filter messages routed among agents.Figure 8 depicts cosine similarities among 100 randomly sampled agents’ profile embeddings.Source: §B.2 · p. 14, Figure 8 · p. 15

B.3 Simulation

• The simulation appendix summarizes randomly sampled agents’ actions and generates scenarios every three steps of simulated modulator actions.Source: §B.3 · p. 15

Quality analysis

• MATRIX-Gen-DPO has significantly more good or excellent instructions than MATRIX-Gen-SFT, while neither dataset has instructions rated very poor.Most instructions in both datasets are rated average or above using Llama-3-8B-Instruct ratings.Source: §C.1 · pp. 15–16

Difficulty analysis

• MATRIX-Gen-DPO contains no very easy instructions and predominantly medium and hard ones, whereas MATRIX-Gen-SFT skews toward easy and medium difficulty.Source: §C.1 · p. 16

C.2 Data leackage analysis

• The analysis embeds training data and test questions, compares L2 distances, and checks both seed profiles and final synthetic datasets for overlap.The examined datasets were MATRIX-Gen-SFT and MATRIX-Gen-Code.Source: §C.2 · pp. 16–17
• No evidence of data leakage was found in MATRIX-Gen-SFT or MATRIX-Gen-Code, although their closest matches covered topics also discussed in test sets.The authors report no significant overlap based on L2-norm comparisons.Source: §C.2 · p. 17

C.3 Matrix-Gen-Reason

• Long reasoning responses may add noise, while very short ones may omit reasoning examples; filtering removes excessively long samples and makes think-length proportions more uniform.The analysis examines think-token lengths across datasets, as shown in Figure 10.Source: §C.3 · pp. 17–18, Figure 10 · p. 19

D Examples of generated instruction [Warning: Potentially Harmful Content!]

• Examples span healthcare improvement planning, personal financial guidance, and coding tasks in the generated instruction sets.Source: D Examples of generated instruction… · pp. 21, 30