Source-linked AI summary
SynCo: Data Synthesis Co-Training for Self-Evolving LLMs via Multi-Agent Reinforcement Learning
Wei Yang, Shawn Li, Yuehan Qin, Yawei Wang, Mingxi Wang, Shixuan Li, Tiankai Yang, Jiate Li, Jesse Thomason, Xuezhe Ma, Yue Zhao
TL;DR
Static or separately updated training data can fall out of step with an agent’s changing capabilities. SYNCO jointly trains a task Synthesizer and Reasoner from shared online interaction, adapting the training distribution as the Reasoner evolves. Across eight mathematical benchmarks, it improves over a frozen Synthesizer, with gains primarily from previously unsolved problems.
Problem
Static or separately updated training experience can become mismatched to changing agent capabilities, limiting sustained improvement.
Source: §1 · pp. 1–2Method
SYNCO jointly trains a Synthesizer that generates capability-conditioned tasks and a Reasoner that learns from them, using shared rollout outcomes to update both agents.
Source: §1 · p. 2Results
Across eight mathematical benchmarks, SYNCO improves over a frozen Synthesizer, with gains primarily arising from previously unsolved problems.
Source: §5 · p. 11Takeaways & Limitations
Outcome-grounded teachability is strongly aligned with gradient-bearing interactions, while static task properties such as quality or novelty are insufficient to identify useful training experience.
Source: §5 · p. 11The reported experiments cover eight mathematical reasoning benchmarks.
Source: §5 · p. 11
Abstract
from arXiv · showhide
Self-evolving LLM agents promise to improve autonomously through continual interaction and learning, reducing their dependence on manually curated supervision. Realizing this promise requires not only updating the agent, but also evolving its training experience as its capabilities change. However, most existing pipelines rely on static datasets or separately updated synthesis models, causing previously useful tasks to become trivial while overly difficult tasks remain uninformative. This growing mismatch between agent capability and training experience limits sustained self-improvement. To address this problem, we propose SynCo, an agentic data synthesis co-training framework for self-evolving LLMs based on multi-agent reinforcement learning. SynCo jointly optimizes two independently parameterized agents: a Synthesizer that constructs training tasks from the Reasoner's evolving capability state, and a Reasoner that learns from the resulting experience. Each synthesized task induces multiple Reasoner rollouts whose outcomes provide complementary rewards to both agents. Correctness feedback improves the Reasoner, while task quality, answer reliability, and outcome-grounded teachability guide the Synthesizer. Their updates are fed back into subsequent synthesis rounds, allowing the task-solving policy and its training distribution to evolve together. Extensive experiments across eight mathematical reasoning benchmarks demonstrate that SynCo substantially outperforms a broad range of existing synthetic-data methods and controlled baselines, achieving the strongest overall performance while deriving most of its gains from previously unsolved problems.
1 Introduction
Self-evolving agents need training experience that changes with their capabilities, yet many pipelines use static data or update synthesis separately from learning. SYNCO jointly trains a Synthesizer and Reasoner through shared online interaction, with experiments showing gains over a frozen Synthesizer that mostly come from previously unsolved problems.
1 Introduction
- Static or separately refreshed training data can become mismatched to changing agent capabilities, leaving formerly useful tasks trivial and overly difficult tasks uninformative.The introduction links this mismatch to limits on training efficiency and sustained improvement.Source: §1 · p. 1
- Prior adaptive synthesis methods may respond to learner failures or difficulty, but many still generate from an earlier checkpoint and train afterward rather than jointly optimizing synthesis and task learning.The paper frames joint optimization from shared online interaction as the gap SYNCO addresses.Source: §1 · p. 2
- SYNCO jointly optimizes a Synthesizer and Reasoner in one online multi-agent process, using interaction feedback to adapt both task solving and task generation.The Synthesizer creates tasks from the Reasoner’s evolving capability state; correctness rewards guide the Reasoner, while quality, reliability, and teachability guide the Synthesizer.Source: §1 · p. 2
- Experiments across mathematical reasoning benchmarks show SYNCO outperforms a frozen-Synthesizer baseline, with most gains arising from previously unsolved problems.The contribution summary identifies this as a consistent result across diverse benchmarks.Source: §1 · p. 2
2 Related Work
Related work spans synthetic-data generation and self-evolving agents. SYNCO is positioned among approaches that adapt training experience, while emphasizing joint online optimization of synthesis and learning.
2 Related Work
- Synthetic-data methods span instruction bootstrapping, reasoning-oriented pipelines, and approaches targeting realism, diversity, verifiability, or learner relevance.Examples include Self-Instruct, WizardLM, STaR, MetaMath [Yu et al., 2023], and DeepSeekMath [Shao et al., 2024].Source: §2 · p. 2
- Many related data-generation pipelines construct offline corpora or adapt generation separately from downstream policy optimization.The cited work includes methods spanning synthetic supervision for reasoning, tool use, web interaction, and executable environments.Source: §2 · p. 3
- Self-evolving frameworks couple experience generation with learner improvement through evolving proposer, teacher, curriculum, environment, memory, or workflow roles.Examples include R-Zero [Huang et al., 2025], Socratic-Zero [Wang et al., 2025a], and Agent0 [Xia et al., 2025].Source: §2 · p. 3
3 Method
SYNCO jointly trains a Synthesizer that adapts task generation to the Reasoner and a Reasoner that learns from generated tasks. Shared rollout outcomes guide both agents, and feedback is retained to renew the context and future task distribution.
3.1 Problem Formulation: Agentic Data Synthesis Co-Training
- SYNCO formulates synthesis and reasoning as coupled policies trained from the same online batch, with the Synthesizer setting the Reasoner’s task distribution and the Reasoner supplying synthesis feedback.Both policies use agent-specific group-relative objectives rather than sequential synthesis followed by learning.Source: §3.1 · pp. 3–4
3.2 Capability-Conditioned Agentic Data Synthesis
- The Synthesizer conditions task generation on a dynamic context of learner capabilities, task-pool coverage, and rejected-sample feedback.It produces structured, verifiable task cards with a problem, pedagogical intent and estimated difficulty, plus a reference solution and answer; the Reasoner receives only the problem.Source: §3.1 · pp. 3–4, §3.2 · p. 4
- A verifier screens generated tasks for validity and consistency, excluding invalid tasks from Reasoner training while returning negative feedback to the Synthesizer.When enabled, an independent consensus check rejects tasks whose inferred answers do not support the provided reference.Source: §3.2 · p. 4
3.3 Interaction-Grounded Multi-Agent Co-Training
- Reasoner rollouts provide correctness rewards for task solving and outcome-grounded feedback for synthesis; teachability favors tasks with mixed success rather than uniformly solved or failed outcomes.Quality and answer reliability constrain feasible synthesis, while realized Reasoner behavior determines which valid tasks are useful for learning.Source: §3.3 · p. 5
3.4 Closed-Loop Agent–Data Self-Evolution
- SYNCO records tasks, validation results, rollouts, success rates, rewards, and rejection reasons in persistent memory to renew the next synthesis context.Because task utility changes with the Reasoner, a formerly mixed-outcome task may become easy, while another may enter the learnable region; the next distribution changes through both renewed context and an updated synthesis policy.Source: §3.4 · pp. 5–6
4 Experiments
Across eight mathematical reasoning benchmarks, the experiments compare SYNCO with synthetic-data methods and controlled baselines, using Qwen3-8B as the default backbone and Qwen3-4B for cross-scale validation. The analyses examine overall performance, curriculum evolution, sources of gains, synthesis signals, efficiency, learner fit, and optimization dynamics.
4.1 RQ1: Does Co-Training Data Synthesis Improve Reasoning Performance?
- 78.72% average accuracy gives SYNCO a 3.76-point lead over the strongest external baseline, ScaleQuest-Math, and a 2.87-point lead over SYNCO (Frozen Synth.).SYNCO is best or tied-best on six of eight benchmarks and second on the other two.Source: §4.1 · pp. 6–7
4.2 RQ2: How Does the Synthesized Curriculum Evolve During Co-Training?
- Gradient-bearing tasks fall from 31.3% to 9.5% across generation buckets as all-correct outcomes rise from 34.4% to 59.9%, while quality stays high and novelty increases from 0.659 to 0.775.Across the full pool, teachability correlates with gradient-bearing status at r = 0.934, unlike novelty at r = −0.041.Source: §4.2 · p. 7, Figure 2 · p. 7
4.3 RQ3: Where Do the Gains from Co-Training Arise?
- 45 of the 56 additional solved examples come from the Learn subset, while pooled Learn accuracy rises from 27.11% to 32.23% and Retain changes from 98.52% to 98.95%.The pattern is consistent across all four benchmarks.Source: §4.3 · pp. 7–8, Table 2 · p. 8
4.4 RQ4: Which Synthesis Signals Align with Gradient-Bearing Tasks?
- 64.6% of top-quartile tasks ranked by teachability are gradient-bearing, versus 16.2% in the pool; ranking by the SYNCO gated reward yields 62.3%.Quality and novelty alone yield 12.8% and 12.0%, respectively; the analysis reranks an already collected pool and is diagnostic rather than causal.Source: §4.4 · p. 8, Figure 3 · p. 8
4.5 RQ5: Can the Synthesizer Anticipate Gradient Utility Before Rollout?
- Gradient-bearing rates remain nearly constant at 16.8%, 16.0%, and 16.0% across difficulty groups, although harder labels shift outcomes from all-correct toward all-incorrect.The learnable label predicts gradient-bearing status poorly (r = 0.009; 16.8% precision; 22.3% recall).Source: §4.5 · p. 9, Figure 4 · p. 9
4.6 RQ6: How Efficient Is Online Synthesizer–Reasoner Co-Training?
- SYNCO raises the productive-task rate from 13.7% to 14.5% (4.38 to 4.64 tasks per batch) with mean update time essentially unchanged at 45.8 s versus 45.1 s.The variants share the same Reasoner optimization budget and per-task rollout configuration.Source: §4.6 · p. 9, Table 3 · p. 9
4.7 RQ7: Does SynCo Better Match the Learner’s Capability Boundary?
- 22.8% of SYNCO tasks are gradient-bearing, compared with 4.1% for MetaMathQA, 10.9% for ScaleQuest-Math, and 9.4% for OpenMathInstruct-2 under the same Reasoner and rollout protocol.SYNCO places more probability mass at intermediate success rates, near the current learner’s capability boundary.Source: §4.7 · p. 10, Figure 5 · p. 10
4.8 RQ8: How Does Co-Training Affect the Reasoner’s Optimization Dynamics?
- 0.890 end-of-training reward for SYNCO exceeds 0.783 for the frozen-Synthesizer baseline, with trajectories separating after approximately 35 steps.Both variants improve rapidly early; SYNCO then maintains a higher moving-average reward.Source: Figure 6 · p. 10, §4.8 · p. 11
5 Conclusion
SYNCO jointly optimizes task synthesis and reasoning through shared online interaction. Across eight mathematical benchmarks, it improves over a frozen Synthesizer, with gains primarily from previously unsolved problems.
5 Conclusion
- SYNCO jointly optimizes task synthesis and reasoning through shared online interaction.Source: §5 · p. 11
- Across eight mathematical benchmarks, SYNCO improves over a frozen Synthesizer, with gains primarily arising from previously unsolved problems.The conclusion also reports that outcome-grounded teachability aligns strongly with gradient-bearing interactions, unlike static properties such as quality or novelty.Source: §5 · p. 11
A Related Work
- Developments in autonomous and multi-agent systems motivate frameworks where agents construct and adapt the experience used for subsequent training.Source: A Related Work · p. 16
A.1 Synthetic Data for LLM Post-Training
Prior synthetic-data methods improve data realism, diversity, verifiability, or learner relevance, while self-evolving systems use agent feedback to adapt tasks. SYNCO distinguishes itself by jointly optimizing synthesis and learning using outcome-grounded task utility.
A.1 Synthetic Data for LLM Post-Training
- Synthetic-data research spans instruction and reasoning supervision as well as interactive-agent experience, improving realism, diversity, environmental grounding, and verifiability.These pipelines commonly use offline corpora or generators fixed or updated separately from the downstream policy.Source: §A.1 · p. 16
- Reasoning-oriented synthesis emphasizes solution quality and answer verification, while some approaches target persistent learner weaknesses.These methods support scalable reasoning supervision without process-level human preference annotations.Source: §A.1 · p. 16
- In group-relative reinforcement learning, tasks with uniformly correct or uniformly incorrect rollouts provide no discriminative policy signal, motivating synthesis based on realized teachability.The training distribution can therefore evolve with the learner rather than only with static task properties.Source: §A.1 · pp. 16–17
A.2 Self-Evolving LLMs and Agent–Data Co-Training
- Self-evolving systems use task-proposal and task-solving feedback loops to adapt mathematical curricula and broader agent experiences.Examples include systems for search, tool use, service dialogue, and general agent workflows.Source: §A.2 · p. 17
- SYNCO updates independently parameterized synthesis and reasoning policies from the same online interaction batch, unlike approaches using separate or alternating updates.It also distinguishes task feasibility from realized optimization utility by analyzing whether tasks are gradient-bearing.Source: §A.2 · p. 17
B Method
SYNCO frames task synthesis and reasoning as coupled policies trained from shared online interactions. It conditions synthesis on the learner’s state and uses verified rollout outcomes to guide both agents.
B.1 Problem Formulation: Agentic Data Synthesis Co-Training
- The independently parameterized Synthesizer controls the Reasoner’s training experiences, while Reasoner behavior determines task-level feedback to the Synthesizer.The policies have distinct but mutually dependent objectives, making synthesis and learning a coupled optimization problem rather than sequential stages.Source: §B.1 · pp. 17–18
- The synthesis context summarizes recent Reasoner performance, capability weaknesses, task-distribution coverage, and feedback on rejected or redundant tasks.The current synthesis policy and learner state induce the training distribution rather than an externally fixed dataset.Source: §B.1 · pp. 17–18
- For each valid task, the Reasoner generates independent rollouts that are checked against a verified reference answer, producing interaction feedback for both policies.Reasoner rewards encourage correct, properly formatted solutions; the Synthesizer receives a separate task-level reward.Source: §B.1 · p. 18
B.2 Reward and Verification Details
- Every synthesized task is checked for completeness, validity, answer consistency, output validity, and redundancy, with an independent consensus check for reference-answer support.The answer checker supports normalized exact, numerical, symbolic, and set- or tuple-level equivalence.Source: §B.2 · pp. 18–19
- The gated Synthesizer reward combines task quality and answer reliability with teachability determined by the current Reasoner’s rollout outcomes.Teachability peaks when half the rollouts succeed and falls toward zero when all succeed or all fail; quality and reliability constrain optimization to trustworthy tasks.Source: §B.2 · p. 19
C.1 Benchmarks
- Evaluation covers eight mathematical benchmarks spanning elementary word problems, competition mathematics, and higher-level quantitative reasoning, with answer accuracy reported on standard test splits.Source: §C.1 · p. 19
C.2 Baselines
- Comparisons include fixed synthetic-data methods, rejection-sampling SFT, an alternating self-evolution baseline, a matched frozen-Synthesizer variant, released models, and controlled synthetic corpora.The matched frozen variant keeps online generation and Reasoner training while disabling Synthesizer updates.Source: §C.2 · pp. 19–20
C.3 Implementation Details
On Qwen3-4B, SYNCO achieves a 75.20% average across eight benchmarks, exceeding the matched frozen-Synthesizer variant. The implementation uses separately initialized agents, while prompts condition synthesis on learner diagnostics and require independently verifiable tasks.
C.3 Implementation Details
- The Synthesizer and Reasoner are independently parameterized Qwen3-8B copies trained with GRPO; the main comparison is also repeated using Qwen3-4B.Each iteration contains 32 tasks and four independent Reasoner trajectories per admitted task.Source: §C.3 · pp. 20–21
C.4 Agent Prompt Templates
- Synthesis prompts use pool statistics, Reasoner diagnostics, and rejection feedback, and require one original, well-posed task with a unique verifiable answer.The Reasoner receives only the generated problem statement; reference solutions and pedagogical metadata are reserved for validation and reward computation.Source: §C.4 · pp. 21–23
- Prompts favor tasks likely to yield mixed success across independent Reasoner attempts, since uniformly solved or failed tasks provide little group-relative learning signal.Source: §C.4 · p. 22
D.1 Generalization Across Model Scales
- 75.20% average accuracy for SYNCO exceeds 73.72% for SYNCO (Frozen Synth.) on Qwen3-4B across eight benchmarks.This is a 1.48 percentage-point gain over the matched frozen variant; SYNCO improves on seven of eight benchmarks and has the strongest overall average.Source: Table 4 · p. 23, §D.1 · p. 23
D.2 How Does the Synthesized Curriculum Evolve During Co-Training?
Only 16.2% of verified tasks initially provided gradient-bearing feedback, and that share fell as the Reasoner improved. Teachability aligned with this learning utility far more closely than novelty or intended difficulty did.
D.2 How Does the Synthesized Curriculum Evolve During Co-Training?
- 475 of 2,940 verified tasks (16.2%) were gradient-bearing, compared with 994 (33.8%) uniformly incorrect and 1,471 (50.0%) uniformly correct tasks.A task is gradient-bearing when Reasoner rollouts receive non-identical rewards and produce non-zero group-relative advantages; this coincided with 0 < SR < 1 in the collected data.Source: §D.2 · p. 23, Table 5 · p. 24
- Gradient-bearing tasks fell from 31.3% in the earliest bucket to 9.5% in the final bucket, while all-correct tasks rose from 34.4% to 59.9%.The shift was driven primarily by tasks becoming uniformly solvable, even as quality remained high and novelty increased; the useful region therefore moved with the Reasoner’s capability frontier.Source: §D.2 · pp. 23–24
- Teachability correlated with gradient-bearing outcomes at r = 0.934, while novelty correlated at r = −0.041 and gradient-bearing rates across intended-difficulty groups stayed near the 16.2% pool prevalence.Teachability was also nearly uncorrelated with raw success rate (r = −0.028), indicating that it captures heterogeneous rollout outcomes rather than task easiness or difficulty.Source: §D.2 · p. 24
D.3 Attributing Gains to Capability Acquisition
SYNCO’s terminal advantage over a frozen Synthesizer came mainly from solving examples the initial model had missed. This recovery-heavy pattern held across all four benchmarks.
D.3 Attributing Gains to Capability Acquisition
- SYNCO improved recovery on initially incorrect examples by 5.12 points, from 27.11% to 32.23%, while retention improved by 0.43 points, from 98.52% to 98.95%.Learn contributed +21 of +27 on GSM-Hard, +9 of +11 on MATH-500, +11 of +13 on GSM8K, and +4 of +5 on SVAMP.Source: §D.3 · p. 24
- Across all four benchmarks, Learn contributed 45 of the 56 additional solved examples (80.4%) over SYNCO (Frozen Synth.).The Learn subset comprises examples initially answered incorrectly by Qwen3-8B; Figure 7 attributes the terminal advantage between Learn and Retain.Source: §D.3 · pp. 24–25, Figure 7 · p. 25
- The decomposition links the minority of tasks producing non-degenerate group-relative feedback to gains predominantly on newly solved evaluation examples, across saturated and higher-headroom benchmarks.The reported pattern supports capability acquisition beyond the Reasoner’s initial competence rather than merely consolidating existing performance.Source: §D.3 · p. 25
D.4 Synthesis Signals and Realized Gradient Utility
Outcome-grounded teachability best identified tasks with realized optimization utility, while quality and reliability served as gates and novelty or intended difficulty offered little utility discrimination.
D.4 Synthesis Signals and Realized Gradient Utility
- Quality and reliability averaged 0.962 and 0.965 after verification, so their weak utility correlations reflect limited discrimination among admitted tasks, not that they are unnecessary.They constrain the gated reward toward trustworthy tasks, while teachability identifies realized learning utility; quality-only and novelty-only rankings fall below pool prevalence.Source: §D.4 · pp. 25–26
- Teachability correlated with gradient-bearing outcomes at r = 0.934, compared with r = −0.041 for novelty, and correlated with raw Reasoner success rate at only r = −0.028.This alignment reflects heterogeneous rollout outcomes rather than simply favoring easy or hard tasks.Source: §D.4 · p. 25
- Ranking the top quartile by SYNCO’s gated reward raised the gradient-bearing rate from 16.2% to 62.3%; teachability alone reached 64.6%, versus 12.8% for quality and 12.0% for novelty.The Figure 3 reranking analysis measures enrichment over the verified-task pool; Table 6 describes the synthesis-signal diagnostics.Source: §D.4 · pp. 25–26, Table 6 · p. 26
D.5 Why Ex-Ante Difficulty Does Not Predict Gradient Utility
The Synthesizer’s difficulty estimates captured broad task-difficulty shifts but did not identify tasks likely to produce gradient-bearing feedback. That utility depended on observing the current Reasoner’s outcomes.
D.5 Why Ex-Ante Difficulty Does Not Predict Gradient Utility
- Learnable tasks averaged a 0.625 success rate, versus 0.484 for hard/too-hard tasks; uniformly correct outcomes fell from 53.0% to 39.9% as uniformly incorrect outcomes rose from 30.2% to 44.1%.The Synthesizer therefore captured the broad direction of difficulty and could shift generation toward harder regions.Source: §D.5 · p. 26
- Gradient-bearing rates remained 16.8%, 16.0%, and 16.0% across learnable, mid-band, and hard/too-hard tasks, near the 16.2% pool rate.Greater nominal difficulty mainly shifted probability from uniformly correct to uniformly incorrect outcomes; either uniform outcome can remain uninformative for group-relative optimization.Source: §D.5 · pp. 26–27
- The learnable label correlated with gradient-bearing tasks at only 0.009, with 16.8% precision and 22.3% recall; outcome-grounded teachability correlated at r = 0.934.Difficulty is a generation intent, whereas teachability uses observed Reasoner rollouts to identify realized learning utility.Source: §D.5 · p. 27
D.6 Training Efficiency, Gradient Utilization, and Data Production
The efficiency analysis finds that SYNCO’s computational profile remains comparable to Frozen, while SYNCO modestly increases productive-task yield and generates verified training data within the training loop.
D.6 Training Efficiency, Gradient Utilization, and Data Production
- 45.1 s versus 45.8 s mean synchronized update times and 343.5 s versus 341.6 s median end-to-end step times indicate comparable SYNCO and Frozen computational profiles.The runs were executed independently, so the small wall-clock differences do not show that either variant is intrinsically faster.Source: §D.6 · p. 27
- 14.5% under SYNCO versus 13.7% under Frozen were productive tasks, or 4.64 versus 4.38 per batch of 32, without added rollout budget or Reasoner optimization cost.The increase is modest and suggests better allocation of a fixed interaction budget toward tasks providing non-degenerate group-relative feedback.Source: §D.6 · p. 28
- 91.9% of 3,200 SYNCO task candidates passed verification across 100 training steps, yielding 2,940 accepted tasks without human annotation or external LLM API calls.Generation, verification, and Reasoner rollouts are included in measured runtime, so internal data production is not cost-free.Source: §D.6 · p. 28