PaperlayerView exact revision on arXiv

Source-linked AI summary

ADE: Agentic Data Evolution Framework for Human-Centered Objectives

Yang Yu, Yilin Jiang, Zexuan Fei, Yiming Luo, Xingkai Song, Kaiyi Huang, Aimin Zhou, Xin Lin, Fei Tan

arXiv:2608.23719v1cs.CL

TL;DR

Human-centered objectives are non-executable and context-dependent, making reliable verification and scalable supervision difficult. ADE organizes synthetic supervision as evolving data snapshots updated by Observation–Variation–Selection with conservative admission. The framework improves intrinsic and extrinsic outcomes, receives human preference, and shows gains across methods, scales, and tasks, while remaining subject to scope and exploration–reliability limitations.

  • Problem

    Human-centered objectives are weakly verifiable, lacking scalable measurement tools or reliable supervision signals for alignment.

  • Method

    ADE evolves synthetic supervision snapshots through Observation–Variation–Selection, routed critiques, comparative selection, and admission that retains the parent when evidence is insufficient.

  • Results

    ADE shows stable improvements across intrinsic, extrinsic, and human-calibrated validation, with consistent gains across post-training methods, model scales, and tasks beyond educational objectives.

  • Takeaways & Limitations

    ADE provides a practical framework for accumulating supported improvements in synthetic supervision for human-centered objectives under weak verification.

  • Takeaways & Limitations

    The tutoring objectives and rubrics are grounded in Chinese K–12 contexts, and elitist admission may reject uncertain variants whose benefits are difficult to verify.

Abstract

from arXiv · show

Aligning large language models to human-centered objectives is difficult when targets are non-executable and context-dependent, limiting reliable verification and scalable supervision. Although synthetic data expands coverage, weak verification shifts the bottleneck from generation to selection. Noisy signals destabilize iterative refinement and can cause silent regressions. We propose Agentic Data Evolution (ADE), a data-centric framework that organizes synthetic supervision as evolving data snapshots. ADE improves data snapshots through a closed-loop Observation-Variation-Selection (OVS) procedure, where a steady-state admission mechanism acts as a quality ratchet that conservatively gates updates for sustained cross-round improvement. We validate these improvements through complementary intrinsic trend tracking and extrinsic post-training evaluation. On DEV300, ADE raises the intrinsic win rate from 50% to 75.81% and the extrinsic win rate from 55.20% to 68.86%, consistent performance gains across diverse benchmarks. Blind expert evaluation further confirms this, with a 66.11% preference for evolved answers. These gains extend across post-training methods, model scales, and tasks beyond the target weakly verifiable educational objectives. Resources are available at https://github.com/ZeroLoss-Lab/Agentic-Data-Evolution.

1 Introduction

Human-centered alignment targets are weakly verifiable and context-dependent, making scalable supervision and reliable candidate selection difficult. ADE addresses this by evolving synthetic supervision through evidence-based, non-regressive updates, with gains supported by complementary validation.

  • Motivation: Weakly verifiable objectives lack scalable measurement tools or reliable supervision signals, limiting alignment progress.Examples include value orientation, affective support, and creative innovation in educational settings.
  • Motivation: Synthetic supervision expands coverage, but weak verification shifts the bottleneck from candidate generation to deciding which revisions to retain.Iterative acceptance can compound regressions in pedagogical appropriateness, learner support, or objective alignment.
  • Approach: ADE reframes supervision construction as continuous data evolution using Observation–Variation–Selection with role-specialized agents and comparative admission.It retains the current answer when comparative evidence is insufficient, while supported improvements accumulate across rounds.
  • Evidence: ADE is evaluated through intrinsic, extrinsic, and human-calibrated validation, with experts preferring evolved answers in 66.11% of cases.Additional experiments report consistent gains across post-training methods, model scales, and tasks beyond educational objectives.
  • Contribution: The work formulates weakly verifiable supervision construction as data evolution and demonstrates ADE through intrinsic, extrinsic, and human-calibrated evidence.

2 Related Work

Human-centered objectives are context-dependent, rubric-mediated, and often lack unique ground-truth answers. ADE treats weak verification as a design constraint and uses complementary validation rather than relying on a single judge signal.

  • Alignment under Weak Verification: Human-centered objectives such as value orientation, affective support, and creative innovation are structurally harder to align than tasks with executable feedback.They depend on context and lack unique ground-truth answers.
  • Synthetic Data and Agentic Systems: ADE validates evolved data with intrinsic, extrinsic, and human checks instead of relying on a single judge signal.

3 Preliminaries

The paper represents supervision as question–answer pairs organized into evolving dataset snapshots. An iterative operator generates, revises, and selects pairs, with progress assessed through three validation views.

  • Problem Formulation: Supervision is represented as question–answer pairs, with t indexing an evolving data snapshot.
  • Problem Formulation: The iterative operator Φ maps snapshots by generating new pairs, revising existing pairs, and selecting or discarding candidates.
  • ADE Formulation: ADE implements Φ with role-prompted LLM agents following Observation–Variation–Selection and assesses progress through intrinsic, extrinsic, and human-calibrated validation.

4 Methodology

ADE evolves tutoring supervision through routed critique, complementary mutation, comparative selection, and conservative admission. Its validation framework measures cross-round quality, downstream transfer, and agreement with human judgments.

  • 4 Methodology: ADE evolves each instance through observation, variation, and selection, committing updates only when comparative evidence supports improvement.The parent is retained otherwise, providing steady-state non-regression protection.
  • 4.1 Observation–Variation–Selection: Observation routes each instance to a primary objective and factorizes critique into general and dimension-specific assessments.The resulting critique summaries serve as variation evidence.
  • 4.1 Observation–Variation–Selection: Variation uses conservative and aggressive proposers to target observed weaknesses while balancing preservation of structure with larger exploratory revisions.Rejected offspring and rationales can guide explicit self-correction in subsequent mutation.
  • 4.1 Observation–Variation–Selection: Selection separates candidate comparison from survivor admission, allowing the best available candidate to be rejected when it remains too weak to commit.Admission compares the parent and preference winner under the routed objective and general quality constraints.
  • 4.1 Observation–Variation–Selection: Rejected survivors and rationales feed back into mutation, while elitist commitment forms a quality ratchet that accepts only evidence-supported updates.
  • 4.2 Data Construction: ADE begins with a 10,000-pair tutoring snapshot covering three routed objectives: value orientation, affective support, and creative innovation.The initial data are generated through a concept–topic, scenario, and question pipeline with approximately balanced objective coverage.
  • 4.3 Validating Evolving Data Quality: Intrinsic validation compares answers across evolution rounds, while extrinsic validation tests transfer to post-trained models on held-out benchmarks and downstream tasks.Human-calibrated validation anchors automatic preference trends to independent expert judgments.

5 Experiments

ADE is evaluated through intrinsic, extrinsic, human-calibrated, transfer, and ablation studies, testing reliable evolution, post-training transfer, broader generalization, and operator contributions. Results show sustained improvements under weak verification, transfer across settings, and measurable dependence on objective-specific critique and conservative admission.

  • Experimental questions: The experiments test reliable cross-round improvement, transfer beyond protocol-aligned preferences, and generalization across methods, scales, and tasks.These questions are evaluated using intrinsic and extrinsic validation on DEV300, transfer experiments, and out-of-domain verifiable benchmarks.
  • Intrinsic validation: ADE achieves a monotonic increase in preference win rates against D(0) across the first three evolution rounds with little late-round penalty.The result supports OVS as a non-regression mechanism under weak verification, while recursive refinement baselines can regress substantially.
  • Extrinsic validation: Models post-trained on ADE-evolved snapshots improve behavior on DEV300 under extrinsic validation across routed objectives.Extrinsic evaluation complements intrinsic comparisons by testing whether evolved supervision transfers to post-trained model behavior.
  • Out-of-domain transfer: MATH-500 accuracy increases by 1.20 points and ToxiCN F1 increases by 1.18 points when comparing models trained on D(0) and D(4).These positive changes indicate transfer to two verifiable tasks beyond the target educational objectives.
  • Ablation studies: Removing dimension-specific critics lowers Overall win rates from 75.06 to 53.92 at Round 1 and from 75.81 to 65.03 at Round 4.The degradation concentrates on value orientation and affective support, indicating that general critiques are insufficient for routed objectives.
  • Ablation studies: Removing SELECT lowers Overall win rates by 6.53 points at Round 1, while removing ADMIT causes a 1.92-point average decline per round.The ablations associate comparative selection with filtering adverse offspring and admission with preventing silent quality erosion.

6 Discussion

ADE frames weakly verifiable supervision as continuous cross-round data evolution, using conservative evidence-based admission to accumulate supported improvements while limiting uncertain updates. Complementary validation indicates gains across evaluation streams, methods, model scales, and tasks.

  • ADE reframes synthesis for weakly verifiable objectives as continuous cross-round data evolution rather than one-shot generation.
  • Dimension-specific critics provide routed, factorized evidence, while relative comparison and non-regression admission convert noisy judgments into conservative update decisions.
  • The quality-ratchet mechanism accumulates supported improvements while reducing the risk of committing uncertain revisions.
  • ADE’s observed gains extend across post-training methods, model scales, and tasks beyond the target weakly verifiable educational objectives.

7 Conclusion

ADE applies iterative data evolution with routed critiques, reflective mutation, comparative selection, and elitist admission to weakly verifiable human-centered objectives. The study reports stable improvements and identifies domain coverage, inference cost, and exploration as practical boundaries.

  • ADE formulates supervision construction as iterative data evolution using routed critiques, reflective mutation, comparative selection, and elitist admission.
  • Across the studied alignment objectives, ADE yields stable multi-round intrinsic improvements, downstream post-training transfer, and human-calibrated support.
  • The educational testbed covers three objectives grounded in Chinese K–12 contexts, so normative and affective judgments may reflect culturally specific expectations.
  • ADE increases inference-time compute and latency relative to single-pass synthesis or one-shot rewriting, although the added cost is confined to inference and can be parallelized.
  • Elitist admission may reject uncertain but potentially valuable variants when their benefits are difficult to verify in a single comparison, especially for creative innovation.

A Implementation Details of OVS

The OVS implementation routes each instance to one primary objective, gathers factorized critique, generates conservative and aggressive revisions, and admits only non-regressive updates. Rejected answers and rationales feed subsequent mutation rounds.

  • Observation: ROUTE assigns each instance exactly one primary objective so later critique, selection, and admission use a single evaluation criterion.
  • Observation: CRITIQUE evaluates content, writing quality, and the routed objective, producing evidence consumed by MUTATE and SELECT.
  • Variation: MUTATE generates conservative and aggressive candidates with different revision magnitudes while preserving task intent and the routed objective.
  • Selection: When a candidate is rejected, ADE records the rejection rationale and passes it to the next MUTATE call.
  • Selection: SELECT compares the parent and two offspring under the routed objective, while ADMIT performs a final non-regression check before committing an update.

B Implementation Details of D(0)

D(0) bootstraps ADE with realistic, controllable K–12 tutoring interactions spanning three human-centered objectives. The construction pipeline instantiates and filters concepts, generates topic scenarios, and derives instructional questions under structured controls.

  • Objectives and Scope: D(0) grounds supervision in realistic educational interactions, covers three co-occurring human-centered objectives, and exposes controls for isolating supervision quality.
  • Objectives and Scope: The scaffold records grade band, role view, activity domain, task type, artifact, question type, and constraints for controllable, reproducible coverage.
  • Objectives and Scope: The objectives are Value Orientation, Affective Support, and Creative Innovation, defining educational norms, emotional guidance, and novel feasible ideas respectively.
  • Concept Instantiation and Filtering: Concept instantiation samples dimension-constrained cues and structured attributes, then applies an LLM plausibility judge to remove inconsistent or unrealistic specifications.
  • Scenario and Question Generation: Topic scenarios are generated from accepted concept instantiations, after which tutoring questions are derived using role view and question type.

C Evaluation Benchmarks and Protocols

The evaluation uses held-out benchmarks aligned with ADE’s three educational objectives and pairwise judgments designed to reduce presentation bias. DEV300 is constructed with balanced, non-overlapping instances, while comparison criteria cover understanding, values, creative innovation, and affective support.

  • DEV300: DEV300 is a held-out benchmark designed to match ADE’s three objectives, with 300 instances distributed evenly across them.Questions are generated using the same concept-composition and question-generation pipeline as D(0).
  • DEV300: 100 instances per objective are sampled after rejecting questions with BLEU-2 similarity above 0.7 to any D(0) question.This procedure enforces non-overlap before uniform sampling.
  • Pairwise evaluation: Pairwise evaluation uses three independent trials with swapped presentation order, and invalid judge outputs are treated as TIE.Swapping reverses the meanings of A and B in the win count to reduce position bias.
  • Evaluation criteria: The judge compares answers across question understanding, value orientation, creative innovation, and affective support.Criteria include completeness, responsible value judgments, multiple perspectives, feasible solutions, emotional regulation, and constructive handling of interpersonal situations.
  • Evaluation criteria: Edu-Values assesses educational-value alignment and teacher-facing competencies, emphasizing normative educational viewpoints and transferable pedagogical competencies.The related benchmark is used to assess dimensions relevant to pedagogical intent and decision structure beyond surface fluency.
  • Evaluation criteria: The EduBench subset contains 90 balanced instances across nine task types, grouped into value orientation and affective support categories, with creative innovation also represented.The supplied passage specifies 40 value-orientation and 20 affective-support instances before introducing the creative-innovation allocation.

C.4 Out-of-Domain Tasks

ADE’s supervision is tested beyond its weakly verifiable educational targets on mathematical reasoning and content moderation. The evaluation compares models trained on D(0) and D(4) using deterministic decoding and task-specific metrics.

  • Out-of-domain tasks: MATH-500 and ToxiCN test whether ADE-evolved supervision transfers to out-of-domain mathematical reasoning and toxicity detection.MATH-500 is associated with creative-innovation routing, while ToxiCN is associated with value-orientation routing.
  • Evaluation setup: Models trained on D(0) and D(4) are compared under the same target model and post-training recipe, with temperature set to 0.0.MATH-500 uses accuracy, whereas ToxiCN uses F1.
  • Evaluation setup: The MATH-500 inference prompt requests step-by-step solutions with the final answer placed in boxed notation.The prompt frames the model as a helpful mathematical assistant.
  • Evaluation setup: The ToxiCN prompt asks a content-moderation assistant to classify text as safe or unsafe after analyzing potentially offensive content.Unsafe content includes insults, threats, profanity, hate speech, harassment, and other offensive language.

D Human Evaluation Details

Human evaluation compares D(0) and D(4) answers through blinded, randomized pairwise judgments by three annotators. Agreement is substantial both among annotators and between human consensus and the automatic judge.

  • Human evaluation protocol: Three human annotators independently evaluated 300 paired D(0) and D(4) instances in randomized, blinded order.They selected the better-aligned response or marked TIE when the responses were indistinguishable.
  • Human evaluation protocol: Annotators followed the LLM judge’s pairwise A/B/TIE guideline, were recruited through an open call, and were paid USD 15 per hour.Average completion time was approximately two hours.
  • Annotators: The three experts included two participants with education-related bachelor’s degrees and one with a computer-science master’s degree.All had prior rubric-based text-evaluation experience and were independent of ADE’s design and evaluation protocol.
  • Agreement: Fleiss’s κ was 0.7751 among human annotators, while Cohen’s κ was 0.7149 between human majority vote and the automatic judge.Both statistics are reported as indicating substantial agreement.

E Settings and Supplementary Results

Supplementary results examine training implementation and reinforcement-learning robustness across ADE’s three evaluation dimensions. The reported comparisons indicate gains across multiple post-training recipes, while the pipeline uses full-parameter fine-tuning and substantial GPU resources.

  • Training settings: All runs use full-parameter fine-tuning with DeepSpeed on a single node containing eight NVIDIA H100 GPUs.Training uses BF16, gradient checkpointing, a global batch size of 8, 16 accumulation steps, and a learning rate of 3 × 10−5.
  • Supplementary results: Table 8 reports win rates for PPO, GRPO, and DPO across value orientation, affective support, and creative innovation.Each reinforcement-learning strategy is evaluated relative to the SFT baseline on ADE-evolved snapshot D(4).
  • Training settings: The pipeline generates approximately 1 M tokens per hour on the stated eight-H100 configuration.Construction of D(0) is training-free and requires LLM deployment only for agent inference.
  • Supplementary results: All three reinforcement-learning methods improve over the SFT baseline on ADE-evolved supervision.The passage presents this as evidence that gains are not tied to one post-training recipe.

F Case Study

The case studies illustrate how ADE’s evolved answers address weak-verification failure modes through clearer constraints, responsibilities, and actionable support. Across rounds, representative value-orientation and affective-support trajectories show improving win rates while revisions are conservatively admitted.

  • Case trajectories: ADE’s evolved answers more often prioritize safety or non-escalation, specify intervention responsibilities, and organize support into inspectable plans.These changes target missing conditional constraints, blurred responsibility boundaries, unstable decision policies, and over-edit drift.
  • Case trajectories: Non-regression admission favors revisions that strengthen constraint clarity, decision stability, and objective consistency without uncontrolled verbosity or cosmetic reformulation.Candidates are admitted only when comparative judgment supports improvement over the current answer.
  • Value orientation: The value-orientation example evolves advice on protecting a minor by adding safer intervention, evidence documentation, support-seeking, and escalation guidance.The Round 4 version prioritizes safety and recommends staff or security support before confrontation when the responder feels unsafe.
  • Value orientation: 91.7% was the Round 4 win rate for the value-orientation case, compared with 66.7% in Round 1.The figures mark additions and deletions relative to Round 0.
  • Affective support: 75% was the Round 4 win rate for the affective-support case.The trajectory figure compares revisions against Round 0.
Loading 2608.23719v1…