PaperlayerView exact revision on arXiv

Source-linked AI summary

DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedback

Zaid Khan, Elias Stengel-Eskin, Jaemin Cho, Mohit Bansal

arXiv:2410.06215v3cs.CLcs.AIcs.LG

TL;DR

Human-led creation of training data targeted to model weaknesses is labor-intensive, and testbeds for developing teachers on complex tasks are lacking. DATAENVGYM provides iterative teacher environments with differing levels of structure and student feedback. Example teachers improve students across its environments and domains, while the testbed supports analysis of training dynamics and skill discovery.

  • Problem

    Developing automated teachers for complex tasks lacks testbeds that simulate iterative data generation or evaluate the components involved.

    Source: §1 · p. 2

  • Method

    DATAENVGYM provides teacher environments where agents plan and generate training data from student-performance states, then receive feedback after student retraining and evaluation.

    Source: §1 · p. 2

  • Results

    Example teachers improve students in all DATAENVGYM environments across visual question answering, mathematics, programming, and tool use.

    Source: §5 · pp. 12–13

  • Takeaways & Limitations

    DATAENVGYM supports analysis of training dynamics, performance across difficulty levels, and the impact of skill-discovery quality, as well as evaluation of agents, engines, and feedback mechanisms.

    Source: §5 · p. 13

    Requiring improvement on specified skills may constrain teachers suboptimally, while unconstrained teaching may not suit users seeking targeted control.

    Source: §3.1 · p. 9

Abstract

from arXiv · show

The process of creating training data to teach models is currently driven by humans, who manually analyze model weaknesses and plan how to create data that improves a student model. Approaches using LLMs as annotators reduce human effort, but still require humans to interpret feedback from evaluations and control the LLM to produce data the student needs. Automating this labor-intensive process by creating autonomous data generation agents - or teachers - is desirable, but requires environments that can simulate the feedback-driven, iterative, closed loop of data creation. To enable rapid, scalable testing for such agents and their modules, we introduce DataEnvGym, a testbed of teacher environments for data generation agents. DataEnvGym frames data generation as a sequential decision-making task, involving an agent consisting of a data generation policy (which generates a plan for creating training data) and a data generation engine (which transforms the plan into data), inside an environment that provides student feedback. The agent's goal is to improve student performance. Students are iteratively trained and evaluated on generated data, and their feedback (in the form of errors or weak skills) is reported to the agent after each iteration. DataEnvGym includes multiple teacher environment instantiations across 3 levels of structure in the state representation and action space. More structured environments are based on inferred skills and offer more interpretability and curriculum control. We support 4 domains (math, code, VQA, and tool-use) and test multiple students and teachers. Example agents in our teaching environments can iteratively improve students across tasks and settings. Moreover, we show that environments teach different skill levels and test variants of key modules, pointing to future work in improving data generation agents, engines, and feedback mechanisms.

1 INTRODUCTION

DATAENVGYM addresses the lack of testbeds for developing automated teachers on complex tasks by simulating iterative, feedback-driven data generation. It provides environments with different levels of structure and reports that example agents improve student performance across tasks.

1 INTRODUCTION

  • The testbed addresses the absence of environments for developing automated teachers on complex tasks, a need arising from the labor and repetition of human-led data improvement.The motivating process involves identifying weaknesses, creating targeted data, retraining, and revising data based on performance.Source: §1 · pp. 1–2
  • DATAENVGYM frames teaching as repeated planning, data generation, student training, and evaluation that returns performance feedback to the teacher.The policy plans data creation; the engine executes the plan, and the environment supplies the next state.Source: §1 · p. 2, Figure 1 · p. 2
  • Its environments range from open-ended error-based states and actions to skill-structured representations, including a skill-tree setting with explore and exploit actions.Skill-based teaching offers more interpretable control and supports human- or model-specified skills and curriculum control.Source: §1 · pp. 2–3
  • Example agents improve students by averages of 4.43% on GQA, 4.82% on MATH, 1.80% on LiveCodeBench, 15.17% on NaturalBench, and 20.81% on MnMs.These are average absolute-accuracy gains across environments.Source: §1 · p. 3
  • With State policies outperform No State policies by 3.5% in OPEN-ENDED, 2.05% in SKILL-LIST, and 1.08% in SKILL-TREE.The comparison tests whether policies use student feedback states.Source: §1 · p. 3

2 DATAENVGYM ENVIRONMENTS AND AGENTS

DATAENVGYM provides teacher environments in which agents use student-performance feedback to plan and generate training data, with environment structure determining the state and available actions. Its modules support iterative training and evaluation, skill-based organization, policy decisions, and execution of data-generation plans.

2 DATAENVGYM ENVIRONMENTS AND AGENTS

  • Three environments range from direct data generation from per-example performance to skill-based states and actions, with skill structures supporting targeted control and interpretability.OPEN-ENDED maps per-example predictions to data plans; SKILL-LIST reports performance by skill; SKILL-TREE adds hierarchical skills and explore/exploit actions.Source: §2 · pp. 3–5, Figure 2 · p. 4

2.1.1 TRAINER AND EVALUATOR

  • The trainer updates the student from generated data, and the evaluator measures the updated model to provide the next performance state.The baseline uses supervised fine-tuning and evaluates on standard training splits from the tested datasets.Source: §2.1.1 · p. 5

2.1.2 SKILL DISCOVERY

  • Skill discovery labels evaluated examples with skills inferred from training data or user-specified target skills, supplying skill inputs to SKILL-LIST and SKILL-TREE.The baseline assigns a skill to each task instance with an LLM, then groups similar skills into mutually exclusive categories.Source: §2.1.2 · pp. 5–6

2.1.3 SKILL ORGANIZATION

  • The SKILL-TREE organization module turns skills into a forest of hierarchical trees that encode skill relationships and store metadata such as data allocation.The default forest can be replaced by other user-provided implementations.Source: §2.1.3 · p. 6

2.2.1 DATA GENERATION POLICY

  • The policy maps the environment’s student-performance state to a data-generation plan, using explore actions to add subskills and exploit actions to change their data allocations in SKILL-TREE.OPEN-ENDED and SKILL-LIST plans specify training examples directly; SKILL-TREE plans use discrete structural or allocation actions.Source: §2.2.1 · p. 6

2.2.2 DATA GENERATION ENGINE

  • The engine executes a plan to produce training examples, either formatting specified datapoints or interpreting SKILL-TREE actions and generating data for its subskills.For SKILL-TREE, explore invokes subskill discovery, while exploit adjusts subskill budgets before the engine generates the allocated data.Source: §2.2.2 · p. 7

3 EXPERIMENTS

Experiments test DataEnvGym across visual question answering, mathematics, programming, and tool-use, using instruction-tuned students and comparing state-conditioned with state-free teaching. Students generally benefit from teaching, while results vary by environment and task.

3.1 PRIMARY RESULTS: VQA, MATHEMATICS, PROGRAMMING

  • With State generally outperforms No State; SKILL-TREE leads on GQA (+5.58%), while OPEN-ENDED leads on MATH (+7.66%).On LiveCodeBench, policies without state information do not improve the student; on MATH, they improve students in all environments.Source: §3.1 · p. 8, Table 2 · p. 8

3.2 ADDITIONAL MULTIMODAL AND TOOL-USE BENCHMARKS

  • On NaturalBench, state-conditioned data improves every reported metric, including Group Score, which shows no increase without state conditioning.Across NaturalBench and MnMs, task and error conditioning yields more effective training data.Source: §3.2 · p. 9
  • For MnMs, treating tool combinations as skills requires sampling, and randomly selecting skills below near-perfect student accuracy is the most effective tested method.The standard skill-discovery pipeline does not produce skills tightly coupled to the tools required by MnMs.Source: §3.2 · p. 9

3.3 ANALYSIS: DIFFICULTY/RARITY, TRAINING DYNAMICS, SKILL DISCOVERY QUALITY, AND QUALITATIVE EXAMPLES

  • In SKILL-TREE MATH training, gains are largest around difficulty 3.5 and generally greater for more frequent skills; edge cases can show little gain or decreases.Calculus and Optimization show close to zero difference, while Trigonometry decreases; the authors offer saturation and data-generation difficulty as possible explanations.Source: §3.3 · p. 10, Figure 4 · p. 10
  • Training trajectories differ by task: SKILL-TREE improves for more iterations on GQA, plateaus on MATH, and is truncated after two data-generation rounds on LiveCodeBench.Figure 5 distinguishes each student's maximum performance from continued training beyond that point.Source: Figure 5 · p. 10, §3.3 · p. 11
  • Oracle skills improve teaching on GQA and MATH, indicating that skill-discovery components can be tested modularly using student performance.GQA uses human-annotated oracle skills; MATH's oracle skills are inferred from test data, and the authors note this privileged-information setup.Source: §3.3 · p. 11, Table 5 · p. 11
  • GPT-4o-mini produces less effective data than GPT-4o, but its reported gains are higher in skill-structured environments than in OPEN-ENDED.For GPT-4o-mini, reported overall gains are +1.81% in SKILL-TREE, +1.75% in SKILL-LIST, and +0.77% in OPEN-ENDED.Source: §3.3 · p. 11, Table 6 · p. 11
  • Generated examples generally match the target tasks' training-data styles across GQA, MATH, and LiveCodeBench, with qualitative VQA examples showing corrected responses after training.The examples include improved material identification and correction of a foreground-background confusion.Source: §3.3 · pp. 11–12

4 RELATED WORK

Prior environment-generation work largely developed students in restricted settings, while DATAENVGYM makes data-generation agents the focus in more realistic domains. Unlike fixed-dataset distillation, its teaching loop generates inputs adaptively from student feedback.

4 RELATED WORK

  • Prior environment-generation work often centered on restricted settings such as simple games and on developing students, whereas DATAENVGYM focuses on teachers in more realistic domains.Its environments include automatic skill discovery for mathematics, visual question answering, and programming.Source: §4 · p. 12
  • Earlier environment-generation work includes adapting game instances to teach core skills from agent progress, while DATAENVGYM applies feedback-driven generation to training data for students.EnvGen is cited as an example of adapting game instances; DATAENVGYM treats students as part of the environment and targets the data-generation process.Source: §4 · p. 12
  • DATAENVGYM differs from typical knowledge distillation by generating model inputs dynamically from student feedback rather than using a fixed dataset or generating all inputs at once.The framework can also use different student-training methods, including symbolic distillation and logit-based losses.Source: §4 · p. 12

5 CONCLUSION

DATAENVGYM provides modular teacher environments across four domains and reports that its example teachers improve students in all environments. The conclusion analyzes training dynamics, difficulty levels, and skill-discovery quality; the appendix adds related work, method details, qualitative examples, and further experiments.

5 CONCLUSION

  • DATAENVGYM is a modular testbed with three environment instantiations across visual question answering, mathematics, programming, and tool use, and its example teachers improve students in all environments.The paper analyzes training dynamics, performance across difficulty levels, and the impact of skill-discovery quality.Source: §5 · pp. 12–13

APPENDIX

  • The appendix adds related work, method details, qualitative examples, generated-data accuracy experiments, and ablations of data versus training time.Source: APPENDIX · p. 18

A ADDITIONAL RELATED WORK

  • The additional related work connects skill discovery to DATAENVGYM’s interpretable skill-based environments and frames model-weakness discovery as cooperative student-teacher improvement rather than adversarial defense.The paper also presents downstream student improvement as an extrinsic way to evaluate skill-discovery methods.Source: A ADDITIONAL RELATED WORK · p. 18

B ADDITIONAL METHOD DETAILS

The multimodal OPEN-ENDED loop is modular: changing the evaluation dataset targets a different data distribution, and the policy and engine can also be changed.

B ADDITIONAL METHOD DETAILS

  • In the GQA OPEN-ENDED example, changing the evaluation dataset targets a different data distribution, and the data-generation policy and engine are independently modifiable.Source: B ADDITIONAL METHOD DETAILS · p. 18

B.1.1 GQA

The appendix gives training configurations for GQA and LiveCodeBench/MATH and identifies the LLM modules and prompt templates used in the environments.

B.1.1 GQA

  • For GQA, PaliGemma-3b-pt-224 is trained for 10 epochs with AdamW, LoRA, BF16, and batch size 16; Figure 6 depicts the modular OPEN-ENDED multimodal loop.The GQA configuration uses learning rate 2 × 10^-5, weight decay 10^-6, LoRA rank 16 and alpha 32, and dropout 0.05; the caption notes that math or code generation may not require a data-generation engine.Source: §B.1.1 · pp. 18–19, Figure 6 · p. 19

B.1.2 LIVECODEBENCH AND MATH

  • LiveCodeBench and MATH training uses Alpaca-format instruction-response pairs, Adam, FP16, three epochs, and LoRA on all linear layers.Training uses batch size 16, a cosine learning-rate schedule with 0.1 warmup ratio, maximum length 1024, and LoRA rank 16, alpha 32, and dropout 0.05.Source: §B.1.2 · p. 19

B.2 LLM DETAILS

  • GPT-4o-2024-08-06 supplies skill discovery, data-generation policy, and data-generation engine modules, while the appendix lists prompts for discovery and domain- and environment-specific generation.The calls use temperature 0 and top-p 1.0; structured outputs are parsed with Instructor.Source: §B.2 · p. 19

B.3 DATA GENERATION DETAILS

The section documents prompt templates for skill discovery and domain-specific data generation, alongside termination and best-student selection details.

B.3 DATA GENERATION DETAILS

  • Episodes terminate at a fixed iteration limit, and validation accuracy selects the best student; the figures provide skill-discovery and domain-specific generation prompt examples.The prompt figures cover skill discovery for SKILL-TREE and SKILL-LIST, plus GQA, MATH, and LiveCodeBench generation.Source: §B.3 · p. 19, Figure 7 · p. 20, Figure 8 · p. 21
  • In the SKILL-TREE environment, the baseline policy episode produced the top-performing student after 20 steps at approximately 3k data points; in OPEN-ENDED, it did so after 5 steps and approximately 3k data points.Source: §B.3 · p. 20
  • For MATH, the SKILL-TREE baseline produced its best student after 20 steps using 2.8k datapoints.The SKILL-LIST baseline used roughly 2.1k datapoints and 4 iterations; OPEN-ENDED used 752 datapoints over 10 iterations.Source: §B.3 · p. 20
  • For LiveCodeBench, the SKILL-TREE baseline produced its best student after 10 iterations using 3138 datapoints.OPEN-ENDED used 1.6k datapoints over 5 iterations, while SKILL-LIST used 675 datapoints over 11 iterations.Source: §B.3 · p. 20

B.4 RESOURCE COSTS

The appendices report resource costs, describe the skill forest and its baseline policy, and specify validation and test splits.

B.4 RESOURCE COSTS

  • Coding has the highest token expenditure and multimodal tasks the highest GPU time; GPT-4o-mini reduces costs while retaining performance improvements.OPEN-ENDED code runs cost less than $3, and training typically takes under 30 minutes even on one A6000 GPU.Source: §B.4 · p. 21, §B.5 · p. 22, Table 7 · p. 22

B.5 SKILL FOREST DETAILS

  • The skill forest organizes high-level skill domains into subskills, tracking names, data allocations, training-data performance, and child skills.It supports fine-grained progress tracking and prioritization of data production.Source: §B.5 · pp. 21–23
  • The forest evolves by growing subskills and rebalancing data allocations, actions that correspond to exploring and exploiting the skill tree.The structure supports resource allocation and prioritization across skills.Source: §B.5 · p. 23

B.6 SKILL-TREE POLICY

  • The SKILL-TREE baseline alternates exploration and exploitation until reaching a subskill limit, then allocates data until each subskill reaches capacity.During growth, exploitation resets allocations; the filling phase applies per-action allocation limits.Source: §B.6 · p. 24

B.7 VALIDATION AND TEST SPLITS

  • GQA validation and test sets use stratified samples; MATH validation samples levels and topics, while LiveCodeBench validation uses release-two problems absent from release one.LiveCodeBench validation contains 100 problems; the official MATH test set and first LiveCodeBench release serve as test sets.Source: §B.7 · p. 24

C QUALITATIVE EXAMPLES

The examples illustrate discovered skills and show qualitative changes in a PaliGemma-3B student's answers after training on generated data.

C QUALITATIVE EXAMPLES

  • The examples pair discovered skills with teacher-generated datapoints, including images generated in their entirety.The illustrated domains are GQA, MATH, and LiveCodeBench.Source: C QUALITATIVE EXAMPLES · p. 24
  • Training corrected two illustrated VQA errors: identifying a chair as plastic and distinguishing a giraffe in the foreground from its background.The first correction may reflect debiasing in the synthetic data's representation of chair materials.Source: C QUALITATIVE EXAMPLES · p. 24

D ACCURACY ON GENERATED TRAINING DATA

The analysis tests students on data generated in the following iteration, finding improvement over time but not a monotonic or one-step process.

D ACCURACY ON GENERATED TRAINING DATA

  • Students improve on data generated in the next iteration, which they have not yet seen at evaluation time.The analysis uses iteration n accuracy on data generated in iteration n + 1 for math and multimodal environments.Source: D ACCURACY ON GENERATED TRAINING DATA · p. 24
  • Accuracy improves over successive iterations, indicating that teacher-generated data is consistent and students gradually improve at its target distribution.The reported trend applies to the analyzed math and multimodal environments.Source: D ACCURACY ON GENERATED TRAINING DATA · p. 24
  • Improvement on the generated-data distribution is neither monotonic nor achieved in a single iteration.Source: D ACCURACY ON GENERATED TRAINING DATA · p. 24

E ABLATION: DISENTANGLING THE EFFECT OF ADDITIONAL TRAINING FROM ADDITIONAL DATA

The ablation compares training with all generated data against a smaller subset trained for an equivalent number of epochs. Less data with more epochs produced substantially smaller improvements, while extending training without additional data typically hurt performance.

E ABLATION: DISENTANGLING THE EFFECT OF ADDITIONAL TRAINING FROM ADDITIONAL DATA

Related papers

Loading 2410.06215v3…