PaperlayerView exact revision on arXiv

Source-linked AI summary

MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verification

Saptarshi Sengupta, Harsh Vashistha, Kristal Curtis, Akshay Mallipeddi, Abhinav Mathur, Joseph Ross, Liang Gou

arXiv:2412.04494v2cs.CL

TL;DR

Customer-assistant testing needs representative queries, while trajectory evaluation often relies on LLM judges. MAG-V generates customer-like questions and reverse-engineers alternatives for deterministic verification; its simple ML baselines can match more expensive, capable models.

  • Problem

    Testing custom assistants requires representative customer queries, but collecting them from customers is time-consuming, and trajectory verification can rely on prompt-sensitive, inconsistent API-based LLM judges.

    Source: §1 · pp. 1–2

  • Method

    MAG-V uses LLM agents to generate customer-like questions and reverse-engineers alternate questions from responses for deterministic trajectory verification.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Simple ML baselines with feature engineering can match the performance of more expensive and capable models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Cheaper models such as GPT-4o-mini or open-source models may alleviate the API costs of generating alternate questions and trajectories at scale.

    Source: §4.2 · p. 6

    Scaling with sample count remains to be tested, and the authors plan to improve TF-IDF grounding and smooth labels for ambiguous trajectories.

    Source: §5 · p. 6

Abstract

from arXiv · show

Extending the capabilities of Large Language Models (LLMs) with functions or tools for environment interaction has led to the emergence of the agent paradigm. In industry, training an LLM is not always feasible because of the scarcity of domain data, legal holds on proprietary customer data, rapidly changing business requirements, and the need to prototype new assistants. Agents provide an elegant solution to the above by relying on the zero-shot reasoning abilities of the underlying LLM and utilizing tools to explore and reason over customer data and respond to user requests. However, there are two concerns here: (I) acquiring large scale customer queries for agent testing is time-consuming, and (II) high reliance on the tool call sequence (or trajectory) followed by the agent to respond to user queries may lead to unexpected or incorrect behavior. To address this, we propose MAG-V, a multi-agent framework to first generate a dataset of questions that mimic customer queries; and second, reverse-engineer alternate questions from the responses for trajectory verification. Initial results indicate that our synthetic data can improve agent performance on actual customer queries. Furthermore, our trajectory verification methodology, inspired by distant supervision and using traditional machine learning (ML) models, outperforms a GPT-4o judge baseline by 11% accuracy and matches the performance of a GPT-4 judge on our constructed dataset. Overall, our approach is a step towards unifying diverse task agents into a cohesive framework for achieving an aligned objective.

1 Introduction

Customer-assistant deployment needs realistic test queries, but collecting them from customers is time-consuming; agent trajectory verification also presents challenges. MAG-V addresses both with synthetic-query generation and deterministic trajectory verification.

1 Introduction

  • Customer queries are needed to test assistants before deployment, but asking customers for them is time-consuming and does not scale well.The paper notes that training or fine-tuning per customer is often infeasible because of insufficient data and privacy concerns.Source: §1 · p. 1
  • Agent trajectories are sequences of tool calls, and verifying their correctness is non-trivial; LLM judges can be prompt-sensitive and API-inconsistent.Source: §1 · pp. 1–2
  • MAG-V is a multi-agent framework for generating questions that mimic customer queries and verifying agent trajectories deterministically, without LLM feedback.Its verification approach draws on distant supervision, discriminative models such as SVMs, and LLM response-verification methods.Source: §1 · p. 2
  • Simple ML baselines with feature engineering can match the performance of more expensive and capable models.Source: §1 · p. 2

2 Related Work

Prior work spans LLM-based synthetic-data generation and trajectory evaluation, but the cited approaches retain limitations relevant to MAG-V’s focus on customer-style questions and deterministic verification.

2 Related Work

  • LLM-generated synthetic data has been used for inductive reasoning and question answering, while multi-agent generation for more complex data remains nascent.The cited examples include preference-alignment datasets, conversation datasets, and transformations of seed data at large scales.Source: §2 · p. 2
  • Prior multi-agent data-generation work includes preference-alignment and linear-programming dialogue datasets, whereas MAG-V prioritizes questions and tool-calling.The paper also notes reliance on LLM-as-a-judge or human evaluators in those approaches.Source: §2 · p. 2
  • Trajectory-verification studies use judge models for win-rate or step-by-step tool-call evaluation, retaining LLM-based evaluation that MAG-V seeks to avoid.Source: §2 · p. 2

3 Methodology

MAG-V uses three agents to generate customer-like questions and verify tool-call trajectories. Its method seeds synthetic data from engineer-written queries, then compares trajectories from response-derived alternate questions.

3 Methodology

  • MAG-V assigns query generation, answering, and response-based question generation to investigator, assistant, and reverse engineer agents.The investigator and reverse engineer use GPT-4o-mini; the assistant uses GPT-4o-mini for data generation and GPT-4o for verification.Source: §3 · pp. 2–3

3.1 Data Generation

  • Nineteen engineer-written questions seeded generation; in-context learning produced 190 synthetic questions, filtered to 45, with the most common of five trajectories selected for annotation.The investigator executes a randomly sampled seed trajectory and writes ten questions in the style of an example question.Source: §3.1 · p. 3
  • Two domain experts labeled trajectories as correct or incorrect, agreed on 34 of 45 cases, and produced a final dataset of 30 incorrect and 15 correct trajectories.Their Cohen’s Kappa was 0.42; a third expert resolved disagreements.Source: §3.1 · p. 3

3.2 Trajectory Verification

  • For each base question, three response-derived alternate questions yield alternate trajectories, whose agreement with annotated trajectories grounds verification.The approach draws on distant supervision and backward question generation, treating high trajectory variance among similar questions as lower confidence.Source: §3.2 · p. 3
  • Six trajectory similarity features plus base-question TF-IDF feed discriminative classifiers evaluated with stratified ten-fold cross-validation.The study tests seven ML models and reports mean accuracy and F1 across folds and three random seeds.Source: §3.2 · p. 4

3.2.1 GPT-baseline

  • GPT-4 and GPT-4o judges were tested on the same cross-validation splits and seeds using the annotation rubric and example correct and incorrect pairs.The study also varied label order to address recency bias and tested with and without Autogen’s default system prompt.Source: §3.2.1 · pp. 4–5

4 Experiments

Experiments test synthetic questions as in-context examples and compare trajectory verification with GPT judges. The results show that these samples helped on one customer query, while feature-based ML performed strongly in verification.

4.1 Utilizing Synthetic Data

  • Adding synthetic questions and their most common trajectories as in-context examples helped the assistant identify the correct time range on its first try.Without those examples, GPT-4o used the wrong time range, while GPT-4o-mini retried unsuccessfully even after 10 attempts.Source: §4.1 · p. 5

4.2 Trajectory Verification

  • Including tool arguments improved every ML model, because arguments gave the models finer-grained features and a clearer boundary between correct and incorrect trajectories.Without arguments, identical tool-call sequences could occur in both classes.Source: §4.2 · p. 5
  • 11% accuracy and 4% F1 improvement over GPT-4o were achieved by k-NN, the strongest ML model across accuracy and F1.The comparison uses all features; Figure 2 presents accuracy and F1 for the ML models and GPT baselines.Source: §4.2 · p. 5, Figure 2 · p. 6
  • In the feature ablation, models using only EDIT distance came close to GPT-4 and outperformed GPT-4o in accuracy.The authors reason that EDIT distance gives more signal by measuring the work needed to align trajectories rather than simply penalizing mismatches; Table 1 compares the models with GPT baselines.Source: §4.2 · p. 6, Table 1 · p. 6
  • The verification approach requires three alternate questions and trajectories, increasing API calls and potentially adding cost at scale with expensive models.The authors say cheaper models can alleviate this cost, while their method is deterministic under the same conditions.Source: §4.2 · p. 6

5 Conclusion and Future Work

The conclusion presents MAG-V as a framework for generating synthetic questions with LLM agents and verifying answer trajectories. It identifies scale, question grounding, and label granularity as areas for future work.

5 Conclusion and Future Work

  • MAG-V combines LLM-agent generation of synthetic questions with verification of the trajectories used to answer them.The verification method is deterministic and reduces reliance on API-based LLMs to judge trajectory veracity.Source: §5 · p. 6
  • Future work includes testing how verification scales with sample count and improving how TF-IDF question features ground trajectories to their questions.The authors identify both as areas needing further work.Source: §5 · p. 6
  • The authors plan to replace binary trajectory labels with three classes because some incorrect trajectories may be judged correct given question complexity.They also plan to test their models in that setting.Source: §5 · p. 6

A Filtering Generated Questions

The filtering process clusters question embeddings to select a diverse set, then removes questions that are generic or concern a platform aspect under development.

A Filtering Generated Questions

  • BGE embeddings were reduced to two dimensions with UMAP and grouped by HDBSCAN into 11 manually checked clusters.The clusters were checked for diversity in themes such as service information and hardware issues.Source: A Filtering Generated Questions · pp. 9–10
  • Selecting the five questions nearest each cluster centroid yielded 55 questions, then removing 10 unsuitable questions left a 45-question dataset.Removed questions were generic, required no tool calls, or concerned a platform aspect under active development.Source: A Filtering Generated Questions · p. 10

B Trajectory Features

Trajectory similarity is measured with six features spanning exact equality, sequence edits, graph structure, semantic similarity, argument overlap, and shared starting-call sequences.

B Trajectory Features

  • EM is a binary measure that equals 1 when BT and AT match verbatim and 0 otherwise.Source: B Trajectory Features · p. 10
  • EDIT is the minimum number of insertions, deletions, or substitutions needed to convert AT into BT.Source: B Trajectory Features · p. 10
  • GEDIT measures the edits needed to make the base and alternate trajectory graphs isomorphic, operating on nodes and edges.Source: B Trajectory Features · p. 10
  • SS measures cosine similarity between the base and alternate trajectories embedded with BGE-Large.Source: B Trajectory Features · p. 10
  • AO measures common arguments across tool-call pairs with an F1 score based on precision and recall.Precision divides common arguments by the arguments in AT; recall divides them by the arguments in BT.Source: B Trajectory Features · p. 10
  • LCSS compares the longest common tool-call sequence from the first call, using precision and recall normalized by AT and BT call counts.Precision divides the shared sequence length by AT calls; recall divides it by BT calls.Source: B Trajectory Features · p. 10

Related papers

Loading 2412.04494v2…