PaperlayerView exact revision on arXiv

Source-linked AI summary

TestAgent: Automatic Benchmarking and Exploratory Interaction for Evaluating LLMs in Vertical Domains

Wanying Wang, Zeyu Ma, Xuhong Wang, Yangchun Zhang, Pengfei Liu, Mingang Chen

arXiv:2410.11507v5cs.AIcs.CL

TL;DR

Vertical-domain evaluation remains costly to scale, and static single-turn formats do not capture multi-turn professionalism and stability. TESTAGENT combines RAG-based benchmark generation and two-stage criteria creation with RL-guided adaptive questioning. Experiments across medical, legal, and government domains demonstrate its adaptability and discriminative power.

  • Problem

    Vertical-domain benchmarks are limited and costly to construct, while static single-turn evaluation does not assess multi-turn professionalism and stability.

    Source: §1 · pp. 1–2

  • Method

    TESTAGENT generates domain questions and two-stage criteria from user-provided knowledge sources, then uses RL-guided interactions to adapt question types to model responses.

    Source: §1 · p. 2

  • Results

    Experiments across medical, legal, and government domains demonstrate TESTAGENT's adaptability and discriminative power.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TESTAGENT offers efficient benchmark construction and insights into LLM performance in vertical-domain evaluation.

    Source: §4 · p. 8

Abstract

from arXiv · show

As Large Language Models (LLMs) are increasingly deployed in highly specialized vertical domains, the evaluation of their domain-specific performance becomes critical. However, existing evaluations for vertical domains typically rely on the labor-intensive construction of static single-turn datasets, which present two key limitations: (i) manual data construction is costly and must be repeated for each new domain, and (ii) static single-turn evaluations are misaligned with the dynamic multi-turn interactions in real-world applications, limiting the assessment of professionalism and stability. To address these, we propose TestAgent, a framework for automatic benchmarking and exploratory dynamic evaluation in vertical domains. TestAgent leverages retrieval-augmented generation to create domain-specific questions from user-provided knowledge sources, combined with a two-stage criteria generation process, thereby enabling scalable and automated benchmark creation. Furthermore, it introduces a reinforcement learning-guided multi-turn interaction strategy that adaptively determines question types based on real-time model responses, dynamically probing knowledge boundaries and stability. Extensive experiments across medical, legal, and governmental domains demonstrate that TestAgent enables efficient cross-domain benchmark generation and yields deeper insights into model behavior through dynamic exploratory evaluation. This work establishes a new paradigm for automated and in-depth evaluation of LLMs in vertical domains.

1 Introduction

Vertical-domain evaluation needs scalable, domain-specific benchmarks and dynamic interactions, but existing approaches rely on manual construction and static single-turn formats. TestAgent combines RAG-based question generation and two-stage criteria generation with RL-guided multi-turn evaluation.

1 Introduction

  • Manual vertical-domain benchmark construction is labor-intensive and must restart from scratch when new domains emerge, limiting efficient assessment.Existing benchmarks largely target general domains, with some domain-specific datasets such as LegalBench and MedBench still relying heavily on manual collection.Source: §1 · p. 1
  • Static single-turn evaluations miss the depth of interrelated domain knowledge and can overlook contradictory responses that undermine trust in professional consultations.Manually constructing interactive dialogues would further increase evaluation costs.Source: §1 · p. 1
  • 93.3% of surveyed LLM companies prioritize domain-specific evaluation, while over 90% of providers struggle to build efficient benchmarks and more than 70% need dynamic evaluation.The survey covered 50 LLM companies.Source: Figure 1 · p. 2, §1 · p. 2
  • TestAgent uses RAG to generate questions from user-provided domain knowledge and refines topic-level criteria into question-relevant criteria for automated benchmark construction.The criteria process first generates coarse topic-level criteria via in-context learning, then refines them using retrieved question-relevant information.Source: §1 · p. 2
  • Its RL-guided interaction approach selects subsequent question types from target-model responses to probe domain expertise and stability across turns.The framework combines automated cross-domain benchmark generation with dynamic evaluation and reports experiments in medical, legal, and government domains.Source: §1 · p. 2
  • TestAgent aims to deliver comprehensive evaluation metrics aligned with real-world requirements through multi-turn assessment of expertise and stability.The introduction frames these dimensions as important for vertical-domain evaluation.Source: §1 · p. 2

2 Methodology

TESTAGENT builds domain-specific questions and criteria from user-provided knowledge, then uses response-guided multi-turn interactions to assess models across dynamism, professionalism, and stability.

2 Methodology

  • TESTAGENT combines RAG-based benchmark construction with RL-guided multi-turn interactions to evaluate professionalism and stability across vertical domains.The framework generates questions and two-stage criteria from domain documents, then uses response quality to guide interaction.Source: §2 · p. 2

2.1.1 Topic-based Initial Question Generation

  • For each user topic, RAG retrieves relevant domain-document knowledge to prompt the kernel model to generate role-play initial questions representative of downstream applications.The retrieved material comes from a domain-specific knowledge base constructed from sources such as textbooks and real-world case records.Source: §2.1.1 · pp. 2–3

2.1.2 Two-Stage Criteria Generation

  • Two-stage criteria generation first creates structured topic-level criteria from demonstrations, then refines them with retrieved question-specific details to produce verifiable evaluation points.The resulting criteria are reused throughout the multi-turn sequence initiated by the initial question.Source: §2.1.2 · pp. 3–4

2.2.1 Follow-up Question Generation

  • Because retrieved criteria cover related knowledge, they support contextually related follow-up questions, while fixed follow-ups alone cannot probe dynamic behaviors such as consistency and stability.The framework therefore adjusts its question strategy in real time using feedback from prior responses.Source: §2.2.1 · p. 4

2.2.2 RL-based Subsequent Question Decision

  • The RL policy chooses between follow-up and challenge questions using response scores, score changes, response similarity, and the previous action.Its reward encourages challenges for unchanged scores and follow-ups when score changes exceed a threshold; the design aims to probe knowledge boundaries and stability.Source: §2.2.2 · pp. 4–5

2.3 Multi-Dimensional Evaluation Metrics

  • Evaluation covers dynamism through multi-turn scores and score changes, professionalism through accuracy, satisfaction, and clarity, and stability through coherence and question- and challenge-handling.Professionalism and stability are qualitatively evaluated by GPT-4o.Source: §2.3 · p. 5

3 Experiments

Experiments assess TESTAGENT across three vertical domains and benchmark settings through human meta-evaluation, model comparisons, and tests of adaptive multi-turn evaluation. Results show reliable evaluation data and scoring, while dynamic and RL-guided evaluation reveal model differences and response-dependent questioning.

3.1 Experiment Setup

  • Experiments cover government, medical, and legal domains, using DeepSeek-V3 as kernel model and evaluating seven target LLMs plus three medical-specific models.Knowledge sources include medical disease documents and department introductions, legal crime and judgment data, and government fund-management data.Source: §3.1 · p. 5

3.2 Benchmark Comparison

  • TESTAGENT uniquely combines from-scratch cross-domain vertical benchmark generation with exploratory multi-turn interactions that adapt questions to response quality.The compared existing benchmarks rely on static datasets or target general capabilities.Source: §3.2 · pp. 5–6, Table 2 · p. 6, Table 1 · p. 6

3.3 Meta-Evaluation

  • Human ratings found consistently high question and criteria quality with minimal variance, and experts reported high agreement with kernel-model scoring.GPT-4o achieved mean agreement µ = 0.974 with variance σ2 = 0.025; DeepSeek-V3 achieved µ = 0.957 with σ2 = 0.040.Source: Table 3 · p. 6, §3.3 · p. 6

3.4 Main Results

  • Across medical, legal, and government evaluations, GPT-4o ranks highest in medical and legal, while GPT-4o and Qwen2.5-72B-Instruct share top government Scores.Medical Qwen2.5-72B-Instruct leads stability; DeepSeek-V3 leads legal rational analysis and error correction; no government model exceeds a Score of 60. Evaluations used 100 conversations of six turns per domain.Source: §3.4 · pp. 6–7, Table 4 · p. 7

3.5 Activating Existing Benchmarks

  • On SQuAD, TESTAGENT turns independent questions into dynamic multi-turn interactions, and all models achieve Scores above 95.The dynamic metrics reveal multi-turn contextual understanding and knowledge stability; DeepSeek-V3 attains the highest expertise and exceptional stability.Source: Table 5 · p. 7, §3.5 · pp. 7–8

3.7 RL Improves Evaluation Quality

  • TESTAGENT achieves near-perfect Flaw Detection and Exploratory Inquiry Rates (≈1), versus near-random rates (≈0.5) for the Untrained baseline.The Q-Embedding variant fails to learn effective strategies, while a direct LLM strategy consistently chooses follow-ups regardless of response quality.Source: Figure 3 · p. 8, §3.7 · p. 8

3.6 Effect of Multi-turn Evaluation

  • On legal judgment analysis, dynamic multi-turn evaluation has 4.3× greater discriminative power than static single-turn evaluation.Static questions show minimal performance differences, whereas dynamic evaluation incorporates multi-turn professional dialogue and consistency.Source: Figure 4 · p. 8, §3.6 · p. 8

4 Conclusion

TESTAGENT automates vertical-domain benchmark construction and uses RL-guided multi-turn interaction to assess LLM performance. Experiments demonstrate the framework’s versatility and effectiveness for evaluating professionalism and stability.

  • TESTAGENT uses RAG to automatically construct benchmarks from vertical-domain documents across multiple domains.Source: §4 · p. 8
  • Its RL-based exploratory interaction dynamically selects questions based on model performance to assess professionalism and stability across multi-turn dialogues.Source: §4 · p. 8
  • Experiments demonstrate the framework’s versatility and effectiveness, and it offers insights into LLM performance in vertical-domain evaluation.Source: §4 · p. 8

A RL Algorithm

Algorithm 1 iteratively refines question generation to explore a target model’s performance, using its responses to guide multi-turn interactions and recording transitions for policy improvement.

A RL Algorithm

  • The procedure iterates until its step limit, with the stated aim of improving question generation to explore the target model’s performance more effectively.It initializes a random policy and an empty transition buffer.Source: A RL Algorithm · p. 12
  • At the outset, the kernel model generates a question and criteria from the knowledge base and topic, then the target model answers and receives a criteria-based score.The initial state repeats that score across its three components, and the interaction history stores the question, answer, and score.Source: A RL Algorithm · p. 12
  • Each turn uses the selected action to generate either a history-based follow-up question or a challenge, with both conditioned on the criteria.The interaction runs across the specified turns per episode.Source: A RL Algorithm · p. 12
  • The procedure scores response A′t+1 against criterion C as S(C, At+1) = K(ps, C, A′t+1), then adds (st, at, rt, st+1) to buffer B.Source: A RL Algorithm · p. 12

B Related Work

Prior work includes static multi-turn benchmarks and dynamic evaluation frameworks, but the cited literature identifies limited vertical-domain automation and cross-domain benchmark generation. TestAgent combines automated domain-specific construction with response-adaptive interactions.

Multi-turn Evaluation

  • MT-bench, MT-bench-101, MT-Eval, and a user-reported scenario dataset represent prior approaches to multi-turn evaluation.They use two-turn questions, a hierarchical ability taxonomy, augmented datasets, or authentic user feedback, respectively.Source: B Related Work · p. 12

Dynamic Evaluation

  • DyVal, KIEval, AgentBench, and MINT explore dynamic evaluation, while existing benchmarks predominantly use static datasets rather than response-adaptive questions.The cited approaches address reasoning, repeated confirmation, open-ended agent settings, or tool-supported multi-turn tasks.Source: B Related Work · p. 12
  • Existing dynamic frameworks focus mainly on general capabilities, while vertical benchmarks such as LegalBench and MedBench rely heavily on manual collection and lack cross-domain generation from scratch.The passage identifies limited scalability, adaptability, and dynamic multi-turn dialogue generation as related shortcomings.Source: B Related Work · p. 13
  • TestAgent proposes automated benchmark construction for arbitrary vertical domains and dynamically builds multi-turn dialogues from real-time model responses.It frames question generation as an RL-guided choice between follow-up questions and challenges based on model responses.Source: B Related Work · p. 13

C Industry Survey Subjects

The survey spans companies in three layers of the LLM industry chain: infrastructure, models and tools, and applications.

C Industry Survey Subjects

  • The infrastructure layer survey covers 16 companies, including computing centers and cloud providers.Named examples include Shanghai Supercomputer Center and Alibaba Cloud East China Intelligent Computing Center.Source: C Industry Survey Subjects · p. 13
  • The model and tools layer includes 20 surveyed companies, with examples such as Shanghai SenseTime Intelligent Technology and Shanghai Artificial Intelligence Laboratory.Source: C Industry Survey Subjects · p. 13
  • The application layer comprises 34 surveyed companies, including firms in technology, digital health, and artificial intelligence.Examples include Transwarp Technology, iFLYTEK, and Silicon Intelligence Technology.Source: C Industry Survey Subjects · p. 13

D Case Study

The medical-domain cases illustrate TestAgent’s evaluation process, including benchmark generation and automatic assessment against realistic needs.

D Case Study

  • The case study uses examples from the medical domain to illustrate the evaluation process.Source: D Case Study · p. 13
  • The cases are presented as evidence that TestAgent can generate benchmarks for vertical domains.Source: D Case Study · p. 13
  • The described process includes automatic assessment intended to meet realistic needs.Source: D Case Study · p. 13

Initial Questions for Disease Topic

The section lists example disease-topic questions covering symptoms, clinical manifestations, emergency treatment, diagnostic criteria, causes, and prevention.

General Criterion for Disease Topic

This section presents a disease-topic criterion spanning disease fundamentals, risk and care, medications, and diet, and illustrates its use in evaluating allergic-shock follow-up interactions with summarized answers and criterion-based scores.

General criterion

  • The criterion requires disease name, description, classification, and prevention information, including the disease’s full name and feasible preventive measures.The allergic-shock example specifies allergy-history review, minimizing unnecessary injections, preferring oral formulations, post-injection observation, and skin-prick testing.Source: General Criterion for Disease Topic · pp. 14–15
  • It also assesses causes, pathogenesis, and symptoms, with the example describing allergen-triggered IgE responses and a subsequent Type I hypersensitivity reaction.Example symptoms include hives, breathing difficulty, dizziness, pallor, fainting, altered consciousness, palpitations, and hypotension.Source: General Criterion for Disease Topic · pp. 14–15
  • Additional dimensions cover insurance, incidence, susceptible populations, transmission, comorbidities, and the appropriate consultation department.For anaphylactic shock, the example gives 0.001% incidence, identifies allergic constitution as a risk, specifies non-contagiousness, and recommends the emergency department.Source: General Criterion for Disease Topic · p. 14
  • The criterion requests main pharmacological and non-pharmacological treatment approaches, approximate treatment duration, disease cure-rate data, common medications, estimated treatment-cost range, and possible diagnostic examinations and tests.Source: General Criterion for Disease Topic · p. 14
  • Dietary guidance, suitable recipes, recommended medication brands or manufacturers, and detailed medication information complete the 24-part criterion.The example provides suggested and prohibited foods and extensive drug-manufacturer listings, but reports no recommended recipes.Source: General Criterion for Disease Topic · pp. 14–16

Example interaction

  • Scores rose from 2/25 initially to 3/25 after turn 1 and 3.5/25 after turn 2, despite repeated emergency-treatment summaries.The initial response received credit only for disease name and treatment method; later turns added credit for causes and partial pathogenesis.Source: General Criterion for Disease Topic · pp. 17–19
  • At turn 3, the model affirmed vascular spasms and neural reflexes as main mechanisms, while the evaluation remained at 3.5/25.The criterion’s example instead describes an allergen-triggered immune response involving IgE and Type I hypersensitivity.Source: General Criterion for Disease Topic · pp. 15, 19

General Criterion for Disease Topic

  • The symptoms response lists breathing difficulty, rapid heartbeat, confusion, cyanosis, gastrointestinal symptoms, dizziness, and other allergic-shock symptoms, then describes emergency treatment.The summarized treatment includes intravenous epinephrine, vital-sign monitoring, CPR or artificial respiration if breathing difficulty occurs, and hospital transfer for supportive care.Source: General Criterion for Disease Topic · pp. 19–20
  • Both allergic-shock responses score 6/25, earning full credit for disease name, causes, symptoms, and treatment method while leaving many other criteria unaddressed.Both receive partial credit for disease description and pathogenesis; classification, prevention, insurance status, incidence probability, vulnerable population, and numerous other criteria score zero.Source: General Criterion for Disease Topic · p. 20
  • The pathogenesis response names allergen-triggered reactions, abnormal immune responses, vascular spasms, neural reflexes, and associated neurological and respiratory symptoms, while noting that each case needs individual assessment.Its summarized answer likewise emphasizes emergency intervention and hospital treatment.Source: General Criterion for Disease Topic · p. 20

E Prompt

The appendix prompts specify how to generate topic criteria, refine question-specific criteria, score responses, and produce initial, follow-up, and challenge questions.

Prompt for General Domain-level Criteria

Prompt for Refined Question-level Criteria

Prompt for Scoring

  • The scoring prompt asks evaluators to list each criterion’s maximum score, score the response criterion by criterion, and report a total.Source: Prompt for Scoring · p. 21

Prompt for Initial Questions

Prompt for Follow-up Questions

Prompt for Challenges

Prompt for Qualitative Evaluation

The qualitative evaluation prompt assesses professionalism and stability across responses, with separate dimensions and strict numerical scoring instructions.

  • Professionalism is assessed through factual accuracy, meeting user needs, and clarity.Source: Prompt for Qualitative Evaluation · p. 22
  • Stability is assessed through consistency, handling challenges, and responding to follow-up questions.The prompt says longer responses are not always better and prefers concise responses that meet the stated requirements.Source: Prompt for Qualitative Evaluation · p. 22
  • Evaluators score each dimension from 1 to 10, then assign separate overall professionalism and stability scores using strict quality-based rules.The rubric assigns 1–2 for irrelevant, fundamentally erroneous, or harmful responses; 9–10 requires substantially exceeding the reference and near-perfect performance.Source: Prompt for Qualitative Evaluation · p. 22
  • The required output returns integer scores in a dictionary, including dimension scores and overall scores.Source: Prompt for Qualitative Evaluation · p. 23
Loading 2410.11507v5…