PaperlayerView exact revision on arXiv

Source-linked AI summary

RL-ADA: A World-Feedback Framework for Adversarially Robust Enterprise Dialogue Agents

Ram Narayanan, Harshit Rajgarhia, Abhishek Mukherji

arXiv:2609.02902v1cs.CL

TL;DR

Enterprise dialogue robustness requires large volumes of labelled interaction data, yet privacy-sensitive, costly-to-annotate logs create an annotation bottleneck. RL-ADA co-evolves support and adversarial customer agents using outcome-based world feedback; in a banking proof of concept, routing errors were eliminated and strict PASS doubled from 25% to 50% over five cycles.

  • Problem

    Enterprise dialogue robustness requires large labelled datasets despite privacy-sensitive, costly-to-annotate logs, and annotation-free methods leave asymmetric support-agent/adversarial-customer settings unaddressed.

    Source: §1 · p. 1

  • Method

    RL-ADA co-evolves support and adversarial customer language-model agents using measurable-outcome world feedback, opposing rewards, and a fixed judge without human annotation.

    Source: §1 · p. 1

  • Results

    Strict PASS doubled from 25% to 50%, and all routing errors were eliminated on a 12-scenario held-out benchmark over five cycles.

    Source: §8 · p. 5

  • Takeaways & Limitations

    The authors identify empirical demonstration of cross-domain transfer as the immediate next step.

    Source: §8 · p. 5

    Evidence is preliminary: experiments cover one banking domain, few matches, and held-out scenarios, so cross-domain generalization and finer statistical separation are unestablished.

    Source: §6 · p. 4, §8 · p. 5

Abstract

from arXiv · show

Deploying task-oriented dialogue agents in enterprise customer support faces a persistent annotation bottleneck: robust training requires labelled interaction data at scale, yet enterprise conversational logs are privacy-sensitive and expensive to annotate, while user behaviour evolves faster than labelling pipelines can keep pace. We present RL-ADA (Reinforcement Learning with Adversarial Dialogue Agents), a co-evolutionary training framework that eliminates this bottleneck by replacing human labels with \emph{world feedback}: consequence-based reward signals derived directly from measurable interaction outcomes. A Customer Support Agent (DA, 3B parameters) and an Adversarial Customer Agent (CA, 7B parameters) co-evolve in an adversarial arena guided by a fixed automated judge: the DA is rewarded for correctly handling multi-turn customer conversations to successful resolution, while the CA is rewarded for producing realistic, intent-concealing utterances that cause misroutes, creating asymmetric adversarial pressure through opposing but independently structured rewards. An isolation gym iteratively retrains the weaker agent on prior-failure transcripts, requiring no human annotation at any stage. In a banking customer support proof of concept, tool-routing errors are eliminated and the strict end-to-end PASS rate doubles over five co-evolutionary cycles, driven solely by automated arena reward with no labelled data. We additionally observe the emergence of \textbf{Contextual Camouflage}, an adversarial strategy in which the CA learns to embed intent within dense realistic customer detail purely from reward pressure, with direct implications for enterprise red-teaming and robustness evaluation.

1 Introduction

RL-ADA addresses the enterprise dialogue annotation bottleneck by training asymmetric support and adversarial customer agents with outcome-based world feedback instead of human labels. Its introduction also specifies an isolation gym, a rolling win-rate stopping criterion, and a banking support instantiation.

1 Introduction

  • Enterprise dialogue agents need labelled data to remain robust to unpredictable, adversarial, evolving users, but privacy-sensitive logs and costly annotation create a bottleneck.Agents that perform well on evaluation benchmarks may still degrade under adaptive pressure from real users.Source: §1 · p. 1
  • RLHF [12] relies on costly human annotation, while self-play [4] assumes symmetric agents and risks non-stationarity and catastrophic forgetting.Neither directly handles annotation-free customer support with distinct support and adversarial customer roles, action spaces, capacities, and rewards.Source: §1 · p. 1
  • RL-ADA trains a 3B Customer Support Agent and 7B Adversarial Customer Agent using measurable outcome-based world feedback, with no human annotation.The co-evolutionary loop combines rule-based outcome signals with a fixed automated judge, whose reliability is evaluated against GPT-4o-mini.Source: §1 · p. 1
  • The isolation gym retrains the weaker agent on a 70:30 failure/success transcript mix, while rolling win-rate stability serves as a macro-level stopping criterion.The paper frames that criterion as a practical surrogate for empirical ε-Nash convergence and the isolation gym as a training-side mechanism for a post-deployment data flywheel.Source: §1 · p. 1
  • In the banking instantiation, the DA routes free-text utterances across six API tools and 78 customer intents using call outcomes without human labels.Customers do not name their intent directly, and adversarial phrasing can trigger the wrong tool.Source: §1 · p. 1

2 Related Work

RL-ADA is positioned as an outcome-feedback alignment alternative and an asymmetric adversarial dialogue co-evolution framework, distinct from fixed-target red-teaming and cooperative-user evaluation.

RLHF and AI feedback

  • RL-ADA replaces human-preference feedback [12] and AI feedback [1] with interaction-outcome episode scores, removing annotation dependence while introducing judge-reliability uncertainty.The framework uses structured episode scores rather than human labels.Source: §2 · p. 1

Self-play and game-theoretic RL

  • RL-ADA’s isolation gym is a one-oracle-per-cycle Double Oracle inner loop [8], with curated failure/success curricula and freeze-one-train-one training to avoid asymmetry-induced cycling [2] [10].This differs from symmetric self-play exemplified by AlphaGo, OpenAI Five [4], and AlphaStar.Source: §2 · p. 2

LLM red-teaming and concurrent work

  • Unlike fixed-target automated red-teaming [17] [13] and safety/reasoning self-play [18] [7], RL-ADA co-evolves a 3B DA and 7B CA for asymmetric tool-routing attacks under an explicit convergence criterion.The adversarial customer agent co-evolves with the dialogue agent rather than attacking a fixed model.Source: §2 · p. 2

Task-oriented dialogue evaluation

  • Where cooperative-user benchmarks [19] [3] evaluate realistic task completion, RL-ADA adversarially trains its customer agent to cause failures.The contrast is the customer agent’s adversarial objective rather than cooperative task completion.Source: §2 · p. 2

3 System Architecture

RL-ADA co-trains a 3B Customer Support Agent (DA) and a 7B Adversarial Customer Agent (CA) using a fixed automated judge and outcome-based rewards. Its architecture combines bootstrapping, adversarial matches, failure-focused retraining, and a win-rate stopping rule.

3 System Architecture

  • RL-ADA assigns the DA to clarify, route tool calls, and end calls; the CA generates customer-style utterances designed to elicit misroutes; a fixed Judge scores terminal episode quality.The DA is Qwen2.5 3B, while the CA and Judge are Qwen2.5 7B; training proceeds across three phases.Source: §3 · p. 2

3.1 Automated Judge

  • NeutralJudge matches GPT-4o-mini in resolution-detection F1 (0.807 vs. 0.821), ranks quality tiers in the same order (ρ=0.833), and achieves hallucination-detection F1 1.000.The evaluation uses 38 conversations; the tier Spearman score uses the 8 synthetic conversations only. Table 1 compares NeutralJudge and GPT-4o-mini.Source: §3.1 · p. 2, Table 1 · p. 2

3.2 Phase 0 – Bootstrap

  • The DA is bootstrapped with supervised tool-routing demonstrations and refined with GRPO, while the CA has no SFT initialization and learns against a fixed DA checkpoint.At each turn, the DA sees only conversation history and chooses whether to call a tool, speak, or end the call; 78 intent strings map many-to-one to six tools, and reward depends on outcomes rather than intent labels.Source: §3.2 · p. 2

3.3 Reward Design

  • The DA’s total reward combines rollout and terminal-judge feedback with turn-level signals, while required-step failures incur mandatory deductions and the CA is rewarded for inducing misroutes.Missing identity verification deducts −1.2 and a missing domain tool deducts −1.8; the CA reward includes format, realism, concealment, and inverse-DA terms. Reward weights were chosen from early stability observations and are tunable by domain.Source: §3.3 · pp. 2–3

3.4 Adversarial Arena

  • Each arena match runs 180 episodes across 10 scenarios, measures DA wins over CA-winnable scenarios, and sends the match outcome to the weaker agent’s Isolation Gym if training continues.Fraud-escalation scenarios are tracked separately as a safety reference check; matches use the latest checkpoints of both agents.Source: §3.4 · p. 3

3.5 Isolation Gym

  • The Isolation Gym retrains the weaker agent against a frozen opponent using recent transcripts mixed 70:30 between failures and successes, with one GRPO sample per dialogue decision point.The DA and CA gyms share this transcript-based structure but use different generation counts, context and completion lengths, and reward functions; the CA reward combines format, realism, concealment, and inverse-DA terms.Source: §3.5 · p. 3

3.6 Stopping Criterion

  • Training stops after at least 3 matches when DA win rate reaches 0.60 for two consecutive matches and the latest inter-match change is ≤0.05, or when 8 matches are exhausted.Win-rate convergence against a frozen opponent is used as a practical exploitability proxy, since exact best-response computation is intractable [8].Source: §3.6 · p. 3

4 Experiments

Experiments use 4-bit LoRA training and evaluate agents through arena matches and held-out scenarios. Checkpoint names distinguish the baselines from agents after successive isolation gym cycles.

4.1 Setup

  • All agents use 4-bit LoRA on Qwen2.5-3B for the DA and Qwen2.5-7B for the CA and Judge.Full hyperparameters and training rationale are in Appendix B.Source: §4.1 · p. 3
  • Arena matches average 18 episodes per scenario across 10 scenarios and 2 runs, while held-out evaluation uses 12 fixed scenarios.Source: §4.1 · p. 3
  • DA0 is SFT-initialized and GRPO-warmed, whereas CA0 is trained against fixed DA0 without SFT; later DA1, DA2, CA1, and CA2 checkpoints follow gym cycles.Each subscript increment represents one completed isolation gym cycle for that agent.Source: §4.1 · p. 3

5 Results

Across five matches, arena outcomes fluctuated around adversarial gym cycles; after five co-evolutionary cycles, DA2 eliminated held-out routing errors and doubled strict PASS, while the CA developed Contextual Camouflage.

5.1 Arena Progression

  • DA win rate dropped at matches 3 and 5 after CA gym cycles, then recovered after DA retraining, a pattern consistent with co-evolutionary pressure.The pattern is qualified by only five matches and high per-match variance (±8pp); stopping fired at match 5 (|δ|=0.02 ≤ ε, P=2), with three of eight maximum cycles unused.Source: §5.1 · p. 3, Figure 2 · p. 4, Table 3 · p. 4

5.2 DA Improvement on Held-out Evaluation

  • 50% strict PASS for DA2 versus 25% for DA0, while DA2 routed correctly on every held-out scenario versus 75% in DA0.The fixed held-out set had 12 scenarios and is described as indicative; PASS required correct routing, lookup_account first, rDA ≥2.0, and a clean ending. The aggregate FAIL rate remained 33%, shifting from routing errors to procedural and conversation-quality failures; training used only automated arena reward, without labelled data.Source: §5.2 · pp. 3–4, Table 4 · p. 4

5.3 Emergent Behaviour: Contextual Camouflage

  • Direct Intent rises (25→34→37) as Vague/Indirect falls (25→16→13), indicating increasingly specific misdirection through realistic contextual detail rather than vagueness.The CA embeds intent among merchant names, transaction amounts, and incident context; this qualitative behavior emerges from reward for maximizing DA misroutes without explicit specification.Source: §5.3 · p. 4

6 Limitations •

The framework’s limitations concern configurable stopping criteria, narrow evaluation scope, untested reward and hyperparameter choices, transfer to live deployment, and role consistency under longer training.

Stopping criterion generality

  • Stopping uses fixed pragmatic competency and stability thresholds; configurable per-domain gates covering scenario coverage, compliance, and customer-segment floors remain future work.Production criteria would combine non-negotiable scenario coverage, domain-specific compliance gates, and customer-segment performance floors.Source: §6 · p. 4

Evaluation scope

  • Evaluation is limited to one banking domain, few matches, and held-out scenarios, leaving cross-domain generalisation and finer-grained statistical separation unestablished.Source: §6 · p. 4

Ablations and sensitivity

  • Reward-component contributions remain unquantified, while reward weights, thresholds, the 70:30 transcript mix, and gym hyperparameters were chosen pragmatically without systematic sweeps.Systematic sweeps and reward ablations are left to future work.Source: §6 · p. 4

Live-deployment transfer

  • The data flywheel is demonstrated only in arena training; production requires a deployment-time outcome signal in place of the judge’s terminal score.Existing telemetry such as resolution markers, callback rates, and escalation traces could provide that signal without new annotation.Source: §6 · p. 4

Role-reversal under prompt mitigation

  • CA outputs occasionally drift toward bank-agent impersonation; stronger system prompts mitigate this, but longer-training robustness is unproven and a role-consistency reward remains future work.Source: §6 · p. 4

7 Production Deployment Considerations

Production deployment requires an outcome-based feedback loop that remains future work, a compliance-focused gate separate from training convergence, and domain-specific configuration of otherwise domain-agnostic components.

Toward a post-deployment data flywheel

  • Failure transcripts feed retraining in a 70:30 failure/success mix without human annotation, but production still needs measurable outcome signals to replace the judge’s terminal score.Potential signals include resolution markers, callback rates, and agent escalations; closing the end-to-end loop is left to future work.Source: §7 · p. 5

Deployment gating

  • Production promotion requires non-negotiable scenarios to meet a higher threshold, such as ≥90% DA win rate against the latest CA, independently of training convergence.Examples include fraud detection, account compromise, and identity verification; this gate is orthogonal to the §3.6 macro stopping criterion.Source: §7 · p. 5

Domain transferability

  • RL-ADA’s arena, isolation gym, and world-feedback reward transfer across domains when outcomes are measurable, with intent maps, judge validation, and deployment gates configured per domain.The 78-intent banking taxonomy can be replaced by a domain-specific intent-to-tool map, and scenario thresholds can reflect compliance and escalation costs.Source: §7 · p. 5

8 Conclusion · A Intent-to-Tool Taxonomy · B Training Configurations

RL-ADA uses outcome-based world feedback to train adversarial dialogue agents without human annotation, while the accompanying taxonomy groups intents into routing tools and exposes escalation imbalance as a challenge. The reported results are preliminary and limited to one banking domain.

8 Conclusion

  • All routing errors were eliminated on a 12-scenario held-out benchmark, strict PASS rose from 25% to 50%, and average episode reward rose from +1.58 to +2.16 over five cycles.Training used only automated arena reward and no labelled data.Source: §8 · p. 5
  • The win-rate trajectory was non-monotonic, with DA performance dropping after CA isolation-gym cycles; a win-rate-stability stopping criterion identified convergence at cycle 5, leaving three of eight cycles unused.Source: §8 · p. 5
  • The trained CA developed Contextual Camouflage, embedding intent in dense, specific customer detail rather than naming it directly, solely under reward pressure.Source: §8 · p. 5
  • The findings are preliminary and from one banking domain; empirical cross-domain transfer is the immediate next step, although the components rely on measurable outcomes rather than domain-specific labels.Source: §8 · p. 5

A Intent-to-Tool Taxonomy

  • Banking77 intents and six core synthetic labels are grouped many-to-one into six routing tools: 78 intent strings in total, with 38 assigned to transfer_to_human.Core intents are used in all arena scenarios; Banking77 extensions add vocabulary diversity during SFT and GRPO warm-up.Source: A Intent-to-Tool Taxonomy · p. 6, Table 6 · p. 6
  • The transfer_to_human skew creates class imbalance: over-escalation can score well on coverage while failing on the five specific tools, a failure mode shown by DA0 that CA-winnable scenarios are designed to stress-test.Source: A Intent-to-Tool Taxonomy · p. 6

B.1 DA Training: Baseline vs. Isolation Gym … C.1 Contextual Camouflage — CA Wins by Burying Intent

The supplied passages contrast DA training before and after isolation-gym standardization, describe the CA gym’s training constraints, and show a CA2 utterance that leads DA2 to misroute a duplicate-charge dispute. The episode illustrates intent concealed by self-doubt phrasing.

B.1 DA Training: Baseline vs. Isolation Gym

  • DA0 preceded standardization of the isolation-gym protocol and used a higher learning rate and more training steps.DA1 and DA2 were produced using the standardized protocol parameters listed in Table 7.Source: §B.1 · p. 6, Table 7 · p. 6
  • DA1 and DA2 used the standardized isolation-gym parameters listed in Table 7.The passage distinguishes these runs from DA0, which used a higher learning rate and more steps before the protocol was standardized.Source: §B.1 · p. 6, Table 7 · p. 6

B.2 CA Isolation Gym

  • All CA isolation-gym cycles producing CA1 and CA2 used the configuration summarized in Table 8.The passage specifies the shared cycle configuration but does not provide its individual parameter values here.Source: §B.2 · p. 6, Table 8 · p. 6
  • The CA gym uses cosine learning-rate decay, fewer generations per step than the DA gym, and a tighter KL constraint (β=0.05) to keep outputs natural-sounding.The passage attributes these choices to the CA’s shorter utterance space and its natural-language-register constraint.Source: §B.2 · p. 6

C.1 Contextual Camouflage — CA Wins by Burying Intent

  • DA2 escalates a duplicate-charge dispute to a human instead of using the expected dispute_charge tool after CA2 describes possible duplicate drinks or payment confusion.The scenario is dispute-duplicate-v2; DA2 first looks up the account, then transfers to a human.Source: §C.1 · p. 6
  • The CA embeds dispute intent in self-doubt phrasing, leading the DA to escalate rather than file a dispute.The passage characterizes this as the CA burying intent in realistic customer detail.Source: §C.1 · p. 6

C.2 DA Improvement — Before and After Training · D System Prompts

The before-and-after example shows DA2 resolving an unrecognized charge dispute correctly without new labelled examples; the system prompts define the DA’s tools and call protocol and the customer caller’s role and style.

C.2 DA Improvement — Before and After Training

  • For an unrecognized $89 AMZN charge, DA0 wrongly transferred to a human, while DA2 used lookup_account then dispute_charge and closed the call correctly.The example shows DA2 identifying the dispute intent and completing the call.Source: §C.2 · p. 6
  • DA2 correctly identifies the dispute intent and closes the call without any new labelled examples.Source: §C.2 · p. 6

D.1 DA System Prompt (SFT + GRPO)

  • The DA prompt lists account lookup, transaction retrieval, charge disputes, card-status checks, fund transfers, and human escalation as available tools.Source: §D.1 · p. 6
  • The DA prompt requires lookup_account first, one domain tool next, then an explanation and end_call when the customer is satisfied.Source: §D.1 · p. 6
  • The DA prompt requires valid JSON with thinking, action_type, content, and tool_args fields.The action_type options are speak, tool_call, or end_call.Source: §D.1 · p. 6

D.2 CA System Prompt

  • The CA prompt casts it as a confused or frustrated banking customer who called support and must not offer help or diagnose.It directs the caller to express the problem naturally and indirectly in replies of 1–2 sentences.Source: §D.2 · p. 7

E Per-Scenario Match 5 Breakdown

In Match 5, CA2 pressure concentrates on ambiguous-intent scenarios, while DA2 retains near-perfect accuracy on clear-intent scenarios.

E Per-Scenario Match 5 Breakdown

  • CA2 pressure concentrates on dispute_duplicate, unknown_merchant, and internal_transfer, while DA2 retains near-perfect accuracy on clear-intent scenarios.Figure 3 reports 18 episodes per scenario; solid bars mark CA-winnable scenarios and hatched bars mark DA-baseline fraud scenarios.Source: Figure 3 · p. 7, E Per-Scenario Match 5 Breakdown · p. 7

Related papers

Loading 2609.02902v1…