PaperlayerView exact revision on arXiv

Source-linked AI summary

RL-ADA: A World-Feedback Framework for Adversarially Robust Enterprise Dialogue Agents

Ram Narayanan, Harshit Rajgarhia, Abhishek Mukherji

arXiv:2609.02902v1cs.CL

TL;DR

Enterprise dialogue agents need robust training data, but privacy-sensitive, expensive-to-annotate logs create a bottleneck as user behaviour evolves. RL-ADA replaces labels with world feedback in asymmetric co-evolution, eliminating routing errors and doubling strict PASS in a banking proof of concept, while remaining preliminary in scope.

  • Problem

    Enterprise dialogue robustness is limited by the privacy sensitivity and annotation cost of interaction logs, while user behaviour evolves faster than labelling pipelines.

  • Method

    RL-ADA co-evolves a 3B Customer Support Agent and 7B Adversarial Customer Agent using opposing automated rewards, a fixed judge, and isolation-gym retraining on failure transcripts without human annotation.

  • Results

    All routing errors were eliminated on a 12-scenario held-out benchmark, while strict PASS doubled from 25% to 50% over five co-evolutionary cycles.

  • Takeaways & Limitations

    Automated arena reward can support adversarial robustness training for asymmetric enterprise dialogue agents in the evaluated banking setting.

  • Takeaways & Limitations

    Results are preliminary: experiments cover a single banking domain with few matches and held-out scenarios, so cross-domain generalisation is not established.

Abstract

from arXiv · show

Deploying task-oriented dialogue agents in enterprise customer support faces a persistent annotation bottleneck: robust training requires labelled interaction data at scale, yet enterprise conversational logs are privacy-sensitive and expensive to annotate, while user behaviour evolves faster than labelling pipelines can keep pace. We present RL-ADA (Reinforcement Learning with Adversarial Dialogue Agents), a co-evolutionary training framework that eliminates this bottleneck by replacing human labels with \emph{world feedback}: consequence-based reward signals derived directly from measurable interaction outcomes. A Customer Support Agent (DA, 3B parameters) and an Adversarial Customer Agent (CA, 7B parameters) co-evolve in an adversarial arena guided by a fixed automated judge: the DA is rewarded for correctly handling multi-turn customer conversations to successful resolution, while the CA is rewarded for producing realistic, intent-concealing utterances that cause misroutes, creating asymmetric adversarial pressure through opposing but independently structured rewards. An isolation gym iteratively retrains the weaker agent on prior-failure transcripts, requiring no human annotation at any stage. In a banking customer support proof of concept, tool-routing errors are eliminated and the strict end-to-end PASS rate doubles over five co-evolutionary cycles, driven solely by automated arena reward with no labelled data. We additionally observe the emergence of \textbf{Contextual Camouflage}, an adversarial strategy in which the CA learns to embed intent within dense realistic customer detail purely from reward pressure, with direct implications for enterprise red-teaming and robustness evaluation.

1 Introduction

RL-ADA addresses the annotation bottleneck in robust enterprise dialogue training by using asymmetric, annotation-free co-evolution driven by measurable interaction outcomes. Its banking instantiation targets adversarial tool-routing failures across realistic customer interactions.

  • Enterprise dialogue robustness requires large labelled datasets, but privacy-sensitive logs and expensive annotation create a persistent bottleneck.
  • Existing RLHF and self-play methods do not directly support asymmetric customer-support agents with different action spaces, capacities, and reward structures.
  • RL-ADA replaces human labels with measurable world-feedback rewards while co-evolving a 3B Customer Support Agent and 7B Adversarial Customer Agent.
  • The co-evolutionary loop uses an arena and isolation gym, retraining the weaker agent on a 70:30 failure/success transcript mix.
  • The banking instantiation routes utterances across six API tools and 78 intents, with customers concealing intent rather than naming it directly.

2 Related Work

RL-ADA builds on automated feedback, self-play, adversarial testing, and task-oriented dialogue evaluation, but combines them in an asymmetric co-evolutionary framework for tool-routing robustness.

  • Feedback and reinforcement learning: RL-ADA replaces human or AI preference feedback with structured episode scores derived from interaction outcomes.This removes annotation dependence but leaves judge reliability as an explicit concern.
  • RL-ADA framework: The three-phase loop bootstraps both agents, measures relative arena win rates, and retrains the weaker agent on failure transcripts.The loop structure is summarized in Figure 1.
  • Self-play and game-theoretic reinforcement learning: Its isolation gym resembles a single-oracle Double Oracle inner loop, using curated failure/success curricula instead of uniform population replay.The dialogue setting is asymmetric, and simultaneous co-training can induce cycling dynamics.
  • LLM red-teaming: Unlike automated adversarial testing against fixed targets, RL-ADA co-evolves its adversarial customer with the dialogue agent.It targets tool-routing correctness with asymmetric model sizes of 3B and 7B under an explicit convergence criterion.
  • Task-oriented dialogue evaluation: Unlike cooperative-user benchmarks, RL-ADA trains the customer agent to cause failures rather than simulate realistic task completion.This shifts evaluation from cooperative task success toward adversarial robustness.

3 System Architecture

RL-ADA uses a dialogue agent, an adversarial customer, and a fixed judge in a three-phase training architecture. Arena matches determine relative competency, while isolation retraining targets recent failures until empirical convergence criteria are met or training limits are reached.

  • Roles and training phases: RL-ADA comprises a 3B Customer Support Agent, a 7B Adversarial Customer Agent, and a 7B fixed Judge.The DA handles clarification, tool routing, and call termination; the CA generates misroute-inducing customer utterances; the Judge scores terminal episode quality.
  • Roles and training phases: The fixed NeutralJudge is a local 7B scorer intended to replace proprietary judging while providing the sole automated training signal.Its required reliability dimensions are resolution detection, quality ranking, and hallucination detection.
  • Initialization and optimization: The DA begins with Banking77-derived tool-routing demonstrations and GRPO refinement, whereas the CA learns entirely from reward pressure without SFT initialization.The CA is trained against a fixed DA checkpoint during initialization.
  • Environment and agent decisions: The DA acts under partial observability, choosing tools, speech, or termination from dialogue history without seeing the CA’s hidden intent.Its structured JSON output includes explicit reasoning before each action.
  • Reward design: The environment reward combines turn-level rules with a terminal judge score clipped to [−2, +2].Mandatory deductions of −1.2 for missed identity verification and −1.8 for a missing domain tool ensure procedural failures remain negative.
  • Reward design: The CA reward combines format, realism, concealment, and inverse-DA-routing terms, with weights selected from early stability observations and tunable by domain.The inverse-routing signal rewards misroutes and penalizes correct DA routing.
  • Arena evaluation: Each arena match evaluates 180 episodes across 10 scenarios, computes DA win rate over CA-winnable scenarios, and separately tracks fraud-escalation scenarios.The current DA and CA checkpoints oppose one another in each match.
  • Isolation gym: The weaker agent is retrained against a 70:30 failure/success transcript mix while the opponent remains frozen.Sliding-window transcript slicing creates GRPO training examples at each agent decision point and preserves successes against catastrophic forgetting.

4 Experiments

The experiment uses a defined checkpoint progression and evaluates agents with repeated arena and held-out protocols.

  • Arena matches use 18 episodes per scenario across 10 scenarios, averaged over 2 runs, while held-out evaluation uses 12 fixed scenarios.
  • DA0 is the SFT-and-GRPO-warm-up baseline, while CA0 is trained against fixed DA0 without SFT initialization.
  • Each checkpoint subscript increment represents one completed isolation gym cycle, producing DA1, DA2, CA1, and CA2.

5 Results

Across five arena matches, DA performance fluctuates after CA gym cycles and then recovers, while held-out evaluation shows eliminated routing errors and a doubled strict PASS rate. The CA also develops Contextual Camouflage, becoming more specifically misleading rather than merely vague.

  • 5.1 Arena Progression: At matches 3 and 5, DA win rate drops after CA gym cycles and recovers after DA retraining, consistent with co-evolutionary pressure.The interpretation is limited by five matches and ±8pp per-match variance.
  • 5.1 Arena Progression: The stopping criterion fires at match 5 with |δ|=0.02 ≤ε and P=2, leaving three of eight maximum cycles unused.
  • 5.2 DA Improvement on Held-out Evaluation: 100% correct tool routing after five cycles eliminates routing errors, up from 75% in DA0; strict PASS rises from 25% to 50%.PASS requires correct routing, lookup_account first, r_DA ≥2.0, and a clean ending.
  • 5.2 DA Improvement on Held-out Evaluation: The aggregate FAIL rate remains 33%, but DA2 failures shift from routing errors to procedural and conversation-quality criteria.
  • 5.3 Emergent Behaviour: Contextual Camouflage: Contextual Camouflage embeds intent in dense customer detail, while Direct Intent increases from 25 to 34 to 37 and Vague/Indirect decreases from 25 to 16 to 13.The behavior emerges from reward pressure maximizing DA misroutes; the authors treat it as a qualitative observation.

6 Limitations •

The study’s conclusions are bounded by a single banking domain, small evaluation scale, pragmatically chosen settings, and arena-only demonstration of the data flywheel.

  • Evaluation covers one banking domain with few matches and held-out scenarios, so cross-domain generalization and finer-grained statistical separation remain unestablished.
  • Reward components, thresholds, transcript mixing, and gym hyperparameters were chosen pragmatically; systematic sensitivity sweeps and reward ablations remain future work.
  • The data flywheel is demonstrated within arena training rather than live deployment, which requires deployment-time outcome signals to replace the judge’s terminal score.

7 Production Deployment Considerations

Production use requires deployment gates and outcome signals beyond the training loop. RL-ADA’s components are intended to be configurable across domains, but production transfer and end-to-end feedback remain future work.

  • Training-to-Deployment Flywheel: The Isolation Gym can feed failure transcripts into retraining through a 70:30 failure/success mix without human annotation.Closing the production loop requires deployment-time signals such as resolution markers, callback rates, or escalations to replace the judge’s terminal score.
  • Deployment Gating: Production deployment requires a stronger gate for compliance-critical scenarios, independent of the macro training stopping criterion.Examples include fraud detection, account compromise, and identity verification scenarios requiring higher thresholds before promotion.
  • Domain Configuration: RL-ADA’s arena, isolation gym, and world-feedback rewards require measurable interaction outcomes, while intent maps and gates are configured per domain.The banking taxonomy can be replaced with a domain-specific intent-to-tool map, and the judge must be re-validated against an appropriate reference.

8 Conclusion

RL-ADA trains asymmetric support and adversarial customer agents using world feedback and reports improved routing and PASS performance in a banking proof of concept. Its conclusions remain preliminary, with domain transfer requiring empirical validation.

  • Conclusion: RL-ADA eliminates routing errors and doubles strict PASS from 25% to 50% over five co-evolutionary cycles without labelled data.Average episode reward rises from +1.58 to +2.16, and convergence is identified at cycle 5 by macro-level win-rate stability.
  • Conclusion: The trained CA develops Contextual Camouflage, embedding intent in dense, specific customer detail through reward pressure alone.The arena win-rate trajectory is non-monotonic, with DA performance dropping after CA isolation-gym cycles.
  • Conclusion: These results are preliminary because experiments cover a single banking domain with limited matches and held-out scenarios.Empirical cross-domain transfer remains the immediate next step.
  • Intent-to-Tool Taxonomy: The banking taxonomy maps 78 intent strings many-to-one onto six routing tools, with transfer_to_human containing 38 intents.Core intents appear in all arena scenarios, while Banking77 extensions provide vocabulary diversity during warm-up.
  • Intent-to-Tool Taxonomy: The transfer_to_human skew creates class imbalance: escalation over-generalisation can score well on coverage while failing on five specific tools.The arena’s CA-winnable scenarios are designed to stress-test those non-escalation buckets.

B.1 DA Training: Baseline vs. Isolation Gym

The appendix contrasts baseline and isolation-gym training configurations and illustrates a CA-generated camouflage case that still induces a DA routing error. The CA hides dispute intent behind self-doubt phrasing, causing escalation instead of dispute filing.

  • DA Training: DA0 predates the standardised isolation-gym protocol and used a higher learning rate and more training steps than DA1 and DA2.DA1 and DA2 use the standardised isolation-gym parameters listed in Table 7.
  • CA Training: CA1 and CA2 use a cosine learning rate, fewer generations per step, and β=0.05 to keep outputs within natural-language register.These are the configuration choices listed for all CA isolation-gym cycles in Table 8.
  • Camouflage Example: In the dispute-duplicate-v2 scenario, the expected tool is dispute_charge, but DA2 routes to transfer_to_human after lookup_account.The CA’s utterance expresses uncertainty about duplicate drinks or payment confusion rather than naming the dispute intent directly.
  • Camouflage Example: The CA embeds dispute intent behind self-doubt phrasing, while the DA escalates instead of filing a dispute.This example demonstrates how contextual customer wording can trigger a misroute.

C.2 DA Improvement — Before and After Training

A concrete dispute example shows DA2 correcting a routing failure after training. It follows the required call sequence, files the dispute, explains the outcome, and ends the call without new labelled examples.

  • Before Training: The utterance reports an unrecognised AMZN MKTP charge for $89, while DA0 incorrectly calls transfer_to_human.This provides the before-training baseline for the dispute-routing example.
  • After Training: DA2 correctly executes lookup_account followed by dispute_charge, explains that the charge was disputed, and ends the call.The sequence satisfies the demonstrated tool-routing and conversation-ending requirements.
  • After Training: DA2 identifies the dispute intent and closes the call without any new labelled examples.The improvement is attributed to the post-training agent behavior shown in this example.
  • Agent Protocol: The support agent must choose among tool calls, speech, and ending the call while following a fixed sequence beginning with lookup_account.Its output format requires valid JSON with an action type, content, and tool arguments.
  • Agent Protocol: The customer agent is instructed to express problems naturally and indirectly in one or two sentences while waiting for the support agent to help.This interaction style creates the conversational conditions for indirect intent expression.

E Per-Scenario Match 5 Breakdown

Match 5 shows DA2 winning more often than CA2 overall, while adversarial pressure is concentrated in ambiguous-intent scenarios and DA2 remains highly accurate on clear-intent cases.

  • 62% DA2 win rate versus 38% CA2 win rate across CA-winnable Match 5 scenarios.The comparison covers 18 episodes per scenario.
  • CA pressure concentrates on ambiguous-intent scenarios including dispute_duplicate, unknown_merchant, and internal_transfer.
  • DA2 retains near-perfect accuracy on clear-intent scenarios.
Loading 2609.02902v1…