PaperlayerView exact revision on arXiv

Source-linked AI summary

ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interaction

Yongqi Tong, Tan Li Hui Faith, Choy Zhen Wen Marcus, Zhou Jin, Kewei Fu, Jiang-Ming Yang, Jianshe Li, Xin Zhang

arXiv:2608.13622v1cs.AIcs.CL

TL;DR

Open-ended interaction lets agents use multiple valid strategies, but standard group-based RL can compare heterogeneous rollouts unfairly and bias learning toward reward-preferred styles. ARC conditions rollout comparisons on strategy families, and training with INTER3-86K improves open-ended agentic RL rewards while INTER3 reduces TTFT from 4.91s to 1.27s versus a think-then-act baseline.

  • Problem

    Standard group-based RL can confound advantage estimates when heterogeneous interaction strategies are normalized together, making reward-model bias influence policy updates beyond task appropriateness.

  • Method

    ARC conditions rollout comparison groups on a single strategy family, while INTER3 separates user-visible communication from latent reasoning and tool execution.

  • Results

    Training on INTER3-86K yields higher and more stable rewards for open-ended agentic RL, though improvements are not uniform across backbones or downstream metrics.

  • Takeaways & Limitations

    ARC is a targeted correction for comparison bias in multi-strategy interaction rather than a universally dominant optimizer.

  • Takeaways & Limitations

    ARC depends on normalized interaction traces and strategy labels that may reflect annotation bias or deployment-specific patterns.

Abstract

from arXiv · show

Open-ended real-world interaction admits multiple valid behaviors: an agent may answer directly, ask for clarification, provide progress updates, or confirm before acting. This flexibility breaks a core assumption behind group-based RL: rollouts compared within a group are no longer guaranteed to be behaviorally comparable. As a result, reward-model preferences over interaction style can distort relative advantages and steer optimization toward reward-preferred behaviors rather than context-appropriate ones. We formalize this as a \textit{reward fairness problem} and propose \textbf{ARC} (Advantage Regularization via Conditioning), a training recipe that restores fairer relative comparison through strategy-conditioned rollout grouping, together with hybrid rewards and entropy regularization. We study ARC in our proposed \inter, a novel paradigm for responsive, steerable, and execution-aware user-agent interaction that decouples user-visible communication from latent reasoning and tool use. \inter\ also provides the annotation and distillation pipeline for constructing \inter-86K, our strategy-annotated training corpus for supervised and RL training. Empirically, ARC substantially strengthens the core $τ/τ^2$ tool-use benchmarks, while \inter\ reduces time-to-first-token from 4.91s to 1.27s relative to a think-style baseline. Together, these results suggest that a central bottleneck in open-ended interactive learning is not only how agents are rewarded, but whether their behaviors are compared fairly in the first place. The ARC implementation and \inter-86K training data will be released.

1 Introduction

Open-ended interaction permits multiple context-dependent behaviors, making standard group-based RL vulnerable to unfair relative comparisons across strategies. The paper introduces ARC for strategy-conditioned advantage estimation and INTER3 for collecting realistic, channel-separated interaction data.

  • Motivation: Standard group-based RL is most interpretable when compared rollouts occupy a locally comparable response region, but cross-strategy variation can distort centered rewards.The introduction identifies qualitatively different behavioral modes as a source of unfair relative comparison.
  • Motivation: Open-ended dialogue states can support several appropriate actions, including direct answers, clarification, progress updates, or confirmation before irreversible actions.The best action depends on missing information, action reversibility, and the evolving user-agent state.
  • ARC: ARC assigns training-time strategy instructions, compares rollouts only within strategy-conditioned groups, and removes the instruction at inference so the policy selects strategies autonomously.Its objective combines a hybrid reward with entropy regularization, while strategy conditioning removes inter-strategy variance from the ideal centered-advantage decomposition.
  • INTER3: INTER3 separates user-visible communication from latent reasoning and tool execution while supporting explicit strategy control and user interruption.The framework is designed to expose diverse valid interaction behaviors rather than only trajectories with verifiable task outcomes.
  • Combined contribution: Together, INTER3 supplies realistic open-ended interaction data, while ARC provides fairer comparison of diverse valid behaviors during optimization.The two components address data collection and relative-comparison problems jointly.

2 Related Work

Prior work advances tool-augmented reasoning, interleaved responsiveness, policy-gradient/RLHF optimization, group-relative methods, and guidance-augmented RL. INTER3 instead separates visible interaction from latent reasoning and tool execution, while ARC uses auxiliary instructions to define cleaner comparison classes for group-relative optimization.

  • Tool-augmented and interleaved agents: Tool-augmented agents advance reasoning, planning, and external action, while interleaving methods improve responsiveness by alternating thought with partial output.INTER3’s architectural departure separates visible interaction from latent reasoning and tool execution.
  • RL and group-relative optimization: Policy-gradient, RLHF, and group-relative methods such as GRPO are effective when samples within a group are meaningfully comparable.This work instead focuses on open-ended interaction, where valid rollouts may differ in strategy.
  • Guidance-augmented RL: Guidance-augmented RL methods introduce auxiliary signals to improve exploration, mitigate sparse rewards, or reuse successful experience.The cited neighboring approaches include SAGE, Scaf-GRPO, LUFFY, and ExGRPO.
  • ARC’s methodological distinction: ARC also uses an auxiliary instruction, but to define a cleaner comparison class for group-relative optimization rather than to make the task easier.Additional discussion of adjacent tool-use, interaction, and guidance literatures is deferred to Appendix B.

3 The INTER3 Setting

INTER3 separates user-visible communication from hidden reasoning and structured tool use, enabling streaming interaction before execution finishes. Its unified, strategy-conditioned runtime and INTER3-86K corpus support measurable comparison of diverse interaction behaviors while reducing perceived latency.

  • Channel-separated interaction: INTER3 separates interaction from execution: <answer> spans stream to users, while reasoning and tool calls remain internal until results return to context.This lets user-visible communication begin before the hidden reasoning-and-tool trajectory terminates.
  • Channel-separated interaction: Streaming <answer> spans substantially reduce perceived latency while preserving continuous interaction during execution.The latency benefit follows from emitting answers before hidden execution completes.
  • Runtime and training substrate: A unified async runtime interleaves hidden reasoning, user-visible answers, tool calls, interruptions, and resumed continuations as first-class events.Partial outputs, interruptions, tool results, and continuations are normalized into one trace format for training and post-deployment data conversion.
  • Strategy conditioning: INTER3 organizes behaviors into Progress Update, Clarify First, Alignment Check, and Direct Answer families used for training-time comparison, not inference-time requirements.The strategy definitions align prompting, annotation, and rollout grouping with ARC conditioning families.
  • Strategy conditioning: 86.8K examples comprise INTER3-86K, spanning tool use, multi-hop QA, and logical reasoning, with 57.9K SFT and 28.9K RL examples.The corpus is strategy-annotated and includes strategy instructions for its RL portion.

4 INTER3-86K Construction

INTER3-86K is built from real-world deployment traces and curated, teacher-augmented public data, then annotated with interaction strategies for ARC-conditioned training. Its construction emphasizes context-adapted behavioral diversity through a verified four-strategy taxonomy and separate SFT and RL data pipelines.

  • Strategy taxonomy: INTER3 defines four high-level interaction families and nine concrete strategies, with ARC conditioning rollout groups on the high-level family.The finer-grained strategies support analysis and interface documentation.
  • Data sources: INTER3-86K combines de-identified deployment traces from a global payment platform with curated public benchmarks, teacher-driven augmentation, distillation, and synthetic trajectories.Deployment traces include interruption, redirection, clarification, progress updates, and multi-step tool use.
  • Dataset composition: The SFT split contains 57.9K examples spanning tool use, multi-hop QA, and logical reasoning, while the RL set contains approximately 28.9K examples.The RL set reuses seven ToolMind sub-datasets and adds strategy instructions during training.
  • Dataset composition: 75.3% of the RL training data uses Progress Update, while Tool Use comprises 72.8% of the full training set.Multi-hop QA and Logical Reasoning account for 19.8% and 7.1%, respectively, providing complementary reasoning diversity.
  • Strategy annotation: Each example receives one of four strategy labels—Progress Update, Direct Answer, Clarify First, or Alignment Check—through model-based labeling and collaborative verification.Two language models annotate independently; agreement, confidence-based resolution, and human review handle disagreements.

5 Our Methodology: Advantage Regularization via Conditioning

ARC addresses unfair relative comparison in open-ended RL by grouping rollouts according to interaction strategy, so advantages are computed within comparable behavioral subspaces. Its objective combines strategy-conditioned policy gradients with entropy regularization, while theoretical analyses explain variance reduction and identify train-inference mismatch as a limitation.

  • Core mechanism: ARC computes relative advantages within strategy-conditioned rollout groups rather than across heterogeneous communication behaviors.This changes the comparison unit in group-based RL to a strategy-conditioned subspace.
  • Reward fairness problem: Reward-model bias can contaminate advantages when equally high-quality responses use different strategies, skewing updates toward reward-preferred rather than task-appropriate behaviors.Exposure, length, and style biases make this problem especially pronounced when interaction appropriateness lacks clear ground truth.
  • Training procedure: ARC assigns each example a strategy instruction, samples M rollouts conditioned on it, computes within-strategy advantages, and applies an entropy-regularized policy update.At inference, the strategy instruction is removed and the model autonomously selects an appropriate strategy.
  • Variance analysis: Under the idealized variance analysis, strategy-conditioned sampling removes the between-strategy mean-shift term from the centered-advantage variance decomposition.The result is an estimator-level variance-reduction claim, not an end-to-end convergence guarantee.
  • Limitation: Training-time strategy instructions create a train-inference mismatch because they are removed at deployment.Progressive instruction removal is studied separately as further analysis rather than as part of ARC itself.
  • Theoretical scope: Theoretical sample-efficiency results are intended as idealized estimator-level scaling interpretations rather than calibrated empirical speedup or full convergence guarantees.The analysis motivates reduced gradient-sampling burden from cleaner within-strategy comparisons.

6 Experiments

Across RL backbones, ARC most consistently improves in-domain multi-turn tool use, with additional gains in selected out-of-domain reasoning settings. Ablations show benefits across model scales and strategy configurations, while persistent training-time conditioning is preferable to instruction removal or deployment-time hints.

  • In-domain performance: ARC consistently improves in-domain tool use across RL backbones, with the largest gain for GRPO at +5.37 average points.For GRPO, τ-airline rises from 31.33 to 44.00, τ-retail from 40.29 to 50.00, and τ2-airline from 36.67 to 48.00.
  • Out-of-domain reasoning and tradeoffs: GRPO + ARC improves AIME 2026 reasoning from 31.67 to 40.83, while PPO and DAPO show mixed out-of-domain effects.ARC also provides modest gains on instruction-following and alignment metrics.
  • Training dynamics and robustness: ARC maintains or improves rewards across the training budget, whereas GRPO peaks mid-training and then declines, especially for tool-call and answer rewards.Format reward saturates for both methods, so the divergence reflects execution and answer quality rather than syntactic compliance.
  • Scale robustness: At 4B scale, ARC outperforms no-think, think, and the corresponding 4B GRPO baseline, increasing the five-task average by about 53% over 4B no-think.This supports improved training comparison signals rather than reliance on larger model capacity.
  • Curriculum learning: Default ARC achieves the strongest τ/τ2 tool-use performance and lowest off-strategy mean and variance; linear removal is intermediate, while constant removal is weakest.Linear removal modestly improves some reasoning metrics but reduces agentic performance, whereas constant removal performs worst overall.
  • Strategy scaling and deployment hints: 36.1% total improvement emerges from incremental strategy scaling, including 99% and 71% gains on τ-bench and τ2-bench, respectively.Alignment Check contributes meaningfully despite comprising only 2.5% of the data; inference-time hints instead yield averages of 41.73 without removal, 39.58 randomized, and 39.85 matched.

7 Conclusion · A Proofs and Theoretical Analysis · A.1 Proof of Theorem 5.1 (Variance Amplification)

ARC addresses reward unfairness in open-ended interaction by grouping rollouts by strategy, improving relative-advantage interpretability without changing the underlying optimizer. The paper’s analysis supports this as a targeted, estimator-level correction, while noting limitations in strategy abstraction, domain scope, annotation dependence, and theoretical coverage.

  • 7 Conclusion: Heterogeneous strategies in shared relative-reward pools confound advantage estimates, so comparisons may reflect reward-model bias rather than intrinsic response quality.This is the central reward fairness problem motivating ARC.
  • 7 Conclusion: ARC conditions rollout comparison classes so each group contains one strategy family, improving relative-advantage interpretability without altering policy optimization.INTER3 exposes the issue by separating user-facing communication from latent reasoning and tool execution.
  • 7 Conclusion: ARC is most effective on in-domain agentic benchmarks with prevalent multi-strategy interaction.Training on INTER3-86K produces higher and more stable rewards, including less post-peak degradation under fixed compute budgets.
  • 7 Conclusion: ARC’s improvements are not uniform across backbones or downstream metrics, so it is a targeted correction for comparison bias rather than a universally dominant optimizer.The conclusion frames ARC as especially useful where multi-strategy interaction creates comparison bias.
  • 7 Conclusion: The theoretical results explain how inter-strategy reward variance can reduce sample efficiency and how restricting comparisons can reduce variance under strategy-dependent reward bias.These are stylized estimator-level characterizations, not end-to-end convergence guarantees.
  • Limitations: The study’s limitations include coarse strategy taxonomy, uncertain generalization beyond open-ended tool use, dependence on normalized traces and labels, and incomplete convergence theory.Labels and traces may encode annotation bias or deployment-specific patterns.
  • A.1 Proof of Theorem 5.1 (Variance Amplification): The appendix formalizes the variance claims by distinguishing target strategy s★ assigned before rollout generation from realized strategy s_i=f(y_i), which may differ under imperfect adherence.The proof decomposes reward into a realized-strategy mean capturing reward-model bias and i.i.d. zero-mean noise, then derives the centered advantage from both components.

A.2 Proof of Theorem 5.2 (Variance Reduction via Conditioning) … C.1 Final Reward Used in Main Experiments

The appendices show that strategy conditioning removes between-strategy variation under perfect compliance, while imperfect compliance attenuates the variance reduction and the sample-efficiency argument remains idealized. Related-work discussion positions INTER3 around user-visible communication during long-running actions, and the main experiments use a gated, execution-grounded reward.

  • A.2 Proof of Theorem 5.2 (Variance Reduction via Conditioning): Under perfect compliance, conditioning on the target strategy equalizes strategy-specific means and removes the between-strategy mean-shift term from centered-advantage variance.With imperfect compliance, a residual between-strategy term remains, attenuating the predicted variance reduction.
  • A.3 Derivation for Theorem 5.3 (Idealized Sample-Complexity Comparison): The gradient-variance comparison assumes factorized conditional mixed moments, comparable score-function second moments, and ARC’s perfect-compliance idealization.The derivation frames the result as an analytic comparison between strategy-conditioned and standard sampling with equal group size N.
  • A.3 Derivation for Theorem 5.3 (Idealized Sample-Complexity Comparison): The sample-efficiency derivation is a stylized variance-to-sample-efficiency translation under a sub-Gaussian gradient-estimation assumption, not a full convergence proof.The stated scaling concerns achieving gradient error at most ε with probability at least 1 − δ.
  • A.4 Effective Variance Analysis for Curriculum Learning: The curriculum mixes conditioned and unconditioned groups, producing a monotonic variance increase that exposes the model to progressively noisier optimization landscapes as capability grows.This describes the expected batch advantage variance across training step t.
  • A.5 Information-Theoretic Perspective: In the idealized perfect-compliance setting, conditioning removes between-strategy variation from within-group comparisons and emphasizes the within-strategy reward signal I(r; y | x, s).The strategy-linked dependence term I(r; s | x) is treated as orthogonal to within-strategy quality assessment.
  • B Extended Related Work: Unlike work focused on adding tools, planning, or thought-answer interleaving, INTER3 exposes interaction as a first-class user-visible channel while execution continues.The related-work discussion distinguishes this design from methods mainly addressing the sequencing of thought and answer tokens.
  • B Extended Related Work: INTER3 targets latency from long-running external actions by making communication independent from execution, addressing the cost of opaque waiting.The design connects to broader work on transparency, feedback, and controllability in human-AI interaction.
  • C.1 Final Reward Used in Main Experiments: The full INTER3 reward combines format validity, exact tool-call correctness, and an answer score, while treating this construction as a setting-specific stabilization choice rather than ARC’s core mechanism.The components are Rfmt ∈ {0, 1}, Rtool ∈ {−1, 0, 1}, and Rans ∈ {0, 0.50, 1}.

C.2 Detailed Reward Ablation

The ablation examines reward construction and judge selection for multi-channel INTER3 outputs, emphasizing execution-grounded supervision without strategy-dependent bias. Exact, gated tool-execution rewards and the 235B judge provide the strongest reported execution-focused signals, while ARC benefits most from strategy-conditioned grouping with tuned entropy regularization.

  • Reward design: Reward design targets internal reasoning, tool calls, and user-facing answers while avoiding strategy-dependent bias.The study varies both reward construction and the model judging semantic answer quality.
  • Reward construction: About 42% higher 𝜏Avg. results from the final gated reward design versus the weakest variant.The final design combines stricter execution matching with tighter coupling between execution and semantic reward, but broader reasoning-style evaluations decline.
  • Reward judge: The 235B judge gives the strongest 𝜏/𝜏2 performance and best overall balance under the controlled Qwen3-8B comparison.This consistency led to selecting the 235B judge for the main experiments and tool-execution verification.
  • ARC training settings: Strategy-conditioned grouping provides ARC’s largest single gain over base GRPO, while entropy value 0.001 yields the highest average score.Entropy regularization alone slightly decreases performance, but combining it with strategy-conditioned grouping improves the overall operating point.
  • Reward construction: Negative penalties discourage incorrect tool execution and sharpen optimization toward valid trajectories.The final reward variant explicitly penalizes spurious or malformed calls.
  • Reward construction: Gated semantic reward is issued only when exact tool execution is correct, coupling answer quality with execution validity.Earlier variants apply semantic reward unconditionally, allowing fluent but unsupported answers to receive credit.

D Entropy Analysis · E Annotation Guidelines for INTER3-86K

Entropy analysis identifies strategy-conditioned grouping as ARC’s primary improvement mechanism, while entropy regularization serves as a stabilizer only after grouping. The annotation guidelines assign interaction strategies for progress updates, direct answers, clarification, and alignment checks.

  • D Entropy Analysis: Table 9 fixes the reward family and varies strategy-conditioned grouping and entropy regularization to separate ARC’s core mechanism from setting-specific stabilizers.The first block examines improvement over GRPO; the second examines entropy sensitivity after grouping is enabled.
  • D Entropy Analysis: Strategy-conditioned grouping is the primary source of improvement over base GRPO when reward design is held fixed.This supports ARC’s main causal story.
  • D Entropy Analysis: Entropy regularization alone does not explain the gain and can weaken performance.Entropy becomes useful after grouping cleans up the comparison class, stabilizing multi-channel generation rather than driving the main improvement.
  • E Annotation Guidelines for INTER3-86K: The annotation guidelines provide detailed strategy-assignment instructions for both models and human annotators.These guidelines define how interaction strategies are assigned in INTER3-86K.
  • E Annotation Guidelines for INTER3-86K: Progress Update means proactively reporting current progress during simultaneous tool calls or multi-step tasks, including after each completed sub-task.The purpose is to reduce waiting anxiety.
  • E Annotation Guidelines for INTER3-86K: Direct Answer means responding concisely without tools when information is already known or contextually available.The guideline emphasizes clarity and avoiding verbosity.
  • E Annotation Guidelines for INTER3-86K: Clarify First means asking directly for missing information rather than guessing, and executing tools only after user confirmation.This strategy suits vague, ambiguous, or incomplete requests.
  • E Annotation Guidelines for INTER3-86K: Alignment Check means restating requirements and executing only after confirmation, especially for misunderstood, important, or irreversible operations.The strategy reduces mistakes caused by misunderstanding.

F Mechanism-Level Analysis and Label-Noise Robustness … I.3 TTFT Measurement Protocol

ARC’s mechanism analyses show reduced realized reward variance and retained—but weakened—performance under noisy strategy labels. The appendix specifies interaction-strategy examples, structured interleaved inference, benchmark protocols, and TTFT measurement conditions.

  • F Mechanism-Level Analysis and Label-Noise Robustness: ARC lowers the realized inter-/intra-strategy reward variance ratio from 0.417 under GRPO to 0.074, an approximately 82% reduction.Confidence intervals are wide because few prompt groups contain multiple realized strategies.
  • F Mechanism-Level Analysis and Label-Noise Robustness: Under 50% strategy-label corruption, ARC achieves a tool-use average of 35.01 versus 33.35 for GRPO, below clean ARC at 41.73.Corrupted instructions are replaced with uniformly sampled alternative strategies; results use one training seed and three evaluation runs.
  • G Interaction Strategy Examples: The interaction examples illustrate progress updates, direct answers, clarification before action, and alignment confirmation as distinct response strategies.These examples span tool use, user-visible status messages, missing-information requests, and confirmation before sending data.
  • G.1 Progress Update Strategy: Progress-update behavior interleaves user-visible status messages with tool calls, while direct-answer behavior presents the requested answer without the demonstrated multi-step workflow.The examples include fetching Acme Corp financial statements and answering China’s capital directly.
  • G.3 Clarify First Strategy: Clarify-first and alignment-check strategies gather missing details or confirm scope before executing booking and sending operations.The examples request date, time, capacity, recipient, and exact scope before tool execution.
  • H System Prompt: The system prompt separates hidden internal reasoning from user-visible <answer> content and internal <tool_call> segments, enabling streaming updates during reasoning and execution.It requires paired XML-style tags and instructs the model to summarize tool outputs rather than expose them verbatim.
  • I.1 Inference Configuration: Evaluations use controlled inference settings, including concurrency=16 parallel requests on the same H200 GPU and the structured interface used during training.The appendix standardizes configurations across reported benchmarks and supports fine-grained TTFT measurement through partial answers.

I.4 Evaluation Uncertainty and Training-Seed Robustness … J.2 Compute Resources

ARC’s tool-use gains are evaluated under both repeated-checkpoint evaluation and independent training seeds, with robust improvements for PPO and GRPO but little change for DAPO. The experiments also report RL hyperparameters and compute details for reproducibility.

  • I.4 Evaluation Uncertainty and Training-Seed Robustness: ARC’s evaluation analysis separates stochastic variability from optimization randomness during training.Repeated evaluations use the same trained checkpoint, while training-seed analysis compares independently trained models.
  • I.4 Evaluation Uncertainty and Training-Seed Robustness: Evaluation uncertainty is measured as mean ± standard deviation over N=3 independent runs of each trained checkpoint.The reported Tool-use Avg. averages the five displayed τ-bench and τ2-bench metrics.
  • I.4 Evaluation Uncertainty and Training-Seed Robustness: Training-seed robustness is measured over three independently trained models with different seeds.Repeated evaluations are first averaged within each seed, then the mean and standard deviation are reported across the three seed-level scores.
  • I.4 Evaluation Uncertainty and Training-Seed Robustness: 33.60→38.45 and 33.35→41.73: ARC raises Tool-use Avg. for PPO and GRPO, respectively.DAPO results are substantially closer, indicating that ARC’s effect depends on the underlying RL backbone.
  • I.4 Evaluation Uncertainty and Training-Seed Robustness: 31.81→36.76 and 34.74→37.53: ARC consistently improves PPO and GRPO Tool-use Avg. across training seeds.DAPO remains essentially unchanged at 34.69 versus 34.77, which is not interpreted as meaningful improvement.
  • J.2 Compute Resources: All experiments ran on NVIDIA H200 GPUs with NVIDIA driver 570.158.01 and CUDA 13.0.These compute-resource details specify the hardware and software environment used for the experiments.
  • J Training Details: Table 14 details the RL training hyperparameters used in the main experiments.These settings are provided as part of the paper’s training details.
Loading 2608.13622v1…