PaperlayerView exact revision on arXiv

Source-linked AI summary

ToSCA: Leveraging Hierarchical Reinforcement Learning on Temporal and Strategic Abstractions of Conversational Agents

Xiaoyu Wang, Qingqing Gu, Yue Zhao, Teng Chen, Yuqi Cao, Xiaokai Chen, Hongyan Li, Luo Ji

arXiv:2608.21969v1cs.CLcs.HCcs.LG

TL;DR

Conversational LLMs face supervision, preference-alignment, and topic-distribution challenges that motivate reinforcement learning approaches. ToSCA addresses the gap between utterance-level strategic planning and token-level response generation with hierarchical reinforcement learning, and experiments show it surpasses prompting, supervised, and RL baselines across in-domain and out-of-domain conversations.

  • Problem

    Conversational LLMs face bottlenecks in data supervision, nuanced human-preference alignment, and broad topic distributions that challenge traditional fine-tuning methods.

  • Method

    ToSCA uses a two-level MDP in which a DQN-trained high-level critic selects explicit strategies and a PPO-trained low-level actor-critic generates conditioned response tokens, with KL and intrinsic-motivation rewards.

  • Results

    ToSCA surpasses prompting, supervised, and RL baselines on daily-life and emotional-support conversations, with an out-of-domain test demonstrating good generalization and reasonable strategic planning.

  • Takeaways & Limitations

    The hierarchical framework connects explicit utterance-level strategic planning with token-level response generation for conversational agents.

  • Takeaways & Limitations

    Compared with conventional supervised and RL methods, ToSCA adds training complexity and may require careful tuning for stable convergence; its study focuses mainly on conversations.

Abstract

from arXiv · show

Humans have multiple levels of temporal abstractions on daily interaction and thinking, such as concept perception and strategic planning. Inspired by this nature, we propose a two-level hierarchical reinforcement learning (RL) framework for conversational agents, bridging the gap between previous token-level or utterance-level RL methods. Developed on a two-level MDP, the token-level response decoding is conditioned on the utterance-level action, the explicit textual strategies. Based on theoretical derivation and efficiency consideration, we use DQN to solve the high-level critic and PPO to solve the low-level actor-critic. To further alleviate the reward sparsity and facilitate the convergence, we also design the dual-granularity reward mechanism, in which the utterance-level satisfaction score is integrated with token-level intrinsic motivation and K-L penalty. Experiments on both daily and emotional support conversations show that our method outperforms versatile baselines in strategy determination and response quality. Our implementation is available at https://github.com/AaronJi/ToSCA.

1 Introduction

ToSCA addresses limitations of token-level conversational reinforcement learning by using hierarchical temporal and strategic abstractions. It combines explicit utterance-level strategy planning with token-level generation and dual-granularity rewards, then evaluates the approach across conversational domains.

  • Token-level RL faces sparse feedback and high computational overhead, especially because meaningful user feedback often arrives only after a complete assistant utterance.
  • ToSCA guides token-level response generation with an utterance-level strategic planner, bridging explicit strategic planning and detailed response decoding.
  • The framework uses explicit textual strategies rather than continuous latent intents, improving the interpretability of high-level strategic actions.
  • ToSCA combines a high-level DQN critic with a low-level PPO actor-critic within a two-level hierarchical reinforcement-learning framework.
  • Its dual-granularity reward mechanism adds token-level KL penalty and intrinsic motivation to utterance-level user satisfaction, addressing reward sparsity and convergence.
  • Experiments cover DailyDialog, ESConv, and out-of-domain EmpatheticDialogues, with ToSCA surpassing prompting, supervised, and reinforcement-learning baselines.

2 Preliminaries

The paper frames reinforcement learning as discounted-return optimization over Markov decision processes. It introduces Q-learning and PPO as complementary value-based and actor-critic approaches for learning policies and value functions.

  • An MDP is defined by state set S, action set A, reward set R, transition function T, and reward discount factor γ.
  • A reinforcement-learning policy π(a|s) is optimized to maximize the discounted cumulative expected return J.
  • Q-learning learns the optimal state-action value function Q*(s, a), selecting actions that maximize estimated return.
  • Deep Q-learning approximates Bellman-error minimization with deep value networks, using a periodically synchronized target network.
  • PPO uses an actor-critic architecture in which a generative policy network acts and a separate value network serves as the critic.
  • PPO’s objective includes advantage-based policy optimization, KL regularization toward the SFT model, and an auxiliary pretraining objective.

3 Method

ToSCA formulates multi-turn strategy-based conversation as a hierarchical MDP, where utterance-level strategies guide token-level response generation. It combines a discrete high-level critic, token-level actor-critic training, and dual-granularity rewards, with convergence supported under stated assumptions.

  • Hierarchical MDP: ToSCA represents multi-turn conversations with high- and low-level MDPs: strategies guide utterance-level decisions, while responses are generated token by token.The high-level action performs strategic abstraction and conditions the low-level response action.
  • RL Components and Objectives: ToSCA uses a high-level critic QH for discrete strategic bootstrapping and a low-level actor-critic (πL, QL) trained with token-level PPO.The overall architecture is named CriticH-ActorL-CriticL; DQN trains QH while PPO jointly trains πL and QL.
  • The High-Level MDP: The generative high-level value function averages logits of action tokens from an LLM conditional forward pass, and DQN trains this strategic critic.At inference, the strategy with the maximum available-strategy logit is selected.
  • System Configuration: The high-level MDP uses a discrete strategy action set, while the low-level MDP generates the response conditioned on the selected strategy.Strategies are represented by possible strategy indices in a multiple-choice instruction, and the low-level action is the response-token sequence.
  • The Reward Mechanisms: The dual-granularity reward combines utterance-level satisfaction with token-level KL penalty and intrinsic motivation.The satisfaction score is generated by an LLM judge on a 0-to-5 scale, while the KL term constrains deviation from the reference policy and intrinsic motivation aligns context, strategy, and response.
  • Theoretical Derivation: Under the everywhere and Markovian strategy assumptions, the value function converges when the deterministic high-level policy and both critics are solved interleaved.The paper states that this theorem suggests the determined strategy can converge to the ground truth.

4 Experiment

Experiments evaluate ToSCA across daily-life and emotional-support conversations, including out-of-domain transfer, using strategy, response-quality, reward, ablation, and human-evaluation analyses. ToSCA generally achieves strong strategy determination, response quality, cross-domain performance, and generalization.

  • Experimental setup: The experiments address adaptation across conversation domains and generalization when strategy annotations are unavailable.DailyDialog and ESConv provide in-domain evaluations, while EmpatheticDialogues provides an out-of-domain test.
  • Experimental setup: ToSCA is evaluated against direct inference, prompting, supervised, reinforcement-learning, and hierarchical-reinforcement-learning baselines.The compared methods include DQN, PPO, DAT, and ArCher alongside prompting and finetuning approaches.
  • Training analysis: ToSCA training exhibits loss convergence and increasing rewards and values on DailyDialog.Average rewards and values are compared with Direct and Raw dataset references in Table 2.
  • Daily-life conversations: 58.91 MaF1 is ToSCA’s best strategy-determination result on DailyDialog despite competitive rather than best bias performance.The model achieves accuracy without ground-truth strategy reward signals, benefiting from alignment between human annotations and automatic strategy evaluation.
  • Daily-life conversations: ToSCA achieves the highest B-2, R-L, and D-2 response results on DailyDialog, surpassing prompting, finetuning, RL, and structured baselines.B-2 and R-L measure similarity to ground truth, while D-2 measures response diversity.
  • Emotional-support conversations: On ESConv, ToSCA achieves the best or second-best performance across all metrics, while on EmpatheticDialogues it outperforms baselines in the zero-shot test.The reported transfer success is associated with empathy elements included in the reward signal derived from daily dialogues.
  • Ablation study: Removing either hierarchical component causes a significant performance drop, while removing intrinsic motivation moderately decreases automatic metrics.The ablations compare the full model with token-level PPO, a DQN strategy planner guiding a frozen LLM, and a version without intrinsic motivation.
  • Discussion: Intrinsic motivation prevents failure to follow the Question strategy during multi-turn training.The mechanism enforces strategy-aligned behavior when fine-tuning might otherwise ignore prompt constraints for higher rewards.

5 Related Work

Prior hierarchical dialogue methods address multiple generation granularities, but ToSCA adopts discrete, explicit strategy abstraction at the high level. It connects utterance-level strategic actions with token-level response generation within an options-based HRL structure.

  • Hierarchical reinforcement learning: Hierarchical reinforcement learning organizes decision-making through multi-level MDPs, including goal-conditioned and options-based frameworks.ToSCA belongs to the options-based category, where the two action levels retain semantic connections.
  • Dialogue reinforcement learning: Utterance-level RL addresses token-level reward sparsity, while DAT uses continuous utterance actions and ArCHer uses a learned high-level critic.ToSCA shares the two-level HRL structure but replaces continuous high-level abstraction with discrete strategy abstraction.

6 Conclusion

ToSCA combines strategic, utterance-level actions with low-level response generation in a hierarchical dialogue agent. It uses DQN and PPO with auxiliary rewards, and experiments report effectiveness across conversation domains.

  • ToSCA uses a high-level critic to produce strategy-level actions that guide a low-level policy generating detailed responses.
  • The framework trains the high-level critic with DQN and the low-level actor and critic with PPO.
  • KL penalty and intrinsic motivation rewards are added to improve semantic alignment between strategic actions and generated responses.
  • Experiments on daily-life, emotional-support, and out-of-domain conversations demonstrate ToSCA's effectiveness.

Limitations

The hierarchical framework adds training complexity compared with conventional supervised and reinforcement-learning methods. Stable convergence may therefore require careful hyperparameter tuning, and the study remains focused on conversations.

  • The hierarchical framework introduces additional training complexity compared with conventional supervised and RL methods.
  • Careful tuning may be required to ensure stable convergence.
  • The study mainly focuses on conversations rather than generalized, self-evolving hierarchical reasoning and thinking AI.

A Implementation Details

Implementation uses prompts for high-level strategy selection and low-level response generation, with GPT-4o evaluating responses across five binary dimensions. KL divergence is introduced as a distribution-comparison measure.

  • The high-level prompt asks a language model to select one strategy from candidate options for a multi-turn dialogue.
  • The low-level prompt conditions response generation on the selected strategy and dialogue context.
  • GPT-4o evaluates responses for Acceptance, Effectiveness, Sensitivity, Fluency, and Emotion, each scored 0 or 1.
  • KL divergence measures the difference between probability distributions P and Q using their probability density functions p(x) and q(x).

B.1 Datasets

The experiments use DailyDialog and ESConv for in-domain evaluation and EmpatheticDialogues for out-of-domain testing, with dataset-specific annotations and splits. Sensitivity analyses select γ = 0.85 and reward weights β1 = 0.01 and β2 = 0.01.

  • Datasets: DailyDialog contains 13,118 multi-turn daily-life dialogues annotated with 7 emotions and 4 strategies.
  • Datasets: ESConv contains 1,000+ emotional-support dialogues annotated with 11 emotions and 8 strategies organized across three stages.
  • Datasets: EmpatheticDialogues focuses on emotion-centered conversations, includes user-emotion annotations, and has no strategy annotations.
  • Data splitting: DailyDialog uses 11,118 training, 1,000 validation, and 1,000 test conversations, while ESConv uses a random 9:1 training-test split.
  • Data splitting: EmpatheticDialogues uses approximate 80%/10%/10% train-validation-test partitions, grouping identical initial situational descriptions to avoid leakage.
  • Sensitivity analysis: γ = 0.85 is selected because it corresponds to higher Acc and MaF1, lower bias, and similar generative-metric trends.
  • Sensitivity analysis: β1 = 0.01 and β2 = 0.01 are empirically identified as optimal reward-component weights, with both auxiliary loss terms improving performance as weights increase.

C Detailed Evaluation Methods

The evaluation uses strategy-classification metrics, automatic generation metrics, and human ratings to assess response quality and conversational behavior. Baselines are grouped by increasingly guided response-generation procedures and implemented on a shared LLM backbone where possible.

  • Strategy determination: Strategy accuracy measures the fraction of correctly predicted strategies against annotated dataset strategies.
  • Strategy determination: Macro-F1 averages label-level F1 scores, whereas Micro-F1 aggregates precision and recall across all instances.
  • Strategy determination: Preference bias is quantified with Bradley-Terry strategy preferences and computed as the standard deviation across strategies.
  • Generative metrics: Bleu-2 uses modified n-gram precision with a brevity penalty based on prediction and reference lengths, while Rouge-L uses longest-common-subsequence F-measure.
  • Generative metrics: Dist-2 measures response diversity through distinct unigrams and bigrams normalized by the number of generated tokens.
  • Human evaluation: Eight evaluators independently rate responses under predefined criteria, with extreme scores removed before averaging across judgments.

E Theoretical Proofs

The theoretical analysis recasts ToSCA’s high-level controller as an SMDP whose action duration equals the low-level output length, then establishes convergence under finite-space, finite-variance, and discount assumptions. The accompanying experiments report reward-model agreement, training behavior, and strategy-conditioned performance.

  • Hierarchical MDP formulation: ToSCA extends the traditional MDP into a two-level hierarchical MDP, equivalent to an SMDP on a unified time scale.
  • Hierarchical MDP formulation: The SMDP adds a duration-distribution function F, and ToSCA sets F(t | s, a) = L, where L is the maximum low-level output length.
  • Theoretical derivation: Maximizing the high-level value function produces the optimal strategy under the reformulated SMDP Bellman equation.
  • Convergence proof: SMDP Q-learning convergence is established under finite state and action spaces, finite reward variance, and a discount condition below one.
  • Empirical validation: GPT-4o scores significantly correlate with human judgments across six evaluation dimensions, with all Spearman coefficients above 0.8 and p-values below 0.01 or 0.001.
  • Training behavior: ToSCA’s training curves show decreasing losses, bounded auxiliary signals, and a gradually increasing utterance-level satisfaction reward.

F.7 Case analysis

Case analyses show ToSCA using explicit dialogue strategies to guide daily conversations and sustain topics, while also exposing failures from unsuitable strategies and reinforcement-learning hallucinations. Per-strategy evaluations further indicate balanced performance and stronger Question and Inform results.

  • Human evaluation: ToSCA’s responses align with human preferences better than baselines in averaged evaluations on DailyDialog and ESConv.
  • Typical good cases: ToSCA uses dialogue strategies more effectively than Direct to guide a cooking dialogue and sustain its topic.GPT-4o evaluation reports higher reward and long-term value for the ToSCA case.
  • Bad case analysis: Some low-quality outputs arise from inappropriate dialogue strategies or hallucinations during reinforcement-learning training.A Commissive response to a direction inquiry is described as stiff and unnatural.
  • Bad case analysis: Even with the correct Directive strategy, the model can produce semantically unclear utterances such as “Upon completion...”.
  • Per-strategy analysis: Question and Inform strategies show relatively higher Acc, MaF1, and B-2 when metrics are grouped by predicted strategy labels.
Loading 2608.21969v1…