ToSCA: Leveraging Hierarchical Reinforcement Learning on Temporal and Strategic Abstractions of Conversational Agents

• Token-level RL faces sparse feedback and high computational overhead, especially because meaningful user feedback often arrives only after a complete assistant utterance.Source: §1 · p. 1
• ToSCA guides token-level response generation with an utterance-level strategic planner, bridging explicit strategic planning and detailed response decoding.Source: §1 · p. 2
• The framework uses explicit textual strategies rather than continuous latent intents, improving the interpretability of high-level strategic actions.Source: §1 · p. 2
• ToSCA combines a high-level DQN critic with a low-level PPO actor-critic within a two-level hierarchical reinforcement-learning framework.Source: §1 · p. 2
• Its dual-granularity reward mechanism adds token-level KL penalty and intrinsic motivation to utterance-level user satisfaction, addressing reward sparsity and convergence.Source: §1 · p. 2
• Experiments cover DailyDialog, ESConv, and out-of-domain EmpatheticDialogues, with ToSCA surpassing prompting, supervised, and reinforcement-learning baselines.Source: §1 · p. 2

• An MDP is defined by state set S, action set A, reward set R, transition function T, and reward discount factor γ.Source: §2.1 · p. 2
• A reinforcement-learning policy π(a|s) is optimized to maximize the discounted cumulative expected return J.Source: §2.1 · p. 2
• Q-learning learns the optimal state-action value function Q*(s, a), selecting actions that maximize estimated return.Source: §2.2 · pp. 2–3
• Deep Q-learning approximates Bellman-error minimization with deep value networks, using a periodically synchronized target network.Source: §2.2 · p. 3
• PPO uses an actor-critic architecture in which a generative policy network acts and a separate value network serves as the critic.Source: §2.3 · p. 3
• PPO’s objective includes advantage-based policy optimization, KL regularization toward the SFT model, and an auxiliary pretraining objective.Source: §2.3 · p. 3

Hierarchical MDP

• ToSCA represents multi-turn conversations with high- and low-level MDPs: strategies guide utterance-level decisions, while responses are generated token by token.The high-level action performs strategic abstraction and conditions the low-level response action.Source: §3.1 · p. 3

RL Components and Objectives

• ToSCA uses a high-level critic QH for discrete strategic bootstrapping and a low-level actor-critic (πL, QL) trained with token-level PPO.The overall architecture is named CriticH-ActorL-CriticL; DQN trains QH while PPO jointly trains πL and QL.Source: §3.2 · pp. 3–4, §3.3 · p. 4, §3.4 · p. 4

The High-Level MDP

• The generative high-level value function averages logits of action tokens from an LLM conditional forward pass, and DQN trains this strategic critic.At inference, the strategy with the maximum available-strategy logit is selected.Source: §3.3 · p. 4

System Configuration

• The high-level MDP uses a discrete strategy action set, while the low-level MDP generates the response conditioned on the selected strategy.Strategies are represented by possible strategy indices in a multiple-choice instruction, and the low-level action is the response-token sequence.Source: §3.1 · p. 3, §3.3 · p. 4, §3.5 · p. 5

The Reward Mechanisms

• The dual-granularity reward combines utterance-level satisfaction with token-level KL penalty and intrinsic motivation.The satisfaction score is generated by an LLM judge on a 0-to-5 scale, while the KL term constrains deviation from the reference policy and intrinsic motivation aligns context, strategy, and response.Source: §3.5 · pp. 4–5

Theoretical Derivation

• Under the everywhere and Markovian strategy assumptions, the value function converges when the deterministic high-level policy and both critics are solved interleaved.The paper states that this theorem suggests the determined strategy can converge to the ground truth.Source: §3.7 · p. 5

Experimental setup

• The experiments address adaptation across conversation domains and generalization when strategy annotations are unavailable.DailyDialog and ESConv provide in-domain evaluations, while EmpatheticDialogues provides an out-of-domain test.Source: §4 · p. 5, §4.2 · p. 6
• ToSCA is evaluated against direct inference, prompting, supervised, reinforcement-learning, and hierarchical-reinforcement-learning baselines.The compared methods include DQN, PPO, DAT, and ArCher alongside prompting and finetuning approaches.Source: §4.4 · p. 6

Training analysis

• ToSCA training exhibits loss convergence and increasing rewards and values on DailyDialog.Average rewards and values are compared with Direct and Raw dataset references in Table 2.Source: §4.5 · p. 6, Table 2 · p. 6

Daily-life conversations

• 58.91 MaF1 is ToSCA’s best strategy-determination result on DailyDialog despite competitive rather than best bias performance.The model achieves accuracy without ground-truth strategy reward signals, benefiting from alignment between human annotations and automatic strategy evaluation.Source: §4.5 · p. 7
• ToSCA achieves the highest B-2, R-L, and D-2 response results on DailyDialog, surpassing prompting, finetuning, RL, and structured baselines.B-2 and R-L measure similarity to ground truth, while D-2 measures response diversity.Source: §4.3 · p. 6, §4.5 · p. 7

Emotional-support conversations

• On ESConv, ToSCA achieves the best or second-best performance across all metrics, while on EmpatheticDialogues it outperforms baselines in the zero-shot test.The reported transfer success is associated with empathy elements included in the reward signal derived from daily dialogues.Source: §4.6 · p. 7

Ablation study

• Removing either hierarchical component causes a significant performance drop, while removing intrinsic motivation moderately decreases automatic metrics.The ablations compare the full model with token-level PPO, a DQN strategy planner guiding a frozen LLM, and a version without intrinsic motivation.Source: §4.6 · p. 7, Table 6 · p. 8

Discussion

• Intrinsic motivation prevents failure to follow the Question strategy during multi-turn training.The mechanism enforces strategy-aligned behavior when fine-tuning might otherwise ignore prompt constraints for higher rewards.Source: §4.7 · p. 8

Hierarchical reinforcement learning

• Hierarchical reinforcement learning organizes decision-making through multi-level MDPs, including goal-conditioned and options-based frameworks.ToSCA belongs to the options-based category, where the two action levels retain semantic connections.Source: §5.1 · p. 8

Dialogue reinforcement learning

• Utterance-level RL addresses token-level reward sparsity, while DAT uses continuous utterance actions and ArCHer uses a learned high-level critic.ToSCA shares the two-level HRL structure but replaces continuous high-level abstraction with discrete strategy abstraction.Source: §5.2 · p. 8

• ToSCA uses a high-level critic to produce strategy-level actions that guide a low-level policy generating detailed responses.Source: §6 · p. 8
• The framework trains the high-level critic with DQN and the low-level actor and critic with PPO.Source: §6 · pp. 8–9
• KL penalty and intrinsic motivation rewards are added to improve semantic alignment between strategic actions and generated responses.Source: §6 · p. 9
• Experiments on daily-life, emotional-support, and out-of-domain conversations demonstrate ToSCA's effectiveness.Source: §6 · p. 9

• The hierarchical framework introduces additional training complexity compared with conventional supervised and RL methods.Source: Limitations · p. 9
• Careful tuning may be required to ensure stable convergence.Source: Limitations · p. 9
• The study mainly focuses on conversations rather than generalized, self-evolving hierarchical reasoning and thinking AI.Source: Limitations · p. 9

• The high-level prompt asks a language model to select one strategy from candidate options for a multi-turn dialogue.Source: §A.1 · p. 12
• The low-level prompt conditions response generation on the selected strategy and dialogue context.Source: §A.1 · p. 12, §A.2 · p. 12
• GPT-4o evaluates responses for Acceptance, Effectiveness, Sensitivity, Fluency, and Emotion, each scored 0 or 1.Source: §A.1 · p. 12
• KL divergence measures the difference between probability distributions P and Q using their probability density functions p(x) and q(x).Source: §A.2 · p. 12

Datasets

• DailyDialog contains 13,118 multi-turn daily-life dialogues annotated with 7 emotions and 4 strategies.Source: §B.1 · p. 12
• ESConv contains 1,000+ emotional-support dialogues annotated with 11 emotions and 8 strategies organized across three stages.Source: §B.1 · pp. 12–13
• EmpatheticDialogues focuses on emotion-centered conversations, includes user-emotion annotations, and has no strategy annotations.Source: §B.1 · p. 13

Data splitting

• DailyDialog uses 11,118 training, 1,000 validation, and 1,000 test conversations, while ESConv uses a random 9:1 training-test split.Source: §B.1 · p. 13
• EmpatheticDialogues uses approximate 80%/10%/10% train-validation-test partitions, grouping identical initial situational descriptions to avoid leakage.Source: §B.1 · pp. 13–14

Sensitivity analysis

• γ = 0.85 is selected because it corresponds to higher Acc and MaF1, lower bias, and similar generative-metric trends.Source: §B.2 · p. 14, Figure 4 · p. 14
• β1 = 0.01 and β2 = 0.01 are empirically identified as optimal reward-component weights, with both auxiliary loss terms improving performance as weights increase.Source: §B.2 · p. 14

Strategy determination

• Strategy accuracy measures the fraction of correctly predicted strategies against annotated dataset strategies.Source: §C.1 · p. 14
• Macro-F1 averages label-level F1 scores, whereas Micro-F1 aggregates precision and recall across all instances.Source: §C.1 · p. 15
• Preference bias is quantified with Bradley-Terry strategy preferences and computed as the standard deviation across strategies.Source: §C.1 · p. 15

Generative metrics

• Bleu-2 uses modified n-gram precision with a brevity penalty based on prediction and reference lengths, while Rouge-L uses longest-common-subsequence F-measure.Source: §C.2 · p. 15
• Dist-2 measures response diversity through distinct unigrams and bigrams normalized by the number of generated tokens.Source: §C.2 · p. 15

Human evaluation

• Eight evaluators independently rate responses under predefined criteria, with extreme scores removed before averaging across judgments.Source: §C.3 · pp. 15–16

Hierarchical MDP formulation

• ToSCA extends the traditional MDP into a two-level hierarchical MDP, equivalent to an SMDP on a unified time scale.Source: E Theoretical Proofs · p. 18
• The SMDP adds a duration-distribution function F, and ToSCA sets F(t | s, a) = L, where L is the maximum low-level output length.Source: E Theoretical Proofs · p. 18

Theoretical derivation

• Maximizing the high-level value function produces the optimal strategy under the reformulated SMDP Bellman equation.Source: E Theoretical Proofs · p. 18

Convergence proof

• SMDP Q-learning convergence is established under finite state and action spaces, finite reward variance, and a discount condition below one.Source: E Theoretical Proofs · p. 18

Empirical validation

• GPT-4o scores significantly correlate with human judgments across six evaluation dimensions, with all Spearman coefficients above 0.8 and p-values below 0.01 or 0.001.Source: §F.1 · p. 19

Training behavior

• ToSCA’s training curves show decreasing losses, bounded auxiliary signals, and a gradually increasing utterance-level satisfaction reward.Source: §F.2 · p. 19, Figure 5 · p. 19

Human evaluation

• ToSCA’s responses align with human preferences better than baselines in averaged evaluations on DailyDialog and ESConv.Source: Table 13 · p. 20

Typical good cases

• ToSCA uses dialogue strategies more effectively than Direct to guide a cooking dialogue and sustain its topic.GPT-4o evaluation reports higher reward and long-term value for the ToSCA case.Source: §F.7 · pp. 19–20

Bad case analysis

• Some low-quality outputs arise from inappropriate dialogue strategies or hallucinations during reinforcement-learning training.A Commissive response to a direction inquiry is described as stiff and unnatural.Source: §F.7 · p. 20, Table 19 · p. 22
• Even with the correct Directive strategy, the model can produce semantically unclear utterances such as “Upon completion...”.Source: §F.7 · p. 20

Per-strategy analysis

• Question and Inform strategies show relatively higher Acc, MaF1, and B-2 when metrics are grouped by predicted strategy labels.Source: Table 18 · p. 22