Source-linked AI summary
DialogueReason: Rule-Based RL Sparks Dialogue Reasoning in LLMs
Yubo Shu, Zhewei Huang, Xin Wu, Chen Hu, Shuchang Zhou, Daxin Jiang
TL;DR
Large reasoning models achieve strong results but rely mainly on monologue reasoning, which can exhibit limited diversity and coherency. The paper assesses these issues with Compound-QA and develops DialogueReason, a dialogue-based approach trained with PPO and rule-based rewards. On MATH, AIME, and GPQA, dialogue reasoning outperforms monologue reasoning on compound questions, with improved diversity and coherency.
Problem
Large reasoning models perform strongly on math and science tasks, but monologue reasoning can reuse fixed strategies and shift attention unnecessarily, limiting diversity and coherency.
Source: §1 · pp. 1–2Method
The paper introduces Compound-QA to assess reasoning diversity and coherency, then trains dialogue reasoning structured around agents, an environment, and interactions using PPO with rule-based rewards.
Source: §1 · p. 2Results
Dialogue reasoning outperforms monologue reasoning on MATH, AIME, and GPQA compound QA tasks, with greater accuracy and improved reasoning diversity and coherency.
Source: §1 · p. 2Takeaways & Limitations
The authors identify potential benefits of dialogue reasoning for interpretability, controllability, user interaction, and multi-agent system design.
Source: §1 · p. 2Compound-QA jointly stresses diversity and coherency, making it difficult to isolate weaknesses in either dimension.
Source: §5 · p. 13
Abstract
from arXiv · showhide
We propose DialogueReason, a reasoning paradigm that uncovers the lost roles in monologue-style reasoning models, aiming to boost diversity and coherency of the reasoning process. Recent advances in RL-based large reasoning models have led to impressive long CoT capabilities and high performance on math and science benchmarks. However, these reasoning models rely mainly on monologue-style reasoning, which often limits reasoning diversity and coherency, frequently recycling fixed strategies or exhibiting unnecessary shifts in attention. Our work consists of an analysis of monologue reasoning patterns and the development of a dialogue-based reasoning approach. We first introduce the Compound-QA task, which concatenates multiple problems into a single prompt to assess both diversity and coherency of reasoning. Our analysis shows that Compound-QA exposes weaknesses in monologue reasoning, evidenced by both quantitative metrics and qualitative reasoning traces. Building on the analysis, we propose a dialogue-based reasoning, named DialogueReason, structured around agents, environment, and interactions. Using PPO with rule-based rewards, we train open-source LLMs (Qwen-QWQ and Qwen-Base) to adopt dialogue reasoning. We evaluate trained models on MATH, AIME, and GPQA datasets, showing that the dialogue reasoning model outperforms monologue models under more complex compound questions. Additionally, we discuss how dialogue-based reasoning helps enhance interpretability, facilitate more intuitive human interaction, and inspire advances in multi-agent system design.
1 Introduction
Long-CoT reasoning models can perform strongly on math and science, yet monologue reasoning may lack diversity and coherency. The paper introduces Compound-QA to probe these weaknesses and DialogueReason as a dialogue-based approach, reporting stronger performance on compound questions.
1 Introduction
- Although large reasoning models show strong math and science performance, monologue reasoning can reuse fixed strategies and shift attention unnecessarily, fragmenting the process.The paper associates these patterns with low reasoning diversity and coherency.Source: §1 · pp. 1–2
- Compound-QA concatenates independently solvable problems in one prompt to assess reasoning diversity and coherency, exposing monologue weaknesses through metrics and reasoning traces.Source: §1 · p. 2
- DialogueReason frames reasoning through agents, an environment, and interactions, and uses PPO with rule-based rewards to train Qwen-QWQ-32B and Qwen2.5-Base-32B.Source: §1 · p. 2
- Dialogue reasoning outperforms monologue reasoning on MATH, AIME, and GPQA, with greater accuracy on compound QA tasks and improved reasoning diversity and coherency.The authors also identify potential benefits for interpretability, controllability, user interaction, and multi-agent systems.Source: §1 · p. 2
2 Monologue Reasoning Inspection
This section defines diversity and coherency as complementary reasoning demands, introduces Compound-QA to test both through concatenated questions, and uses QWQ-32B experiments and traces to expose monologue weaknesses.
2 Monologue Reasoning Inspection
- The section examines monologue reasoning through the paired criteria of diversity and coherency, then uses Compound-QA experiments and traces to identify limitations.Its overview describes an inspection of monologue reasoning patterns followed by an evaluation task and experiments.Source: §2 · p. 2
2.1 Diversity and Coherency in Thinking
- Diversity explores distinct perspectives and domains, while coherency sustains a reasoning path; effective reasoning alternates exploration with consolidation.The section associates low coherency with attention shifts and repetitive “wait” or “alternative” behaviors.Source: §2.1 · pp. 2–3
2.2 Compound-QA Task
- Compound-QA concatenates independently solvable questions, requiring varied strategies and sustained paths; increasing k lengthens context and makes switching and attention harder.The task is designed for objectively verifiable answers, and k controls the degree of difficulty.Source: §2.2 · pp. 3–4
2.3 Results
- As cbK rises, accuracy declines while reasoning length generally grows; the decline is sharper on AIME24 and GPQA Diamond than MATH-500.At cbK10 on AIME24, inference length instead decreases; the authors suggest this may reflect exceeding model capacity and a possible reasoning collapse.Source: §2.3 · p. 4, Table 1 · p. 5
2.4 Analysis
- Monologue traces reveal shallow oscillation across tasks, answering only the first question, and reusing its strategy until later questions go unfinished.These cases illustrate fragmented processing, failure to allocate reasoning paths to later questions, and rigidity within the first question’s reasoning framework.Source: §2.4 · p. 5, Figure 3 · p. 6
3 DialogueReason
DialogueReason organizes reasoning as dialogue among agents in a defined environment, with structured interactions, and trains models to adopt this pattern. The section reports stronger performance on compound questions and illustrates dialogue reasoning in both GPQA and an out-of-distribution example.
3.1 Reasoning Pattern Perspective: Dialogue-Based Reasoning
- DialogueReason defines reasoning through agents, an environment, and interactions; structured turn-taking and conversational boundaries are intended to support diverse, coherent reasoning.Agents have characters and objectives; the environment can manage task progression, while interactions include dialogue among agents and feedback between agents and the environment.Source: §3.1 · p. 7
- Rule-based PPO trains Qwen-QWQ-32B and Qwen2.5-Base-32B to adopt dialogue reasoning, and both acquire the pattern.DialogueReason-Base more closely resembles natural conversation, while DialogueReason-QWQ retains more monologue-like expression within each character’s turn.Source: §3.1 · p. 8
3.2 Results
- At cbK=10 on MATH-500, accuracy is 91.48% for DialogueReason versus 89.63% for QWQ; at cbK=8 on AIME, it is 50% versus 35.09%.Figure 5 plots higher remaining rates for DialogueReason across Overall, MATH-500, AIME24, and GPQA-Diamond as compound question size grows.Source: §3.2 · pp. 8–9, Figure 5 · p. 9
3.3 Analysis
- In the GPQA case study, scenes and characters change between questions, and dialogue participants propose, refine, and agree on answers.The authors say distinct scenes and character boundaries prevent interference between questions and support reasoning diversity and dialogue coherence.Source: §3.3 · p. 10, Figure 6 · p. 10
3.4 Out of Distribution Question
- On an out-of-distribution question requiring different perspectives, dialogue assigns viewpoints to characters and explores them more structurally than monologue reasoning.The paper frames this as an exploration of generalization beyond tasks where correctness can be checked with a matching verifier.Source: §3.4 · p. 11, Figure 7 · p. 11
4 Discussion
The discussion presents DialogueReason as a way to make reasoning more interpretable and compatible with monologue reasoning, while offering users clearer opportunities to intervene. It also describes how dialogue-based interaction could support more flexible multi-agent reasoning.
4.1 Interpretability
- Dialogue turns may create clearer semantic boundaries, helping readers recognize reasoning stages and transitions without line-by-line analysis.The authors suggest the listener or interacting party in dialogue may encourage more understandable, structured traces, especially during extended thought chains.Source: §4.1 · pp. 11–12
4.2 Compatibility with Monologue Reasoning
- Dialogue reasoning retains compatibility with monologue reasoning by making implicit cognitive agents, such as reflection or analogy agents, explicit and interactive.The authors describe this as a more structured, modular form of internal deliberation.Source: §4.2 · p. 12
4.3 Controllability and Interaction
- Dialogue offers clearer boundaries for user intervention or automatic step-level verification, and users may configure scenarios or characters or intervene during specific steps.The authors suggest this could enhance transparency and adaptability in practical settings.Source: §4.3 · p. 12
4.4 Multi-Agent Training
- Dialogue-based multi-agent reasoning may reduce reliance on manually predefined characters and workflows by learning to generate agents and coordinate their responsibilities.The authors propose integrating character definition, segmentation, and switching into an end-to-end RL optimization pipeline.Source: §4.4 · p. 12
5 Limitations and Future Work
The authors identify limits in model coverage, task and domain scope, and the Compound-QA evaluation design. They also propose exploring Compound-QA as an RL training objective.
Model scope
- The study primarily trains and evaluates Qwen-series models, limiting the model-family scope of its findings.The authors characterize the results as promising insights into dialogue-style reasoning and identify further research as necessary.Source: §5 · p. 12
Task and domain scope
- Experiments cover math and science tasks—MATH, AIME-24, and GPQA-Diamond—which may not capture reasoning demands in broader domains.The authors plan to extend evaluation to domains such as commonsense and multi-hop QA.Source: §5 · p. 12
Compound-QA as a training objective
- Compound-QA is used only for evaluation in this study, although the authors propose testing it as an RL training objective.They suggest compound-style questions in training data or RL episodes could help models internalize and generalize multi-step reasoning strategies.Source: §5 · p. 12
Entangled evaluation
- Compound-QA simultaneously stresses reasoning diversity and coherency, making it difficult to isolate weaknesses in either dimension.The authors propose separate targeted benchmarks for diversity, such as strategy switching, and coherency, such as intermediate-step consistency.Source: §5 · p. 13
6 Related work
The related-work discussion covers limitations identified in RL-based reasoning models and prior analyses and interventions on thinking patterns. It contrasts these approaches with DialogueReason, which uses RL to train dialogue reasoning and reports more stable performance as task complexity increases.
6.1 RL-Based Reasoning Model
- Prior work reports attention shifts and inefficient, repetitive reasoning in RL-driven models, while extensive self-reflection alone does not significantly improve accuracy (Wang et al., 2025; Chen et al., 2024; Liu et al., 2025).These findings motivate the paper’s focus on reasoning diversity and coherency in multitask scenarios.Source: §6.1 · p. 13
6.2 Thinking Pattern of Reasoning
- Prior analyses discuss prompt sensitivity and comparisons of unstructured monologue with structured strategies, whose benefits may vary by model size (Liu et al., 2025; Guo et al., 2025; Wen et al., 2025).The supplied discussion says structured strategies can benefit smaller models but hinder larger ones.Source: §6.2 · p. 13
- Prior methods include hierarchical RL over thinking templates, dialogue between decomposer and solver characters, dialogue chain-of-thought, generation-and-verification feedback, and graph-based thought structures (Yang et al., 2025; Han et al., 2023; Chae et al., 2023; Zhu et al., 2023; Besta et al., 2024).The paper distinguishes its approach by training dialogue simulations end to end with RL.Source: §6.2 · pp. 13–14
- DialogueReason uses RL to train a dialogue pattern that balances diversity and coherency on compound questions and maintains more stable reasoning performance as task complexity increases.The authors describe end-to-end training in which models dynamically instantiate agents, configure environments, and manage interactions.Source: §6.2 · p. 14
7 Conclusion
The paper presents dialogue-style reasoning as an alternative for monologue models on compound questions and introduces Compound-QA to assess diversity and coherency together. It reports greater robustness across math and science tasks, while identifying broader domains, disentangled evaluation, and Compound-QA training as future directions.
Conclusion
- Dialogue-style models trained with RL exhibit greater robustness across math and science tasks, according to the paper’s conclusion.The conclusion also notes future work on broader domains, separating evaluation dimensions, and Compound-QA as a potential training objective.Source: §7 · p. 14
- Compound-QA jointly assesses reasoning diversity and coherency, and the paper uses it to highlight limitations of monologue-style reasoning on compound questions.Source: §7 · p. 14
- The authors identify broader-domain evaluation, disentangled metrics, and investigation of Compound-QA as a training objective as future work.Source: §7 · p. 14