PaperlayerView exact revision on arXiv

Source-linked AI summary

PCQC: Privileged Counterfactual Question Credit for Multi-Turn Medical Dialogue

Chenxuan Li, Jiayi Wan, Xinrong Chen, Zhongyu Zhao, Xuecheng Shang, Peixing Wan

arXiv:2609.27987v1cs.LG

TL;DR

Medical dialogue policies need to learn which questions uncover diagnostically useful information, but final-diagnosis supervision does not distinguish the contribution of individual questions or provide question-level feedback for unexecuted alternatives. PCQC uses privileged patient facts to score same-state candidate question–answer pairs and train question selection alongside terminal RL. Across four benchmarks, it outperforms GRPO and ATPO while using fewer inquiry turns than GRPO.

  • Problem

    Final-diagnosis feedback evaluates a consultation as a whole rather than distinguishing the contribution of individual questions, and it provides no question-level feedback for unexecuted alternatives.

    Source: §1 · p. 1

  • Method

    PCQC uses privileged patient facts to answer candidate questions at the same dialogue state, scores their diagnostic utility, and turns comparisons into question-level credit for executed and unexecuted candidates alongside terminal GRPO.

    Source: §1 · p. 2

  • Results

    63.10% mean diagnostic accuracy versus GRPO (58.72%) and ATPO (58.89%), with 33.1% fewer inquiry turns than GRPO across four medical dialogue benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    At deployment, the learned policy requires neither full patient information nor the diagnostic scorer.

    Source: §1 · p. 2

    PCQC is evaluated only in benchmark-based simulated medical dialogue, not clinical deployment, so the results are not evidence of clinical safety or effectiveness.

    Source: Ethics Statement · p. 10

Abstract

from arXiv · show

Large language models (LLMs) have made substantial progress on medical question-answering, yet effective medical dialogue also requires learning to ask questions that uncover relevant patient information. To train such dialogue policies, a common pipeline combines supervised fine-tuning with reinforcement learning (RL) based on final diagnostic correctness. However, this outcome-based supervision does not directly distinguish the contributions of individual questions and provides no question-level feedback for unexecuted alternatives. To address this gap, we introduce PCQC (Privileged Counterfactual Question Credit), which uses privileged patient information during training to learn from questions never asked. During training, PCQC makes alternative questions directly comparable at the same dialogue state by using privileged patient facts to construct their answers. A frozen diagnostic scorer evaluates the diagnostic utility of each resulting question-answer pair by how strongly it supports the correct diagnosis. PCQC turns these comparisons into relative question credit that teaches the policy which questions to favor, directly supervising both executed and unexecuted questions alongside outcome-based RL without requiring complete rollouts for the unexecuted alternatives. Extensive experiments across four medical benchmarks demonstrate that PCQC achieves 63.10% mean diagnostic accuracy, outperforming GRPO and ATPO by 4.38 and 4.21 percentage points, respectively. These gains are achieved with 33.1% fewer inquiry turns than GRPO.

1 Introduction

Medical dialogue policies must learn which questions elicit diagnostically useful information, but final-diagnosis feedback does not distinguish individual questions. PCQC uses privileged patient facts to compare candidate questions at the same state, and across four benchmarks it outperforms GRPO and ATPO while using fewer inquiry turns than GRPO.

1 Introduction

  • Final-diagnosis feedback evaluates a consultation as a whole, while question selection determines what information becomes available for later inquiries and diagnosis.The paper identifies a need for supervision that distinguishes the value of individual inquiry decisions.Source: §1 · p. 1
  • PCQC uses privileged patient facts to answer candidate questions at the same dialogue state, exposing evidence from unexecuted alternatives without further dialogue rollouts.A frozen scorer assesses how strongly each candidate question–answer pair supports the correct diagnosis, and the resulting comparisons yield relative question credit.Source: §1 · p. 2
  • 63.10% mean diagnostic accuracy, versus GRPO (58.72%) and ATPO (58.89%), with 33.1% fewer inquiry turns than GRPO across four medical dialogue benchmarks.PCQC outperforms both baselines on all four benchmarks.Source: §1 · p. 2
  • Relative question credit supervises both executed and unexecuted questions alongside terminal GRPO, jointly training question selection and final diagnosis.This extends learning beyond branches explored during rollout.Source: §1 · p. 2

2 Related Work

Prior work studies interactive medical agents and finer-grained credit assignment using explored interactions, future returns, or information acquired during dialogue. PCQC addresses the complementary case of assigning diagnostic credit to alternative questions that are never executed.

2.1 Medical Dialogue Agents

  • Medical dialogue agents study sequential information acquisition and learn question-selection policies, while PCQC focuses on assigning diagnostic credit among same-state alternatives, including unexecuted inquiries.Related approaches include MediQ, AMIE, AgentClinic, DoctorAgent-RL, Doctor-R1, and ALFA (Li et al., 2024; Tu et al., 2025; Schmidgall et al., 2026; Feng et al., 2025b; Lai et al., 2026; Li et al., 2025).Source: §2.1 · p. 3

2.2 Fine-Grained Credit Assignment

  • Credit-assignment methods use future returns, repeated states, or expanded trajectories; medical examples include ATPO and IGRPO (Cao et al., 2026).VinePPO estimates intermediate values from Monte Carlo continuations, while GiGPO groups actions at shared anchor states (Kazemnejad et al., 2025; Feng et al., 2025a).Source: §2.2 · p. 3
  • Information-based methods derive feedback from realized or explored interactions, whereas PCQC uses privileged patient facts to score unexecuted alternatives at the same state without further rollouts.Examples include ProMed, IGPO, TIPS, and InfoPO (Ding et al., 2026; Wang et al., 2026; Xie et al., 2026; Kong et al., 2026).Source: §2.2 · p. 3

3 Privileged Counterfactual Question Credit

PCQC constructs same-state question–answer candidates using patient facts, scores their diagnostic utility, and converts within-state comparisons into question-level credit. It combines that supervision with terminal GRPO and does not require privileged information or the scorer at deployment.

3 Privileged Counterfactual Question Credit

  • The framework constructs and evaluates same-state candidate interactions, derives relative question credit, and combines it with terminal GRPO.The policy samples candidate questions, a frozen patient responder answers using patient facts, and a frozen diagnostic scorer assigns utilities before within-state normalization.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Clinical Inquiry as Selective Information Acquisition

  • Clinical inquiry is formulated as sequential information acquisition: the policy observes the initial patient information and preceding exchanges, then asks a question or makes a final diagnosis.The patient responder conditions its answer on the patient facts and asked question; the answer is appended to the dialogue state, and the final diagnosis receives terminal evaluation.Source: §3.1 · p. 4

3.2 Learning from Questions Never Asked

  • At each supervised state, sampled candidate questions are answered using the same patient facts, but only the executed question–answer pair advances the consultation.The other candidates provide counterfactual interactions without further dialogue rollouts.Source: §3.2 · pp. 4–5

3.3 Relative Privileged Question Credit

  • PCQC normalizes candidate diagnostic utilities within each state, assigning positive credit above the group mean and negative credit below it when utilities vary enough.When their standard deviation falls below τ = 10^-4, credit is suppressed; ϵ_norm = 10^-6 stabilizes normalization.Source: §3.3 · p. 5
  • Relative credit can equivalently compare candidates’ diagnostic gains from a shared baseline, preserving the ordering induced by reductions in correct-option surprisal.Its magnitude is determined by normalized diagnostic gains on the probability scale.Source: §3.3 · p. 5

3.4 Joint Optimization of Inquiry and Diagnosis

  • PCQC applies question credit to doctor-generated candidate-question tokens and combines its PPO-style clipped objective with terminal GRPO feedback.The question-credit objective applies to executed and unexecuted candidates; a coefficient β balances the question-level and trajectory-level objectives.Source: §3.4 · p. 6
  • At deployment, the policy uses the observed dialogue, clinical task, and answer options without privileged patient information, counterfactual candidate construction, or the diagnostic scorer.The joint objectives train the policy to acquire diagnostically useful information while optimizing the final diagnostic outcome.Source: §3.4 · p. 6

4 Experiments

Across four medical dialogue benchmarks, PCQC combines a shared interaction protocol with counterfactual question supervision, achieving higher diagnostic accuracy and fewer inquiry turns than GRPO. Further experiments compare question-level supervision configurations and test whether diagnostic utility predicts downstream success.

4.1 Experimental Setup

  • The evaluation uses a shared 14,256-case training set, a 1,268-case MedQA development split, and three held-out benchmarks; all policies follow the same interaction protocol.The policy is limited to 10 dialogue turns per consultation, including the final answer; RL metrics are means and sample standard deviations across three runs.Source: §4.1 · pp. 6–7

4.2 Diagnostic Performance and Inquiry Efficiency

  • 63.10% mean diagnostic accuracy: PCQC exceeds GRPO (58.72%) and ATPO (58.89%) by 4.38 and 4.21 percentage points, while using 33.1% fewer inquiry turns than GRPO.The accuracy advantage holds on all four benchmarks; Figure 3 summarizes the joint accuracy and inquiry-turn comparison.Source: §4.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7

4.3 Value of Counterfactual Candidate Supervision

  • PCQC reaches 59.60% mean accuracy across MedQA, MedicalExam, and MedMCQA, 5.36 percentage points above executed-local feedback and 10.22 points above terminal-only training.Executed-local feedback raises mean accuracy from 49.38% to 54.24%; PCQC also improves accuracy on all three benchmarks while requiring fewer inquiry turns than either control.Source: §4.3 · p. 8, Table 3 · p. 8
  • With the same 222-update budget, directly supervising unexecuted candidates raises four-benchmark mean accuracy from 59.95% to 63.02% and reduces mean inquiry turns from 2.153 to 1.573.On MedQA, PCQC reaches 61.59% accuracy versus 58.20% for Same-State-Q1, with inquiry turns falling from 2.096 to 1.574.Source: §4.3 · p. 9

4.4 Diagnostic Utility Predicts Downstream Success

  • 65.14% downstream diagnostic success from highest-utility selection exceeds uniform selection by 4.20 percentage points and the executed question by 5.47 points.Among grounded candidates, highest-utility selection improves success over uniform selection by 3.52 percentage points; Table 4 and Figure 4 present the continuation-based evaluation.Source: §4.4 · pp. 9–10, Figure 4 · p. 9, Table 4 · p. 9

5 Conclusion

PCQC learns from questions never asked by using privileged patient information to compare candidate question–answer interactions at the same dialogue state. Across four benchmarks, it improves diagnostic accuracy over GRPO and ATPO while requiring fewer inquiry turns than GRPO.

5 Conclusion

  • PCQC uses privileged patient information to compare candidate question–answer interactions at the same dialogue state and derive relative credit for executed and unexecuted questions without further dialogue rollouts.It combines this question-level supervision with terminal GRPO.Source: §5 · p. 10
  • 63.10% mean diagnostic accuracy across four benchmarks outperforms GRPO and ATPO, while inquiry turns are 33.1% fewer than with GRPO.Controlled experiments find gains beyond feedback on executed questions, and diagnostic utility identifies questions associated with higher downstream diagnostic success.Source: §5 · p. 10
  • Training-time patient information can supervise information-acquisition decisions as well as final diagnostic outcomes.Source: §5 · p. 10

Ethics Statement

PCQC is evaluated in benchmark-based simulated medical dialogue, and the reported results are limited to those experimental settings rather than clinical deployment.

Ethics Statement

  • PCQC is studied in benchmark-based simulated medical dialogue and is not evaluated for clinical deployment.Source: Ethics Statement · p. 10
  • The experiments use existing clinical benchmark cases, without prospective patient interaction or collection of new patient data.Source: Ethics Statement · p. 10
  • Reported diagnostic accuracy and inquiry behavior characterize these experiments and are not evidence of clinical safety or effectiveness.Source: Ethics Statement · p. 10

AI Use Statement

The paper describes AI assistance in research and manuscript work, and documents the sources, preparation, and overlap checks for its training and evaluation datasets.

AI Use Statement

  • Generative AI assisted manuscript writing and editing, literature discovery, selected research and coding tasks, and model judging in an answer-grounding analysis.Authors reviewed all AI-assisted outputs and take responsibility for the scientific claims, results, and manuscript.Source: AI Use Statement · p. 10

A.1 Datasets

  • Training uses 4,157 conversations for supervised initialization and 14,256 RL cases from the released ATPO collection.The RL cases comprise 9,400 MEDIQ/MedQA and 4,856 MedMCQA cases; conversion preserves source messages and target responses.Source: §A.1 · p. 13
  • Evaluation uses released MedQA, MedicalExam, and MedMCQA sets, plus 139 of 140 iCRAFT-MD cases after excluding one with conflicting answer text and option index.The iCRAFT-MD case set and evaluated policies are fixed before evaluation.Source: §A.1 · p. 13
  • Overlap checks found no normalized full-prompt matches for the three ATPO evaluation sets, but found two MedQA case-and-problem matches and lexical near-duplicate candidates among 50 cases.For iCRAFT-MD, checks against SFT and RL data found no exact matches or lexical matches above the specified retrieval thresholds.Source: §A.1 · p. 13

A.2 Training Configuration

The appendix specifies model initialization, optimization settings, comparison protocols, question-supervision controls, and evaluation and statistical procedures for the training configuration.

A.2 Training Configuration

  • Table 5 summarizes the models and sampling groups used for the reported comparisons.Source: Table 5 · p. 13, §A.2 · p. 13
  • The shared Qwen3-8B actor receives assistant-token supervision for three epochs, while a separate frozen Qwen3-8B patient responder is trained for one epoch.The actor schedule uses 96 updates and the responder schedule uses 32 updates.Source: §A.2 · p. 13
  • GRPO and PCQC use full-parameter BF16 training with shared optimizer settings; PCQC additionally optimizes question credit with β = 1.Terminal rewards are binary diagnostic correctness; PCQC uses AdamW, constant learning rate, and gradient clipping at 1.0.Source: §A.2 · p. 13
  • ATPO follows its U1+U2 configuration with tree budget M = 128, four candidates, variance-threshold pruning, and a separately configured critic and actor.Source: §A.2 · p. 13
  • Methods use the released MedQA dialogue set for development and checkpoint selection, then evaluate selected checkpoints on MedicalExam, MedMCQA, and iCRAFT-MD without further selection.GRPO, ATPO, and PCQC each use three independent RL runs.Source: §A.2 · p. 14

A.3 Evaluation and Metrics

  • Question-supervision controls compare terminal-only training with executed-local feedback, while evaluation reports diagnostic accuracy, inquiry attempts, generation length, and case-level uncertainty.Diagnostic accuracy counts missing or invalid answers as incorrect; paired 95% confidence intervals use 10,000 case-level bootstrap resamples, and exact two-sided McNemar tests compare discordant predictions.Source: §A.2 · p. 14, §A.3 · p. 14

A.4 Full-System Training Cost

On MedQA, PCQC Run A achieves higher diagnostic accuracy than terminal GRPO at a lower reported GPU-hour cost, with fewer consultation trajectories; the cost includes RL-system components but excludes shared SFT and separate evaluation.

A.4 Full-System Training Cost

  • Table 6 GPU-hours multiply recorded RL training time by all eight allocated H800 GPUs through the reported checkpoint, including policy optimization, rollout, patient responses, and diagnostic scoring.Shared SFT and separate benchmark evaluations are excluded.Source: §A.4 · p. 14, Table 6 · p. 14
  • 44.5% fewer complete consultation trajectories were used by PCQC Run A than terminal GRPO: 113,664 versus 204,800.Source: §A.4 · p. 14
  • 61.59% diagnostic accuracy at 70.62 H800 GPU-hours for PCQC Run A (GT = 4) compares with 56.07% at 72.68 GPU-hours for terminal GRPO (GT = 32) on MedQA.Source: Table 6 · p. 14, §A.4 · p. 14

B Question-Credit Implementation

PCQC builds same-state counterfactual question–answer pairs from privileged patient facts, scores their diagnostic utility, and uses relative credit to supervise question selection. The implementation also specifies how candidate answers, diagnostic utility, and question-token updates are constructed.

B.1 Information Access

  • Patient facts supply answers for candidate questions, while the diagnostic scorer evaluates observed question–answer evidence and uses the correct label to select its output probability.The information-access table distinguishes the patient responder’s access to facts from the scorer’s evidence inputs.Source: Table 7 · p. 15, §B.1 · p. 15

B.2 Interaction Protocol and Prompts

  • The actor emits either a question or final answer, and the patient responder answers only from fixed patient facts, without adding medical inference.If no fact directly answers the question, the responder returns a fixed cannot-answer phrase; decoding uses temperature 0.8 and a 256-token limit.Source: §B.2 · p. 15

B.3 Diagnostic Utility Construction

  • The scorer normalizes final-token log probabilities across answer options, and the correct-option probability defines diagnostic utility.Subtracting the shared pre-exchange probability before normalization gives the same relative question credit as Equation 5.Source: §B.3 · p. 16

B.4 Candidate Construction and Question-Token Optimization

  • Four consultations per case supply supervised states, where four actor responses form candidate groups and only the first valid question advances the consultation.Question credit applies only to actor tokens after the first Question: marker; terminal and question losses are normalized separately and combined at β = 1 with reference-policy KL regularization.Source: §B.4 · p. 16

B.5 Executed-Question Local Credit

  • Executed-local standardizes diagnostic gains across valid executed-question events; unexecuted candidates are scored but receive no local gradient.Credit is zero with fewer than two events or standard deviation below 10^-4, and its objective is combined with terminal GRPO at β = 1.Source: §B.5 · p. 16

C Properties of Relative Question Credit

  • Subtracting a shared baseline shifts candidate gains’ mean but preserves sample variance, so utilities and gains yield identical threshold decisions and normalized credit.When the group standard deviation meets the threshold, credit is the normalized mean of pairwise utility differences; positive credit marks above-average diagnostic gain.Source: C Properties of Relative Question Credit · pp. 16–17

D.1 Benchmark Results and Interaction Metrics

PCQC’s update-222 checkpoints retain the same performance pattern across the primary benchmarks and independent iCRAFT-MD evaluation, while question-selection analyses compare supervision variants and candidate-selection rules. In the grounded-answer analysis, highest-utility selection exceeds uniform selection among cases with multiple eligible candidates.

D.1 Benchmark Results and Interaction Metrics

  • 63.10% mean diagnostic accuracy is reported across PCQC’s three update-222 checkpoints, which preserve the same pattern across the three primary benchmarks and independent iCRAFT-MD evaluation.The mean equally weights all four benchmarks.Source: §D.1 · p. 17, Table 8 · p. 17
  • PCQC reduces mean actor-generated tokens per consultation relative to terminal GRPO on MedQA, MedicalExam, and MedMCQA.Table 10 reports PCQC entries averaged over three independent RL runs and reductions relative to GRPO.Source: §D.1 · p. 18, Table 10 · p. 18

D.2 Decomposing Candidate Question Supervision

  • Same-State-Q1 retains all four candidates for relative credit but supervises only the executed question, while PCQC averages question loss over all candidate tokens.Table 12 compares both methods at 222 updates, with one run each and equal weighting across four benchmarks.Source: §D.2 · p. 18, Table 12 · p. 19

D.3 Downstream Evaluation of Candidate Questions

  • The downstream evaluation samples four initial-state candidates for each of 128 MedQA cases and generates eight continuations per question–answer pair.A candidate’s downstream diagnostic success is the fraction of continuations reaching the correct final diagnosis; selection is compared with the executed question and uniform selection.Source: §D.3 · p. 19

D.4 Answer Grounding and Continuation Sensitivity

  • 3.52 percentage points: among 113 cases with at least two eligible candidates, highest-utility selection improves downstream diagnostic success over uniform selection from that same eligible set.Eligible answers are supported by patient facts, confined to the requested information, and do not disclose the decision target.Source: §D.4 · p. 19, Table 13 · p. 19
Loading 2609.27987v1…