PaperlayerView exact revision on arXiv

Source-linked AI summary

MOCC-R1: Reinforcing Reasoning-Response Consistency for Multimodal Counselor Response Generation

Wenjie Zheng, Qiming Xie, Jianfei Yu, Rui Xia

arXiv:2609.17180v1cs.AI

TL;DR

MCRG lacks sustained professional counseling supervision and an objective enforcing consistency between counseling reasoning and generated responses. The paper introduces MOCC and MOCC-R1, which combines structured cold-start supervision with GRPO rewards for grounded plan coherence and plan execution. On MOCC, MOCC-R1 achieves the strongest overall results among evaluated training-based systems and reduces inconsistency from 43% to 12% against matched outcome-level GRPO.

  • Problem

    MCRG lacks sustained human-recorded counseling data involving qualified counselors and dedicated objectives for consistency across the structured counseling decision chain.

  • Method

    MOCC-R1 trains structured client-state and plan outputs with verified cold-start supervision, then uses GRPO rewards for grounded plan coherence and plan execution.

  • Results

    43% to 12%: MOCC-R1 reduces reasoning–response inconsistency against the matched Outcome-level GRPO baseline and achieves the strongest overall results among evaluated training-based systems.

  • Takeaways & Limitations

    The findings establish chain-level reasoning–response consistency as a distinct objective rather than a by-product of response-level optimization.

  • Takeaways & Limitations

    The consistency construct evaluates semantic correspondence among generated fields, but does not establish causal mediation, reveal latent computation, or constitute a clinically validated formulation.

Abstract

from arXiv · show

Multimodal counselor response generation (MCRG) aims to generate an appropriate counselor response from multimodal dialogue histories. Progress is limited by two gaps: first, existing datasets rarely capture sustained, human-recorded counseling interactions conducted by qualified counselors; Second, existing methods do not explicitly optimize consistency between counseling reasoning and the generated response, potentially undermining the reliability of MCRG systems. Thus, we introduce MOCC, a multimodal counseling conversation corpus containing over 200 hours of interactions involving 154 credential-verified counselors. Based on MOCC, we propose MOCC-R1, a two-stage framework for optimizing reasoning-response consistency. Cold-start supervised fine-tuning trains the model to generate a structured trajectory consisting of client-state understanding, a response intent that links a counseling principle to a planned action, and the final response. Reinforcement learning (RL) then rewards grounded plan coherence and plan execution, encouraging the inferred state and plan to be supported by the dialogue context and the response to realize that plan. Experiments demonstrate the effectiveness of the proposed MOCC-R1.

Introduction

MCRG is constrained by limited supervision from sustained professional counseling and by training objectives that do not enforce consistency across counseling reasoning and responses. MOCC-R1 addresses these gaps with a counseling corpus and a two-stage consistency-optimized framework.

  • Motivation: MOCC-R1 addresses limited professional counseling supervision and missing chain-level consistency objectives in multimodal counselor response generation.Existing resources often emphasize empathy, use scripted or synthetic interactions, or provide short role-play sessions; existing objectives separately target intermediate reasoning or response-level qualities.
  • MOCC: MOCC contains 482 human-recorded sessions totaling approximately 203 hours and involving 154 credential-verified counselors.The corpus includes real-client and simulated-client sessions segmented into 3,709 problem-centered dialogue units and 183K utterances, averaging 22 turns per unit.
  • MOCC-R1: MOCC-R1 uses cold-start supervised fine-tuning to generate an evidence-grounded client state, counseling principle, planned action, and counselor response.The structured decision chain links understanding and planning to the generated response.
  • MOCC-R1: GRPO adds rewards for grounded plan coherence and plan execution, assessing whether the context supports the state and plan and whether the response realizes that plan.The consistency objective concerns coherence among the explicitly generated components rather than latent reasoning or actual counselor cognition.
  • Results: 43% to 12%: MOCC-R1 reduces reasoning–response inconsistency against a matched outcome-only GRPO baseline while outperforming existing task-specific MCRG baselines overall.The reported experiments on MOCC support consistency-aware optimization as the framework’s central empirical contribution.

Related Work

Prior multimodal counseling resources provide limited supervision for sustained professional interactions, while reasoning-aware methods and RL objectives do not separately optimize the semantic link between counseling plans and responses.

  • Multimodal counseling datasets: Existing multimodal mental-health datasets commonly rely on reenacted cases, short role-play sessions, television scripts, or synthetic dialogues and imagery.These settings differ from sustained counselor response generation based on professionally conducted human interactions.
  • Reasoning-aware generation: Reasoning-aware response-generation systems model empathy, strategy planning, counseling principles, or multimodal cues, but their structured reasoning targets differ across approaches.Prior systems separate emotion understanding from support-strategy reasoning or use multi-hop psychotherapy reasoning.
  • RL for structured counseling reasoning: Existing counseling RL methods optimize trustworthiness, structured empathy, reasoning steps, response preference, format, emotion, or strategy correctness.None treats the semantic link between a proposed counseling intervention and the final response as a distinct non-component objective in the supplied comparison.

Dataset

MOCC is built from counseling recordings that are filtered, transcribed, aligned, de-identified, segmented, and annotated into a multimodal dataset with sustained dialogue units. Its sessions involve credential-verified counselors and cover client presenting problems and dataset statistics documented in the accompanying tables and figure.

  • Construction pipeline: The construction pipeline filters counseling segments, extracts and aligns speech transcripts, verifies speakers, segments utterances, removes identifying information, and annotates presenting problems.Manual review supports segment selection, timestamp and speaker verification, de-identification, and session segmentation.
  • Dataset comparison: Table 1 compares multimodal mental-health support datasets using text, audio, video, sticker, and image modalities and identifies MOCC sessions as counselor-conducted.The table distinguishes real-client and simulated-client sessions according to their source descriptions.
  • Dataset statistics: MOCC retains 202.58 hours from an original 277.36-hour corpus, yielding 482 sessions involving 265 clients and 154 counselors.The retained dataset’s overall statistics are summarized in Table 2, while Figure 2 displays the distribution of client presenting problems.
  • Quality control: Trained annotators verify retained segments, transcript–video alignment, and speaker-role assignments, while a clinical annotation team reviews de-identification and presenting-problem taxonomy decisions.The clinical team includes one clinical psychologist and two trained graduate annotators.

Methodology

MOCC-R1 formulates counselor response generation as an inspectable chain from multimodal context to client state, counseling principle and action, then response. Cold-start SFT teaches this structure, while GRPO combines response-quality rewards with consistency checks for grounded planning and plan execution.

  • Task Formulation: MOCC-R1 generates a structured output linking multimodal dialogue context to client-state understanding, response intent, and counselor response.Response intent consists of a transtheoretical counseling principle and a planned action, exposing the chain x → s → (p, a) → ŷ.
  • Cold-Start Supervision: Cold-start SFT uses verified pseudo-annotations to train the structured decision chain, initializing both the trainable and frozen GRPO policies.The annotations are supervision targets rather than ground-truth traces of private counselor reasoning.
  • GRPO Optimization: GRPO evaluates each sampled tuple with an outcome reward for response quality and a consistency reward for coherence across the generated decision chain.The consistency judge receives neither the reference response nor pseudo-annotations, measuring context-grounded coherence within the generated tuple.
  • Consistency Reward: The consistency reward checks whether the inferred state and plan are grounded and coherent, and whether the response realizes the planned action under the stated counseling principle.Grounded Plan Coherence covers x → s → (p, a), while Plan Execution covers (p, a) → ŷ.
  • Consistency Reward: The chain-level consistency score uses the weaker of grounded plan coherence and plan execution, preventing strength at one stage from offsetting failure at the other.The judge maps Full, Substantial, Weak, and None to 1.0, 0.6, 0.3, and 0.0, respectively.
  • Reward Composition: Final optimization retains response quality as the primary signal and applies consistency as a bounded multiplicative factor rather than an additive bonus.The multiplier lies in [α, 1], preserving the full outcome reward at full consistency and assigning zero final reward when the outcome reward is zero.

Experiments

The experiments compare general-purpose MLLMs and task-specific MCRG systems on MOCC using response-quality and consistency metrics. Structured-output methods are evaluated through grounded plan coherence and plan execution.

  • Baseline Systems: The study compares five proprietary and three open-weight general-purpose MLLMs under an identical three-shot prompt and 128-token output limit.Training-based baselines include ESCoT, M2CoSC, Kardia-R1, and MultiMood.
  • Baseline Systems: The matched comparison isolates the effect of the consistency reward.
  • Evaluation Metrics: Response quality is measured with similarity, perplexity, diversity, empathy-gap, and context-aware safety metrics.All metrics are higher-is-better except perplexity and the three empathy gaps.
  • Evaluation Metrics: Consistency is the percentage of samples rated at least Substantial on both Grounded Plan Coherence and Plan Execution.

Experimental Settings

MOCC-R1 and its baselines use controlled backbone, optimization, and evaluation settings. Under these settings, MOCC-R1 achieves the strongest overall response-quality results and substantially improves reasoning–response consistency.

  • Training Settings: All trainable systems use LoRA with r=32 and α=64, AdamW with weight decay 0.01, and seed 42.SFT and GRPO use separately specified batch, learning-rate, rollout, clipping, and KL settings.
  • Results: General-purpose MLLMs lag on ground-truth-aligned generation metrics, whereas task-specific training markedly improves response quality.
  • Results: MOCC-R1 leads training-based methods in response similarity, empathy alignment, and lexical diversity while maintaining comparable safety.
  • Results: MOCC-R1 reaches 88.17% consistency, reducing inconsistency from 43.28% to 11.82%.Grounded Plan Coherence rises from 84.80% to 95.75%, while Plan Execution rises from 59.92% to 89.00%.

Ablation Studies

Ablations show that outcome-level optimization improves response quality but does not fully align intermediate reasoning with the final response. Adding consistency optimization improves both plan realization and plan-supported response alignment.

  • Ablation Studies: Across response-only and structured-output settings, GRPO improves response quality over SFT, while the consistency reward further improves overall performance.The consistency-aware objective achieves the best balance across the evaluated dimensions.
  • Deep Study of Counseling Reasoning–Response Consistency: MOCC-R1 reduces failure to realize an input-supported plan to 6.03%, from 18.21% with Structured SFT and 19.77% with Outcome-level GRPO.The reductions are 12.18 and 13.74 percentage points, respectively.
  • Deep Study of Counseling Reasoning–Response Consistency: The two failure analyses indicate that MOCC-R1 improves alignment throughout the counseling decision chain.
  • Deep Study of Counseling Reasoning–Response Consistency: MOCC-R1 reduces high-quality responses unsupported by their plans to 11.00%, versus 39.34% for Structured SFT and 40.08% for Outcome-level GRPO.Unsupported cases arise from plans ungrounded in the input or responses that fail to follow grounded plans.

Human Evaluation

A human evaluation compares MOCC-R1 with Outcome-level GRPO on fluency, helpfulness, and reasoning–response consistency. MOCC-R1 obtains substantially more wins than losses across these judgments.

  • Human Evaluation: On 300 randomly sampled test instances, MOCC-R1 receives substantially more wins than losses in consistency and helpfulness while also improving fluency.One clinical psychologist and two graduate students independently made win, tie, or loss comparisons.

Case Study

The case study contrasts consistency failures from baseline systems with MOCC-R1’s grounded reasoning and faithful response execution. Table 7 presents the example, while the consistency evaluation rates the MOCC-R1 chain fully on both dimensions.

  • Case Study: GPT-5.5 rates MOCC-R1 as Full for both grounded plan coherence and plan execution.The evaluation reports Full on each of the two consistency dimensions.
  • Case Study: Table 7 marks generated spans associated with score deductions and highlights consistency dimensions rated Weak or None.The case study is presented on a MOCC test example with visual annotations for problematic generated content and ratings.
  • Case Study: MOCC-R1 remains grounded in the reported context and executes its corresponding reality-testing plan in the response.The baselines infer unsupported concerns and deviate from their stated plans, producing failures in both grounded plan coherence and plan execution.
  • Case Study: The comparison motivates optimizing the entire reasoning-response chain rather than response-level qualities alone.The authors connect the baseline failures to the insufficiency of response-level optimization alone.

Conclusion

The conclusion presents MOCC as a sustained human-recorded counseling resource and MOCC-R1 as a chain-level consistency framework. On MOCC, MOCC-R1 achieves the strongest overall results among evaluated training-based systems and reduces inconsistency relative to Outcome-level GRPO.

  • Conclusion: MOCC contains 482 real- and simulated-client sessions totaling approximately 203 hours and involving 154 credential-verified counselors.The corpus addresses scarce supervision from sustained human-recorded counseling.
  • Conclusion: MOCC-R1 achieves the strongest overall results among the evaluated training-based systems on MOCC.The conclusion reports this as the paper’s overall comparative outcome.
  • Conclusion: MOCC-R1 links multimodal context to client-state understanding, a principle-guided action plan, and the final response.Its training combines verified cold-start supervision with GRPO rewards for grounded plan coherence and plan execution.
  • Conclusion: 43% to 12%: MOCC-R1 reduces inconsistency relative to the matched Outcome-level GRPO baseline.Pairwise human evaluation also favors MOCC-R1 outputs for helpfulness and reasoning–response consistency.
  • Conclusion: The findings establish chain-level consistency as a distinct objective rather than a by-product of response-level optimization.This conclusion follows the reported inconsistency reduction and human-evaluation preference.

Example 1: Reflecting an internal conflict

The example illustrates a cold-start pseudo-annotation workflow for multimodal counseling inputs and a frozen judge that evaluates consistency across evidence, reasoning, and response. The judge separately scores grounded plan coherence and plan execution, then uses the weaker component for the chain-level reward.

  • Example 1: Reflecting an internal conflict: The cold-start example begins with a presenting problem and multimodal dialogue context as inputs to the annotation process.The displayed example uses an emotional-distress presenting problem and a counselor-client dialogue excerpt.
  • Example 1: Reflecting an internal conflict: The frozen consistency judge evaluates internal consistency among the evidence, counseling reasoning, and final response.It is instructed to assess semantic consistency rather than personal preference and not introduce unsupported diagnoses or facts.
  • Example 1: Reflecting an internal conflict: The judge independently assesses grounded plan coherence and plan execution using the presenting problem, multimodal context, and generated state–intent–response chain.The four ordinal labels are Full, Substantial, Weak, and None, mapped to 1.0, 0.6, 0.3, and 0.0.
  • Example 1: Reflecting an internal conflict: The chain-level consistency reward is the minimum of the grounded-plan and plan-execution scores.Neither component can compensate for the other in reward computation.
Loading 2609.17180v1…