PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs

Xinming Wang, Weinong Wang, Hongming Yang, Yansong Lin, Zheng Ruan, Shangpin Peng, Qiming Peng, Nan Qiao, Fengyuan Lu, Guoqing Ma, Marito Li, Songyang Zhang, Saiyong Yang, Han Hu, Yonglong Tian, Xu-Yao Zhang

arXiv:2608.12781v2cs.CV

TL;DR

Hybrid-thinking MLLMs can vary user-visible response behavior when switching between deliberative and non-thinking inference, so correctness alone is insufficient for evaluating interface consistency. The paper introduces PatternEval to diagnose these failures and PatternRM plus PatternRL to reduce them; experiments show lower non-thinking failures with largely preserved task accuracy, although residual failures and capacity-dependent accuracy costs remain.

  • Problem

    Hybrid-thinking interfaces should preserve acceptable, mode-consistent response patterns because correctness-driven evaluation does not directly constrain leakage, repetition, contradiction, or unsupported response behavior.

  • Method

    PatternEval evaluates four response-pattern failures across multimodal prompts, while PatternRM detects them and PatternRL applies category-specific penalties during reinforcement learning.

  • Results

    Non-thinking inference has substantially higher response-pattern failure rates, while PatternRL reduces non-thinking Trigger by 13.08 and 14.35 percentage points on Qwen3-VL-4B and Qwen3-VL-8B, respectively, with aggregate accuracy changing by less than one percentage point.

  • Takeaways & Limitations

    Explicitly optimizing response patterns provides a practical path toward more reliable hybrid-thinking MLLMs while largely preserving task accuracy.

  • Takeaways & Limitations

    PatternRL does not eliminate response-pattern failures entirely, and its accuracy cost is larger for the 4B model than the 8B model.

Abstract

from arXiv · show

Hybrid-thinking multimodal large language models (MLLMs) allow a single model to alternate between deliberative thinking and latency-efficient non-thinking inference. Although these modes differ in reasoning budget, their delivered responses should satisfy the same user-facing standard. Correctness alone may not characterize this response quality; we therefore evaluate task accuracy and response-pattern failures as complementary outcomes. We study this gap through \textbf{response-pattern alignment}: whether thinking and non-thinking interfaces preserve acceptable final-response behavior. We introduce \textbf{PatternEval}, a failure-enriched diagnostic benchmark comprising 2,415 multimodal prompts spanning visual perception and grounding, structured image understanding, and multimodal knowledge reasoning. PatternEval tests four recurrent failures: chain-of-thought leakage, response repetition, logical contradiction, and performative reasoning. Response-pattern failures are widespread across models from different providers, with non-thinking inference exhibiting substantially higher failure rates and thereby creating systematic misalignment between thinking and non-thinking interfaces. Motivated by this diagnosis, we develop \textbf{PatternRM}, a response-level reward model, and \textbf{PatternRL}, which introduces pattern-specific penalties during reinforcement learning. Experiments on Qwen3-VL-4B and Qwen3-VL-8B show that incorporating pattern-specific penalties into reinforcement learning can mitigate cross-mode misalignment while incurring a marginal task performance trade-off. Together, PatternEval and PatternRL provide an evaluation-and-training framework for aligning user-visible response patterns across hybrid-thinking interfaces.

1 Introduction

Hybrid-thinking MLLMs should preserve acceptable user-facing response patterns across thinking and non-thinking modes, not merely answer correctly. PatternEval exposes widespread failures and mode misalignment, while PatternRL reduces these failures with little aggregate-accuracy change.

  • Hybrid-thinking interfaces make reasoning effort controllable, but final responses should remain grounded, coherent, concise, and free of unintended process narration across modes.
  • PatternEval comprises 2,415 multimodal prompts across three task families and tests four recurrent response-pattern failures.The task families cover visual perception and grounding, OCR and structured-image understanding, and multimodal knowledge reasoning.
  • Non-thinking Trigger rates are consistently higher than thinking rates, with gaps reaching 48.64% even among frontier models.This indicates that strong task performance does not guarantee stable response behavior across inference interfaces.
  • PatternRM recognizes four response-pattern failures, and PatternRL incorporates category-specific penalties into reinforcement learning.The approach translates response-pattern evaluation into an explicit post-training objective.
  • PatternRL reduces response-pattern failures relative to correctness-only BaseRL while preserving aggregate accuracy and improving response usability in broader task training.On Qwen3-VL-4B and Qwen3-VL-8B, non-thinking Trigger falls by 13.08 and 14.35 percentage points, respectively, while aggregate accuracy changes by less than one percentage point.

2 Related Work

Prior work studies how to control explicit reasoning in hybrid-thinking models and how to evaluate multimodal outputs beyond correctness. These directions connect reasoning-mode control with broader assessments of robustness, calibration, truthfulness, safety, instruction following, hallucination, visual consistency, and trustworthiness.

  • Chain-of-thought prompting and self-consistency can improve answer accuracy by eliciting or sampling intermediate reasoning.
  • Reasoning-specialized models make deliberation explicit as test-time computation, while Qwen3 and GLM-4.5 expose thinking and direct-response modes within one model.
  • Research on controlling reasoning computation includes learned routing or mode tokens that decide whether a model should reason.
  • General evaluation benchmarks cover robustness, calibration, truthfulness, safety, and instruction following beyond task correctness.
  • Multimodal evaluation suites additionally assess expert reasoning, hallucination, visual consistency, and trustworthiness.

3 PatternEval

PatternEval is a failure-enriched multimodal stress test that measures task correctness and user-visible response-pattern failures across heterogeneous tasks and inference modes. It operationalizes four failure labels and evaluates them with calibrated verifier and pattern judges.

  • Benchmark Construction and Composition: PatternEval is a selected stress test for conditional robustness, not an estimate of natural task or failure prevalence in deployment.Its construction concentrates difficult prompts that expose the targeted failures while capping high-volume source categories.
  • Benchmark Construction and Composition: PatternEval contains 2,415 prompts across visual perception and grounding, OCR and structured-image understanding, and multimodal knowledge reasoning.The benchmark spans nine constituent categories organized into three task families and is designed to expose targeted response-pattern failures.
  • Evaluation Protocol: The taxonomy covers chain-of-thought leakage, response repetition, logical contradiction, and performative reasoning in final user-visible responses.CoT and repetition primarily capture undesirable form, whereas contradiction and performative reasoning require semantic and multimodal consistency judgments; labels may overlap under a CoT-priority attribution rule.
  • Evaluation Protocol: Evaluation scores each user-facing response for correctness and pattern failures while excluding any separately returned reasoning-channel output.Verifier and pattern judges independently assess outputs, with correctness summarized by Acc and undesirable response patterns by Trigger; mode gaps are reported as ∆acc and ∆pat.
  • Meta-Judge Analysis: CoT leakage and repetition are judged more reliably than contradiction and performative reasoning across candidate judges.The latter labels require finer-grained semantic assessment involving cross-sentence consistency, visual grounding, and evidence-supported analysis.

4 Experiments

Matched evaluation across 25 model configurations shows persistent thinking–non-thinking response-pattern gaps, with non-thinking generally more failure-prone. Further analyses link failures to model capability, task category, response length, correctness, and inference mode.

  • Main Results: Every evaluated model pair has a positive response-pattern gap, and 17 of 25 pairs exceed 20 percentage points.The gap persists across model families, architectures, and parameter scales.
  • Main Results: CoT leakage is the most prominent failure mode and the dominant source of bad-pattern triggers, especially under non-thinking inference.The operational label concerns observable response form, not private internal computation.
  • Main Results: 12.75% CoT leakage and 8.49% response repetition exceed 4.72% performative reasoning and 2.74% logical contradiction across model–mode rows.Because CoT-priority attribution handles co-occurring failures, these marginal rates are not independent prevalence estimates.
  • Model capability: From 4B to 397B-A17B, non-thinking PatternEval Acc rises from 45.96% to 61.96%, yet the cross-mode Trigger gap remains 21.36–26.63 percentage points.Thinking-mode Trigger falls from 22.19% to 4.23%, while larger-model non-thinking Trigger remains between 28.45% and 33.00%.
  • Failure-prone constituent categories: Reasoning, STEM, and OOD perception show some of the largest non-thinking Trigger peaks, whereas content recognition and chart understanding generally show lower rates.Category ordering varies substantially across models, so similar aggregate Trigger can conceal different localized weaknesses.
  • Response length, correctness, and Trigger: Response length is positively associated with Trigger, with Pearson correlations of r = 0.64 in non-thinking and r = 0.84 in thinking.The relationship is descriptive rather than causal, and incorrect aggregates are generally longer and more failure-prone than correct aggregates.
  • Response length, correctness, and Trigger: In the longest sextile, Trigger reaches approximately 86% and 56% for incorrect and correct non-thinking responses, versus 52% and 22% under thinking.Across length bins, non-thinking remains more failure-prone, and long correct non-thinking responses demonstrate that correctness alone is insufficient.

5 Pattern-Aware Post-Training

PatternRM and PatternRL add response-pattern penalties to correctness-oriented post-training, targeting user-visible failures while retaining the primary verifier objective. PatternRL reduces non-thinking failures but introduces a capacity-dependent accuracy trade-off.

  • PatternRM: PatternRM is trained on consensus labels for chain-of-thought leakage, repetition, contradiction, and performative reasoning.The supervision corpus yields 57,578 SFT instances after filtering and oversampling.
  • PatternRM evaluation: Direct-prediction PatternRM attains 71.3% macro-F1 versus 70.2% for its thinking counterpart while avoiding additional decoding.The comparison concerns predictive performance and decoding cost rather than measured end-to-end latency.
  • Reward design: PatternRL combines verifier rewards for answer correctness with PatternRM penalties for four response-pattern failures.Incorrect responses receive zero reward, while correct responses receive scores between 0.9 and 1.
  • PatternEval results: PatternRL substantially reduces overall non-thinking Trigger rates across most failure types and task families while preserving comparable accuracy on PatternEval.The method mitigates degradation associated with correctness-only reinforcement learning.
  • Accuracy trade-off: PatternRL weakens aggregate task accuracy relative to BaseRL, with a larger decline for Qwen3-VL-4B than Qwen3-VL-8B.The larger model largely preserves performance, with limited declines in math and logic and a slight improvement in document understanding.
  • Limitations: Residual response-pattern failures indicate that a lightweight reinforcement-learning reward does not fully correct behaviors potentially embedded during earlier training stages.The authors suggest higher-quality supervision and earlier pattern constraints during midtraining or supervised fine-tuning.

6 Conclusion

The paper studies response-pattern alignment in hybrid-thinking MLLMs and finds a consistent gap between thinking and non-thinking interfaces. PatternEval, PatternRM, and PatternRL provide an evaluation-and-training approach that reduces non-thinking failures while largely preserving task accuracy.

  • Conclusion: PatternEval evaluates four user-visible response failures across matched thinking and non-thinking interfaces.The conclusion frames response-pattern alignment as complementary to answer correctness.
  • Conclusion: Non-thinking inference produces substantially more response-pattern failures, including in frontier models.This is described as a consistent mode-dependent gap.
  • Conclusion: PatternRM and PatternRL reduce non-thinking failures while largely preserving task accuracy.The framework targets response behavior without treating controllable reasoning effort as sufficient by itself.

B.1 Category-wise Results

Category-level results show positive cross-mode Trigger gaps across multiple PatternEval categories rather than a single task family. Scale does not uniformly reduce Trigger values, supporting category-specific analysis.

  • Category-wise results: Positive cross-mode Trigger gaps occur across multiple constituent task categories, not only one task family.Some of the largest descriptive differences appear in OOD perception and knowledge-intensive reasoning.
  • Category-wise results: Content recognition and chart understanding often show smaller cross-mode gaps than OOD perception and knowledge-intensive reasoning.These category-level point estimates characterize PatternEval but do not identify the cause of mode differences.
  • Category-wise results: Scale does not uniformly reduce every constituent-category Trigger value across model families.Thinking-mode Trigger sometimes declines with scale, while non-thinking values remain elevated or vary non-monotonically in OOD perception, STEM, and general reasoning.

C Meta-Judge Analysis

The calibration pipeline separates candidate discovery from benchmark scoring by using independent annotations, consensus and disputed strata, and controlled sampling. This procedure supports selection of an operational judge for response-pattern evaluation.

  • Calibration pipeline: Three judges independently annotate four PatternEval failure labels, with complete agreement defining consensus and disagreement defining disputed samples.The two strata are sampled at an approximate 7:3 ratio.
  • Calibration pipeline: The selected 2,500 fixed responses are used after controlling the predicted positive rate near 70%.The calibration pipeline separates this candidate-discovery process from benchmark scoring.

C.2 Failure-Label Calibration Behavior

Judge performance varies by response-pattern category: leakage and repetition are easier to detect than contradiction and performative reasoning across all nine configurations.

  • CoT leakage F1 ranges from 90.2% to 95.3%, while repetition F1 ranges from 81.6% to 88.2%.
  • Contradiction F1 ranges from 56.1% to 62.5%, and performative reasoning F1 ranges from 41.8% to 64.5%.
  • GPT-5.5 achieves the highest aggregate F1 overall across the nine judge–input configurations.
  • Image access improves aggregate F1 for Seed-2.0-Pro and Kimi-K2.6, while Seed-2.0-Pro with image access attains the highest contradiction result.

D PatternRL Training

PatternRL uses PatternRM’s four category decisions as weighted auxiliary penalties during reinforcement learning, with stochastic application and bounded reward fusion.

  • PatternRM converts four response-pattern decisions into a category-weighted auxiliary reward for PatternRL.
  • Logical contradiction and performative reasoning each receive weight 0.02, whereas CoT leakage and repetition each receive weight 0.05.
  • PatternRM is independently invoked with probability 0.6 for each rollout; otherwise its auxiliary contribution is zero.
  • Simultaneous penalties accumulate to an auxiliary-reward floor of −0.1, and the fused scalar reward is clipped to [0, 1].
  • The reported PatternRL configuration corresponds to a single run because multi-seed training manifests are unavailable.

D.3 Training Parameters

The training-parameter section documents PatternRL’s configuration and the meta-judge’s operational labeling procedure for four response-pattern failures.

  • D.3 Training Parameters: Table 8 summarizes PatternRL’s optimization hyperparameters, rollout settings, sequence-length budgets, reward configuration, and distributed-training setup.
  • Meta-judge prompt: The meta-judge receives the user question, associated image, and model response, then independently assigns four response-pattern labels.
  • Meta-judge prompt: Judgments must rely on observable response text, use images for image-related verification, and avoid treating ordinary factual or visual errors as response patterns.
  • Decision order: The fixed decision order evaluates leakage first, repetition second, and contradiction and performative reasoning afterward.
  • Chain-of-Thought Leakage: Chain-of-thought leakage requires explicit exposed internal reasoning, while polished user-facing analysis without such traces is not leakage.
  • Response Repetition: Repetition is marked when unnecessary repeated content materially harms information efficiency, readability, or answer quality.
  • Performative Reasoning: Performative reasoning is analysis-like content that supplies no valid information supporting the final conclusion, including generic image-based claims without concrete visible evidence.
  • Logical Contradiction: Logical contradiction requires two mutually incompatible claims about the same object, quantity, or conclusion that remain jointly presented as valid.
Loading 2608.12781v2…