PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning-Driven Multimodal LLM for Domain Generalization

Zhipeng Xu, Zilong Wang, Xinyang Jiang, Dongsheng Li, De Cheng, Nannan Wang

arXiv:2602.23777v1cs.AI

TL;DR

Domain generalization needs models that remain reliable under unseen domains, yet existing benchmarks and methods largely emphasize feature-level invariance rather than explicit reasoning. The paper constructs DomainBed-Reasoning and proposes RD-MLDG, combining direct classification with self-aligned reasoning supervision. Across standard DomainBed benchmarks, RD-MLDG achieves the best reported performance among strong baselines.

  • Problem

    Domain generalization lacks benchmarks that explicitly capture reasoning, while conventional methods mainly evaluate feature-level invariance under domain shift.

  • Method

    RD-MLDG integrates class-relevant reasoning chains with label supervision through MTCT and SARR.

  • Results

    RD-MLDG achieves the best reported performance among strong baselines on DomainBed, including PACS, VLCS, OfficeHome, and TerraInc.

  • Takeaways & Limitations

    Reasoning can serve as a complementary signal for robust out-of-domain generalization when integrated with direct classification and self-aligned supervision.

Abstract

from arXiv · show

This paper addresses the domain generalization (DG) problem in deep learning. While most DG methods focus on enforcing visual feature invariance, we leverage the reasoning capability of multimodal large language models (MLLMs) and explore the potential of constructing reasoning chains that derives image categories to achieve more robust predictions under domain shift. To this end, we systematically study the role of reasoning in DG using DomainBed-Reasoning, a newly constructed extension of DomainBed dataset, in which each sample is paired with class-relevant reasoning chains. Our analysis reveals two key challenges: (i) fine-tuning MLLMs with reasoning chains for classification is more challenging than direct label supervision, since the model must optimize complex reasoning sequences before label prediction; and (ii) mismatches in reasoning patterns between supervision signals and fine-tuned MLLMs lead to a trade-off between semantic richness (informative but harder to optimize) and optimization efficiency (easier to optimize but less informative). To address these issues, we propose RD-MLDG (Reasoning-Driven Multimodal LLM for Domain Generalization), a framework with two components: (i) MTCT (Multi-Task Cross-Training), which introduces an additional direct classification pathway to guide reasoning supervision; and (ii) SARR (Self-Aligned Reasoning Regularization), which preserves the semantic richness of reasoning chains while mitigating reasoning-pattern mismatches via iterative self-labeling. Experiments on standard DomainBed datasets (PACS, VLCS, OfficeHome, TerraInc) demonstrate that RD-MLDG achieves state-of-the-art performances, highlighting reasoning as a promising complementary signal for robust out-of-domain generalization.

1 INTRODUCTION

Domain generalization seeks reliable prediction under unseen domain shifts, but conventional methods mainly target feature-level invariance. This paper introduces reasoning-chain supervision and RD-MLDG to make class-relevant, interpretable reasoning contribute to robust out-of-domain prediction.

  • Domain generalization aims to transfer models trained on source data to unseen domains despite substantial performance degradation under domain shift.
  • Existing DG methods emphasize feature-level invariance, often missing higher-level cross-domain commonalities.
  • DomainBed-Reasoning extends DomainBed by pairing samples with class-relevant reasoning chains to study reasoning under domain shift.
  • Reasoning supervision is harder than direct labels because models must optimize complex intermediate chains before predicting the final class.
  • RD-MLDG combines MTCT’s direct classification pathway with SARR’s self-aligned supervision to integrate informative reasoning while addressing optimization and pattern-mismatch challenges.
  • RD-MLDG achieves state-of-the-art performance on PACS, VLCS, OfficeHome, and TerraInc benchmarks.

2 RELATED WORK

Prior DG methods largely improve representation-level robustness, while MLLMs offer reasoning and cross-modal generalization capabilities that motivate reasoning-based alternatives. However, MLLMs can still underperform specialized vision encoders on fundamental classification benchmarks.

  • Traditional DG approaches use invariant representations, augmentation, meta-learning, adversarial training, and regularization, but primarily operate at the representation level.
  • MLLMs provide strong vision–language reasoning and multimodal generalization capabilities, motivating their use beyond feature-level DG methods.
  • Despite these capabilities, recent studies report that MLLMs underperform vision encoders on fundamental classification benchmarks.

3 DOMAINBED-REASONING

Existing DG datasets lack explicit reasoning annotations, limiting analysis of how reasoning supports generalization. DomainBed-Reasoning addresses this gap with structured, label-independent reasoning chains filtered for coherence.

  • Existing DG datasets contain only input–label pairs, making reasoning under domain shift difficult to evaluate.
  • DomainBed-Reasoning adds structured reasoning chains to samples from PACS, VLCS, OfficeHome, and TerraInc.
  • GPT-4o generates five-stage chains comprising SUMMARY, CAPTION, REASONING, REFLECTION, and CONCLUSION without using ground-truth labels.
  • Rejection sampling retains candidates containing all required components and coherent conclusions, producing a dataset for reasoning and classification evaluation under domain shift.

4 CHALLENGES OF REASONING CHAIN IN DG

The analysis identifies two barriers to reasoning-based DG: reasoning chains are difficult to optimize for classification, and supervision from different reasoning sources follows mismatched patterns. Richer chains are more informative but harder to fit, whereas self-generated chains are easier to optimize but less semantically detailed.

  • The study therefore frames reasoning supervision as a trade-off between semantic richness and optimization efficiency under domain shift.
  • 4.1 OPTIMIZATION GAP IN REASONING-CHAIN SUPERVISION: +43.28%p ground-truth token probability improvement appears when zero-shot InternVL3-8B uses reasoning, but reasoning SFT is 0.93%p less accurate than direct label SFT.
  • 4.1 OPTIMIZATION GAP IN REASONING-CHAIN SUPERVISION: Reasoning-only SFT is less effective because models must fit intermediate reasoning steps before producing the final label, slowing convergence and classification learning.
  • 4.2 MISMATCHES IN REASONING PATTERNS ACROSS SOURCES: +1.88%p token-probability shift follows GPT-4o reasoning fine-tuning, compared with +29.74%p for InternVL3-8B’s self-generated reasoning.
  • 4.2 MISMATCHES IN REASONING PATTERNS ACROSS SOURCES: GPT-4o chains provide richer contextual descriptions but are harder to optimize, while self-generated chains are easier to fit and focus more directly on category cues.

5 METHOD

RD-MLDG formulates domain generalization as learning predictions for unseen target domains without target data, using direct classification and reasoning supervision together. MTCT stabilizes reasoning optimization with a classification pathway, while SARR iteratively replaces external reasoning chains with label-consistent self-generated chains.

  • Overall Framework: RD-MLDG jointly uses direct classification and reasoning-chain supervision to learn predictions that generalize from source domains to unseen targets.The framework integrates reasoning into DG through MTCT and SARR.
  • Multi-Task Cross-Training: MTCT trains reasoning and no-thinking classification prompts together, using direct labels as a stable anchor for fitting intermediate reasoning steps.The reasoning loss is normalized by chain length so longer chains do not dominate gradients.
  • Evaluation: The benchmark evaluates multi-source DG with average target-domain accuracy, while Table 1 reports comparisons across DomainBed datasets.Results are averaged over three random seeds, and the supplied table caption identifies best and second-best performances.
  • Self-Aligned Reasoning Regularization: SARR repeatedly retains self-generated reasoning chains whose final conclusions match ground-truth labels, then fine-tunes on those chains with classification prompts.This iterative procedure is intended to make supervision informative while easier for the model to fit.
  • Analysis: Figure 5 compares token-probability distributions for direct prediction, reasoning-only SFT, and MTCT SFT on TerraInc source-domain training data.The figure separates all-token distributions from class-token distributions.

6 EXPERIMENTS

RD-MLDG is evaluated on four DomainBed benchmarks under leave-one-domain-out testing and achieves the strongest reported overall performance. Ablations show that MTCT and SARR improve reasoning-based classification by stabilizing optimization and aligning supervision with model-generated reasoning.

  • 6.1 RESULTS ON MULTIPLE DOMAIN GENERALIZATION: 86.89% average accuracy makes RD-MLDG the best-performing method, surpassing GPT-4o at 83.46% and DGCLDTP at 83.19%.RD-MLDG also leads VLCS at 87.03%, OfficeHome at 91.73%, and TerraInc at 70.65%, while reaching 98.13% on PACS.
  • 6.2 ABLATION STUDY: MTCT improves the reasoning-only baseline by up to +5.36%p when combined with SARR, supporting direct classification as a stable guide for reasoning optimization.On InternVL3-8B, MTCT adds +1.81%p on OfficeHome and +2.63%p on TerraInc without SARR, then +0.83%p and +5.36%p with SARR.
  • 6.2 ABLATION STUDY: MTCT raises high-confidence class-token probability from 86.33% to 90.23% while reducing low-confidence tokens from 7.59% to 3.19%.The accompanying analysis finds that MTCT strengthens class-token fitting more than the modeling of all semantically rich reasoning details.
  • 6.2 ABLATION STUDY: SARR improves direct reasoning-chain supervision by up to +2.14% on OfficeHome and adds up to +3.46%p when combined with MTCT.The gains indicate that self-labeled reasoning retains class-relevant information while producing signals that are easier for the model to fit.
  • 6.2 ABLATION STUDY: SARR lowers fitting pressure on high-entropy domain-specific tokens and shifts optimization toward class-relevant tokens on TerraInc and OfficeHome.Class-relevant token entropy decreases faster for self-generated reasoning, while GPT-4o chains retain richer but harder-to-fit domain-specific details.
  • 6.2 ABLATION STUDY: 70.06% at N = 1 increases to 70.59% at N = 2 and 70.65% at N = 3, after which accuracy remains stable between 70.50% and 70.60%.The N = 1 to N = 2 improvement is significant, whereas the N = 2 to N = 3 improvement is not; token distributions likewise stabilize for N ≥3.

7 CONCLUSION

The paper concludes that reasoning can complement domain generalization by providing semantically stable signals, while RD-MLDG addresses the optimization and pattern-mismatch challenges through MTCT and SARR. Quantitative validation reports substantially lower cross-domain divergence for reasoning-chain embeddings than for visual embeddings.

  • 7 CONCLUSION: RD-MLDG combines class-relevant reasoning chains with label supervision through MTCT and SARR to address optimization difficulty and reasoning-pattern mismatch.The conclusion reports the best performance among strong DomainBed baselines.
  • 7 CONCLUSION: Reasoning-chain embeddings reduce average TerraInc MMD from 0.239 to 0.099, a 58.6% reduction compared with visual embeddings.The comparison uses identical domain splits and supports the paper’s hypothesis that reasoning chains are less sensitive to style, background, and environmental shifts.

A.2 REJECTION RATE ANALYSIS IN SARR ITERATIONS

SARR progressively filters self-generated reasoning chains whose conclusions disagree with labels, reducing rejection and improving semantic alignment across iterations. The procedure repeatedly generates, validates, retains, and fine-tunes on correctly concluded reasoning chains.

  • A.2 REJECTION RATE ANALYSIS IN SARR ITERATIONS: SARR’s rejection rate drops from 39.51% at N = 0 to 19.23% after the first iteration.The downward trend indicates that later self-generated reasoning increasingly leads to correct conclusions rather than being randomly filtered.
  • A.2 REJECTION RATE ANALYSIS IN SARR ITERATIONS: MTCT first constructs classification and reasoning prompts, computes both losses, and updates the model with their sum before SARR begins.This two-stage procedure supplies the direct classification pathway used to guide reasoning supervision.
  • A.2 REJECTION RATE ANALYSIS IN SARR ITERATIONS: Each SARR round retains generated reasoning only when its extracted conclusion matches the ground-truth label, then fine-tunes on the retained pairs.The algorithm repeats this process for N rounds using a combined classification and reasoning loss.

B RESULTS ON SMALLER OPEN-SOURCE MODEL

RD-MLDG remains effective on smaller open-source multimodal models with weaker reasoning ability. On LLaVA-1.5-7B, MTCT and SARR progressively improve TerraInc accuracy despite the weaker backbone.

  • B RESULTS ON SMALLER OPEN-SOURCE MODEL: LLaVA-1.5-7B improves from 62.07% to 63.92% with MTCT and to 65.15% after adding SARR.InternVL3-2B also shows similar component gains, indicating that the improvements are not restricted to one backbone.

C RESULTS ON SINGLE DOMAIN GENERALIZATION

Single-domain generalization evaluates transfer from one source domain to unseen targets, where limited inter-domain variation makes the setting especially challenging. RD-MLDG achieves a substantially higher OfficeHome average accuracy than prior baselines.

  • Single-domain generalization trains on one source domain and tests on the other three, preventing reliance on inter-domain variation during training.
  • OfficeHome compares RD-MLDG with invariant-representation, augmentation, adversarial, and single-domain adaptation methods.
  • 89.37% average accuracy versus 62.03% for the best baseline PAPT shows RD-MLDG’s large OfficeHome single-domain generalization margin.The comparison uses one source domain and three unseen target domains.

D RESULTS ON BASE-TO-NEW EXPERIMENT

The base-to-new FGVC-Aircraft experiment tests whether reasoning transfers from familiar aircraft categories to semantically and compositionally distinct unseen categories. RD-MLDG substantially improves both harmonic-mean and unseen-class accuracy.

  • FGVC-Aircraft emphasizes structural and compositional attributes, making reasoning important beyond surface-level visual cues.
  • The base-to-new protocol trains only on 50 base classes and evaluates on both those classes and 50 unseen new classes.Performance is reported separately for base and new classes together with harmonic mean H.
  • Harmonic-mean accuracy rises from 19.01% to 34.63%, while unseen-new-class accuracy increases from 15.44% to 24.33%.The evaluation splits 100 aircraft categories into 50 base and 50 new classes.
  • The results indicate that RD-MLDG benefits conceptual generalization rather than only robustness to stylistic or appearance-level variation.

E EXPLORING THE EFFECTIVENESS OF RD-MLDG ON VQA AND VE (VISUAL ENTAILMENT)

RD-MLDG is extended from visual classification to VQA and visual entailment using structured reasoning supervision. Across both tasks, MTCT and SARR improve performance and support the framework’s reported task generality.

  • RD-MLDG is presented as task-agnostic for tasks following a vision → reasoning → output pathway, without changing model architecture or optimization.
  • The method’s reasoning challenges and MTCT/SARR mechanisms are studied within the standardized DomainBed setting and stated to be independent of the downstream task.
  • The VQA and VE extensions augment each sample with SUMMARY, CAPTION, REASONING, REFLECTION, and CONCLUSION components.This format aligns the tasks with DomainBed-Reasoning so both MTCT and SARR can be applied.
  • On VQA, InternVL3-8B improves from 73.60% with reasoning-only supervision to 77.28% with MTCT and SARR; on VE, it reaches 72.09% from 70.59%.The results are reported on the VOLDOGER benchmark across multimodal tasks.
  • After MTCT and SARR, InternVL3-8B surpasses most listed zero-shot multimodal LLMs in average accuracy across four visual styles for both VQA and VE.
Loading 2602.23777v1…