PaperlayerView exact revision on arXiv

Source-linked AI summary

Turning Drift into Constraint: Robust Reasoning Alignment in Non-Stationary Multi-Stream Environments

Xiaoyu Yang, En Yu, Wei Duan, Jie Lu

arXiv:2510.04142v3cs.CVcs.AIcs.LG

TL;DR

Multi-source reasoning alignment must handle drifting and conflicting source trajectories rather than passively aggregate them. APO recasts drift as negative constraints, and experiments report superior robustness and generalization using only 10% of the data typically required by standard alignment methods.

  • Problem

    Multi-source reasoning alignment lacks a robust way to handle non-stationary, divergent, and conflicting reasoning trajectories from different models.

  • Method

    APO formulates multi-source alignment as constraint satisfaction, using consensus as positive signals and drifting conflicts as negative constraints for label-free optimization.

  • Results

    APO achieves superior robustness and generalization while using only 10% of the data typically required by standard alignment methods.

  • Takeaways & Limitations

    The study establishes a principled path for autonomous label-free model evolution by synthesizing robust consensus from diverse sources.

  • Takeaways & Limitations

    The authors identify improving efficiency and reducing computational cost in large-scale multimodal settings as future work.

Abstract

from arXiv · show

This paper identifies a critical yet underexplored challenge in reasoning alignment from multiple multi-modal large language models (MLLMs): In non-stationary environments, the diverse reasoning distributions of source models often evolve unpredictably, transmitting systematic biases and drift to the target model. To address this, we formulate multi-source reasoning alignment as a constraint satisfaction problem under concept drift theory. We propose Autonomous Preference Optimization (APO), a novel framework that treats inter-model divergences not as noise, but as dynamic negative constraints. APO operates via a two-stage protocol: first, supervised bootstrapping projects the target model into the capability union of source models; second, constraint-aware optimization synthesizes a consistent consensus manifold by explicitly suppressing drifting trajectories via a multi-negative Plackett-Luce objective. Extensive experiments on chest X-ray interpretation demonstrate that our 7B model achieves superior robustness, outperforming even proprietary source models in average accuracy. Furthermore, we release CXR-MAX, a large-scale benchmark comprising 170,982 reasoning trajectories from seven large-scale MLLMs to facilitate research on reasoning alignment under drift. Code and data are available at: https://github.com/XiaoyuYoung/APO.

1. Introduction

The introduction frames multi-source reasoning alignment as a non-stationary concept-drift problem: source diversity can improve reasoning while transmitting divergent biases. It proposes treating disagreement as negative constraints and introduces APO and CXR-MAX to pursue robust alignment research.

  • Motivation: Source models exhibit inter-model drift because differing pre-training biases and architectures produce divergent reasoning distributions.The paper analyzes these dynamics through concept drift theory.
  • Empirical observations: Qwen-VL-Max favors high-precision, concise reasoning, whereas GPT-4o favors high-recall, expansive elaboration, placing the true reasoning manifold within their consensus.The observation presents complementary divergence rather than a single authoritative reasoning trajectory.
  • Empirical observations: Naive imitation of drifting source streams makes the target model internalize their combined biases, producing hallucinations and semantic inconsistencies.The paper identifies this as distributional corruption rather than synthesis of source strengths.
  • Proposed perspective: APO reframes multi-source reasoning integration as constraint satisfaction, using source consensus as positive signals and disagreements as actionable negative constraints.This changes alignment from passive imitation to learning both what to follow and what to avoid.
  • Contributions: 10% of the data typically required by standard alignment methods is sufficient for APO to achieve superior robustness and generalization while mitigating individual-source drift.The result is reported across diverse benchmarks.
  • Contributions: CXR-MAX comprises over 170k reasoning trajectories with fine-grained alignment annotations for studying inter-model dynamics and reasoning consistency in high-stakes domains.The benchmark is released as a testbed for future alignment-under-drift research.

2. Methodology

The methodology frames multi-source reasoning alignment as constraint satisfaction under non-stationary concept drift and implements it through supervised bootstrapping followed by constraint-aware APO. APO synthesizes consensus preferences while suppressing multiple drifting source trajectories, supported by the CXR-MAX benchmark.

  • Two-Stage Protocol: APO uses two stages: supervised bootstrapping projects the target into the sources’ collective capability space, then constraint-aware optimization extracts a consistent trajectory.Bootstrapping assimilates heterogeneous source knowledge into a foundational feature space rather than following a single source.
  • Theoretical Formulation: Multi-source reasoning alignment is modeled as concept drift because joint distributions over source reasoning states evolve non-stationarily across reasoning steps.Source models may converge on an inference result at one step yet diverge substantially in their rationales later, creating an evolving supervision signal.
  • Constraint-Aware Optimization: APO treats synthesized consensus as the preferred output and source-generated drifting trajectories as multiple negative constraints in a Plackett-Luce preference objective.This generalizes pairwise Bradley-Terry ranking to simultaneous comparison against an ensemble of drifting outputs.
  • Constraint-Aware Optimization: Minimizing the APO loss increases consensus likelihood relative to the bootstrapped reference while decreasing likelihood for specific drifting source patterns.The resulting objective converts inter-model drift into supervision by suppressing probability mass in drifting regions of the reasoning space.
  • Benchmark: CXR-MAX aggregates 170,982 reasoning-trajectory distillation instances from seven publicly available MLLMs across 14 thoracic pathologies for multi-source chest X-ray alignment.The benchmark is designed to capture dynamic inter-model conflicts that single-source or static-consensus benchmarks miss.

3. Experiments

Experiments evaluate APO under limited-data, non-stationary multi-stream conditions using a Qwen2.5-VL (7B) target trained only on MIMIC-CXR classification labels. Across disease classification, proprietary-model comparisons, reasoning metrics, zero-shot transfer, and ablations, APO achieves strong accuracy, consistency, and robustness to drift.

  • Experimental setup: APO uses only 1/10 of MIMIC-CXR and excludes original radiology reports, emphasizing knowledge transfer from drifting teachers under limited-data conditions.The target is Qwen2.5-VL (7B), trained for one epoch in each of supervised bootstrapping and autonomous preference optimization with batch size 2.
  • Disease classification: 0.78 average performance establishes a new state-of-the-art on MS-CXR-T, nearly 9% above second-best CoCa-CXR despite using only 10% of the data.Baseline methods use the full training set with radiologist reports, whereas APO relies solely on reasoning alignment from drifting source models.
  • Disease classification: 0.96 on pneumothorax and 0.84 on consolidation surpass the runner-up by 0.23 and 0.14, respectively, supporting APO’s constraint-aware optimization.The paper attributes these gains to suppressing drifting uncertainties and reinforcing consensus around critical visual cues.
  • Robustness to source-model drift: 0.78 average accuracy surpasses every evaluated proprietary source MLLM, including GPT-5 and Claude Sonnet-4, despite the target’s smaller parameter scale.Across individual pathologies, the target generally provides stable second-best performance, including regimes where source-model accuracy gaps exceed 0.60.
  • Reasoning consistency: 0.19 BLEU-4 and 0.21 METEOR indicate improved reasoning consistency and lexical alignment over state-of-the-art methods.These results are presented as evidence that APO internalizes medical logic rather than merely mimicking teacher outputs.
  • Generalization and alignment comparisons: 0.85 AUC on Open-I and 0.83 on ChestXray14 demonstrate zero-shot cross-domain adaptability, while APO reaches 0.78 average on MS-CXR-T against DistiLLM-2 and ABKD.The paper links these gains to filtering inconsistent signals and maintaining transferable reasoning from report generation to fine-grained disease classification.
  • Ablation: APO delivers significant accuracy gains across all diseases over MT and SPD, whereas MT produces marginal overall gains and deterioration on most diseases.The ablation attributes APO’s advantage to blocking concept-drift transmission while constructively learning from all source models.

4. Conclusions and Limitations

The paper introduces APO as a constraint-satisfaction approach that treats inter-model drift as dynamic negative constraints for robust reasoning alignment. It also identifies domain-specific bias and computational efficiency in large-scale multimodal settings as areas for future work.

  • Conclusions: APO formalizes inter-model drift as dynamic negative constraints, suppressing drift and synthesizing a robust consensus manifold from diverse sources.This reframes alignment as a constraint satisfaction problem and supports autonomous label-free model evolution.
  • Limitations and Future Work: Future work will address domain-specific biases and improve efficiency while reducing computational cost in large-scale multimodal settings.

Impact Statement

The work follows the ICML Code of Ethics and reports no human or animal experimentation. Its datasets were sourced under relevant usage guidelines, with no personally identifiable information or privacy and security concerns reported.

  • Ethics and privacy: The study involved no human subjects or animal experimentation and adheres to the ICML Code of Ethics.The authors state that all datasets, including CXR-MAX, were used in compliance with relevant requirements.
  • Ethics and privacy: The research used datasets sourced under relevant usage guidelines, avoided discriminatory outcomes, and used no personally identifiable information.The authors report no experiments raising privacy or security concerns and commit to transparency and integrity.

A. Related Works · A.1. Concept Drift

The related-work discussion frames concept-drift mitigation through three major methodological families and situates this study in distribution-oriented approaches. It also highlights OBAL as a two-stage multistream adaptation pipeline.

  • A. Related Works: Lu et al. organize concept-drift mitigation into error rate–driven adaptation, distribution-aware approaches, and multi-hypothesis frameworks.
  • A. Related Works: Error rate–driven adaptation is identified as one dominant family in concept-drift mitigation.
  • A. Related Works: Distribution-aware approaches constitute a second dominant family in the surveyed concept-drift literature.
  • A. Related Works: Multi-hypothesis frameworks form the third dominant family described by the survey.
  • A.1. Concept Drift: This study is situated within the distribution-oriented stream of concept-drift research.
  • A.1. Concept Drift: OBAL addresses multistream classification with a two-stage pipeline for increasingly complex learning environments.
  • A.1. Concept Drift: OBAL first uses AdaCOSA to capture evolving inter-stream correlations.
  • A.1. Concept Drift: OBAL then applies Gaussian Mixture Model–driven weighting to counter asynchr

A.2. Reasoning Alignment for LLMs and MLLMs

Reasoning alignment has shifted from compressing representations to synchronizing robust reasoning capabilities across LLMs and MLLMs. Multi-source alignment remains difficult because drifting sources provide biased or conflicting supervision, motivating APO’s constraint-satisfaction formulation.

  • Reasoning alignment now transfers reasoning processes alongside capabilities such as robustness, safety, and logical behavior, rather than only probability distributions.
  • Multi-modal alignment bridges visual perception and textual reasoning by synchronizing visual encoders with large language decoders.Contrastive grounding and generative frameworks established this direction.
  • Multi-source reasoning alignment seeks to synthesize diverse capabilities from heterogeneous source models, extending earlier computer-vision work to MLLMs.Recent examples address out-of-context news detection and streaming-data continual learning.
  • Existing methods largely assume stationary sources, but distribution shifts expose concept drift that produces biased or conflicting supervision and dynamic inconsistencies.These limitations motivate APO, which frames multi-source alignment as constraint satisfaction for robust consensus synthesis.

A.3. Reinforced Fine-tuning in LLMs

Reinforcement learning has become central to post-training alignment in LLMs, beginning with RLHF and expanding toward diverse preference-optimization and self-training paradigms.

  • Reinforcement Learning Foundations: RLHF introduced a paradigm for aligning LLM behavior with human values, establishing reinforcement learning’s role in post-training alignment.The passage credits Christiano et al. (2017) with pioneering RLHF and cites Ouyang et al. (2022) for alignment with human values.
  • Reinforcement Learning Foundations: OpenAI-o1 demonstrated the practical utility of preference-driven modeling in reinforcement-learning-based alignment.The passage identifies OpenAI-o1 as an initial implementation of preference-driven modeling.
  • Beyond Classical RLHF: ReST advances iterative self-training by generating policy-driven samples and refining them through offline RL.ReST represents a paradigm extending beyond classical RLHF.
  • Beyond Classical RLHF: DPO reconceptualizes alignment as direct preference optimization through implicit reward modeling.DPO is presented as a complementary paradigm that extends beyond the classical RLHF formulation.

B. CXR-MAX Dataset

CXR-MAX presents image–radiology-report samples and the prompting setup used to collect reasoning trajectories from multiple MLLMs. The examples show models grounding atelectasis and pleural-effusion judgments in radiographic signs such as volume loss, mediastinal shift, costophrenic-angle blunting, and meniscus signs.

  • Dataset construction: CXR-MAX samples pair chest DR images with ground-truth radiology reports and model-generated reasoning for training and validation.The section explicitly showcases the image, report ground truth, and generated samples used in the study.
  • Dataset construction: The collection uses a shared prompt asking MLLMs to identify the basis for diagnoses in a patient’s chest DR image.The prompt begins by describing a patient chest DR image and supplying diagnosed diseases for explanation.
  • Reasoning examples: Additional reasoning references devices and clinical uncertainty, including pacemakers, possible compressive atelectasis, and recommendations to correlate clinically or pursue follow-up imaging.The examples mention pacemaker visibility, compression by effusion, and conditional follow-up with radiographs, ultrasound, or CT.

C. Implementation Details

The implementation uses Qwen2.5-VL (7B) with AdamW-based supervised pre-distillation, cosine learning-rate decay, and specified regularization settings. APO then uses a lower learning rate without warmup while freezing the visual encoder and text decoder.

  • Qwen2.5-VL (7B) serves as the pre-trained model for supervised pre-distillation.
  • 10,686 training steps use AdamW with learning rate 1e-4, β = (0.9, 0.98), weight decay 0.05, dropout 0.1, and batch size 2.The schedule warms up for 20 steps before cosine decay to 10−7.
  • APO uses learning rate 2×10−5 without warmup and retains batch size 2.
  • APO freezes the visual encoder and text decoder, runs for 12,132 steps on 2 × 2 NVIDIA A100 GPUs, and otherwise retains the fine-tuning parameters.
Loading 2510.04142v3…