PaperlayerView exact revision on arXiv

Source-linked AI summary

Sci-CoE: Co-evolving Scientific Reasoning LLMs via Geometric Consensus with Sparse Supervision

Xiaohan He, Shiyang Feng, Songtao Huang, Lei Bai, Bin Wang, Bo Zhang

arXiv:2602.12164v1cs.AI

TL;DR

Scientific reasoning lacks the clear verification signals that support self-evolution in coding and mathematics. Sci-CoE jointly trains a Solver and Verifier with geometric rewards for consensus, reliability, and diversity, improving reasoning across scientific domains under limited supervision.

  • Problem

    Scientific reasoning lacks clear verification signals because solutions have multiple pathways, heterogeneous criteria, and complex intermediate logic requiring expert assessment.

  • Method

    Sci-CoE jointly optimizes a Solver and Verifier within one LLM using sparse supervision followed by geometric-reward-driven self-iteration on unlabeled data.

  • Results

    Sci-CoE improves reasoning accuracy and robustness across diverse scientific domains while enhancing the reliability and diversity of verification strategies.

  • Takeaways & Limitations

    Sci-CoE supports scientific self-evolution under limited supervision and facilitates more robust, diverse evaluation of scientific questions.

  • Takeaways & Limitations

    On UGPhysics, Sci-CoE does not improve monotonically across all sub-disciplines, likely because training-data distributions mismatch specific physics topics.

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated exceptional reasoning capabilities, and co-evolving paradigms have shown promising results in domains such as code and math. However, in scientific reasoning tasks, these models remain fragile due to unreliable solution evaluation and limited diversity in verification strategies. In this work, we propose Sci-CoE, a two-stage scientific co-evolving framework that enables models to self-evolve as both solver and verifier through a transition from sparse supervision to unsupervised learning. In the first stage, the model uses a small set of annotated data to establish fundamental correctness judgment anchors for the Verifier. In the second stage, we introduce a geometric reward mechanism that jointly considers consensus, reliability, and diversity, driving large-scale self-iteration on unlabeled data. Experiments on several general scientific benchmarks demonstrate that Sci-CoE enhances complex reasoning capabilities and exhibits strong scalability, facilitating the construction of more robust and diverse evaluation systems. Codes are available at https://github.com/InternScience/Sci-CoE.

1. Introduction

Sci-CoE addresses the difficulty of self-evolving scientific reasoning without clear verification signals by jointly training a Solver and Verifier within one LLM. Its geometric reward promotes reliable, diverse verification strategies and supports stable self-evolution under limited supervision.

  • Motivation: Self-evolving reinforcement learning improves LLM reasoning through feedback, but Zero RL still fundamentally depends on annotated datasets for reward calculation.Self-play reduces reliance on external supervision by assigning models interacting roles such as challenger, solver, or verifier.
  • Motivation: Scientific reasoning is harder to self-evolve than coding or mathematics because it rarely provides clear ground-truth or explicit verification signals.Coding and mathematics can use ground-truth solutions or verification methods such as unit tests to assess correctness.
  • Sci-CoE: Sci-CoE integrates a Solver and Verifier within a single LLM, jointly optimizing solution generation and correctness evaluation through interactive reinforcement learning.The framework supports self-evolution without ground-truth solutions or predefined verification procedures.
  • Geometric reward: A geometric reward represents verification strategies in a latent geometric space and encourages reliability and diversity to prevent consensus collapse.The mechanism enables stable unsupervised evolution and sustains self-evolution without ground-truth supervision.
  • Results: Experiments across diverse scientific domains show that Sci-CoE improves reasoning accuracy and robustness while cultivating effective multi-perspective verification behaviors.The framework remains effective without explicit verification signals and achieves strong reasoning performance under limited supervision.

2. Related Work

Related work advances scientific LLMs through generalist architectures and domain-specific adaptations, while self-evolution research has explored iterative generation–verification frameworks, especially for code.

  • Scientific LLMs: Scientific LLM research spans generalist architectures and domain-specific adaptations for complex reasoning tasks.Examples include Intern-S1’s multimodal MoE with Mixture-of-Rewards reinforcement learning and SciReasoner.
  • Scientific LLMs: Domain-specific systems connect scientific modalities and specialized data, covering molecular structures, medical vision-language reasoning, fMRI-to-text decoding, and astronomy.AstroMLab 3 uses high-quality data curation to enable an 8B-parameter model to match GPT-4o performance in astronomy.
  • Self-Evolution LLMs: Early self-evolution LLMs used self-play between generation and verification, with code-focused systems jointly refining implementations and tests or improving coders and unit testers through reinforcement learning.Sol-Ver iteratively refines code implementations and test cases, while CURE uses dynamic interaction to mutually enhance an LLM coder and unit tester.

3. Methodology

Sci-CoE trains a shared Solver–Verifier model in two stages, moving from sparse labeled anchors to fully unsupervised co-evolution on unlabeled scientific questions. Its closed-loop framework uses external judging, PPO, consensus-based solution rewards, and geometric verification rewards to jointly improve solving and evaluation.

  • Framework: Sci-CoE jointly trains a Solver and Verifier with shared parameters, treating solution generation and verification as complementary capabilities in a closed-loop process.The Solver generates explicit reasoning and answers, while the Verifier produces diverse strategies; both are optimized with PPO using a binary verification matrix from an external judge model.
  • Stage 1: Anchored Learning: Training begins with 1%-10% of scientific questions annotated with ground-truth answers to establish stable reference anchors for solving and verification.Solver rewards use exact correctness, while verifier rewards favor strategies that pass correct solutions and reject incorrect ones.
  • Stage 1: Anchored Learning: Because shared-parameter joint optimization may be unstable during anchoring, each PPO iteration sequentially updates the model with solution data and then strategy data.This procedure applies the two objectives in sequence after collecting rollout samples and their rewards.
  • Stage 2: Unsupervised Co-Evolution: Stage 2 scales training to unlabeled scientific questions by replacing absolute correctness with relative agreement and structural consensus, allowing Solver and Verifier to supervise each other.This fully unsupervised mechanism addresses the absence of ground-truth answers through mutual consistency between solutions and verification strategies.
  • Stage 2: Unsupervised Co-Evolution: The Solver reward uses solution pass rates across verification strategies, with τ=0.8 identifying high-consensus solutions, while verifier rewards model consistency, reliability, and diversity geometrically.Reliability is based on proximity to the cluster center in latent representation space, where strategies are embedded as semantic vectors.

4. Experiments

Sci-CoE is evaluated with Qwen base policies across diverse scientific datasets and challenging reasoning benchmarks, consistently improving performance over corresponding models. Experiments further show scalability with unlabeled data, the importance of anchored learning, and benefits from geometric rewards for reliable, diverse verification.

  • Experimental Setup: Sci-CoE uses Qwen2.5-7B-Instruct and Qwen3-8B as Solver–Verifier policies, with Qwen3-235B-A22B providing external verification judgments.Training data span MegaScience, Numinamath, ScienceQA, and CaseHold across mathematics, physics, chemistry, and biology.
  • Benchmarks: Evaluations use MMLU-Pro, GPQA-Diamond, and UGPhysics to test general and domain-specific scientific reasoning across challenging disciplines.The study follows each dataset’s official evaluation scripts for accuracy and comparability.
  • Performance on Scientific Reasoning Benchmarks: 4.04%: Sci-CoE with Qwen3-8B raises GPQA-Diamond accuracy from 36.87 to 40.91, while MMLU-Pro accuracy improves 1.15%, from 63.19 to 64.34.UGPhysics overall accuracy also increases by 1.97% for the 7B base model and 1.34% for the 8B base model, despite non-monotonic sub-discipline changes.
  • Performance and Scalability: Sci-CoE consistently outperforms comparable same-scale baselines on MMLU-Pro and UGPhysics, including Llama-3.1-8B-Instruct, Ministral-8B-Instruct-2410, Mathstral-7B-v0.1, Yi-1.5-9B-Chat, and Mistral-Small-Instruct-2409.Increasing Stage 2 unlabeled data from 18k to 30k yields continuous reasoning improvements without evident saturation.
  • Evolutionary Trends and Co-evolving Iterations: Stage 1 rapidly improves verification strategy accuracy, enabling higher-quality Solver feedback during subsequent unsupervised co-evolution and improving solution, verification, and Best-of-N performance progressively.The Verifier also supplies an internal reward signal that helps select correct reasoning trajectories among multiple candidates at inference time.
  • Ablations and Reward Analysis: Skipping Anchored Learning causes substantial performance degradation, whereas only 0.4k annotated samples improve overall performance and bootstrap Solver and Verifier capabilities.Geometric Reward preserves high reliability and diversity with decent consistency, unlike Naive Reward, which achieves high consistency at severe diversity cost.

5. Conclusion

Sci-CoE improves scientific reasoning under minimal supervision by modeling verification strategies geometrically. Experiments show improved reasoning accuracy and robustness, with effective scaling to large unlabeled datasets.

  • Conclusion: Sci-CoE is a scientific co-evolving framework designed to improve LLMs’ scientific reasoning under minimal supervision.The framework is presented as the paper’s central contribution.
  • Conclusion: Verification strategies form a structured, learnable space whose reliability and diversity can be encouraged through geometric modeling.This geometric perspective is identified as a key insight of the framework.
  • Conclusion: Sci-CoE improves reasoning accuracy and robustness and scales effectively to large unlabeled data.These outcomes are reported in the experimental results.

A. Appendix · A.1. Training Data

The appendix describes the composition of Sci-CoE’s training data at different scales, including the discipline subsets represented in MegaScience.

  • A.1. Training Data: Table 5 presents the training-data composition across different scales.The passage identifies this table as covering training data composition at multiple scales.
  • A.1. Training Data: The table includes a third column labeled “Disciplines.”This column is explicitly identified in the passage.
  • A.1. Training Data: “Disciplines” represents the subset composition of disciplines in MegaScience.The passage defines the meaning of the third column.
  • A.1. Training Data: The appendix frames training data as varying by scale.The table title specifies different scales as the organizing dimension.
  • A.1. Training Data: MegaScience is the dataset whose discipline subset composition is reported.The passage specifically names MegaScience in explaining the discipline column.
  • A.1. Training Data: The reported training-data information concerns both scale and discipline composition.These are the two aspects explicitly identified by the table title and column description.

A.2. Experiment Details

Sci-CoE uses two training stages, with sparse-data anchored learning followed by unsupervised co-evolution at different data scales. Joint Solver–Verifier training uses PPO with KL regularization, while vLLM generates solution and verification rollouts.

  • Sampling Configuration: At each sampling step, vLLM generates rollouts for solutions and verification strategies.Sampling produces Solver and Verifier trajectories from scientific questions.
  • Training Stages: Unsupervised Co-evolution trains for 300 steps in the 18k-scale setting and 500 steps in the 30k-scale setting.The optimization budget increases with the data scale.
  • Optimization: Joint Solver–Verifier training uses Proximal Policy Optimization with one update and one training epoch per update.PPO is used for the joint optimization procedure.
  • Optimization: KL regularization is enabled with coefficient 0.01, using the K3 estimator.Both settings are specified for PPO optimization.

A.3. Prompt

The prompt section defines separate instructions for solution generation, verification-strategy generation, ground-truth judgment, and strategy-based solution judgment. These prompts require structured reasoning while constraining final outputs to specified formats or binary correctness labels.

  • Solution generation: Solution generation instructs the assistant to reason step by step and place the final answer within \\boxed{}.
  • Verification-strategy generation: Verification-strategy generation asks for one high-level strategy rather than a solution, using the problem as input.The strategy is intended to evaluate the correctness and quality of a provided solution.
  • Verification-strategy generation: The verification strategy should be specific, actionable, and discriminating, with checks such as reverse calculations, alternative methods, units, laws, and known principles.
  • Ground-truth judgment: Ground-truth judgment compares a generated answer with a reference solution for mathematical, notational, and physical equivalence, outputting only “True” or “False.”
  • Solution-strategy judgment: Solution-strategy judgment reviews a generated solution according to the supplied verification strategy and likewise outputs only “True” or “False.”The reviewer must follow the strategy’s steps or criteria rather than verify the solution independently.

A.4. Case Study

The case study applies Sci-CoE-7B to a benzene–toluene distillation design problem and illustrates two complementary verification strategies. These strategies check input specifications, theoretical-plate calculations, and feed-plate placement using established distillation methods.

  • Case Study: The example asks for the theoretical plates and feed-plate location needed to separate a 40 mole% benzene feed into specified top and bottom products.The column uses a reflux ratio of 3 kmol/kmol product, with 90 mole% benzene in the top product and ≤10 mole% in the bottom product.
  • Step Check: The Step Check strategy verifies feed composition, product specifications, reflux ratio, plate count, and feed-plate location sequentially.It proposes Fenske-Underwood-Gilliland or McCabe-Thiele for plate counting and the Kirkbride equation or graphical McCabe-Thiele analysis for feed-plate placement.
  • Step Check: The Step Check strategy expects top and bottom compositions of 0.90 and ≤0.10, respectively, with reflux ratio R = 3.It also expects the plate count to fall within the calculated range and the feed location to balance vapor and liquid flow rates.
  • Property Validation: The Property Validation strategy checks whether theoretical-plate calculations and feed-plate placement conform to distillation-tower design principles and operating conditions.It compares plate counts with McCabe-Thiele or balance-equation results and checks feed-plate consistency with feed rate and composition.
  • Property Validation: The second strategy separately verifies the plate count against the reflux ratio and product compositions before checking the feed-plate location.The feed enters at its boiling point, and the expected products remain 90 mole% benzene at the top and ≤10 mole% benzene at the bottom.
Loading 2602.12164v1…