PaperlayerView exact revision on arXiv

Source-linked AI summary

Training a Scientific Reasoning Model for Chemistry

Siddharth M. Narayanan, James D. Braza, Ryan-Rhys Griffiths, Albert Bou, Geemi Wellawatte, Mayk Caldas Ramos, Ludovico Mitchener, Samuel G. Rodriques, Andrew D. White

arXiv:2506.17238v2cs.LG

TL;DR

The paper investigates whether language-model reasoning can extend beyond mathematics and programming to chemistry, especially difficult molecular-design problems. It post-trains ether0 with distillation and reinforcement learning on experimentally grounded chemistry tasks, finding that it outperforms frontier LLMs, experts, and specialized models, particularly on open-answer tasks. The model is data efficient but remains limited in distributional coverage, general instruction following, and tool use.

  • Problem

    The paper addresses whether reasoning models can generalize to chemical reasoning and molecular design beyond the mathematics and programming domains where they were previously successful.

  • Method

    ether0 is a 24B-parameter model trained with reinforcement learning, reasoning distillation, and curriculum optimization on experimentally grounded chemistry problems represented with molecular structures.

  • Results

    ether0 significantly outperforms frontier LLMs, domain experts, and specialized models on held-out chemical reasoning tasks, particularly open-answer tasks.

  • Takeaways & Limitations

    Reasoning models can be post-trained for chemistry without additional domain pretraining and can support molecular design across drug-discovery stages.

  • Takeaways & Limitations

    ether0 may generalize poorly beyond its organic-SMILES training distribution, loses some general instruction-following and chat abilities, and was not trained for tool calling.

Abstract

from arXiv · show

Reasoning models are large language models that emit a long chain-of-thought before answering, providing both higher accuracy and explicit reasoning for their response. A major question has been whether language model reasoning generalizes beyond mathematics, programming, and logic, where most previous work has focused. We demonstrate that reasoning models can be post-trained for chemistry without additional domain pretraining, and require substantially less data compared to contemporary domain-specific models. We report ether0, a 24B parameter LLM (based on Mistral-Small-24B) that can reason in natural language and respond with chemical structures. This reasoning model was trained with reinforcement learning on 640,730 experimentally-grounded chemistry problems across 375 tasks ranging from synthesizability, to blood-brain barrier permeability, to human receptor activity, to scent. Our model exceeds general-purpose chemistry models, frontier models, and human experts on molecular design tasks. It is also more data efficient relative to specialized models. We anticipate that this method can be applied to train data-efficient language models specialized for tasks across a wide variety of scientific domains.

1 Introduction

The paper asks whether reasoning models can extend beyond mathematics and programming to chemistry, where inverse molecular-design problems are difficult despite verifiable solution quality. It introduces ether0, which reasons in natural language, outputs SMILES structures, and is trained with optimized reinforcement learning and distillation.

  • Motivation: Scientific inverse problems often permit solution assessment but make solution generation difficult, as in designing molecules with desired solubility.This motivates applying reasoning models to chemistry and other physical-science domains.
  • Motivation: Chemistry provides a suitable testbed because molecules have compact text representations in SMILES, avoiding modality-specific encoders.The paper also emphasizes the importance of molecular generation and editing for clinical and commercial applications.
  • Contribution: ether0 reasons in natural language and outputs molecular structures as SMILES strings for chemical reasoning tasks.The model targets molecular design, completion, modification, and synthesis.
  • Contribution: ether0 outperforms frontier LLMs, human experts, and general chemistry models on the chemical reasoning tasks considered.The reported comparison covers the paper’s evaluated chemical tasks rather than all chemistry benchmarks.
  • Training approach: The training approach combines reasoning distillation, a dynamic curriculum, and expert-model initialization to improve reinforcement-learning efficiency.The authors also analyze data efficiency, failure modes, and reasoning behavior.

2 Chemical Reasoning Tasks

The chemical-reasoning dataset contains experimentally grounded molecule and reaction problems represented as SMILES, spanning diverse molecular design and property tasks with verifiable rewards.

  • Dataset: 640,730 chemical reasoning problems cover 18 tasks, with questions and answers expressed as molecules or reactions in SMILES.Tasks include subtasks such as changing solubility while preserving a scaffold or functional groups.
  • Dataset: The dataset uses synthesized molecules and bases questions and answers on physical experiments rather than hypothetical structures.Some tasks additionally check whether proposed molecules are plausibly synthesizable.
  • Task types: Tasks include solubility editing, IUPAC-to-SMILES conversion, SMILES completion, molecular-formula design, functional-group design, and structural elucidation.Verification uses tools such as RDKit, exmol, KDESol, and similarity thresholds where appropriate.
  • Dataset organization: Table 1 organizes verifiable reward tasks by category, verifier type, templates, and data source.MCQ denotes multiple-choice questions, while dagger-marked tasks also perform a reasonable-molecule check.

3 Background

The paper uses supervised fine-tuning to initialize reinforcement learning, then applies Group Relative Policy Optimization using sampled completion groups, rewards, and clipped policy updates.

  • Initialization: Supervised fine-tuning initializes the reinforcement-learning policy and can be viewed as expert iteration or knowledge distillation when demonstrations come from another policy.The demonstration dataset is denoted D_demo and the source policy π′.
  • GRPO: GRPO samples G completions for each question, assigns rewards, and uses their relative performance to form advantages.The sampled completions are y_1, ..., y_G and their rewards are r_1, ..., r_G.
  • GRPO: The per-group objective optimizes the current policy against rollout probabilities while incorporating a reference-policy KL penalty.The objective is defined for a single problem and a group of completions.
  • Optimization: The clipped update uses the standard PPO clip function to limit policy-ratio changes while weighting updates by advantages.The clip function combines the ratio, advantage, and ε clipping parameter.
  • Optimization: The global GRPO objective averages the per-group objective over the training set, with Algorithm 1 providing the full procedure.This connects individual sampled groups to optimization across dataset D.

4 Training

Training alternates supervised distillation and GRPO through task-specific specialists, distillation into a generalist, and all-task reinforcement learning, with curricula and problem rewriting supporting optimization.

  • Training pipeline: The four-stage pipeline consists of long-chain-of-thought SFT, task-specific specialist GRPO, specialist-to-generalist distillation, and generalist GRPO.The procedure alternates distillation and GRPO phases.
  • Model setup: The model is based on Mistral-Small-24B-Instruct-2501 and uses special tokens to separate reasoning from answers.The intended sequence contains thought and answer boundaries around the natural-language reasoning and molecular output.
  • Initialization: SFT warm-starts reinforcement learning with rejection-sampled reasoning traces whose answers are valid SMILES or SMIRKS and pass reasoning-quality checks.The traces are generated from DeepSeek-R1 and filtered for formatting, chemical validity, and relevant reasoning.
  • Initialization: 14,021 shorter demonstration traces were produced after long reasoning sequences proved inefficient because sampling dominated training time without improving accuracy.The traces summarize R1-generated reasoning before extracting thought and answer components.
  • Specialist RL: Specialist GRPO groups related chemistry tasks into specialists, while the final system combines them through distillation and all-task GRPO.The training design yields seven total specialists and uses binary format and accuracy rewards, with one softer specialist exception.
  • Curriculum: Trivial reward groups can reach 90% of a batch and contribute no policy gradient, motivating a curriculum buffer of recently non-trivial problems.The buffer reduces the trivial-group fraction without additional computational cost, subject to a bound on curriculum sampling.
  • Robustness: Problem rewriting varies phrasing and can add distracting information while retaining relevant content to encourage generalization to unseen formulations.Two rewriting prompts are used in equal proportion.
  • Generalist RL: The combined all-task GRPO phase uses an online curriculum, binary accuracy rewards, a final molecule-quality bonus, problem rewriting, and safety alignment.This phase follows specialist distillation into a single generalist model.

5 Results

ether0 matches or exceeds specialist-model performance after all-task reinforcement learning and outperforms general-purpose, chemistry-specific, and frontier models on key evaluations. Results also show substantial data efficiency, inference-time gains from one-shot prompting, and selective benefits from explicit reasoning.

  • 5.1 Model Performance: Final all-task reinforcement learning recovered distillation-related performance drops, matching or exceeding the corresponding specialist models across tasks.Task-specific RL improved all tasks, while distillation caused declines in some categories before the final RL phase restored performance.
  • 5.1 Model Performance: ether0 achieved the highest accuracy on all open-answer categories and competitive performance on multiple-choice questions against general-purpose and chemistry-specific models.The authors attribute stronger open-answer margins partly to more training problems and the non-uniqueness of many answers, which permits exploration without memorization.
  • 5.1 Model Performance: 80% of unsafe questions were refused without meaningful degradation on the measured capability tasks after safety alignment.The safety procedure was evaluated separately, and an annotated model response is provided in Figure 4.
  • 5.2 Data Efficiency: 70% accuracy was reached after 46,000 reaction examples, versus 64.1% for Molecular Transformer trained on the full USPTO dataset.Molecular Transformer retrained on the smaller dataset failed to exceed 30% accuracy, a threshold ether0 surpassed after seeing 10% of the available training data.
  • 5.2 Data Efficiency: 50.1% zero-shot accuracy rose above all evaluated frontier models with one-shot prompting, despite matching only 52.2% for o1-2024-12-17 in the zero-shot comparison.The one-shot setup supplied a distractor as a labeled example while preserving the same chance baseline by removing that distractor from the question.
  • 5.3 Reasoning Performance and Behavior: Explicit reasoning consistently outperformed non-reasoning training, while cognitive behaviors emerged selectively on tasks where structured reasoning was advantageous.Functional-group and molecule-formula tasks showed increased behavior counts and completion lengths, whereas solubility editing and SMILES completion showed no clear reasoning benefit.

6 Limitations

ether0 may struggle outside its training distribution and loses general instruction-following and chat capabilities after intensive reinforcement learning. Its training also omitted tool calling, which limits direct use in tool-heavy small-molecule workflows.

  • 6 Limitations: ether0 may generalize poorly to inorganic chemistry because training primarily used SMILES strings for organic molecules.The authors specifically mention crystal-structure generation as an expected weak area.
  • 6 Limitations: Intensive reinforcement learning reduced the base model’s general instruction-following and multi-turn chat capabilities.This limitation concerns capabilities outside the chemistry tasks targeted during training.
  • 6 Limitations: Because tool calling was not included in training, ether0 does not directly support tool-heavy small-molecule design workflows.The authors note that future work could integrate chemistry reasoning and tool calling in one model.

7 Conclusion

The paper concludes that reasoning models can solve challenging chemical reasoning questions and that ether0 performs strongly across evaluations. It also analyzes how reasoning helps, where the model fails, and how efficiently it uses data.

  • ether0 is a 24B-parameter reasoning model trained on challenging molecular design, completion, modification, and synthesis tasks.
  • On held-out evaluations, ether0 significantly outperforms frontier LLMs, domain experts, and specialized models, especially on open-answer tasks.
  • The training pipeline interleaves reinforcement learning with verifiable rewards and behavior distillation.
  • The paper examines ether0’s reasoning behavior, failure modes, and data efficiency to identify where reasoning helps and how it changes during training.
  • The authors release the model weights, benchmark data, and reward functions for future scientific reasoning research.

NeurIPS Paper Checklist

The checklist asks whether the abstract and introduction accurately represent the paper’s contributions and scope. It emphasizes that claims should be clearly stated, supported by results, and appropriately bounded in their expected generality.

  • The checklist assesses whether the abstract and introduction include the paper’s contributions and scope accurately.
  • Claims should match the theoretical and experimental results presented in the paper.
  • The abstract and introduction should state important assumptions and limitations and avoid implying broader generalization than the results support.

2. Limitations

The paper reports limitations involving reproducibility, statistical significance, and societal-impact discussion, while also documenting open-access and experimental-detail practices.

  • Reproducibility: The authors publicly release reward functions, test data, training-data sources, and prompt templates through a GitHub repository.These materials support reproduction of the reported experiments.
  • Experimental details: The paper states that all hyperparameters and training methods are extensively described, including details for every training phase.Section 2, supplemental Section C.2, and Section 4 provide data, training, and hyperparameter details.
  • Statistical significance: The paper reports no replicated runs for each task, so it does not provide error bars or other statistical-significance information.The authors cite the high cost of training LLMs and frontier-model API calls as the reason.
  • Societal impact: The paper does not deeply explore the societal risks associated with misuse of chemistry reasoning models.It briefly discusses drug-discovery applications and associated misuse risks, but does not examine them in depth.

C.3 Reward Function Implementation

Reward implementation combines exact matching, property and constraint checks, and plausibility screening to evaluate generated molecules and reactions.

  • Implementation: Reward functions combine Python code, remote calls, and database look-ups to evaluate chemistry tasks.Exact-match tasks use canonicalized molecular or string comparisons, while open-answer tasks apply property and constraint checks.
  • Plausibility checks: Plausibility checks validate molecular valence, ring structures, and atom fragments against synthesized source molecules.The source collections include ChEMBL and COCONUT molecules filtered for evidence of synthesis.
  • Plausibility checks: The conservative plausibility check avoids reward hacks such as inserting peroxides or hydrazines to satisfy target properties.The authors found this check essential for generating plausible molecules.
  • Evaluation: The plausibility check is also applied during evaluation and rejects many answers in molecule-completion and molecular-formula tasks.This evaluation-stage filter therefore affects measured task outcomes, not only training rewards.

D.1 Reasoning Quality Filtering

Reasoning quality is initially high after supervised fine-tuning but degrades during task-specific reinforcement learning. Filtering low-quality traces before distillation marginally improves reasoning quality and reliably raises test performance.

  • The quality judge classified traces as GREAT, ALRIGHT, or BAD using an LLM, with regex-based detection of non-English language.
  • After initial SFT, reasoning quality is almost entirely GREAT, but degradation begins during task-specific RL, especially for IUPAC name, solubility edit, and retrosynthesis.
  • Filtering bad reasoning before distillation produced marginally higher-quality reasoning and reliably improved test-set performance by a few percentage points.Human reviewers preferred reasoning from the filtered RL run.
  • Filtering mattered operationally because, after Arabic characters were removed, the model began using Cyrillic characters during all-task RL.
  • The methodology leaves reasoning largely unconstrained beyond basic formatting, while LLM judges can occasionally misclassify traces.Regex checks found such judge mistakes in fewer than 0.1% of judged traces.

E.1 Task-Specific RL

Task-specific reinforcement learning uses shared GRPO settings and supports new chemical reasoning behaviors through trial and error. Training traces and evaluation analyses track reasoning behaviors, answer outcomes, and capability changes across chemistry tasks.

  • Task-specific RL uses shared settings including a 2048-token completion limit, GRPO epochs µ=1, temperature 1.0, KL penalty β=0.005, and learning rate 10^-6.
  • The training omits top-K, nucleus, and beam sampling because top-K sampling caused unstable learning with K=50.
  • Safety alignment makes ether0 refuse 80% of unsafe questions without meaningfully degrading capability on measured tasks.
  • Reinforcement learning can bootstrap correct retrosynthesis completions from zero success without initial SFT, while ether0 uses SFT on rejection-sampled long chain-of-thought sequences as a warm start.
  • Evaluation analyses use Llama-3.3-70B-Instruct to automatically count specified reasoning behaviors in generated samples.
  • Figures S4 and S5 track reasoning-behavior counts and answer-outcome distributions during training across chemistry tasks.

F.3 Advantage-Based Curriculum Ablation

The advantage-based curriculum increases the fraction of sampled problems that provide useful non-zero-advantage learning signals. This produces higher starting holdout accuracy and faster improvement than training without a curriculum.

  • With a curriculum, the non-trivial fraction rises from 30% to 50–60% early and remains above 40%, whereas it drops to nearly 20% without one.The comparison uses identical reaction-prediction specialist training runs with and without a curriculum.
  • The curriculum causes holdout accuracy to start higher and increase faster.
  • Figure S5 tracks reasoning-behavior counts and answer outcomes throughout training on SMILES completion, solubility edit, retrosynthesis, and multiple-choice tasks.
  • Table S2 reports average human accuracy with minimum and maximum scores, while Table S3 evaluates model-generated reasoning traces.
  • Human expert evaluation includes held-out open-ended and multiple-choice questions and separate assessments of ether0 and DeepSeek-R1 reasoning traces.
Loading 2506.17238v2…