PaperlayerView exact revision on arXiv

Source-linked AI summary

How Post-Training Shapes Biological Reasoning Models

Lukas Fesser, Hanlin Zhang, Michelle M. Li, Eric Wang, Bryan Perozzi, Shekoofeh Azizi, Sham M. Kakade, Marinka Zitnik

arXiv:2606.16517v2cs.LGq-bio.QM

TL;DR

How different post-training stages shape biological reasoning and generalization remains poorly understood. The paper evaluates more than 100 models across DNA, RNA, and proteins, finding that CPT, SFT, and RL produce distinct generalization dynamics, with brief SFT and larger RL allocations yielding the strongest ID-OOD trade-off.

  • Problem

    How different post-training stages shape reasoning and generalization in biological models remains poorly understood, despite strong empirical gains.

  • Method

    The study controls backbone, CPT, SFT, and RL across more than 100 models spanning DNA, RNA, and proteins with matched ID and OOD evaluations.

  • Results

    Post-training stages have distinct effects: CPT improves downstream performance, SFT raises ID performance but narrows OOD robustness, and RL improves transfer beyond the training distribution.

  • Takeaways & Limitations

    Under limited compute, the strongest ID-OOD trade-off comes from brief SFT, larger RL allocations, and asymmetric adaptation capacity across stages.

  • Takeaways & Limitations

    Conclusions rest on a limited set of tasks, benchmarks, and model families, and their broader applicability remains unclear.

Abstract

from arXiv · show

Scientific reasoning models for biology combine language models with foundation models trained on multimodal biological data, including DNA, RNA, and proteins. These models are built through post-training, yet how each stage shapes reasoning and generalization remains poorly understood. We study when post-training improves performance and when it induces over-specialization. Across genomics, transcriptomics, and proteins, we train and evaluate more than 100 biological reasoning models under controlled variation in backbone, continued pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning (RL), measuring both in-domain (ID) and out-of-domain (OOD) performance. We find that each post-training stage reshapes generalization in a distinct way rather than contributing uniform gains. CPT improves downstream performance by aligning models with biological language. SFT consistently increases ID performance but causes OOD performance to peak early and decline as models fit the training distribution. RL, when applied to strong SFT checkpoints with aligned rewards, improves OOD performance and partially recovers generalization. These results show that biological reasoning does not improve monotonically with additional supervision or compute. Instead, performance depends on how training stages are composed. Under fixed post-training budgets, the strongest ID-OOD trade-off comes from brief SFT, larger RL allocations, and asymmetric adaptation capacity across stages.

1 Introduction

Biological reasoning models must generalize across mechanisms and processes that differ from training examples, making biology a stringent testbed. A controlled study of more than 100 models shows that post-training effects are stage-dependent rather than monotonically beneficial, with brief SFT and larger RL allocations yielding the strongest ID-OOD trade-off.

  • Post-training effects on biological reasoning and generalization remain poorly understood across continued pre-training, supervised fine-tuning, and reinforcement learning.Existing studies often examine only one modality, benchmark family, or post-training stage at a time.
  • Biological out-of-domain problems often involve unseen pathways, diseases, species, and perturbations with different mechanisms and biological processes.High benchmark performance therefore does not necessarily indicate robust biological reasoning.
  • Present work: The study trains and evaluates more than 100 biological reasoning models across genomics, transcriptomics, and proteins under matched model, task, and data settings.The controlled design isolates backbone choice, continued pre-training, supervised fine-tuning, and reinforcement learning.
  • Biological reasoning does not improve monotonically with additional post-training, and training dynamics remain consistent across biological modalities.The study examines when post-training improves generalization versus primarily increasing fit to the training distribution.
  • The strongest ID-OOD trade-off comes from combining brief SFT with larger RL allocations under limited compute.This result provides a practical design principle for composing post-training stages.

2 Background and Related Work

Biological foundation models span DNA, RNA, gene expression, and proteins, with newer systems combining language models, biological inputs, and structured context. Their development uses distinct training stages, whose effects can saturate, reverse, or fail to generalize uniformly.

  • Biological Foundation Models and Multimodal Reasoning: Biological foundation models increasingly cover DNA, RNA, gene expression, and proteins, while newer systems combine language models with biological inputs and structured context.Much earlier work focused on representations or prediction within a single modality.
  • Training Dynamics in LLMs: Training Stages Are Not Uniformly Additive: Modern language models are adapted through continued pre-training, supervised fine-tuning, and sometimes reinforcement learning, with each stage serving a distinct role.Training proceeds as a sequence rather than a single optimization pass.
  • Training Dynamics in LLMs: Training Stages Are Not Uniformly Additive: Additional training can saturate or reverse performance gains, and targeted evaluation improvements need not translate into better generalization.Prior work, including PYTHIA and EVOLM, analyzes checkpoint- and stage-dependent training dynamics and non-monotonic behavior.

3 Experimental Setup: Tasks, Data, and Training Stages

The experiments isolate post-training effects across DNA, RNA, and protein reasoning tasks using comparable model families and ID/OOD evaluation splits. Models combine general-purpose LLM backbones with frozen biological foundation models and undergo controlled CPT, SFT, and RL stages.

  • Tasks and data: The setup evaluates DNA, RNA, and protein tasks, with each task combining natural-language context, modality-specific biological inputs, and ID/OOD splits.
  • Tasks and data: Pathway prediction, drug target identification, and protein function prediction test mechanistic pathway reasoning, therapeutic target selection, and multimodal protein-function inference.
  • Models: Main experiments use Qwen3-1.7B and Qwen3-4B backbones, with Gemma 4 E2B as an ablation for cross-family training dynamics.
  • Models: Each modality couples the LLM to frozen biological foundation models through trainable projection layers: Evo2-1B for DNA, TranscriptFormer for RNA, and ESM-3 for proteins.
  • Training stages: The post-training pipeline comprises continued pre-training on biological text, supervised fine-tuning on task-specific reasoning examples, and reinforcement learning with task-aligned rewards.

4 Results: How Training Stages Shape Biological Reasoning in LLMs

Post-training stages reshape biological reasoning models differently: SFT raises in-domain performance but can reduce robustness, whereas RL and CPT improve out-of-domain generalization under suitable conditions. The strongest fixed-budget trade-off combines brief SFT, larger RL allocations, and asymmetric adaptation capacity.

  • Supervised fine-tuning: SFT rapidly improves ID performance but causes OOD performance to peak early and decline as models specialize to the training distribution.This pattern holds across DNA, RNA, and proteins, with increasing data behaving more reliably than increasing epochs.
  • Supervised fine-tuning: 20K examples raise Q1-P-S ID Fmax from 0.17 to 0.23 and OOD Fmax from 0.11 to 0.22.Increasing dataset size improves both domains more reliably than increasing SFT epochs, though returns diminish.
  • Reinforcement learning: RL improves both ID and OOD performance from strong SFT checkpoints, with the largest gains occurring OOD within the first few RL epochs.In DNA, OOD accuracy rises by about 0.05 across the RL sweep, while protein gains are larger.
  • Continued pre-training: CPT improves the effectiveness of downstream SFT and RL, especially OOD for smaller models adapting first to biological language.For Q1-D OOD, CPT improves SFT effectiveness by 0.2 and RL effectiveness by 0.08.
  • Backbone scaling: Stronger backbones raise absolute performance and make RL gains more reliable without removing the SFT-induced ID-OOD trade-off.G-R outperforms Q1-R by around 0.17 in- and out-of-domain after one epoch, while larger models improve more steadily under RL.
  • Post-training composition: The best pipelines use larger SFT capacity, smaller RL capacity, and a small SFT allocation followed by a larger RL allocation.Increasing SFT rank from 16 to 64 or 256 helps ID performance, whereas increasing RL rank beyond 16 yields smaller gains and can reduce OOD performance; early mixed schedules reach about 0.78 OOD accuracy in DNA.

5 Discussion

The discussion frames CPT, SFT, and RL as distinct contributors to biological reasoning: CPT adapts biological language, SFT builds task competence, and RL improves out-of-distribution transfer. It emphasizes that effective progress depends on controlling generalization dynamics, not simply increasing model size or post-training compute.

  • Distinct roles of post-training: RL generally improves OOD performance at comparable ID performance across DNA, RNA, and protein tasks.Figure 9 shows RL shifting checkpoints toward stronger OOD performance across modalities.
  • Distinct roles of post-training: CPT adapts models to biological language, SFT establishes task competence, and RL improves transfer beyond the training distribution.These stages should not be treated as interchangeable sources of compute.
  • SFT and generalization: 18 percentage points: RNA OOD accuracy drops by roughly this amount from its peak as SFT continues.The decline is approximately monotonic rather than plateauing, illustrating SFT-induced over-specialization.
  • Limitations and future work: The study’s conclusions rely on a limited set of tasks, benchmarks, and model families, leaving their extension to other scientific reasoning settings uncertain.The evaluated tasks span DNA, RNA, and protein reasoning with biologically meaningful OOD splits.
  • Broader implications: Progress in scientific reasoning depends on understanding how training stages shape generalization, not only on larger models or more post-training compute.The discussion argues that preserving transfer across biological systems may be as important as scaling model size.

Appendices … A Additional Details on Biological Reasoning Tasks

The appendices document task construction, experimental configurations, figure-level result tables, and additional experiments covering scaling, adaptation, data allocation, and ablations. They also organize detailed inputs, training choices, and supplementary analyses across biological reasoning tasks.

  • A Additional Details on Biological Reasoning Tasks: Appendix A details pathway prediction, target identification, protein function prediction, and example prompts and inputs for each task.The task descriptions include dataset construction and simplifications relative to MEDEA and BioReason-Pro.
  • B Experimental Setup: Appendix B specifies base models, tokenization, input representations, continued pre-training, supervised fine-tuning, and reinforcement-learning setups.It also covers LoRA configurations, trainable-parameter choices, hyperparameters, context windows, sequence lengths, optimization, compute resources, and training budgets.
  • B Experimental Setup: Appendix B separately records compute resources and training budgets used in the experiments.This section complements the model, optimization, and parameterization details provided elsewhere in the experimental setup.
  • C Main-Results Tables: Appendix C collects supplementary results for Figures 2, 3, 4, 5, 6, and 8.The figure-specific tables are organized as six subsections covering the paper’s main-result figures.
  • D Additional Experiments: Appendix D examines scaling for biological non-reasoning tasks and DNA LoRA rank allocation.These experiments extend the analysis beyond the core reasoning-task setup and vary adaptation capacity.
  • D Additional Experiments: Appendix D tests reinforcement-learning data variation under fixed epochs and data allocation between supervised fine-tuning and reinforcement learning under a fixed budget.These experiments isolate how training data and stage allocation affect post-training comparisons.
  • D Additional Experiments: Appendix D includes reward-model ablations and ID/OOD split ablations.These analyses probe the sensitivity of results to reward modeling and in-domain/out-of-domain partition choices.

A.1 Pathway Prediction Dataset and Construction … B.1 Base Models, Tokenization, and Input Representations

The paper constructs three multimodal biological reasoning benchmarks and standardizes their inputs for controlled experiments. The tasks combine structured biological evidence with sequence or representation-level inputs, while simplifying prior agentic or ontology-enriched setups where specified.

  • A.1 Pathway Prediction Dataset and Construction: The pathway benchmark contains 1,449 examples spanning 298 pathway networks and 37 unique diseases, integrating KEGG variants with disease and variant metadata.It represents pathways as molecular interaction networks linked to variants through semi-automated genomic mappings.
  • A.1 Pathway Prediction Dataset and Construction: Pathway examples pair reference and variant DNA sequences with pathway definitions and causal reasoning traces generated using Claude 3.7 Sonnet.The traces incorporate KEGG disease context and have a mean length of 303.8 words.
  • A.2 Target Identification Dataset and Simplifications Relative to MEDEA: Target identification converts MEDEA’s agentic workflow into fixed-input reasoning over disease, cell type, five candidate genes, and transcriptomic evidence.The benchmark preserves the core target-selection signal while removing planning, tool use, retrieval, and literature reconciliation.
  • A.3 Protein Function Prediction Dataset and Simplifications Relative to BioReason-Pro: The protein benchmark adapts BioReason-Pro’s experimentally supported UniProt-based task into a compact multimodal prediction format.Inputs combine protein annotations such as InterPro domains, organism, localization, interactions, and GO terms, while the paper omits BioReason-Pro’s GO graph input.
  • A.4 Example Prompts and Inputs for Each Task: Representative prompts ask models to infer pathway effects, select disease- and cell-type-specific targets, or predict protein functions from structured biological context.Examples include a SOD1 allele associated with amyotrophic lateral sclerosis and selection of PIK3CG for follicular lymphoma.
  • B.1 Base Models, Tokenization, and Input Representations: DNA experiments use Qwen3-1.7B and Qwen3-4B text backbones, with native tokenizers, language-model-only CPT, and frozen Evo2-1B conditioning for downstream SFT and RL.A trainable linear projection prepends DNA hidden states to text embeddings.
  • B.1 Base Models, Tokenization, and Input Representations: RNA experiments use the same Qwen3-1.7B and Qwen3-4B backbones and native tokenizers, coupling a frozen TranscriptFormer encoder through a trainable projection during SFT and RL.Candidate-gene representations cover normal and disease states and are prepended before prompt tokens.
  • B.1 Base Models, Tokenization, and Input Representations: Protein experiments use Qwen3-1.7B and Qwen3-4B-Thinking with native BPE tokenization and a frozen ESM-3 small encoder whose projected layer-37 embeddings occupy protein placeholders.The interface omits the GO-graph encoder and trains the protein projection layer with LoRA adaptation.

B.2 Continued Pre-training (Mid-training) Setup … C Main-Results Tables

The paper evaluates controlled biological post-training pipelines spanning CPT, SFT, RL, LoRA adaptation, and compute budgets across DNA, RNA, and protein settings. The supplied methods establish fixed data splits, staged training recipes, checkpoint-selection rules, and ID/OOD evaluation procedures.

  • B.2 Continued Pre-training (Mid-training) Setup: CPT mid-trains both Qwen3 backbones on 200,000 FineFineWeb biology documents with the next 5,000 held out for evaluation.This yields a fixed 200K/5K train/eval split.
  • B.2 Continued Pre-training (Mid-training) Setup: CPT uses one epoch of causal-LM training with 1024-token inputs, AdamW, cosine decay, 3% warm-up, bf16 precision, and gradient clipping 1.0.The sweep varies learning rates 1×10−5 and 3×10−4 and accumulation steps 64 and 128, selecting checkpoints by validation loss.
  • B.4 Reinforcement-Learning (GRPO) Setup: RL attaches fresh LoRA adapters after merging SFT adapters, uses frozen modality encoders with trainable projections, and evaluates both ID and OOD splits.DNA and RNA use eight rollouts per prompt; protein RL warm-starts from matching SFT checkpoints and uses GO-F1 rewards.
  • C Main-Results Tables: The supplied passages define experimental configurations and evaluation procedures but contain no main-results table values for section C.Therefore, no quantitative main-result claim is added beyond the reported training and compute settings.
  • B.5 LoRA Configurations and Trainable-Parameter Choices: LoRA capacity is asymmetric across stages: DNA uses r = 32 for SFT and RL, whereas RNA uses r = 64 for SFT and r = 16 for RL.Protein SFT uses r = 128; protein RL trains the reattached adapter and projection while freezing the underlying Qwen3-4B-Thinking weights.
  • B.6 Hyperparameters, Context Windows, Sequence Lengths, and Optimization: The study fixes core optimization settings across stages, including AdamW weight decay 0.01, gradient clipping 1.0, and bf16 mixed precision.The documented sweeps cover CPT learning rates and accumulation, protein SFT/RL data fractions, protein RL epochs, and DNA/RNA SFT/RL epochs.
  • B.7 Compute Resources and Training Budget: All runs use one NVIDIA H100 or H200 GPU, with no multi-node training required for reported results.The dominant compute cost is protein RL: 8 runs at up to 7 days each, approximately 56 H200-days, plus approximately 20 GPU-days for paired evaluation.

C.1 Results for Figure 2 … D Additional Experiments

The supplied passages describe numerical-result tables covering Figures 2–6 and 8, spanning SFT, variable-data training, RL, CPT, backbone, and fixed-budget allocation experiments. No specific passage details the additional experiments in Section D.

  • C.1 Results for Figure 2: Table 2 reports Figure 2 results for DNA and RNA accuracy and protein propagated unweighted Fmax, comparing the base model with SFT across epochs.Values include the base model at epoch 0 and post-SFT performance for indicated epoch counts.
  • D Additional Experiments: No supplied passage gives specific findings for D Additional Experiments.The provided evidence consists of table descriptions for Sections C.1–C.6 only.
  • C.2 Results for Figure 3: Table 3 reports Figure 3 protein-function results under fixed-compute, variable-data SFT, using one SFT epoch and varying training-example counts.Metrics are propagated unweighted Fmax on ID and OOD test splits, with the base model at 0K examples as reference.
  • C.3 Results for Figure 4: Table 4 reports Figure 4 performance after varying RL epochs, using accuracy for DNA and RNA and propagated unweighted Fmax for proteins.Protein RL was evaluated through 8 epochs.
  • C.4 Results for Figure 5: Table 5 reports Figure 5’s CPT ablation on DNA and RNA ID and OOD accuracy, contrasting non-CPT and CPT-initialized post-training configurations.The configurations include Base, SFT, SFT+RL, CPT+SFT, and CPT+SFT+RL.
  • C.5 Results for Figure 6: Table 6 reports Figure 6’s Gemma4-E2B RNA backbone ablation using RNA ID and OOD accuracy across SFT and RL epoch sweeps.RL is initialized from the strongest SFT checkpoint and evaluated through 16 epochs.
  • C.6 Results for Figure 8: Table 7 reports Figure 8’s fixed-budget SFT–RL allocation experiment, with eight total epoch-level passes distributed between supervised fine-tuning and reinforcement learning.DNA and RNA are evaluated with ID and OOD accuracy, and values are reported as proportions.

D.1 Scaling Post-Training for Biological Non-Reasoning Tasks · D.2 DNA LoRA Rank Allocation

On VEP-Non-SNV, supervised fine-tuning improves accuracy through 16 epochs before degrading, showing non-monotonic scaling even for biological prediction. DNA pathway prediction benefits from higher LoRA rank during SFT but lower rank during RL.

  • D.1 Scaling Post-Training for Biological Non-Reasoning Tasks: VEP-Non-SNV differs from KEGG-derived pathway prediction because it primarily involves classification rather than multi-step mechanistic inference over molecular networks.The task is therefore less dependent on explicit chain-of-thought-style mechanistic reasoning.
  • D.1 Scaling Post-Training for Biological Non-Reasoning Tasks: VEP-Non-SNV is a classification-style task predicting coding non-SNV pathogenicity and, when pathogenic, the associated disease phenotype.Inputs include paired reference and variant DNA sequences with gene and chromosome context.
  • D.1 Scaling Post-Training for Biological Non-Reasoning Tasks: The dataset uses ClinVar coding non-SNVs affecting at most 64 base pairs, with clinical-review and GRCh38.p14 transcript-matching filters.The benchmark also uses stratified train/test partitioning and paraphrased prompts.
  • D.1 Scaling Post-Training for Biological Non-Reasoning Tasks: Table 8 shows that moderate additional supervision improves VEP-Non-SNV performance, whereas excessive SFT begins to degrade the final metric.Performance improves through 16 epochs before declining at 32 epochs.
  • D.1 Scaling Post-Training for Biological Non-Reasoning Tasks: Qwen3-1.7B accuracy rises from 0.7123 at one SFT epoch to 0.8316 at sixteen epochs, then declines at thirty-two epochs.This demonstrates non-monotonic SFT scaling in a less explicitly reasoning-oriented biological prediction task.
  • D.2 DNA LoRA Rank Allocation: For DNA pathway prediction, higher LoRA rank benefits SFT, while lower rank is sufficient for RL across both ID and OOD tasks.The results indicate that optimal adaptation capacity is asymmetric across training stages.

D.3 Fixed Epoch, Variable Data during RL

With RL fixed to one epoch, increasing the number of training examples produces non-monotonic protein function prediction performance. ID and OOD F1 peak at intermediate data budgets, then decline with further scaling.

  • D.3 Fixed Epoch, Variable Data during RL: 0.952 ID F1 peaks at 12K examples, while 0.956 OOD F1 peaks at 4K under GRPO with β = 10−3.The experiment uses Qwen3-4B-Thinking trained for one epoch with strong KL regularization.
  • D.3 Fixed Epoch, Variable Data during RL: 0.884 OOD F1 at 20K examples falls from its 0.956 peak, showing that additional RL data can substantially reduce performance.Both ID and OOD metrics decline after their respective intermediate-budget peaks.
  • D.3 Fixed Epoch, Variable Data during RL: RL compute allocation across data volume, like epoch scaling, requires calibration because increasing training resources does not monotonically improve biological reasoning.The result reinforces the need to tune RL budgets for maximum impact.

D.4 Fixed Budget, Variable Data Allocation between SFT and RL · D.5 Reward Model Ablations

Under a fixed 20K-example post-training budget, mixed SFT–RL allocation outperforms either endpoint, with the strongest results from brief SFT followed by majority RL. The reward-model ablation audits OOD reward hacking across RNA RL checkpoints, distinguishing format-valid but incorrect outputs.

  • D.4 Fixed Budget, Variable Data Allocation between SFT and RL: The experiment fixes the post-training budget at 20K examples while varying only the SFT-versus-RL data allocation.Each allocation preserves the total number of post-training examples.
  • D.4 Fixed Budget, Variable Data Allocation between SFT and RL: Mixed SFT–RL allocations outperform both pure SFT and pure RL on protein function prediction.Pure RL is substantially better than pure SFT but remains below mixed allocations.
  • D.4 Fixed Budget, Variable Data Allocation between SFT and RL: 0.9470 ID F1 and 0.9685 OOD F1 are achieved by the 20% SFT/80% RL allocation.This split gives the strongest reported ID and OOD performance under the fixed data budget.
  • D.4 Fixed Budget, Variable Data Allocation between SFT and RL: The allocation results indicate that RL benefits from a modest supervised warm start when the total data budget is fixed.The strongest regime assigns a small fraction of examples to SFT and the majority to RL.
  • D.5 Reward Model Ablations: The reward-hacking audit evaluates OOD reward behavior across RNA RL checkpoints, including the epoch 0 SFT initialization.Epoch 0 denotes the checkpoint before RL.
  • D.5 Reward Model Ablations: Format-only success counts outputs that satisfy formatting requirements but contain an incorrect final answer.This metric is used in the RNA RL checkpoint audit.

D.6 ID/ OOD Split Ablations · E Asset Licenses and Redistribution Status

The cell-type ablation preserves the pattern of stronger in-domain fitting but weaker OOD generalization under SFT, while GRPO substantially restores OOD performance. The paper also documents the external assets and their licensing or terms of use.

  • D.6 ID/ OOD Split Ablations: The RNA ablation replaces held-out disease with held-out cellular context, defining OOD examples by regulatory T, exhausted T, or myeloid cell types.The alternative split uses 1,418 training examples, 75 validation examples, and 102 OOD test examples.
  • D.6 ID/ OOD Split Ablations: The ablation keeps the task format, model architecture, and evaluation protocol unchanged while altering only the biological axis of distribution shift.Models receive disease, cell type, five candidate genes, and aligned TranscriptFormer representations, then use greedy exact-match generation.
  • D.6 ID/ OOD Split Ablations: Table 12 reports RNA target-identification performance under the held-out cell-type split using exact-match accuracies expressed as percentages under greedy generation.OOD examples are those whose canonical cell type is regulatory T cell, exhausted T cell, or myeloid cell.
  • D.6 ID/ OOD Split Ablations: 54.7% ID accuracy after one epoch for Qwen3-1.7B rises to 62.7% by eight SFT epochs, while OOD accuracy falls from 60.8% to 39.2–43.1%.This reproduces the pattern of improved in-domain performance alongside degraded OOD generalization under supervised fine-tuning.
  • D.6 ID/ OOD Split Ablations: 95.1% OOD accuracy after GRPO improves on the four-epoch SFT initialization of 39.2%, following an increase to 88.2% after one RL epoch.Starting GRPO from the four-epoch SFT checkpoint reverses the later-SFT decline in OOD performance.
  • E Asset Licenses and Redistribution Status: Table 13 lists the existing assets used in the work together with their licenses or terms of use.The supplied passage identifies the table’s purpose but does not provide individual asset entries.
Loading 2606.16517v2…