PaperlayerView exact revision on arXiv

Source-linked AI summary

How Post-Training Shapes Biological Reasoning Models

Lukas Fesser, Hanlin Zhang, Michelle M. Li, Eric Wang, Bryan Perozzi, Shekoofeh Azizi, Sham M. Kakade, Marinka Zitnik

arXiv:2606.16517v2cs.LGq-bio.QM

TL;DR

How post-training stages shape biological reasoning and generalization remains poorly understood. In a controlled study across genomics, transcriptomics, and proteins, the authors find that stages affect generalization differently, with the strongest ID-OOD trade-off coming from brief SFT and larger RL allocations.

  • Problem

    How different post-training stages shape biological reasoning and generalization remains poorly understood.

    Source: §1 · p. 2

  • Method

    The authors conduct a controlled study of more than 100 models across genomics, transcriptomics, and proteins, varying backbone, CPT, SFT, and RL while measuring ID and OOD performance.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Across modalities, CPT aligns models with biological language, SFT establishes task competence, and RL shifts checkpoints toward stronger OOD performance at comparable ID performance.

    Source: §5 · p. 10

  • Takeaways & Limitations

    When OOD robustness matters, the authors suggest using CPT for biological-language alignment, enough SFT for task competence, and later post-training for RL.

    Source: §5 · p. 10

    The conclusions rest on a limited set of tasks, benchmarks, and model families, leaving their extension to other scientific reasoning settings and richer biological workflows unclear.

    Source: §5 · p. 10

Abstract

from arXiv · show

Scientific reasoning models for biology combine language models with foundation models trained on multimodal biological data, including DNA, RNA, and proteins. These models are built through post-training, yet how each stage shapes reasoning and generalization remains poorly understood. We study when post-training improves performance and when it induces over-specialization. Across genomics, transcriptomics, and proteins, we train and evaluate more than 100 biological reasoning models under controlled variation in backbone, continued pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning (RL), measuring both in-domain (ID) and out-of-domain (OOD) performance. We find that each post-training stage reshapes generalization in a distinct way rather than contributing uniform gains. CPT improves downstream performance by aligning models with biological language. SFT consistently increases ID performance but causes OOD performance to peak early and decline as models fit the training distribution. RL, when applied to strong SFT checkpoints with aligned rewards, improves OOD performance and partially recovers generalization. These results show that biological reasoning does not improve monotonically with additional supervision or compute. Instead, performance depends on how training stages are composed. Under fixed post-training budgets, the strongest ID-OOD trade-off comes from brief SFT, larger RL allocations, and asymmetric adaptation capacity across stages.

1 Introduction

Across more than 100 biological reasoning models, post-training stages produce distinct effects on in-domain and out-of-domain performance rather than uniform gains. The strongest ID–OOD trade-off comes from brief SFT, larger RL allocations, and asymmetric adaptation capacity.

1 Introduction

  • Biology is a stringent generalization test because unseen pathways, diseases, species, and perturbations can involve different mechanisms, so familiar-benchmark performance may not transfer.Additional post-training or larger models can raise in-domain performance without improving biological generalization.Source: §1 · p. 2
  • The study addresses limited understanding of stage interactions and generalization amid confounded comparisons, sparse biological OOD evaluation, and inconsistent OOD definitions.Existing studies often examine one modality, benchmark family, or post-training stage at a time and focus on final performance rather than training dynamics.Source: §1 · p. 2
  • CPT improves downstream adaptation, SFT raises ID performance while narrowing OOD robustness, and RL improves OOD performance from strong SFT checkpoints.These stage-specific effects were measured across genomics, transcriptomics, and protein tasks.Source: §1 · p. 2
  • The controlled study trains more than 100 models across genomics, transcriptomics, and proteins, isolating backbone, CPT, SFT, and RL effects on ID and OOD performance.Matched model families, tasks, and data settings support controlled comparisons across stages.Source: §1 · p. 2
  • Biological reasoning does not improve monotonically: the strongest ID–OOD trade-off uses brief SFT, larger RL allocations, and asymmetric adaptation capacity.The paper presents these as practical design principles under limited compute.Source: §1 · p. 3

2 Background and Related Work

Biological foundation models have moved from modality-specific prediction toward language-model systems for biological reasoning, but fragmented methods leave training effects under distribution shift poorly understood. Prior language-model work likewise treats post-training stages as distinct, with gains that may not translate into robust generalization.

Biological Foundation Models and Multimodal Reasoning

  • Biological foundation models span DNA, RNA, gene expression, and proteins, while newer systems combine language models with biological inputs for reasoning over pathways, cell states, or protein function.Much earlier work centers on representations or predictive tasks within a single modality.Source: §2 · p. 3
  • Architectural, data, supervision, and evaluation differences fragment the literature, making comparisons difficult and leaving no unified account of training choices under distribution shift.The passage identifies this gap despite progress in multimodal biological reasoning.Source: §2 · p. 3

Training Dynamics in LLMs: Training Stages Are Not Uniformly Additive

  • Continued pre-training expands domain vocabulary and structure, supervised fine-tuning specializes task behavior, and reinforcement learning aligns outputs to reward signals [56] [57] [58].These stages have distinct roles; reinforcement learning is also sensitive to reward design and sampling.Source: §2 · p. 3
  • Training gains can saturate or reverse, and targeted-evaluation improvements need not improve generalization [59] [60] [61].PYTHIA [62], EVOLM [4], and fine-tuning scaling-law work [63] examine training dynamics and adaptation interactions.Source: §2 · p. 3
  • Continued pre-training research studies schedules, data mixing and replay, transfer versus degradation, soft-masking, and reading-comprehension reformats [64] [65] [66] [67] [68], treating it as a transition to downstream adaptation [70].Supervised fine-tuning and reinforcement learning can yield strong task gains but reduce robustness when scaled or applied incautiously [72].Source: §2 · p. 3

3 Experimental Setup: Tasks, Data, and Training Stages

The study controls post-training comparisons across DNA, RNA, and protein reasoning tasks with ID/OOD splits, using general-purpose LLM backbones and a three-stage CPT–SFT–RL pipeline. Biological inputs are represented by frozen foundation models coupled through trainable projection layers.

3 Experimental Setup: Tasks, Data, and Training Stages

  • The setup isolates post-training effects while keeping biological tasks, model families, and evaluation splits comparable across modalities.Evaluation spans DNA, RNA, and proteins, with in-domain and out-of-domain splits.Source: §3 · p. 3, §3.1 · p. 3

3.1 Biological Reasoning Tasks and Evaluation Splits

  • Pathway prediction tests variant-to-phenotype mechanistic reasoning on the KEGG-derived BIOREASON benchmark [1], with ID/OOD splits by pathway network.Inputs pair reference and variant DNA sequences with textual pathway and gene context; OOD examples come from unseen networks.Source: §3.1 · p. 4
  • Drug-target identification adapts MEDEA to fixed-input reasoning, using four diseases for training and ID evaluation and hepatoblastoma for OOD.The model selects a therapeutic target for a disease and cell type from candidate genes and TranscriptFormer embeddings.Source: §3.1 · p. 4
  • Protein-function prediction builds on BIOREASON-PRO, using multimodal protein context and species-based splits with two held-out OOD species.Inputs include protein embeddings, organism and domain annotations, and protein–protein interactions; the study removes ontology graph inputs.Source: §3.1 · p. 4

3.2 Language Models, Biological Foundation Models, and Post-Training Pipeline

  • Main experiments use Qwen3-1.7B and Qwen3-4B backbones, with Gemma 4 E2B as a backbone ablation [75].Frozen biological models provide DNA, RNA/transcriptomic, and protein representations through trainable projection layers.Source: §3.2 · p. 4
  • The controlled pipeline varies CPT on biological text, task-specific SFT, and reward-based RL from an SFT checkpoint, including compute, backbone, and LoRA-rank ablations.CPT uses next-token prediction, SFT trains on target responses, and RL optimizes a task-aligned reward rather than imitating reference traces.Source: §3.2 · p. 4

4 Results: How Training Stages Shape Biological Reasoning in LLMs

Across biological reasoning tasks, post-training stages reshape generalization differently: SFT improves in-domain fit but can reduce out-of-domain robustness, while CPT and RL can strengthen out-of-domain performance. Training allocation matters: brief SFT followed by more RL produces the strongest ID–OOD trade-off under the tested fixed epoch-level schedule.

4.1 Supervised Fine-Tuning Increases Accuracy but Narrows Generalization

  • SFT raises ID performance, but OOD performance peaks earlier and declines with more epochs; increasing training data instead yields more stable OOD gains that eventually plateau.The pattern spans DNA, RNA, and proteins; under fixed compute, data scaling is the more robust strategy when additional supervision is available.Source: Figure 2 · p. 5, §4.1 · pp. 5–6, Figure 3 · p. 6

4.2 Reinforcement Learning Recovers Generalization After Fine-Tuning

  • Starting from strong SFT checkpoints, RL improves both ID and OOD performance, with the largest gains generally OOD and concentrated in the first few epochs.The gains diminish with later RL epochs.Source: Figure 4 · p. 6, §4.2 · p. 6

4.3 Continued Pre-Training Establishes the Foundation for Biological Reasoning

  • CPT improves the effectiveness of downstream SFT and RL, with markedly larger gains after RL and in OOD settings, especially for the smaller model.For Q1-D OOD, CPT improves the effectiveness of SFT and RL by 0.2 and 0.08, respectively.Source: Figure 5 · p. 7, §4.3 · p. 7

4.4 Backbone Strength Shifts Performance Ceiling but Not Training Dynamics

  • Stronger backbones raise post-training performance and make RL gains more reliable, but do not remove the SFT-induced ID–OOD trade-off or change the basic training dynamics.Gemma improves monotonically by 0.08 ID and 0.15 OOD between RL epochs 1 and 16.Source: Figure 6 · p. 7, §4.4 · p. 8

4.5 Adaptation Capacity Should Be Allocated Asymmetrically Between SFT and RL

  • The best adaptation allocation is asymmetric: SFT benefits from higher LoRA rank, while RL performs best around rank 16 and can lose OOD performance at higher ranks.The pattern holds for both Q1-R and Q4-R; the highest ID regions occur at SFT rank 256.Source: Figure 7 · p. 8, §4.5 · pp. 8–9

4.6 Optimal Post-Training Requires Balancing SFT and RL

  • With eight epoch-level passes, a small SFT warm start followed by more RL gives the strongest ID–OOD trade-off, while heavier SFT favors ID performance.The comparison is not strictly FLOP-matched because RL epochs cost more than SFT epochs.Source: Figure 8 · p. 9, §4.6 · p. 9

5 Discussion

Post-training stages shape biological reasoning differently: CPT aligns models with biological language, SFT establishes task competence, and RL can improve transfer beyond the training distribution. The discussion emphasizes that preserving cross-system transfer depends on how training is composed, not simply on more compute or longer optimization.

5 Discussion

  • Across DNA, RNA, and protein tasks, RL shifts checkpoints toward stronger OOD performance at comparable ID performance, after CPT aligns biological language and SFT establishes task competence.The stages are not interchangeable sources of compute; the suggested recipe is to use enough SFT, then allocate later post-training to RL when OOD robustness matters.Source: Figure 9 · p. 9, §5 · p. 10
  • In RNA experiments, OOD accuracy drops by roughly 18 percentage points from its peak as SFT continues, following an approximately monotonic decline rather than a plateau.Biological OOD cases can involve unseen pathways, diseases, species, and perturbations with different mechanisms, making training-distribution gains coexist with reduced transfer across biological systems.Source: §5 · p. 10
  • Conclusions are limited by few tasks, benchmarks, and model families; RL dependencies and compute normalization remain undercharacterized, and final-answer accuracy cannot verify valid intermediate reasoning.Future work should test broader benchmarks and study reward design, compute-normalized stage allocation, and adaptation capacity.Source: §5 · p. 10
  • Scientific-reasoning progress depends on controlling how training stages shape transfer, not merely on scaling model size or post-training compute.The discussion argues that the strongest models preserve the ability to transfer across biological systems rather than optimizing longest on a fixed distribution.Source: §5 · p. 10

Appendices … B.1 Base Models, Tokenization, and Input Representations

The appendices detail how the pathway, target-identification, and protein-function benchmarks are constructed and simplified, then specify their example inputs and model representations. Across DNA, RNA, and protein experiments, frozen biological encoders are paired with trainable projections for downstream stages, while CPT uses text backbones alone.

A.1 Pathway Prediction Dataset and Construction

  • The pathway benchmark contains 1,449 examples across 298 networks and 37 diseases, with paired reference/variant sequences and inputs designed to support mutation-to-pathway-to-phenotype reasoning.Examples average roughly 4,000 base pairs, and most mutations differ by 1–3 nucleotides; inputs combine variant details, network definitions, and gene context with mechanism-to-disease answers and reasoning traces.Source: §A.1 · p. 19
  • BIOREASON’s pathway traces average 303.8 words, and this work splits data by pathway network so OOD evaluation tests transfer to unseen molecular systems.The traces verbalize how mutations affect genes, propagate through pathway interactions, and relate to disease; they were generated with Claude 3.7 Sonnet and grounded in KEGG disease context.Source: §A.1 · p. 19

A.2 Target Identification Dataset and Simplifications Relative to MEDEA

  • The MEDEA-derived target benchmark covers 2,400 analyses across five diseases and 29 cell types, using candidate sets with one supported target and four negatives.Targets are defined by disease-context differential expression plus disease-target evidence; the adapted task supplies disease, cell type, candidates, and transcriptomic evidence rather than requiring tool-based research planning.Source: §A.2 · p. 19
  • The fixed-input target task uses TranscriptFormer embeddings for five candidate genes in normal and disease states, with four diseases for ID splits and hepatoblastoma reserved for OOD testing.This preserves target selection while removing planning, retrieval, execution, and literature-reasoning demands of MEDEA’s agentic workflow.Source: §A.2 · p. 20

A.3 Protein Function Prediction Dataset and Simplifications Relative to BioReason-Pro

  • The BioReason-Pro-derived protein benchmark uses experimentally supported GO annotations and multimodal protein context, while this version removes the GO graph encoder and retains temporal evaluation.The source dataset has 133,492 proteins from 3,135 organisms; the temporal holdout contains 8,630 proteins and 230,824 propagated GO annotations, with training and validation before November 2022 and newly annotated test proteins from March 2023–February 2024.Source: §A.3 · p. 20

A.4 Example Prompts and Inputs for Each Task

  • Example prompts pair DNA reference and variant sequences with pathway context, ask target selection from candidate genes using RNA representations, or request protein-function summaries from protein and biological context.The examples include a pathway question about disease effect, a cell-type-specific target question, and a protein prompt organized around molecular function, biological process, and cellular component.Source: §A.4 · pp. 20–21

B.1 Base Models, Tokenization, and Input Representations

  • DNA and RNA use Qwen3-1.7B or Qwen3-4B native-tokenizer backbones for CPT on biological text alone; downstream stages prepend projected hidden states from frozen Evo2-1B or TranscriptFormer encoders.DNA inputs are clipped to 2,048 nucleotides, retaining 1,024 on each flank around the variant; RNA representations cover candidate genes in normal and disease states and precede prompt tokens.Source: §B.1 · p. 22
  • Protein experiments use Qwen3-1.7B or Qwen3-4B-Thinking with a frozen ESM-3 small encoder, projecting layer-37 residue embeddings into protein placeholder positions while training the projection and LoRA adapter.Prompts include task instructions, protein and organism information, sequence, and a request for GO identifiers across molecular function, biological process, and cellular component; the GO-graph encoder is omitted.Source: §B.1 · p. 22

B.2 Continued Pre-training (Mid-training) Setup … C Main-Results Tables

The excerpts describe a multimodal post-training pipeline spanning broad biological-language adaptation, modality-specific SFT and RL, and stage-specific choices for trainable parameters, optimization, and compute. They specify training designs and evaluation conditions, but provide no main-results comparisons.

B.2 Continued Pre-training (Mid-training) Setup

  • CPT uses 200,000 FineFineWeb biology documents for training and 5,000 held-out documents for evaluation, without downstream benchmark examples or reasoning traces.The setup treats CPT as broad biological-language adaptation under realistic pretrained-model conditions, not as a strictly decontaminated intervention; training uses one epoch of causal next-token prediction, and the best checkpoint is selected by validation loss.Source: §B.2 · pp. 22–23

B.3 Supervised Fine-Tuning Setup

  • DNA, RNA, and protein SFT pair Qwen text backbones with frozen modality-specific encoders and trainable projections or adapters.DNA uses Evo2-1B and sweeps 1–32 epochs; RNA uses TranscriptFormer embeddings prepended to text embeddings to predict a target gene; protein SFT uses ESM-3 small and sweeps data fractions or epochs. The recipes use single-GPU bf16 training with AdamW.Source: §B.3 · p. 23

B.4 Reinforcement-Learning (GRPO) Setup

  • DNA and RNA GRPO use eight rollouts per prompt, a format-and-correctness reward, and a fresh RL adapter attached after merging the selected SFT LoRA.RNA starts from the strongest SFT checkpoints identified in its sweep and tests 1, 2, 4, 8, or 16 GRPO epochs; correctness is based on an exact held-out target-gene match after normalization, with additional structure and conciseness rewards [1].Source: §B.4 · pp. 23–24
  • Protein GRPO warm-starts from the matching SFT adapter and projection, uses propagated GO-F1 reward, and evaluates both ID and OOD splits.The SFT-initialized policy serves as the KL reference; the reported evaluation selects a checkpoint by centered 50-step rolling mean training reward and uses sampling at T = 0.7.Source: §B.4 · pp. 24–25

B.5 LoRA Configurations and Trainable-Parameter Choices

  • Trainable capacity differs by modality: DNA uses rank-32 adapters in SFT and RL, RNA uses rank 64 in SFT and rank 16 in RL, and protein SFT uses rank 128.The DNA and RNA encoders remain frozen while their projections train; protein RL warm-starts from SFT and trains its adapter and projection while freezing the underlying Qwen3-4B-Thinking weights.Source: §B.5 · p. 25

B.6 Hyperparameters, Context Windows, Sequence Lengths, and Optimization

  • The optimization plan combines shared AdamW settings with sweeps over CPT learning rates, protein data fractions, and DNA/RNA SFT and RL epochs.Common settings include β1 = 0.9, β2 = 0.999, ε = 10−8, weight decay 0.01, and gradient clipping 1.0; CPT tests learning rates 10−5 and 3×10−4, while protein data sweeps cover 20–100%.Source: §B.6 · pp. 25–26

B.7 Compute Resources and Training Budget

  • The reported protein RL sweep is the largest compute cost at approximately 56 H200-days for training plus 20 GPU-days for paired ID/OOD evaluations.The runs use one GPU each; DNA/RNA CPT and protein SFT each consume approximately 24 GPU-days.Source: §B.7 · p. 26

C.1 Results for Figure 2 … D Additional Experiments

The tables document the metrics and experimental sweeps underlying Figures 2–6 and 8, covering supervised fine-tuning, reinforcement learning, continued pre-training, backbone choice, and fixed-budget SFT–RL allocation.

C.1 Results for Figure 2

  • Table 2 reports DNA and RNA accuracy and protein propagated unweighted Fmax for the base model and supervised fine-tuning epoch sweep.Values cover epoch 0 and the indicated SFT epochs.Source: Table 2 · p. 26

C.2 Results for Figure 3

  • Table 3 reports protein propagated unweighted Fmax on ID and OOD splits for one-epoch SFT runs varying the number of training examples.The table includes the base model at 0K examples.Source: Table 3 · p. 27

C.3 Results for Figure 4

  • Table 4 reports DNA and RNA accuracy and protein propagated unweighted Fmax across reinforcement-learning epochs, with protein RL evaluated through epoch 8.Source: Table 4 · p. 27

C.4 Results for Figure 5

  • Table 5 compares DNA and RNA ID/OOD accuracy for non-CPT and CPT-adapted backbones under selected SFT and SFT+RL configurations.The base column is the non-CPT backbone before task-specific post-training.Source: Table 5 · p. 28

C.5 Results for Figure 6

  • Table 6 reports RNA ID/OOD accuracy for Gemma4-E2B across an SFT epoch sweep and an RL epoch sweep initialized from the strongest SFT checkpoint.The RL sweep runs through 16 epochs, and the SFT sweep includes the base model at epoch 0.Source: Table 6 · p. 28

C.6 Results for Figure 8

  • Table 7 reports DNA and RNA ID/OOD accuracy as proportions for SFT–RL allocations within a fixed budget of eight epoch-level passes.Source: Table 7 · p. 28

D.1 Scaling Post-Training for Biological Non-Reasoning Tasks · D.2 DNA LoRA Rank Allocation

On VEP-Non-SNV, SFT accuracy improves through 16 epochs and declines by 32; for DNA pathway prediction, higher LoRA rank benefits SFT, while lower rank suffices for RL.

D.1 Scaling Post-Training for Biological Non-Reasoning Tasks

  • The VEP experiment tests whether more SFT compute improves a less reasoning-oriented biological prediction task monotonically, contrasting it with pathway prediction’s multi-step mechanistic inference [1].The paper presents VEP-Non-SNV as a non-reasoning control alongside its biological reasoning experiments.Source: §D.1 · pp. 28–29
  • The VEP-Non-SNV task predicts coding non-SNV effects, whereas pathway prediction requires multi-step inference over molecular networks.Source: §D.1 · pp. 28–29
  • VEP-Non-SNV is a classification task predicting benign or pathogenic status from paired DNA sequences and context, with disease phenotype predicted for pathogenic variants.Source: §D.1 · p. 29
  • The benchmark uses ClinVar coding non-SNVs filtered by sequence length, clinical review status, and transcript matching, and includes stratified splits and paraphrased prompts [1].The cited benchmark contains 36,088 core non-SNV entries.Source: §D.1 · p. 29
  • Accuracy rises from 0.7123 at one epoch to 0.8316 at sixteen, then declines at thirty-two for Qwen3-1.7B on VEP-Non-SNV.The results show that moderate additional SFT improves performance, but excessive training degrades the final metric.Source: Table 8 · p. 29, §D.1 · p. 29

D.2 DNA LoRA Rank Allocation

  • For DNA pathway prediction, higher LoRA rank benefits SFT, while lower rank is sufficient for RL on both ID and OOD tasks.The figure indicates that adaptation capacity should differ across training stages.Source: Figure 10 · p. 29

D.3 Fixed Epoch, Variable Data during RL

With RL fixed to one epoch, protein-function F1 peaked at intermediate data budgets and declined with further scaling. The findings indicate that RL data volume, like epoch count, requires careful calibration.

D.3 Fixed Epoch, Variable Data during RL

  • Under a fixed single-epoch schedule, additional RL data did not substitute for multi-epoch exploration and could over-constrain the policy.At 20K examples, both ID and OOD F1 were below their respective peaks.Source: §D.3 · p. 29
  • RL compute allocation—whether measured in epochs or data volume—requires careful calibration to maximize its impact on biological reasoning.Source: §D.3 · p. 30
  • ID F1 peaks at 0.952 with 12K examples, while OOD F1 peaks at 0.956 with 4K; OOD F1 falls to 0.884 at 20K.The comparison is for Qwen3-4B-Thinking trained with GRPO at β = 10−3 for one epoch, measured by propagated GO-F1.Source: §D.3 · p. 29, Table 9 · p. 30

D.4 Fixed Budget, Variable Data Allocation between SFT and RL · D.5 Reward Model Ablations

With a fixed 20K-example budget, a mixed SFT–RL allocation outperforms either endpoint on ID and OOD protein function prediction, with the strongest results at 20% SFT and 80% RL. The next subsection audits OOD reward hacking across RNA RL checkpoints.

D.4 Fixed Budget, Variable Data Allocation between SFT and RL

  • 0.9470 ID F1 and 0.9685 OOD F1 are achieved by the 20% SFT/80% RL allocation, the strongest fixed-budget results reported.The comparison uses a fixed 20K-example post-training budget.Source: Table 10 · p. 30, §D.4 · p. 30
  • Table 10 varies the SFT/RL data fractions while holding the total post-training budget at 20K examples.It reports F1 on ID and OOD protein function prediction splits.Source: Table 10 · p. 30, §D.4 · p. 30
  • The strongest ID and OOD performance occurs in a mixed SFT–RL regime rather than at either endpoint.The 20%/80% split assigns a small fraction of examples to SFT and the majority to RL.Source: §D.4 · p. 30
  • Pure SFT performs poorly, while pure RL is substantially better but remains far below mixed allocations.This comparison is reported for the fixed-total-data setting.Source: §D.4 · p. 30
  • The results indicate that RL benefits from a modest supervised warm start even when the total data budget is fixed.The strongest reported split allocates 20% of examples to SFT and 80% to RL.Source: §D.4 · p. 30

D.5 Reward Model Ablations

  • Table 11 audits OOD reward hacking across RNA RL checkpoints, defining format-only success as format validity with an incorrect final answer.Epoch 0 denotes the SFT initialization before RL.Source: Table 11 · p. 30

D.6 ID/ OOD Split Ablations · E Asset Licenses and Redistribution Status

Under an RNA split that holds out cell types rather than disease, the SFT–RL generalization contrast persists; the next section inventories existing assets and their licenses or terms of use.

D.6 ID/ OOD Split Ablations

  • The alternative RNA target-identification split labels regulatory T, exhausted T, and myeloid cell types as OOD, with remaining examples used for training and validation.This changes the distribution-shift axis from held-out disease to held-out cellular context.Source: §D.6 · pp. 30–31
  • The cell-type split contains 1,418 training examples, 75 validation examples, and 102 OOD test examples.OOD examples are defined by canonical cell type.Source: §D.6 · pp. 30–31
  • The ablation preserves the main RNA task format, model architecture, and evaluation protocol, using greedy generation with exact match against the target gene.Inputs include disease, cell type, five candidate genes, and aligned TranscriptFormer representations.Source: §D.6 · p. 31
  • Under the cell-type split, Qwen3-1.7B OOD accuracy fell from 60.8% after one SFT epoch to 39.2% at four epochs, then rose from 39.2% at GRPO initialization to 95.1% after eight RL epochs.The table reports RNA target-identification exact-match accuracy under this split.Source: §D.6 · p. 31, Table 12 · p. 31
  • Qwen3-1.7B ID accuracy rose from 54.7% after one SFT epoch to 62.7% after eight, while GRPO from the four-epoch checkpoint increased ID accuracy from 58.7% to 73.3%.The SFT–RL contrast also appeared when OOD meant held-out cellular contexts rather than held-out hepatoblastoma disease.Source: §D.6 · p. 31

E Asset Licenses and Redistribution Status

  • Table 13 lists the existing assets used in the work and their licenses or terms of use.The supplied passage does not provide individual assets or license details.Source: Table 13 · p. 32

Related papers

Loading 2606.16517v2…