PaperlayerView exact revision on arXiv

Source-linked AI summary

Best Practices for Biorisk Evaluations on Open-Weight Bio-Foundation Models

Boyi Wei, Zora Che, Nathaniel Li, Udari Madhushani Sehwag, Jasper Götting, Samira Nedungadi, Julian Michael, Summer Yue, Dan Hendrycks, Peter Henderson, Zifan Wang, Seth Donoughe, Mantas Mazeika

arXiv:2510.27629v4cs.CRcs.AI

TL;DR

Open-weight bio-foundation models raise dual-use concerns, and evidence is limited on whether data filtering withstands malicious fine-tuning or probing. This paper introduces BIORISKEVAL to assess sequence modeling, mutational effects, and virulence. Its findings indicate that excluded capabilities may be recovered through fine-tuning and that pretrained representations can retain probe-accessible signals.

  • Problem

    It is unclear whether pretraining data filtering reduces dual-use risks in bio-foundation models when adversaries can fine-tune them, or whether harmful knowledge remains latent and probe-accessible.

    Source: §1 · p. 1

  • Method

    BIORISKEVAL evaluates bio-foundation model capabilities in sequence modeling, mutational effect prediction, and virulence prediction.

    Source: §1 · p. 2

  • Results

    Linear probes on Evo 2 hidden-layer representations predict mutational effects and virulence comparably to a model trained without data filtering, while fine-tuning can recover excluded knowledge in some cases.

    Source: §1 · p. 2, §5 · pp. 8–9

  • Takeaways & Limitations

    Data filtering alone may fall short as a safety measure, so safety strategies should also account for adversarial fine-tuning and probing.

    Source: §5 · p. 9

    The study evaluates only Evo 2, and its virulence data cover Influenza A, leaving performance across other viral families untested.

    Source: §5 · p. 9

Abstract

from arXiv · show

Open-weight bio-foundation models present a dual-use dilemma. While holding great promise for accelerating scientific research and drug development, they could also enable bad actors to develop more deadly bioweapons. To mitigate the risk posed by these models, current approaches focus on filtering biohazardous data during pre-training. However, the effectiveness of such an approach remains unclear, particularly against determined actors who might fine-tune these models for malicious use. To address this gap, we propose BioRiskEval, a framework to evaluate the robustness of procedures that are intended to reduce the dual-use capabilities of bio-foundation models. BioRiskEval assesses models' virus understanding through three lenses, including sequence modeling, mutational effects prediction, and virulence prediction. Our results show that current filtering practices may not be particularly effective: Excluded knowledge can be rapidly recovered in some cases via fine-tuning, and exhibits broader generalizability in sequence modeling. Furthermore, dual-use signals may already reside in the pretrained representations, and can be elicited via simple linear probing. These findings highlight the challenges of data filtering as a standalone procedure, underscoring the need for further research into robust safety and security strategies for open-weight bio-foundation models.

1. Introduction

Open-weight bio-foundation models raise dual-use concerns, while current evaluations leave uncertain whether filtering withstands malicious fine-tuning or whether harmful knowledge remains probe-accessible. BIORISKEVAL evaluates three capability areas and finds that fine-tuning and probing may recover capabilities despite pretraining data filtering.

1. Introduction

  • Evaluation gaps include the lack of a bio-foundation risk framework and limited evidence on filtering under malicious fine-tuning or probing.It remains untested whether filtered capabilities are trivial to relearn or whether latent representations retain relevant knowledge.Source: §1 · p. 1
  • BIORISKEVAL evaluates harmful capabilities through sequence modeling, mutational effect prediction, and virulence prediction.The framework addresses dual-use risk in bio-foundation models.Source: §1 · p. 2
  • The evaluation tests malicious fine-tuning for generalization to unseen sequences and the minimum compute needed to recover excluded knowledge.The paper identifies data-exclusion practices as limited safety and security mechanisms.Source: §1 · p. 2
  • Linear probes on Evo 2 hidden-layer representations predict mutational effects and virulence comparably to a model trained without data filtering.This result is reported without fine-tuning.Source: §1 · p. 2

2. Related Work

Bio-foundation models support sequence generation and prediction tasks, and attackers may target them during deployment or fine-tuning. Prior work examines several dual-use risks, but most attack–task combinations and malicious fine-tuning risks for bio-foundation models remain underexplored.

2. Related Work

  • Table 1 frames sequence generation and regression/classification as model uses and deployment and fine-tuning as attack stages; most combinations remain underexplored.Applications include mutational effect prediction and functional genomics analyses.Source: Table 1 · p. 2, §2 · p. 3
  • Data filtering is used during pretraining to mitigate risks, and Evo 2 excluded eukaryotic viral data for biosecurity concerns.Filtering has also been studied as a safeguard in language models.Source: §2 · p. 3
  • Prior bio-foundation risk evaluations include perplexity, mutational effects, protein generation, ancestry bias, and guided search for eukaryotic viral sequences.Language-model studies also assess bio-risk questions, virology troubleshooting, ProtocolQA, and tacit-knowledge probing.Source: §2 · p. 3
  • The cited work identifies a lack of research assessing fine-tuning-related dual-use risks in bio-foundation models.Wallace et al. [51] propose worst-case estimation of knowledge gains after fine-tuning in language models.Source: §2 · p. 3

3. Evaluating Bio-Foundation Models’ Harmful Capabilities

The threat model considers adversaries using open model weights to design pathogenic eukaryotic viruses through generation, trait optimization, or virulence ranking. BIORISKEVAL assesses related capabilities through sequence modeling, mutational effect prediction, and virulence prediction.

3.1 Threat Model

  • Adversaries may generate viable viral sequences, optimize hazardous traits, or rank candidates by predicted virulence using fine-tuning or representation probing.Defenders seek to constrain malicious utility while preserving scientific value after releasing model weights.Source: §3.1 · p. 3

3.2 BIORISKEVAL

  • BIORISKEVAL assesses sequence modeling, mutational effect prediction, and virulence prediction on eukaryotic viral sequences.These are the framework's three dimensions of harmful capability.Source: §3.2 · pp. 3–4
  • Sequence modeling is assessed by perplexity on human-host-tagged viral sequences, with 10,247,388 unique sequences across 57 families.The dataset spans 131 genera and 1,795 species.Source: §3.2 · p. 4
  • Mutational effects are evaluated on 16 human-virus Deep Mutational Scanning datasets using absolute Spearman correlation between experimental fitness and model scores.A larger |ρ| indicates stronger agreement with experimental rankings.Source: §3.2 · p. 4
  • Virulence prediction uses Influenza A mouse LD50 records for the BALB/C strain and Pearson correlation of probe predictions with observed values.The eight RNA segments of each strain are concatenated for evaluation.Source: §3.2 · p. 4

4. Experimental Results

Across Evo2-7B’s sequence, mutational-effect, and virulence evaluations, filtering did not preclude capability recovery. Recovery varied with taxonomic distance, while predictive signal was present in both fine-tuned outputs and base-model hidden representations.

4. Experimental Results

  • The experiments use Evo2-7B as a controlled baseline because its pretraining deliberately excluded eukaryotic viral sequences, and evaluate sequence modeling, mutational effects, and virulence.The sequence-generalization tests compare held-out species with held-out genera, representing different evolutionary distances.Source: §4 · p. 4, §4.1 · p. 4, Table 3 · p. 5

4.1.1 Inter-Species Generalizability

  • Within 50 fine-tuning steps, held-out species perplexity reached the benign IMG/PR level in both case studies, using 0.72 H100 GPU hours.The authors attribute this transfer in part to typically high structural similarity among species within a genus; they say representative species could streamline malicious fine-tuning.Source: §4.1.1 · p. 5

4.1.2 Inter-Genus Generalizability

  • At genus scale, 1,000 steps lowered Papillomaviridae test perplexity by approximately 20%, while 2,000 steps produced no significant decrease for Orthoherpesviridae.Both remained less generalizable than the species-level cases, making recovery across genera more difficult and computationally demanding.Source: §4.1.2 · p. 5

4.2.1 Log-Likelihood-Based Mutational Effect Prediction

  • After 2,000 fine-tuning steps, Evo2-7B’s mean |ρ| rose from 0.034 to 0.164, narrowing the gap with bio-foundation models trained without filtering.The experiment used 28.9 H100 GPU hours; the evaluated sequences concern human viruses.Source: §4.2.1 · p. 6

4.2.2 Predicting Mutational Effect with Linear Probe

  • Without further fine-tuning, linear probes on Evo2-7B reached |ρ| values of 0.159 and 0.151, and the best result was 52.8% above LLaMA-3.1-8B-Instruct.Additional fine-tuning did not substantially improve hidden-layer probe performance, while log-likelihood predictions rose across fine-tuning checkpoints.Source: §4.2.2 · p. 7

4.3 Latent Virulence Knowledge Persists Despite Data Filtering

  • The base Evo2-7B model reached a maximum Pearson correlation of 0.46 for virulence prediction, 77% above LLaMA-3.1-8B-Instruct.The evaluated target was continuous LD50 over whole viral genomes; the authors argue this differs from binary prediction of human single-nucleotide variants.Source: §4.3 · p. 8
  • Fine-tuning for 200 steps increased maximum virulence-probe Pearson correlation by 5%, while a significant perplexity drop within 100 steps did not strongly affect prediction performance.The authors report that perplexity may not be a suitable proxy for latent virulence encoding.Source: §4.3 · p. 8
  • Virulence-probe expressiveness declined beyond layer 28 as representation magnitude grew drastically, which the authors suggest may explain the collapse.The probing results were closely related to layer-wise representation magnitude but had little correlation with perplexity distribution.Source: §4.3 · p. 8, Figure 4 · p. 8

5. Discussion

BIORISKEVAL finds that filtering is not tamper-resistant in some Evo 2 settings and warns against relying on it as the sole defense. The authors call for safety strategies that account for fine-tuning and probing.

5. Discussion

  • BIORISKEVAL assesses dual-use risks across sequence modeling, mutational-effect prediction, and virulence prediction.The authors describe the framework as a comprehensive assessment across these three dimensions.Source: §5 · p. 8
  • Data filtering may fail to prevent harmful capabilities from emerging through fine-tuning or latent representations, but the authors do not claim it is completely ineffective.They caution against relying on filtering alone and call for more robust safety strategies for open-weight bio-foundation models.Source: §5 · p. 9
  • Future safety evaluations should account for adversarial fine-tuning and probing, which the authors describe as practical for adversaries and potentially able to expand the “bubble of risk” [53].The authors also note that other harmful capabilities, including viral protein sequence generation and host range, remain unexplored.Source: §5 · p. 9
  • The study evaluates only Evo 2, and its virulence results cover Influenza A; performance on other viral families remains untested.The authors identify limited data availability as a constraint and call for evaluation of additional models and harmful capabilities.Source: §5 · p. 9

Ethics Statement

The authors say their findings do not establish that Evo 2 currently poses a real-world threat, and that the study does not expand the threat landscape. They judge the benefits of releasing the research to outweigh potential misuse.

Ethics Statement

  • Improved mutational-effect prediction after fine-tuning or probing does not directly establish a real-world threat, since overall correlation remains modest.The authors describe current capabilities as limited and not readily exploitable for malicious purposes.Source: Ethics Statement · p. 9
  • All datasets and codebases used for inference and fine-tuning were publicly available before the study, so the authors say it does not expand the threat landscape.Source: Ethics Statement · p. 9
  • The authors conclude that the benefits of releasing the research outweigh its potential misuse.Source: Ethics Statement · p. 9

A.1 Fine-tuning Datasets Curation Process

The fine-tuning data workflow prepares viral sequences through dataset-specific preprocessing, a train–validation split, and concatenation into fixed-length segments. The nucleotide-conversion pipeline reconstructs DMS sequences from protein data but does not reproduce every sequence used in Evo 2 evaluations.

A.1 Fine-tuning Datasets Curation Process

  • Fine-tuning datasets pass through reverse-complement addition, train–validation splitting, and concatenation followed by partitioning into 32,000-token segments.DNA-virus datasets receive reverse complements, while RNA-only datasets omit that stage; 10% of examples are reserved for validation and 90% for training.Source: §A.1 · p. 13, Figure 5 · p. 13
  • The curation process removes empty-genus-tag sequences for Papillomaviridae excluding Gammapapillomavirus and sequences containing “NNN” for Orthoherpesviridae excluding Simplexvirus.These exclusions are intended to avoid train–test overlap and unstable gradient computation, respectively.Source: §A.1 · p. 13

A.2 Protein Sequences to Nucleotide Sequences Conversion Pipeline

  • ProteinGym DMS protein sequences require conversion to nucleotide sequences, but the conversion is degenerate because most amino acids have multiple possible codons.Source: §A.2 · p. 13
  • The pipeline finds or constructs wild-type nucleotide sequences using BLAST matches and seeded random codon replacement, then applies the DMS mutations.Source: §A.2 · p. 13
  • The pipeline does not recreate exact Evo 2 evaluation sequences because it randomly selects among viable codons, and two DMS files could not be reconstructed.The authors consider the approach valid because DMS experiments introduce mutations artificially; they also report reproducing a mean absolute Spearman rank below 0.5 matching S2B in.Source: §A.2 · p. 13
  • The reconstruction is framed as an attacker’s best attempt to recover nucleotide sequences and predict the unchanged ground-truth DMS labels.Source: §A.2 · p. 14

A.3 Evaluation Datasets

The evaluation datasets cover viral DMS tasks and Influenza A virulence prediction, with the DMS protein sequences converted to nucleotide counterparts for genomic-model evaluation. The appendix also points to dataset and split summaries in Tables 4 and 5.

A.3 Evaluation Datasets

  • The zero-shot mutational-effect evaluation converts protein sequences from 16 human-virus DMS datasets into nucleotide counterparts for Evo 2.The datasets are drawn from ProteinGym and correspond to virus DMS studies listed by the Evo 2 paper.Source: §A.3 · p. 14
  • Probe experiments sample 624 balanced examples per converted DMS dataset, or the largest balanced subset when fewer are available, with 7,384 training and 1,868 test samples total.The samples are split into balanced 80% train and 20% test sets; ESM2 uses protein sequences for its masked-marginal calculations.Source: §A.3 · p. 14
  • Virulence probing uses Influenza A LD50 data and corresponding protein sequences, converts them to nucleotides, concatenates the eight genomic segments, and probes hidden representations.Source: §A.3 · p. 14

A.4 DMS Datset Overview

B. Experiment Details

The experiment details specify the computing setup, fine-tuning configuration, probing procedures, and correlation metrics used to evaluate model capabilities.

B.1 Hardware Configuration

  • Experiments used four NVIDIA H100-80GB GPUs on an Amazon p5.48xlarge platform with the NVIDIA BioNeMo Framework.The platform uses an AMD EPYC 7R13 processor.Source: §B.1 · p. 15

B.2 Fine-tuning Configuration

  • Fine-tuning used the configuration summarized in Table 6.The supplied passage identifies the table as containing fine-tuning hyperparameter configurations but gives no values.Source: §B.2 · p. 15, Table 6 · p. 15

B.3 Probing Configuration

  • Probing used continuous LD50 labels for regression with a closed-form linear probe and binary lifestyle labels for classification with a sigmoid linear probe.Classification used batch size 128 and Adam optimization.Source: §B.3 · p. 15

B.4 Evaluation Metrics

  • Spearman rank correlation compares mutant DMS-value ranks with ranks from model-derived scores, while Pearson correlation evaluates ground-truth versus predicted LD50 values.The Spearman calculation uses covariance and standard deviations of the rank variables; Pearson uses covariance and standard deviations of the values.Source: §B.4 · pp. 15–16

C. Evo 2 representation Analysis

Evo 2’s deep-layer representations grow in magnitude, yet final-layer RMSNorm supports meaningful outputs; the analysis attributes this growth to architectural features rather than the inference pipeline.

StripedHyena2 architecture

  • A Hyena operator forms q, k, and v streams from residual input x using Toeplitz filters, then combines them through inner convolution G and projection M to produce y.Source: C. Evo 2 representation Analysis · p. 16
  • StripedHyena-2 uses short explicit, medium regularized, and long implicit Hyena filters; Evo 2 orders them SE-MR-LI and inserts Rotary Attention at layers 3, 10, 17, 24, and 31.The filter types capture local patterns, control frequency response and gain, or implement long-range filtering, respectively.Source: C. Evo 2 representation Analysis · p. 16

Potential sources of representation growth

  • Missing residual-stream layer normalization may compound magnitude across layers, while input-dependent gated convolutions can yield a quadratic increase within a layer.Source: C. Evo 2 representation Analysis · p. 16

Representation magnitude and output quality

  • Final-layer RMSNorm rescales representations before logits, so output quality depends on direction and meaningful sequences remain possible despite deep-layer magnitude growth.Source: C. Evo 2 representation Analysis · p. 17
  • The official Vortex inference pipeline reproduces the layer-wise magnitude pattern, supporting the attribution of representation explosion to model architecture rather than the inference pipeline.Source: C. Evo 2 representation Analysis · p. 17
Loading 2510.27629v4…