AmbiModBench: Benchmarking Gene Perturbation Prediction Beyond Shared Responses

• Unmeasured perturbations matter because the intervention space across genes, combinations, cell types, and contexts exceeds what any screen can assay.Source: §1 · p. 1
• Absolute scores can reward shared transcriptional responses without showing whether predictions distinguish the evaluated target from a training-mean reference.Source: §1 · p. 2
• Coverage depends on representation-space proximity and how tightly nearby representations constrain response variation, not training-set size alone.Source: §1 · p. 2
• AmbiModBench combines a paired training-mean contrast, gene-coordinate permutation calibration, and a coverage condition to assess prediction beyond the training mean.Source: §1 · p. 2
• Across six RPE1 splits, kernel ridge gains 2.31 percentage points in top-100 direction agreement over the training mean, with a 95% interval of [1.45, 3.18].Source: §1 · p. 2

• Perturbation models encode structure through latent shifts, composed perturbation and context representations, gene relationships, or protein-sequence representations.Source: §2 · p. 3
• Zero-shot learning predicts unseen categories using fixed representations, a setting AmbiModBench examines for perturbation targets.Source: §2 · p. 3
• The Replogle K562 and RPE1 datasets and the Norman CRISPR activation screen provide complementary target populations and intervention types for benchmark evaluation.Source: §2 · p. 3
• Prior benchmarking work has identified score equivalence, metric-specific biological signals, and loss of gene-level signatures despite favorable quantitative scores.Source: §2 · pp. 3–4
• AmbiModBench distinguishes its evaluation by comparing predictions with a same-split reference, checking sensitivity to wrong gene assignments, and assessing representation-level coverage.Source: §2 · p. 4

3.1 TASK DEFINITION

• The held-out target is the unit of generalization, excluded from model fitting, representation transformation, and hyperparameter selection.Source: §3.1 · p. 4

3.2 BENCHMARK DATASETS

• K562 has 490 targets across 15 evaluation instances, RPE1 has 1,089 eligible targets across six splits, and Norman has 105 eligible single-target perturbations across three splits.Source: §3.2 · p. 4

3.3 PREDICTORS AND REPRESENTATIONS

• The six fixed predictors include the training mean reference, zero response, nearest embedding, ridge, kernel ridge, and a spectrally normalized response network.Source: §3.3 · p. 4
• RPE1 uses ProTrek-650M protein-sequence representations, while K562 and Norman use ESM2; regression readouts fit principal components on outer-training targets before row normalization.Source: §3.3 · pp. 4–5

4.1 PAIRED TRAINING-MEAN CONTRAST

• The paired effect subtracts the training-mean reference score from the model score, using a reference fitted separately on each split and the common reporting unit across datasets and gene selections.A positive effect measures improvement over a response that does not use the evaluated target’s identity.Source: §4.1 · p. 5

4.2 DIRECTION AGREEMENT AND INFERENCE

• Direction agreement scores sign matches among the k largest measured responses, gives zero predictions half credit, and uses target-cluster bootstrap intervals conditional on fitted splits.Gene sets are determined only during scoring; cross-dataset selection uses max(20, round(0.02G)) genes.Source: §4.2 · p. 5

4.3 GENE-IDENTITY CALIBRATION

• Gene-coordinate permutation tests whether a readout retains gene identity, while independent replicates separately calibrate magnitude-sensitive scores.The permutation screen compares an independent-replicate response estimate with a copy whose gene coordinates are permuted.Source: §4.3 · pp. 5–6

4.4 TRAINING COVERAGE

• Coverage combines the test-to-training radius with an ambiguity modulus: nearby embeddings support prediction only when they constrain response variation, and training-set size alone specifies neither condition.Under the stated smoothness assumption, the conditional bound separates coverage from response-estimation noise; the fixed-test subsampling experiment refits the procedure across nested training-set fractions.Source: §4.4 · p. 6

5.1 HIGH ABSOLUTE SCORES DO NOT IMPLY TARGET-SPECIFIC PREDICTION

• 2.31 percentage points: RPE1 kernel ridge improves top-100 direction agreement over the training mean, with a 95% target-cluster bootstrap interval of [1.45, 3.18].The gain counts additional correctly signed genes among the 100 strongest measured responses, averaged over targets.Source: §5.1 · pp. 6–7, Figure 2 · p. 7
• After excluding seed42, kernel ridge gains 2.36 [1.49, 3.26] percentage points; every additional split has a positive point estimate, and all four gene-cutoff intervals exclude zero.Across all six splits, gains are 2.45 percentage points for top 2% and 3.57 for top 20.Source: §5.1 · p. 7

5.2 MAGNITUDE-BASED SCORES DO NOT VERIFY GENE ASSIGNMENT

• At mixture weight 0.2, K562 squared error reaches its minimum even as Pearson correlation and top-20 recall increase across the mixture.The passage attributes the squared-error minimum to shrinking toward a noisy cohort-mean reference.Source: §5.2 · p. 7, Figure 3 · p. 8
• Energy and Wasserstein distances over marginal response values remain unchanged after gene-coordinate scrambling, whereas five gene-aligned readouts respond to scrambling.This means the marginal distances evaluate response magnitudes without establishing whether the affected genes were predicted correctly.Source: §5.2 · pp. 7–8, Figure 3 · p. 8

5.3 DETECTABLE GAIN DEPENDS ON COVERAGE, NOT TRAINING-SET SIZE

• Kernel-ridge gain rises from 0.01 percentage points at 10% training to 1.33, 1.72, 2.08 and 2.38 at successive fractions, becoming detectable between 10% and 25%.At 10%, the interval contains zero; the fixed-test subsampling experiment refits transformations and references.Source: §5.3 · p. 8, Figure 4 · p. 9

5.4 CROSS-DATASET EVALUATION REGIMES

• Across the three datasets, the protocol separates a regime with positive paired gain from one at the training mean, while CRISPRa remains unresolved because its interval includes zero.The paper cautions against averaging these regimes into a single verdict.Source: §5.4 · p. 9

6 DISCUSSION AND CONCLUSION

• AmbiModBench qualifies scores through gain beyond a shared-response reference, gene-specific information retained by the score, and training coverage supporting the held-out target.It pairs a training-mean contrast with gene-permutation and replicate calibration, and uses a coverage radius, ambiguity modulus, and fixed-test subsampling.Source: §6 · p. 9
• RPE1 sequence-embedding readouts yield positive directional gain across repeated target holdouts, increasing as more training perturbations are observed; K562 shows no detectable advantage over the training mean.The RPE1 finding identifies a setting where additional response data supports detectable out-of-target prediction; Norman’s interval includes zero and is unresolved.Source: §6 · p. 9
• The coverage bound assumes a finite Lipschitz constant that is not estimated, leaving the bound qualitative; empirical estimation of the ambiguity modulus and radius δt remains open.The benchmark also fixes one family of sequence-derived representations and compares three datasets.Source: §6 · p. 9
• The response network uses a single fixed configuration rather than a searched grid, and the paired contrast is reported only for direction readouts; extending it to effect-size and differential-expression scores remains future work.Removing the auxiliary response-to-embedding penalty restores a response-aligned residual component, but not an effect as large as one additional correct gene, so the verdict is unchanged.Source: §6 · p. 10

AI USE STATEMENT

• Generative AI provided feedback on evaluation-protocol design, helped formulate and revise mathematical arguments, and assisted with interpreting experiment summaries.The authors say complete proofs appear in Appendix A.Source: AI USE STATEMENT · p. 10
• The authors report no AI use for synthetic dataset generation, analyzed-data cleaning or reformatting, translation, or qualitative or thematic data analysis.Quantitative plots and tables were produced from experimental output files.Source: AI USE STATEMENT · p. 10
• Recommended AI-assisted tasks included manuscript editing, literature and reference checks, analysis and visualization scripts, and parts of the benchmark overview figure.The authors reviewed assisted work against proofs, original sources, benchmark records, and rendered figure labels and effects.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Appendix B documents response construction, representations, tuning, subsampling, and target-cluster inference, while Appendix A gives assumptions and proofs.Supporting tables report six predictors across three direction-score selections; analysis records include the locked configuration, per-target scores, and hyperparameter choices.Source: REPRODUCIBILITY STATEMENT · p. 10
• The authors state that code and data are available at the listed project repository.The repository address is https://anonymous.4open.science/r/ambimod-4D5B/.Source: REPRODUCIBILITY STATEMENT · p. 10
• The statement points readers to dataset and target-holdout definitions in Section 3 and primary scores in Section 4.The accompanying records preserve configuration, per-target scores, and hyperparameter choices used to generate summaries.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

• The study analyzes existing perturbation datasets rather than collecting new human-subject data, and its laboratory-context results do not establish clinical efficacy or safety.Source: ETHICS STATEMENT · p. 10

A.1 PROOF OF THE CONDITIONAL COVERAGE BOUND

• The conditional coverage bound concerns prediction of the population mean; scoring against an independent noisy response adds that response’s error variance.The proof chooses a nearest training embedding, and the theorem assumes a fixed representation and finite response-specific Lipschitz constant that it does not estimate.Source: §A.1 · p. 13

A.2 SHRINKAGE TOWARD A KNOWN SHARED RESPONSE

• With independent zero-mean errors, the expected squared-error minimizer for shrinkage toward a known shared response is a⋆=∥s∥2^2/(∥s∥2^2+Vq), when the denominator is positive.If both terms are zero, every mixture weight has the same risk.Source: §A.2 · p. 13

A.3 ESTIMATED SHARED RESPONSES AND SHARED CONTROLS

• When the shared response is estimated, the squared-error expansion includes cross-moments among its estimation error, the prediction error, and the evaluation error.These cross-moments must be retained when prediction and reference share observations or response estimates share control cells.Source: §A.3 · p. 13

A.4 GENE-PERMUTATION INVARIANCE

• Distances based only on marginal distributions of response-coordinate values are unchanged by permuting gene coordinates, including the one-dimensional Wasserstein and energy distances used in the screen.Distances on joint distributions of cell vectors are not covered by this invariance statement.Source: §A.4 · p. 14

B.1 RESPONSE CONSTRUCTION AND TARGET ELIGIBILITY

• The dataset protocols define responses, eligibility, gene sets, and target splits separately for K562, RPE1, and Norman.K562 uses five partitions per target and 1,000 selected genes; RPE1 has 1,089 targets and 4,063 common genes with six splits; Norman retains 105 targets and 5,045 genes across three splits.Source: §B.1 · p. 14

B.2 REPRESENTATIONS AND READOUTS

• Readouts use sequence-derived embeddings, training-only PCA transformations, and ridge, kernel-ridge, nearest-embedding, mean, zero, and response-network predictors.The six full-training RPE1 splits select ridge α=3.1623 and kernel-ridge (α, γ)=(1, 0.5); Table 2 gives kernel-ridge search grids.Source: §B.2 · pp. 14–15, Table 2 · p. 15

B.3 FIXED-TEST TRAINING SUBSAMPLING

• Five nested training prefixes (87, 218, 436, 653, and 871 targets) are evaluated against the same 218 test targets within each RPE1 split.Five repeats are run per fraction across five additional splits; repeats at 100% reuse the same training targets.Source: §B.3 · p. 15

B.4 SCORING AND UNCERTAINTY

• Direction agreement scores genes by descending absolute observed response, with a shared within-target ordering across predictors and half credit for zero predictions.The top-2% rule selects max(20, round(0.02G)) genes at scoring time.Source: §B.4 · p. 15
• Bootstrap intervals quantify target-sampling variation conditional on evaluated splits and fitted models, not uncertainty from resampling cells, controls, or the full training procedure.The analysis uses paired target-split records and target-cluster resampling; stored bootstrap-tail summaries are not permutation-test probabilities.Source: §B.4 · pp. 15–16

B.5 INDEPENDENT-REPLICATE CALIBRATION

• K562 calibration mixes a cohort-mean reference with replicate-specific references and evaluates the mixtures against independent partitions.The cohort mean is a calibration device, not a training mean from a particular target holdout; the displayed sensitivity ratio is descriptive.Source: §B.5 · p. 16

B.6 SOFTWARE AND REPRODUCIBILITY

• The robustness return records software versions, locked configurations, input manifests, per-target scores, hyperparameter receipts, and scripts identifying each evaluation.Supporting tables cover primary RPE1 analyses and predictor results, the fixed-test coverage curve, and independent-replicate K562 calibration.Source: §B.6 · p. 16, C SUPPORTING RESULTS · p. 17, Table 7 · p. 19

C SUPPORTING RESULTS

• Tables 3–6 report primary RPE1 and coordinated predictor results, while Tables 7–8 cover fixed-test coverage and independent-replicate calibration.Table 3 includes 1,308 records from 792 targets in the all-six analysis and 1,090 records from 724 targets in the five-additional analysis.Source: C SUPPORTING RESULTS · p. 17, Table 3 · p. 17, Table 7 · p. 19

C.1 ABLATION OF THE AUXILIARY PENALTY

• Removing the auxiliary penalty restores residual–truth correlation (+0.202 median, positive in all six splits), but its +0.00008 paired effect [−0.00001, 0.00017] does not reach one additional correct gene.The reported verdict is unchanged.Source: §C.1 · p. 19, Table 9 · p. 20
• Removing early stopping raises the amplitude ratio without a detectable effect, so early stopping is not the cause of the reported result.The ablation compares the frozen configuration with a capacity arm without early stopping.Source: §C.1 · p. 19, Table 9 · p. 20
• The spectral-normalization argument gives a soft four-layer Lipschitz bound; observed residual amplitudes are about 1–2%, versus a bound of about 3–6%.The bound follows from the architecture rather than from the data; the ablation is RPE1-only.Source: §C.1 · p. 19, Table 9 · p. 20

D COORDINATED CROSS-DATASET COMPARISON

• 2.61 [1.60, 3.65] percentage points for kernel ridge and 1.66 [0.74, 2.61] for ridge are the RPE1-minus-K562 top-2% gain differences.The comparison uses the same score formula and primary method family across all three datasets.Source: Table 10 · p. 21, D COORDINATED CROSS-DATASET COMPARISON · p. 21
• RPE1 combines positive paired gains with a training-coverage response, K562 readouts lie near the training mean, and Norman’s wider interval includes zero.Norman’s comparison covers CRISPRa with 49 unique evaluation targets, so that regime is reported as unresolved rather than negative evidence.Source: D COORDINATED CROSS-DATASET COMPARISON · p. 21