PaperlayerView exact revision on arXiv

Source-linked AI summary

The microscope is the mask: privileged views and labels from a cryo-ET forward model

Bogdan Toader, Kiarash Jamali, Tanmay A. M. Bharat, Sjors H. W. Scheres

arXiv:2609.04325v1cs.CVcs.LGmath.OC

TL;DR

Real cryo-ET tomograms are degraded and crowded, and annotations are missing for many small and rare proteins. CARNIVAL trains on simulator-generated paired views and privileged labels, then outperforms TomoTwin on real-tomogram classification and detection without finetuning. The evaluation is limited to the tested dataset and processing conditions, and broader assessment remains needed.

  • Problem

    Cryo-ET tomograms are degraded by low dose and restricted tilt angles, and real annotations miss small and rare proteins.

    Source: §1 · p. 1

  • Method

    CARNIVAL uses forward-model paired views and simulator-provided protein positions and identities to train nuisance-invariant dense representations.

    Source: §1 · p. 2

  • Results

    CARNIVAL outperforms TomoTwin on real-tomogram classification and detection without finetuning; on filtered tomograms, its overall F1 is 0.88 versus 0.83.

    Source: §1 · p. 2, §3 · p. 5

  • Takeaways & Limitations

    CARNIVAL’s F1 stays relatively constant as simulated block density rises, while the baseline degrades at high density.

    Source: §3 · p. 5

    Evaluation across more experimental and processing conditions is needed, along with targeted ablations of unreliable corruption axes.

    Source: §4 · p. 6

Abstract

from arXiv · show

We explore the use of simulated data for training a model for protein annotation in crowded cryo-electron tomography volumes reconstructed from images collected at limited tilt angles and severely corrupted by the measurement operator. Firstly, we leverage the corruptions imposed by the forward model to generate domain-specific augmented paired views of the exact same scene for an invariance objective integrated into the LeJEPA self-supervised training framework. Secondly, we use additional information from the simulation pipeline such as the positions and identity of proteins in the simulated volumes to inform the architecture of the model and the loss function, so that semantic information is localised at protein positions in the resulting dense feature volume. The resulting model, CARNIVAL, is evaluated without finetuning on classification and detection tasks in real tomograms, using a benchmark dataset containing multiple protein types and two tomogram processing types. We show that CARNIVAL outperforms a state-of-the-art model trained using a contrastive objective on simulated data but without forward model-based paired views or privileged information.

1 Introduction

Cryo-ET tomograms are degraded and crowded, while real annotations miss many small and rare proteins. CARNIVAL uses simulated paired views and simulator information to learn representations that outperform TomoTwin on real-tomogram classification and detection.

1 Introduction

  • Real tomograms combine low-dose noise and a restricted tilt range with dense, unlabelled cellular material, while annotations miss small and rare proteins.The paper therefore uses simulation as a training-data source at the scale required.Source: §1 · p. 1
  • CARNIVAL uses the forward model to create physically grounded paired views of the same scene and simulator information to learn nuisance-invariant representations.The missing wedge is imposed by microscope acquisition, rather than being a freely chosen masking hyperparameter.Source: §1 · p. 2
  • On real tomograms, CARNIVAL’s embeddings outperform TomoTwin on classification and detection without finetuning.TomoTwin is also trained on simulated data, but lacks forward-model-based paired views and privileged information.Source: §1 · p. 2

2 Physically-grounded views and privileged labels

The method uses simulated views and labels to train CARNIVAL to recognize proteins across acquisition and scene variation. Paired-view invariance, identity-based localization, and auxiliary segmentation use complementary information from the simulator.

2 Physically-grounded views and privileged labels

  • CARNIVAL combines same-scene measurement-corrupted views, views with changed protein orientations and backgrounds, and complete protein identity and position labels.The identity and position labels are privileged training information, absent at deployment.Source: §2 · pp. 2–3
  • The method targets invariance to modeled corruptions, not reproduction of simulated densities, and samples plausible parameter ranges without dataset-specific calibration.Coverage cannot compensate for a corruption axis missing from the forward model, and inaccurate nuisance distributions can still affect representations.Source: §2 · p. 3

2.1 Paired views

  • The forward model maps structures plus a random cellular background to tomograms through acquisition, reconstruction, and processing operators.Acquisition parameters, including tilt angles, defocus, and noise colour, are sampled during training.Source: §2.1 · p. 3
  • Three clean views preserve the scene in two cases and redraw protein orientations in the third; each is paired with a corrupted view from the forward model.The first two views draw separate acquisition parameters and backgrounds; the third shares these with the first view.Source: §2.1 · p. 3

2.2 Identity loss across scenes

  • The training objective combines paired-view invariance with scene-level identity loss, regularization, and an auxiliary segmentation loss.Scene labels support embeddings read out at protein locations and segmentation training on protein and membrane masks.Source: §2.2 · p. 4

3 Experiments

CARNIVAL is evaluated without finetuning on real tomograms under filtered and denoised processing conditions. It outperforms TomoTwin across the reported classification and detection comparisons, while its embeddings remain comparatively robust as simulated scene density rises.

3 Experiments

  • Using proteins’ ground-truth positions, CARNIVAL achieved overall F1 0.88 versus TomoTwin’s 0.83 on filtered tomograms; on denoised tomograms, CARNIVAL was unchanged while TomoTwin fell to 0.74.Source: §3 · p. 5
  • CARNIVAL leads in all six detection conditions, using weighted means of best F4 scores over threshold sweeps.On filtered TS_5_2, CARNIVAL’s reported VLP and apoferritin scores exceed TomoTwin’s, though TomoTwin leads on two other types by small margins.Source: §3 · p. 5, Table 1 · p. 5
  • As simulated block density rises, CARNIVAL’s F1 stays relatively constant while the baseline degrades at high density.The comparison changes multiple design choices, including multi-protein blocks and scene-level identity loss, so it does not isolate one variable.Source: §3 · p. 5

4 Discussion

The discussion identifies missing or unreliable corruption axes and limited evaluation as key constraints. Related work situates CARNIVAL among self-supervised learning, physical-system representations, cryo-ET annotation, and domain randomization approaches.

4 Discussion

  • Widening sampled parameter ranges cannot compensate for a corruption axis missing from the forward model, and no signal identifies that omission.The discussion distinguishes this from mis-set ranges, which wider sampling can address.Source: §4 · p. 5
  • Radiation-damage and noise-colour models are empirical approximations, and the authors do not establish how to sample an unreliable corruption axis.The chosen forms prioritize simplicity and runtime for on-the-fly GPU generation; approximate corruptions risk over-invariance.Source: §4 · p. 6
  • CARNIVAL’s capabilities still need evaluation across more experimental and processing conditions using broader benchmarks.The authors also call for targeted ablations to quantify each simulation axis.Source: §4 · p. 6

A Related work

  • Unlike random patch masking in masked modelling, CARNIVAL’s mask is the acquisition-determined missing wedge applied in Fourier space.The wedge depends on tilt angles, and arbitrary protein orientations cause different spectral losses.Source: A Related work · p. 9
  • For frozen cryo-ET specimens, the forward operator replaces time evolution as the source of paired views, and identity information surviving the instrument projection judges the latent.Source: A Related work · p. 9
  • Unlike prior cryo-ET embedding methods described here, CARNIVAL uses forward-model corruptions for paired views and generates scenes and tomograms during training.The cited methods use generic image augmentations when pairing views; stored simulated datasets bound parameter-space coverage.Source: A Related work · p. 10
  • Compared with appearance randomization, CARNIVAL’s views vary through the microscope forward model and add changes in background environment and protein orientation.Holding the clean scene fixed while redrawing parameters makes view pairs provide an invariance target.Source: A Related work · p. 10

B Model architecture and training

CARNIVAL uses an encoder, a training-only readout, and an optional segmentation head, trained jointly. Training uses multiple clean and corrupted views of shared scenes, with staged finetuning and substantial compute.

Model architecture

  • The encoder maps 64^3 input volumes to dense 64-channel features at 32^3 resolution, while a training-only readout maps protein-centered feature patches to 512-dimensional embeddings.An optional segmentation head predicts protein and membrane masks at input resolution; the full model has 9,708,706 parameters.Source: B Model architecture and training · p. 10

Training

  • Training used AdamW with a 5e-5 learning rate, 5e-4 weight decay, 1,000 warmup steps, and cosine annealing to a 5e-8 minimum.Source: B Model architecture and training · p. 10
  • Each batch sample is represented by six views: three clean tomograms and three corresponding corrupted tomograms, extracted from the same positions with small per-view jitter.The views include different backgrounds and protein orientations; the effective batch size was 96 across four GPUs.Source: B Model architecture and training · p. 11
  • The final checkpoint followed an initial 100,000-step run and eight sequential finetuning rounds of 10,000–30,000 steps each.The authors note that a future version would train from scratch using the full set of transformations throughout.Source: B Model architecture and training · p. 11
  • The initial 100,000-step training run took approximately 25 days on one node with four NVIDIA GH200 Grace Hopper Superchips.Additional finetuning extended the time on the same node linearly.Source: B Model architecture and training · p. 11

C Training data

Training scenes use foreground target proteins, smaller background proteins, and simulated membranes. The simulator creates multiple views of shared scenes with controlled changes in background, orientation, and measurement conditions.

Training data

  • The model’s foreground protein library contains 112 PDB IDs from TomoTwin’s training set, restricted to proteins larger than 100 kDa.The authors describe training on broader PDB and AlphaFold databases as a future goal.Source: C Training data · p. 11
  • Foreground protein densities are generated from atomic models, while 1,000 smaller protein volumes of up to 30 kDa supply the random crowded background.The density volumes are formed by placing Gaussians at atom coordinates and rescaling to 64^3 at 10 Å/voxel.Source: C Training data · p. 11
  • Every 50 steps, foreground proteins and a simulated membrane are placed in a slab, then three clean views vary background or protein orientation before corruption generates three paired views.The pipeline samples protein positions with Poisson disk sampling, avoids overlaps, and returns protein coordinates, identities, and protein and membrane masks.Source: C Training data · p. 12

D Full list of transformations in the measurement operator

The paper refers readers to Table 2 for the full set of transformations used to generate physically grounded tomogram augmentations.

Full list of transformations in the measurement operator

E Experiments: further details

The experiments evaluate CARNIVAL on real tomograms in filtered and denoised forms, using classification and detection measures tailored to each task. The section also reports the sampled forward-model settings and experimental resources.

E Experiments: further details

  • The public test set supplies each tomogram as a band-pass filtered reconstruction and a version additionally denoised with DenoisET.The same deposited tomograms are used at approximately 10 Å voxel spacing, without resampling, normalisation, or finetuning.Source: E Experiments: further details · p. 12
  • Detection uses F4, which weights recall 16 times more heavily than precision because missed particles cannot be recovered downstream.The authors follow the Object Identification challenge's metric for comparability.Source: E Experiments: further details · p. 12
  • Experiments used TomoTwin version 09.2023 and ran for approximately 12 hours on a cluster node with six NVIDIA GeForce RTX 5090 GPUs.The node also had an AMD EPYC 9354 processor and 512 GB RAM.Source: E Experiments: further details · p. 12
  • Table 3 reports per-protein-type F1 scores and an overall average across protein types for classification with k = 10 nearest neighbours.Source: Table 3 · p. 14

E.1 Classification

The classification experiment tests whether embeddings at known protein positions separate protein types, independent of tomogram, orientation, and surrounding material. Embeddings from all tomograms of one processing type are classified by leave-one-out nearest-neighbour voting.

E.1 Classification

  • At ground-truth positions, cosine distances test whether embeddings separate protein types regardless of tomogram, orientation, or neighbourhood material.The experiment evaluates embedding geometry with detection assumed, rather than testing detection itself.Source: §E.1 · p. 14
  • Embeddings from 121 tomograms per processing type are pooled and classified by leave-one-out k-nearest-neighbour voting with k = 10 and uniform weights.Per-type F1 scores and the macro average over six types are reported in Table 3 and Figure 2A.Source: §E.1 · p. 14

E.2 Detection

Detection is evaluated by scoring candidate locations against reference embeddings and sweeping a distance threshold to form precision-recall curves. Per-type maximum F4 scores are reported across three tomograms, each in filtered and denoised form.

E.2 Detection

  • CARNIVAL feature volumes use overlapping 64^3 blocks at stride 8, retaining central 48^3 regions and blending overlaps with a Hann window.TomoTwin uses stride 2; both methods produce feature volumes at half tomogram sampling and are scored at identical candidate positions.Source: §E.2 · p. 14
  • For each of 30 draws, five reference particles per type are sampled; local minima of nearest-template cosine-distance volumes become candidates, evaluated across 500 thresholds.Candidates within four feature voxels of a same-type ground-truth particle count as true positives, with one-to-one greedy matching; types with fewer than 10 remaining instances are excluded from figures and weighted means.Source: §E.2 · p. 14
  • Table 4 reports per-type maximum F4 over the threshold sweep, with standard deviations across 30 template draws and weighted-mean rows matching Table 1.Bold marks the higher value within each processing condition.Source: Table 4 · p. 15
  • The reported per-type F4 values are taken from draw-averaged precision-recall curves, and Figures 4–6 mark those operating points across three tomograms.The figures cover filtered and denoised versions; one tomogram lacks VLPs and β-amylase, while another is dominated by apoferritin.Source: §E.2 · p. 15, Table 4 · p. 15

E.3 Crowding

The crowding experiment compares embeddings as crowding increases in simulated tomograms. CARNIVAL's F1 remains nearly flat across the density range, while the baseline degrades at high density.

E.3 Crowding

  • Crowding is varied through minimum centre-to-centre separation across seven values, from 18 to 40 voxels, in simulated tomograms containing 2,000 proteins.Each non-overlapping 64^3 block is labelled by one plus the number of other proteins inside it.Source: §E.3 · p. 15
  • The two models use the same readout: averaging spatial feature vectors over a spherical region of radius 2 feature voxels at the feature-volume centre.Per-density macro-average F1 is computed using leave-one-out k-nearest neighbours with k = 10; the comparison keeps types represented at at least 14 density levels.Source: §E.3 · p. 16
  • CARNIVAL’s F1 is nearly flat across the density range, while the baseline degrades at high density.Both models used the same encoder backbone and were trained for 100,000 steps; the baseline differs in its one-protein blocks, whole-block readout, and lack of Lscene.Source: §E.3 · p. 16

Related papers

Loading 2609.04325v1…