PaperlayerView exact revision on arXiv

Source-linked AI summary

TomoTransformer: Towards a Foundation Model for CT Reconstruction

AmirEhsan Khorashadizadeh, Benjamín Béjar

arXiv:2609.37605v2cs.CVcs.LG

TL;DR

Sparse-view CT reconstruction models are often tied to acquisition protocols, detector dimensions, or narrow training distributions. TomoTransformer predicts missing views using local tokens in a disentangled back-projection space. The authors report strong benchmark comparisons and zero-shot transfer to real experimental nanoscale brain data.

  • Problem

    Many supervised sparse-view CT methods are tied to projection geometry, detector dimensions, or narrow training distributions, limiting flexibility across acquisition settings.

    Source: §1 · p. 2

  • Method

    TomoTransformer uses self-attention over local filtered projections in a disentangled back-projection space to predict missing views across varying angles, view counts, and detector sizes.

    Source: §1 · p. 2

  • Results

    TomoTransformer significantly outperforms ViewTrans and matches or exceeds protocol-specific baselines, while generalizing across anatomies, materials, and resolutions.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The pretrained model transfers zero-shot to real experimental nanoscale brain tomography data collected at the Swiss Light Source.

    Source: §1 · p. 3

Abstract

from arXiv · show

Supervised deep learning has advanced sparse-view tomographic reconstruction. However, conventional models, which typically map filtered back-projection (FBP) images or sinograms to clean reconstructions, are brittle under distribution shifts. Because they require retraining whenever projection counts and angles, detector resolutions, or data distributions change, their deployment in real-world applications remains limited. To address this, we introduce TomoTransformer, a transformer-based architecture that treats each \textit{local} filtered projection as an individual token and predicts missing views via self-attention. Crucially, TomoTransformer operates in a \emph{back-projection space} that separates projections across spatial locations, making view interpolation geometrically well-posed and invariant to detector size. This design yields a single foundation model that can process any number of input projections, at arbitrary angular locations and detector dimensions, and query any number of target angles without retraining. Trained on a large-scale dataset spanning diverse medical CT anatomies and natural images, TomoTransformer generalizes effectively across anatomies, materials, and resolutions. Extensive evaluations on several benchmark sparse-view datasets show that TomoTransformer significantly outperforms concurrent multi-purpose models like ViewTrans and matches or exceeds strong protocol-specific baselines, while remaining fully agnostic to the number of input and target projections. Furthermore, the model demonstrates robust zero-shot generalization on real experimental nanoscale brain data collected from an X-ray synchrotron, showcasing its practical utility for real-world applications.

1 Introduction

Sparse-view CT models are often tied to acquisition geometry, detector dimensions, or narrow training distributions. TomoTransformer predicts missing views in a local disentangled back-projection space and is reported to generalize across protocols and data settings.

1 Introduction

  • Existing supervised methods commonly require fixed projection counts and angular ranges, fixed image and detector sizes, and narrow training distributions.For example, a model trained on 60-view sinograms may require retraining or architectural changes for a 90-view acquisition.Source: §1 · p. 2
  • TomoTransformer significantly outperforms ViewTrans and matches or exceeds protocol-specific baselines trained for each acquisition protocol.It remains agnostic to the number of input and queried projections.Source: §1 · pp. 2–3
  • TomoTransformer treats each local filtered projection as a token and predicts missing views via self-attention in a disentangled back-projection space.The authors state that spatially decoupling projections makes interpolation geometrically well-posed and invariant to detector dimensions.Source: §1 · p. 2
  • A single trained model handles varying projection counts, angular configurations, and detector sizes without retraining or architectural changes.The model is trained on a large-scale dataset spanning medical CT anatomies and natural images.Source: §1 · p. 2
  • The pretrained model transfers zero-shot to real experimental nanoscale brain tomography data collected at the Swiss Light Source.The data are described as nanoscale brain ptychography X-ray tomography.Source: §1 · pp. 2–3
  • A model pretrained on clean images achieves zero-shot denoising on unseen noisy projections without fine-tuning.The authors present this as evidence of practical utility for real-world imaging tasks.Source: §1 · p. 3

2 Related work

Related work includes supervised reconstruction networks and generative-prior methods. The cited approaches differ in their handling of acquisition variability, computational demands, and detector or data-distribution scope.

2 Related work

  • Supervised CNN, unrolled-solver, and transformer reconstructors remain tied to fixed acquisition protocols and detector resolutions, requiring retraining when projection counts or angles change.Transformer methods capture global sinogram structure, while unrolled methods embed the Radon operator to improve data consistency and generalization.Source: §2 · p. 3
  • Iterative model-based methods use hand-crafted priors such as total variation and wavelet sparsity to partially suppress artifacts, at the cost of slow optimization.Supervised deep learning is described as offering faster inference and higher reconstruction quality.Source: §2 · p. 3
  • Diffusion methods decouple a learned generative prior from the forward operator, theoretically allowing one model to accommodate any acquisition protocol.They require hundreds to thousands of score evaluations per reconstruction and use a prior specific to the imaged distribution and detector size.Source: §2 · p. 3

3 Computed tomography

The section introduces parallel-beam CT measurements and filtered back-projection. It then describes how individual angle-specific back-projections are separated in the representation used by TomoTransformer.

3 Computed tomography

  • In parallel-beam CT, measurements are acquired at discrete projection angles and detector positions, forming columns of a sinogram.The attenuation field is illuminated by parallel X-ray beams at different angles.Source: §3 · pp. 3–4
  • FBP filters each sinogram column along the detector dimension, interpolates the filtered values, and back-projects them along sinusoidal trajectories in the reconstruction domain.Each resulting single-view back-projection is evaluated on the reconstruction grid; averaging across angles gives the FBP reconstruction.Source: §3 · p. 4
  • Equation (3) isolates the contribution of one view at each reconstruction coordinate, and stacking these single-view back-projections across angles forms the disentangled representation.The paper names this representation the disentangled back-projection space and uses it to establish the TomoTransformer architecture.Source: §3 · p. 4

4 TomoTransformer: a flexible reconstruction model

TomoTransformer predicts missing views in a disentangled back-projection space, using angle-conditioned tokens and local patches to support flexible, scalable reconstruction. Its geometric tokenizer compresses each patch to a feature that the decoder turns back into an output patch.

4 TomoTransformer: a flexible reconstruction model

  • The model addresses four requirements: flexibility in input views and angles, flexibility in detector dimensions and resolution, scalability, and generalization across heterogeneous data.Its design combines disentangled-space prediction, angle-conditioned tokens, local patches, and large-scale heterogeneous training.Source: §4 · p. 4
  • Because views share a coordinate system in back-projection space, missing-view prediction becomes spatially aligned interpolation, and self-attention handles variable input tokens and queried views.The figure describes processing varying numbers of input projections and generating arbitrary numbers of local projections.Source: §4 · pp. 4–5, Figure 1 · p. 5
  • Angle embeddings are defined for real-valued angles, enabling unseen target angles; attention also uses a learned function of pairwise angular differences.The encoder processes existing tokens, while the decoder combines their representations with a learnable tensor for queried tokens.Source: §4 · pp. 5–6
  • Local P × P patches reduce memory and computation, decouple tokens from full image and detector dimensions, and provide more training patches per image.The authors also state that local prediction reduces task complexity and improves generalization.Source: §4 · pp. 4, 6

4.1 Geometric tokenizer and decoder

  • A single-view back-projection patch inherits parallel-stripe structure, so the geometric tokenizer exploits that structure rather than encoding the full two-dimensional patch directly.The stripe pattern is determined by a one-dimensional signal swept along the stripe direction.Source: §4.1 · p. 6
  • The tokenizer samples a local back-projection patch along a diagonal near the stripe normal to extract a concise one-dimensional feature for encoding.The decoder maps a predicted embedding back to a one-dimensional feature and sweeps it perpendicular to the queried angle to reconstruct the output patch.Source: §4.1 · p. 6

5 Experiments

Across fixed and variable sparse-view protocols, TomoTransformer is evaluated on heterogeneous CT and natural-image data, as well as unseen noise and real nanoscale brain measurements. Results show strong comparisons with existing methods and zero-shot transfer without fine-tuning.

5.1 Experimental setup

  • Training uses about 650,000 CT slices and natural images at native resolutions; evaluation uses simulated parallel-beam data and PSNR and SSIM against dense-view targets.The training set includes 570,000 CT slices from 1,340 volumes and 80,000 natural images; evaluation metrics are computed on reconstructed images.Source: §5.1 · p. 7

5.2 Fixed sparse-view reconstruction

  • On average, TomoTransformer outperforms competing baselines in the fixed 64-to-256 protocol, while its variable-sparsity model performs comparably without retraining.The reported PSNR and SSIM values are averaged over 32 test slices per dataset.Source: §5.2 · p. 7

5.3 Varied sparse-view reconstruction

  • At both 128 and 64 measured views, TomoTransformer outperforms ViewTrans when reconstructing a 256-view target, with the visual difference especially stark at 64 views.At 64 views, ViewTrans produces swirling artifacts absent from the reference; small sinogram prediction errors can be amplified into structured image artifacts under heavy undersampling.Source: §5.3 · pp. 8–9, Figure 4 · p. 9

5.4 Zero-shot denoising

  • Predictions from a noiseless-trained TomoTransformer yield cleaner reconstructions than noisy FBP inputs, while complementary re-masking outperforms blind-spot re-masking by denoising measured and unmeasured views.When measured noisy views are included as usual, their residual noise dominates and lowers PSNR.Source: §5.4 · pp. 9–10, Figure 5 · p. 10

5.5 Real experimental data

  • Without fine-tuning, TomoTransformer recovers fine cellular structure from real nanoscale brain projections, and querying 256 or 512 views from the same 128 inputs further improves quality.The comparison uses a dense 512-view FBP reference; the pretrained ViewTrans cannot be applied to these measurements, while the baselines show hallucinated texture and blocking artifacts.Source: §5.5 · p. 10, Figure 6 · p. 11

6 Conclusion

TomoTransformer is a transformer architecture for sparse-view CT reconstruction that disentangles projection angles from local back-projection patches. The conclusion emphasizes its robustness across datasets and sparsity levels, including out-of-distribution data without fine-tuning.

6 Conclusion

  • TomoTransformer is a transformer-based architecture for sparse-view CT reconstruction.Its representation disentangles projection angles and local back-projection patches.Source: §6 · p. 11
  • The model achieves robust performance across a variety of datasets and sparsity levels.The conclusion also reports effective handling of out-of-distribution data without fine-tuning.Source: §6 · p. 11
  • Future work will explore adapting the architecture to other tomographic modalities, including cone-beam CT.Source: §6 · p. 11

A TomoTransformer architecture

TomoTransformer processes local patches from single-view back-projections as angle-conditioned tokens. Its geometric attention incorporates known angular relationships, while its shared tokenizer and detokenizer and fixed-size patch processing support varied input geometries.

A TomoTransformer architecture

  • Each token is a 32 × 32 patch cropped from one single-view back-projection and reduced to a 46-dimensional feature before projection to the 512-dimensional embedding.The patch is sampled along its maximum-τ diagonal chord at 46 equispaced points; the same tokenizer-detokenizer pair is shared across patch locations and projection views.Source: A TomoTransformer architecture · p. 16
  • Fourier features of each viewing angle are projected and added to encoder and decoder inputs through separately gated positional embeddings.The two learnable gates are initialized to 0 and 0.5, respectively.Source: A TomoTransformer architecture · p. 16
  • The MAE-style backbone encodes observed tokens and decodes a complete sequence, filling target positions with a learnable mask token.Both encoder and decoder use four transformer blocks of width 512 with eight attention heads.Source: A TomoTransformer architecture · p. 16
  • TomoTransformer handles irregularly spaced real projections by conditioning on each token’s actual angle, unlike fixed-geometry baselines in the 64-to-256 comparison.The Figure 7 comparison applies those baselines to 617 real projections that are not uniformly spaced.Source: Figure 7 · p. 17
  • The geometric attention mechanism adds a learned function of pairwise angular differences to attention logits, slightly enhancing reconstruction quality and convergence speed.The final layer of the shared two-layer GELU bias network is initialized to zero, so training begins from standard dot-product attention.Source: A TomoTransformer architecture · pp. 16, 18, Table 5 · p. 18

B Data simulation and training

Training uses simulated parallel-beam projections converted into view-wise back-projections, with both fixed and varied sparse-view sampling protocols. The model learns from unobserved target views using an ℓ1 token loss.

B Data simulation and training

  • Training slices are normalized, circularly masked, projected with ODL, ramp-filtered, and back-projected view by view before patch extraction.Each iteration samples 32 axial slices at native resolution from the dataset mixture.Source: B Data simulation and training · p. 18
  • The fixed protocol uses 64 uniformly spaced inputs from a 256-angle grid, while the varied protocol resamples view counts and perturbs angles during training.In the varied protocol, full-view counts range from 128 to 512 and angle jitter has standard deviation 0.05 times nominal angular spacing.Source: B Data simulation and training · pp. 18–19
  • The training loss is the ℓ1 distance between predicted and ground-truth tokens at unobserved target views.Models are trained with AdamW for 10^6 iterations using a cosine-decayed learning rate and gradient clipping.Source: B Data simulation and training · p. 19

C Inference

Inference divides images of arbitrary size into fixed-size patches, predicts only unobserved target views, and averages patch reconstructions. Measured views remain unchanged, preserving exact data consistency.

C Inference

  • Inference tiles an arbitrary-size image with non-overlapping patches and shifts boundary patches inward, averaging any overlaps.Because the network operates on fixed-size patches, inference is independent of image dimension and detector resolution; it was applied to 768 × 768 brain data after training up to 512 × 512.Source: C Inference · p. 19
  • Predicted tokens replace only unobserved target positions, while measured views are left untouched to preserve exact data consistency.Source: C Inference · p. 19
  • The final patch reconstruction averages across all target tokens and across patches.Source: C Inference · p. 19

D.1 Image-domain post-processing baselines

The image-domain post-processing baselines map sparse-view FBP patches to dense-view counterparts, enabling comparison with TomoTransformer under shared data and reconstruction settings. Their architectures and parameter counts were adjusted to similar capacity, with some implementation-specific differences.

D.1 Image-domain post-processing baselines

  • All five image-domain baselines map sparse-view FBP patches to dense-view counterparts, sharing TomoTransformer’s training data, ramp filter, and back-projection operator.They are tested patch-wise with the same tiling, stride, and overlap-averaging scheme.Source: §D.1 · p. 19
  • U-Net and DRUNet retain encoder–decoder designs, while NAFNet uses NAFBlocks and Restormer uses MDTA/GDFN transformer blocks.Their reported parameter counts are 12.13 M for U-Net, 12.76 M for DRUNet, 11.77 M for NAFNet, and 11.98 M for Restormer.Source: §D.1 · pp. 19–20
  • CTformer uses a token-to-token dilated performer tokenizer and predicts a residual subtracted from the input, with its configuration expanded to 10.72 M parameters.The baselines’ capacities were adjusted to approximately 12.71 M parameters where feasible.Source: §D.1 · pp. 19–20
  • The baselines use 32 × 32 patches and TomoTransformer’s optimization recipe, but train with image-domain ℓ1 loss rather than per-projection supervision on unobserved views.CTformer uses a 64 × 64 tokenizer grid because its public implementation hard-codes that size.Source: §D.1 · p. 20

D.2 ViewTrans baseline

ViewTrans is reimplemented as a sinogram-domain, view-as-token transformer with geometry-aware attention and a differentiable FBP head. Its configuration is adapted to parallel-beam geometry and the benchmark resolution, then trained under TomoTransformer’s schedule for comparison.

D.2 ViewTrans baseline

  • ViewTrans treats each full sinogram row across detector bins as a token and uses five pre-LayerNorm encoders, beginning with geometry-aware self-attention.Its input includes pseudo-full views generated by linear interpolation, and a differentiable FBP head is trained with image-domain ℓ2 loss.Source: §D.2 · p. 20
  • For parallel-beam geometry, its attention uses normalized detector coordinates in place of the original fan-angle encoding, while retaining angular and observed/interpolated maps.Queries and keys concatenate layer-normalized sinogram features with the geometry maps.Source: §D.2 · p. 20
  • At detector width M = 362, ViewTrans decouples inner attention width from M and projects features back to RM for its residual skip connection.The adapted model has 13.06 M parameters, close to TomoTransformer’s 12.71 M.Source: §D.2 · pp. 20–21
  • ViewTrans uses the same ramp filter and back-projection operator as TomoTransformer, with predictions supervised against dense-view FBPs from that operator.This places both models in an identical reconstruction domain.Source: §D.2 · p. 21
  • The baseline is trained on LoDoPaB-CT with TomoTransformer’s dynamic sparse-view schedule and training length.Its schedule uses cosine decay, Adam at 5 × 10^-4, gradient clipping at 1.0, and 8 slices per gradient step for 10^6 iterations.Source: §D.2 · p. 21

Related papers

Loading 2609.37605v2…