PaperlayerView exact revision on arXiv

Source-linked AI summary

Structure-Aware Sparse-View X-ray 3D Reconstruction

Yuanhao Cai, Jiahao Wang, Alan Yuille, Zongwei Zhou, Angtian Wang

arXiv:2311.10959v3eess.IVcs.CV

TL;DR

Sparse-view X-ray 3D reconstruction must capture internal structure, yet existing NeRF approaches overlook important X-ray properties. SAX-NeRF combines Lineformer and MLG sampling, and reports improvements over prior NeRF-based methods on novel view synthesis and CT reconstruction.

  • Problem

    Existing NeRF methods overlook X-ray imaging properties, limiting their ability to capture structural contents in sparse-view 3D reconstruction.

    Source: §1 · pp. 1–2

  • Method

    SAX-NeRF combines Lineformer, which models dependencies within X-ray line segments, with MLG ray sampling for projection context and geometry.

    Source: §1 · p. 2

  • Results

    SAX-NeRF outperforms prior NeRF-based methods across X3D scenes, including by 12.56 dB in PSNR over NAF and over 2.49 dB over previous NeRF-based methods on CT reconstruction.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    The paper concludes that Lineformer and MLG sampling address structural modeling in sparse-view X-ray reconstruction, with X3D covering wider application scenarios.

    Source: §5 · p. 8

Abstract

from arXiv · show

X-ray, known for its ability to reveal internal structures of objects, is expected to provide richer information for 3D reconstruction than visible light. Yet, existing neural radiance fields (NeRF) algorithms overlook this important nature of X-ray, leading to their limitations in capturing structural contents of imaged objects. In this paper, we propose a framework, Structure-Aware X-ray Neural Radiodensity Fields (SAX-NeRF), for sparse-view X-ray 3D reconstruction. Firstly, we design a Line Segment-based Transformer (Lineformer) as the backbone of SAX-NeRF. Linefomer captures internal structures of objects in 3D space by modeling the dependencies within each line segment of an X-ray. Secondly, we present a Masked Local-Global (MLG) ray sampling strategy to extract contextual and geometric information in 2D projection. Plus, we collect a larger-scale dataset X3D covering wider X-ray applications. Experiments on X3D show that SAX-NeRF surpasses previous NeRF-based methods by 12.56 and 2.49 dB on novel view synthesis and CT reconstruction. Code, models, and data are released at https://github.com/caiyuanhao1998/SAX-NeRF

1. Introduction

The introduction frames sparse-view X-ray 3D reconstruction as a low-dose problem and argues that existing NeRF methods do not adequately model X-ray internal structure. SAX-NeRF combines a line-segment transformer and structured ray sampling, and the paper reports strong results across X3D applications.

1. Introduction

  • Sparse-view X-ray reconstruction can reduce exposure in circular cone-beam scanning, and the paper studies novel view synthesis and CT reconstruction.The two tasks provide complementary 3D representations from 2D projections.Source: §1 · p. 1
  • Existing paired learning methods require projection-CT training data that is difficult to collect, and they generalize poorly across application domains.CT volumes may be inaccessible; collecting paired data is labor-intensive and harmful to health.Source: §1 · p. 1
  • Existing NeRF methods overlook X-ray penetration and attenuation, which reveal internal structures that provide clues for 3D reconstruction.Visible-light imaging relies on reflection and mainly captures external features.Source: §1 · p. 2, Figure 2 · p. 2
  • SAX-NeRF uses Lineformer to model internal dependencies along X-ray line segments and MLG sampling to capture local context and global geometry in projections.The introduction also presents X3D as a larger-scale dataset spanning wider X-ray applications.Source: §1 · p. 2
  • Over 12 dB average improvement on X3D scenes for novel view synthesis is reported for SAX-NeRF over state-of-the-art NeRF-based methods.The introduction reports this average across all scenes.Source: §1 · p. 2

2. Related Work

Related work describes neural rendering approaches for X-ray, established cone-beam CT reconstruction families, and the computational challenge of applying Transformers to X-ray rendering.

2.1. Neural Rendering

  • NeRF-based X-ray rendering methods may underperform because RGB methods do not account for the imaging differences, and MLP-based NAF has limitations capturing complex 3D and 2D structures.NeRF represents objects with an implicit function of color and volume density; later work includes finer details and feature encodings.Source: §2.1 · p. 2

2.2. Cone Beam CT Reconstruction

  • Analytical CT methods can work with hundreds of projections but struggle with sparse views, while optimization-based methods iteratively minimize an energy function and take a long time.CNN and diffusion approaches can perform well but require paired training data; the paper develops NeRF-based algorithms to avoid these restrictions.Source: §2.2 · pp. 2–3

2.3. Vision Transformer

  • Vanilla Transformers are computationally expensive with respect to input-point count, leaving their potential for X-ray neural rendering under-explored.The paper positions its work as addressing that research gap.Source: §2.3 · p. 3

3. Method

SAX-NeRF models X-ray radiodensity along rays and trains on rendered projection intensities. Lineformer captures within-ray structural dependencies, while MLG sampling targets foreground pixels and patches.

3.1. Overall Framework

  • X-ray radiodensity is mapped from 3D point positions and integrated along each ray to predict projection intensity, with training minimizing squared error against observed intensities.The rendering follows exponential attenuation; the loss compares predicted and ground-truth intensities in the training ray batch.Source: §3.1 · pp. 3–4

3.2. Line Segment-based Transfomer

  • Lineformer partitions points on each X-ray into segments and applies self-attention within each segment to model internal 3D structural dependencies.Its LS-MSA has linear complexity in the number of input points, unlike the quadratic global-attention mechanism described for vanilla Transformers.Source: §3.2 · p. 4

3.3. Masked Local-Global Ray Sampling

  • Naive scattered-pixel sampling can miss neighboring-pixel context and waste samples on background rays that do not penetrate the object.Those uninformative rays do not characterize object radiodensity and can degrade model efficiency.Source: §3.3 · p. 5
  • MLG uses a binary foreground mask, sampling patches for local context and additional foreground pixels outside those patches for global shape and geometry.The resulting batch combines patch-level and pixel-level rays.Source: §3.3 · p. 5

4. Experiment

SAX-NeRF is evaluated on X3D for sparse-view novel view synthesis and CT reconstruction, with comparisons, ablations, parameter analyses, and tests using fewer training projections. It outperforms the compared methods on both main tasks and benefits from its Lineformer and MLG components.

4.1. Experimental Settings

  • Experiments use X3D’s 15 scenes across four applications, with 50 projections for training and 50 for NVS testing per scene.X3D projections are generated from CT volumes with 3% noise over 0°–180°; experiments use 320 sampled points per X-ray and PSNR and SSIM metrics.Source: §4.1 · p. 5

4.2. Main Results

  • 12.56 dB: SAX-NeRF surpasses NAF in NVS PSNR, and it outperforms the compared methods across all scenes.It also exceeds NAF by 0.0209 in SSIM; visual results show clearer textures, finer details, and more complete geometry than previous methods.Source: §4.2 · p. 6, Table 1 · p. 6, Figure 5 · p. 6
  • Over 2.49 dB: SAX-NeRF outperforms previous NeRF-based methods in CT reconstruction, and it yields the best results on all scenes.Its gains over optimization-based and analytical methods exceed 4.92 dB and 12.13 dB, respectively; visual comparisons show sharp details while preserving smooth homogeneous regions.Source: §4.2 · pp. 6–7, Table 2 · p. 7, Figure 6 · p. 7

4.3. Ablation Study

  • 13.40 dB and 3.04 dB: jointly using LS-MSA and MLG improves the baseline on NVS and CT reconstruction, respectively.LS-MSA alone adds 9.98 and 2.65 dB, while MLG alone adds 5.54 and 1.09 dB on those tasks.Source: §4.3 · p. 8, Table 3 · p. 8
  • On bonsai, successively adding LS-MSA and MLG recovers more structural content and fine-grained texture than the baseline.The baseline loses tree-branch geometry in projections and blurs basin edges in CT slices.Source: §4.3 · p. 8, Figure 7 · p. 8
  • With M = 160 and S = 4, SAX-NeRF reaches its best performance; across tested settings, it beats the baseline by over 7.81 dB on NVS and 1.64 dB on CT.The analysis varies line-segment count M and patch size S while holding total sampled points and X-rays constant.Source: §4.3 · p. 8, Table 3 · p. 8
  • Across different numbers of training projections, SAX-NeRF reliably surpasses state-of-the-art methods by large margins on both NVS and CT reconstruction; with only 60% of projections, it still outperforms other algorithms on NVS.Figure 8 analyzes the number of training projections.Source: §4.3 · p. 8, Figure 8 · p. 8
  • Lineformer outperforms a vanilla Transformer by 5.30 dB on NVS and 1.28 dB on CT while using 3.41% of its computational complexity.The comparison replaces LS-MSA with G-MSA.Source: §4.3 · p. 8

5. Conclusion

SAX-NeRF targets sparse-view X-ray 3D reconstruction with Lineformer and MLG ray sampling, alongside the X3D dataset. Experiments on X3D show it significantly surpasses state-of-the-art algorithms on novel view synthesis and CT reconstruction.

5. Conclusion

  • Lineformer partitions each X-ray into line segments and computes self-attention within them to model 3D structural dependencies.Source: §5 · p. 8
  • MLG ray sampling combines pixel- and patch-level sampling on informative foreground regions to extract 2D geometry and contextual representations.Source: §5 · p. 8
  • X3D covers wider X-ray application scenarios, and experiments show SAX-NeRF significantly surpasses state-of-the-art algorithms on novel view synthesis and CT reconstruction.Source: §5 · p. 8

Related papers

Loading 2311.10959v3…