PaperlayerView exact revision on arXiv

Source-linked AI summary

Two-Stage Mesh Deep Learning for Automated Tooth Segmentation and Landmark Localization on 3D Intraoral Scans

Tai-Hsien Wu, Chunfeng Lian, Sanghee Lee, Matthew Pastewait, Christian Piers, Jie Liu, Fang Wang, Li Wang, Chiung-Ying Chiu, Wenchi Wang, Christina Jackson, Wei-Lun Chao, Dinggang Shen, Ching-Chang Ko

arXiv:2109.11941v4cs.CV

TL;DR

Manual tooth segmentation and landmark annotation are time-consuming and expertise-dependent, while dental landmark localization remains relatively underexplored. TS-MDL segments teeth and uses per-tooth regions for landmark heatmap regression. The authors report improved segmentation accuracy and speed, and better landmark-localization results for two-stage methods than single-stage methods.

  • Problem

    Manual segmentation and landmark annotation are time-consuming and expertise-dependent, and relatively few studies address tooth landmark localization.

    Source: §I · pp. 1–2

  • Method

    TS-MDL uses iMeshSegNet to segment teeth, then applies PointNet-Reg to per-tooth regions to regress landmark heatmaps.

    Source: §I · p. 2

  • Results

    iMeshSegNet segmentation refined by graph cut achieved DSC 0.964±0.054, and two-stage methods consistently outperformed single-stage methods for landmark localization.

    Source: 1) Tooth Segmentation Results · p. 6, 2) Landmark Localization Results · p. 6

  • Takeaways & Limitations

    The authors report that the straightforward two-stage strategy leads to promising accuracy in anatomical landmark localization.

    Source: §I · p. 2

    The dataset contains only 136 samples, and surface-only scans can leave landmarks in overlapped areas difficult to locate accurately.

    Source: D. Limitations and Future Work · p. 8

Abstract

from arXiv · show

Accurately segmenting teeth and identifying the corresponding anatomical landmarks on dental mesh models are essential in computer-aided orthodontic treatment. Manually performing these two tasks is time-consuming, tedious, and, more importantly, highly dependent on orthodontists' experiences due to the abnormality and large-scale variance of patients' teeth. Some machine learning-based methods have been designed and applied in the orthodontic field to automatically segment dental meshes (e.g., intraoral scans). In contrast, the number of studies on tooth landmark localization is still limited. This paper proposes a two-stage framework based on mesh deep learning (called TS-MDL) for joint tooth labeling and landmark identification on raw intraoral scans. Our TS-MDL first adopts an end-to-end \emph{i}MeshSegNet method (i.e., a variant of the existing MeshSegNet with both improved accuracy and efficiency) to label each tooth on the downsampled scan. Guided by the segmentation outputs, our TS-MDL further selects each tooth's region of interest (ROI) on the original mesh to construct a light-weight variant of the pioneering PointNet (i.e., PointNet-Reg) for regressing the corresponding landmark heatmaps. Our TS-MDL was evaluated on a real-clinical dataset, showing promising segmentation and localization performance. Specifically, \emph{i}MeshSegNet in the first stage of TS-MDL reached an averaged Dice similarity coefficient (DSC) at \textcolor[rgb]{0,0,0}{$0.964\pm0.054$}, significantly outperforming the original MeshSegNet. In the second stage, PointNet-Reg achieved a mean absolute error (MAE) of $0.597\pm0.761 \, mm$ in distances between the prediction and ground truth for $66$ landmarks, which is superior compared with other networks for landmark detection. All these results suggest the potential usage of our TS-MDL in orthodontics.

I. INTRODUCTION

Automating tooth segmentation and landmark annotation matters for orthodontic treatment planning, but anatomical variation and incomplete scans make these tasks challenging. TS-MDL pairs tooth segmentation with landmark heatmap regression, addressing the limited work on dental landmark localization.

I. INTRODUCTION

  • Automated segmentation and landmark annotation support patient-specific orthodontic planning, but manual work is tedious and expertise-dependent, while scan and tooth variation complicate automation.Challenges include abnormal, disarranged, or missing teeth and incomplete scans.Source: §I · p. 1
  • Deep learning methods have improved dental surface labeling, but few address landmark localization, which is sensitive to tooth shape and local geometric detail.Landmark counts also vary across tooth positions.Source: §I · p. 1
  • TS-MDL combines tooth segmentation with landmark heatmap regression, using segmentation to define per-tooth regions for landmark localization.The framework builds on the correlation between a tooth’s landmarks and its local geometry.Source: §I · p. 2

A. Deep Learning-based 3D Dental Mesh Labeling

  • CNN-based dental segmentation methods typically convert meshes or point clouds to regular image-like inputs, losing fine-grained geometry, while later GNN methods label raw dental surfaces.Prior GNN approaches include tooth labeling and instance segmentation.Source: A. Deep Learning-based 3D Dental Mesh… · p. 2
  • MeshSegNet uses graph-constrained modules to integrate multi-scale local context, but large adjacency-matrix computations impose heavy computational requirements.The paper identifies this computational burden as a key limitation of its predecessor.Source: A. Deep Learning-based 3D Dental Mesh… · p. 2

B. Learning-based Landmark Localization

Prior landmark-localization approaches include coordinate regression and heatmap-based methods, but dental mesh landmark detection—especially directly on 3D data with graph neural networks—remains limited. The study uses annotated intraoral scans with tooth and landmark targets.

B. Learning-based Landmark Localization

  • Dental-specific landmark methods can be limited to predefined features, while medical-image approaches do not address irregular dental meshes.Some earlier methods identify only their specifically defined dental feature.Source: B. Learning-based Landmark Localization · pp. 2–3
  • Direct coordinate regression from high-dimensional inputs is technically challenging, while Gaussian heatmaps recast landmark localization as a regression task.Medical-imaging work has also combined heatmap regression with spatial configuration or segmentation.Source: B. Learning-based Landmark Localization · p. 3
  • Landmark detection on dental meshes remains limited, particularly methods that work directly on 3D data with graph neural networks.Existing approaches include point-cloud keypoint identification and mesh-to-flat-torus conversion.Source: B. Learning-based Landmark Localization · p. 3

A. Dataset

  • The dataset comprises 136 patients’ raw upper intraoral scans, with 14 teeth and 66 landmarks per scan except in a few cases with missing teeth.The landmarks are used in orthodontic superimposition and calculation of tooth movement.Source: A. Dataset · p. 3
  • Two experienced orthodontists manually annotated and checked each scan to establish ground truth for segmentation and localization.The study’s learning targets are 14 teeth and 66 anatomical landmarks.Source: A. Dataset · p. 3, Figure 2 · p. 3

B. TS-MDL

TS-MDL first segments teeth on a downsampled scan with iMeshSegNet, then uses the segmentation to crop tooth-specific ROIs for landmark heatmap regression with PointNet-Reg.

B. TS-MDL

  • TS-MDL segments the downsampled scan with iMeshSegNet, then crops each tooth’s ROI from the original mesh for PointNet-Reg landmark heatmap regression.The two stages correspond to tooth segmentation and heatmap regression.Source: B. TS-MDL · pp. 3–4
  • Each tooth ROI is processed by its own PointNet-Reg to regress landmark heatmaps, narrowing localization from the full scan to the corresponding tooth region.The paper states this ROI restriction improves localization efficiency and accuracy.Source: B. TS-MDL · p. 4
  • iMeshSegNet forms its input by downsampling scans from about 100,000 cells to 10,000 and encoding each cell with 15 normalized features.The features comprise vertex coordinates, a cell normal, and relative position on the whole surface.Source: 1) iMeshSegNet for Tooth Segmentation · p. 4
  • EdgeConv replaces MeshSegNet’s symmetric average pooling to avoid the large adjacency-matrix computations and memory use that arise at scale.The two N × N adjacency matrices and their matrix multiplications drive the stated computational and memory burden.Source: 1) iMeshSegNet for Tooth Segmentation · p. 4
  • The first graph-constrained module uses EdgeConv to propagate neighboring-cell context, while the second uses k-NN graphs with k = 6 and k = 12 for multi-scale context.The resulting feature matrices are concatenated and fused.Source: 1) iMeshSegNet for Tooth Segmentation · p. 4
  • iMeshSegNet combines local-to-global features and predicts per-cell probabilities for 14 teeth, then graph-cut refinement addresses isolated false predictions and non-smooth boundaries.The refined labels are projected from the downsampled mesh to the original scan using an RBF-kernel SVM.Source: 1) iMeshSegNet for Tooth Segmentation · pp. 4–5

2) Point-Reg for Landmark Localization:

PointNet-Reg localizes landmarks by regressing heatmaps from each tooth’s mesh, reflecting that a landmark’s location is determined by its associated tooth’s shape.

2) Point-Reg for Landmark Localization:

  • A tooth’s landmarks are localized from its own mesh because their positions are determined by that tooth’s shape.The paper gives mesial and distal contact points as examples, describing them as contact-area centers usually in the upper third of most teeth’s crowns.Source: 2) Point-Reg for Landmark Localization · p. 5
  • PointNet-Reg replaces PointNet’s final softmax with sigmoid to output per-cell Gaussian landmark heatmaps, then selects each heatmap’s maximum-valued cell as the predicted coordinate.The selected centroid cells lie on the original intraoral scan.Source: 2) Point-Reg for Landmark Localization · p. 5

1) Data Augmentation:

Training data are expanded with geometric transformations, and landmark targets are represented as Gaussian heatmaps centered on landmark locations.

1) Data Augmentation:

  • Each of 136 original and flipped scans is augmented with random translation, rotation, and rescaling to generate 20 new cases per scan.The scans include annotations for 66 landmarks; each axis has a 50% probability of the stated transformation ranges.Source: 1) Data Augmentation · p. 5

2) Heatmap Generation:

  • Ground-truth landmark heatmaps use Gaussian functions centered at landmark coordinates, with maximum height 1.0 and RMS width 5.0 mm.The heatmap value is defined using the distance between a cell centroid and the landmark.Source: 2) Heatmap Generation · p. 5

3) Training Procedure:

Training uses separate objectives and sampled mesh inputs for segmentation and landmark regression. Inference transfers segmentation back to the original resolution before processing individual-tooth ROIs; experiments use a 65:15:20 train-validation-test split and evaluate both tasks with specified metrics.

3) Training Procedure:

  • Stage 1 minimizes generalized Dice loss and Stage 2 minimizes MSE, using AMS-Grad Adam; training samples 9,000 cells for iMeshSegNet and 3,000 for PointNet-Reg.The inputs are sampled from scans downsampled to approximately 10,000 cells and original-resolution ROIs of approximately 3,050 cells, respectively.Source: 3) Training Procedure · p. 5

4) Inference Procedure:

  • At inference, the downsampled Stage 1 segmentation is transferred to the original resolution, then 14 tooth ROIs feed separate PointNet-Reg models to predict 66 landmark heatmaps.The experiments localize 66 landmarks on 14 upper-dental-model teeth; extension to more landmarks, teeth, or lower models is described as straightforward, with in-house tests and data not shown.Source: 4) Inference Procedure · p. 5

A. Experimental Setup

  • 136 samples were split into training, validation, and test sets at a 65:15:20 ratio, with training and validation augmented.Segmentation was evaluated using DSC, sensitivity, PPV, and HD; landmark localization was evaluated using MAE.Source: A. Experimental Setup · pp. 5–6

B. Competing Methods

iMeshSegNet significantly outperformed MeshSegNet on all four reported segmentation metrics, and was faster in training and inference. The landmark comparison includes two-stage ROI-based methods and single-stage whole-scan methods.

B. Competing Methods

  • Landmark comparisons included two-stage methods that apply per-tooth ROI regression and single-stage methods that regress heatmaps from the whole dental model.For segmentation, iMeshSegNet was compared with MeshSegNet; the authors note that MeshSegNet had previously been compared with other state-of-the-art methods,.Source: B. Competing Methods · p. 6

1) Tooth Segmentation Results:

  • iMeshSegNet significantly outperformed MeshSegNet on DSC, SEN, PPV, and HD; after graph-cut refinement, its DSC was 0.964±0.054.The per-tooth comparison also reports DSC and HD for the two methods.Source: 1) Tooth Segmentation Results · p. 6, Table II · p. 6, Figure 5 · p. 6
  • iMeshSegNet trained in 253.6 seconds per epoch for 30 samples, roughly 20 times faster than MeshSegNet, and segmented an unseen scan in about 0.62 seconds, approximately 8.6 times faster.The comparison used the same implementation strategy and computational environment.Source: 1) Tooth Segmentation Results · p. 6

2) Landmark Localization Results:

The two-stage methods consistently outperformed the single-stage methods for landmark localization, and iMeshSegNet+PointNet-Reg also outperformed the alternative two-stage model. The strongest errors were observed on first molars, where incomplete posterior scans may leave less information.

2) Landmark Localization Results:

  • The two-stage methods consistently outperformed the single-stage methods, and iMeshSegNet+PointNet-Reg outperformed iMeshSegNet+iMeshSegNet-Reg in MAE.The authors suggest that ROI localization reduces the value of the second iMeshSegNet-Reg model’s localized-feature advantage, while the small dataset may also contribute to overfitting and weaker test generalization.Source: 2) Landmark Localization Results · pp. 6–7
  • For incisor, canine, premolar, and molar landmarks, iMeshSegNet+PointNet-Reg had MAEs of 0.51, 0.72, 0.51, and 0.70 mm, respectively, versus 0.65, 0.71, 0.86, and 0.96 mm in Ref..The authors describe these results as comparable to, and slightly better than, Ref., while noting the absence of a public intraoral-scan dataset and mesh-segmentation benchmark for fair comparison.Source: 2) Landmark Localization Results · p. 7
  • First-molar landmark errors were higher; the authors note that incomplete posterior scanning may leave molars with less complete information.The paper also reports that molar segmentation accuracy is lower than that of other teeth.Source: 2) Landmark Localization Results · p. 7
  • The qualitative comparison identifies iMeshSegNet+PointNet-Reg as having the best landmark prediction among the four methods, consistent with the quantitative comparison.Figure 8 shows different teeth against ground truth; its green and purple circles distinguish anterior from posterior landmarks.Source: 2) Landmark Localization Results · p. 7, Figure 6 · p. 7

V. DISCUSSION

For tooth segmentation on 3D intraoral scans, static adjacency performed better than dynamic adjacency on DSC, SEN, and PPV. The authors motivate static adjacency by the geometric distribution and arrangement of human teeth.

A. Static Adjacency vs. Dynamic Adjacency

  • Static adjacency, defined in input Euclidean space, outperformed dynamic adjacency on DSC, SEN, and PPV for tooth segmentation on 3D intraoral scans.The authors argue static adjacency is more reasonable for contextual feature extraction given human teeth’s geometric distribution and arrangement; Table V compares the two strategies.Source: A. Static Adjacency vs. Dynamic… · p. 7, Table V · p. 8
  • Dynamic adjacency is defined in high-dimensional feature spaces to model nonlocal dependencies between points or cells along the network’s forward path.This is the approach attributed to Wang et al..Source: A. Static Adjacency vs. Dynamic… · p. 7

B. Clinically Acceptable Errors

TS-MDL localized 66 landmarks with a mean absolute error of 0.597 ± 0.761 mm; ceiling analysis points to landmark regression refinement as the best improvement target, while incomplete surface scans constrain predictions in overlapped regions.

B. Clinically Acceptable Errors

  • Errors above 0.5 mm can still be clinically acceptable because a passing ABO case has a score of 27 points or less.The simplest computational criterion checks whether predicted landmarks lie within a circle of 2 mm radius; ABO treats 0.5 mm as an awareness distinction and penalizes deviations over 0.5 mm and 1 mm.Source: B. Clinically Acceptable Errors · p. 8

C. Ceiling Analysis

  • 0.087 mm improvement in overall MAE followed perfect segmentation as Stage 1 input, suggesting landmark-regression refinement is the best future improvement target.The analysis particularly emphasizes the 0.5 mm accuracy requirement.Source: C. Ceiling Analysis · p. 8, Table VI · p. 8

D. Limitations and Future Work

  • The dataset contains only 136 samples, and surface-only scans can leave overlapped landmarks inaccurately predicted when the mesh is incomplete.The authors propose dental mesh repair at the end of Stage 1 as future work to reconstruct incomplete areas.Source: D. Limitations and Future Work · p. 8

VI. CONCLUSION

  • 0.597 ± 0.761 mm MAE was achieved by TS-MDL for localizing 66 landmarks, supporting its potential use in orthodontic applications.The conclusion describes the framework as using iMeshSegNet segmentation masks as individual ROIs and PointNet-Reg to predict landmark heatmaps.Source: §VI · p. 8

Related papers

Loading 2109.11941v4…