PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic Image Oversampling for Rare-Class Representation Learning in Deep Facial Phenotyping

Miles Liao, Zhanliang Wang, Hongzhuo Chen, Kai Wang

arXiv:2610.23730v1cs.CV

TL;DR

Scarce, unevenly distributed patient photographs constrain learning facial variation for rare genetic disorders, and synthetic images may not improve recognition of real patients. The study tests disease-specific synthetic oversampling across image-based and multimodal training setups. Mean recognition accuracy improves in both, but responses differ across models and disorders.

  • Problem

    Patient images are scarce and unevenly distributed across disorders, limiting observations of disease-associated facial variation and constraining deep facial phenotyping.

    Source: §I · p. 1

  • Method

    The study adds disease-specific synthetic facial images to a fixed real training set and evaluates increasing augmentation levels in image-based and multimodal models.

    Source: §I · p. 2, §V · p. 6

  • Results

    Mean recognition accuracy improves in both training setups, with modest gains for GestaltMatcher-Arc and continued gains across tested levels for GestaltMML.

    Source: §I · p. 2

  • Takeaways & Limitations

    Synthetic oversampling is a supported option for limited-data facial phenotyping, but its amount should be assessed against disease-level outcomes and the intended recognition objective.

    Source: §I · p. 2, §V · p. 6

    GestaltMML improvements cannot be attributed solely to synthetic image quantity because real images use clinical and placeholder text, while synthetic images use only placeholder text.

    Source: §V · p. 6

Abstract

from arXiv · show

Recent deep facial phenotyping methods can support the recognition of genetic syndromes, but patient images are often scarce and unevenly distributed across disorders. Synthetic facial images offer a way to supplement the training data without collecting additional patient photographs. Their usefulness, however, depends on whether they improve recognition of real patients: generated images may introduce artifacts or repeat features already represented in the training set. It is therefore necessary to examine both how much synthetic augmentation to use and which disorders benefit. We investigate disease-specific synthetic oversampling using images from the Phenotypic Disease Image Database to augment a subset of the GestaltMatcher Database. We compare increasing augmentation levels in an image-based model, GestaltMatcher-Arc, and a multimodal model, GestaltMML, while keeping the real training and patient-disjoint evaluation partitions fixed. Repeated training runs assess overall and disease-specific accuracy, and an exploratory embedding analysis examines changes in class compactness and separation. Augmentation improves mean recognition accuracy in both training setups, although gains remain modest in GestaltMatcher-Arc and continue across the tested levels in GestaltMML. These aggregate improvements are accompanied by uneven disease-level responses, including reduced accuracy for some disorders. In the embedding analysis, disorders with fewer real training images tend to gain more class separation, but geometric changes do not consistently track classification performance. Together, the findings support synthetic oversampling for limited-data facial phenotyping and emphasize the need to assess its amount and benefit for individual disorders within each training setup.

I. INTRODUCTION

Unevenly distributed, scarce patient images can limit learning facial variation for rare genetic disorders. This study tests whether disease-specific synthetic oversampling improves recognition of real patients across image-based and multimodal models.

I. INTRODUCTION

  • Sparse syndrome classes may represent only a narrow portion of disease-associated facial variation, limiting how well learned features generalize to other patients.Collecting additional photographs for rare conditions is difficult, motivating ways to supplement disease-specific training examples while evaluating on real patients.Source: §I · p. 1
  • Existing imbalance methods can reweight examples or alter learning and prediction, but they do not add observations of facial variation.Generated images could add disease-specific examples, but artifacts, repeated features, or source-data biases mean downstream recognition must establish their usefulness.Source: §I · p. 1
  • PDIDB synthetic images augment a fixed real GMDB training set, with nested augmentation levels tested in GestaltMatcher-Arc and GestaltMML.The real partitions are patient-disjoint and fixed; synthetic images are used only for training, and evaluation reports overall and disease-specific accuracy.Source: §I · pp. 1–2
  • Synthetic augmentation raises mean recognition accuracy in both training setups, with modest gains for GestaltMatcher-Arc and continued gains across tested levels for GestaltMML.Disease-level responses are uneven, and the best overall-accuracy level does not always give the highest mean per-class accuracy.Source: §I · p. 2
  • Exploratory GestaltMatcher-Arc analysis associates fewer real training images with larger gains in class separation, but geometric changes do not consistently track classification performance.The association does not establish that smaller class size causes greater augmentation benefit.Source: §I · p. 2

A. Deep Facial Phenotyping

  • GestaltMML combines facial images with clinical information, while this study examines how synthetic facial images affect both multimodal and image-based recognition.DeepGestalt and GestaltMatcher provide image-based facial phenotyping and patient-matching approaches.Source: A. Deep Facial Phenotyping · p. 2

B. Long-Tailed Recognition and Synthetic Oversampling

Long-tailed recognition methods address class imbalance through changes to learning or prediction, while synthetic oversampling adds generated examples. The study evaluates synthetic images alongside fixed real-image partitions and checks potential overlap with evaluation images.

B. Long-Tailed Recognition and Synthetic Oversampling

  • Long-tailed methods adjust example influence or prediction scores, whereas synthetic oversampling adds generated examples to underrepresented classes.The cited approaches include class reweighting, class-dependent margins, focal loss, decoupled training, logit adjustment, and residual correction.Source: B. Long-Tailed Recognition and… · p. 2
  • PDIDB supplies disease-conditioned StyleGAN3 facial images, whose usefulness is assessed through nested augmentation levels on a fixed real-image evaluation partition.Disease-specific accuracy and embedding geometry complement aggregate performance; synthetic artifacts, repeated features, and source-data biases may limit usefulness.Source: B. Long-Tailed Recognition and… · p. 2

A. Study Design and Dataset

  • The GMDB subset contains 1,847 real images across 10 disorders, with a fixed patient-disjoint split of 1,289 training and 558 evaluation images.Training classes range from 312 Cornelia de Lange images to 41 for 22q11.2 deletion, a largest-to-smallest ratio of 7.6; partitions stay fixed across models, conditions, and seeds.Source: A. Study Design and Dataset · p. 2
  • Potential synthetic-to-evaluation overlap was screened using exact and perceptual hashes plus nearest-neighbor cosine similarity in GestaltMatcher-Arc embeddings.Embedding similarities were calibrated against real-training-to-evaluation nearest-neighbor similarities.Source: A. Study Design and Dataset · p. 2

B. Synthetic Image Augmentation

The study progressively augments fixed real training data with nested synthetic subsets and compares both models across augmentation conditions and training seeds.

B. Synthetic Image Augmentation

  • PD25–PD100 add increasing portions of the available disorder-specific synthetic pool to the real-image Baseline.The conditions are built from PDIDB synthetic facial images generated with a disease-conditioned StyleGAN3 model.Source: B. Synthetic Image Augmentation · p. 2
  • Nested synthetic subsets remain fixed across training seeds, while the study design compares GestaltMatcher-Arc and GestaltMML under the augmentation conditions.The real partitions and synthetic subsets are fixed; both models use three training seeds per condition.Source: Figure 1 · p. 3
  • Table I reports fixed patient-disjoint real Train and Eval partitions alongside the synthetic image counts added under PD25–PD100.Baseline contains real training images only.Source: Table I · p. 3

C. Model Training

  • Both models use the same real partitions and synthetic subsets across three training seeds, which vary training stochasticity rather than the selected images.The seeds control random initialization and data shuffling.Source: C. Model Training · p. 3

1) GestaltMatcher-Arc:

GestaltMatcher-Arc uses a frozen pretrained face backbone with trainable projection and classifier layers, and evaluates synthetic oversampling alongside class-frequency loss weighting. Classification metrics are summarized across seeds on the real-image evaluation partition.

1) GestaltMatcher-Arc:

  • GestaltMatcher-Arc freezes its pretrained ResNet-50 backbone and trains a 512-dimensional feature projection and 10-class classifier.Training uses Adam with separate learning rates for the projection and classifier.Source: 1) GestaltMatcher-Arc · p. 3
  • Synthetic oversampling is tested with class-frequency-weighted cross-entropy, and Baseline and PD50 are also repeated with unweighted cross-entropy as an ablation.The class weight uses each class's training count and the smallest training count.Source: 1) GestaltMatcher-Arc · p. 3

2) GestaltMML:

  • GestaltMML pairs each real image with clinical-text and placeholder-text examples, but synthetic images only with placeholders because clinical metadata are unavailable.This yields 2,578, 3,011, 3,444, 3,876, and 4,305 training pairs for Baseline, PD25, PD50, PD75, and PD100, respectively.Source: 2) GestaltMML · p. 3

D. Classification Evaluation

  • The selected checkpoint maximizes overall Top-1 accuracy on the real-image evaluation partition, which is also used to report classification performance.Classification metrics are means and sample standard deviations across three seeds; disease-level differences are paired against Baseline within each seed.Source: D. Classification Evaluation · pp. 3–4
  • Overall Top-1 accuracy weights evaluation images equally, while mean per-class Top-1 accuracy weights classes equally.Top-3 and Top-5 accuracy count a prediction as correct when the true label appears among the corresponding number of highest-ranked predictions.Source: D. Classification Evaluation · pp. 3–4

E. Embedding Analysis

The exploratory embedding analysis compares Baseline and PD50 representations using cosine-distance measures of global and disease-level class structure.

E. Embedding Analysis

  • Seed-11 Baseline and PD50 runs were compared on the same real evaluation images using L2-normalized, 512-dimensional embeddings.PD50 was selected after classification results because it had the highest mean overall Top-1 accuracy across training seeds.Source: E. Embedding Analysis · p. 4
  • The mean cosine-distance silhouette score summarizes global class structure, using within-class distance and the nearest other-class mean distance for each image.The passages define a(i) as the mean distance to images in the same class and b(i) as the smallest mean distance to another class.Source: E. Embedding Analysis · p. 4
  • Disease-level measures capture centroid compactness, nearest-centroid separation, and their ratio; lower compactness means tighter clusters and higher separation means more distant class centroids.All quantitative measures were calculated in the original embedding space; PCA and UMAP were used only for visualization.Source: E. Embedding Analysis · p. 4

F. Statistical Analysis

The analysis measures relative embedding-metric changes from Baseline to PD50 and tests their association with real training class size.

F. Statistical Analysis

  • Relative change was calculated for each embedding metric from Baseline to PD50.The equation expresses the change as a percentage relative to the Baseline value.Source: F. Statistical Analysis · p. 4
  • Spearman rank correlation with two-sided p-values assessed associations between real training class size and relative metric changes.The analyses covered 10 disorders and one training seed and were interpreted as exploratory associations.Source: F. Statistical Analysis · p. 4

IV. EXPERIMENTS

The experiments compare nested synthetic-augmentation levels in GestaltMatcher-Arc and GestaltMML on fixed patient-disjoint partitions. Mean overall accuracy rose with augmentation in both setups, with different response patterns.

A. Experimental Setup

  • The experiments compared Baseline and four nested augmentation conditions (PD25–PD100) across three seeds, with fixed patient-disjoint partitions for both models.Augmentation effects were compared within each model because their architectures and training procedures differ.Source: A. Experimental Setup · p. 4

B. Overall Classification Performance

  • GestaltMatcher-Arc mean overall Top-1 accuracy rose from 89.19 ± 0.92% at Baseline to 90.20 ± 0.58% at PD50, while GestaltMML rose from 68.82 ± 1.53% at Baseline to 74.31 ± 0.90% at PD100.Arc differences among augmented conditions were small relative to seed variation; MML mean overall accuracy increased across tested levels.Source: B. Overall Classification Performance · pp. 4–5
  • Arc mean per-class Top-1 accuracy peaked at PD25 (89.15 ± 0.80%), rather than at the PD50 overall-accuracy peak, and its paired PD50 gain was similar with and without class-frequency weighting.The paired PD50 gain was +1.02 ± 1.72 percentage points without weighting and +1.02±1.49 points with weighting.Source: B. Overall Classification Performance · p. 5

C. Disease-Specific Performance

Disease-level accuracy responses were uneven, and exploratory embedding comparisons showed geometric changes that varied across disorders and measures.

C. Disease-Specific Performance

  • GestaltMatcher-Arc disease-specific accuracy varied across augmentation conditions and training seeds, so higher mean overall accuracy did not imply improvement for every disorder.Source: C. Disease-Specific Performance · p. 5
  • At PD100, GestaltMML mean Top-1 accuracy improved for seven disorders but decreased for Rubinstein-Taybi, Kabuki, and Angelman.The largest gains were KBG (+16.05 percentage points), Williams-Beuren (+14.63), and Smith-Magenis (+12.90).Source: C. Disease-Specific Performance · p. 5

D. Embedding-Space Analysis

  • In the seed-11 Baseline–PD50 comparison on 558 real evaluation images, silhouette score rose from 0.228 to 0.267, while nearest-centroid separation increased for nine of ten disorders.The silhouette increase was 16.9%; Cornelia de Lange was the only disorder with reduced separation (−9.86%).Source: D. Embedding-Space Analysis · p. 5
  • Compactness improved for eight disorders, while the separation-to-compactness ratio increased for nine; Figure 5 shows a shared UMAP projection, but quantitative results use the original embedding space.Smith-Magenis and 22q11.2 deletion had slightly increased mean distance to their class centroids despite increased nearest-centroid separation.Source: D. Embedding-Space Analysis · p. 5

E. Association With Real Training Class Size

The analysis associates smaller real training classes with larger relative gains in nearest-centroid separation, while the discussion cautions that representation geometry does not consistently explain recognition outcomes. The two training setups respond differently to added synthetic images, supporting setup-specific assessment of augmentation amount.

E. Association With Real Training Class Size

  • Across ten disorders, real training class size was inversely associated with relative nearest-centroid separation change (Spearman ρ = −0.707, p = 0.022).In the seed-11 analysis, disorders with fewer real training images tended to show larger relative separation gains at PD50.Source: E. Association With Real Training… · p. 5
  • Compactness change correlated at ρ = 0.527 (p = 0.117), while separation-to-compactness ratio change correlated at ρ = −0.588 (p = 0.074).These are exploratory associations across disorders in one training run and do not establish that smaller class size causes greater augmentation benefit.Source: E. Association With Real Training… · p. 5

F. Discussion

  • GestaltMatcher-Arc gains modestly with little variation across augmented conditions, whereas GestaltMML continues to improve across tested levels.Because the models differ in backbones, trainable parameters, optimization, and input construction, the comparison does not identify which component explains their responses.Source: F. Discussion · p. 5
  • The class-size association concerns centroid separation, not direct recognition benefit, and the geometric results alone cannot explain accuracy changes across repeated runs.Cornelia de Lange syndrome improved in mean classification accuracy under some conditions while its separation decreased in the seed-11 PD50 analysis.Source: F. Discussion · p. 5
  • The evaluation covers ten disorders, one synthetic image source, and one patient partition; checkpoint selection and reporting share that partition.The representation analysis uses one seed and a condition selected after examining classification results; clinical utility has not been established.Source: F. Discussion · p. 5

V. CONCLUSION

Disease-specific synthetic oversampling improves mean recognition accuracy in both evaluated training setups, but responses vary by disorder and augmentation level. The authors recommend assessing outcomes for the intended recognition objective and further testing stability and generalizability.

V. CONCLUSION

  • Synthetic oversampling improves mean recognition accuracy in both setups, with modest gains for GestaltMatcher-Arc and continued gains across tested levels for GestaltMML.The augmentation level with the highest overall accuracy does not always have the highest mean per-class accuracy.Source: §V · p. 6
  • The authors recommend evaluating augmentation against the intended recognition objective and disease-level outcomes; geometric changes do not establish the mechanism behind accuracy changes.Further evaluation should use separate validation and test partitions, external patient cohorts, and alternative synthetic subsets and generators.Source: §V · p. 6
  • Because real GestaltMML images used clinical and placeholder text while synthetic images used placeholder text only, gains cannot be attributed solely to synthetic image quantity.The authors propose controlled comparisons with matched training budgets and image-text ablations to distinguish image quantity from optimization and input-composition effects.Source: §V · p. 6

REPRODUCIBILITY STATEMENT

  • Training, evaluation, and analysis code, experiment configurations, and aggregate results are available in the GitHub repository; restricted patient data and checkpoints are not redistributed.Experimental details are provided in the Methods and Appendix.Source: REPRODUCIBILITY STATEMENT · p. 7

AI USAGE STATEMENT

  • The authors used ChatGPT 5.6 and Claude Sonnet 5.5 for manuscript editing, organization, formatting, and code assistance, then reviewed the generated material.They state that they independently verified results, analyses, references, and scientific claims and take responsibility for the final manuscript.Source: AI USAGE STATEMENT · p. 7

APPENDIX A COMPLETE CLASSIFICATION RESULTS

The appendix captions specify the classification metrics, class-weighting sensitivity comparison, and representation-geometry measures reported in three tables; the supplied passages do not include their numerical results.

APPENDIX A COMPLETE CLASSIFICATION RESULTS

  • Table II reports sample-weighted Top-k accuracy and equally class-averaged Mean Top-1 across ten disorders, with means and standard deviations over three seeds.Its colors mark the highest- and lowest-observed means within each architecture and metric.Source: Table II · p. 8

APPENDIX B CLASS-WEIGHTING SENSITIVITY ANALYSIS

  • Table III compares GestaltMatcher-Arc under class-frequency weighting and defines Δ as the paired seed-wise PD50-minus-Baseline difference in overall Top-1 accuracy.Values are mean and sample standard deviation across three training seeds.Source: Table III · p. 8

APPENDIX C GESTALTMATCHER-ARC EMBEDDING ANALYSIS

  • Table IV defines compactness as mean cosine distance to the class centroid and separation as cosine distance to the nearest foreign-class centroid.Lower compactness and higher separation are better; Sep./Comp. is their ratio, and changes are relative percentages.Source: Table IV · p. 8

Related papers

Loading 2610.23730v1…