Source-linked AI summary
Robust and generalizable embryo selection based on artificial intelligence and time-lapse image sequences
Jørgen Berntsen, Jens Rimestad, Jacob Theilgaard Lassen, Dang Tran, Mikkel Fly Kragh
TL;DR
Evidence remains limited on how embryo-selection models generalize to new clinics and across clinical conditions, and on biological support for their predictions. This study evaluates an automated deep-learning model using embryo time-lapse sequences; it reports KID-embryo AUC 0.67 and generalization across clinics and subgroups, with performance comparable to or better than the manual model.
Problem
Evidence on embryo-selection model generalization to new clinics and subgroup performance across conditions is limited, and the paper identifies biological justification as an open concern.
Source: Introduction · p. 4Method
The authors developed a fully automated deep-learning embryo-selection model using 115,832 time-lapse sequences, without embryologist assessment or annotation.
Source: Conclusion · p. 21Results
The model had AUC 0.67 for KID embryos and was reported to perform better than the state-of-the-art morphokinetic KIDScore D5 v3 model, with generalization to held-out clinics and across age, insemination, and incubation strata.
Source: Conclusion · pp. 21–22Takeaways & Limitations
The model's reported performance was obtained without embryologist assessment or annotation.
Source: Conclusion · p. 21A high AUC indicates sorting within a cohort and does not imply a high implantation rate.
Source: Clinic hold-out test · p. 19
Abstract
from arXiv · showhide
Assessing and selecting the most viable embryos for transfer is an essential part of in vitro fertilization (IVF). In recent years, several approaches have been made to improve and automate the procedure using artificial intelligence (AI) and deep learning. Based on images of embryos with known implantation data (KID), AI models have been trained to automatically score embryos related to their chance of achieving a successful implantation. However, as of now, only limited research has been conducted to evaluate how embryo selection models generalize to new clinics and how they perform in subgroup analyses across various conditions. In this paper, we investigate how a deep learning-based embryo selection model using only time-lapse image sequences performs across different patient ages and clinical conditions, and how it correlates with traditional morphokinetic parameters. The model was trained and evaluated based on a large dataset from 18 IVF centers consisting of 115,832 embryos, of which 14,644 embryos were transferred KID embryos. In an independent test set, the AI model sorted KID embryos with an area under the curve (AUC) of a receiver operating characteristic curve of 0.67 and all embryos with an AUC of 0.95. A clinic hold-out test showed that the model generalized to new clinics with an AUC range of 0.60-0.75 for KID embryos. Across different subgroups of age, insemination method, incubation time, and transfer protocol, the AUC ranged between 0.63 and 0.69. Furthermore, model predictions correlated positively with blastocyst grading and negatively with direct cleavages. The fully automated iDAScore v1.0 model was shown to perform at least as good as a state-of-the-art manual embryo selection model. Moreover, full automatization of embryo scoring implies fewer manual evaluations and eliminates biases due to inter- and intraobserver variation.
1 Vitrolife A/S, Jens Juuls Vej 20, 8260 Viby J, Denmark
The listed affiliation is Harrison AI in Sydney, Australia.
- Harrison AI is listed at 60 Margaret St, Sydney NSW 2000, Australia.Source: §1 · p. 1
Introduction
Embryo selection models increasingly automate IVF assessment, but evidence on their generalization across clinics and biological support remains limited. This study develops a fully automated deep-learning model intended to address those gaps.
Introduction
- Prior embryo-selection models have targeted outcomes including blastocyst development, genetic status, gestational sacs, and live birth.Some newer methods directly estimate transfer outcomes from images, including time-lapse sequences used to predict fetal heartbeat.Source: Introduction · p. 3
- Earlier studies report improved outcomes with time-lapse and morphokinetic selection compared with standard incubation, while other work calls for in-house model validation.Source: Introduction · p. 3
- Morphokinetic analysis can improve assessment consistency and reduce inter- and intra-observer variation, yet embryo selection remains challenging in busy IVF laboratories.Automatic scoring methods have progressed from traditional computer vision to deep learning.Source: Introduction · p. 3
- Limited evaluation of generalization across clinical practices motivates testing models across clinics and conditions.Deep learning models can overfit to training data, and the paper emphasizes evaluation on varied practices, incubation conditions, and insemination methods.Source: Introduction · p. 4
- The paper aims to develop a fully automated embryo-selection model that generalizes, is robust, and can be supported by biological evidence.Deep-learning models are described as black boxes, raising discussion about biological justification for computer-generated selection algorithms.Source: Introduction · p. 4
Materials and methods
The study used retrospective, de-identified embryo data from multiple IVF clinics, trained a time-lapse deep-learning model, and evaluated clinic generalization and morphokinetic relationships.
Description of data
- The final dataset comprised 115,832 embryos labeled fetal-heartbeat positive or negative and was randomly split 85% for training and 15% for independent testing.Unknown and pending embryos were excluded; data were collected retrospectively from 18 clinics and provided in de-identified form.Source: Description of data · pp. 5, 7
Deep learning model
- The model used 128 time-lapse frames sampled one hour apart, with I3D convolutional features pooled and passed to a bidirectional LSTM and fully connected output layers.Training included sequence and image augmentation and predicted fetal heartbeat plus an auxiliary discard output.Source: Deep learning model · pp. 7–9
Clinic hold-out test
- Clinic generalization was assessed with leave-one-clinic-out testing on 12 clinics, training each model on the remaining clinics.Clinics included in this analysis each had more than 250 KID embryos.Source: Clinic hold-out test · p. 9
Morphokinetic annotations
- Morphokinetic annotations included pronuclear fading, cleavage timings, time to blastocyst, and inner-cell-mass and trophectoderm grades.Direct cleavage was defined by less than five hours between tPNf and t3 for DC1-3, or between t3 and t5 for DC2-5; KIDScore required annotations of PNs, t2, t3, t5, tB, ICM, and TE.Source: Morphokinetic annotations · p. 9
Statistical analysis
- AUCs were estimated from ROC curves, differences between AUCs were tested with DeLong’s test, and morphokinetic group differences were tested with Mann–Whitney–Wilcoxon.A p-value below 0.05 was considered significant.Source: Statistical analysis · p. 10
Results
On an independent test set, the model sorted KID embryos with AUC 0.67 and the whole embryo cohort with AUC 0.95. Clinic hold-outs and subgroup analyses showed generally similar sorting performance, while scores varied with morphokinetic and morphological features.
Results
- AUC was 0.67 (95% CI 0.64–0.69) for KID embryos and 0.95 (95% CI 0.95–0.96) for the whole cohort in the independent test set.The test set contained 17,249 embryos, including 2,212 KID embryos.Source: Results · p. 11, Table 2 · p. 11
Sub-group analysis
- Across age groups, KID-embryo AUCs ranged from 0.63 to 0.69, with the lowest value for ages 30–34; AUCs were 0.69 for ICSI, 0.67 for IVF, 0.65 for D5, and 0.66 for D6.Cryopreserved transfers had AUC 0.65, significantly below the 0.69 for fresh transfers.Source: Sub-group analysis · p. 12
Clinic hold-out test
- Clinic hold-out KID-embryo AUCs ranged from 0.60 to 0.75, with clinics 1 and 4 significantly below the other clinics in the test data.The clinic comparison included clinics with more than 250 KID embryos.Source: Clinic hold-out test · pp. 12–13, Table 3 · p. 12
Correlation with morphology and morphokinetic annotations
- Model scores differed across direct-cleavage groups and decreased with slower blastocyst development and lower ICM and TE grades.For blastocysts, no-DC scores differed from both DC1-3 and DC2-5, while the two direct-cleavage groups did not differ significantly; the annotated-subset comparison with KIDScore found AUC 0.92 versus 0.89 for the whole cohort, and 0.67 versus 0.66 for KID embryos with no significant difference.Source: Correlation with morphology and… · pp. 13, 15, Comparison with the KIDScore D5 v3 model · p. 15
Discussion
The model’s reported performance depends on the embryo-ranking task, and subgroup and clinic hold-out analyses indicate where its sorting capability varies. Its scores also track established morphology and development parameters, although those parameters were not training targets.
Discussion
- The whole-cohort AUC was 0.95 versus 0.93 for IVY, while KID-embryo sorting was harder than sorting across the wider range of the whole cohort.The authors say the relevant measure depends on whether the task is sorting all embryos or pre-screened transfer candidates, and propose reporting both.Source: Discussion · p. 17
Sub-group analysis
- Only the cryopreserved subgroup had a significantly lower AUC; the authors suggest vitrification, warming, and endometrium preparation may make implantation sorting more difficult.They also note that including age may improve overall performance without improving sorting within one patient’s embryo cohort; age-subgroup sorting for STORK and AIR E remained undocumented.Source: Sub-group analysis · p. 18
Clinic hold-out test
- Clinic 1 and 4 had lower AUCs, which the authors relate to transferred KID cohorts likely dominated by homogeneous top-quality blastocysts.A high AUC does not itself imply a high implantation rate; AUC differences can also reflect cohort and clinical-practice differences.Source: Clinic hold-out test · p. 19
Correlation with morphology and morphokinetic annotations
- Scores decreased with direct cleavages, and the decrease was smaller for blastocysts than for the whole embryo cohort.The authors connect the smaller blastocyst effect to a self-correction hypothesis, while noting the score was not trained on these biological parameters.Source: Correlation with morphology and… · pp. 19–20
- Scores were associated with time to blastocyst and with ICM and TE grades, decreasing as grades declined.The authors conclude that the model’s scores correlate with established embryo development and morphology parameters.Source: Correlation with morphology and… · pp. 19–20, Conclusion · p. 22
Model comparisons
- The model had significantly higher AUC for whole-cohort sorting than KIDScore D5 v3, but no significant advantage for KID embryos.The authors note that iDAScore v1.0 was trained on discarded embryos whereas KIDScore D5 v3 was trained only on transferred KID embryos.Source: Model comparisons · p. 21
Conclusion
- iDAScore v1.0 was reported to outperform KIDScore D5 v3 overall, while the KID-embryo comparison showed no significant difference.The comparison describes the automated model as better overall than the manual, annotation-based model.Source: Conclusion · p. 21, Model comparisons · p. 21