Source-linked AI summary
Machine learning approach for early detection of autism by combining questionnaire and home video screening
Halim Abbas, Ford Garberson, Eric Glover, Dennis P Wall
TL;DR
Existing autism screening is limited by costly, time-intensive tools and by machine-learning data that differ between controlled clinical training and home application. This paper combines a short parent questionnaire and analyst-tagged home videos, using clinical ADI-R and ADOS data with feature and decision strategies for these constraints. In a clinical study of 162 children, the resulting screeners significantly improved accuracy over established tools.
Problem
Affordable, reliable early autism detection is needed because existing tools can be costly and time-intensive, while clinical training data are sparse, imbalanced, and mismatched with home-screening conditions.
Method
The paper trains questionnaire and video machine-learning classifiers from ADI-R and ADOS data, combines their outputs, and uses feature engineering plus inconclusive decisions to address home-application limitations.
Results
The new screening tools significantly improved performance over established autism screeners, with the combined assessment further boosting performance on the clinical study sample.
Takeaways & Limitations
Machine learning can improve the effectiveness of affordable autism screeners when methods address clinical-to-home data shifts and other clinical-data challenges.
Takeaways & Limitations
Short unsupervised home videos differ from 45-minute expert ADOS observations, increasing unobserved or inconclusive behaviors and reducing reliability of assessed severity or frequency.
Abstract
from arXiv · showhide
Existing screening tools for early detection of autism are expensive, cumbersome, time-intensive, and sometimes fall short in predictive value. In this work, we apply Machine Learning (ML) to gold standard clinical data obtained across thousands of children at risk for autism spectrum disorders to create a low-cost, quick, and easy to apply autism screening tool that performs as well or better than most widely used standardized instruments. This new tool combines two screening methods into a single assessment, one based on short, structured parent-report questionnaires and the other on tagging key behaviors from short, semi-structured home videos of children. To overcome the scarcity, sparsity, and imbalance of training data, we apply creative feature selection, feature engineering, and novel feature encoding techniques. We allow for inconclusive determination where appropriate in order to boost screening accuracy when conclusive. We demonstrate a significant accuracy improvement over standard screening tools in a clinical study sample of 162 children.
1 Introduction
Early autism diagnosis is important but delayed by inadequate screening capacity and tools that can be costly or time-intensive. The paper develops parent-questionnaire and home-video machine-learning screeners, combines them, and reports higher performance than conventional tools in a 162-child clinical sample.
- Motivation: Early diagnosis can improve outcomes, yet autism is typically not diagnosed before age 4 in the United States, and about 27% of cases remain undiagnosed at age 8.The passage attributes this delay primarily to insufficient screening tools and a shortage of specialists.
- Existing tools: Common autism screeners use parent- or practitioner-completed score sheets compared against predetermined thresholds.Examples include M-CHAT and CBCL.
- Existing tools: Cognoa provides a machine-learning screening alternative used by more than 150,000 parents through web and smartphone applications.Most users are parents of children aged 18–30 months.
- Contribution: The paper presents a short parent questionnaire and a home-video screener designed to be reliable, cost-effective, and completable within minutes.The video screener uses analysts’ identification of behaviors in two or three short videos recorded by parents on mobile devices.
- Contribution: The questionnaire screener uses patterns from the gold-standard ADI-R while replacing its lengthy professional interview with a shorter unsupervised parent questionnaire.ADI-R contains 93 multipart questions and can be costly and time-consuming to administer.
- Contribution: The video screener uses behavioral patterns from ADOS to classify behaviors observed by analysts in short parent-recorded home videos.ADOS is a standardized clinical examination administered by trained clinicians.
- Results: The combined screener achieved higher accuracy than either individual screener, while the new tools surpassed M-CHAT and CBCL on 162 clinically diagnosed children.The paper also introduces an inconclusive outcome to improve accuracy for conclusive classifications at the cost of lower coverage.
2 Data
Training data came from multiple repositories of ADOS and ADI-R records, while clinical validation used children assessed at autism centers with standardized instruments and psychologist-confirmed diagnoses.
- Training data: Training data combined ADOS and ADI-R score sheets from multiple repositories covering boys and girls aged 18–84 months.Because the repositories were autism-imbalanced, control data were supplemented with additional ADI-R interviews.
- Clinical validation: The clinical validation sample included 230 children aged 18–72 months assessed at three United States autism centers.Each child received ADOS and appropriate standard screeners, with diagnosis determined by a licensed psychologist.
- Clinical validation: Parents of 162 validation-sample children completed the short questionnaire and submitted the required home videos.These children formed the subset used to evaluate the proposed screeners requiring mobile-device data.
- Dataset composition: Table 1 breaks down the training and clinical-validation datasets by age group and condition type.The negative class includes neurotypical children and children with developmental delays or other non-autism conditions.
3 Approach
The approach trains separate machine-learning classifiers from clinical ADI-R and ADOS data, then combines their outputs into a home-administered screening assessment while addressing degradation caused by the change from clinic to home.
- Classifier design: Two independent classifiers were trained: a parent-questionnaire model from item-level ADI-R data and a video model from ADOS score sheets with diagnostic labels.Their outputs were combined into one screening assessment.
- Deployment setting: The proposed screeners shift assessment from professional, standardized clinical administration to unsupervised home completion taking minutes.ADI-R and ADOS typically take hours and require trained professionals.
- Deployment setting: The change from clinical training conditions to home application causes data degradation and bias that are expected to reduce screening accuracy.The methodology applies adjustments intended to mitigate these effects.
3.1 Parent questionnaire
The parent-questionnaire pipeline progressively improves a generic classifier through robust feature selection, age-specific modeling, severity-aware encoding, aggregate features, and an inconclusive-result option.
- 3.1.1 Generic ML baseline variant: A generic random-forest baseline one-hot encoded 155 ADI-R columns and selected the top 20 features using feature importance.Age and gender were also included as features.
- 3.1.2 Robust feature selection variant: Sparse, small training data and correlated ADI-R questions made generic feature selection unstable across runs.Different competing feature sets could appear similarly powerful during training.
- 3.1.2 Robust feature selection variant: Robust feature selection used 100 bootstrapped selections, retained the top 30 recurring candidates, and performed a final selection run.Each iteration used a weight-balanced 90% random sample and selected 20 top features.
- 3.1.3 Age silo variant: Age-silo modeling trained separate classifiers for children younger than 4 and children aged 4 or older.The split served as a proxy for language ability and enabled developmentally specialized features.
- 3.1.4 Severity-level feature encoding variant: Severity-level encoding groups semantically meaningful answer levels while collapsing irrelevant codes, reducing sparsity and over-fitting relative to one-hot encoding.The encoding reflects increasing symptom severity and distinguishes it from non-severity response categories.
- 3.1.5 Aggregate features variant: Aggregate features summarize the selected responses with minimum, maximum, and average severity and counts by severity level.These condensations are useful for sparse, shallow, wide training data and reflect specialist ADI-R interpretation.
- 3.1.6 Inconclusive results variant: An inconclusive-result variant replaces a binary cutoff with a cutoff range or uses a second binary-classification strategy.The cutoff range trades lower effective coverage for higher accuracy among conclusive subjects.
3.2 Video
The video screener uses analyst-tagged behaviors from short home videos and adapts training to account for missing or unreliable observations. Binary presence encoding improves reliability, while injected missing values reduce reliance on unobserved behaviors.
- Video screener: Analysts rate target behaviors in two or three 1-minute home videos, and a classifier trained on ADOS item-level data predicts clinical diagnosis.The videos are semi-structured, recorded by parents on mobile phones, and analyzed for behavior presence and severity.
- Model training: The video classifier used decision forests with selected questions, inconclusive outcomes, bootstrapped tuning, stratified validation, and age-label balancing.These adjustments were designed to address sparse, imbalanced data and the shift from clinical training conditions to home screening.
- Video screener: Video screening faces a train–application mismatch because expert 45-minute ADOS observations are replaced by unsupervised short home videos.The application setting increases unobserved or inconclusive behaviors and makes severity estimates less reliable.
- Feature encoding: Collapsing severity levels into binary presence improves feature reliability but sacrifices information about symptom severity.A value of 1 indicates observed behavior; a value of 0 may reflect failure to observe it rather than true absence.
- Missing-value handling: Training samples were duplicated with selected 1-to-0 flips so random forests would be less likely to infer conclusions from feature absence.The injected zeros were calibrated so the sample-weighted positive-to-negative ratio for zero values was approximately 50% for each feature.
3.3 Combination
The questionnaire and video classifiers are combined into an overall screener by combining their numerical outputs, with age-specific models and inconclusive criteria.
- Combination: Logistic regression combines the numerical outputs of the parent questionnaire and video classifiers into a single screening assessment.Because the component methods were siloed by age group, separate combinators were trained for each age group, with optimal inconclusive criteria selected afterward.
4 Results
The study evaluates increasingly refined questionnaire classifiers and finds that the questionnaire outperforms established screeners, while combining questionnaire and video inputs further improves performance. Allowing inconclusive outcomes produces additional accuracy gains among conclusive cases.
- Training performance: Bootstrapped cross-validation metrics were reported for increasingly effective parent-questionnaire classifier variants, with the baseline expressed as a confidence interval.The interval reflects performance variation caused by unstable generic feature selection across runs.
- Clinical evaluation: Parent questionnaires were evaluated on the clinical sample using classification variants built from the paper’s feature-selection scheme.The clinical-sample metrics are reported for the questionnaire-based variants.
- Questionnaire performance: The parent-questionnaire approach outperformed established screening tools including MCHAT and CBCL on the clinical sample.The comparison is shown through ROC curves.
- Combination performance: Combining questionnaire and video classifiers further boosted performance on the clinical study sample.The combined assessment exceeded the performance of the individual screening approaches.
- Inconclusive outcomes: Allowing inconclusive determinations up to 25% of the time improved accuracy among conclusive cases.The trade-off is higher accuracy for cases receiving autism or not-autism determinations, with lower effective coverage.
5 Conclusion
The study reports significant improvement over established autism screening tools in a clinical trial and highlights pitfalls of applying machine learning clinically. Its evaluation compares questionnaire- and video-based algorithms, alone and combined, with MCHAT and CBCL baselines, including settings that allow inconclusive determinations.
- The clinical trial demonstrated a significant improvement over established autism screening tools for young children.The authors also quantified benefits from addressing important pitfalls in clinical machine-learning applications.
- ROC curves compare questionnaire-based autism screening techniques on a clinical sample of 168 children.Six children were excluded from this sample because videos were unavailable.
- ROC curves compare questionnaire-based, video-based, and combined algorithms against MCHAT and CBCL baselines.A separate evaluation allows inconclusive determinations for up to 25% of cases.
A Progressive sampling on training datasets
Progressive sampling was used to test whether training datasets contained enough data for stable machine-learning classifiers. The evaluation used bootstrapped cross-validation and AUC across increasingly large training-set proportions.
- Progressive sampling assessed whether available training data were sufficient for stable machine-learning classifiers.
- Bootstrapped cross-validation computed AUC for optimized random-forest classifiers trained on increasingly larger training-set proportions.
- The plots demonstrated stable learning of ensemble decision trees as training-set size increased.
B Differences between training and application environments
Medical screening models were trained on rigorous clinical measurements but applied to less controlled proxy data, creating environment differences that can introduce confounding bias. These biases may not be detected by training-data cross-validation alone.
- Training data came from rigorous clinical evaluations, whereas mobile screening uses proxies acquired in less controlled settings.
- The mismatch between clinical training features and application proxies is expected to considerably degrade classifier performance in mobile screening.
- Confounding biases can arise in application data through mechanisms absent from the training data.
- Cross-validation or similar analyses of training data alone cannot probe inaccuracies introduced by these application-specific biases.
C Age binned results
Results were analyzed separately for younger and older children, with figures comparing questionnaire, video, and combined algorithms against established screeners. Some age-specific analyses also restricted evaluation to children without inconclusive outcomes.
- Results were split separately for young and old children in Figures 5 through 8.
- For children under four, figures compare questionnaire, video, and combined algorithms with MCHAT and CBCL.
- For children under four, a restricted analysis excluded inconclusive outcomes while allowing at most 30% inconclusive cases.
- For children aged at least four, figures compare questionnaire and video algorithms with established MCHAT and CBCL screeners.
- For children aged at least four, a restricted analysis excluded inconclusive outcomes while allowing at most 30% inconclusive cases.