Source-linked AI summary
Efficient Leakage-Free Neural Architecture Search under Leave-One-Subject-Out Evaluation
Heinke Hihn
TL;DR
LOSO evaluation makes leakage-free NAS expensive because fully nested searches scale as O(N^2). The paper introduces a block-based approach that shares NAS runs across subjects, improving BioVid accuracy from 82.79% to 83.39% while reducing parameters by up to 99.2%.
Problem
Fully nested NAS under leakage-free LOSO evaluation requires independent architecture searches for each subject and scales as O(N^2).
Method
The method partitions subjects into disjoint outer blocks, uses subject-disjoint inner folds for architecture selection, and performs final LOSO evaluation without using held-out subjects during optimisation.
Results
83.39% mean accuracy versus 82.79% for the early-fusion baseline, with parameter reductions of 76.7% to 99.2%.
Takeaways & Limitations
The approach provides a computationally feasible LOSO NAS procedure while improving the baseline and substantially reducing model size.
Takeaways & Limitations
Architectural components and parameter counts exhibit high variance, and the paper leaves architecture deployment selection and broader settings for future work.
Abstract
from arXiv · showhide
Leave-One-Subject-Out (LOSO) evaluation estimates generalisation performance for subject-based classification but makes Neural Architecture Search (NAS) computationally expensive because a fully nested implementation requires N independent architecture searches and, assuming approximately linear training cost, scales as O(N^2). We propose a leakage-free, block-based approach that shares NAS runs across subjects. On the BioVid Heat Pain dataset, our approach increased the mean accuracy from 82.79% to 83.39% while reducing the number of parameters by up to 99.2%.
1 Method
The paper addresses the O(N^2) cost of fully nested NAS under leakage-free LOSO evaluation by sharing architecture searches across disjoint subject blocks. Each block uses subject-disjoint inner validation to select an architecture before final fitting and held-out evaluation.
- Motivation: Fully nested LOSO-NAS scales as O(N^2) when NAS and fitting costs grow approximately linearly with the number of subjects.Each of N folds independently searches architectures and fits models using the remaining N−1 subjects.
- Block NAS: Block NAS partitions subjects into disjoint outer blocks and performs one architecture search per block instead of one per held-out subject.For each block, NAS uses the subjects outside that block as its training set.
- Inner validation: Subject-disjoint inner folds estimate a proxy for LOSO generalisation by training each candidate architecture on K−1 folds and validating on the remaining fold.The candidate model is independently initialised for each inner fold.
- Architecture selection: The block selection score rewards mean subject-level validation accuracy and penalises high inter-subject variance, using β = 1.0.The score is based on the mean and sample standard deviation of subject-level accuracies.
- Final LOSO evaluation: After selection, the chosen architecture is fitted using all non-target subjects and evaluated once on the held-out subject, with final performance reported as mean accuracy across targets.The held-out subject remains excluded throughout optimisation and final fitting.
2 Results
The method was evaluated on binary no-pain versus pain classification using BioVid and an early-fusion baseline. Five block searches replaced 87 independent searches, reducing runtime while improving accuracy and substantially reducing model size.
- Experimental setup: Five NAS studies replaced 87 independent searches for 87 BioVid subjects partitioned into five outer blocks and three inner folds.The implementation used a Google Colab L4 22 GB GPU instance.
- Computational cost: Approximately 90h of runtime for full 87-subject NAS, training, and evaluation fell to approximately 12h under the same configuration.The reported comparison was measured on the same hardware and configuration.
- Accuracy: 83.39% mean accuracy improved on the 82.79% early-fusion baseline.The evaluation used binary no-pain versus pain classification on the BioVid heat pain dataset.
- Model size: 76.7% to 99.2% parameter reductions produced models with 61.4 k to 1.79 M parameters instead of 7.7 M.These results are reported for the early-fusion comparison.
3 Discussion and Conclusion
The paper proposes a computationally feasible NAS procedure for LOSO evaluation that improves the baseline while reducing parameter counts. It identifies architectural and deployment questions for future work, including multiclass and late-fusion settings.
- Contribution: The proposed method makes NAS computationally feasible in the LOSO setting.The contribution is presented as a block-based approach that shares NAS runs across subjects.
- Conclusion: The method improved the baseline and substantially reduced parameter counts, increasing model efficiency.The conclusion states this efficiency outcome without extending it beyond the evaluated setting.
- Limitation: High variance in architectural components and parameter counts remains a limitation requiring mitigation.The paper also calls for work on selecting an architecture for deployment.
- Future work: Future evaluations should test multiclass classification, late-fusion architectures, and other LOSO classification problems.Affective computing is given as one example of another target problem.