Source-linked AI summary
Efficient Leakage-Free Neural Architecture Search under Leave-One-Subject-Out Evaluation
Heinke Hihn
TL;DR
LOSO evaluation makes leakage-free NAS expensive because fully nested searches scale as O(N^2). The paper introduces a block-based approach that shares NAS runs across subjects, improving BioVid accuracy from 82.79% to 83.39% while reducing parameters by up to 99.2%.
Problem
Fully nested NAS under leakage-free LOSO evaluation requires independent architecture searches for each subject and scales as O(N^2).
Source: §1 · p. 1Method
The method partitions subjects into disjoint outer blocks, uses subject-disjoint inner folds for architecture selection, and performs final LOSO evaluation without using held-out subjects during optimisation.
Source: §1 · pp. 1–2Results
83.39% mean accuracy versus 82.79% for the early-fusion baseline, with parameter reductions of 76.7% to 99.2%.
Source: §2 · p. 2Takeaways & Limitations
The approach provides a computationally feasible LOSO NAS procedure while improving the baseline and substantially reducing model size.
Source: §3 · p. 2Architectural components and parameter counts exhibit high variance, and the paper leaves architecture deployment selection and broader settings for future work.
Source: §3 · p. 2
Abstract
from arXiv · showhide
Leave-One-Subject-Out (LOSO) evaluation estimates generalisation performance for subject-based classification but makes Neural Architecture Search (NAS) computationally expensive because a fully nested implementation requires N independent architecture searches and, assuming approximately linear training cost, scales as O(N^2). We propose a leakage-free, block-based approach that shares NAS runs across subjects. On the BioVid Heat Pain dataset, our approach increased the mean accuracy from 82.79% to 83.39% while reducing the number of parameters by up to 99.2%.
1 Method
The paper addresses the O(N^2) cost of fully nested NAS under leakage-free LOSO evaluation by sharing architecture searches across disjoint subject blocks. Each block uses subject-disjoint inner validation to select an architecture before final fitting and held-out evaluation.
Motivation
- Fully nested LOSO-NAS scales as O(N^2) when NAS and fitting costs grow approximately linearly with the number of subjects.Each of N folds independently searches architectures and fits models using the remaining N−1 subjects.Source: §1 · p. 1
Block NAS
- Block NAS partitions subjects into disjoint outer blocks and performs one architecture search per block instead of one per held-out subject.For each block, NAS uses the subjects outside that block as its training set.Source: §1 · p. 1
Inner validation
- Subject-disjoint inner folds estimate a proxy for LOSO generalisation by training each candidate architecture on K−1 folds and validating on the remaining fold.The candidate model is independently initialised for each inner fold.Source: §1 · p. 1
Architecture selection
- The block selection score rewards mean subject-level validation accuracy and penalises high inter-subject variance, using β = 1.0.The score is based on the mean and sample standard deviation of subject-level accuracies.Source: §1 · p. 2
Final LOSO evaluation
- After selection, the chosen architecture is fitted using all non-target subjects and evaluated once on the held-out subject, with final performance reported as mean accuracy across targets.The held-out subject remains excluded throughout optimisation and final fitting.Source: §1 · p. 2
2 Results
The method was evaluated on binary no-pain versus pain classification using BioVid and an early-fusion baseline. Five block searches replaced 87 independent searches, reducing runtime while improving accuracy and substantially reducing model size.
Experimental setup
- Five NAS studies replaced 87 independent searches for 87 BioVid subjects partitioned into five outer blocks and three inner folds.The implementation used a Google Colab L4 22 GB GPU instance.Source: §2 · p. 2
Computational cost
- Approximately 90h of runtime for full 87-subject NAS, training, and evaluation fell to approximately 12h under the same configuration.The reported comparison was measured on the same hardware and configuration.Source: §2 · p. 2
Accuracy
- 83.39% mean accuracy improved on the 82.79% early-fusion baseline.The evaluation used binary no-pain versus pain classification on the BioVid heat pain dataset.Source: §2 · p. 2
Model size
- 76.7% to 99.2% parameter reductions produced models with 61.4 k to 1.79 M parameters instead of 7.7 M.These results are reported for the early-fusion comparison.Source: §2 · p. 2
3 Discussion and Conclusion
The paper proposes a computationally feasible NAS procedure for LOSO evaluation that improves the baseline while reducing parameter counts. It identifies architectural and deployment questions for future work, including multiclass and late-fusion settings.
Contribution
- The proposed method makes NAS computationally feasible in the LOSO setting.The contribution is presented as a block-based approach that shares NAS runs across subjects.Source: §3 · p. 2
Conclusion
- The method improved the baseline and substantially reduced parameter counts, increasing model efficiency.The conclusion states this efficiency outcome without extending it beyond the evaluated setting.Source: §3 · p. 2
Limitation
- High variance in architectural components and parameter counts remains a limitation requiring mitigation.The paper also calls for work on selecting an architecture for deployment.Source: §3 · p. 2
Future work
- Future evaluations should test multiclass classification, late-fusion architectures, and other LOSO classification problems.Affective computing is given as one example of another target problem.Source: §3 · p. 2