Source-linked AI summary
Infant Care Video Dataset for Classification of Interventions Using Transformers
Igor Bogdanov, James Green
TL;DR
Routine NICU interventions are frequently under-reported, motivating automated video-based detection. The paper introduces the ICVD and benchmarks temporal video transformers, which substantially outperform a framewise approach and support temporal modeling for intervention classification.
Problem
Routine NICU interventions can be inconsistently documented, with reporting rates for some procedures as low as 31%, creating a data gap for healthcare automation research.
Method
The paper constructs the 4,144-video ICVD across 12 standardized simulated NICU procedures and fine-tunes pretrained TimeSformer and MotionFormer models using controlled variability and privacy-preserving manikin recordings.
Results
93.97% and 93.17% top-1 accuracy were achieved by TimeSformer and MotionFormer, while the framewise approach reached 23.17%, producing a 70.80% performance gap.
Takeaways & Limitations
The ICVD provides a foundation for automated intervention documentation, and the results support temporal modeling for healthcare intervention classification.
Takeaways & Limitations
The manikin-based dataset may not represent real NICU variability, and its 12 classes exclude some procedures while overlapping interventions remain a challenge.
Abstract
from arXiv · showhide
Healthcare documentation in the neonatal intensive care unit (NICU) presents significant challenges, with nurses spending approximately 25\% of their time on record-keeping, while up to 60\% of interventions remain undocumented. Motivated by the need to detect interventions from video automatically, we present the Infant Care Video Dataset (ICVD), a collection of 4,144 videos spanning 12 simulated intervention classes designed for developing automated documentation systems. Our manikin-based approach systematically varies conditions, such as camera angle and clinician skin tone, while ensuring privacy compliance. Using video transformer architectures (TimeSformer and MotionFormer), we establish strong baseline performance (93.97\% and 93.17\% top-1 accuracy) among the 12 infant care classes. Our ablation study comparing temporal models with a framewise approach (23.17\% accuracy) demonstrates a 70.80\% performance gap, validating the need for temporal modeling. The ICVD provides a foundation for developing automated documentation systems to reduce clinical burden in neonatal care environments and improve existing practices.
I. INTRODUCTION
NICU documentation is time-intensive and incomplete, while privacy constraints and the lack of specialized datasets hinder automated intervention detection. The paper introduces ICVD and benchmarks transformer-based classification on simulated NICU procedures.
- 25-50% of clinicians’ time is consumed by documentation, while up to 60% of routine interventions go undocumented.
- Privacy barriers and the absence of publicly available NICU routine-care video datasets constrain automated documentation development.
- ICVD contains 4,144 simulated-care videos spanning 12 standardized NICU procedures.
- The intervention classes represent frequent hands-on NICU activities that require documentation but are inconsistently reported.
- TimeSformer and MotionFormer achieved 93.97% and 93.17% top-1 accuracy, respectively, while temporal modeling outperformed framewise classification by 70.80%.
A. Medical Video Datasets
Existing medical video datasets provide limited support for routine NICU intervention recognition because relevant neonatal footage is scarce or privacy-restricted. ICVD addresses this gap with broader, privacy-preserving intervention coverage.
- Surgical video datasets support workflow benchmarks, but neonatal datasets largely target physiological monitoring rather than care interventions.
- The NICU Care Activities dataset contains 289 hours of real infant footage covering three interventions but is inaccessible because of privacy issues.
- Transformer architectures are increasingly used in healthcare action recognition because they capture long-range temporal dependencies.
- Prior NICU studies detected three caregiving activities or clinician presence, whereas this work extends classification to a broader intervention set using privacy-preserving data.
- Documentation compliance for patient repositioning was as low as 31%, while an automated system improved compliance to 82%.
- Nurses may record hundreds of data points per shift and often leave the care space for data entry, consuming substantial time.
III. DATASET DESIGN AND COLLECTION
ICVD was designed around clinically relevant, discrete intervention classes while combining hierarchical procedures with controlled recording variability and ethical accessibility. The selected activities emphasize frequent, documentable hands-on care.
- A. Design Principles: Five principles guide ICVD: clinical relevance, action discreteness, hierarchical complexity, standardized variability, and ethical accessibility.
- A. Design Principles: The dataset includes 12 distinct intervention classes representing frequent hands-on care activities that are often underreported.
- B. Procedure Selection: Pacifier, diaper-related, and feeding interventions cover comfort, frequent daily care, and growth-related procedures.
- B. Procedure Selection: Crib handling captures removing and replacing the patient, while hygiene includes wiping the body and face.
- B. Procedure Selection: Temperature management includes covering and uncovering the patient, and composite sequences such as diaper changes combine atomic actions.
C. Recording Methodology
The recording methodology uses multiple camera types and controlled environmental variation to capture diverse perspectives. Raw footage is segmented into privacy-preserving clips, yielding a high-quality dataset after review.
- C. Recording Methodology: Eight cameras, including smartphones and baby monitors, were arranged around the crib at varied angles and similar distances.
- C. Recording Methodology: Recording sessions varied lighting, crib-sheet color, and five hand-covering types alongside camera diversity.
- C. Recording Methodology: Approximately 48 hours of raw footage were segmented into 3- to 40-second clips, each showing one complete intervention performed by the clinician’s hands on the manikin.
- C. Recording Methodology: 4,144 clips were produced from a target of 4,160, with a 99.6% acceptance rate after quality control.
IV. DATASET PROPERTIES AND ORGANIZATION
The ICVD standardizes video recordings to preserve intervention-relevant visual and temporal information while addressing privacy concerns. Clips use consistent formats, dimensions, and processing, with frame rates retained across camera outputs.
- Videos use .mov or .mp4 containers with H.264 encoding at approximately 270 kbps bitrate.
- Clips are presented in landscape orientation and resized to 456×256 pixels while maintaining the original aspect ratio.
- Frame rates range from 24 to 60 fps depending on the camera output, providing temporal resolution for intervention motion.
- Audio tracks were removed during processing to address privacy concerns and reduce file sizes.
- Clips average approximately 17 seconds, with longer recordings corresponding to more complex procedures such as diaper changing.
- The dataset occupies approximately 2.21 GB, with individual clips averaging 570-600 KB.
B. Temporal Characteristics
Temporal sequences are necessary for distinguishing interventions with similar visual content, so the ICVD incorporates complete videos and controlled variability across recording conditions. The dataset varies backgrounds, hand coverings, performers, camera perspectives, and lighting.
- Temporal Characteristics: Covering and uncovering share visual elements but differ in temporal order, making static image analysis insufficient for distinguishing them.
- Temporal Characteristics: The ablation study compared frame-based models with whole-video transformers and found that temporal modeling substantially improved classification, especially for visually similar intervention pairs.
- Data Variability: The dataset varies backgrounds, hand coverings, performers, and camera perspectives to support robustness across recording conditions.
- Data Variability: ICVD includes two background conditions, five hand-covering types, two performers, and eight camera perspectives.
- Data Variability: Sample frames illustrate diversity across intervention classes, camera angles, hand coverings, and light or dark background sheets.
- Data Variability: Recording sessions also include both artificial and natural lighting scenarios.
D. Organization and Distribution
The ICVD is organized hierarchically by dataset split, intervention class, and lighting condition, while maintaining broadly balanced class distributions. One class is larger because it was repeated under two diaper conditions.
- Organization and Distribution: The dataset uses a 70-15-15 train-validation-test split, 12 intervention categories, and dark or light background conditions.
- Organization and Distribution: Most intervention classes contain 310-340 recordings, indicating broadly balanced class distributions.
E. Access Protocol and Reproducibility
The ICVD uses controlled access and pretrained video transformers to support reproducible research and establish classification baselines. Temporal transformer models substantially outperform the framewise comparison on the test set.
- Access Protocol and Reproducibility: Researchers can request ICVD access through the dataset website, and approved users receive a non-exclusive license.
- Modeling and Evaluation: TimeSformer and MotionFormer were pretrained on Kinetics-400 and fine-tuned on the training dataset using distinct spatiotemporal attention mechanisms.
- Results: 93.97% top-1 and 100% top-5 accuracy were achieved by TimeSformer, while MotionFormer reached 93.17% top-1 and 99.84% top-5 accuracy.
- Results: The framewise approach achieved 23.17% top-1 accuracy, creating a 70.80% performance gap relative to the temporal model.
1) Class-Level Performance Analysis:
Temporal models classify most intervention classes with near-perfect accuracy, while composite and temporally defined activities expose important failure modes. Across aggregate metrics, temporal models substantially outperform framewise analysis.
- Six intervention classes achieved 100% accuracy with TimeSformer, while three more exceeded 97%.The perfect classes were cribtake, wiping-face, pacifier-give, crib-put, blanket-uncover, and diaper-remove.
- 97.92% accuracy was achieved for the feeding class.
- 85.42% accuracy for diaper-apply and 41.30% for diaper-change marked the main TimeSformer challenges.Diaper-change was often confused with diaper-remove, consistent with its composite structure.
- 28.26% accuracy for MotionFormer on diaper-change confirms the difficulty of distinguishing composite activities from their component actions.
- Framewise analysis frequently confused actions defined by reverse temporal order, producing near-random classification between visually similar pairs.Its precision, recall, and F1 score were 23.08%, 23.00%, and 19.45%, respectively.
- 100% top-5 accuracy for TimeSformer and 99.84% for MotionFormer accompanied macro-averaged precision, recall, and F1 scores above 91%.Both temporal models substantially outperformed the framewise model in the aggregate performance comparison.
VI. FUTURE WORK AND LIMITATIONS
Future work must address the gap between controlled manikin recordings and real NICU care, broaden intervention coverage, and improve robustness and clinical translation.
- Manikin-based recordings lack variability from infant movement, body size, and unpredictable responses in real NICU care.The authors call for validation on ethically approved recordings of actual NICU footage.
- The 12-class intervention set excludes procedures such as respiratory and resuscitation care and models interventions as isolated events.Real NICU care can involve related or overlapping interventions, motivating broader classes and multi-label formulation.
- Controlled recording conditions do not fully capture variable lighting and constrained viewing angles in actual NICU monitoring systems.Future work should evaluate robustness across broader video qualities and camera viewpoints.
- Clinical translation requires converting detected interventions into context-aware documentation aligned with clinical standards and EHR integration.
VII. CONCLUSION
The paper introduces ICVD and benchmarks temporal transformer architectures for neonatal intervention classification. Its results support temporal modeling while positioning privacy-compliant simulated data as a foundation for automated documentation research.
- ICVD contains 4,144 videos spanning 12 neonatal care intervention classes and includes benchmark metrics for temporal transformer architectures.
- 70.80% performance gap between temporal and framewise approaches provides evidence for temporal modeling in healthcare intervention classification.
- Nurses spend 25-50% of their time on documentation, while up to 60% of routine interventions go undocumented.These figures motivate automated intervention detection as a response to the documentation burden.
- The dataset establishes a foundation for automated documentation systems that could reduce NICU administrative burden and improve record completeness and accuracy.
- Privacy-compliant simulated data provides a starting point for connecting computer vision technologies with clinical documentation needs.