PaperlayerView exact revision on arXiv

Source-linked AI summary

InfantNet: A Large Scale Dataset for Infant Body Pose and Shape Estimation

Zhipeng Hui, Jiahao Wang, Xun Dong, Guofeng Zhang, Xingrui Wang, Jiawei Peng, Qihao Liu, Xiaoding Yuan, Yi Zhang, Junjie Oscar Yin, Wufei Ma, Qinfan Lyu, Sara N. Lupolt, Keeve Edward Nachman, Adam Kortylewski, Alan Yuille

arXiv:2610.40467v1cs.CV

TL;DR

Infant pose estimation lacks large-scale, high-quality annotated datasets, including the 3D supervision needed to train and evaluate modern modeling methods. InfantNet combines 108,902 infant RGB images with 2D keypoints and a curated 11,642-image subset of 3D SMIL annotations, using iterative annotation and validation. Adult estimators generalize poorly, while InfantNet fine-tuning improves 2D performance by 6.2% AP and reduces 3D MPJPE by 78%.

  • Problem

    Infant pose estimation lacks large-scale, high-quality annotated datasets, including the 3D supervision needed to train and evaluate modern modeling methods.

    Source: §1 · p. 2

  • Method

    InfantNet combines 108,902 infant RGB images with 2D keypoints and a curated 11,642-image subset of 3D SMIL annotations, using iterative annotation and validation.

    Source: §1 · p. 2

  • Results

    Adult estimators generalize poorly, while InfantNet fine-tuning improves 2D performance by 6.2% AP and reduces 3D MPJPE by 78%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    InfantNet provides a benchmark and data resource for infant pose and shape estimation, with reported downstream use in hand-to-mouth behavior detection for exposure-risk assessment.

    Source: §1 · p. 2

    3D pose and shape annotations cover a curated subset of 11,642 images from InfantNet, which comprises more than 100K RGB images.

    Source: §1 · p. 2

Abstract

from arXiv · show

Infant pose and shape estimation is essential for applications in childcare, developmental monitoring, and medical diagnosis. However, existing methods and datasets are largely designed for adults, and direct transfer to infants fails due to substantial differences in body proportions, articulation limits, and frequent self-occlusion. To address this gap, we introduce InfantNet, the largest real-image infant dataset to date, comprising 108,902 RGB images of infants aged 6-18 months. Each image is annotated with 2D keypoints, and a curated subset of 11,642 images additionally includes 3D pose and shape annotations with full SMIL parameters, verified through multi-view rendering and reprojection-error filtering. We use an iterative annotation pipeline to ensure high fidelity across both 2D and 3D labels. InfantNet establishes a large-scale, comprehensive benchmark for infant 2D keypoint detection and 3D pose-and-shape recovery. Baseline experiments demonstrate that state-of-the-art adult pose estimators do not generalize well to infants, whereas fine-tuning on InfantNet yields a consistent improvement. The gains are even more pronounced for 3D pose and shape estimation, where SMIL-based estimation demonstrates robust performance with accuracy comparable to results reported for SMPL-based models. By releasing the InfantNet dataset and benchmark, we provide a vital resource for advancing infant pose analysis and related healthcare applications.

1. Introduction

InfantNet addresses the shortage of high-quality infant pose data with large-scale 2D annotations and a curated 3D subset. Its benchmark reports improved 2D and 3D performance after fine-tuning on InfantNet.

1. Introduction

  • An iterative pipeline combines human supervision and optimization-based refinement for 2D labels, while 3D SMIL parameters undergo multi-view rendering and reprojection checks.Frames with reprojection error above 16 pixels are discarded.Source: §1 · p. 2
  • The collection uses caregiver consent and digital face-masking for HomeVisit, and controlled academic licensing and a non-identification agreement to limit misuse.YouTube material is not redistributed; URLs and anonymized derivatives are released instead.Source: §1 · p. 2
  • Fine-tuning on InfantNet substantially narrows the adult-to-infant generalization gap, improving 2D performance by 6.2% AP and reducing 3D MPJPE by 78%.Source: §1 · p. 2
  • The authors report that their model enables more accurate detection of infant hand-to-mouth behavior for environmental exposure-risk assessment.Source: §1 · p. 2
  • InfantNet contains 108,902 RGB images with 2D keypoints and a curated 11,642-image subset with 3D SMIL labels.The dataset supports evaluation of both 2D and 3D pose and shape estimation.Source: §1 · p. 2
  • The paper trains an end-to-end framework to regress SMIL parameters from images using optimization-generated and verified pseudo-ground truth.The authors describe this as the first attempt to directly regress SMIL parameters from images with neural networks.Source: §1 · p. 2

2. Related Work

Prior infant datasets and methods are limited in scale, modality, or 3D supervision, while adult-trained estimators face infant-specific domain differences. InfantNet is presented as a real-world RGB benchmark with consistent 2D and 3D annotations.

2.1. Infant Datasets

  • Earlier infant datasets include small clinical or supine video collections, activity labels, synthetic depth sequences, and sparse keypoint images; InfantNet adds real RGB data with 2D and 3D annotations.The paper also lists segmentation masks and bounding boxes among InfantNet’s annotations.Source: §2.1 · p. 3

2.2. Infant Pose and Shape Estimation

  • Infant 2D estimators and 3D approaches remain constrained by infant-specific proportions, articulation, occlusion, limited labels, or specialized sensors; InfantNet supplies a real-world RGB benchmark with consistent 2D and 3D annotations.Earlier 3D approaches include depth-sensor prediction and stereo triangulation; SMIL relies on limited, low-quality depth sequences, while some alternatives use synthetic renderings or pressure mats.Source: §2.2 · p. 3

2.3. Adult pose and shape estimation

  • Adult regression models such as HMR, VIBE, PARE, and 4D-Humans use feedforward prediction but generalize poorly to infants; this work adapts them using InfantNet and SMIL annotations.The paper identifies infant body proportions, articulation limits, and frequent occlusions as domain differences.Source: §2.3 · p. 3

3. InfantNet Dataset

This section describes InfantNet’s data governance, collection and annotation workflows, SMIL fitting, and dataset characteristics.

3.1. Ethical Design and Data Governance

  • HomeVisit data were collected under IRB approval and caregiver consent; both sources are face-masked, while YouTube releases only URLs and anonymized derivatives.Access is controlled by academic licensing and a Data Use Agreement prohibiting biometric identification.Source: §3.1 · pp. 3–4

3.2. Data Collection

  • InfantNet combines YouTube videos with 15 consented HomeVisit sessions recorded by four synchronized, orthogonally positioned cameras; frame sampling is capped at 2 FPS.HomeVisit frames are calibrated per session to support SMIL fitting.Source: §3.2 · p. 4, §3.3 · p. 4

3.3. Data Annotation

  • RTMO-L supplies initial keypoints for human correction and inspector review; SAM2 generates boxes from visible-keypoint prompts, covering masks and occluded joints.Correction and verification repeat until the inspector approves annotations for each recording session.Source: §3.3 · p. 4

3.4. Fitting SMIL on Infant Images

  • SMIL fitting optimizes shape β and pose θ using data, prior, and auxiliary losses, then filters annotations using multi-view checks and a 16-pixel reprojection-error threshold.Auxiliary losses cover landmark alignment, table contact, temporal smoothness, and self-intersections.Source: §3.4 · pp. 4–5

3.5. Data Summary

  • 115.0 is InfantNet’s post-filtering pose-diversity score, exceeding several widely used human datasets; it was 138.2 before filtering.The score measures average standard deviation of pelvis-centered 3D joint coordinates.Source: §3.5 · p. 6, Figure 4 · p. 6
  • 3D annotations achieve 9.10 px mean reprojection error and 83.04%/98.40%/99.99% PCK@0.05/0.1/0.2; independently optimized candidates agree at 3.1 mm mean pairwise MPJPE.The annotations differ from initial annotations by 80.6 mm MPJPE.Source: §3.5 · p. 6
  • 75.58% of frames have fully occluded keypoints, and 92.24% have fully or partially occluded keypoints.Among occluded keypoints, 64.5% are self-occluded, 30.0% externally occluded, and 5.5% out of frame.Source: §3.5 · p. 6

4. Experiments

The experiments benchmark adult-oriented 2D and 3D estimators on InfantNet, comparing direct transfer with infant-specific adaptation. They also test source-specific and cross-domain 2D performance and a hand-to-mouth application.

4.1. 2D Keypoint Estimation

  • Fine-tuning improved 2D performance by 6.2% AP and 5.2% AR on average; ViTPose-huge achieved 0.9477 AP and 0.9582 AR.The table reports the best adapted results as 94.77% AP and 95.82% AR.Source: Table 3 · p. 6, §4.1 · p. 7
  • For direct 2D transfer, Sapiens achieved the highest AP at 0.9008, while ViTPose-large and ViTPose-huge achieved the best AR at 0.9199.Transformer models, particularly Sapiens and ViTPose, showed the strongest direct transfer.Source: §4.1 · pp. 6–7

4.2. 3D Pose and Shape Estimation

  • SMIL adaptation reduced MPJPE by 78% and PA-MPJPE by 54% on average; 4D-Humans reached 84.73 mm MPJPE and 46.74 mm PA-MPJPE.The adapted models improved across all architectures, and 4D-Humans had the strongest results.Source: Table 4 · p. 7, §4.2 · p. 8

4.1. 2D Keypoint Estimation

  • Without InfantNet fine-tuning, models mislocalized joints under self-occlusion and atypical poses; higher input resolutions outperformed lower ones.Fine-tuning enabled more accurate joint localization in the challenging scenarios shown.Source: Figure 5 · p. 7
  • Fine-tuning improved AP by 6.1% on HomeVisit and 5.0% on YouTube, while ViTPose-H leg PCK rose from 0.793 to 0.828 and remained the hardest region.Legs were more challenging than head, torso, or arms before fine-tuning.Source: §4.1 · p. 7, Table 5 · p. 7

4.2. 3D Pose and Shape Estimation

  • Direct adult-model inference was less accurate than infant-specific modeling; among adult-centric models, PARE achieved 364.49 mm MPJPE and 120.06 mm PA-MPJPE.The visual comparison contrasts direct inference with adult models against training with infant models.Source: §4.2 · pp. 7–8, Figure 6 · p. 8

4.3. Real-world Application

  • The infant-specific 4D-Humans model detected hand-to-mouth behavior with 83% accuracy, versus 30% for the adult-oriented baseline.The behavior was identified using normalized hand-to-mouth Euclidean distances.Source: §4.3 · p. 8

5. Conclusion

The conclusion presents InfantNet as a large-scale infant pose-and-shape dataset and a benchmark for addressing adult-model limitations. It reports improved 2D and 3D estimation and a hand-to-mouth application, while identifying expansion of 3D labels as future work.

5. Conclusion

  • InfantNet is presented as the first large-scale dataset dedicated to infant pose and shape estimation, addressing limitations of adult-centric models.The authors aim to expand 3D labels across more diverse domestic environments and ages using human-in-the-loop active learning.Source: §5 · p. 8
  • The authors report that training on InfantNet bridges the adult-to-infant domain gap and enables accurate 2D and 3D estimation.They also describe the framework as a foundation for quantifying hand-to-mouth behavior for exposure-risk assessment.Source: §5 · p. 8

A.1. 2D Source-Specific Evaluations

The appendix evaluates 2D estimators separately on YouTube and HomeVisit, using source-specific fine-tuning and direct inference. It reports stronger direct-transfer results for ViTPose models and gains after fine-tuning on either source.

A.1. 2D Source-Specific Evaluations

  • On YouTube, ViTPose-large led direct inference with 0.9233 AP and 0.9443 AR; on HomeVisit, ViTPose-large led AP at 0.8901 and ViTPose-huge led AR at 0.9120.Across both subsets, ViTPose models consistently had the strongest direct-inference results, and larger variants outperformed smaller counterparts.Source: §A.1 · p. 11
  • Fine-tuning on YouTube yielded average relative gains of 5.0% AP and 3.7% AR; ViTPose-huge led AP at 0.9446 and ViTPose-large led AR at 0.9600.The reported gains were consistent across architectures.Source: §A.1 · p. 11
  • Fine-tuning on HomeVisit yielded average relative gains of 6.1% AP and 5.3% AR; ViTPose-large achieved 0.9433 AP and 0.9567 AR.Res152 256x192 also improved from 0.8700 to 0.9308 AP and from 0.8893 to 0.9428 AR.Source: §A.1 · p. 11

A.2. 2D Cross-Subset Evaluations

Cross-subset evaluation fine-tunes on one InfantNet source and tests on the other without using target-source training data. Transfer was asymmetric: YouTube training improved HomeVisit performance substantially, while HomeVisit training yielded only marginal gains on YouTube.

A.2. 2D Cross-Subset Evaluations

  • The evaluation compares direct inference with cross-subset fine-tuning in both directions, reporting AP, AP.5, AP.75, and AR.The table’s rightmost blocks report cross-subset results for YouTubeTest and HomeVisitTest.Source: §A.2 · pp. 11–12, Table S1 · p. 12
  • HomeVisit-to-YouTube fine-tuning yielded only 0.5% AP and 0.3% AR average relative gains, amid a mismatch between the subsets’ visual conditions.HomeVisit has fixed viewpoints, stable lighting, and clean indoor scenes; YouTube has greater camera, compression, occlusion, and appearance variability.Source: §A.2 · p. 12
  • YouTube-to-HomeVisit fine-tuning yielded average relative gains of 5.2% AP and 4.7% AR, attributed to YouTube’s larger and more diverse training distribution.The authors report that all model families benefited from this transfer direction.Source: §A.2 · p. 12

A.3. 3D Infant Pose and Shape Annotations

InfantNet’s fitted SMIL meshes capture diverse infant poses and body shapes as 3D annotations. The benchmark passages also report gains from fine-tuning and cross-subset performance.

A.3. 3D Infant Pose and Shape Annotations

  • InfantNet’s fitted SMIL meshes illustrate diverse infant poses and body shapes used as 3D annotations for training end-to-end models.The visualizations complement the samples shown in the main paper.Source: §A.3 · p. 12
  • Fine-tuning substantially improves 2D pose-estimation performance across architectures, with average relative gains of 6.1% AP and 5.3% AR for source-specific adaptation.Cross-subset fine-tuning yields relative gains of 5.2% AP and 4.7% AR over direct inference.Source: Table S2 · p. 13
  • Infant models consistently improve 3D accuracy across architectures when trained on HomeVisit and tested on YouTube.The benchmark reports MPJPE and PA-MPJPE in millimeters.Source: Table S3 · p. 13

A.4. Occlusion and Edge Case Analysis

Infant-specific 4D-Humans performance remains strong on occluded frames and samples with rare poses or viewpoints. Figure S2 illustrates these challenging cases.

A.4. Occlusion and Edge Case Analysis

  • 91.28 mm MPJPE and 48.84 mm PA-MPJPE on frames with ≥5 occluded keypoints, comparable to Table 4.The evaluation focuses on frames with at least five occluded keypoints.Source: §A.4 · p. 13
  • On the top 10% rare-pose samples, the model achieves 103.53/57.25 mm MPJPE/PA-MPJPE; on the top 10% rare-viewpoint samples, it achieves 85.47/46.28 mm.Rare poses and viewpoints are defined using 5th-NN distances in pelvis-centered SMIL-joint space and normalized camera-pose space, respectively.Source: §A.4 · p. 13
  • Figure S2 shows examples of model outputs for occlusion, rare poses, and rare viewpoints.These are the edge-case categories discussed in the evaluation.Source: §A.4 · p. 13

A.5. 3D Cross-Subset Evaluations

Cross-subset evaluation finds that infant models retain an advantage over adult baselines despite accuracy drops across data sources. End-to-end regression is faster and more reliable than optimization-based fitting, face masking has little effect on performance, and the paper plans a privacy-protected release.

A.5. 3D Cross-Subset Evaluations

  • Cross-subset results report 84.50 mm MPJPE and 42.40 mm PA-MPJPE, suggesting InfantNet’s 3D infant pose and shape annotations retain robustness despite capture-environment differences.Infant models still outperform adult baselines across architectures.Source: §A.5 · pp. 14–15

A.6. End-to-End vs Optimization

  • End-to-end models run in milliseconds rather than optimization-based SMIL fitting’s 38.7020 seconds per image; HMR is fastest at approximately 3.25 ms.Figure S3 contrasts deformed optimization meshes with anatomically coherent end-to-end 4D-Humans meshes that better align with the infant silhouette.Source: §A.6 · p. 15, Table S4 · p. 15, Figure S3 · p. 16

A.7. 3D Evaluation with Face-Masked Data

  • Face masking limits performance variation to approximately 1–5 mm, with PARE improving from 83.73 mm to 78.48 mm MPJPE and 4D-Humans changing from 84.73 mm to 86.57 mm.The authors report that masking removes identifiable facial information while preserving structural and geometric cues needed for reconstruction.Source: Table S5 · p. 15, §A.7 · p. 16

A.8. Public Release

  • The authors plan to release training and evaluation annotations with strict privacy safeguards, along with fine-tuned model checkpoints.They also plan to release the data-processing and end-to-end modeling implementation for reproducibility.Source: §A.8 · p. 16

Related papers

Loading 2610.40467v1…