Source-linked AI summary
ChildGaze: A Benchmark Dataset for Collaborative Behavior Understanding in Children
Sindhuja Penchala, Saketh Reddy Kontham, Prachi Bhattacharjee, S. Nima Mahmoodi, Daniel Fonseca, Sareh Karami, Mehdi Garemani, Sudip Mittal, Shahram Rahimi, Noorbakhsh Amiri Golilarz
TL;DR
Gaze-target estimates do not necessarily indicate whether a child participates in a shared activity. ChildGaze adds child-wise collaborative and non-collaborative labels to ChildPlay videos, with face and hand annotations organized at multiple levels. Validation reports κ = 0.8631 for collaboration labels and overall mean IoU = 0.808 for bounding boxes.
Problem
Knowing a child’s gaze target does not directly establish participation in a shared activity, which may differ across children and interactions.
Source: §I · p. 1Method
Results
Validation found Cohen’s κ = 0.8631 for collaboration labels and overall mean IoU = 0.808 for bounding-box annotations.
Source: §V · p. 8Takeaways & Limitations
ChildGaze adds explicit collaboration labels to gaze-centered child data, supporting study of child engagement, child-adult interaction, peer interaction, and multi-person behavior.
Source: §I · p. 2
Abstract
from arXiv · showhide
Understanding collaborative behavior in children is important for analyzing social participation, peer interaction, shared attention, and engagement during play and learning activities. Reliable recognition of these cues can support research in child development, educational analysis, and human-centered computer vision. However, estimating where a child is looking does not necessarily reveal whether the child is actively participating in a shared activity. To support this higher-level analysis, we introduce ChildGaze, a child-centered behavioral annotation dataset built on the ChildPlay video collection [1]. ChildGaze introduces two behavioral labels, collaborative and non-collaborative, assigned independently to each child within a frame. The dataset provides face, left-hand, and right-hand bounding boxes for children and adults and organizes the annotations at the row, person, and frame levels. The current release contains 27 annotated video files, 10,641 frames, and 73,268 body-part annotation rows. Annotation reliability was evaluated on 1,187 frames using independent annotations from two annotators. The collaboration labels achieved 93.16% raw agreement and a Cohen's kappa of 0.8631, while bounding-box annotations achieved an overall mean IoU of 0.808. Baseline experiments with pretrained ViT and Swin Transformer models achieved up to 97.44% child-person-level accuracy and 96.80% frame-level accuracy, respectively. These results show that ChildGaze provides a reliable benchmark for studying collaborative behavior in naturalistic child-adult and peer interactions.
I. INTRODUCTION
Gaze-target estimates do not establish whether a child participates in a shared activity, motivating annotations that represent each child’s collaboration state. ChildGaze adds these labels to ChildPlay videos and provides a person-wise annotation and validation framework for collaborative-behavior research.
I. INTRODUCTION
- Gaze-target location alone cannot establish participation: a child may look toward an adult without joining a task or collaborate with a peer without continuously looking at the adult.Independent play can also occur while other people are present.Source: §I · p. 1
- ChildGaze labels each child as collaborative when participating in a shared interaction, task, or activity, and non-collaborative when disengaged or engaged independently.Adults are treated as collaborative partners because they typically guide, support, demonstrate, or participate in the selected videos’ activities.Source: §I · p. 1
- Person-wise labels preserve different collaboration states among children in the same frame, while frame labels summarize those states by majority vote or mark ties undetermined.A single scene label would erase within-frame differences; the frame label is Yes or No according to the majority, and Undetermined when counts tie.Source: §I · p. 2, Figure 1 · p. 2
- The work introduces ChildGaze on ChildPlay videos, with dense frame-level person annotations, a validation framework, and baseline experiments using contemporary visual models.Its stated contributions include labels assigned independently to each child and a benchmark for collaborative-behavior recognition.Source: §I · pp. 2–3
II. RELATED WORK
The related-work section reviews prior research most closely connected to ChildGaze, spanning gaze following, child-centered behavioral datasets, collaborative engagement, and annotation reliability.
II. RELATED WORK
- The review covers gaze following and child-centered behavioral datasets as research areas related to ChildGaze.It emphasizes work most closely related to the paper.Source: §II · p. 3
- The section also reviews prior work on collaborative engagement and social interaction understanding.Source: §II · p. 3
- Video annotation reliability is another stated focus of the related-work review.Source: §II · p. 3
A. Gaze Following and Gaze-Target Estimation
Prior gaze-target research estimates where a person is looking, while child-centered datasets extend visual analysis to naturalistic child behavior and interaction. These resources provide context for ChildGaze’s focus on explicit collaboration labels.
A. Gaze Following and Gaze-Target Estimation
- Sharingan jointly models multiple people to predict gaze targets, while Gaze-LLE uses frozen pretrained DINOv2 features with a lightweight person-specific decoder; neither explicitly represents a child’s participation in shared activity.Earlier gaze-following work formulated target prediction using head appearance and scene information, while child-specific modeling and domain adaptation were also studied.Source: A. Gaze Following and Gaze-Target… · p. 3
B. Child-Centered Gaze and Behavior Datasets
- ChildPlay provides gaze annotations from children playing and interacting with adults in naturalistic settings, and its results indicate child and adult gaze prediction can differ.The passage notes child-specific annotations can improve performance.Source: B. Child-Centered Gaze and Behavior… · p. 3
- ChildPlay-Hand, MMASD, and Autism Gaze Target extend child-centered analysis to hand-object interaction, multimodal behavior, and gaze targets for young autistic children.These resources cover social attention, actions, and interaction context in child-specific settings.Source: B. Child-Centered Gaze and Behavior… · p. 3
C. Engagement and Collaborative Behavior Recognition
Prior behavior datasets describe classroom actions or collaborative engagement, while reliable behavioral labels require clear criteria and independent checks. ChildGaze combines person-level collaboration labels with gaze context and face and hand annotations, validated through agreement measures.
C. Engagement and Collaborative Behavior Recognition
- Classroom behavior labels such as hand raising, reading, and writing describe individual actions rather than whether a child participates collaboratively with another person.Source: C. Engagement and Collaborative… · p. 3
- Penchala et al. [9] used gaze, interaction, and peer collaboration cues to classify engagement, whereas ChildGaze constructs and validates reusable person-wise collaboration annotations.Source: C. Engagement and Collaborative… · p. 3
D. Video Annotation and Dataset Validation
- Because collaboration labels interpret social participation, dataset reliability depends on clear criteria and independent annotation checks.ChildGaze evaluates collaboration-label agreement and bounding-box consistency on an independently annotated held-out subset using Cohen’s kappa and IoU-based agreement.Source: D. Video Annotation and Dataset… · pp. 3–4
- ChildGaze combines gaze context, face and hand annotations, multi-person support, and explicit collaboration labels at multiple annotation levels, unlike prior datasets focused on narrower aspects.Table I compares annotation types, spatial cues, multi-person support, behavioral labels, annotation granularity, and evaluation methodology.Source: D. Video Annotation and Dataset… · p. 4, Table I · p. 4
III. CHILDGAZE DATASET CONSTRUCTION AND ANNOTATION
ChildGaze extends ChildPlay videos with person-wise collaboration labels and dense annotations of children and adults, structured for validation and downstream modeling.
III. CHILDGAZE DATASET CONSTRUCTION AND ANNOTATION
- ChildGaze extends naturalistic ChildPlay videos with person-wise collaboration labels and dense frame-level annotations for children and adults.The workflow prepares the videos, annotates faces and hands, assigns child-person labels, summarizes them at frame level, and exports annotations for validation and modeling.Source: §III · p. 4
A. Dataset Preparation
- Annotators mark each visible person’s face and hands with bounding boxes, propagate boxes from keyframes, review corrections, and export structured CSV records.Children and adults have separate classes; instance identifiers distinguish people in multi-person frames.Source: A. Dataset Preparation · p. 4
B. Collaboration Annotation Scheme
- Children receive independent collaborative or non-collaborative labels based on participation in an ongoing shared interaction, task, or activity.Independent play, disengagement, or unrelated activity is non-collaborative; continuous gaze toward an adult is not required when shared-task participation is clear.Source: B. Collaboration Annotation Scheme · pp. 4–5
- Adults are recorded as collaborative interaction partners, while the behavioral collaboration decision is defined primarily for children.The annotation scheme reflects adults’ typical roles in the selected interactions, including guiding, demonstrating, encouraging, and assisting children.Source: B. Collaboration Annotation Scheme · p. 5
C. Annotation Levels
ChildGaze organizes annotations at row, person, and frame levels, with person-level labels serving as the primary unit for behavioral analysis.
Row level
- Each visible or reliably localized face, left hand, and right hand is stored as a separate row, with unavailable body parts omitted.A person may contribute up to three rows in one frame.Source: C. Annotation Levels · p. 5
Person level
- Each child is counted once per frame and receives an independent collaboration label, even when other children in the frame have different states.Person level is the primary unit for behavioral analysis.Source: C. Annotation Levels · p. 5
Frame level
- A frame is collaborative or non-collaborative when most children have that state; ties are marked undetermined.The frame-level label is derived from the underlying person-level annotations.Source: C. Annotation Levels · p. 5
D. Dataset Statistics
The release combines annotations across files, frames, body parts, people, and frame-level collaboration states, with person-level labels retaining individual differences in multi-child scenes.
D. Dataset Statistics
- 27 annotated files contain 10,641 frames and 73,268 body-part annotation rows across children and adults.Source: D. Dataset Statistics · p. 5
- Among 13,330 child instances, 5,959 (44.7%) are collaborative and 7,371 (55.3%) non-collaborative; frames include 4,258 collaborative, 5,730 non-collaborative, and 653 undetermined.Undetermined frames contain equal numbers of collaborative and non-collaborative children.Source: D. Dataset Statistics · p. 5
- Adults appear in most annotated frames and provide interaction context, but their collaboration labels are not classification targets.Face and hand counts vary when hands are occluded, outside the image, or unsuitable for reliable annotation.Source: D. Dataset Statistics · p. 5
- The person-level distribution preserves a natural, relatively balanced class prevalence, and multi-child scenes make child-specific labels the most appropriate primary benchmark representation.The difference between frame and child-instance counts reflects multi-child interactions.Source: D. Dataset Statistics · pp. 5–6
IV. ANNOTATION VALIDATION AND RELIABILITY
Independent annotations on a validation subset show high agreement on child collaboration labels, assessed with chance-corrected and spatial consistency measures.
A. Validation Subset
- The validation subset comprised 1,187 frames from three video segments, approximately 11.2% of the 10,641-frame dataset.Grouping child face and hand annotations yielded 1,418 shared child person-instances for label-agreement evaluation.Source: A. Validation Subset · pp. 6–7
IV. ANNOTATION VALIDATION AND RELIABILITY
- Two annotators independently labeled the same validation frames using the same annotation ontology and collaboration criteria.Validation assessed child collaboration labels separately from spatial bounding-box annotations.Source: A. Validation Subset · p. 6, §IV · p. 6
B. Agreement on Collaboration Labels
- 93.16% raw agreement and Cohen’s kappa of 0.8631 were achieved across 1,418 child person-instances.The annotators agreed on 644 collaborative and 677 non-collaborative instances, with 97 disagreements.Source: Table IV · p. 7, B. Agreement on Collaboration Labels · p. 7
- Cohen’s kappa measures collaboration-label agreement while accounting for chance agreement based on the annotators’ marginal label distributions.Bounding-box consistency was assessed with IoU, agreement at an IoU threshold of 0.5, and missed-instance rate.Source: §IV · p. 6, B. Agreement on Collaboration Labels · p. 7
C. Bounding-Box Agreement
Bounding-box agreement was strongest for faces and adult hands, while child hands were less consistent; overall spatial agreement remained high across all six classes.
C. Bounding-Box Agreement
- Bounding boxes were matched within each frame and body-part class, and agreement was assessed using mean IoU, the 0.5 threshold, and missed-instance rate.The per-class table reports agreement for six child and adult face and hand classes.Source: C. Bounding-Box Agreement · p. 7, Table V · p. 7
- 0.892 mean IoU for child faces and 0.896 for adult faces were the strongest results, with nearly all matched face boxes above the 0.5 threshold.Adult left- and right-hand mean IoU values were 0.839 and 0.871, respectively.Source: C. Bounding-Box Agreement · p. 7
- 0.661 mean IoU for child left hands and 0.689 for child right hands were lower than face and adult-hand agreement, partly because hands are small, moving, or occluded.Small placement differences affect hand IoU more than larger, more stable face regions.Source: C. Bounding-Box Agreement · p. 7
- 0.808 pooled mean IoU, 85.1% of matched boxes at IoU ≥ 0.5, and a 0.2% missed-instance rate summarize agreement across all six classes.Disagreements mainly concerned precise localization of small body regions, rather than whether a person or body part was present.Source: C. Bounding-Box Agreement · p. 7, Table V · p. 7
D. Baseline Experiments
Pretrained ViT and Swin Transformer baselines performed strongly at both frame and child-person levels. Their top accuracy differed by level: Swin led at frame level, while ViT led at child-person level.
D. Baseline Experiments
- Swin achieved a frame-level F1-score of 96.48% and ROC-AUC of 0.9957, while ViT achieved frame-level accuracy of 96.40% and ROC-AUC of 0.9929.At child-person level, ViT achieved an F1-score of 96.97% and ROC-AUC of 1.000; Swin achieved 100% recall.Source: D. Baseline Experiments · pp. 7–8
- 96.80% frame-level accuracy for Swin and 97.44% child-person-level accuracy for ViT were the respective highest accuracies.Table VI summarizes performance and inference efficiency at both recognition levels.Source: D. Baseline Experiments · pp. 7–8, Table VI · p. 8
- 1.64 ms per frame for ViT and 1.94 ms for Swin corresponded to 609.5 and 515.2 frames/s, respectively, using batch size 32 on GPU.The authors interpret the results as an initial indication that the labels contain learnable visual patterns for frame- and person-level recognition.Source: D. Baseline Experiments · p. 8
E. Discussion
ChildGaze supports consistent person-level collaboration annotation in naturalistic scenes, while its validation coverage and binary labels leave room for expansion. The conclusion points to additional data, finer categories, and temporal or multimodal modeling as future directions.
E. Discussion
- 93.16% raw agreement and Cohen’s kappa of 0.8631 support consistent person-level collaboration annotation, while 653 frames had equal numbers of collaborative and non-collaborative children.The discussion notes that a single frame-level label cannot always represent differing participation among children in the same scene.Source: E. Discussion · p. 8
- Validation covered 1,187 frames, about 11.2% of the dataset, so consistency across more videos, activities, and interaction settings remains to be established.The discussion also reports lower IoU for small, frequently occluded child hands than for faces.Source: E. Discussion · p. 8
V. CONCLUSION AND FUTURE WORK
- 10,641 frames, 73,268 body-part rows, and 13,330 child person-instances make up the dataset; baseline accuracy reached 97.44% at child-person level and 96.80% at frame level.The conclusion describes face and hand boxes for children and adults, with annotations organized at row, person, and frame levels.Source: §V · p. 8
- Future work proposes more videos and broader validation, finer collaboration categories, and temporal or multimodal models combining gaze, face, hand, object, and interaction cues.The stated aim is a more complete representation of collaboration over time and improved generalization in naturalistic settings.Source: §V · p. 8