NICE Toolbox: Nonverbal Interpersonal Communication Exploration Toolbox

1 Statement of need

• Manual video annotation is slow, difficult to scale, and hard to make consistent, constraining dataset size through time and budget.Trained labellers traditionally rewatch recordings multiple times.Source: §1 · p. 1
• Combining separate vision and speech libraries into a working workflow requires machine-learning and software-engineering expertise often outside psychology projects.Source: §1 · p. 1
• Public-benchmark rankings may not transfer to a specific study, so researchers need to compare candidate models and configurations on their own recordings.The paper notes that no available framework then integrated such evaluation with multimodal model combination.Source: §1 · p. 1
• NICE Toolbox unifies processing and inference across machine-learning algorithms through configurable experiments, evaluation, visualisation, third-party integration, and tabular export.The exported measurements support further statistical analysis within psychological studies.Source: §1 · p. 2

2 State of the field

• ANVIL and ELAN support frame-accurate behavioural coding and established workflows, but annotation remains time-consuming and labour-intensive.Source: §2 · p. 2
• Open-source automated tools typically cover one signal channel, leaving researchers to reconcile formats, coordinate systems, and sampling rates with custom scripts.Examples include MEA for movement, OpenFace and Py-Feat for facial analysis, and CrisperWhisper and WhisperX for transcription.Source: §2 · p. 2
• EnvisionBox curates code modules, tutorials, and datasets for multimodal social-interaction research, but serves as shared storage rather than unified software.Source: §2 · p. 2
• FaceReader and iMotions offer integrated multimodal analysis, while proprietary licensing, closed algorithms, cost, and vendor-bounded signal support limit academic reproducibility and access.Source: §2 · p. 2
• DISCOVER is a server-based back-end operated through NOVA for exploring content, inspecting recordings, and refining annotations.Source: §2 · p. 2
• NICE Toolbox combines interchangeable models and multimodal detector outputs in automated large-scale processing, with study-specific evaluation and multi-camera detections fused by triangulation into 3D.The paper contrasts this with alternatives that typically accept one camera or leave camera outputs unfused.Source: §2 · p. 2

3 Software design

• The toolbox reuses established open-source models and libraries, keeps detector outputs in consistent component structures, and lets non-programmers configure multimodal analyses without writing code.Source: §3 · p. 2
• Its modules separate detectors, user-specific evaluation, interactive 3D rerun visualisation, and connectors for importing or exporting third-party tools.The example experiment flow combines video, optional audio and calibration with TOML configuration, detector processing, per-subject tabular results, export, evaluation, and visualisation.Source: §3 · p. 2, Figure 1 · p. 3
• TOML configurations specify algorithm parameters, metrics, visualiser layouts, and dataset structure; schema validation catches errors before long-running execution.Source: §3 · p. 2
• Users can derive detector variants by inheriting a parameter template and overriding only changed fields, such as confidence thresholds or backbones.Source: §3 · p. 3
• Detectors form a directed computational graph whose execution is topologically sorted, and detectors producing the same components can be substituted as inputs to downstream detectors.For example, either MMPose or SAM-3D-Body body-joint outputs can feed the same kinematics detector.Source: §3 · p. 3
• Detector models run in isolated virtual or conda environments to accommodate conflicting machine-learning dependencies, with configuration and unified-format outputs passed through files.Source: §3 · p. 3

4 Research impact statement

• In psychotherapy research, pose-based kinematic displacement measures were used across 32 videos of 16 parent–child dyads to distinguish interactive coupling from coincidental covariation and compare play conditions.Source: §4 · p. 3
• A human–AI interaction study used facial-expression, gaze, and posture measurements across 482 sequences recorded with six synchronised cameras alongside self-reported responses.Source: §4 · p. 3
• The rerun visualiser displays triangulated body joints, body mesh, gaze direction, head orientation, and synchronised kinematics for a dyadic interaction.Source: Figure 2 · p. 4
• The toolbox is released under AGPL-3.0 on GitHub and Docker Hub, with documentation, tutorials, and an example dataset.Source: §4 · p. 4

• Claude Sonnet 4.6 and Opus 4.8 assisted software development and manuscript editing; humans reviewed generated code and edited AI-assisted text, with authors responsible for final content.Source: AI usage disclosure · p. 4