HAPACT: A Benchmark For Human-Centric Physical Impact Localization in Movies

1 Introduction

• Impacts are short, forceful events often initiated externally, making them difficult to capture using active interaction cues alone.They complement sustained or continuous interactions as a physical-understanding target.Source: §1 · p. 1
• Action-oriented movies depict bodily impacts from collisions and environmental forces, while text queries specify which impact to localize.The paper also notes that physical-event reasoning in video context can support haptic authoring and experience generation.Source: §1 · p. 2
• HAPACT combines viewer-grounded annotations, text-conditioned target-event localization, and frame-level temporal and SMPL-X mesh supervision.Temporal annotations are provided at 24 FPS.Source: §1 · p. 2
• The region-selection baseline outperforms contact-based models [41], but remains limited to coarse body regions rather than fine-grained body-mesh estimates.The paper presents this as a stronger text-conditioned reference while noting that vertex-level estimation remains challenging.Source: §1 · p. 3
• Human-centric impact localization identifies queried physical impacts in video frames and predicts the affected human-body regions.HAPACT contains over 2.8K video-query pairs and more than 20K fine-grained human-annotated frames.Source: §1 · pp. 2–3
• The benchmark evaluates temporal localization with video moment retrieval models and spatial localization with human-scene contact models, alongside an MLLM baseline.The MLLM baseline reformulates on-body localization as query-guided body-region selection.Source: §1 · p. 3

Body-centric Physical Interaction Datasets

• Body-centric physical interaction datasets cover scene constraints, human-object and human-human interaction, and other cues, but largely emphasize everyday or user-initiated interactions.HAPACT adds short, abrupt external-force impacts in movie videos as a physical cue.Source: §2 · p. 3

Video Moment Retrieval and Spatio-temporal Grounding Benchmarks

• Language-guided video localization benchmarks include moment retrieval across diverse domains, with extensions to corpus-level, egocentric, long-video, and multiple-moment settings.The passage cites for video localization benchmarks and,,, and for the listed extensions.Source: §2 · p. 3
• Existing spatio-temporal video grounding benchmarks localize language-specified targets in frames using image-coordinate boxes or tubes, whereas HAPACT targets frame-level body regions.This representation captures the effect of a queried physical event on the human body.Source: §2 · p. 3

3 HAPACT

• HAPACT contains over 2.8K video-query pairs from 212 movies, drawn from more than five hours of high-resolution video.Clips range from 6 to 8 seconds.Source: §3 · p. 3

3.1 Data Collection and Preprocessing

• Movieclips candidates are genre-filtered, processed by Gemini 2.5 Pro, then human-filtered and manually revised; the candidate pipeline achieves 88.99% mean recall.Candidate segments longer than 8 seconds are excluded.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 Data Annotation

• Twenty-eight trained annotators labeled temporal extents at 24 FPS and painted affected regions on an SMPL-X mesh for each frame.Temporal annotations include every occurrence of the queried impact; three annotations are aggregated by majority vote, with ambiguous cases manually adjudicated.Source: §3.2 · pp. 4–5

3.3 Dataset Analysis

• HAPACT combines text-conditioned temporal grounding with frame-level on-body impact annotations, unlike prior datasets.The paper describes it as the first human-centric video dataset to provide fine-grained spatio-temporal annotations for physical impacts.Source: §3.3 · p. 5
• The 2,884 pairs contain 4,242 moments averaging 4.80 frames (0.20 seconds); 34.1% of queries have multiple moments.HAPACT moments are shorter than those in existing video grounding datasets; queries average 11.1 words, and action and adventure movies appear frequently.Source: §3.3 · pp. 5–6, Table 1 · p. 5
• The head has the highest aggregate impact intensity at 23.3%, followed by the hips at 8.4% and spine at 6.9%.Impacts also frequently occur on the back, buttocks, and palms.Source: §3.3 · p. 6

3.4 Dataset Split

• The 70:15:15 train-validation-test split is movie-disjoint and maintains comparable distributions across selected dataset characteristics.The splits contain 2,019 training pairs, 432 validation pairs, and 433 test pairs.Source: §3.4 · p. 6

4.1 Task Definition

• Impact localization predicts a set of intervals for each queried impact occurrence and, for frames in those intervals, which SMPL-X body vertices are affected.The benchmark evaluates temporal and spatial on-body localization as separate axes.Source: §4.1 · pp. 6–7

4.2 Benchmark Setup

• HAPACT evaluates temporal impact localization and spatial on-body impact localization as its two benchmark axes.The benchmark uses specialized baselines and evaluation metrics for each task.Source: §4.2 · p. 7

4.2.1 Temporal Evaluation

• Temporal evaluation covers 18 retrieval- and generation-based models, using mAP and Recall@1 at tIoU 0.3 plus multi-moment metrics across tIoU 0.1–0.5.The 0.3 threshold reflects short impacts averaging 0.20 seconds; G-mAP is also reported for single-, two-, and three-or-more-moment queries.Source: §4.2.1 · p. 7

4.2.2 Spatial Evaluation

• Spatial evaluation tests three human-scene contact models and a text-conditioned MLLM that selects affected regions from 18 SMPL-X-derived labels.Spatial metrics include frame-wise precision, recall, F1, and geodesic error; MLLM predictions are evaluated with label-level F1.Source: §4.2.2 · pp. 7–8

4.3 Main Results

• Fine-tuning on HAPACT improves all retrieval models, although their best scores remain moderate; QVHighlights pre-training does not consistently transfer, with several models near random.The authors attribute the remaining difficulty to short durations and multiple annotated moments.Source: §4.3 · pp. 8–9, Table 2 · p. 8
• General MLLMs achieve stronger zero-shot temporal localization than VMR-targeted generation models, but GPT-5.4 drops about 51% and 73% from mAP@1_tgt to mAP@2_tgt and mAP@3+tgt.The decline indicates difficulty recovering multiple target moments with accurate boundaries.Source: Table 3 · p. 8, §4.3 · p. 9
• Across representative models, 1–2-frame impacts yield lower mAP than 3–6-frame and 7+-frame impacts.Figure 5 compares lengthwise mAP for the strongest representative model from each family.Source: §4.3 · p. 9, Figure 5 · p. 9
• Contact-based models remain below 9% vertex-level F1 across models, while fine-tuned Cosmos-Reason2-8B with region selection obtains the highest spatial scores but performs unevenly across regions.Its label-level F1 is concentrated on the head, back, and buttocks; balancing label distribution narrows but does not eliminate the disparity.Source: §4.3 · pp. 9–10, Table 4 · p. 9

• HAPACT introduces human-centric impact localization and a fine-grained benchmark for queried physical impacts in time and on the human body.The benchmark extends video-based physical reasoning beyond user-intended interactions to externally driven bodily impacts.Source: §5 · p. 10

A.1 Broader Impacts

• HAPACT frames impacts as human-centric physical cues involving external forces, object collisions, and environmental contact.The paper notes possible applications in physical event analysis, body-aware video understanding, haptic generation, and mixed reality.Source: §A.1 · p. 15
• HAPACT is built from staged movie clips and does not redistribute the underlying copyrighted footage.Source: §A.1 · p. 15
• The dataset contains violence, falls, collisions, and explosions; models transferred beyond its domain should be used with care because predictions may be misinterpreted without context.Source: §A.1 · p. 15

A.2 Video Availability

• The Movieclips YouTube channel offers relatively stable availability compared with videos uploaded by individual users.Source: §A.2 · p. 15
• Only 0.3% of Movieclips-channel videos were removed between January 1 and September 1, 2020, according to Bain et al..Source: §A.2 · p. 15
• For reproducibility, HAPACT provides source video identifiers, trimmed timestamps, IMDb identifiers, annotations, and evaluation scripts, but not the underlying footage.The movie footage remains owned by the original rights holders.Source: §A.2 · p. 15

A.3.1 Video Collection and Preprocessing

• The initial video pool used Movieclips videos available as of August 12, 2025.Source: §A.3.1 · p. 15
• PySceneDetect segmented videos into shots, and the terminal shot was compared with reference outro clips using visual similarity.Source: §A.3.1 · p. 15
• When an outro match was found, the final 10 seconds were removed for videos uploaded before May 2017 and the final 30 seconds for later uploads.Videos without an outro match were left untrimmed.Source: §A.3.1 · p. 15

A.3.2 Genre-Based Scoring Prior from 4DX Films

• Candidate movies were selected for genre similarity to films previously screened in 4DX, using IMDb genre metadata.Source: §A.3.2 · p. 15
• The genre prior combines average empirical frequencies for a movie’s individual genres and, when at least two valid genres are available, its genre pairs.The probabilities are computed from the 4DX-screened films.Source: §A.3.2 · p. 15
• The final score is the sum of the single-genre and genre-pair scores, with the pair score set to zero when fewer than two valid genres are available.Movies with no valid genres receive zero; candidates with S(m) ≥0.1 are retained.Source: §A.3.2 · p. 15

A.3.3 Query-Clip Candidate Generation

• Gemini 2.5 Pro [13] proposed second-level impact segments and descriptions, which were buffered by 2.5 seconds and re-encoded at 24 FPS; GPT-4o [29] and a Stanford Parser filter retained human-involved candidates.Descriptions were screened for sudden, salient forces, and the two human-targeted filters’ accepted candidates were combined.Source: §A.3.3 · p. 16, §A.3.4 · p. 16

A.3.5 Recall Evaluation of the Candidate Pipeline

• Candidate generation and filtering achieved 88.99% mean recall (95% CI: 87.98–90.00%) of human-targeted impacts in the source videos.Authors independently annotated Movieclips videos at one-second resolution and compared those annotations with retained candidates.Source: §A.3.5 · p. 16

A.3.6 Manual Revision

• Two authors accepted candidates only when impacts affected a single target, were visible and reliably inferable, and were physically plausible; valid query edits required review or consensus.Direct human-human impacts were the exception to the single-target rule.Source: §A.3.6 · pp. 16–17

A.4.2 Annotation Instructions

• Annotators marked impact timing itself, excluding approach and aftermath, then labeled affected regions on every interval frame; temporal labels used three-annotator majority voting and spatial labels used an SMPL-X mesh interface.For human-human impacts, annotators marked the grammatical subject; ambiguous temporal cases could be resolved by author consensus.Source: §A.4.2 · p. 17, §A.4.3 · p. 18, §A.4.4 · p. 18

A.4.5 Quality Control and Assessment

• Quality checks found temporal Fleiss’ κ of 0.46 and mean temporal IoU of 0.70, while spatial Fleiss’ κ was 0.53 at vertex level and 0.61 at body-part level, with reference mIoU of 0.42.The reference comparison used 100 samples from PROX and RICH; the body-part agreement exceeded vertex-level agreement.Source: §A.4.5 · pp. 18–19

A.5.2 Split Statistics

• HAPACT queries average 1.47 moments, with a significant share containing at least two; split statistics maintain consistent clip-duration, moment-count, and moment-length distributions.Frequent description nouns such as “man” and “woman,” and verbs such as “strike” and “kick,” reflect the dataset’s human-centric impact focus.Source: §A.5.1 · p. 19, Table 6 · p. 20, §A.5.2 · p. 21

A.6.3 Spatial Localization Evaluation

• Spatial baselines included BSTRO, DECO, and GraphiContact, while Cosmos-Reason2 was evaluated through query-guided selection among 18 body-part labels and label-level F1 scores.Methods requiring auxiliary object meshes or hand-held-object annotations were excluded because those inputs were not consistently available in movie clips.Source: §A.6.3 · pp. 23–24, Figure 13 · p. 23

A.6.4 Ablation Settings

• The 0.05-second sampling interval gave competitive temporal-localization results, fine-tuned pretrained checkpoints generally outperformed training from scratch, and text-image input performed best for spatial localization.These findings motivated the sampling interval, initialization, and modalities used in the main baselines.Source: §A.6.4 · pp. 24, 26, Table 8 · p. 25, Table 9 · p. 25

A.7.1 Analysis of Temporal Bottlenecks in VMR Models

• Shorter impacts strongly affect Sim-DETR and Qwen3-VL-8B, while multiple-moment queries cause larger drops for Qwen3-VL-8B and GPT-5.4 than for Sim-DETR.UniTime remains low across both subgroup analyses; the evaluated models show differing sensitivities to impact length and moment count.Source: §A.7.1 · p. 26, Table 11 · p. 27

A.7.2 HSC Baseline Performance under Different Training Settings

• Training DECO and GraphiContact from scratch does not consistently improve F1 over fine-tuning, so limited HSC transfer is not attributable solely to pretraining.BSTRO was excluded because its released implementation requires unavailable initialization weights; architecture or learning-formulation limitations may also matter.Source: §A.7.2 · p. 26, Table 12 · p. 27

A.7.3 Effect of Region-Label Imbalance on Spatial Performance

• Region balancing improves mean region F1 and reduces Region Gap and Across-Region SD, whereas random down-sampling does not reduce the gap.This suggests that label imbalance partly contributes to lower overall performance but does not fully explain disparities across regions.Source: §A.7.3 · p. 26, Table 13 · p. 27

A.8 Dataset Examples

• HAPACT examples pair movie clips and textual queries with temporal intervals and frame-level on-body impact labels, spanning human, environmental, and external-force impacts.An annotation-visualization video is provided in the supplementary material.Source: §A.8 · p. 28, Figure 15 · p. 28

A.9 Limitations

• Fine-grained spatial annotations are subjective: annotators show similar overall tendencies but may differ on precise regions because of occlusion, narrow contact, and interpretation.HAPACT reports Fleiss’ κ of 0.53 at vertex level and 0.61 at body-part level; quality control includes annotator pre-screening and meta-annotator review. Related agreement values are DAMON κ = 0.656 and EPIC-Contact [4] κ = 0.61.Source: §A.9 · p. 29, Figure 16 · p. 29
• HAPACT labels are not perfectly objective physical measurements; exact vertex choices can remain ambiguous even when annotators agree on the body region.The dataset is better suited to coarse spatial applications such as media-based haptic authoring and haptic experience generation.Source: §A.9 · p. 29
• Gemini 2.5 Pro and GPT-4o contributed to candidate generation and filtering, so lineage-specific biases may favor GPT/Gemini-family models despite human review.Final labels were produced through human review and annotation, which reduces but may not eliminate direct model influence.Source: §A.9 · p. 29
• Because HAPACT is movie-based, domain shifts may limit real-world generalization, and it omits some impacts such as daily bumps, sports impacts, and medical collisions.The pipeline could extend to domain-specific videos, but such extensions and validation are left to future work.Source: §A.9 · p. 29