Source-linked AI summary
Pose-Anchored Optical Flow for Low-Latency Human Action Anticipation in Human-Robot Teaming
Lewis de Zoete Grundy, Chris McCarthy, Christopher Fluke
TL;DR
Human-action anticipation for HRI must balance early intent inference with the limitations of existing pose and dense-motion representations. PoseOFF anchors local optical-flow features to human joints and, across multiple datasets and backbones, improves early-observation performance while supporting lightweight deployment.
Problem
Skeleton representations omit fine-grained motion cues, whereas dense optical flow can be computationally expensive for low-latency HRI perception.
Method
PoseOFF conditions localized optical-flow feature extraction on human pose, aligning motion cues with semantically meaningful joint locations.
Results
Across multiple benchmark datasets and backbone models, PoseOFF consistently improves performance at early observation ratios and achieves comparable or improved accuracy with less observed action.
Takeaways & Limitations
Pose-centred motion representations provide a lightweight enhancement for earlier human-intent understanding in low-latency HRI systems.
Abstract
from arXiv · showhide
Human-robot interaction (HRI) requires robots to interpret human actions early in their execution in order to respond safely, efficiently, and naturally. However, many existing approaches to human action recognition rely either on sparse skeletal representations, which lack fine-grained motion cues, or dense optical flow, which can be computationally expensive for low-latency perception pipelines. In this paper, we propose PoseOFF, a pose-anchored optical flow representation that captures local motion information around human joints to support earlier human intent understanding. By conditioning motion feature extraction on human pose, PoseOFF encodes localised motion dynamics at semantically meaningful body locations, forming a structured motion representation that is explicitly aligned with human kinematics. We evaluate PoseOFF across multiple benchmark datasets and backbone architectures for action anticipation, demonstrating consistent improvements in recognition accuracy, particularly at early observation ratios. Our results show that PoseOFF enables models to achieve comparable or improved performance while observing less of the action sequence, highlighting its effectiveness for early prediction. Importantly, these gains are achieved without requiring full-frame motion processing, making the approach practical for real-time and resource-constrained settings. These findings suggest that pose-centred motion representations such as PoseOFF can enhance the ability of interactive robot systems to infer human actions earlier, supporting more responsive and anticipatory behaviour in human-robot interaction scenarios.
I. INTRODUCTION
Early human-action anticipation is important for safe, coordinated, and natural HRI, but existing pose and dense-motion approaches trade motion fidelity against computational efficiency. PoseOFF addresses this trade-off by anchoring local optical-flow cues to human joints and improves anticipation under partial observation.
- HRI systems need to infer human intent from partial observations to coordinate movements, avoid interference, support assistance, and maintain safety.
- Skeleton representations are efficient and semantically meaningful but discard fine-grained motion, limiting early discrimination between actions with similar poses.
- Dense optical-flow and large spatiotemporal models capture richer dynamics but often impose computational costs unsuitable for low-latency robotic perception.
- PoseOFF anchors localized optical-flow features to human joints, creating a compact motion representation aligned with human kinematics.
- Across multiple benchmark datasets and backbone models, PoseOFF consistently improves performance at early observation ratios and supports comparable or improved accuracy with less observed action.
- The paper positions targeted motion representations as lightweight enhancements for earlier and more responsive HRI.
II. RELATED WORK
Research on action anticipation has progressed from handcrafted and probabilistic methods to recurrent, transformer-based, and multimodal models. Recent approaches model temporal information strongly but commonly rely on dense representations and larger models.
- Early action-anticipation methods used probabilistic models and handcrafted features, followed by recurrent architectures for partial-sequence prediction.
- Transformer-based models such as AVT model long-range temporal dependencies and have demonstrated strong performance.
- Multimodal approaches incorporate complementary cues but typically depend on dense spatiotemporal representations and large models.
B. Skeleton-based Human Action Recognition
Skeleton methods offer efficient, semantically meaningful motion representations but miss fine-grained visual dynamics, while optical-flow approaches are richer yet computationally expensive. The identified gap is direct integration of joint-centred motion into skeleton-based anticipation.
- Skeleton-based methods represent human motion through joint trajectories and remain efficient and robust to background variation.
- Pose trajectories may be ambiguous during early action anticipation, when informative motion cues are present before distinctive configurations emerge.
- Optical-flow and multimodal methods provide complementary dynamic information but commonly process motion as a dense, global signal with substantial cost.
- PoseOFF constructs localized motion features around joints and combines them with pose information without altering skeleton joint relations.
- Joint-centred methods such as JOLO-GCN and H-MoRe treat motion as a separate input stream and focus on action recognition rather than anticipation.
- The literature lacks direct integration of joint-centred motion features within skeleton-based action-recognition frameworks for action anticipation.
III. METHOD
PoseOFF conditions optical-flow extraction on human pose, sampling local motion around each joint and combining it with pose features. The resulting structured representation preserves body alignment while capturing motion at multiple spatial scales.
- The method is designed to add learnable early visual motion cues to skeleton-based architectures while retaining feasibility for real-time settings.
- PoseOFF conditions optical-flow features on human pose and extracts motion dynamics at spatial locations corresponding to body joints.
- The representation focuses on local body regions rather than dense full-frame motion, reducing redundancy while retaining behaviourally relevant dynamics.
- An RGB video provides T frames, from which up to M skeletons with V keypoints are extracted per frame.
- For each joint, PoseOFF extracts an N × N neighbourhood of optical-flow vectors, with dilation controlling the sampled spatial extent.
- Flattened local motion regions are concatenated with pose features to form a unified pose-conditioned representation with C = 2N 2 + 3 channels per keypoint.
(T, M, V, C)
PoseOFF aligns motion features explicitly with human joint locations, producing a structured encoding of local motion dynamics.
- PoseOFF explicitly aligns motion features with joint locations to encode local motion dynamics.
B. Skeleton-based Action Recognition Models
The evaluated skeleton-based models incorporate PoseOFF by learning features from optical-flow windows and concatenating them with skeleton coordinates. The setup uses joint-stream inputs and established implementation settings.
- B. Skeleton-based Action Recognition Models: Three skeleton action-recognition models were modified to incorporate learnable visual motion cues through a CNN embedding layer.The evaluated models were InfoGCN++, MS-G3D, and ST-GCN++.
- B. Skeleton-based Action Recognition Models: The CNN learns features from optical-flow windows and concatenates its output with the original skeleton keypoint coordinates.Figure 4 depicts this PoseOFF embedding-layer design.
- B. Skeleton-based Action Recognition Models: PoseOFF experiments use the skeleton joint stream only because optical-flow windows are extracted at human skeleton-joint locations in image space.
- B. Skeleton-based Action Recognition Models: Temporal masking simulates partial observation by retaining only the initial portion of each sequence and padding the input to the required length.
- B. Skeleton-based Action Recognition Models: Models were trained with configurations aligned to the original implementations, using identical settings for baseline and PoseOFF-augmented models.Optical flow used RAFT, while pose keypoints came from dataset annotations or YOLO-POSE Large for UCF101.
D. Datasets
The evaluation uses NTU RGB+D 60, NTU RGB+D 120, and UCF101, combining controlled datasets with an in-the-wild action-video benchmark. Performance is assessed through accuracy at observation ratios and AUC for anticipation.
- D. Datasets: The experiments use NTU RGB+D 60, NTU RGB+D 120, and UCF101 as evaluation datasets.
- D. Datasets: NTU RGB+D 60 and 120 provide RGB, depth, infrared, and ground-truth 3D coordinates for 25 body joints from Kinect v2 captures.
- D. Datasets: UCF101 contains unconstrained real-world action videos and evaluates performance under in-the-wild conditions across three standard splits.
- D. Datasets: Optical-flow window dilation is 3 for NTU RGB+D and 1 for UCF101, reflecting their different video resolutions.NTU RGB+D videos are 1920 × 1080, while UCF101 videos are 320 × 240.
- D. Datasets: Action anticipation is measured by classification accuracy at each observation ratio and AUC across the evaluated ratios.AUC uses accuracy at each observation ratio and the number of evaluated ratios.
B. Action Classification
PoseOFF improves full-sequence classification and enables equivalent recognition performance from partial observations. Across datasets and models, it reaches baseline accuracy after observing fewer frames.
- B. Action Classification: PoseOFF-augmented models were compared with base models trained on full input sequences without temporal masking for offline action recognition.
- B. Action Classification: 2.14%, 6.84%, and 11.22% average accuracy improvements are reported on NTU RGB+D 60, NTU RGB+D 120, and UCF101, respectively.The UCF101 gain is attributed in the passage to added motion information and scene context when estimated keypoints are misplaced.
- C. Action Anticipation: PoseOFF enables models to reach full-sequence baseline accuracy with fewer observed frames across datasets.Performance is plotted against increasing observation ratios, with full-sequence accuracy shown as a reference.
- C. Action Anticipation: 80% observation suffices for ST-GCN++ and MS-G3D to match the NTU RGB+D 60 baseline, while InfoGCN++ reaches equivalent accuracy at 50%.
- C. Action Anticipation: PoseOFF-augmented models on UCF101 match baseline performance after approximately half of the sequence.
D. Per-Class Performance Analysis
PoseOFF improves recall across most action classes, with the strongest gains for subtle, localized motion and object interactions. Benefits are more variable for large-scale sports and global-motion actions.
- Cross-dataset results: PoseOFF improves recall for 88.33% of NTU RGB+D 60 classes, 98.33% of NTU RGB+D 120 classes, and 80.20% of UCF101 classes.Median recall gains are 2.12% (p < 10^-9), 5.98% (p < 10^-20), and 7.65% (p < 10^-13), respectively.
- Action categories: The largest NTU gains occur in fine-grained daily actions involving hand-object interactions and self-directed activities.Examples include writing, typing, applying cream, and cutting nails, whose subtle local motion is not fully captured by skeletal pose.
- Action categories: UCF101 improvements are strongest for structured body motion and object interaction, but sports-related gains are more variable.Sports classes exhibit both strong improvements and occasional performance drops.
- Qualitative analysis: PoseOFF captures localized motion signatures around joints, including limb rotation and motion toward or away from the camera.Qualitative examples show these cues supporting discrimination under occlusion, subtle motion, and scene clutter.
- Limitations by class: Performance decreases are limited to a small number of classes, typically those dominated by global motion patterns or complex multi-agent interactions.Localized motion cues alone may be insufficient for reliable discrimination in these cases.
E. Performance and Implementation Considerations
PoseOFF adds targeted motion processing while retaining modest inference overhead and compatibility with real-time operation. Integration requires changes only to each model’s embedding layer.
- Computational cost: PoseOFF requires approximately 1060kB of data for a 1920 × 1080 video sequence, versus 60kB for pose-only and 1.66GB for full optical flow.The comparison covers data size and extraction cost across pose-only, PoseOFF, and full optical flow representations.
- Computational cost: PoseOFF increases trainable parameters by an average of 1.88% and adds 5.3 ms and 9.35 ms to postextraction inference time for InfoGCN++ and MS-G3D, respectively.These increases remain small relative to model complexity and are compatible with real-time operation.
- Implementation: PoseOFF integration is limited to the embedding layer of each model, preserving the underlying architecture’s structure and efficiency.This enables straightforward integration into existing skeleton-based pipelines.
V. DISCUSSION AND CONCLUSION
PoseOFF improves early recognition by adding localized motion cues to skeleton-based models with modest overhead. It is particularly useful for subtle local motion, while severe occlusion and highly dynamic global motion remain boundaries.
- Discussion: PoseOFF consistently improves early action recognition under partial observation by capturing fine-grained dynamics absent from skeletal configurations alone.It avoids the overhead of dense optical flow and provides an efficient middle ground between pose-only and computationally intensive motion-based approaches.
- Discussion: Gains are most pronounced for subtle localized motion, including hand-object interactions and self-directed activities.Pose trajectories alone are insufficient for early discrimination in these actions.
- Systems implications: PoseOFF provides a lightweight mechanism for enhancing existing skeleton-based models with targeted motion cues and modest computational overhead.The approach is suited to low-latency and resource-constrained settings.
- HRI implications: By enabling comparable performance with fewer observed frames, PoseOFF supports earlier prediction for motion priming, adaptive planning, and proactive safety responses.These properties are relevant to HRI responsiveness, coordination, and safety.
- Limitations: PoseOFF may be less effective under severe occlusion or highly dynamic global motion because it relies on pose estimation and local motion extraction.Future work targets more efficient motion estimation and real-time embodied-interaction evaluation.