PaperlayerView exact revision on arXiv

Source-linked AI summary

A Black-Box Adversarial Attack on Human Pose Estimation and Keypoint-Based Action Recognition Models

Kacper Mroczek, Michal Kepski

arXiv:2609.08013v1cs.CV

TL;DR

Adversarial attacks on human pose estimation are comparatively underexplored, especially for video and downstream recognition. OKS Attack uses pose similarity as feedback for a temporally coherent black-box attack. It degrades pose quality across evaluated models and reduces downstream action-recognition accuracy in a cross-dataset evaluation.

  • Problem

    Adversarial attacks on human pose estimation remain limited, with most studies focusing on still images and not examining effects on downstream tasks.

    Source: §1 · p. 2

  • Method

    OKS Attack is a decision-based black-box attack that uses Object Keypoint Similarity between clean and perturbed pose predictions as feedback and reuses perturbations to initialize subsequent frames.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    Mean OKS decreases range from 0.0802 to 0.1494 across evaluated models, while downstream accuracy drops range from 6.18 percentage points for ResNet-50 to 13.86 for YOLO-Pose S, with stronger degradation than query-matched random noise for evaluated single-stage models.

    Source: Action recognition results · p. 8, §5 · p. 10

  • Takeaways & Limitations

    The attack applies to both top-down and single-stage pose estimators, and its evaluation finds effects on both top-down pipeline stages as well as single-stage models.

    Source: §5 · p. 10

    The attack uses the pose predicted from the clean frame as its reference, allowing operation without ground-truth keypoint annotations at test time.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Human pose estimation and keypoint-based action recognition models are increasingly deployed as components of video understanding pipelines, yet their vulnerability to adversarial attacks remains insufficiently studied. Temporally coherent black-box attacks have been previously studied in visual object tracking, where the attack feedback can be defined using bounding-box overlap measures such as Intersection over Union (IoU). However, human pose estimation produces keypoint configurations rather than enclosing boxes, making box-level similarity poorly suited for measuring pose degradation. We propose OKS Attack, a decision-based black-box attack that uses Object Keypoint Similarity (OKS) as the attack feedback signal, directly targeting the spatial structure of human poses rather than their enclosing boxes. Experiments on the Penn Action dataset show that OKS Attack consistently reduces pose quality across evaluated pose estimators, with mean OKS decreases ranging from 0.0802 to 0.1494. In a downstream cross-dataset action-recognition evaluation, the attack reduces accuracy by 6.18 to 13.86 percentage points and outperforms query-matched random-noise perturbations. The attack is effective across both top-down and single-stage pose estimation models. The source code will be made publicly available at https://github.com/KacperM33/OKS_attack

1 Introduction

Adversarial attacks on human pose estimation remain limited, particularly for video and downstream tasks. This paper introduces OKS Attack, using keypoint similarity to guide a temporally coherent black-box attack and evaluating its effects on pose estimation and action recognition.

1 Introduction

  • Most prior human-pose attacks target single images and do not examine effects on downstream tasks, despite pose estimation's role in applications such as action recognition.The introduction also notes that HPE has received less adversarial-attack attention than image classification.Source: §1 · pp. 1–2
  • The study evaluates top-down heatmap-based and single-stage pose estimators and tests downstream keypoint-based action recognition, including comparison with query-matched random noise.The work also separates attack effects on the person detector and keypoint estimator in top-down pipelines.Source: §1 · pp. 2–3
  • OKS Attack adapts decision-based black-box attacks to 2D pose estimation by guiding candidate perturbations with scale-normalized Object Keypoint Similarity instead of bounding-box overlap.The attack compares clean and perturbed pose predictions and reuses perturbations to initialize subsequent frames.Source: §1 · p. 2

2 Related Work

Related work describes pose-estimation and action-recognition approaches, then situates adversarial attacks across vision tasks. It identifies a gap in temporally coherent, decision-based black-box attacks using OKS feedback on video-frame pose estimation and downstream action recognition.

2.1 Human Pose Estimation

  • 2D pose estimation includes top-down pipelines with detection followed by per-person estimation, bottom-up joint grouping, and single-stage alternatives that predict people and keypoints together.The section also distinguishes 2D image-plane keypoints from 3D joint locations.Source: §2.1 · p. 3

2.2 Keypoint-Based Action Recognition

  • Keypoint-based action recognition represents pose sequences as joint coordinates, spatio-temporal heatmaps, or skeleton graphs.The approaches use temporal or attention-based models, 3D CNNs, and graph-based models, respectively.Source: §2.2 · p. 3

2.3 Adversarial Attacks

  • Prior video attacks include tracking methods using box-level IoU, while HPE attacks often target still images or use white-box OKS; action attacks commonly perturb skeletons directly.The paper identifies no prior temporally coherent decision-based black-box attack using OKS on video-frame HPE and evaluating downstream keypoint-based action recognition.Source: §2.3 · p. 4

3 Proposed method

OKS Attack adapts decision-based black-box search to pose outputs by measuring the deviation between clean and perturbed pose predictions with OKS. It uses temporal continuity to encourage consistent degradation across video frames.

3.1 Overview

  • The clean-frame pose serves as the reference, and the attack reduces OKS between clean and perturbed predictions without requiring ground-truth annotations.The search begins with a large adversarial perturbation and iteratively reduces its magnitude while preserving the adversarial effect.Source: §3.1 · p. 4, §3.2 · p. 5
  • The perturbation found for one frame initializes attacks on subsequent frames, encouraging temporal consistency; the framework applies to different 2D pose estimators that return keypoints per frame.After optimization, the final perturbation is propagated to the next frame as initialization.Source: §3.1 · p. 5, §3.2 · p. 6

3.2 OKS Attack

  • The attack starts from a noisy image with low pose similarity and seeks to reduce similarity while keeping the added noise small.Source: §3.2 · p. 5
  • Candidate perturbations are normalized to preserve the current perturbation magnitude, queried against the pose estimator, and evaluated using OKS.D denotes a pixel-wise distance measure between images.Source: §3.2 · p. 5
  • The attack combines spatial and temporal OKS scores, using only the spatial term for the first frame and λ to balance pose degradation against temporal consistency.Because lower OKS indicates greater deviation from the reference pose, the lowest-scoring candidate is selected.Source: §3.2 · pp. 5–6

4 Experiments

The experiments evaluate OKS Attack on Penn Action pose estimators and a pretrained cross-dataset action-recognition pipeline. They measure pose degradation, downstream accuracy, and the contributions of detection and pose-estimation stages.

4 Experiments

  • The experiments validate the method on four pose-estimation networks and one keypoint-based action-recognition model using Penn Action.Penn Action is the shared benchmark for assessing the attack’s effects on pose estimation and downstream recognition.Source: §4 · p. 6, §4.1 · p. 6

4.1 Dataset

  • Penn Action supplies 2,326 videos across 15 action classes, with 13 keypoints per frame and action labels for evaluating pose and downstream recognition.These annotations support both pose-estimation assessment and action-recognition evaluation within one dataset.Source: §4.1 · p. 6

4.2 Experiment setup

4.3 Overall Results

  • Action accuracy drops 6.18 percentage points for ResNet-50 and 13.86 for YOLO-Pose S; OKS Attack degrades pose quality across models and beats query-matched noise for single-stage models.The action-recognition evaluation uses an off-the-shelf model not trained on Penn Action, so it measures cross-dataset robustness rather than supervised Penn Action performance.Source: Pose estimation results · p. 7, Table 2 · p. 8, Action recognition results · p. 8

4.4 Ablation Study

  • Lost detections rise from 8.63 to 21.6 per sequence for MobileNetV2 and from 4.91 to 16.32 for ResNet-50 under attack.The results suggest that top-down pose degradation can involve both person localization and pose-estimator errors.Source: §4.4 · p. 9, Table 3 · p. 10
  • ResNet-50’s OKS decrease falls from 0.1316 in the full pipeline to 0.0535 with clean-frame boxes and 0.0411 with ground-truth boxes.The remaining detector-controlled drop, together with nonzero drops for single-stage YOLO-Pose, indicates that the pose estimator is also affected.Source: §4.4 · pp. 9–10, Table 3 · p. 10

5 Conclusions

The paper introduces OKS Attack, a decision-based black-box attack for video-based 2D human pose estimation that uses keypoint similarity as feedback. Penn Action experiments show pose degradation across models, while ablations implicate both detector and pose-estimator stages in top-down pipelines.

5 Conclusions

  • OKS Attack uses Object Keypoint Similarity rather than box overlap as feedback, aligning the attack with structured keypoint outputs.The method is model-agnostic and applies to both top-down and single-stage pose estimators.Source: §5 · p. 10
  • Mean OKS decreases range from 0.0802 to 0.1494 across evaluated models on Penn Action.The attack affects both the person detector and keypoint estimator in top-down pipelines; single-stage results also show pose degradation.Source: §5 · p. 10
  • Future work will examine stronger temporal objectives, physical-world perturbations, and action recognition models trained on the target dataset.Source: §5 · p. 10
Loading 2609.08013v1…