PaperlayerView exact revision on arXiv

Source-linked AI summary

StickFigure: Enabling Pose-Only IoT Cameras for Privacy-Preserving Edge Vision

Aveline Xiaoyuan Han, Chenshu Wu

arXiv:2610.34130v1cs.CV

TL;DR

Indoor cameras can support body-motion sensing, but exporting RGB exposes sensitive appearance and household details. StickFigure keeps RGB local and exports compact pose streams, using an activation-aware backbone and a hierarchical refiner for on-device pose extraction. It reports the highest MCU-scale pose accuracy and HAR utility on two benchmarks while substantially reducing model cost relative to an RTMPose-S-based reference.

  • Problem

    Indoor-camera RGB pipelines expose faces, clothing, and household context, while existing privacy-aware outputs lack an inspectable interface with broad body-motion utility and few address MCU deployability.

    Source: §1 · pp. 1–2

  • Method

    StickFigure keeps RGB local during monitoring and exports compact skeleton streams, using an activation-aware pose backbone and a hierarchical refiner for elbow and wrist coordinates.

    Source: §1 · p. 2

  • Results

    StickFigure achieves the highest pose accuracy and downstream HAR utility among MCU-scale methods on NTU RGB+D and Penn Action, using 96.5% fewer parameters and 99.3% fewer MACs per frame than the RTMPose-S-based reference.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The system demonstrates pose-only monitoring with RGB kept local and compact skeleton streams exported for downstream body-motion sensing.

    Source: §6 · p. 12, §7 · p. 13

    The current system focuses on single-person or primarily single-person activity; tasks requiring faces, text, object identity, fine hand–object state, or detailed scene context are outside its scope.

    Source: §2 · p. 4, §6 · p. 13

Abstract

from arXiv · show

Indoor IoT cameras can support assisted living and activity-aware spaces, but exporting raw RGB frames raises privacy concerns by exposing facial, clothing, and household details that body-motion tasks often do not require. We present StickFigure, a pose-only IoT camera system that, during real-time monitoring, keeps raw frames local and ephemeral while exporting only compact skeleton streams sufficient for downstream human activity recognition (HAR). While pose estimation has matured on GPUs, extracting accurate pose streams on resource-constrained IoT cameras remains difficult. To overcome these challenges and enable on-device pose streams on (almost) any IoT camera, we present two core designs in StickFigure: an activation-aware pose backbone that allocates channel capacity after spatial downsampling and a lightweight hierarchical MLP refiner that selectively corrects elbow and wrist coordinates using intermediate visual features and kinematic context. We fully implement StickFigure and evaluate it on public benchmarks and real-world deployment. Compared with state-of-the-art MCU pose models, StickFigure achieves the highest pose accuracy and downstream HAR utility on both NTU RGB+D and Penn Action, while using 96.5% fewer parameters and 99.3% fewer MACs per frame than RTMPose-S-based baselines, achieving 0.107 s p95 latency on a representative STM32H753 MCU. We believe StickFigure establishes a new operating point for privacy-aware edge vision on massive IoT cameras.

1 INTRODUCTION

Indoor cameras can support body-motion sensing, but transmitting or retaining RGB exposes sensitive appearance and existing privacy-aware outputs either limit reuse or remain opaque. StickFigure keeps RGB local, exports pose streams, and combines MCU-focused model designs with deployment adaptations; it reports the highest MCU-scale pose accuracy and HAR utility on two benchmarks.

Motivation and gap

  • Existing privacy approaches lack an inspectable, semantically clear output interface that retains broad downstream body-motion utility, and few address MCU deployability.Data-access and capture methods have limitations, while task-specific labels constrain reuse and implicit representations remain opaque and vulnerable to reconstruction or attribute-leakage attacks.Source: §1 · pp. 1–2

Pose as the output interface

  • StickFigure selects pose as its output interface because joint geometry and temporal motion support many human-centric HAR tasks without retaining full appearance-rich RGB.The predefined joint semantics make the released representation inspectable and reusable across recognizers, unlike opaque embeddings or fixed activity labels.Source: §1 · p. 2

Evaluation overview

  • Among MCU-scale methods, StickFigure achieves the highest pose accuracy and downstream HAR utility on both NTU RGB+D and Penn Action.It uses 96.5% fewer parameters and 99.3% fewer MACs per frame than the RTMPose-S-based reference, with 0.107 s end-to-end latency per frame on STM32H753.Source: §1 · p. 3
  • The evaluation covers two public benchmarks, three indoor sites, a roughly 40-minute annotated single-person case study, and four hardware platforms.On STM32H753, the real-world HAR deployment reaches 96.45% macro-F1 versus 89.31% for XiNet Pose.Source: §1 · p. 3

System design

  • The system combines a backbone that shifts channel capacity to downsampled stages with a hierarchical MLP that selectively refines elbows and wrists using visual features and kinematic context.The designs target tight MCU memory and compute budgets while preserving distal-joint fidelity.Source: §1 · p. 3, Figure 2 · p. 3

2 MOTIVATION AND SCOPE

StickFigure targets private indoor body-motion sensing with a camera-side trust boundary that prevents RGB frames from leaving the device during monitoring. Its pose interface and evaluation are scoped to primarily single-person activity and exclude tasks requiring detailed appearance or scene information.

Pose interface and joint selection

  • Pose preserves articulated body geometry and temporal motion while suppressing much of RGB’s facial, clothing, background, and scene appearance.Its predefined joint semantics remain interpretable and reusable across body-motion recognizers, unlike opaque learned representations or fixed activity labels.Source: §2 · p. 3

Privacy scope

  • The trusted boundary is at the camera MCU: RGB is discarded before any monitoring image leaves the camera, while exported skeleton histories and downstream applications remain outside that boundary.The 12-joint schema is a compact interface choice, not a universally optimal configuration.Source: §2 · p. 4
  • The current system focuses on single-person or primarily single-person activity; tasks requiring faces, text, object identity, fine hand–object state, or detailed scene context fall outside the pose interface’s scope.The STM32H753 is the primary operating point, with a 256 KiB limit on the largest individual activation tensor.Source: §2 · p. 4

3 SYSTEM DESIGN

StickFigure’s camera pipeline estimates a 12-joint pose, refines distal arm joints, and exports fixed-format skeleton records for downstream HAR while keeping RGB local. Its design combines activation-aware capacity allocation, kinematic refinement, site calibration, and runtime controls for constrained devices and deployment conditions.

3.1 SticKFiguRe Overview

  • The system estimates a 12-joint pose with an activation-aware backbone, selectively refines distal arm joints, and packages the result for skeleton-only export.Downstream software transforms skeleton sequences into temporal motion features for HAR; one-shot site calibration and motion-triggered execution address deployment shift and inactive-period computation.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2.1 Capacity Allocation.

  • Because expanded activation size scales with spatial area, the backbone keeps high-resolution stages narrow and increases channel capacity after downsampling.Its four-stage inverted-bottleneck architecture progressively reduces resolution while increasing channels, omitting squeeze-and-excitation and attention for a compact MCU INT8 operator set.Source: §3.2.1 · p. 4, §3.2.2 · p. 5

3.2.3 Coordinate and Confidence Heads.

  • The coordinate head predicts joint locations directly rather than materializing dense heatmaps, using axis-wise distributions and bounded residuals for sub-bin correction.A separate confidence head outputs one value per joint; the canonical backbone has 284.2K parameters, approximately 29.54M MACs per frame, and 248.64 KiB profiled peak activation memory.Source: §3.2.3 · p. 5

3.3 Hierarchical Distal-Joint Refinement

  • A lightweight, frame-local hierarchical MLP refines only the left and right elbows and wrists, which are sensitive to compression and important for arm-centric activities.It samples six 24-dimensional feature vectors per refined joint from the Stage 1 feature map, concatenates them into a visual descriptor, and appends an 8-dimensional joint-type embedding.Source: §3.3 · p. 5, §3.3.1 · p. 5

3.3.2 Kinematic Update.

  • Refinement proceeds from elbow to wrist, using the shoulder as elbow context and the refined elbow as wrist context; only four distal arm joints change.Residual components are bounded to [−0.10, 0.10], updated coordinates are clipped to [0, 1], and the path adds 10.1K parameters and 0.02M MACs without increasing profiled peak activation memory.Source: §3.3.2 · p. 5

3.4.1 Temporal Representation.

  • HAR uses sequences of predicted skeletons represented with root-centered positions, velocities, and bone features, and each pose estimator is evaluated with its own fine-tuned HAR model.The pose-source-matched protocol exposes the HAR model to that estimator’s coordinate errors and temporal behavior rather than idealized ground-truth poses.Source: §3.4.1 · p. 6, §3.4.2 · p. 6

3.5.1 Calibration Input and Synthesis.

  • One-shot calibration uses a single empty-scene RGB image, composites labeled source-person foregrounds onto the target background, and adapts the estimator without target-person images or pose annotations.Calibration starts from a pretrained model, updates only selected final layers offline, then freezes the adapted model for monitoring; it adds no per-frame camera-path computation.Source: §3.5 · p. 6, §3.5.1 · p. 6, §3.5.2 · p. 6

3.6.1 Continuous Camera Pipeline.

  • The camera pipeline uses alternating DMA capture buffers and a latest-frame-wins policy, discarding processed RGB frames and retaining no RGB history during monitoring.Motion checking runs once per second; detected motion invokes pose inference, and after approximately 5 seconds without motion the system returns to the check state.Source: §3.6.1 · p. 6, §3.6.2 · p. 6

4 EVALUATION

The evaluation uses two public action datasets and controlled MCU-scale baselines to assess pose accuracy, downstream activity recognition, resource use, and deployment performance across hardware platforms.

4.1 Experimental Setup

  • Live evaluation also measures pose and HAR under deployment shift, one-shot calibration, adaptive execution, MCU power, and a separate long-horizon case study.The long-horizon study is described as single-elderly monitoring; deployment evaluation uses three indoor environments.Source: §4.1 · p. 7

4.1.1 Datasets.

  • NTU RGB+D 60 provides the primary benchmark using 20 pose-dominant classes, 13,440 clips, 40 subjects, and three synchronized views; Penn Action tests transfer across 15 actions and varied capture conditions.NTU uses the official cross-subject protocol; Penn Action uses a standard 1,258-sequence training and 1,068-sequence test split, with estimators fine-tuned before evaluation.Source: §4.1.1 · p. 7

4.1.2 Baselines.

  • Controlled MCU-scale comparisons use shared splits, training, and evaluation, with common pose heads isolating backbone design; higher-resource models provide accuracy references.The compared MCU variants include Tiny MobileNet Pose, XiNet Pose, Uniform-Expansion, Progressive-Expansion, the backbone alone, and full SticKFiguRe.Source: §4.1.2 · p. 7

4.1.3 Evaluation Metrics.

  • Evaluation measures FP32 and INT8 PCK@0.10, HAR Top-1 accuracy, fine-arm recall, model resources, and end-to-end p95 pose latency.HAR uses CTR-GCN on predicted 64-frame skeleton sequences; reported resources include parameters, MACs/frame, Flash, and peak activation RAM.Source: §4.1.3 · p. 7

4.1.4 Platforms.

  • The primary live-camera platform pairs an STM32H753ZIT6 with an OV2640 capturing 144 × 144 RGB565 frames, while scaling tests add three edge platforms.Additional platforms are ESP32-S3, Raspberry Pi 4, and Jetson Orin Nano.Source: §4.1.4 · p. 7

4.2 Overall Performance

SticKFiguRe leads MCU-scale methods in pose and HAR on both benchmarks, while resource comparisons and INT8 tests characterize its operating point and robustness.

4.2.1 Pose and Downstream Utility.

  • 78.40% PCK@0.10, 74.99±1.06% HAR Top-1, and 63.10% fine-arm recall on NTU RGB+D accompany SticKFiguRe’s strongest MCU-scale pose and HAR performance on both datasets.On Penn Action it again leads MCU-scale methods in all three metrics; frame-level localization alone does not fully determine skeleton-based HAR utility.Source: §4.2.1 · p. 7, Table 1 · p. 8
  • Fine-arm recall gains exceed aggregate PCK gains on NTU relative to the backbone alone, with improvements concentrated at wrists and elbows.The stated joint-wise pattern connects the targeted refinement to arm-centric activity recognition, while non-MCU references remain more accurate.Source: §4.2.1 · p. 8

4.2.2 Resource Efficiency.

  • 51.6% fewer parameters, 52.1% less Flash, 60.6% fewer MACs/frame, and 12.3% less peak activation RAM than XiNet Pose accompany higher PCK and HAR on both datasets.SticKFiguRe’s 289.18 KiB Flash footprint is the smallest among MCU-scale methods; its 248.64 KiB peak activation RAM is below XiNet Pose and expansion variants, while the refiner does not raise the measured maximum.Source: §4.2.2 · p. 8

4.2.3 INT8 Robustness.

  • 77.95% PCK@0.10 on NTU RGB+D and 77.14% on Penn Action after INT8 conversion show that SticKFiguRe retains accuracy at deployment precision.Progressive Expansion drops sharply on NTU, while the backbone and full model show smaller degradation.Source: §4.2.3 · p. 8, Table 2 · p. 9

4.3.1 Expansion Allocation.

  • The selected (3, 3, 4, 5) schedule allocates added capacity to downsampled stages while keeping the largest individual activation tensor at 248,832 B across schedules.Stage 0 has the largest tensor; later stages retain channel headroom under that fixed tensor-size ceiling. This is distinct from peak activation memory.Source: §4.3.1 · p. 9, Figure 5 · p. 9

4.3 Design Analysis

The design analysis tests how activation-aware expansion allocates capacity and how hierarchical refinement affects distal joints and downstream utility.

4.3 Design Analysis

  • Relative to Peak-Matched Uniform, the selected schedule increased PCK by 1.89 points at a cost of 5.37M additional MACs/frame, yielding 0.352 PCK points per additional 1M MACs/frame.The aggressive later-heavy schedule increased PCK by a further 1.51 points.Source: §4.3 · p. 9

4.3.2 Targeted Distal Refinement.

  • On the full-frame NTU evaluation, wrist and elbow PCK rise by 2.05 and 0.45 points, while other joints remain unchanged at reported precision.Overall NTU PCK rises 0.40 points, HAR Top-1 rises 1.34 points, and fine-arm recall rises 2.7 points; the refiner adds 10.1K parameters and 0.02M MACs/frame while preserving peak activation memory at 254,608 B.Source: §4.3.2 · p. 9

4.4 Live Deployment

Live deployment tests examine one-shot calibration, adaptive power use, and long-horizon activity monitoring. In an annotated 40-minute session, SticKFiguRe improves activity-state accuracy, duration estimation, and transition responsiveness over XiNet Pose.

4.4.1 Deployment Performance.

  • 78.40% PCK@0.10 is the dataset reference; empty-scene one-shot calibration yields 74.16%, 70.12%, and 66.30% in Envs. A, D, and E, while target-person supervision yields 82.33%, 84.29%, and 80.13%.The target-supervised results exceed empty-scene calibration by 8.17, 14.17, and 13.83 points, respectively; Fig. 6(a) compares these calibration results with the dataset-domain reference.Source: §4.4.1 · p. 10, Figure 6 · p. 10
  • 72.22±1.57% and 73.89±2.83% HAR Top-1 in two independently collected Env. A sessions are 2.77 and 1.10 points below the 74.99±1.06% dataset reference.The sessions differ in appearance and lighting; the evaluation measures skeleton-representation utility across capture conditions, not cross-session transfer of a fixed HAR model.Source: §4.4.1 · p. 10, Figure 6 · p. 10

4.4.2 Power and Adaptive Execution.

  • 611.35 ± 3.59 mW at 3.293 V and 8.311 ± 0.067 skeletons/s corresponds to 73.57 ± 0.84 mJ per emitted skeleton during continuous execution.These MCU-side measurements use the production firmware across three 60-s runs after a two-minute warm-up.Source: §4.4.2 · p. 10
  • About 63% lower MCU-side current during inactivity comes from MOTION_CHECK at approximately 66–68 mA versus POSE_INFERENCE at approximately 179–182 mA.MOTION_CHECK checks for motion once per second; detection activates inference, and approximately 5 s of inactivity returns the system to checking. Motion-to-first-skeleton latency is 121 ms, while active throughput is 8.222 FPS versus 8.236 FPS continuously; Fig. 7 traces two entry periods.Source: §4.4.2 · p. 10, Figure 7 · p. 11

4.4.3 Three-day Monitoring Case Study.

  • Across three daytime sessions without dense ground truth, activity-state predictions disagree with XiNet Pose for 7.9% to 10.4% of intervals, and SticKFiguRe has fewer state transitions per hour each day.The transition counts are 98.6 versus 137.6, 105.4 versus 152.7, and 103.0 versus 147.4; without ground truth, these timelines do not establish absolute recognition accuracy.Source: §4.4.3 · p. 10
  • 96.45% four-state macro-F1 versus 89.31% for XiNet Pose; mean absolute activity-duration error falls from 17.62% to 3.15%, first-correct-frame delay from 1.193 s to 0.052 s, and erroneous switches from 208 to 91.These results come from a separate approximately 40-minute session with ground-truth annotations.Source: §4.4.3 · pp. 10–11

4.5 Cross-Platform Scaling

Cross-platform evaluation selects the highest-PCK feasible INT8 configuration for each architecture under a shared real-time gate. SticKFiguRe retains the highest pose and HAR utility across four hardware tiers, with its advantage widening under tighter resource budgets.

4.5 Cross-Platform Scaling

  • Feasible candidates must meet INT8 execution, memory, and storage requirements plus p95 latency ≤125 ms and sustained throughput ≥8 FPS.Among feasible configurations, each architecture’s highest-PCK@0.10 candidate is selected using a one-epoch INT8 validation screen, then trained for eight epochs; test-set pose accuracy and HAR results are excluded from selection.Source: §4.5 · p. 11
  • At ESP32-S3, SticKFiguRe achieves 72.9% INT8 PCK@0.10 and 63.5% HAR Top-1, exceeding the next strongest method by 5.0 and 7.7 percentage points, respectively.Across all four tiers, SticKFiguRe has the highest-utility feasible operating point and degrades more gradually under resource-driven scaling.Source: §4.5 · p. 11
  • Performance differences are small on larger platforms but widen under tighter resource budgets; at ESP32-S3, SticKFiguRe reaches 72.9% INT8 PCK@0.10 and 63.5% HAR Top-1, beating the next strongest method by 5.0 and 7.7 percentage points.Source: §4.5 · p. 11

5 RELATED WORK

Prior work addresses privacy through data access, capture, and output-representation changes, while efficient pose models and runtimes leave continuous Cortex-M pose extraction with downstream utility as an unresolved operating point.

5 RELATED WORK

  • Privacy approaches restrict video access, alter capture, or export task-specific labels; skeleton methods instead encode articulated motion, omitting much appearance and background information without a formal privacy guarantee.The related work also describes systems that select, compress, or reconstruct image-bearing outputs, and prior 2D pose export on more capable embedded hardware.Source: §5 · p. 11
  • Efficient pose models target phones, SBCs, or accelerators, while XiNet-Pose is the closest Cortex-M precedent; parameters and MACs alone do not establish deployability.Activation memory, runtime buffers, workspace, firmware state, quantized operator support, kernel behavior, and joint-localization quality also matter.Source: §5 · p. 11
  • Continuous articulated pose extraction on a Cortex-M camera within tight memory and runtime limits, while retaining downstream temporal utility, remains the unresolved operating point.SticKFiguRe targets this regime by converting local RGB into skeleton streams and explicitly evaluating downstream HAR utility; Fig. 9 describes cross-platform INT8 pose and HAR utility among feasible configurations.Source: §5 · p. 12, Figure 9 · p. 12

6 DISCUSSION AND FUTURE WORK

The discussion frames privacy as a camera-side egress boundary and describes lower-detail calibration inputs, alongside scope limits and possible extensions.

6 DISCUSSION AND FUTURE WORK

  • The trusted boundary sits at the camera MCU, confining raw RGB to the device; SticKFiguRe exports 76 B SFSK records at 8.222 FPS, about 5.0 kb/s.A 144 × 144 RGB565 frame contains 41,472 B, more than 500 times one SFSK record; bounded format and rate could permit admission of only valid records.Source: §6 · p. 12
  • Strong blur largely preserves calibration accuracy, and downsampling to 36 × 36 produces comparable or better PCK across three sites, while 18 × 18 is less consistent.Reducing the calibration snapshot before adaptation can reduce exposed visual detail; the source image could be reduced on-device and immediately discarded.Source: §6 · pp. 12–13
  • The current design targets single-person or primarily single-person monitoring; multi-person pose extraction with temporal association is a proposed extension.The fixed 12-joint stream suits the studied body-motion tasks, while other applications may need different semantic granularity.Source: §6 · p. 13

7 CONCLUSION

StickFigure keeps RGB local during monitoring and exports pose streams for body-motion sensing, jointly addressing pose extraction, MCU constraints, and temporal utility.

7 CONCLUSION

  • StickFigure keeps RGB local during monitoring and exports only pose streams for downstream body-motion sensing.The system evaluates whether those pose streams remain useful beyond frame-level pose accuracy.Source: §7 · p. 13
  • Its activation-aware backbone limits large high-resolution activations through late channel expansion, while a hierarchical refiner improves task-critical distal-joint localization.Source: §7 · p. 13
  • StickFigure sustains real-time execution on an STM32H753 MCU and degrades more gracefully than state-of-the-art baselines as computation resources tighten.Source: §7 · p. 13

A FINE-ARM ACTIVITY SUBSETS

Fine-arm recall averages per-class recall over predefined arm-centric action subsets, with separate subsets for NTU RGB+D 60 and Penn Action.

A FINE-ARM ACTIVITY SUBSETS

  • Fine-arm recall is the macro-average of per-class recall over predefined arm-centric action subsets.Source: A FINE-ARM ACTIVITY SUBSETS · p. 14
  • The NTU RGB+D 60 subset includes nine specified actions, from clapping and hand waving to touch back and use fan.The actions are identified by codes A010, A023, A031, A033, A038, A040, A044, A046, and A049.Source: A FINE-ARM ACTIVITY SUBSETS · p. 14
  • The Penn Action subset includes seven actions: baseball pitch and swing, bowling, golf swing, strumming guitar, tennis forehand, and tennis serve.Source: A FINE-ARM ACTIVITY SUBSETS · p. 14

Related papers

Loading 2610.34130v1…