PaperlayerView exact revision on arXiv

Source-linked AI summary

A Deployment Study of Identity-Gated Drone Gesture Control

Diyari Mohammed Salih, Ilyes Chaabeni, Naima Ait Oufroukh

arXiv:2609.25511v1cs.ROcs.CV

TL;DR

Vision-based gesture interfaces accept commands from any hand in view, a concern in shared indoor spaces. IGATE combines gesture control with face verification and deterministic arbitration, then evaluates the stack on a DJI Tello EDU. Its operational measurements differ from offline figures, including face verification and hover-locked gesture comparisons.

  • Problem

    Vision-based gesture interfaces respond to any hand in view, while predictable shared-space interaction calls for enrolled-operator authorization and deterministic arbitration.

    Source: §I · p. 1

  • Method

    IGATE filters gesture and tracking commands through session-level face verification and deterministic state-machine arbitration, and evaluates the stack in flight on a DJI Tello EDU.

    Source: §I · p. 1

  • Results

    19.3% in-flight false rejection versus 0.32% offline equal error rate; hover-locked RBF-SVM accuracy is 0.850 versus 0.651 for the geometric rule.

    Source: §VII · p. 7

  • Takeaways & Limitations

    Operational and offline figures diverge for different reasons across the stack, and per-frame logging supports diagnosing those gaps.

    Source: §I · p. 1, §VI · p. 7

    The results characterize one enrolled operator across four sessions, and the in-flight classifier comparison suppresses actuation.

    Source: §VI · p. 7

Abstract

from arXiv · show

Vision-based gesture control accepts commands from any hand in the camera field of view, which is unsafe in shared indoor spaces. This paper presents IGate, an identity-gated control stack that includes gesture control and face tracking, in which commands are admitted only when an enrolled operator is verified. The system performs few-shot user enrolment from 20 initial face frames, without prior user-specific training: verification compares an embedding of the current face crop against the enrolled template by cosine similarity, while face tracking uses proportional correction. Gesture control is achieved by classifying extracted hand landmarks using an RBF-SVM trained on a custom dataset. Additionally, a hierarchical finite-state machine handles mode selection, default, and fallback behaviours. The approach is tested on a DJI Tello EDU, each component evaluated offline and in-flight across 270 trials (149 flown). Face verification yields a 0.32% offline equal error rate versus 19.3% in-flight. Under hover-locked conditions, the RBF-SVM gesture classifier outperforms the geometric rule (0.850 vs. 0.651 accuracy), with 82% of this gap stemming from the depth channel. All logs and reproduction scripts will be released.

I. INTRODUCTION

IGATE combines identity-gated gesture control with deterministic arbitration and is evaluated in flight on a DJI Tello EDU. The introduction emphasizes measuring component performance through the aircraft’s video pipeline, where offline and operational results can diverge.

I. INTRODUCTION

  • The contribution includes measuring each component both by its offline metric and by its rate through the drone’s video pipeline, with measurements taken in flight.The system is evaluated on the DJI Tello EDU.Source: §I · p. 1
  • A factor-of-23 false-rejection gap arises from aircraft imagery, while gesture accuracy falls from 0.997 to 0.850 almost entirely through the choice of split.The paper distinguishes image-quality effects from evaluation-protocol effects.Source: §I · p. 1
  • IGATE filters gesture and tracking commands through session-level face verification and a deterministic state machine, excluding a local language model from actuation.The stack combines identity-gated interaction with arbitration among behaviours.Source: §I · p. 1
  • 0.850 versus 0.651 is the hover-locked in-flight accuracy comparison between the RBF-SVM and geometric rule.The paper also reports gesture-classifier accuracies of 0.997, 0.910, and 0.783 across three protocols.Source: §I · p. 1
  • The identity gate is characterized by in-flight false rejection and an operator-standoff authorization envelope, alongside the same model’s offline equal error rate.The evaluation decomposes the operational cost into imagery and operating-point effects.Source: §I · p. 1
  • Six behaviours appeared only in flight, and three were corrected with paired measurements on the same protocol.The accompanying audit script rederives quantitative claims from released logs and reports figures it cannot reproduce.Source: §I · p. 1

II. RELATED WORK

Prior gesture-control systems report curated-dataset accuracy and qualitative flight demonstrations, while face-recognition benchmarks may not reflect aircraft imagery. This paper situates its evaluation around operational video, session-independent gesture splits, and the limits of within-subject results.

II. RELATED WORK

  • Gesture-UAV systems report curated-dataset classification accuracy and qualitative flight demonstrations, but not recognition performance through the aircraft video link or authorization against bystander hands.The cited systems map landmarks or learned features to commands and use dedicated datasets such as UAV-GESTURE.Source: §II · p. 2
  • Still-image face-recognition benchmarks such as LFW may not characterize performance on rolling-shutter, compressed aircraft imagery; Section V-C finds that gap in this setting.Cheng et al. [12] report a large drop between benchmark figures and native surveillance imagery.Source: §II · p. 2
  • Session-independent partitioning removes a different leakage axis from random frame splits, but all four sessions use one operator, making every result within-subject and an upper bound on cross-subject performance.Varga [5] shows that random splits over continuous-recording frames overstate gesture accuracy.Source: §II · p. 2

A. Platform and Perception

The platform runs perception and arbitration on a host PC using the Tello EDU’s video stream. Gesture control compares a smoothed geometric rule with a learned classifier operating on hand landmarks, and the two decode different vocabularies.

A. Platform and Perception

  • The DJI Tello EDU streams to a CPU-only host, where a single-slot decoded-frame buffer bounds staleness at the cost of throughput and frame age is logged.Hand detection runs every second frame and face detection every fourth.Source: A. Platform and Perception · p. 2
  • MediaPipe Hands supplies 21 landmarks flattened into 63 coordinates; image-frame normalization removes hand-position dependence but leaves scale dependence, the strongest single predictor of cross-session transfer.The geometric rule smooths these features with an exponential moving average at α = 0.35 and resets when the hand is lost; the learned classifier receives unsmoothed features.Source: A. Platform and Perception · p. 2

B. Two Gesture Vocabularies

  • The geometric rule reads index-finger direction, while the learned classifier reads whole-hand configuration, so each is cued on its own vocabulary rather than matched inputs.A pose that means FORWARD to one has no reading for the other.Source: B. Two Gesture Vocabularies · p. 2
  • The rule emits lateral or vertical commands when the corresponding index-finger vector component exceeds 0.10, and FORWARD or BACK when the bounding-box area change exceeds 0.18.The rule infers depth from change in area; the learned model standardizes the landmarks and uses an RBF-SVM over seven classes, fitted on a stratified 75/25 split at seed 42.Source: B. Two Gesture Vocabularies · p. 2

C. Identity Gate and Arbitration

IGATE runs perception, authorization, and arbitration on the host, admitting gestures only when a hand is present and the operator is authorized. Its identity gate, face-following logic, and prioritized state machine coordinate control and fallback behavior.

C. Identity Gate and Arbitration

  • At startup, 20 accepted face crops are embedded and averaged into an authorization template; later crops are verified by cosine similarity against it.The embedding network is frozen, so enrolment averages its outputs rather than training on the operator.Source: C. Identity Gate and Arbitration · pp. 2–3
  • Authorization uses a Schmitt trigger: it opens at τon = 0.55 and closes below τoff = 0.45 for three consecutive frames.The opening threshold is higher than the offline equal-error threshold of 0.18.Source: C. Identity Gate and Arbitration · p. 2
  • The aircraft carries no computation: host-side perception, authorization, and arbitration are logged per frame, while the language model remains outside the actuation path.The system sends a structured decision summary to a local model for logging only.Source: Figure 1 · p. 3, C. Identity Gate and Arbitration · p. 3
  • The hand overlay numbers 21 landmarks; joints 5 and 8 define the rule’s directional vector, while joint 20 distinguishes LEFT from BACK.Joint 0 is the wrist and joint 4 the thumb tip.Source: Figure 2 · p. 3
  • Face detection and identity verification use different timing rules: arbitration holds the face predicate for 2.0 s across detection gaps, while identity accepts detections only up to 0.35 s old.The age limit avoids embedding a stale box that may now cover something other than the face.Source: C. Identity Gate and Arbitration · p. 3
  • A gesture is admitted when a hand is present and the operator is authorized, but the system does not associate that hand with the verified face.The gate therefore establishes operator presence, not command provenance.Source: C. Identity Gate and Arbitration · p. 3
  • Face following centers the detection box and holds its area at a fixed fraction of the frame, calibrated to a 0.94 m standoff.The embedding decides whether to follow; the box supplies the tracking correction.Source: C. Identity Gate and Arbitration · p. 3
  • The state machine prioritizes battery failsafe, gesture, face following, bounded search, then hover; battery at or below 15% forces an irreversible landing.Modes are held for at least 1.2 s and released only after the signal is absent for 0.8 s; after 10 s without a sighting, bounded search uses a 28 s yaw sweep.Source: C. Identity Gate and Arbitration · p. 3

IV. EXPERIMENTAL SETUP

The experimental setup combines four varied gesture-capture sessions with logged webcam and flight trials, and measures latency through the drone’s own video pipeline.

IV. EXPERIMENTAL SETUP

  • Run logs include telemetry, decisions, per-frame performance, trial records, and a manifest; latency rates count only executed throttled stages and deduplicate frames by sequence number.The manifest records configuration, Git commit, library versions, and host.Source: §IV · p. 3
  • Four seven-class gesture sessions used 400 images per class under deliberately varied lighting and background conditions, yielding 2,715, 2,345, 2,613, and 2,784 usable samples.Session A used daylight and a uniform blue wall; B and C used artificial light and cluttered backgrounds, while D used daylight and a cluttered room.Source: §IV · p. 3
  • Of 270 valid trials, 149 were flown; all 149 flight trials and all 61 webcam-harness trials produced the specified mode transition.One of 150 flight trials was excluded because it began after a failsafe landing and before the next takeoff; the ceiling result cannot distinguish correct from incorrect underlying perception.Source: §IV · p. 3
  • Across 557 s airborne and 12,419 frames, median camera-to-command latency was 18.6 ms and throughput was 22.3 fps.Frame age contributed a median 4.0 ms; in-flight hand detection rose from 24.4 to 26.7 ms and face embedding from 9.0 to 13.0 ms. Gesture inference averaged 0.84 ms for the SVM and 0.07 ms for the rule, so classifier choice was not a latency trade-off.Source: §IV · p. 4

V. RESULTS

Gesture accuracy falls as evaluation moves from within-session splits to held-out sessions and aircraft-encoded frames. The results expose near-duplicate-frame leakage and substantial session-to-session transfer variation.

A. Gesture Classification at Three Protocols

  • Random within-session splits reached 0.9971 at 400 frames per class, but near-identical frames from continuous recordings crossed partitions, consistent with leakage.Accuracy rose with retained sampling density in session A, from 0.9486 at 100 frames to 0.9840 at 250 and 0.9971 at 400; this pattern is consistent with Varga [5].Source: A. Gesture Classification at Three… · p. 4
  • Leave-one-session-out accuracy averaged 0.910, with transfer varying from 0.471 to 0.988 across ordered session pairs.A post-hoc analysis at n = 12 found hand-scale distribution overlap correlated with transfer at Pearson r = 0.76, versus 0.30 for shared background and −0.09 for shared illumination; the analysis is exploratory.Source: A. Gesture Classification at Three… · p. 4
  • Table III covers hover-locked in-flight gesture classification; its caption specifies unconditioned recall and per-hold majority-vote scoring over 42 holds.The caption also identifies bracketed recall as conditioned on detection.Source: Table III · p. 4
  • On 2,947 aircraft-encoded cued frames, the SVM reached 0.783 accuracy and 0.809 over 42 holds; hand detection was complete, so the loss was misclassification.The drone was powered and streaming on the ground with propellers removed; each hold was scored over a six-second capture after a five-second gesture-forming period.Source: A. Gesture Classification at Three… · p. 4
  • Gesture accuracy was 0.997 within session, 0.910 under held-out-session evaluation, and 0.783 on aircraft-encoded frames, with each protocol stricter than the last.The three protocols use a within-session split, leave-one-session-out testing, and operational scoring on frames encoded, transmitted, and decoded through the aircraft pipeline.Source: A. Gesture Classification at Three… · p. 4

B. Both Classifiers in Flight

A hover lock lets both classifiers run through the deployed perception and control pipeline without moving the aircraft. In that comparison, the SVM outperforms the rule, while the results distinguish protocol effects from platform effects.

B. Both Classifiers in Flight

  • Hover-lock captures ran perception, gating, arbitration, and command computation as deployed while holding the transmitted radio-control command at zero.Suppression occurred at the transmit boundary, leaving upstream processing and logged commands unchanged.Source: B. Both Classifiers in Flight · p. 4
  • Across 84 separate-protocol holds, with each classifier cued on its own vocabulary at matched standoff inside the authorization envelope, the SVM scored 0.850 per frame and 0.905 per hold, versus rule scores of 0.651 and 0.714.Hand detection exceeded 0.98 for both; smoothing was applied inside the rule and favoured it, so it cannot account for the direction of the gap.Source: B. Both Classifiers in Flight · p. 4
  • Gesture accuracy was 0.783 on the ground and 0.850 airborne, versus 0.910 across sessions and 0.997 within session, making evaluation protocol rather than platform the governing difference.The reported comparisons distinguish the three evaluation protocols from the aircraft’s video pipeline.Source: B. Both Classifiers in Flight · pp. 4–5
  • A preregistered prediction that SVM RIGHT would collapse into BACK was disconfirmed: RIGHT was correct on 0.985 of cued frames and BACK never occurred as an error target.Instead, BACK errors divided between LEFT at 0.33 and RIGHT at 0.30; the preregistration was written before flight but not version-controlled, and assumed an actuation protocol later revised to hover lock.Source: B. Both Classifiers in Flight · p. 5
  • The 0.199 per-frame accuracy gap was driven chiefly by FORWARD and BACK, which contributed 0.162, or 82%; UP and CENTER contributed the remaining 0.037.The SVM’s lowest static-class recall was DOWN at 0.92; LEFT and RIGHT had F1 scores of 0.841 and 0.808 as BACK leaked into both.Source: B. Both Classifiers in Flight · p. 5

C. Identity Verification

Offline verification reaches a 0.32% equal error rate, but deployed false rejection reflects both aircraft imagery and the stricter operating threshold. Authorization is usable from 0.5–1.5 m, while detections beyond 2.0 m can be background false positives.

Offline benchmark

  • 0.32% equal error rate is achieved against 13,410 negatives at threshold τ = 0.18.The positive set contains 76 operator images; the negatives include webcam images and Labeled Faces in the Wild.Source: C. Identity Verification · p. 5

In-flight verification

  • At the offline threshold, false rejection rises from 0.32% to 7.3% in flight; at τon = 0.55 it reaches 35.1%, falling to 19.3% with hysteresis.The fixed-threshold comparison isolates the imagery cost; the higher deployed threshold adds an operating-point cost.Source: C. Identity Verification · p. 5

Operating envelope

  • Authorization is 80–95% from 0.5–1.5 m, falls to 37% at 1.75 m, and is essentially absent at 2.0 m and beyond.The measured usable envelope is 0.5–1.5 m; the follower target corresponds to 0.94 m.Source: Figure 3 · p. 5, C. Identity Verification · p. 5

Limitation

  • The study does not resolve which image-pipeline factor dominates the in-flight verification loss; isolating causes requires captures varying one factor at a time.Candidates include compression, motion blur, illumination, camera motion, and the unaligned crop from a different detector.Source: C. Identity Verification · p. 5

Beyond the envelope

  • Beyond 2.0 m, background-like scores indicate false-positive face detections, although identity verification correctly refuses them.Arbitration is still told that a face is present.Source: C. Identity Verification · pp. 5–6

D. Deployment Behaviours

Six behaviors appeared only in flight, and three were corrected with paired measurements. The deployment findings expose interactions among perception, timing, arbitration, radio conditions, and fallback behavior.

Stale bounding boxes

  • Restricting identity crops to detections no older than 0.35 s reduced background-population scores from 56% to 20% and raised authorized frames from 41% to 68%.Spurious search decisions fell from 130 to 4; the stale face predicate had caused the same held box to be re-embedded.Source: D. Deployment Behaviours · p. 6

Authorization hysteresis

  • Hysteresis reduced mode switches from 78.5 to 15.7 per minute, but authorized 5.6% of impostor frames across 20 trials.Without hysteresis, no impostor was authorized.Source: D. Deployment Behaviours · p. 6

Depth commands

  • Static-pose depth commands reached 0.777 for FORWARD and 0.336 for BACK, versus 0.000 and 0.008 for the motion cue under the same flight protocol.The motion cue lacked a steady state; 67% of BACK holds emitted a spurious FORWARD command.Source: D. Deployment Behaviours · p. 6

Vocabulary packing

  • BACK was also confused with LEFT because the packed gesture vocabulary separates several closed-fist poses by a single digit that landmark estimation did not resolve.On BACK holds misread as LEFT, little-finger extension measured 0.884 versus 0.551 on correctly read frames; the authors attribute this to landmark estimation, not classification.Source: D. Deployment Behaviours · p. 6

Search coverage and battery failsafe

  • The deployed search covered a median 318° over 22 episodes, while stepping down the battery threshold produced eleven automatic landings in one 580 s flight.The 28 s sweep duration matches a full rotation at approximately 13°/s.Source: D. Deployment Behaviours · p. 6

Radio environment

  • Stream rate varied from 4 to 24 fps across 31 runs; in the same room, flight achieved 8.4 fps with 48 reconnections, versus 21.8 fps on the ground without reconnections.The flight link also experienced SDK timeouts while airborne, including a failed emergency stop.Source: D. Deployment Behaviours · p. 6

Reason-only language model

  • The reason-only language model averaged 1,766 ms latency against a 100 ms command interval, and emitted a low-battery sentence at charges as high as 94%.Eleven of 199 calls timed out; these properties motivate excluding the model from actuation.Source: D. Deployment Behaviours · p. 6

VI. DISCUSSION

Deployment failures arose both from interactions among individually correct components and from a gesture vocabulary that exceeded landmark resolution. Per-frame logging and operational measurements exposed issues that scenario validation and component-level metrics missed.

Cross-component failures

  • Three deployment behaviors arose between individually correct components, while vocabulary packing instead exposed a specification fault.The stale crop involved a shared temporal predicate, authorization chatter a correct threshold feeding a state machine, and spurious depth commands a rule evaluated at an uncontrolled rate.Source: §VI · pp. 6–7

Measurement practice

  • Per-frame logs, latency conditioned on execution, and operational rates alongside offline metrics exposed behaviors that component-level evaluation and scenario validation missed.Every trial in scenario validation passed, but that did not reveal the internal failures.Source: §VI · p. 7

Scope of results

  • Absolute rates are specific to this aircraft’s camera, encoder, and radio link, while the authors identify couplings, specification errors, and measurement practice as broader lessons.The 19.3% false-rejection rate, 4–24 fps stream, and 0.5–1.5 m envelope are tied to this hardware configuration.Source: §VI · p. 7

Limitations

  • The in-flight classifier comparison suppresses actuation, uses different vocabularies, and covers one operator across four sessions.The gate verifies an authorized operator is present, not that the commanding hand belongs to that person; radio-environment evidence comes from two sites.Source: §VI · p. 7

VII. CONCLUSION

IGATE was evaluated across 270 logged trials, including 149 flown, and the conclusion emphasizes mismatches between offline and operational measurements. The paper also specifies its biometric scope, data handling, flight safeguards, and release plans.

VII. CONCLUSION

  • Every valid flight trial produced its specified behavior across an evaluation of 270 logged trials, 149 of them flown.The conclusion reports successful specified behavior for every valid flight trial.Source: §VII · p. 7
  • In flight, face verification has 19.3% false rejection against a 0.32% offline equal error rate, while hover-locked gesture accuracy is 0.850 for the RBF-SVM and 0.651 for the geometric rule.The conclusion attributes 82% of the gesture-classifier gap to the depth channel and supports reporting operational rates alongside offline benchmarks.Source: §VII · p. 7

ETHICS, DATA AND CODE AVAILABILITY

  • Operator face images were collected by the operator, public face datasets were used only for offline evaluation, and the enrolled embedding is held in volatile session memory.The embedding is not persisted; the system has no liveness detection or anti-spoofing and is explicitly not a biometric security mechanism.Source: ETHICS, DATA AND CODE AVAILABILITY · p. 7
  • Indoor flights used propeller guards, conservative command limits, and manual supervision; implementation, instrumentation, run logs, and generation scripts will be released.Source: ETHICS, DATA AND CODE AVAILABILITY · p. 7
Loading 2609.25511v1…