AdaptaFace: Adversarial Assistance for Inclusive Facial Gesture Interaction

• Existing personalization either limits users to exposed mapping configurations or requires calibration data and demonstrations, motivating compatibility improvements without model changes or canonical-gesture conformity.Source: §1 · p. 2
• AdaptaFace mediates between user expression and recognizer expectations with subtle, proxy-dependent perturbations, leaving the AU-based recognizer unchanged.Source: §1 · p. 2
• The work combines interviews with 12 PwM, a browser system with four personalization modes, and a four-task evaluation with 16 participants plus offline transfer analyses.Source: §1 · p. 2
• The formative interviews, participant study, and offline validations provide evidence of improved performance, lower workload, greater perceived control, broader usable gesture vocabularies, and limits to cross-recognizer transfer.Source: §1 · p. 2

2.1 Facial Gesture Interaction as Accessible Input

• Facial movements can support accessible interaction through commodity cameras, but users’ subtle, asymmetric, or variable movements may not match canonical recognizer vocabularies.Source: §2.1 · p. 2

2.2 Personalization in Recognition-Based Accessibility

• Personalization commonly changes mappings or adapts models with user-specific data, while an input-side alternative could improve compatibility without changing model parameters.Source: §2.2 · p. 3

2.3 Adversarial Methods as Input-Side Adaptation

• AdaptaFace applies adversarial input transformation to accessibility, keeping model parameters fixed while mediating the image presented to the recognizer.Source: §2.3 · p. 3

3.1 Participants and Method

• The formative study included 12 PwM with varied facial control, asymmetry, fatigue, and movement consistency.Source: §3.1 · p. 3
• The interviews translated documented accessibility barriers and participant experiences into design principles, rather than claiming the barriers themselves were new.Source: §3.1 · p. 3

3.2 Findings

• Participants described canonical gestures that were difficult to reproduce; comfortable movements could be asymmetric or subtle and frequently mismatch recognizers.Source: §3.2 · pp. 3–4
• Repeated demonstrations, recalibration, and adaptation were tiring or frustrating, especially when performance varied with posture, fatigue, or day-to-day condition.Source: §3.2 · p. 4
• Participants valued natural, feasible, socially acceptable gestures and wanted visibility into how the system interpreted their actions.Source: §3.2 · p. 4

3.3 Design Principles

• The six principles call for user-compatible gestures, low-burden personalization, user-defined vocabularies, user control, pipeline compatibility, and low-effort socially acceptable interaction.Source: §3.3 · p. 4

4.1 Compatibility-Layer Logic

• The compatibility layer transforms live facial input so a user-compatible gesture is more likely to map to an intended command, while the recognizer, label space, and application command mapping remain unchanged.Source: §4.1 · p. 4
• Adversarial assistance uses bounded, visually subtle perturbations for recognition enhancement within the native vocabulary or extension toward an existing recognizer output.Source: §4.1 · p. 4

4.2 System Overview

• Registration associates each gesture with a target recognizer output; matched modes select a compatible target once, and runtime applies assistance toward that assigned target.Source: §4.2 · p. 4

4.3 Matching, Mediation, and Direct Triggering

• At runtime, detected gestures condition frame-level perturbation before the assisted image reaches the recognizer; matching-only cannot change AU output when the raw frame is passed through.Source: §4.2 · p. 4, §4.3 · p. 5
• The matcher selects the highest-similarity registered trajectory after it remains top-ranked and above threshold for three consecutive windows, then retrieves its internal target.Source: §4.3 · p. 5

4.4 Adversarial Assistance Mechanism

• A differentiable proxy and targeted loss generate a perturbation bounded by ε; the target is assigned during registration, while the downstream recognizer remains unchanged.Source: §4.4 · p. 5

4.5 Four Personalization Modes

• The four modes combine fixed or matched target assignment with enhancement inside the recognizer vocabulary or extension using user-defined gestures.Source: §4.5 · p. 5

4.6 Parameter Settings

• User-facing controls trade steering effectiveness against visual effects, smoothness, latency, responsiveness, stability, and accidental repeated triggers.Source: §4.6 · p. 6

5.1 Study Design

• The study used a counterbalanced within-subject 2 × 4 × 4 design with 16 adults with motor impairments, each completing 32 planned task blocks.Participants had reduced or unreliable upper-limb function and could perform at least one facial or head gesture.Source: §5.1 · p. 6, §5.2 · p. 6

5.3 Apparatus, Tasks, and Procedure

• Participants used a secure browser client with their own webcams to perform Swipe, Click, Zoom In, and Zoom Out tasks, with gesture choice and review varying by mode.Participants could review or re-record gestures between blocks and take breaks or split sessions.Source: §5.3 · pp. 6–7

5.4 Measures; 5.5 Data Analysis

• Measures covered task performance, block-level completion, event confidence, subjective ratings, gesture mappings, and interview feedback; repeated-measures outcomes used GEE or mixed-effects models.Interview themes contextualized the quantitative findings rather than serving as a standalone qualitative contribution.Source: §5.4 · pp. 7–8, §5.5 · p. 8

5.6.1 Overall Objective Performance.

• Mean completion rate rose from 59.7% to 90.5%, task completion time fell from 16.6 s to 13.4 s, and success-normalized task time fell from 37.7 s to 15.9 s.Mean event-level confidence increased from 0.65 to 0.89.Source: §5.6.1 · p. 8, Table 4 · p. 8
• Completion improved in every mode, with larger descriptive changes in extension than enhancement; no Technology × Mode interaction was significant.Fixed Enhancement rose from 77.3% to 94.8%, Matched Enhancement from 77.6% to 92.7%, Fixed Extension from 27.9% to 82.4%, and Matched Extension from 55.8% to 91.9%.Source: §5.6.1 · p. 8, §5.5 · p. 8, Table 5 · p. 9

5.6.2 Subjective Experience.

• Ease increased from 3.70 to 4.35 and perceived control from 3.15 to 4.27; SUS rose from 47.3 to 70.8 and NASA Raw TLX fell from 5.61 to 4.21.Workload reductions spanned cognitive, physical, temporal, effort, and frustration scores.Source: §5.6.2 · p. 8, Table 6 · p. 9

5.6.3 Gesture Vocabulary Extension.

• All 16 participants introduced personalized gestures in extension modes, and several gestures were reused across tasks to form personal repertoires.Oral/lip gestures were most frequent (40 instances), followed by ocular/gaze/brow (38), cheek (5), and tongue (4).Source: §5.6.3 · p. 8

5.7.1 AdaptaFace preserved participants’ own gesture abilities and preferences

• Participants valued gestures aligned with their motor abilities, including smaller movements that were less tiring or more sustainable than canonical expressions.One participant described using a small mouth movement in custom mode when opening wide was difficult.Source: §5.7.1 · p. 9

5.7.2 AdaptaFace reduced adaptation burden and improved perceived ease of use

• Self-defined gestures reduced the need to exaggerate movements and remember system-imposed gestures, helping explain participants’ reports of lower effort.Participants contrasted Baseline’s wide-mouth expressions with AdaptaFace as “less tiring.”Source: §5.7.2 · p. 9
• Mirror views, live labels, and immediate responses helped participants inspect recognition and mappings, but technical controls could add setup burden.Adjustable assistance supported agency when participants could see its effects and understand the controls.Source: §5.7.2 · p. 9

5.7.3 AdaptaFace expanded participants’ usable gesture vocabulary

• Selectable gestures offered bounded choices, while custom gestures improved personal fit and memorability; participants valued alternatives for fatigue or changing contexts.Preferences for subtle gestures in public also connected vocabulary extension with social comfort.Source: §5.7.3 · p. 9

6 Offline Validation: Cross-Dataset, Cross-Model, and Model-Transfer Analyses

• Aligned generation and evaluation produced the strongest steering, while OpenGraphAU-to-OpenFace transfer was substantially weaker.Table 7 summarizes the aggregate comparison across the three validation settings and both state types.Source: Table 7 · p. 10, §6 · p. 10
• In cross-dataset enhancement, target-confidence success rose from 0.3238 to 0.7591, while target top-1 rose from 0.3853 to 0.7863, showing strong aggregate evidence of successful steering.Source: §6 · p. 10
• In cross-dataset validation, substitution-case target-confidence success increased from 0.0223 to 0.5248, while target top-1 increased from 0.0401 to 0.5917, with perturbation generation and evaluation aligned within the same recognizer.Source: §6 · p. 10
• Cross-model validation with OpenGraphAU showed the same overall direction, but pair-level results varied: some AU pairs approached saturation while others remained difficult.The cross-model setting generated and evaluated perturbations within OpenGraphAU.Source: §6 · p. 10
• In OpenGraphAU-to-OpenFace model transfer, enhancement target-confidence success rose from 0.2407 to 0.2665, and substitution rose from 0.0368 to 0.0432.The authors characterize these gains as modest compared with aligned settings.Source: §6 · p. 10
• Random perturbations stayed close to baseline and below adversarial assistance across settings, consistent with gains depending on targeted steering rather than magnitude alone.Source: §6 · p. 10

7.1 Personalization as Accessibility Mediation

• Adversarial assistance mediates the recognizer-facing image so feasible movements can become usable commands without requiring users to conform to canonical gestures.Enhancement supports the recognizer’s existing vocabulary; extension supports subtler or more personal movements without retraining.Source: §7.1 · pp. 10–11

7.2 Integration and Why Image Mediation Matters

• Image mediation fits applications where camera frames can be intercepted but the existing image recognizer is the available integration boundary; direct triggering is an alternative when an application API permits it.The approach could extend to camera-based hand or body gestures, but would require modality-specific optimization and validation.Source: §7.2 · p. 11

7.3 Scope, Limitations, and Future Work

• Evidence is bounded by aligned online proxy/evaluator testing, weaker cross-recognizer transfer, no component ablation, and remote conditions that were not systematically varied.The authors call for further benchmarking across conditions and recognizer pairings.Source: §7.3 · p. 11

8 Conclusion

• AdaptaFace treats personalization as input-side accessibility mediation for facial interaction.Source: §8 · p. 11
• The complete pipeline improved task performance, perceived control, usability, and workload relative to Baseline, while extension modes broadened usable gesture vocabularies.Source: §8 · p. 11
• The evidence is bounded to proxy-compatible target spaces and does not isolate every pipeline component.The paper presents input-side mediation as a promising direction for more user-compatible facial interaction.Source: §8 · p. 11

A.1 PGD-style Perturbation Optimization

• A small number of gradient updates reduce targeted proxy-model loss, with each perturbation projected into a bounded L∞ norm ball to preserve subtlety and real-time performance.The optimization uses a random start; the perturbation visualization is magnified in Figure 1.Source: §A.1 · p. 12

A.2 Target Selection in Matched Modes

• Matched modes select a recognizer-output target during registration by evaluating candidate confidence on the registered sample, accepting the selection only when confidence criteria are met.Confidence scores may be aggregated across frames or checked for consistent top ranking over a short temporal window.Source: §A.2 · p. 13

A.3 Runtime Gesture Detection

• At runtime, a sliding window of incoming AU vectors is matched against registered trajectories; assistance begins only after the best match clears a threshold and can persist across consecutive windows.The study configuration required three consecutive above-threshold frame-level matches to the same template, suppressing transient matches.Source: §A.3 · p. 13
• After detection, AdaptaFace applies bounded adversarial assistance frame by frame toward the target assigned to that registered gesture.Assistance begins at gesture onset rather than waiting for the full gesture sequence to finish.Source: §A.3 · p. 13

B.1.1 Modeling Setup.

• Full-block completion was analyzed with a binomial GEE, while repeated-measures outcomes were modeled with participant-level random-intercept LMMs.Technology, Task, and Personalization Mode were fixed effects; nonsignificant Technology × Task and Technology × Mode interactions led the main paper to focus on Technology main effects.Source: §B.1.1 · p. 13

B.1.2 Objective Performance Models.

• Objective performance models cover full-block completion, task completion time, and success-normalized task time.The supplied specifications identify a binomial GEE for completion and analogous LMMs for TCT and SNT.Source: §B.1.2 · p. 14

B.1.3 Subjective Rating Models.

• Ease and perceived control ratings used LMMs; SUS and NASA Raw TLX were collected once per technology condition and summarized descriptively.The system-level measures were not repeated-measures model outcomes.Source: §B.1.3 · p. 14

B.2 Formative-Study Interview Procedure

• Formative-study interviews were remote video sessions, recorded with consent and structured to let participants pause or withdraw.The main paper reports the interview sequence and analysis purpose in Section 3.Source: §B.2 · p. 14

B.3 Extended Design Principles from the Formative Study

• Six design principles call for user-compatible and user-defined gestures, low-burden personalization, user control, recognizer compatibility, and low-effort socially acceptable interaction.Together, the principles motivate input-level mediation that preserves users’ expressive habits while helping existing recognizers interpret them more reliably.Source: §B.3 · p. 14

B.4 Remote Study Setup

• Figure 4 presents representative setups for the remote study.The supplied passage identifies the figure as illustrating the evaluation context.Source: §B.4 · p. 14

B.5 Task-Level Performance

• Table 13 gives mean and standard deviation task-level performance across four interaction tasks, with Zoom tasks reported separately.The passage identifies the table’s scope but supplies no task-level values.Source: §B.5 · p. 14, Table 13 · p. 16

B.6 Ease and Control Ratings

• Table 14 reports ease and control ratings overall, by task, and by personalization mode, with higher ratings indicating better outcomes.The supplied passages provide the table scope but no rating values.Source: §B.6 · p. 14, Table 14 · p. 16

B.7 Gesture Vocabulary Inventory

• Table 15 summarizes user-defined gestures from extension-oriented modes, including descriptive completion rates by technology and improvement.The supplied passages do not provide gesture-level values.Source: §B.7 · p. 14, Table 15 · p. 17

C Additional Offline Validation Examples

• Figure 4 shows representative remote-study setups, while Figure 5 presents original inputs, assisted outputs, and normalized perturbation visualizations for cross-dataset, cross-model, and model-transfer settings.The passages describe the figures’ contents but do not state outcomes for those comparisons.Source: Figure 4 · p. 16, Figure 5 · p. 18