Defending Wearable VLMs Against Private Attribute Inference

Zhimin Li; Pan Wang; Jingxian Chen; Yuantao Tang; Anthony Chen; Qian Lou; Jingtong Hu

Source-linked AI summary

Defending Wearable VLMs Against Private Attribute Inference

Zhimin Li, Pan Wang, Jingxian Chen, Yuantao Tang, Anthony Chen, Qian Lou, Jingtong Hu

arXiv:2608.28691v1Aug 26, 2026cs.CVcs.AI

TL;DR

Split VLMs can transmit visual tokens that preserve private attributes even when the requested response is benign, creating a privacy risk at the token interface. The paper introduces TGAP, a residual pre-LLM adapter trained to suppress private evidence while preserving utility, and reports a large source-model privacy reduction with retained relaxed utility. The study also identifies architecture-specific utility losses and other scope limitations.

Problem
Split VLMs expose an understudied risk: downstream components or attackers may recover private attributes from transmitted visual tokens, so privacy and utility must be evaluated together.
Method
TGAP is a small residual adapter before transmission that keeps the VLM backbone frozen and combines utility, identity, semantic privacy, and representation-suppression objectives.
Results
TGAP lowers MiniCPM-V P.Acc from 0.567 to 0.074 while changing U.Rel from 0.766 to 0.744; tradeoff improves across several evaluated VLMs but not uniformly.
Takeaways & Limitations
Securing the compact token interface can reduce source-model privacy leakage and average recovery by freshly trained attackers across four frozen LLM families.
Takeaways & Limitations
The benchmark is a single-image proxy rather than smart-glasses capture, utility is conditional, raw-image transmission bypasses TGAP, and no formal privacy guarantee is provided.

Abstract

from arXiv · show

Wearable VLM pipelines promise continuous multimodal assistance from egocentric visual capture: a user asks a task-driven question about the surrounding scene, and the system uses compact visual tokens to support language reasoning. The challenge motivating this work is that the same egocentric evidence needed for useful assistance can also reveal private attributes about the wearer or nearby bystanders. We investigate this as a joint privacy-utility problem for split VLM inference, where visual encoding occurs within a trusted device boundary but intermediate visual tokens may be transmitted to downstream reasoning components. This exposes an understudied leakage surface: even when final textual responses are benign, external attackers or untrusted downstream components can recover private attributes from transmitted visual tokens. To evaluate this tension, we construct a paired privacy-utility benchmark with 3,221 image-question records, each paired with a utility question and privacy labels covering location, income, sex, and interests. We further propose Token-Guided Attribute Privacy (TGAP), a pre-LLM token disentangler that learns a residual transformation of visual tokens before they leave the trusted boundary. TGAP combines utility preservation, identity regularization, semantic privacy suppression, and image-driven representation suppression, avoiding the utility loss caused by coarse hard or attention masking. On the benchmark used for source-model evaluation, TGAP reduces privacy accuracy from 56.7\% to 7.4\%, a 49.3\% absolute drop, while maintaining relaxed utility at 74.4\%. These results suggest that securing the compact token interface is a practical path toward privacy-preserving wearable multimodal AI.

1 Introduction

Wearable split VLMs can expose private attributes through transmitted visual tokens even when answering benign questions. The paper frames privacy and utility as a joint token-interface problem and proposes TGAP, a residual adapter that suppresses private evidence while preserving task-relevant information.

Transmitted visual tokens can retain location, demographic, financial, or interest-related evidence despite benign requested tasks.
Existing token-level defenses can damage utility because tokens attributed to privacy may also contain evidence required by the task.
The benchmark pairs each utility question with privacy annotations for location, income, sex/gender, and interests across 3,221 image-question records.
TGAP is a small residual adapter placed after the visual resampler, retaining a frozen VLM backbone while optimizing utility, identity, source-model privacy, and representation-suppression signals.
TGAP reduces MiniCPM-V source-model P.Acc from 0.567 to 0.074 while retaining U.Rel of 0.744 versus 0.766 without filtering.

2 Related Work

Prior work addresses wearable inference, multimodal privacy, attackers, and defenses at separate interfaces. This paper targets their intersection through paired privacy-utility evaluation and low-cost mitigation for exposed visual tokens.

Related work covers wearable inference, private-attribute prediction, representation attacks, model-level adaptation, or visual privacy filters largely in isolation.
The related-work gap is paired privacy-utility evaluation with low-cost token-interface mitigation for exposed visual tokens in wearable VLM pipelines.
Split inference distributes VLM computation across mobile, edge, and cloud resources to improve practical latency and compute use.
The paper pairs utility and privacy targets on the same image and evaluates both source-model disclosure and recovery from exposed token representations.
A downstream adversary may combine leaked visual tokens with language-model priors through a malicious app, remote service, or eavesdropping path.
Model-level defenses change outputs but do not constrain tokens available to newly trained downstream attackers, while image-space defenses perturb inputs before encoding.

3 Methodology

The methodology secures the split-inference visual-token interface by evaluating semantic and representation leakage on paired utility-privacy data, then applying TGAP before transmission. TGAP learns a continuous residual token transformation inside the trusted boundary while preserving the intended utility representation.

Threat model and evaluation: The pipeline evaluates source-model disclosure and attacker recovery from transmitted tokens alongside utility on paired single-image records.Source-model prompting measures semantic leakage and utility, while fresh attacker bridges and heads measure recoverability from transmitted representations.
Threat model and evaluation: TGAP transforms post-resampler tokens inside the trusted boundary before transmission or language reasoning, while the source language model remains frozen.The defense replaces z with filtered tokens ˜z = Tθ(z); in the MiniCPM-V instantiation, the token shape is T = 64 and d = 4096.
Scope and boundaries: TGAP protects the token-transmission interface but does not protect raw-image transmission or compromised trusted components, and utility evaluation assumes queries do not request protected attributes.The threat model uses a fresh bridge and output head with a frozen surrogate LLM; it is weaker than a fully white-box adversary and provides no formal privacy guarantee.
Threat model and evaluation: The methodology measures semantic privacy leakage, representation privacy leakage, and utility with task-specific scores, where lower privacy and higher utility indicate stronger defense.Semantic leakage evaluates private responses from the source model, representation leakage evaluates attacker recovery from filtered tokens, and utility evaluates answers to the intended query.
TGAP architecture: The token map adds a gated residual update using layer normalization, a bottleneck MLP, a token-wise sigmoid gate, and a learnable global strength.This controlled feature update changes the token stream without replacing or deleting tokens, unlike hard token dropping or attention masking.
Paired benchmark: The benchmark contains 3,221 image-question records with utility questions and evidence-supported labels for location, income, sex/gender, and interests.Only supported non-uncertain labels enter optimization and evaluation; the benchmark uses Reddit posts and converted VQA images rather than smart-glasses capture.

4 Experiments

The experiments evaluate semantic leakage, TGAP’s source-model privacy–utility performance, objective ablations, and representation-level attackers across split VLM settings. Results show substantial privacy reduction overall, but utility preservation and robustness vary by architecture and attack protocol.

Experimental questions: The benchmark measures privacy recovery and utility jointly, using semantic leakage from model outputs and representation leakage from exposed tokens.Privacy metrics reward lower recovery, while utility metrics reward higher task correctness.
Semantic leakage: Closed-source P.Acc reaches 70.42%, while MiniCPM-V and Qwen2.5-VL reach 56.70% and 47.80% on evidence-supported proxy labels.The remaining mitigation experiments use open-source VLMs with accessible post-resampler token interfaces.
Main privacy–utility results: TGAP reduces MiniCPM-V P.Acc from 0.567 ± 0.003 to 0.074 ± 0.064 while U.Rel changes from 0.766 ± 0.011 to 0.744 ± 0.021.The tradeoff score also improves on Qwen2.5-VL, Phi-3.5-Vision, and GLM-4.1V, but InternVL2.5 shows a larger utility decrease.
Main privacy–utility results: InternVL2.5 is an architecture-specific failure case: TGAP lowers P.Acc to 0.090 ± 0.095 while U.Rel falls from 0.660 ± 0.053 to 0.551 ± 0.024.The authors do not treat this result as evidence of utility preservation across every VLM.
Objective ablation: Removing semantic suppression causes the largest privacy deterioration, increasing P.Acc from 0.074 to 0.553 and P.Cont from 0.037 to 0.179.Removing image-driven suppression or identity regularization also weakens the reported privacy–utility balance.
Representation-level attackers: Adaptive-readout balanced accuracy decreases for every tested attacker, including Qwen 0.482 → 0.386, Phi 0.584 → 0.439, InternVL 0.645 → 0.490, and GLM 0.562 → 0.406.Eleven of twelve attacker–attribute pairs decline, while frozen-decoder transfer BA changes from 0.678 to 0.426 for linear decoders and from 0.650 to 0.408 for MLPs.
General-task utility: TGAP changes TextVQA and DocVQA little but reduces MSVD-QA from 0.566 to 0.521 and ActivityNet-QA from 0.654 to 0.587.The reported controls use different backbones or interfaces and are therefore not combined into a single ranking.

5 Conclusion

The paper studies private-attribute leakage at split VLM token interfaces and proposes TGAP, a residual adapter trained with semantic, representation, utility, and identity objectives. Empirical evaluations report reduced source-model privacy metrics and lower recovery by retrained bridge/head attackers, while the approach remains interface-specific.

TGAP is a small residual adapter that modifies transmitted visual tokens without updating the VLM backbone.The method is trained with semantic, representation, utility, and identity objectives.
Freshly trained bridge/head attackers recover less information on average across four frozen LLM families after TGAP filtering.
Consent-based first-person sequences, temporal utility objectives, and stronger white-box attackers remain future work.

6 Limitations

The evaluation is bounded by proxy data, conditional utility, attacker and threat-model assumptions, and deployment constraints that were not measured. The study also provides no formal privacy guarantee and faces reproducibility limits from data-release restrictions.

The benchmark is a single-image proxy rather than smart-glasses capture, and its four labels are evidence-supported targets rather than verified identities.
Utility is conditional because paired questions avoid protected attributes, while video QA and InternVL2.5 show larger losses than image/OCR tasks.
TGAP does not protect raw-image transmission, and the strongest attacker retrains a bridge/head while freezing its LLM.
Several attribute-level intervals include zero, and the study makes no formal privacy guarantee.
Reddit redistribution and source deletion limit reproducibility, while glasses-side latency, power, and thermal costs remain outside the emulated study.

A Dataset Details

The benchmark pairs utility questions with evidence-gated privacy annotations across four attributes, using curated single-image records and human review. Its protocol specifies annotation scope, auditing, prompt formats, parsing, evaluation, and restricted release of raw Reddit-derived data.

Dataset construction: The benchmark contains 3,221 image-question records, with one utility question and one or more multi-label privacy annotations per record.The records refer to 3,180 unique image files because some source images support multiple questions.
Dataset construction: The collection is a wearable-assistance-motivated single-image proxy spanning indoor scenes, streets and signs, documents, products, food, and personal objects.Its images were not captured by smart glasses and it does not represent continuous first-person video or head motion.
Curation rules: Records were retained only when images and utility questions were usable and attribute evidence clearly linked to the subject.Ambiguous, corrupted, unsupported, or unclear-subject records were removed or marked uncertain.
Paired tasks: The benchmark pairs utility questions with privacy questions covering location, income, sex/gender, and interests.Examples include city names, explicitly reported income, supported sex/gender presentation, and post-associated interests.
Annotation protocol: Evidence-gated curation produced 5,223 privacy annotations, including 4,954 non-uncertain targets used by TGAP and 269 explicit uncertainty annotations excluded from optimization and evaluation.
Annotation quality: Humans reviewed all 5,223 annotations, and a stratified subset of 2,203 annotations was re-audited for incorrect or insufficiently supported labels.Agreement was reported by attribute because open-ended, potentially multi-label interests had lower unanimous agreement than other attributes.
Annotation quality: Income targets required explicit numeric or directly mapped categorical self-reports and were interpreted as evidence-supported coarse ranges rather than verified financial records.All 303 income annotations were re-audited, with one categorical case corrected.
Evaluation protocol: TGAP experiments use a deterministic seed-specific 80/20 split of 3,084 eligible records, yielding 2,467 training and 617 held-out records.

C External LLM Attacker Benchmark

The external-attacker benchmark tests whether private attributes remain recoverable from raw or TGAP-filtered tokens when a frozen LLM receives them through a trainable bridge and output head. Across four LLM families, adaptive-readout macro balanced accuracy decreases after filtering, but several attribute-level effects remain uncertain.

Protocol: The benchmark compares raw and TGAP-filtered post-resampler tokens using independently trained bridges and output heads for each frozen external LLM family.This prevents the filtered condition from being evaluated only by an attacker trained on raw tokens.
Protocol: The attacker freezes the LLM backbone and trains only a bridge from token space into the LLM hidden space plus an attribute output head.
Evaluation details: The evaluation uses balanced accuracy over three attacker-training seeds with matched initialization and minibatch order between raw and filtered conditions.
Results: Macro BA decreases from 0.482 to 0.386 for Qwen, 0.584 to 0.439 for Phi, 0.645 to 0.490 for InternVL, and 0.562 to 0.406 for GLM.
Results: Eleven of twelve attribute-level mean changes are positive, but Phi sex is the only negative mean and its interval contains zero.
Results: Several other attribute-level intervals also contain zero, so those effects are interpreted as uncertain rather than uniform attribute-level suppression.
TGAP mechanism: TGAP applies a gated residual update to post-resampler tokens, changing the interface through controlled feature updates rather than replacing or deleting tokens.The backbone VLM remains frozen, so optimization changes only the pre-LLM token interface.
TGAP mechanism: The TGAP objective combines utility preservation, identity regularization, source-model semantic privacy suppression, and image-driven representation privacy suppression.

D.5 Attention-mask implementation audit

The audit found that the original Qwen and GLM attention-mask rows were ineffective because negative attention bias did not reach the generation backend. After instrumentation, masked outputs became non-identical, while GLM’s zero exact-match scores required interpretation alongside relaxed utility.

Original Qwen and GLM attention-mask rows matched their baselines because the negative attention bias did not reach the generation backend.Attribution and token-position mapping were computed, but the backend used during generation did not apply the bias.
Instrumented-backend reruns verified non-identity masked outputs and runtime application of the attention bias.
Table 13 reports mean ± standard deviation over three seeds on the complete held-out split, while the MiniCPM-V training configuration uses a full objective and named-term ablations.The accompanying hyperparameter and ablation tables identify the shared configuration and one-term-at-a-time changes.
GLM exact match was zero in both rows, while relaxed semantic utility remained nonzero because the checkpoint often produced explanatory or non-canonical answers.The evaluation therefore interprets U.Rel together with U.EM rather than treating zero exact match as zero semantic utility.

D.7 Computing resources and implementation footprint

The deployment-relevant component is a lightweight residual adapter applied before the frozen LLM, while the auxiliary probe shapes training and can be discarded. Reproducibility materials distinguish matched experiments from controls at different interfaces and report modest local resource requirements.

Local runs used a single NVIDIA GeForce RTX 4080 SUPER GPU with 16GB memory, requiring 3.6–5.0 GPU-hours per full-objective seed.The reported three-seed runs include evaluation.
TGAP inference applies a residual token transformation before the frozen LLM consumes visual tokens, and the training probe can then be discarded.
The appendix reports matched evidence only when experiments reuse the corresponding Table 4 checkpoint and seed-specific split without retuning the residual gate.
Controls using different model, token, or pixel interfaces are scoped separately rather than ranked directly against TGAP.The fixed MiniCPM-V checkpoint also delimits transfer to image/OCR and video tasks rather than providing a repeated-run efficacy estimate.

E.2 Full closed-source benchmark

The full closed-source benchmark establishes that high utility and private-attribute recoverability coexist across model families. Token-level diagnostics and frozen-decoder transfer evaluations then assess whether TGAP-filtered tokens remain compatible with privacy decoders learned from raw representations.

The closed-source benchmark evaluates utility and privacy on the same paired dataset to show their coexistence across model families.The main source-model mitigation experiments use an open-source VLM whose intermediate visual-token interface can be modified directly.
The overlap diagnostic computes privacy and utility attribution over the same 64 post-resampler tokens and visualizes their top-token intersection.This motivates learned disentangling because removing shared tokens can damage useful answers.
The overlap score compares the top-k privacy-attributed tokens with the top-k utility-attributed tokens through their intersection and union.
Frozen-decoder transfer trains a private-attribute decoder on raw tokens and evaluates it on raw versus TGAP-filtered tokens.Lower filtered balanced accuracy, macro-F1, and Cohen’s κ indicate weaker direct transfer from raw tokens.
Figure 12 expands the aggregate transfer result by attribute, with lower filtered balanced accuracy indicating weaker direct transfer of image-driven private-attribute decoders.

E.6 Utility beyond the paired benchmark

External-task evaluation supports utility preservation for tested image and OCR tasks but exposes larger decreases on video tasks, whose temporal structure is absent from the single-image checkpoint. Additional controls show that privacy reductions can trade off against utility or arise at different deployment interfaces.

External-task utility: TextVQA and DocVQA change little under the fixed TGAP checkpoint, whereas MSVD-QA and ActivityNet-QA show larger decreases.The checkpoint was trained on single-image inputs and does not model temporal structure.
External-task utility: Utility preservation is therefore supported for the tested image/OCR tasks, not generalized to all VQA or video tasks.
Model-level control: Within CLEAR, LLMU reduces P.Acc by 0.3970 with a U.Rel decrease of 0.0615, while DPO reduces P.Acc by 0.1503 with a U.Rel decrease of 0.0210.Sex P.Acc remains 0.7753 across the three CLEAR checkpoints, and the comparison does not evaluate the representation sent across the split boundary.
Token-space control: At K = 4, Gaussian replacement decreases P.Cont by 0.0154 and U.Rel by 0.0227; replacing 16 tokens lowers U.Rel to 0.6677 without monotonic privacy improvement after K = 8.This fixed-seed control therefore incurs increasing utility cost without consistently improving privacy.
Image-space control: Image-space perturbation reduces privacy leakage on all five VLMs by 0.0014–0.0123, while utility changes range from −0.0261 to +0.0080.Its image-space evaluator makes these values non-comparable directly to the token-interface metrics above.
Scope and implications: The work presents safer smart-glasses assistance as a positive implication while acknowledging that benchmark and evaluation components rely on LLM/VLM systems.