PaperlayerView exact revision on arXiv

Source-linked AI summary

No Frames, Just Frequencies: Bandwidth-Efficient Deployment-Time Adaptation for Smart Glasses

Francesco Corti, Moritz Becker, Olga Saukh

arXiv:2610.29607v1cs.CVcs.LG

TL;DR

Smart glasses need deployment-time adaptation but face privacy and device constraints. NFJF sends compact frequency descriptors to a host, which synthesizes shifted data and adapts a model replica. Across the evaluated tasks, it recovers most supervised-oracle adaptation gain, with descriptor uplinks bounded at 832 B.

  • Problem

    Smart glasses must adapt to distribution shifts while respecting privacy and device constraints.

    Source: §1 · p. 1

  • Method

    NFJF sends on-device frequency descriptors to a host that synthesizes shifted labeled data and adapts a model replica on a compact parameter subset.

    Source: §1 · p. 2

  • Results

    82% on VWW, 93%–95% on ImageNet, and 94%–98% on MS-COCO detection are the oracle adaptation gaps NFJF closes across six backbones and three tasks.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Across the evaluated tasks, NFJF recovers most supervised adaptation gain while keeping per-frame uplink at or below 832 B.

    Source: §7 · p. 8

    The adaptation loop reproduces only corruptions from the modeled ten-operator DAG family; shifts outside it may be mapped to a different corruption and can yield accuracy below No Adapt.

    Source: §7 · p. 8

Abstract

from arXiv · show

Deep neural networks deployed on smart glasses must adapt to changes in the distribution of incoming data while adhering to strict privacy and device constraints. We present No Frames, Just Frequencies, a bandwidth-efficient source-assisted adaptation framework that characterizes distribution shifts using compact frequency descriptors computed on-device. Rather than transmitting camera frames, the glasses compute and send frequency descriptors to a host device that contains a replica of the model. The host device maps the received frequency descriptor to a distribution shift operator that synthesizes out-of-distribution data and adapts a model replica using a pre-defined adaptation protocol. We evaluate the framework on mixtures of ImageNet-C corruptions using convolution-based (ConvNeXtV2-Femto), self-attention-based (DeiT-Tiny), and hybrid (TinyViT-5M) vision models, on person classification (Visual Wake Words), and on object detection (MS-COCO). Our results reveal that adaptation driven solely by the transmitted descriptors recovers $93-95\%$ of the supervised oracle adaptation gain on all three architectures, $82\%$ on Visual Wake Words and $94$--$98\%$ on MS-COCO. Notably, adaptation quality degrades gracefully as the descriptor resolution decreases from $224 $ to $28$ pixels, consistently across architectures, enabling bandwidth reductions from $213\times$ to $784\times$ across the frequency descriptors.

1 Introduction

Smart glasses face distribution shifts under tight privacy, latency, power, memory, and bandwidth constraints. NFJF addresses these constraints by sending compact on-device frequency descriptors to a host that synthesizes shifted data and adapts a model replica.

1 Introduction

  • Smart glasses must process video continuously under tight latency and power budgets, while keeping sensitive visual content such as faces, screens, and locations on-device.Local inference also reduces dependence on network connectivity [56].Source: §1 · p. 1
  • Deployment shifts can degrade predictions, while adaptation adds memory demands because gradient methods store intermediate activations and optimizer states.Wearable-camera streams can change rapidly with illumination and motion blur [53].Source: §1 · p. 1
  • On-device TTA can fail with small batches or noisy gradients, and fixes that accumulate gradients or replay visual samples conflict with glasses’ memory and privacy constraints.Entropy minimization may amplify errors under some shifts [32].Source: §1 · p. 1
  • NFJF computes privacy-enhancing frequency descriptors on glasses, then offloads adaptation to a nearby host using synthetic shifted data and a compact parameter subset.The approach avoids streaming camera frames and characterizes shifts during non-stationary streams.Source: §1 · p. 2
  • NFJF evaluates three architecture families, three frequency descriptors, four input resolutions, and 63 corruption mixtures, then extends evaluation to person classification and object detection.The work also analyzes descriptor accuracy against model-based shift descriptors and measures end-to-end latency on smart-glasses-class hardware.Source: §1 · p. 2

2 Related Work

Prior adaptation approaches either use target samples, assume domain information, or manipulate frequencies offline. NFJF instead estimates shifts online from transmitted frequency descriptors and uses them to synthesize labeled adaptation data on a host.

2 Related Work

  • Unlike TENT [44], SAR [31], and SPA [30], which adapt on the unlabeled test stream, NFJF keeps test frames on glasses and adapts on labeled in-distribution images at a host.This is source-assisted rather than source-free, label-free TTA, in exchange for bandwidth-efficient transmission.Source: §2 · p. 2
  • Unlike domain-adaptation approaches that observe target samples or specify target conditions beforehand, NFJF estimates shifts online from descriptors and synthesizes labeled adaptation data on the host.The comparisons include unsupervised domain adaptation [8], source-free domain adaptation [21], continual TTA, predictive domain adaptation [28], and zero-shot domain adaptation [51].Source: §2 · p. 2
  • Prior frequency manipulation swaps low-frequency amplitudes for offline adaptation [52], while NFJF uses descriptors as a communication primitive and corruption-classifier input.Corruption families concentrate perturbation energy in distinct spatial-frequency bands [54].Source: §2 · p. 2
  • NFJF sends phase-discarded frequency descriptors rather than gradients, avoiding the on-device gradient computation required by federated learning [29].The framework is presented as a privacy-enhancing alternative to gradient sharing [56].Source: §2 · p. 2

3 Background

The system keeps camera frames on the glasses while a host infers corruption stages, synthesizes labeled adaptation batches, and updates a model replica. Only a compact descriptor travels to the host, and only the adapted parameter subset returns.

3 Background

  • At native 224 × 224 input, DWT sends 192 B instead of a 150,528 B RGB frame, a 784× reduction; descriptor reductions range from 213× for DCT to 784× for DWT.At 28 × 28 resolution, a descriptor can contain as few as 14 floats.Source: §3 · p. 2
  • For each incoming frame, the glasses compute a descriptor and send it over a bandwidth-limited link; phase is discarded and magnitude pooled, leaving a lossy summary.Figure 1 depicts this descriptor-only uplink and the host adaptation loop.Source: §3 · pp. 2–3, Figure 1 · p. 3
  • A host-side multi-stage classifier maps each descriptor to corruption-severity distributions for stages in a DAG, with severity zero representing an inactive stage.The classifier is pretrained on descriptors of corruption mixtures modeled as DAGs.Source: §3 · p. 3
  • The host composes predicted corruptions in camera ISP order over stored in-distribution data to construct labeled adaptation batches.These synthetic batches represent the predicted shift.Source: §3 · p. 3, Figure 1 · p. 3
  • The host adapts a model replica with a predefined supervised loss over a compact parameter subspace, freezes the remaining parameters, and returns the updated subset to the glasses.The returned subset is exchanged to complete the adaptation cycle.Source: §3 · p. 3, Figure 1 · p. 3

4 Methodology

NFJF represents visual distribution shifts with on-device frequency descriptors, classifies them as staged corruption graphs, and uses a host to synthesize shifted data for model adaptation.

4.1 Frequency Descriptors

  • RAPSD summarizes radial frequency energy, DCT aggregates block-wise frequency statistics, and DWT captures energy across multiple spatial scales.DWT yields a 48-dimensional descriptor; all descriptors use one channel, red, for the reported payload sizes.Source: §4.1 · pp. 3–4
  • NFJF improves on No Adapt across all three architectures above 28-pixel resolution, while surpassing Oracle - E at every resolution on DeiT-Tiny and TinyViT-5M.For ConvNeXtV2-Femto, it mostly surpasses Oracle - E at 112 and 224 resolutions.Source: §4.1 · p. 4, Figure 2 · p. 4

4.2 Multi-Stage Corruption Classifier

  • A multi-head MLP predicts corruption-severity pairs for four ordered stages—Weather, Blur, Noise, and Digital—from each frequency descriptor.The host applies the predicted operators sequentially to stored clean adaptation images to synthesize corrupted data.Source: §4.2 · pp. 4–5

4.3 Source-Assisted Adaptation Protocol

  • NFJF replaces entropy minimization with cross-entropy on labeled synthetic batches and adapts normalization parameters plus each model’s first block.At higher input resolutions, the adaptation gap against supervised cross-entropy Oracle adaptation narrows.Source: §4.3 · p. 5, Figure 4 · p. 5

5 Experimental Setup

The experiments test NFJF across corruption mixtures, model architectures, and classification and detection tasks, using separated data partitions and comparisons with source-only and oracle adaptation baselines. Reported adaptation gains are positive and approach the supervised Oracle as resolution increases across VWW and MS-COCO.

5.1 Datasets and DAGs

  • 945 corruption mixtures train each descriptor classifier, with signatures extracted from the first image of each ImageNet validation class; test descriptors use a separate image.Source: §5.1 · pp. 5–6

5.2 Adaptation Protocol

  • End-to-end evaluation separates ground-truth-corruption evaluation data from predicted-corruption synthetic adaptation data and repeats each test-DAG cycle three times.Reported values average the three repetitions; figure error bars show standard deviation of means across 63 test DAGs.Source: §5.2 · p. 6

5.3 Evaluation Baselines

  • No Adapt measures shifted-data accuracy without updates, while Oracle - E and Oracle - CE provide unsupervised and supervised adaptation references using restricted parameter updates.Oracle - CE uses ground-truth corruption graphs and labels, serving as the framework's empirical upper bound; Oracle - CE + Reg. adds bounding-box regression for detection.Source: §5.3 · p. 6

5.4 Experiments

  • The ImageNet experiments compare ConvNeXtV2-Femto, DeiT-Tiny, and TinyViT-5M at roughly five million parameters, and test descriptor resolution scaling through reduced-scale JPEG decoding.Spatial reduction diminishes shot-noise signatures while contrast and brightness remain visible at every evaluated resolution.Source: §5.4 · p. 6

5.5 Person Classification and Object Detection

  • VWW uses MobileNetV1 for binary person classification, while MS-COCO evaluates SSDLite-MobileNetV3-Large and YOLOX-Tiny using box mAP.For detection, pixel-level corruptions leave image geometry unchanged, so adaptation-set ground-truth boxes remain valid for synthesized batches.Source: §5.5 · p. 6
  • Every adaptation gain is positive and approaches the supervised Oracle as resolution increases across VWW and MS-COCO, while Oracle - E falls below No Adapt on VWW.Source: Table 2 · p. 7
  • ActStats improves on DCT by 0.67 pp on average and approaches Oracle - CE within 0.71 pp, but costs 2.3× its payload and about 100× its FLOPs.ActStats requires running the deployed model’s first layer and block; frequency descriptors require no model execution.Source: Figure 5 · p. 7

6 Results

NFJF recovers much of supervised-oracle adaptation gain across architectures and tasks, while descriptor resolution, classifier seeds, descriptor choice, privacy leakage, and host runtime shape the results.

6.1 Adaptation Across Architectures and Resolutions

  • 93.1%, 93.3%, and 94.7% of supervised-oracle adaptation gain is recovered on ConvNeXtV2-Femto, DeiT-Tiny, and TinyViT-5M, respectively, with DCT at 224 resolution.Adaptation gains grow monotonically with descriptor resolution; Oracle - E collapses on self-attention and hybrid models.Source: §6.1 · p. 7

6.2 Classifier Seed Sensitivity

  • Classifier-seed variation in post-adaptation accuracy is below variation across transmitted descriptors at all four resolutions, and the resolution trends persist across seeds.Classifier DAG agreement rises from 4% at resolution 28 to 59% at 224.Source: §6.2 · p. 7

6.3 Task Generality

  • 82% of Oracle - CE’s adaptation gain is closed on VWW, with accuracy rising from 65.3% to 72.80% using DWT at 96 resolution.The authors note that the binary task’s limited recoverable margin makes the fraction of the NFJF–oracle gap more informative than adaptation gain.Source: §6.3 · p. 7
  • 93.6% of the achievable gain is recovered on SSDLite320, and 97% on YOLOX-Tiny, with NFJF improving box mAP from 7.4 to 11.8 and 8.8 to 16.9, respectively.The authors relate detection’s higher gap closure than classification to its denser per-image supervision.Source: §6.3 · pp. 7–8

6.4 Descriptor Ablation

  • ActStats exceeds DCT adaptation accuracy by 0.67 pp on average, while FeatFinal trails DCT and DWT on every architecture.ActStats approaches Oracle - CE within 0.63 pp on average, but costs 2.3× the DCT payload and about 100× its FLOPs; low-level spectral content identifies corruption DAGs better than pre-logits embeddings.Source: §6.4 · p. 8

6.5 Privacy Leakage Analysis

  • 58.5±2.4% is the highest eavesdropper balanced accuracy for predicting VWW person presence from a single descriptor, indicating a weak person-presence signal.Increasing resolution from 24 to 48 raises average adaptation gain by 1.0 pp and leakage by 1.3 pp; from 48 to 96, the increases are 0.4 pp and 0.3 pp.Source: §6.5 · p. 8

6.6 Adaptation Latency on Smart Glasses

  • 85±9 s is the measured end-to-end adaptation time on the glasses-class board, with synthesis and host forward-backward computation accounting for 98% of the time.On-device descriptor computation, parameter update, and first adapted inference take 38 ms; transferring the 708 B descriptor and 22 kB update takes 0.14 s, while host-stage energy remains future work.Source: §6.6 · p. 8

7 Discussion

NFJF recovers most supervised adaptation gain across six backbones and three tasks without transmitting camera frames. Its scope is limited to modeled corruptions, and continual adaptation, dynamic descriptor resolution, and vision-language-model transfer remain open directions.

7 Discussion

  • NFJF closes 82% of the oracle adaptation gap on VWW, 93%–95% on ImageNet, and 94%–98% on MS-COCO, with per-frame uplink never exceeding 832 B.Entropy minimization destabilizes on self-attention and hybrid ImageNet architectures and on VWW.Source: §7 · p. 8
  • The adaptation loop handles each test DAG independently from pretrained parameters; continual adaptation to drifting DAGs and dynamic resolution selection are proposed extensions.Progressive JPEG could allow transmission of a low-resolution DCT descriptor while the frame continues decoding.Source: §7 · p. 8
  • The method reproduces only the modeled ten-operator DAG family, and transfer to vision-language models remains untested.Out-of-family shifts are mapped to the most similar modeled DAG and may yield accuracy below No Adapt; filtering unreliable predictions and broadening the corruption family remain future work.Source: §7 · p. 8
Loading 2610.29607v1…