Federated Evaluation and Tuning for On-Device Personalization: System Design & Applications

• On-device processing supports privacy-sensitive ML applications such as predictive keyboards and dictation, where personalization toward users’ vocabulary and language constructs is valuable.Source: §1 · p. 1
• The initial ASR use case required tuning global parameters using data available only on devices, motivating a generic federated evaluation and tuning system.Source: §1 · p. 1
• The generalized system abstracts away application-specific components and supports federated task distribution, on-device evaluation-data storage, and centralized task-result processing.Source: §1 · p. 1
• Federated learning support was added later, but the paper primarily studies on-device personalization applications rather than FL itself.Source: §1 · pp. 1–2

• Unlike SysBon, the proposed system supports arbitrary distributed computation through application-specific plug-ins rather than centering on neural-network training.Source: §2 · p. 2
• Federated tuning primarily learns small global parameter sets centrally from federated evaluation metrics, whereas FL learns global neural-network parameters mainly through on-device gradient updates.Source: §2 · p. 2
• The paper argues that federated tuning reduces privacy exposure relative to FL because it centrally shares evaluation metrics instead of high-dimensional model updates tied to user data.Source: §2 · p. 2
• Only a few FL publications address federated model personalization, with discussing user and context features, multi-task learning, local fine-tuning, and meta-learning connections.Source: §2 · p. 2

3.1 Motivation

• ASR personalization motivated FE&T because the target system uses a statistical grammar requiring application-specific modification code, favoring flexible task execution beyond neural models.Source: §3.1 · pp. 2–3
• The system assumes sufficiently anonymized and siloed data, where user data cannot reasonably be reidentified or aggregated across application-specific silos.Source: §3.1 · p. 3
• Because on-device evaluation data are non-IID and limited per device, meaningful accuracy insight requires aggregating results across many devices.Source: §3.1 · p. 3
• Federated tuning evaluates baseline and parameter-modified ML systems on devices, then uses returned metrics to select subsequent parameter configurations.Source: §3.1 · p. 3

3.2 The Challenge of Ground Truth

• On-device evaluation requires storing system inputs and ground truth, which can be collected implicitly for app suggestions from device context and the app the user launches.Source: §3.2 · p. 3
• ASR ground truth is difficult because WER requires reference transcripts, user corrections bias the data toward detected errors, and explicit labels depend on user participation and consistency.Source: §3.2 · p. 3
• Potential remedies include pseudo-labels from accurate teacher models and proxy metrics based on model confidences, including the ASR confidence-based metric used later.Source: §3.2 · p. 3

4.1 High Level System Design

• Application-specific plug-ins interpret and execute tasks while the core remains task agnostic, enabling arbitrary federated tasks; FL plug-ins additionally use private federated learning components.Source: §4.1 · p. 4

4.2 Device Participation & Data Handling

• Participation requires genuine attested devices, opt-in data collection, power, idleness, and unmetered connectivity, but these conditions can introduce population bias.Source: §4.2 · p. 4
• The on-device store supports encryption, configurable retention, subsampling, deletion after m days, and record caps, while application plug-ins perform data-specific computation.Source: §4.2 · pp. 4–5
• Raw stored data remains on-device, while encrypted task results, telemetry, and differentially processed FL updates are sent to servers after sensitive identifiers are removed.Source: §4.2 · p. 5

4.3 Core System

• The device scheduler samples tasks using server-configured likelihoods and matches device, OS, and data preconditions before plug-in execution.Source: §4.3 · p. 5
• Compared with SysBon’s server-orchestrated pace steering, the system favors asynchronous reporting and central databases or scripts for synchronization rather than fixed simultaneous cohorts.Source: §4.3 · p. 5

4.4 FL Specific Additions

• For FL, encrypted locally noised updates are temporarily stored and centrally aggregated, with local and central differential privacy applied before model release.Source: §4.4 · p. 6
• Compared with SysBon’s secure aggregation, the design accepts central storage of encrypted updates and less efficient coordination, while secure aggregation’s costs grow quadratically with device count.Source: §4.4 · p. 6

5 On-Device Personalization Applications

• Federated tuning supports large-scale personalization applications while keeping evaluation data on devices and processing task results server-side.The paper focuses on news personalization and ASR personalization, while noting that federated evaluation and tuning remain major system uses.Source: §5 · pp. 6–7

5.1 News Personalization

• News tuning uses on-device reading interactions as positive or negative labels, randomized grid search, and server-side clustering to iteratively narrow parameter ranges.The process repeats until convergence, yielding a locally optimal parameter set for the defined tuning dimensions.Source: §5.1 · p. 7
• 1.98% more daily article views followed news-tuning run 1, while run 2 increased daily article views by 1.87% and daily in-app time by 0.90%.Run 1 produced no statistically significant difference in daily time spent.Source: §5.1 · p. 8, Table 2 · p. 8

5.2 ASR Personalization

• Around 13% eWER stabilizes for federated ASR test sets of at least 100k utterances, yet remains 1% absolute different from central-test WER because background-noise audio was scrubbed centrally.The federated sets remain hidden, so actual WER is unavailable for them; the comparison assumes matching data distributions.Source: §5.2 · p. 9
• ASR personalization combines a personalized on-device recognizer with a generic server recognizer, with tuning aimed at minimizing estimated word error rate.The final transcription can be evaluated using actual WER on sampled production requests with human reference transcriptions.Source: §5.2 · pp. 8–9
• Personalized ASR reduces WER by 16.4% on user-vocabulary utterances and by 1.4% on generic-vocabulary utterances affected by system combination.These results concern 3,851 assistant-directed requests whose final transcripts changed through ASR system combination.Source: §5.2 · pp. 9–10, Table 4 · p. 10

• The system supports federated evaluation and tuning for on-device ML personalization, originating from ASR personalization that optimizes global parameters for device-specific models.The paper also describes an additional personalization use case demonstrating federated tuning’s applicability.Source: §6 · p. 10
• Because it supports arbitrary evaluation and tuning task implementations, the system can in principle perform arbitrary distributed computation.The described personalization applications and FL extensions provide partial demonstrations of this flexibility.Source: §6 · p. 10
• The system’s federated learning extensions add support for private federated learning by combining FL with differential privacy guarantees.Source: §6 · p. 10