PaperlayerView exact revision on arXiv

Source-linked AI summary

ElderBench: Benchmarking Autonomous Mobile Agents for Older Adults

Weide Zhan, Qumu Shaqu, Yuanqing Liu, Peng Zhang, Jiahao Liu, Kam Him Lam, Ning Gu, Zhan Hu, Tun Lu

arXiv:2609.04850v1cs.AI

TL;DR

Existing GUI benchmarks rely mainly on explicit instructions, while older adults naturally use indirect, ambiguous, and under-specified language. ElderBench collects and analyzes authentic elderly-oriented tasks, evaluates agents online and offline, and finds substantial performance limitations on these instructions.

  • Problem

    Existing GUI benchmarks provide limited support for elderly-oriented scenarios because they rarely capture naturally occurring older-adult instructions.

  • Method

    ElderBench collects 249 naturally elicited tasks across 20 applications, characterizes their language, and evaluates GUI agents and VLMs with online and offline protocols plus failure analyses.

  • Results

    No evaluated model exceeds 50% overall success rate on naturally elicited older-adult instructions.

  • Takeaways & Limitations

    The findings identify linguistic mismatch and execution bottlenecks as targets for more adaptive and age-inclusive GUI-agent design.

Abstract

from arXiv · show

While autonomous mobile agents hold great potential for assisting older adults with smartphone usage, existing GUI benchmarks mainly rely on explicit, goal-oriented instructions and rarely capture the naturally occurring language patterns of older users, such as indirect speech, referential ambiguity, and under-specified requests. This mismatch between benchmark instructions and real-world elderly interactions may hinder reliable agent deployment. To address this gap, we present ElderBench, the first benchmark for evaluating mobile GUI agents in authentic elderly-oriented scenarios. ElderBench is constructed from 249 naturally elicited smartphone tasks collected from older adults across 20 applications. We first characterize the linguistic divergence between elderly instructions and existing GUI benchmark instructions from syntactic, semantic, and pragmatic perspectives. We then evaluate mainstream GUI agents and Vision-Language Models under both online and offline settings, revealing substantial performance degradation when handling elderly-oriented instructions. Through controlled instruction normalization, failure analysis, and fine-grained linguistic feature analysis, we further identify how elderly-specific language patterns contribute to agent failures. Our findings provide actionable design insights toward more adaptive, interpretable, and age-inclusive GUI agents for older adults.

1 Introduction

ElderBench addresses the mismatch between explicit benchmark commands and naturally occurring older-adult smartphone requests. It introduces an elderly-oriented benchmark and analysis framework to characterize this language and identify agent failures.

  • Existing GUI benchmarks mainly use explicit, goal-oriented instructions, whereas older adults often express smartphone needs indirectly and incompletely.
  • ElderBench contains 249 naturally elicited smartphone tasks collected from older adults across 20 applications.
  • The benchmark characterizes elderly GUI instructions across syntactic, semantic, and pragmatic dimensions.
  • ElderBench evaluates mobile GUI agents and VLMs through online and offline protocols.
  • Controlled normalization, failure analysis, and linguistic factor analysis are used to identify failure sources and inform more adaptive, age-inclusive GUI agents.

2 Related Work

Related benchmarks evaluate GUI agents through online interaction or offline trajectories, but rarely combine target-user elicitation with naturally occurring, population-specific language analysis. ElderBench fills this gap with older-adult instructions collected across applications.

  • Existing mobile GUI benchmarks primarily use online interaction-based or offline trajectory-based evaluation paradigms.
  • Mobile-Bench-v2 extends benchmark realism by testing ambiguous instructions and whether agents recover missing information or request clarification.
  • Existing benchmarks rarely combine target-user elicitation, population-specific design, naturally occurring language phenomena, and systematic linguistic analysis.
  • ElderBench includes 249 tasks across 20 applications, with instructions directly elicited from older adults rather than constructed from predefined templates.
  • Older adults often use contextual descriptions, fragmented expressions, and ambiguous references when seeking technical assistance.
  • ElderBench focuses on linguistic challenges arising from authentic elderly-oriented interactions, complementing prior GUI benchmarks.

3 Data Collection and Linguistic Analysis

The study collects authentic older-adult smartphone instructions and analyzes their linguistic structure and meaning. ElderBench reveals short, heterogeneous, context-dependent language that differs from conventional action-oriented benchmark commands.

  • Data Collection: Researchers recruited 28 older adults aged 59–84 to capture diverse linguistic expressions and smartphone usage patterns.
  • Data Collection: Semi-structured interviews elicited routine smartphone tasks, desired agent assistance, and situations where participants experienced difficulties.
  • Data Collection: 249 valid elderly GUI instructions remained after two invalid instructions were removed from 251 collected instructions.
  • Linguistic Analysis: The analysis compares ElderBench with MobileWorld across syntax, semantics, and pragmatics.
  • Syntactic Dimension: 78.31% of ElderBench instructions contain fewer than 20 characters, while elliptical and narrative structures account for 47.80%.
  • Semantic Dimension: Elderly instructions include 19.28% indirect speech and 12.85% referential ambiguity, often describing desired states or relying on current interface context.
  • Pragmatic Dimension: Elderly requests concentrate on life services, social communication, and system settings linked to daily activities and accessibility needs.

4 Benchmark

ElderBench combines online and offline evaluation to assess mobile GUI agents on elderly-oriented smartphone tasks while balancing real-world fidelity with reproducibility. Its benchmark, execution protocols, verification procedures, and multidimensional metrics define a controlled setting for measuring task completion, latency, and cost.

  • Benchmark design: ElderBench contains 249 tasks across 20 Android applications and uses dual online/offline evaluation to balance interaction fidelity and controllability.The online subset covers live interaction, while the offline subset uses reproducible human-demonstrated task graphs.
  • Benchmark design: The benchmark evaluates single-turn instructions from an initial device state, reflecting both collected task patterns and the operating assumptions of current mobile GUI agents.The design is motivated by independent task requests observed in interviews and the predominantly single-turn operation of current agents.
  • Evaluation protocols: Online evaluation forms a closed-loop perception-action cycle in which agents process screenshots and context, generate structured actions, and execute them through Android Debug Bridge.Agents receive updated GUI states after each action in live applications.
  • Evaluation protocols: Task success requires both agent termination and successful task verification, with automated judging based on final screenshots, intermediate frames, and the active Android Activity.Manual validation of AutoGLM results reported 93.08% agreement between human judgment and the VLM judge.
  • Evaluation protocols: Offline evaluation records multiple human trajectories as task graphs, merges equivalent UI states, and counts success when an agent reaches the graph’s terminal node.The protocol supports multiple valid paths, 50-pixel click tolerance, and fuzzy matching for text inputs.
  • Evaluation metrics: The evaluation suite measures Task Success Rate, Average Inference Latency, and Monetary Cost Metric to capture completion, temporal burden, and deployment cost.Latency aggregates model inference and structured action-parsing overhead across operational steps, while monetary cost tracks input and output token consumption.

5 Experiments

Experiments evaluate mainstream GUI agents and VLMs on ElderBench through online and offline protocols, then analyze normalization effects, failure causes, and linguistic factors. Results show limited overall success, substantial sensitivity to instruction formulation, and bottlenecks concentrated in intent understanding and multi-step execution.

  • Overall Performance: No model exceeds 50% overall SR on naturally elicited older-adult instructions.Gemini-3-Flash leads VLMs at 49.80% SR, while GUI-Owl-1.5 leads GUI agents at 45.38%.
  • Online and Offline Evaluation: Online evaluation generally yields higher SR than offline evaluation, but the subsets differ in tasks and applications.The offline protocol also requires every predicted action to match a valid task-graph edge, while online agents can observe updated states and recover through alternative paths.
  • Efficiency Trade-offs: No model simultaneously optimizes success, latency, and token cost.Qwen3-VL-Flash has the lowest overall AIL at 3.57 seconds but 22.89% SR, while other models provide stronger success or efficiency trade-offs.
  • Controlled Instruction Normalization: Instruction normalization increases successful tasks from 37 to 61 for AutoGLM and from 23 to 46 for Qwen3-VL-Flash.Both gains are significant under exact McNemar’s test (p < 0.001), with paired-bootstrap 95% CIs of [14, 34] and [12, 34] percentage points.
  • Failure Analysis: Intent understanding and capability-boundary errors dominate AutoGLM failures at 57.1%, followed by long-horizon planning and state-tracking errors at 33.3%.Together, these categories account for 90.4% of analyzed failures, exceeding device-environment awareness and visual/action execution errors.
  • Linguistic Factors: Elliptical and indirect-speech instructions are associated with lower success because agents must infer omitted action sequences from user states.Failures also arise through interactions among linguistic properties, including scenario-dependent verbosity effects and conflicts between canonical and indirect speech.

6 Design Insights

ElderBench results motivate agents that interpret older adults’ implicit states, clarify underspecified requests, and adapt interpretation to language and task context.

  • State-Aware Intent Understanding: Agents should translate older adults’ implicit user situations into executable operations through state-aware intent understanding.The recommendation addresses instructions that describe desired states rather than explicit actions.
  • Proactive Clarification: Agents should use proactive clarification and adapt instruction interpretation to both linguistic patterns and task contexts.The paper links this need to the varying effects of linguistic complexity across scenarios.

7 Conclusion

ElderBench is a benchmark built from naturally elicited older-adult smartphone instructions and designed to expose linguistic and execution challenges for mobile GUI agents. Its analyses identify persistent performance limitations and inform more adaptive, age-inclusive agent design.

  • Conclusion: The benchmark combines linguistic characterization, online and offline evaluation, controlled normalization, failure analysis, and feature-level attribution.These components examine both language differences and execution bottlenecks.
  • Data Collection: The dataset was collected from 28 older adults aged 59–84 across multiple Chinese regions and varied smartphone-experience levels.The authors note that the benchmark focuses on Chinese-speaking older adults because of practical collection constraints.
  • Data Collection: Data collection used semi-structured interviews without predefined task lists, instruction templates, or linguistic examples.This procedure was intended to preserve spontaneous older-adult expressions and reduce benchmark-style language bias.
  • Dataset Construction: Two invalid samples were removed from 251 candidate instructions, leaving 249 executable GUI instructions.The filtering excluded instructions with missing goals or unrecoverable ambiguity.
  • Linguistic Analysis: Linguistic analysis covers syntax, semantics, and pragmatics, using LLM-assisted extraction followed by human verification.Average agreement between LLM annotations and manual verification reached 93.10%.

D.1 Syntactic Dimension

ElderBench instructions differ structurally from conventional GUI benchmark commands, with shorter and more heterogeneous forms, including more elliptical and narrative expressions. Linguistic features show distinct relationships with task success across scenarios.

  • Syntactic Categories: ElderBench classifies instructions as Canonical, Narrative, Elliptical, or Compound according to their syntactic structure.Canonical instructions state explicit operational intentions, whereas Narrative and Elliptical forms respectively describe situations or omit execution information.
  • Syntactic Comparison: ElderBench instructions show substantial structural differences from baseline GUI instructions.Existing benchmarks mainly use explicit action-oriented commands, while elderly-oriented instructions are shorter and more diverse.
  • Syntactic Comparison: Elliptical and Narrative expressions occur at substantially higher proportions in ElderBench than in baseline instructions.Figure 2 presents the distribution of the four syntactic structures.
  • Semantic Analysis: The analysis also examines semantic categories including under-specification, disfluency/correction, referential ambiguity, and indirect speech.Indirect speech frequently lacks explicit operational verbs, and semantic distributions are reported separately from syntactic structure.
  • Application Scenarios: Collected instructions mainly concern daily-life scenarios such as life services, social communication, and system settings.These scenarios include shopping, transportation, messaging, payment, font adjustment, volume control, and accessibility configuration.
  • Feature Attribution: Longer instructions correlate negatively with task success at −0.16, while Elliptical and Indirect Speech show correlations of −0.15 and −0.12.Information inquiry correlates positively with success at +0.26, whereas System Settings and Tool Assistance correlate at −0.11 and −0.07.

E.2 Cross Feature Influence

ElderBench models nonlinear interactions among linguistic features, showing that instruction length has scenario-dependent effects and that feature combinations can either help or hinder execution.

  • Cross Feature Influence: The Factorization Machine captures second-order interactions among categorical and continuous linguistic features.Categorical features use one-hot encoding, while instruction length and verb number are standardized.
  • Cross Feature Influence: +0.70: Scenario-specific tolerance for verbosity indicates that longer instructions can be beneficial in some scenarios.
  • Cross Feature Influence: Instruction length alone negatively affects performance, but longer descriptions provide useful constraints in information inquiry tasks.
  • Cross Feature Influence: -0.42: Long descriptions combined with System Settings tasks introduce additional planning difficulty.
  • Cross Feature Influence: -0.34: Canonical syntax combined with Indirect Speech semantics creates conflicting interpretation signals.

F.1 Overall Benchmark Statistics

ElderBench contains 249 smartphone tasks from older adults across 20 Android applications and evaluates them through complementary online and offline protocols. The benchmark uses live interaction for stable environments and human-demonstrated trajectories when privacy, reproducibility, or external-state constraints apply.

  • Overall Benchmark Statistics: ElderBench contains 249 executable smartphone GUI tasks collected from older adults across 20 Android applications.
  • Overall Benchmark Statistics: The online subset contains 130 tasks across 15 applications, while the offline subset contains 119 tasks across five applications.
  • Overall Benchmark Statistics: Task-level protocol assignment depends on execution reproducibility, privacy and account constraints, and external application-state stability.Sensitive, payment, or highly dynamic applications are evaluated offline; relatively stable interaction flows are evaluated online.
  • Overall Benchmark Statistics: The benchmark evaluates applications in reset Android environments with cleared search history, enabled required permissions, and no retained personalized content.
  • Overall Benchmark Statistics: Offline evaluation records human-executed trajectories containing screenshot observations, UI hierarchies, and executed actions.

H.2 Multi-path Task Graph Generation

ElderBench converts multiple valid human demonstrations into task graphs so agents are evaluated against alternative execution paths rather than a single prescribed trajectory.

  • Multi-path Task Graph Generation: Multiple valid demonstrations are collected for each offline task to avoid dependence on a single golden trajectory.
  • Multi-path Task Graph Generation: Demonstration trajectories are converted into task graphs by extracting and matching equivalent UI states, then merging them into graph nodes.
  • Multi-path Task Graph Generation: Human actions between merged states become directed edges, and successful terminal states are marked as DONE nodes.
  • Multi-path Task Graph Generation: The final graph represents multiple valid execution paths.
  • Multi-path Task Graph Generation: An agent action is valid only when it matches an outgoing edge from the current graph node, and success requires reaching a DONE node.

I.1 Single-turn Evaluation

ElderBench evaluates single-turn mobile GUI execution using original and normalized instructions, then analyzes failed trajectories to separate linguistic understanding, planning, environment, and execution problems.

  • Single-turn Evaluation: ElderBench focuses on single-turn GUI task execution because older adults often make standalone requests and current agents primarily process single-turn instructions.
  • Single-turn Evaluation: A normalization study rewrites 100 instructions into concise, explicit, action-oriented forms while preserving task goals and semantic equivalence.
  • Single-turn Evaluation: Qwen3-VL-Flash and AutoGLM are evaluated on both original and normalized instructions for the same tasks and environments.The paired design estimates linguistic-formulation effects rather than task-difficulty differences.
  • Single-turn Evaluation: Among 63 failed AutoGLM tasks, intent-understanding and capability-boundary errors include misinterpreting elderly expressions, unresolved references, and unavailable operations.
  • Single-turn Evaluation: Planning and state-tracking failures include repeating completed operations, missing intermediate steps, and continuing after reaching the goal.
  • Single-turn Evaluation: The evaluation uses ADB-mediated Android interaction with Tap, Long Press, Swipe, Type, Home, and Back actions, normalized to a [0,1000] coordinate system.
Loading 2609.04850v1…