Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security

• Current intelligent personal assistants are restricted by narrow domains, weak intent understanding, reasoning, and execution, and rule-based workflows that require costly manual specifications or demonstrations.Supervised and reinforcement-learning approaches [6] still depend substantially on manual demonstrations and/or predefined reward functions.Source: §1 · p. 4
• LLMs create opportunities to improve assistant scalability through instruction following, commonsense reasoning, zero-shot generalization, autonomous planning, and tool use.Researchers have applied LLMs to agents that use search engines, code interpreters, and third-party APIs to solve complex problems.Source: §1 · p. 4
• Personal LLM Agents integrate LLMs deeply with personal data and devices to automate repetitive, tedious, low-value daily labor and improve users’ work and life.They can build on existing mobile apps and websites while providing ubiquitous intelligent automation, and are envisioned as a major software paradigm for personal computing devices.Source: §1 · p. 4
• The scope emphasizes the personal aspects of Personal LLM Agents—personal data, resources, devices, and personalized services—rather than simply integrating general LLM capabilities into IPAs.The paper presents a route map, design choices, challenges, and possible solutions, while excluding straightforward integration of general language capabilities.Source: §1 · p. 5
• The paper’s expert-informed architecture centers on managing and utilizing personal data and resources, which differentiates Personal LLM Agents, and introduces five intelligence levels inspired by autonomous driving.The survey consulted 25 chief architects, managing directors, and senior engineers or researchers working on IPAs and/or LLMs on personal devices.Source: §1 · p. 5
• The paper surveys three technical dimensions: fundamental capabilities, efficiency, and security and privacy, each decomposed into key research problems and relevant techniques.Capabilities include task execution, context sensing, and memorization; efficiency includes LLM inference, customization, and memory retrieval; security and privacy include data confidentiality, decision reliability, and system integrity.Source: §1 · p. 5

2.1 Timeline View of the Intelligent Personal Assistants History

• Figure 2 divides IPA development into four stages: speech recognition, software integration, mobile assistants, and LLM-based assistants.The stages span the 1950s–1980s, 1990s–late 2000s, early 2010s, and the recent LLM era.Source: §2.1 · pp. 5–6, Figure 2 · p. 6

2.2 Technical View of the Intelligent Personal Assistants History

• The technical view focuses on task automation and distinguishes four concurrently developing solution types rather than a strict chronology.These types are template-based programming, supervised learning, reinforcement learning, and early foundation-model adoption.Source: §2.2 · p. 6

2.2.1 Template-based Programming

• Template-based automation maps commands to predefined workflows, providing reliable and accurate execution but limited scalability and inflexible user experiences.Developers must create templates for supported functions, although end-users can also define repetitive sequences through tools such as Shortcuts.Source: §2.2.1 · pp. 6–7

2.2.2 Supervised Learning Methods

• Supervised learning predicts actions and states from task inputs and GUI states, potentially generalizing to unseen tasks but requiring extensive high-quality human-annotated data.Representative methods learn GUI representations and interaction models from text, images, structures, screenshots, and multimodal UI data.Source: §2.2.2 · pp. 7–8

2.2.3 Reinforcement Learning Methods

• Reinforcement-learning agents learn task automation through interaction and rewards, but huge action spaces, sparse rewards, and difficult reward design limit generalization.Workflows, curricula, factorized action spaces, and hierarchical policies reduce exploration difficulty.Source: §2.2.3 · p. 8

2.2.4 Early Adoption of Foundation Models

• LLMs such as ChatGPT support prompt-customized reasoning and tool use, while commercial products such as Copilot mainly provide shallow integrations amid unresolved efficiency, security, and privacy issues.LLM tools have been applied to browsers, robots, and code interpreters, but deeper functional integration with personal assistants remains an early-stage research direction.Source: §2.2.4 · pp. 8–9

3 Personal LLM Agents: Definition & Insights

• Personal LLM Agents integrate an LLM with personal data, devices, and services to reduce repetitive work and provide personalized assistance.The focus includes personal-data management, smartphone-app use, and deployment on resource-constrained devices.Source: §3 · p. 9

3.1 Key Components

• The architecture uses a foundation model to coordinate local resources, user context and memory, and responsive or proactive skills.Local resources include mobile applications, sensors, and IoT devices; user context and historical memory support personalized services and can be managed with systems such as vector databases.Source: §3.1 · p. 9, Figure 4 · p. 10

3.2 Intelligence Levels of Personal LLM Agents

• The five-level taxonomy progresses from explicit step following (L1), through autonomous execution and planning (L2-L3), to proactive personalization (L4) and an autonomous avatar (L5).L3 agents iteratively generate and execute plans using intermediate feedback, while L5 agents predict activities, collaborate with other agents, and self-evolve.Source: §3.2 · pp. 10–11, Figure 5 · p. 11

3.3 Opinions on Common Problems

• Experts identify efficient data management, work and life assistance, personalized recommendations, autonomous task completion, emotional support, and digital representation as desired agent features.They also highlight multimodal and context-aware intelligence, lightweight-model capability, efficient local-remote collaboration, data privacy, harmlessness, personalization storage, and operating-system support as urgent challenges.Source: §3.3 · pp. 12–14
• 88% of experts prefer edge-cloud collaboration, 58.33% support local deployment, and 81.82% are dissatisfied with cloud-only solutions because of latency, privacy, and cost.The deployment findings are reported in Figure 6.Source: §3.3 · p. 11, Figure 6 · p. 13
• 66.67% of experts favor combining fine-tuning with in-context learning for personalization, while multimodal, voice-based, and autonomous decision-making capabilities are also prioritized.Text is the most preferred modality, Image ranks second, Voice is the most popular interaction method, and 47.83% rank more intelligent autonomous decision-making first.Source: §3.3 · pp. 11–12, Figure 7 · p. 13

4 Fundamental Capabilities

• Personal LLM Agents center on task execution, context sensing, and memorization as three fundamental capabilities for personal assistance.Task execution translates commands or proactively perceived tasks into actions on personal resources; context sensing perceives the user and environment; memorization records user data for recall, knowledge summarization, and self-evolution.Source: §4 · p. 14

4.1 Task Execution

• Task execution requires interpreting verbal or written commands, planning and reasoning over them, and controlling smartphones, computers, or IoT devices through appropriate tools.Voice recognition conveniently converts speech to text, after which agents can manipulate personal data and use UIs when applications lack comprehensive API support.Source: §4.1 · p. 15

4.1.1 Task Automation Methods

• Task automation divides into code-based API calling and UI-based interaction, with UI methods improving flexibility where APIs are unavailable but facing privacy, security, complexity, and representation challenges.Code-based methods include slot filling and LLM program synthesis, while UI agents use text-based or multimodal GUI representations; multimodal approaches require substantial screen recording and current vision-language models have limited reasoning abilities.Source: §4.1.1 · pp. 16–17

4.1.2 Autonomous Agent Frameworks

• Autonomous agent frameworks combine an LLM for planning and self-reflection with memory and tool-use modules, enabling external-tool interaction, context-aware applications, and multi-agent collaboration.For mobile tasks, AutoDroid [94] can learn from app exploration or demonstrations, while extracted activity macros can support automatic task completion.Source: §4.1.2 · p. 17

4.1.3 Evaluation

• UI-task evaluation commonly uses completion rate or manually designed rewards, but neither fully resolves the difficulty of judging alternative valid action sequences and scalable real-world completion.Static datasets can misrepresent accuracy, dynamic simulation can be inefficient and hard to reproduce, and agents may hallucinate task completion; UI representation remains a choice between lightweight text and broadly available vision inputs.Source: §4.1.3 · pp. 17–18, Table 6 · p. 18

4.2 Context Sensing

• Context sensing gathers user or environmental information through hardware and software sources to customize services, support sensing-dependent tasks, enrich ambiguous tasks, trigger proactive assistance, and update agent memory.Hardware sensing uses sensors, wearables, edge devices, and other data sources, while software sensing analyzes existing information such as typing habits and common phrases.Source: §4.2 · pp. 18–19

4.2.1 Sensing Sources

• Sensing sources include hardware devices, software data, and collaborative multi-sensor systems, but effective deployment must select, fuse, and analyze sources while removing less important data to conserve resources.Hardware modalities span microphones, GPS, cameras, wearables, and smart-home devices; software sensing uses app activity, calls, typing, games, text, and speech.Source: §4.2.1 · pp. 19–20
• LLMs can use sensor data as prompts, through encoded embeddings and fine-tuning, or by redirecting raw data to domain-specific models, trading simplicity against computational cost, training difficulty, or limited direct understanding.Prompting is simple but costly for large raw streams and complex modalities; encoding supports complex sensors but is difficult to train; specialized models add interpretable information for LLMs.Source: §4.2.1 · p. 19

4.2.2 Sensing Targets

• Sensing targets comprise environments and users: environment sensing covers scenes, occasions, locations, cultures, and social relationships, while user sensing covers activities, states, health, emotions, goals, profiles, and personality.Scene and occasion awareness can change device behavior and task emphasis, while user sensing is divided into rapidly changing short-term states and more stable but difficult-to-maintain long-term information.Source: §4.2.2 · p. 20
• Short-term sensing captures rapidly changing activities, health, emotions, and stress, whereas long-term sensing infers deeper profiles, personality, psychological states, habits, and life outcomes that are essential for advanced personal agents.Long-term information is more subtle and challenging to perceive and maintain, despite its importance for advanced personalization.Source: §4.2.2 · pp. 20–21
• Existing sensing methods are often limited to particular sensors, applications, or domains, motivating unified representations that let agents combine diverse environmental and user signals and decide what and when to sense with minimal overhead.The section identifies a need for a unified ontology or format for downstream use and policies for selecting sensing actions efficiently.Source: §4.2.2 · p. 21

4.3.1 Obtaining Memory

• Personal LLM Agents obtain memory by logging raw inputs, events, contexts, and activities or inferring higher-level traits, emotions, preferences, and knowledge from raw data [215].Memory may take the form of key-value profiles, timestamped historical sequences, or files such as documents, photos, and videos.Source: §4.3.1 · pp. 21–22

4.3.2 Managing and Utilizing Memory

• Memory management comprises raw-data processing, memory-augmented LLM inference, and agent self-evolution for improving personalized services.These categories address how agents access and transform data, use memories during reasoning, and dynamically update themselves.Source: §4.3.2 · p. 22
• Raw-data interfaces translate user requests into database operations, while stream-based frameworks such as PrivacyStreams unify personal-data access and processing for LLM comprehension.Machine-learning and template-based methods map requests to SQL statements [217].Source: §4.3.2 · p. 22
• Memory-augmented inference uses short-term working memory and long-term episodic or world knowledge to support reasoning, planning, decision-making, and learning [85] [219] [220] [221] [222].Short-term memory carries current inputs, goals, retrieved knowledge, and intermediate reasoning, while long-term memory stores experiences and knowledge that can be retrieved or updated.Source: §4.3.2 · p. 22
• Self-evolution uses memory to acquire and chain reusable skills and to fine-tune LLMs as tasks, knowledge, tools, and device constraints change [233, 234, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245].Parameter-efficient fine-tuning can reduce adaptation costs and support edge devices, while skill repositories enable later reuse.Source: §4.3.2 · pp. 22–23
• Open problems concern organizing huge, heterogeneous, dynamic memories; deciding what to forget; and determining what data, timing, and method should govern self-evolution and foundation-model updates.These issues arise because user memory underpins personalization but inappropriate retained data can harm service quality and efficiency.Source: §4.3.2 · p. 23

Overview

• Efficiency optimization must address inference, customization, and memory retrieval because personal devices have limited hardware resources and power supplies.These low-level processes underpin the agents’ higher-level capabilities, including task execution, context sensing, and memorizing.Source: §5 · p. 23, Figure 9 · p. 23

Inference

• LLM inference is the performance bottleneck because it requires substantial computation and memory across task execution, context sensing, and memory generation.Agents use inference for task decomposition, step solving, context sensing, and memory generation, while tool and sensor costs are harder to estimate.Source: §5 · pp. 23–24

Customization

• Customization can consume substantial computation and storage when agents adapt to users, scenarios, or self-evolution through context conditioning or domain-specific tuning.Customization may use different context tokens or tune the LLM with domain-specific data, and it occurs frequently across deployments and scenarios.Source: §5 · p. 24

Memory manipulation

• Memory manipulation is costly because agents handle longer inputs and manage external memories containing perceptions, profiles, histories, and data files.The efficiency challenge involves both processing longer contexts and acquiring information from an external memory bank.Source: §5 · p. 24

5.1 Efficient Inference

• Inference efficiency is central because LLM inference dominates Personal LLM Agent runtime cost, while this section focuses on model- and system-level approaches rather than application-specific agent designs.The surveyed techniques target model size, latency, memory consumption, and energy consumption.Source: §5.1 · pp. 24–25

5.1.1 Model Compression

• Model compression reduces model size and computation through quantization, pruning, knowledge distillation, and low-rank factorization.PTQ is more flexible for on-device deployment than QAT; WOQ trades compression ratio against perplexity, while activation quantization addresses outliers through methods such as SmoothQuant.Source: §5.1.1 · pp. 25–26

5.1.2 Inference Acceleration

• Inference acceleration targets long-context and small-batch workloads through KV caching, context compression, optimized attention kernels, and speculative decoding.Attention cost grows near quadratically with context length; speculative decoding uses a lightweight draft model and a large oracle model to improve latency while preserving zero bias in generated results [269].Source: §5.1.2 · pp. 26–27

5.1.3 Memory Reduction

• Memory reduction must address model weights in short contexts and the linearly growing KV cache in long contexts.KV quantization and continuous token eviction can reduce memory, whereas one-shot prefill pruning is less effective for generation; offloading across GPU, CPU, and disk provides an orthogonal alternative through FlexGen.Source: §5.1.3 · p. 27

5.1.4 Energy Optimization

• Energy optimization targets computation and memory access through software methods, while specialized hardware including NPUs, TPUs, and FPGAs offers additional efficiency opportunities.Research on energy efficiency is still far from insufficient, although prediction models such as IrEne are limited to GPU host backends and lack generalization to platforms such as mobile phones.Source: §5.1.4 · p. 27
• Despite extensive progress, affordable ubiquitous deployment still requires compact accurate models, efficiency–quality tradeoffs through dynamic inference, and hardware and operating-system evolution.These are identified as open problems for Personal LLM Agents.Source: §5.1.4 · p. 28

5.2.1 Context Loading Efficiency

• Context loading efficiency improves by compressing KV caches and reusing attention states across overlapping prompts, reducing bandwidth demands and inference latency.CacheGen encodes KV caches into compact bitstreams, while Prompt Cache reuses precomputed attention states for recurring prompt segments.Source: §5.2.1 · p. 28

5.2.2 Fine-tuning Efficiency

• Fine-tuning efficiency is pursued through parameter-efficient methods, optimizer design, and curated data, including LoRA, (IA)3, Sophia, LOMO, and phi models.LoRA reduces memory and storage without additional inference latency and supports task switching; (IA)3 trains only learned vectors, while curated small datasets can match larger models on several tasks.Source: §5.2.2 · pp. 28–29
• Personal-device customization must account for limited memory and computing power, potentially using hierarchical storage, while open problems concern context management and customized-agent rollback or transfer.Existing methods often assume the full LLM fits in device memory; open questions address managing contexts across parallel agents, tasks, and users, and moving customized agents between versions or base models.Source: §5.2.2 · p. 29

5.3 Efficient Memory Manipulation

• Efficient external-memory retrieval combines vector indexing and fast search with end-to-end workflow optimization for retrieval-augmented LLM inference.External memories include user profiles, interaction histories, and local files represented as high-dimensional embeddings; retrieved knowledge is injected through prompts or intermediate-layer cross-attention.Source: §5.3 · p. 29

5.3.1 Search Efficiency

• Indexing avoids brute-force O(DN) comparisons by partitioning vectors into traversable tables, trees, or graphs using randomized, learned, or navigable methods.Representative techniques include RPTree, E2LSH, SPANN, NSW, HNSW, and Vamana.Source: §5.3.1 · pp. 29–30
• Hardware-aware indexing extends capacity while preserving efficiency: DiskANN achieves under 10ms latency on billion-point databases, while DiskANN++ raises QPS 1.5x–2.2x without losing accuracy.Approaches include disk-based indexes, software-hardware co-design, CXL memory pooling, and FPGA-based vector search.Source: §5.3.1 · p. 30

5.3.2 Workflow Optimization

• Pipeline methods exploit retrieval–generation parallelism: RaLMSpec uses speculative retrieval with verification, while PipeRAG dynamically adjusts retrieval intervals and search space.These methods use caching, scheduling, asynchronous verification, and algorithm-system co-design to improve performance without increasing end-to-end generation latency.Source: §5.3.2 · p. 30
• Personal LLM Agents additionally need memory systems supporting frequent updates, maintenance, and re-indexing, while compressing accumulated data to control storage and computation costs.Personal-device storage limitations make these requirements distinct from general external-vector-storage challenges.Source: §5.3.2 · p. 31

Security and Privacy

• Personal LLM Agents require protection of user data and service security through confidentiality, integrity, and reliability.Confidentiality prevents unauthorized disclosure, integrity protects decisions from malicious modification, and reliability addresses internal mistakes and dependable, truthful behavior.Source: §6 · p. 31, Figure 11 · p. 31

6.1 Confidentiality

• Personal LLM Agents must protect sensitive user data because they may initiate unseen queries and expose information to other agents or services.Candidate protections include local processing, homomorphic encryption, data masking, and permission access control.Source: §6.1 · pp. 31–32

6.1.1 Local Processing

• Local processing reduces cloud-transmission risks, but device resource constraints can slow inference, limit model accuracy, and make cloud-edge collaboration necessary.Lightweight models, deployment frameworks [362], and compression techniques [363] [246] support edge deployment, although private data cannot always remain local.Source: §6.1.1 · p. 32

6.1.2 Secure Remote Processing

• Secure remote processing uses homomorphic encryption, multi-party communication, or trusted execution environments [372], but faces operation, computation-cost, attack, and latency limitations.HE cannot accurately execute operations such as max, min, and softmax and remains far from agents’ latency demands; Crypten supports MPC tensor and GPU operations, while TEE may suffer attacks.Source: §6.1.2 · p. 32

6.1.3 Data Masking

• Data masking replaces sensitive inputs or encodes them into hidden vectors, but anonymization must balance privacy leakage, inference accuracy, and speed.PII identification has unclear boundaries; EmojiCrypt [375], representation defenses [376, 377, 378, 379], and related methods can improve performance but may leave encoded vectors vulnerable to leakage.Source: §6.1.3 · pp. 32–33

6.1.4 Information Flow Control

• Output privacy requires transparent, permission-based information-flow control because LLMs cannot provide rigorously provable confidentiality and output-data traceability remains open.Users should know who accessed what data, when, why, and how; systematic mechanisms must reconcile symbolic permissions with LLMs’ uninterpretable behavior while balancing efficiency and effectiveness.Source: §6.1.4 · p. 33

6.2 Integrity

• Integrity requires Personal LLM Agents to resist attacks that steal data or assets, disrupt normal operation, or corrupt intended outputs.Traditional threats such as parameter modification, local-data theft, and tampering can be addressed with encryption, permissions, and hardware isolation, but newer attack types also require attention.Source: §6.2 · p. 34

6.2.1 Adversarial Attacks

• Adversarial attacks can manipulate third-party inputs to induce unsafe agent actions, while AutoDAN generates readable jailbreak prompts that bypass perplexity filters.LLM scale and autoregressive generation make some existing defenses computationally difficult, and adversarial training may be impractical; parameter-efficient fine-tuning remains an open direction.Source: §6.2.1 · p. 34

6.2.2 Backdoor Attacks

• Backdoor attacks implant hidden decision logic through poisoned data or modified test-time prompts, and no robust defense yet handles attacks that mimic legitimate behavior.Prompt-triggered methods such as ProAttack and PoisonPrompt alter LLM decision logic, while backward-probability testing can fail when attackers imitate normal behavior.Source: §6.2.2 · p. 34

6.2.3 Prompt Injection Attacks

• Prompt injection can bypass alignment safeguards and induce sensitive operations, while SmoothLLM provides limited protection and prompt transparency offers a mitigation path.Controlling prompt templates, preprocessing third-party content, and encapsulating its origin can help distinguish external content from system prompts; SmoothLLM reduced attack success by about 1% for some models.Source: §6.2.3 · p. 35
• Open problems are detecting third-party tampering by learning normal inputs and behavior, and designing secure, user-friendly verification when agents are uncertain.These challenges reflect diverse, evolving threats and defensive techniques that lag behind them.Source: §6.2.3 · p. 35

6.3.1 Problems

• Reliability problems include hallucinated answers, unrecognized operations that violate executable formats, and sequential reasoning failures involving context and coherence.Personal agents face higher reliability requirements than text-only systems because their outputs must execute actions; proposed remedies include Tree-of-Thought, Cumulative Reasoning, and task-level planning [85] [408] [89].Source: §6.3.1 · pp. 35–36

6.3.2 Improvement

• Improvement methods align model behavior with human values through pre-training, fine-tuning, reward modeling, reinforcement learning, and human oversight.Representative techniques include RLHF, RLAIF [414], and C-RLFT [415].Source: §6.3.2 · p. 36
• Self-reflection improves reliability through iterative feedback, repeated-answer consistency checks, self-debugging, and multi-agent discussion, but increases computational cost.Examples include Reflexion, Self-Debug [419], SelfCheckGPT [420], and mutual discussion and verification [421].Source: §6.3.2 · p. 36
• Retrieval augmentation adds traceable, updateable external knowledge, but noisy context, retriever dependence, and unavailable matching content limit its assistance.Methods address these issues through relevance filtering, self-reflection, retrieval-guided reasoning, and external-tool verification [222] [426] [427] [428] [429] [430].Source: §6.3.2 · pp. 36–37

6.3.3 Inspection

• Inspection verifies generated outputs with constrained generation, self-verification, feasibility checks, and explanations, while intermediate-feature analysis detects signals of false or hallucinatory content.Responsible Task Automation predicts command feasibility and executor completeness, but sensitive-operation detection still needs better accuracy and recall; explanations support significant decisions and debugging.Source: §6.3.3 · p. 37
• Open reliability challenges include authoritative evaluation without costly black-box models or human annotation, rule compliance without sacrificing capability, and interpretable internal mechanisms.Avoiding unreliable behavior remains difficult, especially for hallucination, making evaluation, symbolic constraints, and transparency continuing research directions.Source: §6.3.3 · p. 37

• The paper systematically discusses key opportunities and challenges for Personal LLM Agents using domain expert feedback and an extensive literature review.Source: §7 · p. 38
• Personal LLM Agents still have inadequate task execution, narrow functionality, and unresolved efficiency, reliability, usability, performance, and security issues.Their large-scale parameter requirements also conflict with personal constraints on resources, privacy, and security.Source: §7 · p. 38
• Progress requires joint development of the full software/hardware stack and ecosystem, alongside careful attention to the technology’s responsibility and benign, assistive nature.Source: §7 · p. 38