Natural Language based Context Modeling and Reasoning for Ubiquitous Computing with Large Language Models: A Tutorial

1 INTRODUCTION

• LCaC maps LLM language understanding, reasoning, and planning onto context-aware computing’s three layers: context modeling, context reasoning, and context interfacing.The layers respectively represent context data, interpret it to generate understanding, and interact with users, sensors, and actuators.Source: §1 · pp. 2–3
• Earlier context-aware systems used tuples and ontologies, while rule-based reasoning could lack generalizability or extensibility; the authors present LLM-powered AutoAgents as a feasible direction.Natural-language context representation is considered feasible, although the authors acknowledge ambiguity issues.Source: §1 · pp. 3–4
• LCaC represents user requests, sensor data, and actuator commands as text; an AutoAgent models context, sends requests and context to an LLM, then executes its structured action plan through actuators.The AutoAgent is assumed to run on ubiquitous devices and to interact with users, LLMs, the Web, sensors, and actuators.Source: §1 · p. 4, Figure 2 · p. 4
• The work’s primary contribution is a systematic review and tutorial bridging ubiquitous computing and AI, including LCaC architecture, two showcases, performance factors, and future directions.The showcases concern medication and trip planning; the tutorial also discusses token limits, prompting, trustworthiness, interpretability, domain-specific tuning, task decomposition, and physical embodiment.Source: §1 · pp. 4–5
• The authors identify few efforts at the intersection of context-aware computing and LLM-powered AutoAgents, and position their tutorial as filling that gap with examples, reviews, and discussion.They distinguish this tutorial from prior tutorials on context-aware systems, an LLM review, and an online AutoAgent review.Source: §1 · p. 5

2.1.1 Foundation Models.

• Foundation models learn patterns from large datasets and can be adapted for downstream tasks; they span data modalities and perception or generation tasks.The passage describes them as a base architecture for building AI systems.Source: §2.1.1 · p. 5

2.1.2 Large Language Models.

• LLMs support text understanding and generation, with encoder-only, encoder-decoder, and decoder-only architectures serving different task patterns.The text contrasts BERT-style understanding, encoder-decoder processing and generation, and GPT-style text generation.Source: §2.1.2 · pp. 5–6

2.1.3 Generative Pre-trained Transformer.

• GPT training requires pre-training and supervised fine-tuning, while post-pretraining and reinforcement learning from human feedback are optional stages.Pre-training uses large text collections; supervised fine-tuning uses labeled examples, and post-pretraining can adapt domain-specific text.Source: §2.1.3 · p. 6

2.2.1 Prompting.

• Prompts are designed input texts that guide LLM responses; prompt-based tuning instead optimizes prompts for downstream tasks while keeping model parameters frozen and using labeled question-answer pairs.The background distinguishes discrete and continuous prompts.Source: §2.2.1 · p. 7

2.2.2 Few-shot Prompting and In-Context Learning.

• The news examples use GPT-4 for title classification and personalized recommendations based on a user’s previous reading habits during office hours.The examples illustrate classification and recommendation from prompt-provided task descriptions or demonstrations.Source: §2.2.1 · p. 7, §2.2.2 · p. 8
• Few-shot prompting uses a few examples in the prompt to guide accurate responses without further fine-tuning.The news example uses prior reading records from the same context to personalize recommendation.Source: §2.2.2 · pp. 7–8

2.2.3 Prompting in the General Form.

• In the general question-answering form, a program templates task descriptions and examples with a question, calls GPT-4, and returns its response.The described procedure uses ChatGPT’s API to invoke GPT-4.Source: §2.2.3 · p. 8

2 import openai

• Prompt-based question answering formats task descriptions, examples, and a question into a template, sends it to GPT-4, and returns the response.The examples describe using this template for general-purpose prompting.Source: §2 · p. 9

2.2.4 Prompt-based Tuning.

• Prompt-based tuning methods adapt frozen LLMs using prompts, with discrete and continuous approaches differing in how they modify inputs or add learnable parameters.Discrete methods search for or generate input text; continuous methods add small learnable parameters such as prefixes or layer-wise prompts.Source: §2.2.4 · p. 9

2.2.5 Discussions.

• Pre-training, post-pre-training, supervised fine-tuning, RLHF, in-context learning, and prompt-tuning strategies differ in objectives, training data, and weight updates.In-context learning uses demonstrations without model-weight updates, unlike approaches that update weights.Source: §2.2.5 · p. 10, Table 1 · p. 10

2.3 LLM-powered Applications for Ubiquitous Computing and Computer-Human Interactions

• Reviewed LLM applications span health, writing, gaming, coding, user interaction, and HCI research, including studies published by IMWUT and SIGCHI.Examples include mental-health diagnostics, fitness-tracker search, conversational public-health interventions, personalized game quests, and mobile-UI interaction.Source: §2.3 · pp. 10–11
• These applications are characterized as supporting more natural, personalized, and context-relevant interactions across diverse domains.Source: §2.3 · p. 12

3 LLM-DRIVEN CONTEXT MODELING AND REASONING

• LCaC is introduced as an architecture and LangChain-based implementation for natural-language context modeling and reasoning.Source: §3 · p. 12

3.1 LCaC Architectural Design: An AutoAgent-centric Approach

• The architecture has four components—users, sensors and actuators, an AutoAgent on ubiquitous devices, and LLMs—with the AutoAgent integrating them.LLMs power the AutoAgent's context modeling and reasoning.Source: §3.1 · p. 12
• The AutoAgent converts a user's voice or text instruction into a templated prompt and sends it to the LLM's question-answering API.Source: §3.1 · p. 12
• The LLM returns a structured response that the AutoAgent decodes into an action plan, then executes through sensors and actuators.A single request may require multiple LLM–AutoAgent exchanges.Source: §3.1 · p. 12

3.2 Implementation of an AutoAgent with LangChain

• LangChain implements the AutoAgent with a conversational agent, conversational memory, and an action plan, with optional databases, knowledge graphs, and pre-trained models as tools.Source: §3.2 · p. 13

3.2.1 Conversational Agent.

• Conversational memory retains prior exchanges as state to improve the accuracy and relevance of multi-round conversations, context modeling, and action planning.Source: §3.2.1 · p. 14
• The agent classifies a user's request, selects a prompt template, converts sensor readings into textual context, and sends the resulting prompt to the LLM for an action plan.It then follows that plan using sensors and actuators.Source: §3.2.1 · p. 14

3.2.2 Conversational Memory.

• The current OpenAI chat-completion API does not directly continue conversations through session IDs; LangChain can instead store and append past conversation text, or use deployed LLMs with hidden-state session management.Source: §3.2.2 · p. 14

3.2.3 Action Plan.

• LLM action planning faces domain-specific knowledge gaps, input/output token limits, and challenges in optimal planning and scheduling.JIT-generated code can implement textual actions in a local runtime, typically an isolated sandbox, and can call sensors, actuators, cloud services, or other tools.Source: §3.2.3 · pp. 14–15

3.2.4 Tool-Uses.

• Databases and knowledge graphs provide exact information for context modeling and can support data-driven modeling and rule-based reasoning alongside ontology-based solutions.Source: §3.2.4 · p. 15
• GPT-4 is described as performing cross-table reasoning for database use without prompt demonstrations.The example concerns generating SQL to search movie reviews across three MySQL tables.Source: §3.2.4 · pp. 15–16
• Pre-trained models can extend LCaC perception and AI capacity, with their outputs used as continuous-prompt tokens for LLM reasoning over images.The section also describes GATO as a multi-modal, multi-task, multi-embodiment generalist policy.Source: §3.2.4 · p. 16

3.3 Discussion on the LCaC Architecture

• The paper presents two showcases to illustrate LCaC architecture practices, prompt design, and possible LLM responses.Source: §3.3 · p. 16

4.1.1 System Settings.

• The showcase places a mobile z-arm in an apartment with multiple rooms, where travel between rooms takes about 1 to 2 minutes and the arm must return to its charger.The user's example request is to take two aspirin pills with water at 10 PM in the living room.Source: §4.1.1 · p. 17
• A LangChain-based agent classifies the request, selects and fills a matching prompt template with request and context text, then obtains and follows the LLM's action plan.The simple system may be stateless, without Conversational Memory.Source: §4.1.2 · p. 18

4.1.3 Prompts and Responses for Action Planning.

• The zero-shot prompt yields a schedule for retrieving aspirin, bringing water, delivering both, and returning the mobile z-arm to its charger.The example schedules actions between 9:56 PM and 10:07 PM, but its response format might not be machine-readable.Source: §4.1.3 · p. 18, Figure 10 · p. 18
• Few-shot examples produce a machine-readable action plan, and the response is more concise than the zero-shot version while both plans give correct actions.The few-shot prompt adds examples of expected responses; the zero-shot version tends to list more events.Source: §4.1.3 · p. 18, Figure 11 · p. 19

4.2 Context-aware Personalized Trip Planner and Itinerary Scheduler

• The showcase frames the personalized trip planner as an LLM-powered AutoAgent recommending attractions and scheduling itineraries from travel history and visit context.The example is described as a context-aware personalized trip-planning application.Source: §4.2 · p. 19

4.2.1 System Settings.

• The system uses past travel records to infer interests and mandatory temporal or geographic constraints to shape the current trip schedule.The example records visits to Rome and Barcelona and specifies Paris arrival, events, and departure times.Source: §4.2.1 · p. 19

4.2.4 Action Implementation.

• After users discuss and finalize a plan, the AutoAgent prompts the LLM to generate an action plan and calls third-party APIs to make reservations.The described reservations include hotels, restaurants, and electronic tickets for attractions.Source: §4.2.2 · p. 20

4.2.3 Prompts and Responses for Trip Planning and Itinerary Scheduling.

• The proposed itinerary reflects past interests and accommodates the research talk, banquet, and flight departure constraints in the Paris trip example.It includes a breakfast before the talk, a return to the hotel before the banquet, and an early airport checkout.Source: §4.2.3 · p. 20

• The showcases use natural language and LLMs for context modeling and reasoning, with the resulting action plan followed by the agent to fulfil the request.The agent's execution is part of the demonstrated workflow.Source: §4.3 · p. 20
• Carefully written examples in few-shot prompts can teach GPT-4 to respond in a desired way without fine-tuning.Source: §4.3 · p. 21
• Identical prompts can yield different action plans, although the differing order of fetching aspirin and water did not affect fulfilment in the example.The authors attribute this variability to GPT-4's dynamic, creative generation rather than deterministic output.Source: §4.3 · p. 21
• The authors recommend relatively low generation temperature, syntax analysis or action parsers, and optionally training a custom LLM on annotated requests and plans.They caution against extremely low temperature because it can reduce creativity needed for planning or imagination.Source: §4.3 · p. 22

5.1 Benefits and Limitations of LCaC

• LCaC offers adaptable, user-friendly natural-language interaction and flexibility with new observations, but can introduce ambiguity, inefficiency, and harder-to-interpret reasoning.Widget-based interaction requires users at a mobile terminal; ontology-based reasoning provides exact inference through predefined rules.Source: §5.1 · p. 23

5.2 Performance Affecting Factors

• Token limits create a balance: shorter inputs can omit needed information, while longer inputs may improve performance but increase computational inefficiency.Short outputs risk incompleteness; longer outputs can become irrelevant or repetitive.Source: §5.2 · p. 24
• Hierarchical plans can decompose long-running tasks into subtasks at varying abstraction levels using prompts and techniques such as CoT or ToT.LangChain can integrate these techniques for plans spanning extended periods.Source: §5.2 · p. 24
• Trustworthy, interpretable LLMs matter for LangChain action planning: reliability concerns plan quality and hallucinations, while interpretability supports issue identification and process refinement.The authors connect these qualities with more effective, safe, and robust planning and execution.Source: §5.2 · p. 24

5.3 Future Research Directions

• Future work includes adding vision, acoustic, and haptic modalities and improving LLM architectures, pre-training data, strategies, and tasks.Page limits excluded other directions, including scaling LLMs for emergent abilities.Source: §5.3 · pp. 24–25
• Domain adaptation may use post pre-training, supervised fine-tuning, or prompt tuning; prompt tuning keeps the model frozen while tuning continuous prompts to reduce training memory use.The passage also notes prompt tuning may support adaptation to new domains in modalities such as images.Source: §5.3 · p. 25
• Task decomposition breaks complex tasks into subtasks; CoT provides manageable steps, while ToT explores multiple reasoning possibilities at each step.Prompting, task-specific instructions, or human input can support decomposition and help agents plan ahead.Source: §5.3 · p. 25
• A proposed gateway with unified communication protocols could facilitate AutoAgent interaction with varied forms of physical embodiment.The authors describe tool use and embodiment as ways to perceive and sense the physical environment and respond to changing circumstances.Source: §5.3 · p. 25

6 CONCLUSIONS

• LCaC represents user requests, sensor data, and actuator commands as text; an AutoAgent models context, prompts an LLM for reasoning and an action plan, then executes that plan.The architecture integrates users, LLMs, AutoAgents on ubiquitous devices, and sensors and actuators in LangChain without model fine-tuning.Source: §6 · p. 25
• The two showcases are operating a mobile z-arm in an apartment for assisted living and planning a personalized, context-aware trip itinerary.Source: §6 · p. 26
• The paper analyzes factors that may influence LCaC performance and discusses future directions, anticipating applications in healthcare, transportation, smart homes, and retail.Source: §6 · p. 26