Source-linked AI summary
PARAssist: A Framework for Personalized and Adaptive Robotic Assistance from Ambiguous User Requests
Pourya Aliasghari, Goldie Nejat
TL;DR
Ambiguous service-robot requests may require context beyond the current scene, while users may prefer assistance for particular task demands. PARAssist passively learns those preferences from multimodal user history and evaluates candidate tasks against them, with experiments and ablations supporting personalized disambiguation.
Problem
Existing disambiguation methods generally rely on the present request and scene, while personalization methods mainly use explicit demonstrations or feedback and do not readily transfer across assistive tasks.
Method
PARAssist passively observes user actions, activities, locations, and conversations, infers task-requirement vectors with VLMs, learns an assistance preference model, and evaluates candidate tasks generated from user history and current context.
Results
PARAssist reasoned over user action or conversation history and learned assistance preferences from inferred task-requirement vectors rather than explicit user-provided examples; ablations supported the main components’ contributions.
Takeaways & Limitations
Personalized disambiguation can align suggested assistance with a user’s prior assistance patterns by combining context-aware reasoning with a learned preference model.
Takeaways & Limitations
The demonstrated preference learning uses domestic assistance scenarios and receptionist assistance experiments, while prior single-task personalization does not easily transfer across assistive tasks.
Abstract
from arXiv · showhide
Service robots may encounter ambiguous user requests that require context-aware inference. Users may also have unique preferences with certain tasks when requesting robotic assistance. We introduce PARAssist (Personalized and Adaptive Robotic Assistance), a unique architecture for disambiguating requests in a personalized manner for service robots. PARAssist utilizes vision-language models to determine the physical and cognitive demands of a user's tasks, and passively learns user preferences for assistance by contrasting the demands of tasks the user performs independently with those they request from the robot. When an ambiguous request is received, task candidates are generated from the history of the user's actions, activities, locations, conversations, and requests, as well as the current user and environment state. Task candidates are then evaluated against the learned user preference model to suggest suitable assistance options. Experiments conducted with PARAssist show that personalization can align disambiguation with the task demands of a user's prior assistance requests. An ablation study confirms the contributions of PARAssist's main components in personalizing disambiguation.
I. INTRODUCTION
PARAssist addresses ambiguous robotic requests by combining context-aware task disambiguation with personalization from passively learned user assistance preferences.
- Ambiguous requests may omit object, location, or assistance details that humans infer from social context, prior interactions, and knowledge of users and environments.
- Existing disambiguation methods use general-purpose models and cannot resolve ambiguity according to unique, evolving user preferences.
- PARAssist combines passive multimodal observations of user actions, activities, locations, and conversations with a constantly updated preference model.
- Its candidate generator uses current task context and past user states, while its User Assistance Preference Model is learned without explicit demonstrations or feedback.
- PARAssist evaluates candidate tasks using both user preference and scenario-history relevance scores to suggest assistance for a specific user.
A. Service Robots Handling Ambiguous Requests
Prior work resolves ambiguous robotic requests mainly through clarification, inference, or scene-based candidate suggestions, while personalization typically targets task execution preferences and often requires explicit examples.
- Ambiguity in object selection, locations, spatial relations, and intended assistance is mainly addressed by asking clarification questions.
- Inference-based methods have used fuzzy logic for spatial quantities and transformer models for intent inference from objects handled before an instruction.
- Robot task personalization commonly learns how or when tasks should be performed, often from natural-language examples or visual demonstrations.
- Prior personalization studies include reward learning from trajectory preferences, latent-variable modeling of physical comfort, and preference modeling for task subtasks.
- LLMs and VLMs have personalized sorting, dressing, and object-placement assistance through examples, demonstrations, clarification, fine-tuning, or imitation learning.
C. Summary of Limitations
The paper identifies a gap in using recent user-state history and transferable assistance preferences for disambiguation, then describes PARAssist’s multimodal observation and candidate-generation architecture.
- Summary of Limitations: Existing methods rarely use recent user behavior to infer preferred assistance, while single-task personalization does not transfer easily across assistive tasks.
- PARAssist Architecture: The architecture passively observes users through visual and auditory inputs, analyzes actions, and generates and evaluates candidate assistance tasks from stored user data.
- Summary of Limitations: PARAssist addresses these limitations by grounding disambiguation in user-state history and using an assistance preference model across tasks.
- PARAssist Architecture: The Candidate Generator combines user history, environment information, robot capabilities, and the ambiguous request to produce candidate tasks with relevance scores and requirement vectors.
- PARAssist Architecture: The User Behavior Observer infers actions, locations, and activities from camera frames and temporal action sequences, storing filtered states in short-term memory.
- PARAssist Architecture: The Conversation Observer stores timestamped, speaker-attributed transcripts, while the Action Analyzer infers task-requirement vectors from visual or conversational cues.
IV. TRAINING
The User Assistance Preference Model is updated before each request using requirement vectors from independently performed tasks and robot-assisted tasks.
- The UAPM trains on independently performed tasks labeled 0 and robot-assisted tasks labeled 1, using requirement vectors inferred or generated by the system.
- Random undersampling of label 0 vectors retains all label 1 vectors to reduce bias when users perform many tasks independently but request little robotic assistance.
V. ASSISTANCE DOMAINS
PARAssist was implemented in two assistance domains: domestic tasks at home and receptionist inquiries at a university, using visual and conversational cues for personalization.
- The implementation covers domestic assistance in a home and inquiry assistance for a university receptionist.
- The domains examine personalization during disambiguation using both visual and conversational cues.
A. Assistance Domain 1: Domestic Assistance
The domestic domain uses multimodal observation, task-demand analysis, contextual history, and a preference model to select assistance for household tasks.
- The domestic robot supports everyday physical tasks at home, including retrieving objects.
- The User Behavior Observer processes real-time RGB input or offline videos, with YOLO11n detecting the user.
- The Assistance Candidate Generator combines current room and object context with recent actions, weekly activities, and the user’s last five robot requests.The history also includes the current date and time.
- The robot’s available capabilities include localization, detection, navigation, door operation, pickup, placement, and handover.
- The UAPM is logistic regression selected for simple formulation and suitability for small training sets, reducing overfitting risk.
B. Assistance Domain 2: University Receptionist Assistance
The university receptionist domain uses conversational triggers, environmental context, and robot capabilities to generate and evaluate assistance candidates for inquiries and physical support.
- The User Conversation Observer transcribes receptionist conversations in real time by segmenting microphone audio into utterances.Segmentation uses energy thresholding and voice activity detection, with each segment separated by 1 s of no speech input.
- The Assistance Candidate Generator activates when the receptionist says “hey robot” and makes a request, using medium VLM reasoning effort.
- Its context includes the reception location, possible locations from the floor-layout image, and the most recent user conversation.
- The robot’s capabilities include tracking time and location, retrieving schedules, navigating, checking room occupancy, and making verbal announcements.
- The selector presents a clarification cue for the top candidate task, and an accepted task is sent to the planner.Room occupancy is assessed from an RGB-camera image sent to the GPT-5-nano VLM.
VI. DOMESTIC ASSISTANCE SCENARIOS
Domestic scenarios evaluate PARAssist on ambiguous physical-task requests after observing users’ activities and unambiguous assistance requests across multiple household contexts.
- The study also varies VLM reasoning effort and ablates user history and the UAPM to assess their roles in disambiguation.
- The evaluation includes three scenarios, SC1–SC3, involving physical tasks at home.
- Each scenario provides five videos of real-world user activities for PARAssist to process.
- The first four videos vary by scenario and contain two distinct activities, each repeated twice to extend UAPM training observations.
- Users make eight unambiguous requests during these activities before the fifth video presents a shared laundry-area sequence.
A. Personalized Assistance Response to Ambiguous Requests
PARAssist was evaluated across three scenarios and two VLM reasoning-effort conditions by comparing suggested tasks with scenario-specific assistance criteria. Higher reasoning effort generally improved candidate quality, while latency was substantially higher.
- Scenario setup: The User Behavior Observer identified all activities except one organizing-cleaning-supplies trial that was classified as cleaning.The scenarios contrasted reaching-high, bending-down, and lifting demands between user actions and requested assistance.
- Candidate quality: At least one top-two candidate met both assistance criteria in 9, 10, and 10 MRE trials for SC1, SC2, and SC3, versus 8, 10, and 7 LRE trials.The criteria required including the scenario’s key physical aspect and contributing to starting the washer.
- Candidate quality: The most frequent candidates were moving clothing from the dryer top in SC1, picking up a floor sock in SC2, and retrieving detergent from a stool in SC3.These candidates were consistent across both MRE and LRE conditions.
- Condition effects: Reducing VLM reasoning effort lowered performance, with more infeasible suggestions in SC1 and SC3 or suggestions lacking SC2’s key aspect.SC2 suggestions often included bending down, whereas SC3 infeasibility may reflect inaccurate visual estimation of object weights.
- Condition effects: Assistance Candidate Generator latency was 75.5, 56.2, and 62.3 seconds in MRE for SC1–SC3, versus 23.6, 22.4, and 20.4 seconds in LRE.The reported standard deviations were 17.3, 12.3, and 8.8 seconds for MRE and 6.7, 1.0, and 2.8 seconds for LRE.
B. Ablation Study
The ablation study removed preference scoring, history relevance, or user history to test PARAssist’s components, while real-world experiments examined personalized responses to ambiguous receptionist requests.
- Ablation design: MRE-NoPS removed the Assistance Candidate Evaluator, MRE-NoRS removed relevance ranking by user history, and MRE-NoHist excluded user history from candidate generation.These variants were compared with the full MRE condition.
- Ablation findings: Without preference scoring, none of the top-two candidates included the key aspect in 7 SC1 trials and 6 SC3 trials.The authors interpret this reduction as emphasizing user-specific preference scoring for personalized disambiguation.
- Real-world experiments: In real-world scenarios, the receptionist requested robotic assistance only for long-distance walking or class-schedule memory.The robot used these recent assistance patterns to respond to an ambiguous classroom announcement request.
- Real-world experiments: The robot made a far-away Classroom C announcement in one scenario and a schedule-update announcement in Classroom B in the other.These responses matched the receptionist’s recent assistance requests and actions.
VIII. DISCUSSION
PARAssist personalizes ambiguous-request handling by combining context-aware reasoning with a learned user assistance preference model. The framework also exposes limitations in observation speed and candidate feasibility, while motivating broader validation and application domains.
- PARAssist reasons over user actions or conversations to infer assistance candidates instead of relying primarily on the current request and scene.This context-aware reasoning is combined with learned preferences from inferred task-relevant values.
- PARAssist learns user assistance preferences from inferred task demands rather than explicit user-provided examples.The preference model uses inferred physical and cognitive demands of user actions and robot tasks.
- Periodic VLM inference may miss fast user actions, causing PARAssist to suggest assistance for a task the user already completed.The paper suggests faster inference or buffering captured image frames as mitigations.
- Infeasible assistance candidates can arise in domestic tasks such as laundry when contextual representations and robot capability models are insufficient.Suggested mitigations include semantic maps or scene graphs and more detailed models of manipulable objects.
- Future work will generalize PARAssist to new assistive domains and evaluate user acceptance and disambiguation appropriateness relative to actual intent.