From Language to Drives: A Multimodal Framework for Aligning Autonomous Robots with Human Purposes

• Human purposes are ambiguous and context-dependent, while motivational architectures require structured representations, leaving a semantic gap that can undermine trust and safety.Existing explainability and grounding strategies address aspects of the problem but do not fully embed human purposes into motivational dynamics.Source: §I · p. 1
• LLMs and VLMs provide semantic and cultural priors for interpreting and clarifying purposes, while goal and skill discovery remain grounded in the robot’s developmental learning.The models inform, rather than replace, the motivational dynamics that sustain autonomous development.Source: §I · pp. 1–2
• The framework has three layers: semantic alignment, mission generation, and drive modeling, with an alignment drive prompting clarification until the intended purpose is coherent.Missions are translated into quantitative drive functions embedded in the robot’s motivational system.Source: §I · p. 2
• The work contributes a developmentally grounded framework, integrates LLM- and VLM-based modules as semantic priors, and validates alignment in simulation and on a physical robot.Its stated aim is to sustain long-term growth while remaining aligned with human intentions.Source: §I · p. 2

• SayCan ranks available skills by semantic relevance and uses a value function to assess feasibility, enabling long-horizon instructions when required primitive skills exist.Its behavioral repertoire remains fixed, without generating new goals or expanding its skill space.Source: A. Instruction Following and… · p. 2
• RT-2 maps multimodal inputs directly to low-level actions and supports zero-shot generalization, but lacks explicit motivational or goal representations.The cited work notes concerns about interpretability, regulation, and adaptation in open-ended environments.Source: A. Instruction Following and… · p. 2
• Pragmatic intention methods classify instruction explicitness or use dialogue to increase certainty, while shared autonomy adapts preferences through feedback.These approaches remain within predefined templates or refine existing behaviors rather than supporting autonomous goal discovery or new purposes.Source: A. Instruction Following and… · p. 2

B. Developmental Robotics and Open-Ended Learning

• Developmental systems use intrinsic motivations to acquire competencies and construct skill repertoires, yet typically assume purposes are hard-coded or structured rather than linguistically expressed by people.The unresolved issue is integrating human purposes without disrupting autonomous developmental dynamics.Source: B. Developmental Robotics and… · p. 2

C. Alignment in Embodied Agents

• Instruction-following models and developmental architectures address different pieces of alignment; the missing framework combines multimodal semantic priors with explicit motivation while preserving grounded learning.The proposed work places LLMs and VLMs within a motivation-based architecture to transform human purposes into aligned drives.Source: C. Alignment in Embodied Agents · p. 3

III. PROPOSED APPROACH

• LLMs clarify culturally grounded purposes from multimodal information, then the framework decomposes confirmed purposes into missions and drive functions that guide autonomous behavior.Users specify objectives in natural language without needing to know the robot’s motivational structure.Source: §III · p. 3

A. Multimodal Input

• Visual input is processed by a VLM to describe the scene and infer intent; audio uses Whisper for transcription and gTTS for vocal responses.Source: A. Multimodal Input · p. 3

B. Alignment Engine

• The Alignment Engine reformulates the user’s purpose using language and visual context, asking brief follow-up questions when details are ambiguous or missing.Source: B. Alignment Engine · p. 3

1) Prompt:

• The alignment prompt specifies context, robot and environment descriptions, perceptions, input format, task constraints, and response rules.These components are intended to make responses consistent and compatible with downstream components.Source: 1) Prompt · p. 3

2) Input and Output Configuration:

• The user supplies a natural-language purpose and, when available, an image description; the user decides when alignment is sufficient to proceed.The output is a refined purpose statement, or a standard infeasibility message if the model determines the purpose is unachievable with the robot.Source: 2) Input and Output Configuration · p. 3

C. Mission Generator

• The Mission Generator decomposes an aligned purpose into concrete end-states, each assigned a weight reflecting its relative importance.These weighted missions bridge high-level human intent and the robot’s motivational structure.Source: C. Mission Generator · p. 3

1) Prompt:

• The mission-generation prompt supplies context, robot description and perceptions, mission requirements, interaction and format rules, and predefined internal needs.The module takes the refined purpose and returns an array of [tag, weight] pairs, where each tag names a meaningful end-state and each weight indicates priority.Source: 1) Prompt · p. 4

D. Drive Generator

• A continuous, monotonic drive function quantifies mission urgency from the robot’s current state, reaches zero only when the mission is fulfilled, and rises smoothly with deviation from the desired state.Each generated function is integrated into the motivational system.Source: D. Drive Generator · p. 4

1) Prompt:

• The prompt provides context and definitions, robot description and perceptions, the preceding stage’s mission format, and instructions for generating a function per mission.Source: 1) Prompt · p. 4

2) Input and Output Configuration:

• The module takes the refined purpose and generated mission list, then returns one valid Python drive expression per mission; it reports impossibility if the robot’s capabilities cannot fulfill the purpose.Each output entry pairs the mission tag and weight with its drive expression.Source: 2) Input and Output Configuration · p. 4

IV. EXPERIMENTS AND RESULTS

• The two experiments test purpose alignment and interpretation, then mission and drive generation and their application to robotic tasks.Source: §IV · p. 4

A. Dataset

• The evaluation uses three purposes of increasing complexity: arranging apples by color, sorting colored cylinders into baskets, and stacking cylinders largest-to-smallest at the table center.Source: A. Dataset · p. 4
• For each purpose, the researchers created 15 ambiguous user inputs and used three images of the intended outcome, alternating images across trials.The images expose the VLM to different visual scenarios.Source: A. Dataset · p. 4

B. Semantic Alignment Test

• The test uses four conditions that isolate multimodal input and human feedback, and evaluates three purposes using cosine semantic similarity against ideal descriptions.The purposes are apples arranged by color, cylinders classified by color, and cylinders stacked by size; the test also uses ambiguous example inputs and ideal reference descriptions.Source: B. Semantic Alignment Test · pp. 4–5, Table I · p. 5, Table II · p. 5
• Initial low-detail inputs score 0.4–0.6 in cosine similarity, indicating a semantic gap that the Alignment Engine cannot reliably translate into defined missions and drives.The comparison uses generated descriptions and ideal manually written references, with cosine similarity computed from all-MiniLM-L6-v2 embeddings.Source: B. Semantic Alignment Test · p. 5
• Without user feedback, multimodal input raises similarity relative to the ideal reference, but generated purposes still diverge from that reference.Source: B. Semantic Alignment Test · p. 5
• Interactive conditions produce significantly higher semantic similarity than non-interactive conditions across all three tasks, and scores converge toward high, stable values.Fig. 2 summarizes the similarity distributions across multimodal and unimodal configurations, with and without feedback.Source: Figure 2 · p. 6, B. Semantic Alignment Test · p. 6
• Multimodal interaction reaches convergence in an average of 3 rounds, versus 4.5 rounds for text-only interaction, with some text-only cases requiring up to 11 rounds.Multimodal input does not substantially increase final similarity over linguistic-only interaction when both are interactive.Source: B. Semantic Alignment Test · p. 6

C. Mission and Drive Generation Test

• The test evaluated mission decomposition and drive-function generation separately from feedback by the Alignment Engine.An expert assessed whether mission sets enabled the intended purpose, whether weights were appropriate, and whether drives were mathematically and operationally valid.Source: C. Mission and Drive Generation Test · p. 6
• For the ideal apple-sorting purpose, four missions cover grasping and placing each apple, with later missions receiving higher weights to reflect task order.The associated grasping drives reach zero when the robot holds an apple and its hand coincides with the apple; placement drives reach zero at the designated side of the table.Source: C. Mission and Drive Generation Test · p. 6
• Human-feedback scenarios produced significantly more valid missions than scenarios without user intervention.The ideal purpose achieved 100% mission validity across the three tested purposes.Source: C. Mission and Drive Generation Test · p. 6, Figure 3 · p. 7
• Drive-function validity varied across tasks, with failures in 6 of 15 stacking instances and substantially worse results than for the first two purposes.The authors attribute the weaker stacking results to ambiguity in the purpose that can propagate into inconsistent performance and invalid drives.Source: C. Mission and Drive Generation Test · p. 7
• Repeated runs yielded essentially the same valid drives for well-specified purposes, while ambiguous purposes often produced different drive sets, some invalid.The authors suggest that this variability could prompt further dialogue to clarify the purpose.Source: C. Mission and Drive Generation Test · p. 7

D. Autonomous Robot Test

• The full framework, including three LLM instances for alignment, mission generation, and drive modeling, was integrated into the e-MDB cognitive architecture.The architecture's motivational system is based on needs, missions, and drives.Source: D. Autonomous Robot Test · p. 7
• The physical-robot task was to place a red cylinder inside a box and press a green button when the cylinder was outside reach.The system interpreted the purpose, generated missions and drive functions, and executed the task autonomously.Source: D. Autonomous Robot Test · p. 7
• The experiment ran ten times, with 50 trials per run recording the iterations needed to achieve the human purpose.The results were presented as a comparison with novelty-only behavior.Source: D. Autonomous Robot Test · p. 7
• The robot consistently reached the assigned goal in 3–4 iterations, close to the practical minimum for the objects’ initial positions.The authors report that generated missions and drives supported accurate, efficient task completion and directed behavior compared with novelty-based exploration alone.Source: D. Autonomous Robot Test · p. 7

V. CONCLUSIONS AND FUTURE WORK

• The framework uses three LLM-based components for semantic alignment, mission generation, and drive modeling to ground natural-language goals in a structured, motivation-driven architecture.The authors describe experimental validation in simulation and on a physical robotic platform.Source: §V · pp. 7–8
• Interactive alignment improved semantic precision, and clarification dialogue helped reduce ambiguity in translating purposes into executable motivational structures.The authors report that generated missions and drives satisfied required formal constraints and supported consistent goal-directed behavior when embedded in a cognitive architecture.Source: §V · p. 8
• The authors identify expanding multimodal interaction and investigating adaptive alignment over prolonged human–robot interaction as future work.They describe the system's potential as a semantic mediator within a developmental, motivation-based architecture.Source: §V · p. 8