TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patterns

Motivation

• LLM-based robot planners can translate natural-language instructions into executable plans, but their assumption of clear, complete, task-focused instructions may fail for people living with dementia.The introduction illustrates a resulting unsafe plan that places a knife in a microwave and turns it on.Source: §1 · p. 2

Research questions

• The study evaluates how dementia-associated communication patterns affect robot planning and what strategies can mitigate resulting planning failures.The authors identify five patterns: Referential Imprecision, Object Substitution, Empty Speech, Topic Drift, and Intrusion.Source: §1 · p. 2

Contributions

• TALK-Dem provides a benchmark of 4,800 instructions across 300 household tasks, covering five communication patterns at three intensity levels.Each seed task was conversationalized into a clean control and transformed into 15 pattern-and-intensity variants.Source: §1 · pp. 2–3
• CARE retrieves relevant previously resolved tasks and supplies their task interpretations and reference plans as query-specific in-context demonstrations.Unlike fixed prompting, it adapts the retrieved context to each incoming instruction.Source: §1 · pp. 2–3

Results

• Across patterned conditions, CARE improved average task success by 18.1 percentage points over the vanilla prompt.The evaluation also found that stronger pattern intensity generally led to greater degradation and that patterns induced distinct errors, including referent-resolution errors for Object Substitution, plan-organization errors for Topic Drift, and subgoal omission for Intrusion.Source: §1 · p. 2

Contributions

• The paper reports substantial performance degradation and distinct task-interpretation and planning failure modes across evaluated models.The authors describe their evaluation as comprehensive across open-weight and closed-source models.Source: §1 · p. 3

Embodied task planning

• Existing embodied-planning benchmarks and methods often center on ideal instructions or on ambiguity, uncertainty, and clarification rather than systematically varied dementia-associated communication.Examples include REI-Bench for vague referring expressions, AmbiK for ambiguous kitchen instructions, and DialFRED for task-relevant clarification.Source: §2 · p. 3

Dementia-associated language

• Clinical and psycholinguistic work describes dementia-related language changes across lexical-semantic access, referential expression, and discourse, which TALK-Dem operationalizes as five patterns for embodied interaction.The patterns are Referential Imprecision, Object Substitution, Empty Speech, Topic Drift, and Intrusion.Source: §2 · p. 3

3.1 COMMUNICATION PATTERNS

• The benchmark transforms a clean conversational control into an instruction that preserves the intended task while expressing a dementia-associated communication pattern at a specified intensity.The transformations are conditioned on scene context when needed; Figure 2 presents rows by pattern and columns by increasing intensity.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 TALK-DEM CONSTRUCTION PIPELINE

• TALK-Dem starts from 300 semantically diverse household tasks, conversationalizes each seed instruction, then generates 15 independent pattern-and-intensity variants per control.With clean controls included, the benchmark contains 4,800 instructions; Figure 3 summarizes the construction pipeline.Source: §3.2 · pp. 4–5, Figure 3 · p. 5
• The authors compared linguistic biomarkers in TALK-Dem with those observed in DementiaBank to examine whether generated instructions exhibit associated linguistic characteristics.Further details are provided in Appendix A.3.Source: §3.2 · p. 5

3.3 CONTEXT-AWARE RETRIEVAL FROM EXPERIENCE

• CARE retrieves semantically relevant prior tasks and provides their resolved interpretations and reference plans to the planner for a new instruction.Its memory bank uses held-out tasks disjoint from the evaluation benchmark, unlike fixed prompting or reformulation strategies.Source: §3.3 · p. 5

4.1 EXPERIMENTAL SETUP

• The evaluation uses SayCan in AI2-THOR with six open-weight models and two closed-source models, comparing the vanilla prompt against five mitigation strategies.The strategies are Aware Prompt, Chain-of-Thought, In-Context Learning, Task-Oriented Context Cognition, and CARE.Source: §4.1 · pp. 5–6

4.2 MEASUREMENTS

• Task success rate is the primary performance measure, with success defined by satisfaction of the simulator goal conditions.Clean-conditioned retention compares tasks solved under clean instructions with those solved under a communication-pattern variant; higher retention means more consistent performance across conditions.Source: §4.2 · p. 6

5.1 BENCHMARK RESULTS

• Referential Imprecision and Object Substitution caused the most consistent degradation, while discourse-level patterns had less consistent, model-dependent effects.The reported drops include 22.3 percentage points for Ministral under Referential Imprecision and 9.2 points for Qwen2.5-72B-Instruct under Object Substitution; delayed clarification or repeated task-relevant mentions could also make some tasks easier while disrupting others.Source: §5.1 · pp. 6–7
• Model scale did not consistently predict robustness: larger models sometimes degraded more, while the two closed-source models remained consistently high except claude-opus-5 under Referential Imprecision.Qwen2.5-72B-Instruct outperformed its 7B counterpart on clean instructions and most patterns, but Llama-3.1-70B-Instruct degraded more than its 8B counterpart on several patterns.Source: §5.1 · p. 7

5.2 IMPACT OF PATTERN INTENSITY ON PLANNING ROBUSTNESS

• Across open-weight models, stronger pattern intensity generally increased degradation, with Referential Imprecision showing the largest retention decline, from 82.1% to 54.4%.Object Substitution also declined monotonically, from 80.2% to 69.2%; Intrusion showed no consistent monotonic decrease.Source: §5.2 · p. 7, Figure 5 · p. 7

5.3 MITIGATION PERFORMANCE

• 78.1% average task success for CARE exceeded Vanilla prompt (60.1%) across six open-weight models and 15 pattern–intensity conditions.CARE also outperformed AP (56.2%), CoT (58.0%), ICL (64.9%), and TOCC (62.8%).Source: §5.3 · p. 7, Figure 6 · p. 8
• CARE retrieves semantically relevant resolved tasks and supplies their interpretations and reference plans as query-specific demonstrations.Unlike fixed prompting, retrieval adapts the context to each incoming instruction; the authors suggest this task-relevant evidence helps identify underlying tasks despite communication variation.Source: §5.3 · p. 8

6.1 ERROR ANALYSIS

• The error analysis manually examined 270 sampled cases where models succeeded on clean instructions but failed on patterned variants.The authors identified six failure types spanning entity and attribute interpretation, task completion, plan organization, and execution.Source: §6.1 · p. 8
• Object Substitution most often produced Referent Resolution Errors, while Topic Drift most often produced Plan Organization Errors.Intrusion was most strongly associated with Subgoal Omission; Referential Imprecision more often produced Location and State Interpretation errors, while Empty Speech most often yielded Subgoal Omission.Source: §6.1 · p. 8, Table 1 · p. 9
• Under Object Substitution, Llama-3.1-70B-Instruct selected the book after the instruction corrected the target from book to CD, and failed to re-identify the CD.It identified the CD and completed the task under the clean instruction.Source: §6.1 · p. 8, Figure 7 · p. 9

6.2 SAFETY-RELEVANT ACTIONS

• Among safety-relevant cases, 85 lacked hazard actions under clean instructions but showed them under transformed communications; 10 were still scored as successful.The scanned hazards included inappropriate microwave use, appliances or faucets left running, unsafe knife use or placement, and electronics placed in sinks or bathtubs.Source: §6.2 · p. 9

7 CONCLUSIONS

• TALK-Dem covers 4,800 instructions across 300 household tasks, five communication patterns, and three intensity levels; CARE generally improved task success across conditions.CARE retrieves relevant previously resolved tasks as query-specific demonstrations.Source: §7 · p. 9
• The study’s synthetic patterns may not capture real PLWD language variability; SayCan evaluation in AI2-THOR omitted physical deployment and longitudinal interaction.The authors call for validation with real-world and longitudinal interaction data, additional planning architectures, and physical robots.Source: §7 · p. 9
• TALK-Dem is a controlled benchmark of dementia-associated communication phenomena, not a simulation of PLWD’s full language distribution, and is not intended for clinical use.The synthetic utterances were generated from household-task instructions using author-designed transformation rules.Source: ETHICS STATEMENT · p. 10

A.1 RULE-CONSTRAINED GENERATION

• Generation verification checked task preservation, scene-object mentions, and pattern-specific structure; 4,489 of 4,500 transformed instructions passed all checks.The remaining 11 were retained with failed checks recorded rather than selectively discarded.Source: §A.1 · p. 15

A.2.1 CONVERSATIONALIZATION

• Clean conversational requests varied their framing while preserving task-relevant wording and excluding the communication patterns under study.The prompt prohibited adding, removing, replacing, or reinterpreting task requirements.Source: §A.2.1 · p. 15

A.2.2 PATTERN TRANSFORMATION

• Pattern-specific prompts transformed each clean conversational instruction into five dementia-associated communication patterns at three intensity levels.The passage gives Referential Imprecision at Level 1 as an example.Source: §A.2.2 · p. 16

A.2.3 SCENE LIST

• A listed living-room scene contains only the enumerated household items, including furniture, appliances, and personal objects.The scene inventory names items such as a sofa, microwave, cell phone, and key chain.Source: §A.2.3 · p. 16

A.3 COMPARISON WITH REAL DEMENTIA SPEECH

• The comparison used Cookie Theft descriptions from dementia and healthy-control transcripts, split into word-minimum chunks and sampled to balance the groups.The source comprised 309 dementia and 243 control transcripts; 938 chunks per group were compared.Source: §A.3 · p. 17
• The analysis measured lexical, referential, syntactic, repair, and discourse properties, including task relevance estimated by sentence-level cosine similarity.Markers with significant dementia-control differences were retained, and Cohen’s d represented standardized group differences.Source: §A.3 · p. 17
• For each marker, the Level-3 distribution was compared with dementia and control speech using Wasserstein-1 distance; the signed difference indicates which was closer.The distance is interpreted within each marker because marker units differ.Source: §A.3 · p. 17
• TALK-Dem’s targeted markers generally shifted in the same directions as dementia-versus-control differences, though some inserted discourse markers and repairs exceeded DementiaBank values.The authors interpret this as marker-level alignment, not distributional equivalence, because the datasets differ in task and discourse setting.Source: §A.3 · pp. 17–18

B.1 DETAILED BENCHMARK RESULTS

• Table B1 reports clean-instruction task success and average success-rate changes across three intensity levels for each of five communication patterns.Pattern effects are tested against clean outcomes using paired per-task comparisons.Source: Table B1 · p. 18

B.2 PLAN LENGTH ANALYSIS

• Table B2 compares executed plan lengths for clean–transformed pairs that succeeded under both conditions, pooling three intensity levels.It reports plan length, jointly successful-pair counts, and transformed-minus-clean differences.Source: Table B2 · p. 19

B.3 PATTERN INTENSITY RESULTS

• Table B3 reports clean-conditioned retention across three intensity levels and five communication patterns, alongside each model’s clean-success task count.Retention is scoped to tasks successfully completed under the clean condition.Source: Table B3 · p. 19

B.4 DETAILED MITIGATION RESULTS

• Tables B4–B9 report task success rates under mitigation strategies for six open-weight models.The captions identify AP, CoT, ICL, TOCC, and CARE as the compared strategies.Source: Table B4 · p. 20, Table B5 · p. 20, Table B6 · p. 21

B.5 CLEAN-CONDITIONED RETENTION UNDER MITIGATION STRATEGIES

• Tables B10–B11 compare clean-conditioned retention across mitigation strategies, overall and by communication pattern, with paired tests against CARE.The overall table averages across 15 pattern–intensity conditions per model and includes a six-model average.Source: Table B10 · p. 23, Table B11 · p. 23

B.6 ERROR TAXONOMY

• Table B12 provides the error taxonomy used in the qualitative analysis.Source: Table B12 · p. 23

C.1 DIVERSITY SAMPLING

• Greedy max-min sampling selected 50 trials per task type from 803 candidates, producing 300 tasks across six task types.The selected sets had lower mean pairwise cosine similarity than random subsets for all six task types.Source: §C.1 · pp. 23–24

C.2 HAZARD DETECTION RULES

• Safety-relevant action rules were defined relative to reference plans, and candidate actions were counted only after simulator-state replay confirmed them.Actions required by the task were not flagged; the sequence rules alone did not establish that an action succeeded.Source: §C.2 · p. 24
• Among cases without rule-detected actions under clean instructions, 85 transformed-instruction cases were confirmed after replay; 10 were nevertheless scored as successful.The sequence rules first identified 408 candidates; most rejected candidates involved microwave actions that failed in the simulator.Source: §C.2 · p. 25
• Most confirmed events involved unnecessary microwave use, while unrequested knife use occurred primarily under Referential Imprecision.Source: §C.2 · p. 25

D.1 COMPARISON OF MITIGATION METHODS

• The comparison schematic lists Aware Prompt, Chain-of-Thought, In-Context Learning, TOCC, and CARE as evaluated mitigation methods.Source: Figure D1 · p. 25

D.2 AWARE PROMPT

• Aware Prompt warns that instructions may contain communication irregularities and directs the planner to identify and plan only the requested task from the full utterance.Source: §D.2 · p. 25, Aware Prompt · pp. 25–26

D.3 CHAIN-OF-THOUGHT

• Chain-of-Thought first asks the model to infer the intended household task in one sentence, then gives that interpretation alongside the original prompt for planning.Source: §D.3 · p. 26

D.4 IN-CONTEXT LEARNING

• In-Context Learning uses five fixed demonstrations, one per communication pattern, each pairing an instruction with its task interpretation and reference plan.Source: §D.4 · p. 26

D.5 TASK-ORIENTED CONTEXT COGNITION

• TOCC rewrites an irregular instruction as one clear, complete task instruction, which the planner uses without the original instruction.Source: §D.5 · pp. 26–27

D.6 CONTEXT-AWARE RETRIEVAL FROM EXPERIENCE

• CARE retrieves a memory bank of resolved tasks and supplies the three most similar items as query-specific demonstrations.Its 1,600 memory tasks pair an instruction with a task interpretation and reference plan.Source: §D.6 · p. 27

E.1 ADDITIONAL FAILURE-MODE EXAMPLES

• The appendix illustrates ordering, referent-resolution, state-preservation, location, and omitted-subgoal errors across five cases.Examples include turning on a lamp before picking up a statue, selecting a tomato mentioned in topic drift, and failing to clean a spatula before placing it on a table.Source: Figure E2 · p. 28, Figure E3 · p. 28, Figure E1 · p. 28

• Five examples flag safety-relevant actions involving an empty microwave, a knife placed in a microwave, an appliance left running, a cell phone in a sink, and unrequested knife use.Source: Figure E6 · p. 29, Figure E7 · p. 30, Figure E8 · p. 31