PaperlayerView exact revision on arXiv

Source-linked AI summary

Nutri-ATLAS: Embodied Agent for Tabulated Lookup and Assistance for Smarter nutrition

Uttej Kallakuri, Boxun Hu, Ankur A. Butala, Najim Dehak, Tinoosh Mohsenin

arXiv:2609.32803v1cs.AI

TL;DR

Nutrition assistants can be unreliable when descriptions are ambiguous or nutritional evidence is missing. Nutri-ATLAS combines graph-grounded retrieval, personalized recommendation, hardware-aware LLM selection, and robotic evidence acquisition; its hybrid retriever improves HealthyFoodSubs results, and its evaluations span nutrition tasks and deployment settings.

  • Problem

    Existing LLM nutrition assistants are limited by passive text interaction and static context, leaving them vulnerable when food descriptions are ambiguous or nutritional evidence is missing.

    Source: §I · p. 1

  • Method

    Nutri-ATLAS combines graph and text embeddings in a shared retrieval pipeline with personalized dietary context, hardware-aware LLM selection, and an LLM-guided robot that acquires food-availability evidence.

    Source: §I · p. 2

  • Results

    37.9% MAP, 80.7% RR@5, and 90.1% RR@10: the hybrid variant surpasses the HealthyFoodSubs baseline on all three metrics.

    Source: A. Unified Nutrition Graph Knowledge… · p. 11

  • Takeaways & Limitations

    Nutri-ATLAS grounds food recommendations in structured nutritional evidence and observed availability while evaluating nutrition assistance across retrieval, estimation, and embodied execution.

    Source: §VI · p. 15

Abstract

from arXiv · show

Generative and Agentic IoT systems offer a promising foundation for digital healthcare applications that combine sensing, personalized reasoning, and autonomous interaction in real-world environments. Nutrition assistance is a natural use case, but existing Large Language Model (LLM)-based systems are often limited to passive text interaction and static context, making them unreliable when food descriptions are ambiguous or nutritional evidence is missing. We propose Nutri-ATLAS, an Embodied Agent for Tabulated Lookup and Assistance for smarter nutrition in the real world. It integrates graph-grounded nutrition reasoning, hardware-aware LLM selection, and robot-based evidence acquisition. Nutri-ATLAS builds a unified Food-Nutrient knowledge graph from USDA FoodData Central and FoodKG and learns 64-dimensional GATv2 food and recipe embeddings. A shared hybrid graph-text scoring mechanism supports food nutrition extraction, nutritional gap filling, substitute retrieval, and recipe-level meal composition, while an LLM-guided skill interface navigates landmarks, updates dietary-context and food-accessibility memory, and grounds recommendations in observed food availability. We evaluate Nutri-ATLAS across nutrient estimation, substitution retrieval, recipe recommendation, patient-profile adherence, edge deployment, and real-world embodied execution. On HealthyFoodSubs, the hybrid retriever achieves 37.9% MAP, 80.7% RR@5, and 90.1% RR@10. On NutriBench v2, Dense+GAT retrieval grounds nutrient estimation across nine quantized Qwen3.5-9B configurations. On PFoodReQ, Nutri-ATLAS reaches 78.8% MAP, 83.0% MAR, and 77.5% F1. A patient-profile study shows adherence to allergy and healthy-target constraints for all selected cases.

I. INTRODUCTION

Nutri-ATLAS addresses the challenge of making nutrition recommendations faithful to structured nutrient knowledge, user context, and foods observed in the environment. It combines graph–text retrieval, personalized recommendation, hardware-aware LLM selection, and robotic evidence acquisition.

I. INTRODUCTION

  • Dietary risks accounted for 11 million deaths and 255 million disability-adjusted life years globally in 2017, while dietary tracking and LLM-based nutrition assistance face evidence and usability challenges.Self-reported records can be burdensome and error-prone; LLM assistants remain vulnerable to incomplete evidence, hallucinated nutrient claims, and difficulty enforcing dietary constraints without grounding.Source: §I · p. 1
  • Nutri-ATLAS targets recommendations faithful to structured nutritional knowledge and foods observed in the user’s environment, rather than only to an LLM’s parametric tendencies.The system grounds reported intake, uses dietary context to recommend against identified gaps, and acquires missing environmental evidence through embodied interaction.Source: §I · p. 1
  • The system pairs graph-based food–nutrient representations with semantic text embeddings and an embodied interface for acquiring missing environmental evidence.This responds to the difficulty of matching foods that differ in nutrient composition despite similar names, or have similar nutritional profiles despite different vocabulary.Source: §I · pp. 1–2
  • Nutri-ATLAS learns 64-dimensional GATv2 embeddings for 9,991 foods and 82,238 recipes, alongside complementary 1,024-dimensional text embeddings.The graph embeddings represent nutritional and ingredient structure, while text embeddings capture lexical and cross-language name similarity.Source: §I · pp. 2–3
  • A four-stage hybrid graph–text retrieval pipeline supports food grounding, nutritional gap filling, substitute retrieval, and meal-level composition, while Preference Memory applies restrictions and preferences without model retraining.Dietary restrictions function as hard filters; preferences act as soft reranking signals, and the context conditions recommendation explanations.Source: §I · pp. 2–3
  • Hardware-aware selection evaluates quantized Qwen3.5-9B variants for nutrient-estimation quality and on-device performance, while the robotic skill interface gathers environmental evidence for recommendations.The evaluation includes throughput, tail latency, and energy per generated token; the robot navigates, lists landmarks, queries objects, and updates preference and food-accessibility information.Source: §I · pp. 2–3

II. RELATED WORKS

Prior work covers nutrition assessment, food knowledge graphs, and retrieval-augmented generation, but these strands often lack physically grounded availability evidence. Nutri-ATLAS combines graph–text retrieval across nutrition tasks and connects it to food-availability evidence.

A. LLM-Based Nutrition Assessment and Dietary Assistance

  • LLM nutrition tools can assist with assessment and recommendation, but evaluations report sensitivity to prompting and evidence quality, while prior systems often use text, profiles, or static databases.Nutri-ATLAS instead grounds intake in food–nutrient knowledge and updates recommendations using foods observed in the environment.Source: A. LLM-Based Nutrition Assessment and… · p. 3

B. Food Knowledge Graphs, Substitution, and Recipe Recommendation

  • FoodKG, PFoodReQ, and HealthyFoodSubs provide foundations for structured food reasoning, constrained recommendation, and graph-based substitution, while Nutri-ATLAS unifies these tasks with food-availability evidence.Its shared graph–text primitive spans nutrient grounding, gap filling, substitute retrieval, and meal composition.Source: B. Food Knowledge Graphs,… · pp. 3–4

C. Retrieval-Augmented and Graph-Grounded Generation

  • Nutri-ATLAS fuses nutrition-aware GATv2 embeddings with semantic text embeddings, enabling retrieval of foods that are textually related, nutritionally similar, or both.The shared primitive supports food nutrition extraction, gap filling, substitute retrieval, and recipe-level composition to ground LLM generation in structured nutrient evidence.Source: C. Retrieval-Augmented and… · p. 4

D. Embodied Agents, Memory, and Environmental Evidence

Nutri-ATLAS treats embodiment as a task-specific way to gather physical evidence, while dietary reasoning remains in the graph-grounded recommendation pipeline. It also frames edge deployment as selecting a model against task quality and measured hardware behavior.

D. Embodied Agents, Memory, and Environmental Evidence

E. Efficient LLM Deployment and Hardware-Aware Model Selection

  • Quantized Qwen3.5-9B variants are compared under a fixed Dense+GAT pipeline using nutrient accuracy, task-quality scoring, throughput, tail latency, and energy per generated token.Pareto analysis identifies operating points for accuracy-first, energy-first, performance-first, or balanced deployment on Jetson Orin Nano.Source: E. Efficient LLM Deployment and… · p. 5

A. Problem Setup: Embodied Agentic Nutrition Assistance

  • When its knowledge base is insufficient, the agent navigates to landmarks, queries detected objects, and updates food accessibility before responding to a request for an appropriate, preference-aware recommendation.The setup requires recommendations to be grounded in the user’s available foods.Source: A. Problem Setup: Embodied Agentic… · p. 5

B. Unified food–nutrition graph embeddings

Nutri-ATLAS represents foods and recipes in complementary graph and text embedding spaces, then uses a shared hybrid scoring primitive to support nutrition-aware retrieval. The graph is heterogeneous, with nutrient and substitution supervision and recipe and tag embeddings learned through message passing.

B. Unified food–nutrition graph embeddings

  • A two-layer GATv2 maps the heterogeneous food–nutrient–recipe–tag graph to 64-dimensional embeddings, while Qwen3-Embedding-0.6B encodes food and recipe descriptions into 1,024-dimensional vectors.The graph captures nutrient composition, ingredient cooccurrence, and substitution relationships; the text space captures lexical and semantic description similarity.Source: B. Unified food–nutrition graph… · pp. 5–6
  • The graph contains 9,991 food, 626 nutrient, 82,238 recipe, and 238 tag nodes, with USDA FoodData Central supplying food–nutrient relations and FoodKG supplying recipe and ingredient relations.Food–nutrient edges store log(1 + amount), recipe–food edges encode ingredient containment, and recipe–tag edges assign meal-type and cuisine labels.Source: B. Unified food–nutrition graph… · p. 5
  • Food–nutrient and substitution relations are supervised, while recipe and tag embeddings emerge through heterogeneous message passing.The encoder is trained with a multi-task objective on observed Food→Nutrient edges, using amount, nutrient-edge-existence, and food-substitution losses.Source: B. Unified food–nutrition graph… · p. 6
  • At inference, graph and text similarities are fused by a shared hybrid score; for a free-text query, the top text match supplies a precomputed GAT embedding as a pseudo-anchor.This allows fusion without a graph-inference call at query time.Source: B. Unified food–nutrition graph… · p. 6

C. Personalized Nutrition-Aware Retrieval and Food Recommendation

Nutri-ATLAS uses a four-stage hybrid graph-text retrieval pipeline to ground reported intake, identify nutrient gaps, retrieve substitutes, and compose meals. Preference and food-availability context then inform the generated recommendation.

C. Personalized Nutrition-Aware Retrieval and Food Recommendation

  • The first LLM call uses the meal description, grounded nutrient profile, and preference profile to identify underrepresented nutrients and emit a structured target for the next meal.Source: C. Personalized Nutrition-Aware… · p. 6
  • The four-stage pipeline grounds food mentions in USDA entries, retrieves gap-filling foods, finds nutritionally equivalent substitutes, and ranks recipes for meal composition.The stages use hybrid graph-text scoring; recipe ranking also uses ingredient-overlap and pantry-missing terms, with a five-tier relaxation cascade for nonempty dietary-compliant results.Source: C. Personalized Nutrition-Aware… · pp. 6–7
  • Stage 1 uses a text top-1 match as a pseudo-anchor to hybrid-score all 9,991 foods, then retrieves per-100g nutrients for top-K matches.Source: C. Personalized Nutrition-Aware… · pp. 6–7
  • Stage 2 ranks SQL-filtered, meal-appropriate foods using a hybrid score and structured macro-match term, with the target encoded in text and GAT spaces.The GAT query is a magnitude-weighted blend of nutrient-node embeddings; γ = 0.5.Source: C. Personalized Nutrition-Aware… · p. 7
  • Stage 3 hybrid-ranks available foods, or the full index when availability is unknown, to find substitutes that can be nutritionally similar despite having no name overlap.Source: C. Personalized Nutrition-Aware… · p. 7
  • A tracker records foods as offered and selected, while the second LLM call combines gap analysis, ranked foods and recipes, and preference context into the recommendation.Availability scores rerank candidates and default to 0.5 for foods not yet offered.Source: C. Personalized Nutrition-Aware… · p. 7

D. Hardware-aware multi-objective pareto analysis

Nutri-ATLAS frames quantized-model selection as a multi-objective deployment problem, balancing nutrition quality with throughput and energy use. Its scoring and Pareto analysis support different deployment priorities.

D. Hardware-aware multi-objective pareto analysis

  • Model selection balances nutrition accuracy, generation throughput, and energy efficiency rather than selecting a quantized LLM by nutrient-estimation accuracy alone.The Pareto analysis maximizes accuracy and throughput while minimizing energy per generated token, exposing accuracy-first, energy-first, performance-first, and balanced operating modes.Source: D. Hardware-aware multi-objective… · p. 7
  • The weighted geometric mean penalizes models that perform poorly on any single nutrient category, so one strong metric cannot dominate the nutrition score.The score uses carbohydrate, protein, fat, and energy accuracy under a fixed Dense + GAT retrieval setting.Source: D. Hardware-aware multi-objective… · p. 7
  • The Unified Accuracy Score combines nutrient estimation, graph-based substitution retrieval, and personalized recommendation quality for task-oriented model comparison.The score preserves contributions from all three components through weights whose sum is 1.Source: D. Hardware-aware multi-objective… · p. 7

E. Embodied Nutrition Assistance and System Integration

Nutri-ATLAS connects its nutrition pipeline to an LLM-guided robot skill interface that acquires environmental evidence and updates persistent context. The resulting recommendations can use foods observed in the environment alongside dietary preferences and structured nutrition knowledge.

E. Embodied Nutrition Assistance and System Integration

  • The LLM-guided skill library spans landmark listing, navigation, object querying, preference updates, food-access updates, and nutrition recommendation.The planner checks whether its knowledge is sufficient and gathers environmental evidence through skill calls when needed.Source: E. Embodied Nutrition Assistance and… · pp. 7–8
  • The integrated system uses a Qwen3.5-9B planner, robot navigation, camera-based food detection, persistent preference memory, and nutrition retrieval to ground recommendations in observed food availability.The deployed setup uses a Unitree Go2, RealSense D435, and two Jetson Orin Nano modules.Source: E. Embodied Nutrition Assistance and… · p. 7, Figure 5 · p. 8
  • Three persistent stores track landmark locations, dietary preferences and prior choices, and foods confirmed present; food availability conditions meal composition.Detections are filtered and accumulated across steps to build environmental knowledge over long-horizon interactions.Source: E. Embodied Nutrition Assistance and… · p. 8
  • A real-world interaction shows the robot navigating to inspect foods and update preference and availability stores before recommending a meal that adheres to a stated peanut allergy.The planner’s embodied skill selection is also assessed on EmbodiedBench-ALFRED, while the physical deployment tests the complete pipeline end-to-end.Source: E. Embodied Nutrition Assistance and… · p. 8

IV. NUTRI-ATLAS EVALUATION SETUP

The evaluation covers offline nutrition reasoning, simulated embodied planning, and physical-robot deployment. Its datasets, baselines, and metrics span food substitution, nutrient estimation, personalized recommendation, patient constraints, and embodied task execution.

IV. NUTRI-ATLAS EVALUATION SETUP

  • The evaluation measures learned nutrition-aware representations, retrieval, food grounding, personalized recommendations, simulated skill execution, and feasibility of physical deployment.Source: §IV · p. 8

USDA FoodData Central and Nutri-ATLAS knowledge

  • The knowledge base combines 9,991 canonical foods and 626 nutrients with 82,238 FoodKG recipes, 238 tags, and 752,951 recipe-ingredient links.The graph has 93,093 nodes across food, nutrient, recipe, and tag types, with Food→Nutrient and substitution supervision.Source: USDA FoodData Central and Nutri-ATLAS… · p. 9
  • HealthyFoodSubs tests within-category substitute ranking, NutriBench v2 tests nutrient estimation on 15,617 meal descriptions, and PFoodReQ tests personalized recipe retrieval across 2,269 queries.The patient-profile case study uses 1,000 profiles; EmbodiedBench-ALFRED covers six capability subsets with 50 episodes per subset.Source: USDA FoodData Central and Nutri-ATLAS… · p. 9

B. Baselines

  • Substitution baselines include the reproduced GAT reference and Nutri-ATLAS GAT, text, and hybrid variants; recommendation is compared with BAMnet, and embodied planning with memory-augmented and external planners.Dense + GAT retrieval expands dense USDA candidates with GAT neighbors, retaining matches above cosine similarity 0.60.Source: B. Baselines · p. 9

C. Metrics

  • Metrics include substitution MAP and RR@5/RR@10, nutrient-estimation accuracy and MAE, recommendation MAP/MAR/F1, patient allergen safety and carbohydrate adherence, and embodied task success rate.Real-world evaluation additionally reports navigation success, object inspection success, end-to-end completion, and representative failure modes.Source: C. Metrics · pp. 9–10
  • The real-world workflow depicts a request involving a peanut allergy followed by navigation, food inspection, memory updates, and a grounded recommendation.Source: Figure 6 · p. 10

D. Implementation Details

Nutri-ATLAS combines graph and dense retrieval with quantized LLMs, then connects its nutrition pipeline to a robot through persistent stores and typed skills.

D. Implementation Details

  • The implementation evaluates nine quantized Qwen3.5-9B variants, a two-layer GATv2 encoder, and joint training on nutrient-edge, nutrient-amount, and substitution tasks.The graph encoder produces 64-dimensional embeddings; its losses cover nutrient-edge existence, nutrient amounts, and food substitution.Source: D. Implementation Details · p. 10
  • For nutrition estimation, Dense+GAT retrieval expands dense top-K matches with graph neighbors; substitution and recipe candidates use GAT, text, or equally weighted hybrid similarity.The described system uses USDA-based references in LLM prompts and profiles quantized variants on Jetson Orin Nano for throughput and energy per token.Source: D. Implementation Details · p. 11

E. Robot Platform and Environment Setup

  • The robot maintains landmark, preference, and session food-accessibility stores, with typed skills for navigation, object retrieval, and memory updates.Food recommendation conditions meal composition on current observed availability; the example workflow inspects foods and returns an allergen-safe recommendation using items present.Source: E. Robot Platform and Environment Setup · p. 11
  • The physical workflow navigates to inspect available foods, updates preference and accessibility memory, and composes a meal from foods actually present.The described example is a light, healthy meal request with a reported peanut allergy.Source: E. Robot Platform and Environment Setup · p. 11

V. NUTRI-ATLAS EVALUATION RESULTS

Nutri-ATLAS evaluates graph-based substitution retrieval and personalized meal recommendations, including clinical cases checked against carbohydrate targets and allergen restrictions.

A. Unified Nutrition Graph Knowledge Base Results

B. Personalized and Clinical Food Recommendation Results

1) Clinical Meal Recommendation:

  • Across five patient cases, diabetic patients received meals with 4–6% of energy from carbohydrate, other patients received 52–62%, and all peanut- or gluten-allergic patients received allergen-free recipes.The measured scope excludes sodium and sugar adherence because neither the recipe corpus nor patient records store those values.Source: 1) Clinical Meal Recommendation · p. 12, Table II · p. 12

2) Personalized Food Recommendation Evaluation:

Nutri-ATLAS reports personalized recipe-recommendation results and selects quantized LLM configurations by balancing nutrition accuracy, throughput, and energy.

2) Personalized Food Recommendation Evaluation:

  • 78.8% MAP, 83.0% MAR, and 77.5% F1 on PFoodReQ surpass the baseline by 16.1, 21.2, and 13.8 percentage points, respectively.The study attributes recipe retrieval to combined graph and text similarity with ingredient and nutrient constraints.Source: 2) Personalized Food Recommendation… · p. 13

C. Hardware-Aware LLM selection Results

  • No configuration dominates accuracy, throughput, and energy: Q4-level models favor accuracy, while smaller Q2-level models favor higher throughput or lower energy.The Pareto frontier supports selecting an accuracy-first Q4-level model or an energy-first or performance-first Q2-level model.Source: Figure 7 · p. 13, C. Hardware-Aware LLM selection Results · p. 13
  • Config 2, an energy-first Q2-level model, is used for embodied deployment; Jetson profiling shows utilization and power rise during generation and return to idle between requests.The reported trace supplies throughput and energy-per-token measurements used in hardware-aware selection.Source: C. Hardware-Aware LLM selection Results · p. 13, Figure 8 · p. 13

D. Fully Integrated Robotic Nutrition System Results

The integrated system is tested in simulated embodied tasks and physical home-like deployment, where it navigates, inspects foods, and grounds recommendations in observed availability.

D. Fully Integrated Robotic Nutrition System Results

  • On EmbodiedBench-ALFRED, persistent observed-object memory improves success on recall-dependent subtasks while remaining comparable on single-observation tasks.The comparison covers two quantized backbones over 50 episodes per subtask and includes larger open-source MLLM references.Source: D. Fully Integrated Robotic Nutrition… · p. 13, Figure 9 · p. 14
  • The Config 2 planner handles the Easy physical benchmark tier near-perfectly and degrades on harder tiers, with availability checking the dominant failure mode.The nine-prompt benchmark adds navigation and availability checks at Medium difficulty, then allergen or condition constraints at Hard difficulty.Source: D. Fully Integrated Robotic Nutrition… · p. 14, Table VI · p. 14, Table V · p. 14

VI. CONCLUSION

  • A physical-robot episode grounds prior intake, identifies a nutritional gap, inspects available foods, updates memory, and returns a recommendation adhering to a peanut allergy.The conclusion describes the system as closing the loop between dietary reasoning and foods accessible to the user.Source: D. Fully Integrated Robotic Nutrition… · p. 14, §VI · p. 15

Related papers

Loading 2609.32803v1…