NutriBench-Kitchen: Benchmarking Embodied AI for Nutrition Management

1 Introduction

• Embodied Nutrition Management is the joint ability to derive evolving food states from continuous visual observations and use them for recipe queries and constraint-aware planning.Earlier events may change available inventory even after the ingredient or action is no longer visible; recipe and nutritional knowledge determine whether the resulting state satisfies a request.Source: §1 · pp. 2–3
• NutriBench-Kitchen contains 1,500 manually verified question–answer pairs from 160 cooking sequences across five task families and disjoint Easy and Hard regimes.The regimes differ in whether hints to relevant evidence are provided.Source: §1 · p. 3
• The benchmark tests whether models combine observations over time, propagate ingredient-use changes, and compare inferred inventory with recipe requirements and dietary constraints.Some final states are not shown explicitly and must be inferred from multiple intermediate changes.Source: §1 · p. 3
• Experiments reveal a gulf between current large vision-language models and human performance, while Nutri-Vgent consistently improves its underlying models, especially on state-dependent questions.Errors cluster in quantitative ingredient estimation, long-range state maintenance, and reasoning over interacting inventory and dietary constraints.Source: §1 · pp. 3–4
• Nutri-Vgent separates episodic observations, current food state, and recipe knowledge into distinct memories for long-video reasoning.Its largest gains occur on questions involving earlier ingredient use, accumulated state changes, or multiple interacting constraints.Source: §1 · p. 4

2.1 Nutrition Management in Embodied AI

• Static food resources capture attributes, ingredients, or recipe knowledge, while egocentric datasets capture kitchen activity; neither link visual events to updates in nutrition-relevant kitchen states.The missing connection is determining what changed, what remains, and which decisions remain feasible.Source: §2.1 · p. 4

2.2 Evaluation and Benchmarks

• Existing benchmarks usually cover only one or two lifecycle capabilities, rather than the full loop from video-grounded state updates to constrained recipe, nutrition, and planning decisions.Table 1 defines coverage as an explicit task, annotation target, or evaluation objective, not merely something inferable from raw data.Source: §2.2 · p. 5, Table 1 · p. 5

3.1 Task Scope and Formulation

• The task requires inferring nutrition-relevant kitchen states from long-form observations, maintaining them, and using them to answer recipe and planning questions.Answers may depend on earlier changes that are no longer visible, including ingredient use, and must reconcile the resulting state with recipe or dietary constraints.Source: §3.1 · pp. 5–6
• The benchmark is observational: planning questions assess understanding and decision reasoning, not physical action or closed-loop control.The model receives no simulator feedback and does not change the environment.Source: §3.1 · p. 6

3.2 NutriBench-Kitchen Benchmark Construction

• The benchmark contains 1,500 manually verified question–answer pairs from 160 long-form videos, combining 145 HD-EPIC segments with 15 self-recorded videos.HD-EPIC contributes naturally occurring egocentric activity; self-recorded videos provide controlled state changes with measured quantities and nutritional values.Source: §3.2 · p. 6
• Video processing samples evidence densely around ingredient-changing actions, removes near-duplicate frames, and uses available audio or transcripts as auxiliary annotation context.Visual evidence remains the primary grounding source for questions.Source: §3.2 · p. 6
• A Recipe Knowledge Graph built from more than 10,000 recipes links ingredients, tools, steps, nutrition, and substitutions for knowledge-grounded questions.Ingredient aliases are canonicalized, and recipe relations are resources rather than ground truth for visual events.Source: §3.2 · pp. 6–7
• Model-generated proposals are drafts; annotators verify food identities, states, temporal evidence, answers, and distractors, using measured quantities for self-recorded videos and consensus estimates for HD-EPIC.Candidate questions are filtered for consistency and checked for visual grounding, answer validity, plausible distractors, and task-label fit.Source: §3.2 · pp. 7–8, Figure 2 · p. 7

3.3 Task Families and Evaluation Regimes

• The five task families cover ingredient and quantity recognition, inventory reconstruction, constrained recipe queries, long-range procedural reasoning, and immediate next-step decisions.Long- and short-term planning assess understanding and decision reasoning rather than execution.Source: §3.3 · pp. 8–9
• Easy questions provide a local textual frame hint, while the distinct Hard questions add dialogue context and omit that hint, increasing demands on visual perception and long-range state tracking.Easy and Hard are separate benchmark regimes, not paired versions of the same questions.Source: §3.3 · p. 9

4 Nutri-Vgent

• Nutri-Vgent separates episodic experience, persistent food states, and recipe constraints instead of treating past observations as a homogeneous memory pool.Episodic Memory records temporally grounded events, Food Memory holds the current inventory, and Recipe Memory stores normalized recipe knowledge.Source: §4 · pp. 9–10, Figure 4 · p. 10
• A sliding temporal window converts nutrition-relevant actions into state-changing events, updating the food ledger even after the visual evidence disappears.Food Memory tracks canonical identity, estimated quantity, freshness, location, and timestamp; Episodic Memory preserves the temporal evidence for how the state arose.Source: §4 · p. 10
• For recipe or calorie questions, function calls retrieve dish ingredients and reference weights or candidate foods, which are checked against the food ledger before answering.The answer reconciles video-grounded state with recipe constraints.Source: §4 · p. 11

5 Experiment

• The experiments evaluate how challenging the benchmark is, whether hybrid memory improves long-horizon nutrition management, and what challenges remain.Nutri-Vgent uses dynamically retrieved observations to update episodic and food-state memories.Source: §5 · p. 11, §5.1 · p. 11

5.1 Implementation Details

• The evaluation uses zero-shot baselines with a shared chain-of-thought protocol and reports multiple-choice accuracy and numerical Mean Relative Accuracy across Easy and Hard regimes.The overall score is the unweighted mean across task–regime cells; MRA rewards numerical predictions that meet progressively stricter relative-error tolerances.Source: §5.1 · pp. 11–12

5.2 Main Results

• Human performance averages about 92.0%, while random and frequency baselines average about 13.4% and 19.1%; frequency-based guessing collapses on Hard splits.General-purpose models also struggle with persistent inventory tracking: GPT-4o averages 47.7% and scores 26.7% on Memory Management Easy.Source: §5.2 · pp. 12–13
• Nutri-Vgent achieves an average score of 62.3%, outperforming its base model (53.4%), GLM-4V-9B (50.8%), and GPT-4o (47.7%), while matching Gemini 2.5 Pro (61.1%).Source: §5.2 · p. 13
• Nutri-Vgent rises from 63.4% to 94.1% in Recipe Query and from 14.0% to 73.3% in Ingredient Entry on complex, multi-turn splits.Its Ingredient Entry score also exceeds standard Vgent’s 24.7% peak.Source: §5.2 · p. 14
• Nutri-Vgent’s retrieval and strict knowledge-graph checks can reduce performance on simple Easy tasks, including Ingredient Entry dropping from 30.7% to 20.0%.The paper attributes this trade-off to retrieval overhead and occasional overcorrection in straightforward, fully visible scenes.Source: §5.2 · p. 14

5.3 Ablation Studies

• Full episodic-plus-food memory achieves the highest overall scores—62.1% for Qwen3-VL-4B and 62.3% for 8B—while either memory alone lowers the 8B average from 53.4%.The hybrid 8B model raises Recipe Query from 67.05% to 76.05% and Short-Term Planning from 53.00% to 72.30% versus visual-only input.Source: Table 3 · p. 13, §5.3 · pp. 14–15
• Future work targets multi-agent collaboration and personalized, dynamically updating profiles for health goals and taste preferences.The current framework tracks objective food states and recipe constraints.Source: §5.3 · p. 15

• The paper formalizes Embodied Nutrition Management to connect continuous visual perception with complex dietary reasoning.Source: §6 · p. 15
• NutriBench-Kitchen contains 1,500 manually verified question–answer pairs from untrimmed cooking videos and assesses dynamic inventory tracking, dietary constraints, and long-horizon planning across five dimensions.Source: §6 · p. 15
• Nutri-Vgent decouples episodic observations, persistent food states, and recipe knowledge graphs in a structured memory architecture.Experiments report that it mitigates temporal forgetting and outperforms baselines, particularly in persistent state tracking and constraint satisfaction.Source: §6 · p. 15