NutriBench: A Dataset for Evaluating Large Language Models on Nutrition Estimation from Meal Descriptions

1 INTRODUCTION

• Carbohydrate-estimation errors can contribute to hyperglycemia or hypoglycemia, which may lead to serious short- and long-term complications.People with diabetes use meal carbohydrate estimates to determine insulin doses.Source: §1 · p. 1
• The benchmark addresses a gap: nutrition datasets commonly use tables or meal images, while no dataset was available to evaluate LLMs on natural-language meal descriptions.Everyday-language descriptions can include meal components and serving sizes without real-time input constraints.Source: §1 · pp. 1–2
• NutriBench contains 11,857 human-verified natural-language meal descriptions from real-world intake data across 11 countries, annotated with macronutrients and calories.The benchmark includes carbohydrates, proteins, fats, and calories.Source: §1 · pp. 2–3
• Most LLMs provided more accurate and faster carbohydrate estimates than the nutritionists, while GPT-4o with CoT achieved the highest accuracy at 66.82%.GPT-4o with CoT answered 99.16% of queries; the evaluation also simulated blood-glucose effects across 44,800 simulations, finding LLM estimates could help people with type 1 diabetes maintain safe glucose levels.Source: §1 · p. 2
• The authors evaluate twelve LLMs on carbohydrate estimation using multiple prompting strategies, including Chain-of-Thought and Retrieval-Augmented Generation.The evaluation also includes a study comparing LLM estimates with those of three professional nutritionists.Source: §1 · p. 3

2 RELATED WORK

• Tabular and image-based nutrition resources dominate prior work, but precise searches, multi-item meals, real-time capture, and obscured food components constrain their use.Tabular retrieval can require multiple searches for meals with multiple items; image-based methods may raise privacy concerns.Source: §2 · p. 3
• Earlier natural-language meal-description work matched food items to tabular databases, but its data were not public, preventing evaluation of current LLMs.The authors introduce NutriBench as a publicly available benchmark for this task.Source: §2 · p. 4
• LLMs can hallucinate or provide incorrect or outdated information, while CoT and RAG offer reasoning and retrieved context that had not been assessed for nutrition estimation.CoT prompts step-by-step reasoning; RAG supplies query-relevant information from an external source.Source: §2 · p. 4
• The paper evaluates twelve LLMs with standard, CoT, RAG, and combined RAG-plus-CoT prompting for carbohydrate estimation from natural-language meal descriptions.The authors analyze how prompting strategies affect performance.Source: §2 · p. 4

3 NUTRIBENCH CONSTRUCTION

• NUTRIBENCH combines real-world dietary intake from WWEIA and FAO/WHO GIFT across 11 countries to build a natural-language meal-description benchmark.The descriptions are paired with nutrition labels.Source: §3 · p. 4

3.1 WHAT WE EAT IN AMERICA

• WWEIA dietary recalls provide food names and portions; FNDDS supplies nutrient values, and the 2017–2018 data serve as the reference for cleaning and normalizing records.Entries with missing or invalid information and foods absent from the reference data are excluded.Source: §3.1 · p. 4

3.2 FAO/WHO GIFT

• FAO/WHO GIFT meals from 10 countries are converted from grams to everyday serving units when FDC mappings are available, while unmappable items retain gram amounts.Food items consumed by one person on the same day and occasion are combined into meals; the WWEIA subset has 5,532 descriptions in each of natural and metric servings.Source: §3.2 · p. 5

3.3 GENERATING NATURAL LANGUAGE MEAL DESCRIPTIONS

• GPT-4o-mini generates candidate descriptions, and human verification corrects omitted or inaccurate food names and serving sizes.The verifier checks that descriptions match the meal’s food items and nutrition labels.Source: §3.3 · p. 5

• The evaluation covers 12 open- and closed-source LLMs, including a medical-domain model, on carbohydrate estimation from meal descriptions.Models include Llama, Gemma, Qwen, GPT, and OpenBioLLM variants.Source: §4 · p. 6
• Four prompting approaches are tested: a basic estimate, Chain-of-Thought reasoning, Retrieval-Augmented Generation, and a combination of retrieval and reasoning.RAG retrieves nutrition information for parsed food components and supplies it as context to the model.Source: §4 · p. 6
• RAG and RAG+CoT are evaluated only on GPT-4o-mini and two Llama 3.1 models because of computation constraints.The prompts and hyperparameters are provided in the appendix.Source: §4 · p. 6
• The evaluation reports MAE, Acc@7.5, and answer rate, with an estimate counted correct when it is within ±7.5 g of the ground truth.Models may abstain with “-1” when uncertain; Acc@7.5 uses a conservative threshold motivated by insulin-to-carbohydrate ratios.Source: §4 · p. 6

5 RESULTS

• 66.82% accuracy with a 99.16% answer rate makes GPT-4o with CoT the top model-method pair; Llama 3.1-405B-FP8 with RAG+CoT leads open models at 59.89% accuracy and 96.05% answer rate.Source: §5 · p. 6

5.1 IMPACT OF SERVING DESCRIPTIONS, PROMPTING STRATEGIES, AND RAG

• Most models have higher MAE on natural than metric servings, while GPT-4o and GPT-4o mini perform better with metric servings.The authors hypothesize that metric-serving nutrition information may occur in GPT-family training data.Source: §5.1 · p. 7, Figure 4 · p. 7
• CoT improves answer rate and accuracy across models and mitigates the rising MAE seen with more food items under Base prompting.GPT-4o mini with CoT also exceeds GPT-4o with Base prompting on both accuracy and answer rate.Source: §5.1 · p. 7, Table 2 · p. 7, Figure 5 · p. 7
• RAG and RAG+CoT improve Llama 3.1 performance on metric-serving queries, but RAG offers only minor improvement for GPT-4o-mini and RAG+CoT performs worse than CoT.The authors suggest noisy retrieval may misguide GPT-4o-mini, which may already have relevant metric-serving information from training.Source: §5.1 · pp. 7–8, Figure 6 · p. 8

5.2 VARIATION IN LLM PERFORMANCE ACROSS DIETS AND CULTURES

• GPT-4o with CoT has higher MAE for higher-carbohydrate single-item meals; across countries, its MAE ranges from 2.20 for Nigeria to 15.12 for Sri Lanka.The authors report a correlation between countries’ average meal carbohydrate content and MAE, and note disparities across cultural contexts.Source: §5.2 · p. 8, Figure 7 · p. 8

5.3 FINE-TUNING WITH FDC DATA

• Fine-tuning Gemma2-27B on FDC-derived meal descriptions reduces MAE and raises accuracy and answer rate, but the resulting model still underperforms GPT-4o with CoT.Training uses 600K generated meal descriptions and qLoRA for one epoch.Source: §5.3 · pp. 8–9, Table 3 · p. 9

• Several LLMs provide more accurate estimates than nutritionists on the same 72 queries, while GPT-4o-mini answers them in 2 minutes versus nutritionists’ average 43 minutes.The comparison used sampled U.S. meals, and participants were asked not to consult nutrition apps or the web.Source: §6 · p. 9
• Nutritionists perform better on simpler traditional meals without specific brand information, and lookup resources bring their results to levels comparable with GPT models.Source: §6 · p. 9

7 REAL-WORLD RISK ASSESSMENT

• 20 virtual patients were simulated using insulin sensitivity, carb-insulin ratio, and basal-rate parameters derived from anonymized Tidepool data.Each simulation began with a meal, and glucose response depended on its actual carbohydrate content.Source: §7 · pp. 9–10
• The simulations compare manual insulin dosing using estimated carbs divided by CIR against pump dosing adjusted after estimated carbs are entered.The pump runs the FDA-cleared Loop algorithm; the metabolism model responds to carbohydrate or insulin intake.Source: §7 · pp. 9–10
• GPT-4o with CoT carbohydrate estimates yield the highest %TIR, lowest %TBR, and lowest BGRI in the pump simulations.The supplied table caption reports the comparison but does not provide the underlying values.Source: Table 4 · p. 10, §7 · p. 10
• Figure 8 distinguishes safe glucose (70–180 mg/dl), low glucose (<70 mg/dl), and high glucose (>180 mg/dl) in green, red, and yellow, respectively.The figure caption describes simulated traces for a virtual patient using a pump, comparing carbohydrate estimates from GPT-4o and nutritionists.Source: Figure 8 · p. 10
• The authors describe the case study as indicating that LLMs can be valuable in real-life healthcare settings to help prevent risky outcomes and promote better health.Source: §7 · p. 10

8 CONCLUSION

• NUTRIBENCH is a public benchmark with 11,857 human-verified meal descriptions from real-world dietary data across 11 countries; GPT-4o with CoT surpassed professional nutritionists in accuracy and speed.Source: §8 · p. 10

A MORE EXAMPLES OF GENERATED MEAL DESCRIPTIONS

• The examples show natural-language meals derived from WWEIA and FAO/WHO GIFT, including foods, portions, and carbohydrate, energy, protein, and fat labels.Examples range from a WWEIA wine snack to GIFT meals featuring lentils, quince, millet bread, guava, and mango.Source: §A.1 · p. 15, §A.2 · p. 15

B DETAILS ON MEAL DESCRIPTION GENERATION

• Meal-description generation uses GPT-4o-mini to produce five descriptions per food item, selects one at random, and then refines it before practical use.The examples of human verification correct missing foods, serving sizes, and a wrong serving size.Source: §B.2 · pp. 16–18

C.1 HYPERPARAMETERS.

• For open-source models, generation uses temperature 0.6 and top p 0.9; GPT models use temperature 0.1 and top p 0.1.Source: §C.1 · p. 18

C.2 PROMPTS FOR CARBOHYDRATES ESTIMATION

• For RAG, the method parses a meal into food components, retrieves nutrition information for each, and provides the retrieved context with the meal to the model.Base and CoT instead query the model with the meal description and instructions; prompts for other models use their templates with the same content.Source: §C.2 · p. 18

D CONSTRUCTION OF RAG DATABASE

• RETRI-DB combines FDC entries with identical food descriptions, filters entries without carbohydrate data, removes defined extreme outliers, and uses the remaining median as the label.The resulting database has 450,182 food items with carbohydrate labels, including 13,048 with natural serving sizes; nutrition context is converted to natural-language strings.Source: D CONSTRUCTION OF RAG DATABASE · p. 18

E CONSTRUCTION OF FINE-TUNING DATA

• Fine-tuning data combines 158,246 GPT-generated meal descriptions with 450,182 rule-based descriptions based on FDC food items.GPT-4o-mini generated descriptions for a 98K-food-item sample, and randomly selected groups of 2 to 4 descriptions were combined into a meal.Source: E CONSTRUCTION OF FINE-TUNING DATA · p. 18

F RULE-BASED ALGORITHM FOR OBTAINING NATURAL SERVING UNITS IN THE FAO/WHO GIFT DATA

• FAO/WHO GIFT foods are mapped to FDC items using nearest neighbors in the text-embedding-3-large space with a 0.5 similarity threshold, which achieved 85.5% accuracy on 100 inspected items.A natural serving within 10% of the gram amount is selected; otherwise, the method uses the closest larger serving and a multiple, or smaller fractional and volume-converted servings when needed.Source: F RULE-BASED ALGORITHM FOR OBTAINING… · pp. 19–20

G ADDITIONAL RESULTS ON FINE-TUNING WITH FDC DATA.

• Larger LLaMA 3.1 models generally perform better in the fine-tuning ablation, while Gemma2 consistently outperforms LLaMA 3.1.Fine-tuning was limited by computational constraints to GPT-generated data: 39,745 metric samples and 19,745 natural samples; Table 5 compares MAE and Acc@7.5 across model sizes.Source: G ADDITIONAL RESULTS ON FINE-TUNING… · p. 20, Table 5 · p. 20

H.1 EXAMPLE 1

• One WWEIA-derived example describes a McDonald’s double cheeseburger with a vanilla soft-serve cone, labeled 112.96g carbohydrates, 957.45kcal, 39.45g protein, and 38.86g fat.Source: §H.1 · p. 20

H.2 EXAMPLE 2

• For RAG carbohydrate estimation, the GPT-4o-mini prompt requests total carbohydrate grams, assumes a standard serving when a meal item lacks a serving size, and allows -1 when the answer is unknown.A FAO/WHO GIFT example combines 30g biscuits with 26g sweet bars and 23g peanuts; its labels include 48.9g carbohydrates.Source: §H.2 · pp. 20–21

I ANALYSIS OF HIGH CARB VS. LOW CARB MEALS

• High-carb meals had carbohydrate standard deviation 26.88g versus 2.01g for low-carb meals (P < 0.05), a variability difference the authors say likely contributes to higher errors.High-carb meals exceed the third quartile (33.93g; N=869), while low-carb meals fall below the first quartile (6.47g; N=871).Source: I ANALYSIS OF HIGH CARB VS. LOW CARB… · p. 22
• Reported average portion weights were 95.27 ± 139.89 g for high-carb meals and 226.80 ± 161.73 g for low-carb meals (P < 0.05).The authors suggest larger portions could increase estimation complexity and contribute to higher errors.Source: I ANALYSIS OF HIGH CARB VS. LOW CARB… · p. 22

J ERROR ANALYSIS

• In all 100 sampled queries with absolute error >50g, GPT-4o parsed components and serving sizes, formulated equations, and calculated correctly, but misestimated at least one food item.These item-level estimates led to the overall high errors.Source: J ERROR ANALYSIS · p. 22
• High-error meals averaged 2.13 ± 0.93 food items and 64.58 ± 38.69g carbohydrates, versus 1.81 ± 0.83 items and 24.13 ± 25.27g for low-error meals (P <0.05).Reported portion weights were 268.37 ± 250.08g for high-error meals and 437.28 ± 301.25g for low-error meals (P <0.05).Source: J ERROR ANALYSIS · p. 23
• Meal complexity, greater carbohydrate content, and larger portion sizes are identified as challenges for carbohydrate estimation that future work should address.The comparison found more food items and higher carbohydrate content in high-error meals; the reported portion-weight means were 268.37 ± 250.08g for high-error meals and 437.28 ± 301.25g for low-error meals.Source: J ERROR ANALYSIS · p. 23

K EVALUATING WITH PARAPHRASED PROMPTS

• The robustness test paraphrased instructions while retaining output formats and demonstrations to test whether the model handled rephrased prompts.This setup kept expected outputs consistent across prompt phrasings.Source: K EVALUATING WITH PARAPHRASED PROMPTS · p. 23
• Paraphrasing instructions had minimal impact on GPT-4o-mini’s results, while prompts still needed defined structure, clear output formats, and demonstrations.Table 9 summarizes performance across the different instructions.Source: K EVALUATING WITH PARAPHRASED PROMPTS · p. 23, Table 9 · p. 23

GPT VS. NUTRITIONISTS

• Among 72 meal descriptions, GPT outperformed all nutritionists on 20, while all nutritionists outperformed GPT on 8.The remaining descriptions are not characterized in this comparison.Source: L MORE ANALYSIS ABOUT GPT VS.… · p. 24
• For a complex Burger King breakfast with detailed components, GPT’s MAE was 6.09 versus the lowest nutritionist MAE of 10.09; for macaroni with cheese sauce, GPT’s MAE was 20.84 versus the highest nutritionist MAE of 10.46.The paper describes the first meal as a case where GPT excels and the second as one where nutritionists perform better.Source: L MORE ANALYSIS ABOUT GPT VS.… · p. 24

GPT VS. NUTRITIONNISTS

• Because the nutritionists supplied only carbohydrate estimates, their knowledge of specific meal descriptions was difficult to evaluate directly.The analysis instead examined variance in their estimates.Source: L MORE ANALYSIS ABOUT GPT VS.… · p. 24

GPT VS. NUTRITIONISTS

• For the top 10 high-variance descriptions, GPT’s MAE was 18.9 versus the lowest nutritionist MAE of 34.4; for low-variance descriptions, GPT’s MAE was 4.6 and nutritionists’ MAEs ranged from 3.4 to 4.5.The authors suggest GPT performs better where nutritionists disagree more, possibly reflecting knowledge gaps or unfamiliarity with meals.Source: L MORE ANALYSIS ABOUT GPT VS.… · p. 24
• The follow-up allowed nutritionists to look up food items because they said such tools may be part of their usual workflow, while avoiding direct access to the source database had motivated the initial restriction.The follow-up was intended to make the comparison fairer to their standard methods.Source: L MORE ANALYSIS ABOUT GPT VS.… · p. 24
• With lookups allowed, the one nutritionist result was comparable to CoT GPT-4o; the top-10 high-variance-description MAE fell from 36.7 to 21.4, although GPT still performed better overall.Table 10 reports performance metrics for the lookup condition.Source: L MORE ANALYSIS ABOUT GPT VS.… · p. 24, Table 10 · p. 24