PaperlayerView exact revision on arXiv

Source-linked AI summary

FoodSky: A Food-oriented Large Language Model that Passes the Chef and Dietetic Examination

Pengfei Zhou, Weiqing Min, Chaoran Fu, Ying Jin, Mingyu Huang, Xiangyang Li, Shuhuan Mei, Shuqiang Jiang

arXiv:2406.10261v1cs.CLcs.AI

TL;DR

Culinary and dietary tasks are complex and culturally varied, while comprehensive food-specific data and reliable LLMs remain limited. FoodSky combines the FoodEarth corpus with TS3M and HTRAG; it reports 67.2% accuracy on the Chinese National Chef Exam and 66.4% on the National Dietetic Exam.

  • Problem

    Culinary and dietary tasks interact with culture and chronic disease, while food-specific LLMs and comprehensive food data remain limited.

    Source: §I · pp. 1–2

  • Method

    FoodSky uses the 811K-instruction FoodEarth corpus, TS3M for fine-grained food semantics, and HTRAG for context-aware responses.

    Source: §I · p. 2

  • Results

    67.2% zero-shot accuracy on the Chinese National Chef Exam and 66.4% on the National Dietetic Exam, with reported outperformance of general-purpose LLMs on both.

    Source: §I · p. 3

  • Takeaways & Limitations

    FoodSky aims to enhance culinary creativity and promote dietary health through advice for culinary and dietetic questions.

    Source: §I · p. 3

Abstract

from arXiv · show

Food is foundational to human life, serving not only as a source of nourishment but also as a cornerstone of cultural identity and social interaction. As the complexity of global dietary needs and preferences grows, food intelligence is needed to enable food perception and reasoning for various tasks, ranging from recipe generation and dietary recommendation to diet-disease correlation discovery and understanding. Towards this goal, for powerful capabilities across various domains and tasks in Large Language Models (LLMs), we introduce Food-oriented LLM FoodSky to comprehend food data through perception and reasoning. Considering the complexity and typicality of Chinese cuisine, we first construct one comprehensive Chinese food corpus FoodEarth from various authoritative sources, which can be leveraged by FoodSky to achieve deep understanding of food-related data. We then propose Topic-based Selective State Space Model (TS3M) and the Hierarchical Topic Retrieval Augmented Generation (HTRAG) mechanism to enhance FoodSky in capturing fine-grained food semantics and generating context-aware food-relevant text, respectively. Our extensive evaluations demonstrate that FoodSky significantly outperforms general-purpose LLMs in both chef and dietetic examinations, with an accuracy of 67.2% and 66.4% on the Chinese National Chef Exam and the National Dietetic Exam, respectively. FoodSky not only promises to enhance culinary creativity and promote healthier eating patterns, but also sets a new standard for domain-specific LLMs that address complex real-world issues in the food domain. An online demonstration of FoodSky is available at http://222.92.101.211:8200.

I. INTRODUCTION

FoodSky addresses the difficulty of building food-specific LLMs that can handle fine-grained culinary and dietary knowledge across diverse topics and cultures. It combines a Chinese food corpus with topic-aware modeling and retrieval, and reports results on chef and dietetic examinations.

I. INTRODUCTION

  • Culinary and dietary tasks interact with culture and chronic disease, while prior studies often treated them separately.The introduction also identifies limits in food-specific knowledge and cultural coverage in existing food LLMs.Source: §I · pp. 1–2

II. RELATED WORK

  • Prior food-assistant work includes FoodGPT, ingredient substitution and recipe recommendation, and FoodLMM’s multitask dietary assistance, but FoodGPT did not release a fully trained model.The paper positions FoodSky as a food-domain model combining culinary and nutritional understanding.Source: §I · pp. 1–2, A. Large Language Models · p. 3

I. INTRODUCTION

  • FoodSky combines an 811K-instruction FoodEarth corpus with TS3M for fine-grained food semantics and HTRAG for context-aware, cross-cultural responses.The corpus is described as sourced from authoritative materials and processed through multiple filtering methods.Source: §I · pp. 2–3
  • 67.2% zero-shot accuracy on the Chinese National Chef Exam and 66.4% on the National Dietetic Exam accompany reported outperformance of general-purpose LLMs on both exams.The comparisons include InternLM2 and ChatGPT-3.5.Source: §I · p. 3

B. Food Computing

Food computing includes dietary assistance, recommendation, and recipe retrieval, but the paper identifies a lack of comprehensive Chinese food data. FoodEarth is assembled from varied sources through filtering, annotation, and similarity-based screening.

B. Food Computing

  • Dietary assistants support cooking, nutrition, and diet-health scenarios, alongside food recommendation and recipe retrieval tasks.Related work also includes systems for food recognition, ingredient recognition, segmentation, recipe generation, and nutrition estimation.Source: B. Food Computing · pp. 3–4

III. DATASET

  • FoodEarth contains 811,491 question-answer pairs from diverse Chinese authoritative sources and is presented as a foundation for food- and nutrition-domain fine-tuning.The construction process includes data-source sorting, annotation, and similarity-based screening.Source: §III · p. 4

A. Data Source Sorting

  • Food-related data vary in credibility and may conflict because of differences in taste, measurement, regional food, and cooking methods.The authors describe using varied authoritative sources to capture cooking techniques and personalized nutrition.Source: A. Data Source Sorting · p. 4
  • FoodEarth’s sources include knowledge graph data, authoritative sources, online resources, and public Chinese instruction datasets, organized by authority and timeliness.The collected material includes nutrition and recipe knowledge-graph data and secondary sources such as professional papers and authoritative websites.Source: A. Data Source Sorting · p. 4

B. Data Annotation

  • ChatGPT helps filter food-related raw data and generate questions, while annotators review Q&A accuracy and refine inaccurate or ambiguous pairs.The workflow combines automated processing with manual verification.Source: B. Data Annotation · pp. 4–5

1) Semi-automated Data Filtering:

FoodEarth’s preparation combines manual and automated filtering with expert review, then generates and checks question-answer pairs to establish a food-domain training dataset.

1) Semi-automated Data Filtering:

  • Manual quality and relevance filtering removes noise before ChatGPT classifies text as food-related or irrelevant, followed by expert checks of relevance and trustworthiness.ChatGPT returns 1 for relevant text and 0 for irrelevant text; experts compare the filtered data with current literature and industry reports.Source: 1) Semi-automated Data Filtering · pp. 5–6

2) Semi-automated Data Annotation:

  • ChatGPT generates diverse questions from filtered text, and food-science annotators validate answers and refine or replace imprecise or ambiguous Q&A pairs.A simplified matching algorithm extracts the source paragraph corresponding to each generated question.Source: 2) Semi-automated Data Annotation · p. 6
  • The reviewed FoodEarth dataset covers a wide range of food topics and is presented as a foundation for fine-tuning a Chinese language model for complex domain-specific applications.The authors describe strict quality assurance measures for the dataset.Source: 2) Semi-automated Data Annotation · p. 7

C. Similarity-based Screening

Similarity-based screening reduces repetition in FoodEarth data, with the authors reporting improved generalization. They evaluate text-embedding methods on a small labeled set to select an encoder for filtering.

C. Similarity-based Screening

  • Reducing data redundancy through similarity screening produces higher-quality data and supports the model’s generalization.The authors describe encoding question-answer pairs and using similarity to identify duplicates.Source: C. Similarity-based Screening · p. 7

1) Mini-Test Set:

  • The mini-test set contains 506 samples, each pairing two sentences with a 0/1 label for whether they are similar.Source: 1) Mini-Test Set · p. 7

2) Embedding Methods:

3) Comparison of Different Embedding Methods:

The embedding comparison selects gte-Qwen1.5-7B-instruct for similarity filtering, while FoodSky combines topic-based modeling and retrieval with a Chinese LLaMA-2 backbone fine-tuned for food instructions.

3) Comparison of Different Embedding Methods:

4) Data Screening:

  • After a 0.9 cosine-similarity threshold proved inaccurate for text-similar pairs with different intents, intent-weighted hierarchical clustering selected representatives for expert validation, yielding 811,491 Chinese instructions.Source: 4) Data Screening · pp. 8–9

3) Comparison of Different Embedding Methods:

IV. METHOD

  • FoodSky combines TS3M, which integrates topic-relevant information, with HTRAG, which supports richer, food-relevant responses through retrieval.Source: §IV · p. 9

A. Backbone Model

  • The CLLaMA2 backbone is a Chinese LLaMA-2 model pretrained on a large Chinese corpus and fine-tuned on general-domain Chinese instructions before food-specific fine-tuning.Source: A. Backbone Model · p. 9

B. Instruction Fine-tuning

C. TS3M Classifier

TS3M combines content representations with topic-state representations to capture food-topic semantics and guide context-aware response generation. Its topic indicator token supplies a topic prior to the LLM, while training objectives include language-prediction and topic-hierarchy losses.

C. TS3M Classifier

  • TS3M uses separate content and topic-state branches to combine instruction semantics with selectively integrated topic information.The content branch refines embeddings with linear and convolutional layers; the topic-state branch uses a state space model.Source: C. TS3M Classifier · pp. 10–11
  • The state-space branch updates its latent state from prior state and current input, selectively incorporating instruction information through learned matrices.The paper describes the resulting state as aligned with the input topic and useful for food and dietary advice generation.Source: C. TS3M Classifier · p. 11
  • A gating operation integrates content and topic-state representations, and an encoder with pooling downsamples the result into a topic indicator token.The token embeddings are upsampled and concatenated with instruction embeddings as the LLM input, providing a topic prior for response generation.Source: C. TS3M Classifier · p. 11
  • Training combines next-sequence-prediction and topic-hierarchy losses, with the latter penalizing graph distance between predicted and true topics.The paper says iterative fine-tuning may emphasize language prediction earlier and topic hierarchy more as basic language handling improves.Source: C. TS3M Classifier · pp. 11–12

D. HTRAG

HTRAG augments instruction-tuned FoodSky with retrieved, topic-relevant external knowledge during inference. It uses TS3M's topic indicator to guide hierarchical retrieval, then combines retrieved information with the instruction context for response generation.

D. HTRAG

  • HTRAG integrates retrieved information during inference to provide relevant, context-specific knowledge for food-related responses.The authors state that this is intended to improve response accuracy and information richness.Source: D. HTRAG · p. 12
  • HTRAG uses TS3M's topic indicator and instruction context to retrieve relevant knowledge hierarchically from an external food knowledge base.Retrieval matches against an indexed knowledge base using cosine distance, then filters results by topic and importance.Source: D. HTRAG · p. 12
  • The retrieved documents are integrated with the original instruction representation and passed to the LLM to generate the final response.The paper says this integration supports more informative and comprehensive responses across food-related topics.Source: D. HTRAG · p. 12

A. Experimental Setup

The experiments use FoodSky variants based on 7B- and 13B-parameter backbones and compare them with representative language-model baselines. The setup also identifies the CDE accuracy table as reporting zero-shot and five-shot results across chef and dietetic exams.

1) Implementation Details:

  • FoodSky-7B and FoodSky-13B are food-specific fine-tuned variants based on 7B- and 13B-parameter backbones.The paper reports training the variants on separate eight-GPU servers and using LoRA; the last convergent checkpoint is evaluated.Source: 1) Implementation Details · p. 12

2) Baseline Methods:

  • Table III covers zero-shot and five-shot CDE accuracy for chef and dietetic exams, with mean accuracy also reported.Figure 7 provides example CDE questions and answers with English translations.Source: Table III · p. 13, Figure 7 · p. 13
  • Compared baselines include Chinese-pretrained CLLaMA2-7B, Chinese-QA-and-dialogue-optimized ChatGLM2-6B, practical-scenario InternLM2-7B, open Mistral-7B, multilingual Baichuan2-7B, and open-source Qwen-7B.Source: 2) Baseline Methods · pp. 12–13

3) Evaluation:

FoodSky leads the reported CDE comparisons in zero-shot and five-shot settings, while the evaluation also covers long-form responses and food question answering. The study uses exam accuracy alongside text-overlap, diversity, and GPT-4 judged quality measures.

3) Evaluation:

  • Evaluation includes BLEU, ROUGE, GLEU, and Distinct metrics for generated-answer overlap and diversity, plus GPT-4 subjective scores on FoodQA.Accuracy is used as the principal metric for CDE.Source: 3) Evaluation · pp. 13–14
  • The CDE benchmark uses 628 Chinese Chef Examination and 111 Dietetic Examination multiple-choice questions, with accuracy as its main metric.FoodLongConv contains 22 essay questions; FoodQA contains 25 short-answer and essay questions scored by GPT-4 for fluency, logic, professionality, and information density.Source: 3) Evaluation · pp. 13–14

1) Comparison on the CDE benchmark:

  • 67.1% total accuracy for FoodSky-13B in zero-shot CDE testing exceeded all baselines, including 57.6% for InternLM2-7B and 54.8% for ChatGPT-3.5.FoodSky-13B scored 67.2% on the Chef Exam and 66.4% on the Dietetic Exam; Table III reports the zero-shot and five-shot CDE accuracy comparisons.Source: 1) Comparison on the CDE benchmark · p. 14
  • 69.7% total accuracy for FoodSky-13B led the reported five-shot CDE results, with 69.5% on the Chef Exam and 70.9% on the Dietetic Exam.The passage reports 64.8% total accuracy for InternLM2-7B and 54.5% for ChatGPT-3.5 in this setting.Source: 1) Comparison on the CDE benchmark · p. 14

2) Comparison on Different Topic Categories:

FoodSky performs strongly across food-topic categories, food question answering, and GPT-4-judged response quality. Its results include high benchmark scores and a remaining opportunity to improve question answering with reinforcement learning from human feedback.

2) Comparison on Different Topic Categories:

  • FoodSky-13B achieves the highest accuracy across all five CDE topic categories, with an overall average accuracy of 67.1%.Its category scores are 68.3% in Dietary Science, 68.2% in Food Flavor Profiles, 67.0% in Food Safety Measures, 65.6% in Food Recipes, and 73.1% in Healthy Eating Principles.Source: 2) Comparison on Different Topic… · p. 15

3) Comparison on the Question Answering Task:

  • FoodSky-13B leads FoodQA on most reported metrics, while the authors report a better balance of response quality and diversity than InternLM2-7B and Qwen-7B.Its scores include BLEU-1 24.84%, BLEU-4 8.89%, GLEU 18.26%, ROUGE-1 34.50%, ROUGE-2 14.08%, and ROUGE-L 28.78%; CLLaMA2-7B has the highest Distinct scores, which the authors associate with uncontrolled responses.Source: 3) Comparison on the Question… · p. 15

4) Evaluation with GPT-4 Score:

  • In GPT-4's FoodQA benchmark assessment, FoodSky-7B achieved the highest fluency score, 40; InternLM2-7B and Qwen-7B scored 20%–35% across logic, professional, and informative criteria.The authors say the proposed model’s question-answering ability can be further improved by reinforcement learning with human feedback.Source: 4) Evaluation with GPT-4 Score · pp. 15–16, Figure 8 · p. 16

C. Ablation Studies

Ablations report gains from TS3M and HTRAG, while training-data experiments show that data quality matters alongside volume. Qualitative examples and the conclusion describe FoodSky’s food-domain capabilities and prospective applications.

C. Ablation Studies

  • Adding TS3M raises CDE average accuracy from 53.5% to 57.2%, and adding HTRAG yields 56.3%; using TS3M and RAG together reaches 60.5%.The combined setup also achieves the highest reported FoodQA metrics: BLEU-1 19.78%, BLEU-4 6.63%, GLEU 14.50%, and ROUGE-L 21.15%.Source: C. Ablation Studies · p. 16

D. Evaluation on Different Numbers of Training Data

  • CDE accuracy falls for both model sizes at 780K samples; the authors attribute this to added public teaching data that had not been double-checked for quality.They report removing low-quality and duplicate data for the final 811K-sample FoodEarth dataset.Source: D. Evaluation on Different Numbers of… · p. 17, Figure 9 · p. 17
  • With 811K instructions, FoodSky-13B reaches 67.2% chef-exam accuracy, 66.4% dietetic-exam accuracy, and 67.1% total CDE accuracy.The authors report that the larger model leverages additional data to surpass FoodSky-7B.Source: D. Evaluation on Different Numbers of… · p. 17

E. Qualitative Results

  • In recipe recommendations, FoodSky recognizes ways to make chicken breast more delicious, while Intern and ChatGPT miss culinary nuances and other stated requirements.For adolescent dietary education, FoodSky focuses on fast food’s direct health impacts and recommends moderation; the authors contrast this with Intern’s lengthy advice and ChatGPT’s balanced view.Source: E. Qualitative Results · p. 17

VI. CONCLUSION

  • The conclusion proposes extending FoodSky with user-feedback reinforcement learning, multimodal inputs, and more food-industry data, with prospective applications in food design, safety, and supply-chain management.The authors also report that FoodSky surpasses general-purpose LLMs in chef and nutrition exams.Source: §VI · pp. 17–18

APPENDIX

The appendix documents the study’s data sources and provides additional qualitative model comparisons across user groups.

A. Data Sources Details

  • The listed authoritative sources cover nutrition, clinical nutrition services, medical health, food culture, and academic resources such as CNKI.Table VII is described as listing each source, its content, and the corresponding citation source.Source: A. Data Sources Details · p. 18
  • The knowledge-graph data covers food nutrition and nutrition knowledge collected from authoritative websites during 2021–2022.The authors report that multiple rounds of cleaning and construction produced a detailed knowledge graph.Source: A. Data Sources Details · p. 18

B. More Qualitative Experimental Results

  • The appendix lists public Chinese instruction datasets from platforms including GitHub and Hugging Face, with food-related conversation data screened by keywords.It also describes eight figures comparing model answers to questions from groups including infants, older adults, chronic-disease patients, and pregnant women.Source: A. Data Sources Details · p. 18, B. More Qualitative Experimental Results · p. 18
Loading 2406.10261v1…