PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-tuning Language Models for Recipe Generation: A Comparative Analysis and Benchmark Study

Anneketh Vij, Changhao Liu, Rahul Anil Nair, Theodore Eugene Ho, Edward Shi, Ayan Bhowmick

arXiv:2502.02028v2cs.CLcs.AI

TL;DR

Recipe generation must balance coherent, high-quality recipes with culinary knowledge and dietary safety, particularly for allergen substitution. The paper explores recipe generation and allergen substitution with a multidimensional evaluation framework, finding trade-offs in recipe quality and limits to traditional NLP metrics alone.

  • Problem

    Recipe generation must balance coherence and creativity with culinary knowledge and dietary-restriction awareness to support safe, high-quality recipes.

    Source: §1 · p. 1

  • Method

    The study explores recipe generation and allergen substitution across model scales and uses a multidimensional framework combining recipe-specific, traditional, and LLM-based evaluation.

    Source: §1 · p. 1, §8 · p. 7

  • Results

    Large-scale models produced recipes with higher step complexity and ingredient coverage than prompt-based and RAG-assisted methods, while traditional NLP metrics alone were insufficient for assessing recipe quality.

    Source: §6 · p. 6, §8 · p. 8

  • Takeaways & Limitations

    Recipe systems must balance linguistic coherence, culinary accuracy, and safety, and allergen substitution requires more than simply replacing an ingredient.

    Source: §8 · p. 8

    The study examines comparatively smaller models trained for only 1–2 epochs, limiting its scope relative to larger models and longer training.

    Source: §9 · p. 8

Abstract

from arXiv · show

This research presents an exploration and study of the recipe generation task by fine-tuning various very small language models, with a focus on developing robust evaluation metrics and comparing across different language models the open-ended task of recipe generation. This study presents extensive experiments with multiple model architectures, ranging from T5-small (Raffel et al., 2023) and SmolLM-135M(Allal et al., 2024) to Phi-2 (Research, 2023), implementing both traditional NLP metrics and custom domain-specific evaluation metrics. Our novel evaluation framework incorporates recipe-specific metrics for assessing content quality and introduces approaches to allergen substitution. The results indicate that, while larger models generally perform better on standard metrics, the relationship between model size and recipe quality is more nuanced when considering domain-specific metrics. SmolLM-360M and SmolLM-1.7B demonstrate comparable performance despite their size difference before and after fine-tuning, while fine-tuning Phi-2 shows notable limitations in recipe generation despite its larger parameter count. The comprehensive evaluation framework and allergen substitution systems provide valuable insights for future work in recipe generation and broader NLG tasks that require domain expertise and safety considerations.

1 Introduction

Recipe generation must balance fluent, creative output with culinary expertise and dietary safety. This study compares model architectures, evaluation approaches, and allergen-substitution methods for that task.

1 Introduction

  • The study asks which fine-tuned models generate the best recipes, how to evaluate coherence and dietary safety, and how to produce high-quality allergen-free recipes.It compares architectures and combines traditional, recipe-specific, and LLM-based evaluation.Source: §1 · p. 1

2.1 Language Models in Recipe Generation

  • Earlier recipe-generation work compared BART- and BRIO-based models across mostly English and Vietnamese datasets using ROUGE; this study broadens comparisons across model sizes and architectures.The comparison ranges from SmolLM-135M to Phi-2.Source: §2.1 · p. 2

2.2 Recipe Generation Models

  • A personalized recipe model using attention became a baseline, while health-aware recipe recommendation over a knowledge graph inspired the allergen-substitution system.The cited prior work focused on recipes aligned with user preferences and alternatives considering health factors.Source: §2.2 · p. 2

2.3 Multi-modal Approaches

  • The study adopts prior evaluation metrics and recipe inputs but avoids a ground-truth-focused novel loss to avoid hindering personalization in allergen substitution.Ingredient Coverage was inspired by image-based recipe evaluation, while prior multimodal work also informed the input choices.Source: §2.3 · p. 2

2.4 Evaluation

  • Because overlap metrics can underrate creative recipes, the study combines general metrics with domain-specific measures such as Ingredient Coverage.Related approaches often rely on BLEU, ROUGE, and ingredient-matching F1.Source: §2.4 · p. 2

3 Approach

The approach uses Food.com recipes to fine-tune small and larger language models, then evaluates prompting and retrieval-based routes for allergen substitution.

3.1 Food.com Dataset

  • The RAW_recipes data pipeline extracts names, ingredients, and instructions, standardizes and tokenizes them, and formats name-and-ingredient inputs with instructions as targets.The Food.com dataset contains more than 180,000 recipes.Source: §3.1 · pp. 2–3

3.2 Exploratory Data Analysis

  • 9.66% of unique ingredients appear in 90% of recipes, while the remaining 91.44% appear in 10%.Recipe token lengths also informed context sizing: 99.4% were under 512 tokens and 90.4% under 256.Source: §3.2 · p. 3

3.3 Fine-Tuning Small Scale Models

  • A custom attention encoder-decoder scored near zero, so the study fine-tuned pretrained SmolLM, GPT-2, and T5-small models on 100,000 sampled recipes.The sample was split 80%/10%/10%, with the first 500 test examples used for evaluation.Source: §3.3 · p. 3

3.4 Fine-Tuning Larger Models

  • The 231,637-recipe dataset was split 80%/10%/10% into training, validation, and test; the larger-model stage fine-tuned SmolLM-360M, SmolLM-1.7B, and Phi-2 using QLoRA rank 8.All models trained for 1 epoch; evaluation used the first 500 test samples for both baseline and fine-tuned versions.Source: §3.4 · p. 3

3.5 Allergen Substitution

  • Allergen substitution was explored through two routes: prompting models and a retrieval-augmented system.The prompt-based and RAG-assisted approaches are detailed separately.Source: §3.5 · p. 3

3.5.1 Prompt based Allergen Substitution

  • Prompt-based substitution adds allergens to avoid alongside the recipe name and ingredients, instructing the model to suggest safe replacements while preserving flavor and texture where possible.The prompt also asks for clear, detailed cooking instructions.Source: §3.5.1 · pp. 3–4

3.5.2 RAG-assisted Allergen Substitution System

  • The RAG-assisted system parses generated recipes, checks ingredients against a custom allergen database, and replaces listed allergens with appropriate database ingredients.It uses FAISS similarity search and HuggingFace embeddings.Source: §3.5.2 · p. 4

4 Evaluation Metrics

The evaluation combines traditional language metrics, recipe-specific automatic measures, and LLM judgments. Recipe-specific measures address limitations of overlap metrics for creative outputs.

4.1 Traditional NLP Metrics

  • Traditional evaluation uses BLEU, ROUGE-1/2/L, and perplexity to assess overlap with ground truth and generated-text quality.BLEU and ROUGE compare generated recipes with the ground truth; perplexity is the exponentiated average negative log-likelihood.Source: §4 · p. 4, §4.1 · pp. 4–5

4.2 Recipe Specific Auto Evaluation Metrics

  • Because overlap metrics can score a high-quality creative recipe poorly, the study adds domain-specific measures of ingredient coverage, step complexity, coherence, and cooking time/temperature.The metrics track ingredient use, instruction detail, step ordering and timing, and cooking-parameter completeness; scores range from 0 to 1.Source: §4.2 · p. 5

4.3 LLM-As-A-Judge

  • An LLM-as-a-judge evaluates baseline and fine-tuned recipes across six Likert categories, scoring each from 1 to 5.The study switched from Qwen2.5-1.5B Instruct to Qwen2.5-7B for more accurate scores.Source: §4.3 · p. 5

2. Completeness: Coverage of necessary steps

The section compares small- and large-scale model outputs, then describes prompt-based and RAG-assisted allergen-substitution evaluations using domain-specific metrics and judge assessments.

5. Relevance: Alignment with recipe goals

  • The allergen-safety metric checks whether the allergen is substituted correctly.Source: §5 · p. 5

5.1 Initial Results with Small-Scale Models

  • Table 1 compares recipes generated by small-scale models using BLEU and ROUGE, without a winner reported in the supplied text.Source: §5.1 · p. 5, Table 1 · p. 5

5.2 Results with Large-Scale Models

  • Large-model BLEU and ROUGE scores are low because generated recipes have little overlap with the ground truth, motivating domain-specific evaluation alongside traditional metrics.Tables 2 and 3 compare baseline and fine-tuned versions using traditional and domain-specific auto-evaluation metrics.Source: §5.2 · p. 5, Table 2 · p. 6, Table 3 · p. 6

5.3 Results of Prompt-based Allergy Substitution

  • Prompt-based allergen substitution is evaluated with domain-specific metrics and Qwen2.5-7B judgments on the first 500 test samples.Tables 4 and 5 report these evaluations for baseline and fine-tuned models.Source: §5.3 · pp. 5–6, Table 4 · p. 6

5.4 Results of RAG-Assisted Allergy Substitution

  • RAG-assisted allergen substitution is evaluated with domain-specific metrics and Qwen2.5-7B judgments, with judge evaluation on the first 500 test samples.These results are reported in Tables 6 and 7.Source: §5.4 · p. 6

6 Discussion

This discussion compares recipe generation and allergen substitution across model scales, finding that substitution-related quality trade-offs complicate simple scaling assumptions. It emphasizes domain-specific evaluation and adaptation that protects recipe quality alongside allergen safety.

6 Discussion

  • Large-scale models’ normal recipes had higher step complexity and ingredient coverage than prompt-based and RAG-assisted generations; allergen-substituted recipes gained marginal coherence.Source: §6 · p. 6

2. Fine-tuning Dynamics: The most interesting

  • Phi-2 fine-tuning raised step complexity from 0.82 to 0.99 but degraded the other three metrics, including ingredient coverage and temperature specification.Source: §2 · p. 7
  • Fine-tuned SmolLM-360M and SmolLM-1.7B scored 2.57 and 2.54 in allergen safety, respectively, at the cost of recipe coherence.Source: §2 · p. 7
  • Across all three models, RAG-assisted substitution scored higher on step complexity and temperature/time specification but lower on ingredient coverage than prompt-based; prompt-based led every LLM-judge metric.Source: §2 · p. 7

7 Future Work

  • Future work proposes capability-preserving adaptation, culinary pretraining, joint recipe-quality and safety learning, broader datasets, better metrics, and richer validated knowledge for RAG substitution.Source: §7 · p. 7

8 Conclusion

  • Allergen-substituted recipes showed a comparatively lower trade-off between step complexity and other metrics, but ingredient substitution alone did not ensure high-quality recipes.Source: §8 · pp. 7–8
  • Traditional NLP metrics alone were insufficient; recipe systems need domain-specific measures and must balance coherence, culinary accuracy, and safety.Source: §8 · p. 8

9 Limitations

The study is constrained by compute, evaluation design, and language coverage: it uses smaller models trained for 1–2 epochs, evaluates 500 test recipes, and relies predominantly on English data. The authors also identify stochastic, costly LLM judging as an evaluation limitation.

9 Limitations

  • The study examines comparatively smaller models trained for 1–2 epochs; larger models and more epochs are proposed for future work.The stated aim of that future work is to enhance performance and robustness.Source: §9 · p. 8
  • LLM-as-a-judge scoring is stochastic and computationally expensive per recipe, motivating work on trustworthiness and efficient score calculation.Source: §9 · pp. 8–9
  • Evaluation covered only 500 generated recipes from the test set; expanding the sample could improve statistical significance and generalizability.Source: §9 · p. 9
  • The dataset and generated recipes are predominantly in English; future work could incorporate multiple languages.Source: §9 · p. 9

A Exploratory Data Analysis

This block presents exploratory-analysis visuals, a small-scale output comparison, a RAG-assisted allergen-substitution workflow, prompt-based recipe examples, and a substitution database. The supplied material identifies these items but gives little quantitative detail about the visuals or workflow.

A Exploratory Data Analysis

B Sample Output for Small-Scale Model

  • The small-scale-model sample is labeled as a baseline-versus-fine-tuned output comparison for a chocolate chip cookie recipe.Source: Table 8 · p. 10

C RAG-assisted Allergen Substitution System Workflow

  • The workflow figure is identified as an experimental RAG-based allergen-substitution system workflow.Source: Figure 4 · p. 11

D Sample Generated Recipes from Prompt-based Method

  • Prompt-based cottage-cheese-waffle examples include outputs with incoherent or unrelated recipe content.For example, one output mixes waffle instructions with an extensive list of unrelated items.Source: D Sample Generated Recipes from… · pp. 11–12

F Allergen Substitution Database for RAG-assisted System

G Sample Generated Recipes for RAG-assisted Method

The RAG-assisted sample section juxtaposes baseline and fine-tuned recipe outputs. The shown examples include ingredient substitutions, alongside outputs with unrelated or disorganized content.

G Sample Generated Recipes for RAG-assisted Method

H LLM-As-A-Judge Radar Charts for Prompt-based Method

The three figures are labeled as baseline-versus-fine-tuned comparisons for SmolLM-360M, SmolLM-1.7B, and Phi-2. The supplied figure labels do not state the comparison outcomes.

H LLM-As-A-Judge Radar Charts for Prompt-based Method

  • Figure 5 is labeled as a baseline-versus-fine-tuned SmolLM-360M comparison.Source: Figure 5 · p. 17
  • Figure 6 is labeled as a baseline-versus-fine-tuned SmolLM-1.7B comparison.Source: Figure 6 · p. 17
  • Figure 7 is labeled as a baseline-versus-fine-tuned Phi-2 comparison.Source: Figure 7 · p. 17

I LLM-As-A-Judge Radar Charts for Rag-assisted Method

This material identifies three baseline-versus-fine-tuned radar-chart comparisons for the RAG-assisted method and lists training and allergen-substitution generation configurations.

I LLM-As-A-Judge Radar Charts for Rag-assisted Method

J Training Configurations for Small-Scale Models

K Training Configurations for Large-Scale Models

  • Table 12 presents training configuration details for large-scale models.Source: Table 12 · p. 18

L Hyperparameters for Generation in Prompt-based Allergen Substitution

  • Table 13 lists generation hyperparameters for prompt-based allergen substitution.Source: Table 13 · p. 18

M Hyperparameters for Generation in RAG-assisted Allergen Substitution

  • Table 14 lists generation hyperparameters for RAG-assisted allergen substitution.Source: Table 14 · p. 18

Related papers

Loading 2502.02028v2…