Source-linked AI summary
Validating FKG.in: Soundness Assessment in LLM-Augmented Indian Food Knowledge
Saransh Kumar Gupta, Armaan Shah, Lipika Dey, Partha Pratim Das, Ramesh Jain
TL;DR
LLM-extracted recipe data can contain structural, semantic, numerical, and source-fidelity errors that threaten reliable food knowledge graphs. The paper proposes a layered semi-automated workflow for FKG.in combining automated validation, retrieval-based correction, and human oversight. Evaluation shows widespread failures across validation dimensions, while conservative retrieval-based correction resolves a subset of recurring unseen errors.
Problem
Reliable extraction is difficult because Indian culinary data vary linguistically, regionally, culturally, and structurally, while LLM outputs can hallucinate or diverge from sources.
Method
The paper develops a multi-stage FKG.in workflow combining LLM extraction with rule-based, statistical, learning-based, retrieval, and human-in-the-loop validation and resolution.
Results
Source-fidelity verification identified contradictions in 96.4% of documents, while retrieval covered 53.48% of previously unseen contradictions and conservative automation resolved 311 unseen errors.
Takeaways & Limitations
Layered verification is needed before knowledge-graph ingestion because many extraction failures remain undetected by schema validation alone.
Takeaways & Limitations
Semantically ambiguous, culturally nuanced, and region-specific concepts still require human oversight, while conservative thresholds limit autonomous handling of novel complex inconsistencies.
Abstract
from arXiv · showhide
The online culinary ecosystem is increasingly populated by recipe content generated, modified, or summarized by Large Language Models (LLMs). While often plausible, such outputs may contain hallucinated ingredients, misrepresented quantities, or culturally implausible combinations, limiting their suitability for downstream applications and knowledge graph construction. In this paper, we present a semi-automated soundness assessment workflow for validating structured recipe data extracted and augmented by LLMs from informal culinary sources. Developed as part of FKG(.in), a knowledge graph of Indian food, the pipeline identifies and addresses common failure modes, including structural inconsistencies, semantic and logical incoherence, and deviations from the source text, through a multi-stage process combining formal grammars, vocabulary-based checks, statistical heuristics, Set Transformer-based coherence modeling, and retrieval-based verification. Although evaluated on Indian recipes, the proposed methods are applicable to broader multilingual and multicultural culinary domains. We provide a practical, auditable, and application-agnostic framework for validating LLM-augmented recipe data, thereby strengthening the foundations of machine-readable food knowledge infrastructures in the era of LLM-generated content.
1. Introduction
The paper addresses the scarcity and fragmentation of structured Indian food knowledge and the reliability challenges introduced when LLMs extract information from diverse culinary sources. It presents a semi-automated workflow to validate LLM-extracted data before incorporation into FKG.in.
- Structured Indian food knowledge is scarce and fragmented across diverse informal and formal culinary sources.
- Linguistic, regional, and format variation makes Indian food entities, quantities, and recipe structures difficult to identify and standardize at scale.
- LLM outputs may contain factual inaccuracies, omissions, logical inconsistencies, and hallucinations that affect nutritional, functional, and cultural recipe characteristics.
- The workflow assesses factual correctness, internal consistency, semantic plausibility, and source fidelity using automated checks, modeling, retrieval, and human review.
- Validation is positioned before knowledge-graph ingestion to identify and address inconsistencies in LLM-extracted information.
2. Related Work
Prior work spans general hallucination verification and food-domain language and generation tasks, but validation of extracted food knowledge remains comparatively underdeveloped. The paper motivates a workflow focused on structural consistency, semantic plausibility, and fidelity to source recipes.
- General-purpose hallucination research includes entailment verification, retrieval-verifier pipelines, evidence generation, proof-based methods, and knowledge-graph verification.
- Food-domain studies have examined hallucinations in nutrition and ingredient outputs, alongside culinary understanding, recipe generation, and substitution reasoning.
- Existing food efforts primarily emphasize generation quality, personalization, or domain adaptation rather than validation of extracted recipe data.
- The identified gap concerns verification of structural consistency, semantic plausibility, and source fidelity in LLM-extracted food knowledge.
3. Challenges in Validating LLM-Extracted Food Knowledge
LLM-augmented recipe data can fail structurally, numerically, semantically, culturally, and with respect to source fidelity. These challenges motivate context-aware, layered validation beyond schema matching and lexical checks.
- LLM-extracted recipes can contain hallucinations, factual drift, semantic distortion, context loss, overgeneralization, ambiguity, and structural inconsistencies.
- Such failures can distort recipes, affect nutritional computations, violate cultural expectations, and propagate to downstream applications.
- Structural and numerical inconsistency arises when outputs violate the ontology or express implausible or ambiguous quantities across varied Indian measurement conventions.
- Observed errors include confusing preparation states, misinterpreting transliterated terms, culturally inappropriate substitutions, and misparsing code-switched instructions.
- These examples reinforce the need for context-aware validation that accounts for linguistic, cultural, and contextual complexity.
4. Methodology
The methodology combines LLM extraction with layered rule-based, statistical, learning-based, retrieval, and human-in-the-loop validation and resolution stages. It evaluates ingredients most extensively while applying lighter checks to less ambiguous recipe attributes.
- 4.1. Multi-Stage Soundness Assessment Workflow: The proposed pipeline uses seven sequential stages to identify and address structural, semantic, and source-fidelity issues before knowledge-graph ingestion.
- 4.1. Multi-Stage Soundness Assessment Workflow: Structured recipe-card data provide reference representations, while LLMs extract ingredients, quantities, methods, metadata, and latent culinary attributes from free-text descriptions.
- 4.2. Inconsistency Detection Agent: The Inconsistency Detection Agent combines rule-based, statistical, and machine-learning methods, with ingredient data assessed across all five detection stages.
- 4.2. Inconsistency Detection Agent: JSON Schema validation flags malformed records, missing fields, and datatype mismatches, while dual-tier checks distinguish essential from non-essential attributes.
- 4.2. Inconsistency Detection Agent: A Staple Guard exempts ambient ingredients from fidelity rejections to accommodate implicit culinary conventions.
- 4.3. Inconsistency Resolution Agent: The Resolution Agent progresses from deterministic repairs and canonicalization to patch-based correction, source grounding, rollback, and human adjudication.
- 4.3. Inconsistency Resolution Agent: Human decisions are logged to expand the correction database and refine substitution graphs, co-occurrence priors, and canonicalization rules.
5. Results
Evaluation across 500 recipe documents found inconsistencies spanning validation dimensions, while retrieval-assisted correction reused a small set of validated examples to resolve recurring errors.
- 500 recipe documents were evaluated for inconsistencies relevant to downstream knowledge graph construction.
- Structural violations affected 91.6% of documents, while vocabulary enforcement, statistical grounding, and semantic coherence modeling identified errors in 48.0%, 36.6%, and 68.8%, respectively.Source-fidelity verification found contradictions in 96.4% of documents, many of which would remain undetected through schema validation alone.
- Quantitative hallucinations comprised 49.0% of over 2,000 source-fidelity errors, followed by existential hallucinations at 29.7% and morphological contradictions at 4.0%.These classes indicate that many ingredient-level failures arise from semantic drift and unsupported inference during structured generation.
- 53.48% of previously unseen contradictions were covered at cosine similarity ≥0.88, and 42 validated corrections enabled 311 autonomous resolutions.Automation was restricted to high-confidence exact entity matches and staple or ambient inferences, yielding 20.8% coverage of all unseen errors.
- Ingredient NER Evaluation: Ingredient NER performed best on simple expressions (F1 = 84.2%) and weakest on longer compositional sentences (F1 = 50.3%).Complex constructions achieved F1 = 75.0%, while Hinglish or vernacular expressions achieved F1 = 65.1%; these suites were designed as robustness stress tests.
- Failures across structural, lexical, numerical, semantic, and source-fidelity dimensions support using layered validation rather than a single mechanism.A representative dum aloo walkthrough illustrates stage-wise detection and resolution in Appendix A.
6. Limitations and Future Directions
The framework’s evaluation beyond patch-based resolution remains ongoing, and conservative automation limits handling of novel inconsistencies while current coverage is primarily text-based.
- Systematic evaluation beyond the patch-based resolution component remains ongoing.
- Semantically ambiguous, culturally nuanced, or region-specific culinary concepts still require human oversight in some cases.The detection–resolution feedback loop is not yet fully integrated, and resolution corrections are not systematically reused to improve detection.
- Similarity thresholds of ≥0.88 and strict error archetypes prevent automation drift but limit autonomous handling of novel or highly complex inconsistencies.The resolution agent remains dependent on human-validated correction seeds.
- The framework focuses primarily on text-based recipe sources, leaving video recipes and image-centric blogs to require more sophisticated alignment and grounding.
- Future Directions: Future work targets expanded benchmark datasets, iterative detection–resolution feedback, broader multilingual coverage, retrieval-augmented reasoning, and culturally nuanced substitutions.These efforts aim to reduce manual review while preserving transparency, traceability, and culinary fidelity.
7. Conclusion
The paper presents a layered workflow for validating LLM-extracted Indian recipe knowledge before knowledge graph ingestion, with evaluation showing failures across all validation dimensions.
- The workflow validates structural inconsistencies, numerical anomalies, semantic incoherence, and source-fidelity violations in LLM-extracted recipe data.It combines schema validation, statistical grounding, lexical and semantic consistency checks, source-grounded verification, retrieval-assisted correction, and human review.
- Evaluation across 500 recipe documents found failures across all validation dimensions, including errors that schema validation alone would not detect.
- The framework provides a practical template for validating LLM-generated structured knowledge beyond Indian culinary data.
Declaration on Generative AI
The authors used ChatGPT for language refinement and proofreading and reviewed all AI-assisted text.
- ChatGPT was used for language refinement and proofreading during preparation of the work.
- The authors reviewed all AI-assisted text and take full responsibility for its content.
A. Inconsistency Detection and Resolution: Proof of Concept on a Sample Recipe
A representative dum aloo recipe was used to demonstrate how manually annotated LLM-extracted JSON can contain factual drift, morphological contradictions, and hallucinated metadata. The walkthrough also added inconsistent ingredients to exercise the framework’s coherence capabilities.
- Demonstration Setup: The proof-of-concept uses a raw dum aloo recipe blog document and its corresponding LLM-generated structured JSON representation.The initial extraction was manually annotated for representative inconsistencies.
- Detected Errors: Five factual or morphological errors affected baby potatoes, oil, coriander seeds, and onion quantities or forms.Reported discrepancies included baby potatoes quantity 12.5 versus 13.5, oil quantity 5 versus 2.5, and onion quantity 3 versus 1.
- Detected Errors: The extraction hallucinated a serving size of 6 that was absent from the source.This was categorized as an existential hallucination.
- Demonstration Setup: Two inconsistent ingredients were additionally inserted into the extracted JSON to demonstrate the framework’s coherence capabilities.These additions were part of the demonstration setup rather than the raw extraction.
A.1. Inconsistency Detection Agent Walkthrough
The walkthrough shows detection and resolution agents operating across schema, vocabulary, semantic, and source-grounding checks. Detected inconsistencies are repaired automatically when supported, while difficult numerical discrepancies are escalated for human review.
- Inconsistency Detection Agent Walkthrough: Schema validation flagged missing fields including place_of_origin, ferment_time, knife_cuts, flavor, texture, taste, and related_recipes.The first stage checks extracted JSON against the expected schema.
- Inconsistency Detection Agent Walkthrough: The detection agent identified a morphological contradiction between ‘whole’ and ‘boiled, peeled, and pricked’ baby potatoes.It also classified the unsupported serving size of 6 as an existential hallucination.
- Inconsistency Detection Agent Walkthrough: Ingredient components were mapped against textual bounds to identify factual drift in the quantitative values for Errors 1, 3, and 5.The mapping connected extracted ingredient values with evidence from the source text.
- Inconsistency Resolution Agent Walkthrough: The resolution agent applies schema-driven repairs for missing non-essential fields and uses patch-based semantic correction for contextual contradictions.Missing fields can receive the canonical na value, while semantic correction uses a Weighted Two-Tower embedding strategy and KNN search.
- Inconsistency Resolution Agent Walkthrough: Numerical range errors below the ≥0.88 similarity threshold for autonomous patching are escalated to the Human-in-the-Loop interface.This escalation preserves review for corrections that do not meet the system’s confidence requirement.
B. Error Detection Performance (N=500)
Across a 500-recipe corpus, the detection evaluation organizes errors into structural, lexical, numerical, semantic, and source-fidelity categories. Tables 4 and 5 report the stage-wise results and fidelity-error distribution.
- Corpus Evaluation: The Inconsistency Detection Agent was evaluated across a 500-recipe corpus.The evaluation summarizes detection results across five validation stages.
- Detection Stages 1–4: Table 4 reports results for Stages 1–4: structural violations, unfamiliar ingredient expressions, numerical outliers, and contextually anomalous ingredients.These stages cover schema validity, lexical normalization, numerical plausibility, and semantic coherence.
- Source Fidelity: Table 5 reports Stage 5 source-fidelity errors, including ingredient-level contradictions and metadata discrepancies against the source text.The table focuses on fidelity errors rather than the first four detection categories.