PaperlayerView exact revision on arXiv

Source-linked AI summary

DiningBench: A Hierarchical Multi-view Benchmark for Perception and Reasoning in the Dietary Domain

Song Jin, Juntian Zhang, Xun Zhang, Zeying Tian, Fei Jiang, Guojun Yin, Wei Lin, Yong Liu, Rui Yan

arXiv:2604.10425v1cs.CV

TL;DR

Food-domain VLM evaluation remains limited by coarse categories, single-view imagery, and inaccurate metadata. DiningBench introduces a hierarchical, multi-view benchmark spanning fine-grained classification, nutrition estimation, and VQA, and finds substantial weaknesses in fine-grained discrimination and nutritional reasoning across evaluated models.

  • Problem

    Food benchmarks often rely on coarse-grained categories, single-view imagery, and metadata limitations that inadequately represent real-world dining scenarios.

  • Method

    DiningBench combines hierarchical food tasks, multi-view imagery, hard negatives, nutrition estimation from visual cues, and semantic VQA evaluation.

  • Results

    Evaluation of 29 state-of-the-art models reveals substantial performance gaps, with persistent difficulty in fine-grained visual discrimination and precise nutritional reasoning.

  • Takeaways & Limitations

    DiningBench provides a challenging testbed for advancing visually grounded, domain-aware VLMs for real-world food understanding.

  • Takeaways & Limitations

    The dataset is culturally skewed toward Chinese cuisine, and LLM-assisted nutrition and distractor generation may inherit biases or inaccuracies despite human verification.

Abstract

from arXiv · show

Recent advancements in Vision-Language Models (VLMs) have revolutionized general visual understanding. However, their application in the food domain remains constrained by benchmarks that rely on coarse-grained categories, single-view imagery, and inaccurate metadata. To bridge this gap, we introduce DiningBench, a hierarchical, multi-view benchmark designed to evaluate VLMs across three levels of cognitive complexity: Fine-Grained Classification, Nutrition Estimation, and Visual Question Answering. Unlike previous datasets, DiningBench comprises 3,021 distinct dishes with an average of 5.27 images per entry, incorporating fine-grained "hard" negatives from identical menus and rigorous, verification-based nutritional data. We conduct an extensive evaluation of 29 state-of-the-art open-source and proprietary models. Our experiments reveal that while current VLMs excel at general reasoning, they struggle significantly with fine-grained visual discrimination and precise nutritional reasoning. Furthermore, we systematically investigate the impact of multi-view inputs and Chain-of-Thought reasoning, identifying five primary failure modes. DiningBench serves as a challenging testbed to drive the next generation of food-centric VLM research. All codes are released in https://github.com/meituan/DiningBench.

1 Introduction

DiningBench addresses limitations in food benchmarks by combining hierarchical tasks, multi-view imagery, fine-grained hard negatives, and high-fidelity nutrition alignment. Evaluation of 29 VLMs reveals substantial weaknesses in fine-grained discrimination and nutritional reasoning, while further analyses examine multi-image input, Chain-of-Thought reasoning, and failure modes.

  • Existing food benchmarks often use coarse-grained classification, single-view images, and limited coverage of real-world dining complexity.
  • DiningBench evaluates identification, nutrition quantification, and high-level reasoning through Fine-Grained Classification, Nutrition Estimation, and Visual Question Answering.
  • The benchmark averages 5.27 images per dish and uses same-menu distractors to test multi-view fusion and subtle visual discrimination.
  • Its nutrition alignment integrates merchant metadata with high-quality images, while an AI-assisted curation pipeline supports image assessment, reference matching, and nutrition inference.
  • Evaluation of 29 state-of-the-art models shows significant performance gaps, especially in fine-grained visual discrimination and precise nutritional reasoning.
  • The authors analyze multi-image inputs and Chain-of-Thought reasoning and systematically identify primary failure modes for future research.

2 Related Work

Prior food and multimodal benchmarks established recognition, recipe, and reasoning capabilities but commonly addressed these dimensions separately. DiningBench unifies them in a hierarchical evaluation supported by context-aware hard negatives and multi-view consistency.

  • VQA, GQA, and broader multimodal benchmarks evaluate fine-grained perception, compositional reasoning, cross-modal alignment, instruction following, and factual grounding.
  • Food-101, UEC-Food, VIREO Food-172, ISIA Food-500, and Food2K established or scaled visual food categorization benchmarks.
  • Recipe1M+ and RecipeQA connected food appearance with recipes and procedural knowledge through cross-modal data.
  • DiningBench unifies identification, nutrition estimation, and high-level reasoning in a hierarchical evaluation with context-aware hard negatives and multi-view consistency.

3 DiningBench

DiningBench is a hierarchical, multi-view benchmark that evaluates food-focused VLMs from fine-grained identification through nutrition estimation and higher-order visual reasoning. It combines challenging multi-view data, hard negatives, curated nutrition information, and diverse VQA tasks with systematic evaluation procedures.

  • Task Definitions: DiningBench evaluates VLMs on Fine-Grained Classification, Nutrition Estimation, and Visual Question Answering in a hierarchical task design.
  • Task Definitions: Fine-grained classification presents eight options containing one correct label and seven visually or semantically similar hard distractors from the same merchant category.
  • Nutrition Estimation Dataset: Nutrition estimation predicts Calories, Carbohydrates, Protein, and Fat as continuous values inferred from images, with nutrition data obtained through extraction or LLM-assisted estimation and manual verification.
  • Visual Question Answering: The VQA subset covers cuisine technique, dietary suggestion, multi-image analysis, and counterfactual reasoning, including both single-image and multi-image samples.
  • Evaluation Metrics: Evaluation uses Accuracy for classification, MAE, RMSE, and MAPE for nutrition regression, and an LLM-as-a-Judge Accuracy for semantically evaluated VQA answers.
  • Dataset Statistics: 3,021 unique dishes span 2,060 categories and 15,928 images, averaging 5.27 images per dish across the benchmark.

4 Experiments

DiningBench exposes substantial weaknesses in current VLMs across fine-grained classification, nutrition estimation, and visual reasoning. Multi-view inputs generally help, but gains saturate and depend on model scale, while CoT often harms direct perception and regression tasks.

  • Overall Evaluation: 29 evaluated VLMs show significant gaps in fine-grained visual understanding across DiningBench’s three tasks.The benchmark remains challenging even for advanced models.
  • Fine-Grained Classification: 81.83% accuracy is achieved by Gemini-3-Flash-Preview, compared with 65.26% for GPT-4o and 70.18% for GPT-5 on fine-grained classification.Hard distractors make visually similar dishes difficult to distinguish.
  • Nutrition Estimation: 24.45% Average MAPE for Gemini-3-Pro-Preview and 42.43% for GPT-4o show that nutrition estimation remains especially difficult.The results indicate deficiencies in volumetric reasoning and ingredient analysis.
  • Visual Question Answering: 80.60% accuracy for GPT-4o on VQA indicates stronger general reasoning than the other tasks, but substantial room for improvement remains.Complex food queries still require better visual grounding.
  • Multi-View Inputs: Adding views generally improves Accuracy, MAE, and RMSE, with the largest gain from 1 to 2 images and diminishing returns beyond two.Complementary angles help resolve occlusion and ambiguity, while later additions approach saturation.
  • Multi-View Inputs: Large-scale models improve consistently with more views, whereas smaller models show unstable or worsening MAPE as visual inputs increase.Excessive visual tokens may act as noise or cause information overload in limited-capacity models.
  • Chain-of-Thought Reasoning: CoT often decreases classification accuracy and can drastically increase nutrition-estimation MAPE, especially for smaller open-source models.The reasoning steps may introduce noise rather than refine predictions.
  • Chain-of-Thought Reasoning: CoT produces mixed VQA results and may amplify errors when initial visual grounding is flawed.Selected proprietary models improve, while GPT-4o-mini and smaller Qwen variants degrade.

5 Additional Perspectives of DiningBench

DiningBench’s multi-view and aligned reference data support research beyond benchmark evaluation, including 3D reconstruction and novel-view synthesis.

  • Broader Research Potential: DiningBench averages 5.27 images per dish, offering real-world occlusions and variable lighting for 3D reconstruction and novel-view synthesis research.These conditions are often absent from synthetic datasets.

6 Ethical Considerations

The dataset construction and release follow stated copyright, intellectual-property, privacy, and content-safety procedures.

  • Data Rights: DiningBench uses Meituan-authorized base images and metadata for non-commercial research under the CC BY-NC-ND 4.0 license.The authors state that explicit permission and copyright authorization were obtained.
  • Privacy and Safety: Automated filtering and manual verification were used to ensure food-related images without privacy-compromising elements.The passage truncates before describing the full filtering criteria.

7 Conclusion

DiningBench evaluates food-domain VLMs through hierarchical tasks and multi-view imagery, revealing persistent weaknesses in visual discrimination, nutrition reasoning, and view fusion. The authors position it as a testbed for more visually grounded and domain-aware food understanding.

  • Conclusion: 29 state-of-the-art models exhibit substantial performance gaps, especially on nutrition quantification and subtle visual distinctions.The conclusion also identifies ineffective multi-view fusion as a critical limitation.
  • Implications: The benchmark is intended to support progress toward visually grounded, domain-aware VLMs for real-world food understanding.The stated broader goals include more reliable dietary applications and improved outcomes.

8 Limitations & Potential Risks

DiningBench’s limitations include cultural skew, LLM-assisted data construction, and risks from applying benchmarked VLMs to dietary guidance.

  • The dataset is culturally skewed toward Chinese cuisine, which may affect generalization to underrepresented culinary traditions.The authors note that international dishes were included, but this does not remove the underlying sourcing bias.
  • LLM-assisted generation of nutritional ground truth and distractor selection may introduce latent biases or subtle inaccuracies despite human verification.
  • Errors in fine-grained classification or nutrition estimation could cause incorrect health monitoring or allergen oversight in real-world dietary guidance.The authors therefore call for extreme caution and human-in-the-loop oversight in health-related applications.

A Dataset Case

The paper illustrates DiningBench’s three task formats through examples spanning fine-grained classification, nutrition estimation, and dietary reasoning in VQA.

  • Figure 6 demonstrates fine-grained classification with hard distractors.
  • Figure 7 shows the input and output format for nutrition estimation.
  • Figure 8 illustrates a VQA scenario requiring dietary reasoning.

B.1 Baseline Details

The evaluation covers 29 state-of-the-art VLMs across proprietary and open-source families, and examines how Chain-of-Thought prompting shifts classification and VQA accuracy.

  • 29 state-of-the-art VLMs were evaluated, comprising 10 proprietary and 19 open-source models.
  • The proprietary baselines include Gemini and GPT model series alongside other competitive models.The listed families include Gemini-2.5/3.0, GPT-4o/4.1/5, and o4-mini.
  • The VQA examples include nutrition estimation and dietary-standard reasoning based on fine-grained visual cues.
  • Figure 9 compares Classification Accuracy with and without CoT, while Figure 10 compares the corresponding VQA Accuracy shifts.
  • The open-source evaluation spans Qwen-VL and InternVL models across multiple parameter scales.

B.3 Performance on the English-Translated Dataset

On the English-translated dataset, classification accuracy declines across models while nutrition estimation improves for a significant subset; the dataset audit received unanimous approval.

  • CoT prompting was evaluated by comparing accuracy shifts with and without reasoning for classification and VQA.
  • Fine-Grained Classification accuracy dropped universally across all models on the English dataset.For example, Qwen-3-VL-8B-Instruct declined from 64.15% to 58.56%.
  • Qwen-3-VL-8B-Instruct dropped from 64.15% to 58.56% after translation.
  • The authors attribute the classification decline to a semantic gap between translated dish names and models’ learned cultural or visual associations.
  • Nutrition Estimation performance improved for a significant subset of models, including the Gemini-2.5 and GPT-4o series.
  • Three independent Ph.D. evaluators audited 210 entries, with a 100% approval rate across all evaluators.The sample included 70 entries per task and used the construction-stage inclusion criteria.
  • The benchmark compares model accuracy across Classification, Nutrition Estimation, and VQA, with best and second-best results highlighted in Table 4.
Loading 2604.10425v1…