Source-linked AI summary
FlavourBench: Ranking Frontier Language Models with Executable Culinary Ground Truth
Josef Chen, Erim Hayretci
TL;DR
Culinary reasoning benchmarks often rely on entangled model judges, sparse human preferences, or brittle exact-match scoring. FlavourBench replaces these with executable culinary ground truth and evaluates models on shared tasks with paired uncertainty, producing auditable continuous scores and corrected comparisons.
Problem
Culinary reasoning lacks evaluation that captures differences between plausible decisions without relying on judges entangled with the systems being tested.
Method
FlavourBench freezes continuous scores for every three-of-eight portfolio across shared tasks, then evaluates 27 endpoints using common tasks, anchor-cluster bootstrap bands, and multiplicity-controlled paired tests.
Results
Shared-task comparisons with continuous scores and multiplicity control distinguish supported directional differences from apparent rank order.
Takeaways & Limitations
FlavourBench makes culinary reasoning measurable through a rerunnable, auditable leaderboard and a dense training environment without using one language model to grade another.
Takeaways & Limitations
The benchmark evaluates constrained ingredient selection rather than full recipe generation or kitchen planning, and it does not test transfer from Epicure optimization to human cooking outcomes.
Abstract
from arXiv · showhide
Open-ended language-model benchmarks usually inherit a judge: a human preference panel, another model, or a brittle exact-match key. We introduce FlavourBench, an automated benchmark in which a versioned culinary system supplies dense, executable ground truth. Each task presents eight ingredients and asks for a three-ingredient portfolio; before model execution, Epicure scores all 56 possible portfolios. We evaluate 27 frontier endpoints on an identical 534-task core spanning substitution, pairing, and constrained composition. Every ranked model has exactly 89 valid responses per panel and family (14,418 model-task cells total), eliminating differential missingness from the leaderboard. The FlavourBench Score is the equal-family mean of the frozen task scores. We use 50,000 anchor-cluster bootstrap replicates for simultaneous 95% score bands and 100,000 sign-flip draws for all 351 paired model contrasts, with Holm control. The two independently compiled panels correlate at r = 0.89 (rank rho = 0.80). Grok 4.6 has the largest point estimate at 65.1 (simultaneous 95% CI 61.0-69.2); 101 of 351 model pairs are resolved. The release includes the prompts, all portfolio score maps, raw responses, exact routes, content hashes, and an offline verifier that reconstructs every result.
1 Introduction
FlavourBench addresses entangled model judging and brittle exact-match evaluation by using executable culinary ground truth for dense portfolio decisions. It evaluates models on a shared complete core with a continuous score and reproducible paired comparisons.
- FlavourBench replaces model judges and brittle exact match with an executable culinary environment that preserves differences among plausible portfolios.Epicure supplies task-specific scores without judging models or claiming universal human taste.
- Epicure scores all 56 three-ingredient portfolios before model execution, enabling table-lookup evaluation without post-hoc interpretation.
- 27 endpoints face the same 534 tasks, supporting direct paired comparisons on 534 shared observations per model pair.
- The paper contributes an executable benchmark, a complete-core evaluation with uncertainty and multiplicity control, and a content-addressed reproducibility release.
2 Executable culinary ground truth
FlavourBench fixes a versioned culinary runtime and converts its operations into scored three-of-eight selection tasks. Frozen continuous score maps and strict parsing make model evaluation reproducible and independent of post-hoc judgments.
- The benchmark binds the Epicure release and computational specification by hash, so changing them creates a new task-set identity.
- Epicure assigns utilities to all 56 candidate portfolios, gives invalid constrained portfolios zero, and normalizes valid utilities within each task.
- Each task has a unique 100-point optimum, while chance is the exact mean score over the frozen portfolio set rather than a generic percentage.
- The ranked core contains 534 tasks, with 89 tasks per family in each of two panels and one cluster per anchor ingredient.
- Continuous score maps distinguish many portfolio qualities, while constraint tasks remain sparse because infeasible portfolios score zero.
- Models submit three distinct A–H labels after a FINAL_SELECTION marker, and the parser performs one unordered-set lookup in the frozen score map.
3 Evaluation design
The evaluation uses a balanced common core across endpoint families and culinary task families, with fixed routes and preregistered clustered inference. This design prevents model-specific missingness and controls uncertainty across scores and pairwise contrasts.
- Every endpoint contributes exactly 534 valid responses, and failures determine common-core eligibility rather than receiving zero or being silently dropped.
- The three task families share eight candidates and 56 frozen scores while probing different culinary decisions.
- Task-map diagnostics are computed before model execution using uniformly random portfolio means and median best–second-best margins.
- Routes are fixed before scored blocks, automatic fallback is disabled, and replacement routes trigger complete-block reruns rather than pooled successes.
- Within each panel–family stratum, the first 89 eligible task IDs under fixed SHA-256 ordering form a validity-selected common core.
- Inference uses 50,000 anchor-cluster bootstrap samples for simultaneous score bands and 100,000 sign-flip draws with Holm correction for 351 pairs.
4 Results
The complete-core evaluation yields a statistically cautious leaderboard: Grok 4.6 has the highest point estimate, but only some pairwise differences survive correction. Aggregate scores also conceal distinct family strengths and rank instability.
- 4.1 A statistically resolved FlavourBench leaderboard: 65.1 is Grok 4.6’s largest point estimate, with simultaneous 95% interval [61.0, 69.2]; 101 of 351 paired contrasts remain significant after Holm correction.
- 4.1 A statistically resolved FlavourBench leaderboard: 534 valid parser-confirmed responses per ranked row prevent endpoints from gaining advantage through different task subsets.
- 4.2 Does the ordering replicate on a second task panel?: The two disjoint panels correlate at Pearson r = 0.89 and rank correlation 0.80, serving as descriptive stability diagnostics rather than a selection rule.
- 4.3 Which pairwise conclusions survive correction?: Grey pairwise cells indicate insufficient evidence for a directional claim at familywise α = .05, not equality between models.
- 4.4 Aggregate scores conceal different culinary profiles: Similar aggregate scores can reflect different family strengths because constraint, pairing, and substitution tasks reward distinct culinary capabilities.
- 4.5 Rank uncertainty is visible, not hidden: Bootstrap rank intervals distinguish stable ordering regions from small point-estimate differences that change order under anchor resampling.
- Fixed-hash case studies expose complete prompts, choices, optimal portfolios, raw responses, selected ingredients, and exact scores.
5 From benchmark to training signal
FlavourBench’s exhaustive task maps turn culinary decisions into dense, executable reward signals that support several training paradigms, although the paper evaluates models rather than training them.
- 5 From benchmark to training signal: Epicure assigns every candidate portfolio a deterministic bounded reward, creating an executable culinary environment for learning.For task state x_t and portfolio a, the reward is r(x_t, a) = s_t(a).
- 5 From benchmark to training signal: The full ordering of 56 candidates supports supervised learning from optimal portfolios and preference or contrastive learning from known reward differences.Any candidate pair can provide a contrast whose reward difference is known.
- 5 From benchmark to training signal: Dense rewards also support offline or online policy optimization with held-out anchors.The present experiment does not test whether models trained against Epicure rewards improve.
6 Reproducibility
The release makes FlavourBench reproducible by preserving content-addressed inputs, execution records, statistical artifacts, and offline reconstruction tools.
- 6 Reproducibility: The release includes prompts, candidates, all 56 portfolio scores, oracle provenance, task strata, and digests for the 534 scheduled tasks and unique anchors.These assets define the frozen task inputs and score maps used by the benchmark.
- 6 Reproducibility: Raw responses, parser outputs, scores, usage, latency, finish reasons, and provider-attempt events are retained for the 14,418-cell ranked matrix.Responses are written once under model-slot and cell identifiers.
- 6 Reproducibility: An offline verifier reconstructs leaderboard files, pairwise results, tables, and figures from local files without provider access.The analysis plan, seeds, bootstrap and sign-flip counts, pairwise rows, and diagnostics are also released.
- 6 Reproducibility: 27 frontier models are evaluated on the identical 534-task core.The release also records the model manifest, exact provider routes, fallback policy, parameters, and route digest.
- 6 Reproducibility: Figure 2 displays score intervals and inferential tiers, while unresolved groups should not be interpreted as statistically established orderings.Point ranks are provided for navigation only within unresolved groups.
7 Related work
FlavourBench extends executable-environment evaluation into culinary reasoning, complementing broad, knowledge-focused, contamination-resistant, preference-based, and culinary benchmarks.
- 7 Related work: General-purpose evaluation includes broad suites, difficult knowledge tests, contamination-resistant sets, and human-preference systems.Model judges improve scale but introduce bias and calibration problems.
- 7 Related work: Existing culinary datasets and benchmarks address ingredient networks, recipe understanding, planning, nutrition, and food knowledge.FlavourBench is positioned as complementary rather than as a replacement for these areas.
- 7 Related work: FlavourBench focuses on common-task frontier-model measurement against a versioned culinary decision environment.Its distinguishing elements are exhaustive partial-credit maps and paired inference.
8 Limitations
FlavourBench provides reproducible comparative measurement, but its score, task scope, and common-core estimand remain tied to specific reference choices and completed tasks.
- 8 Limitations: The FlavourBench Score measures agreement with one published Epicure release rather than universal human taste.The reference system defines the task-specific reward surface.
- 8 Limitations: Within-task min–max normalization makes the 0–100 scale comparable while discarding absolute raw-utility magnitudes.Scores therefore preserve relative portfolio quality within tasks, not cross-task raw utility levels.
- 8 Limitations: The tasks cover constrained ingredient selection rather than full recipe generation, sensory execution, safety advice, or long-horizon kitchen planning.The benchmark’s conclusions are consequently limited to its selected decision setting.
- 8 Limitations: Table 3’s ranks summarize uncertainty intervals and Holm-controlled pairwise groups, whereas Table 4 illustrates fixed scores through selected common-core responses.These displays document comparative outcomes and examples rather than expanding the benchmark’s task scope.
- 8 Limitations: The common-core requirement removes differential missingness but restricts the estimand to tasks completed by every endpoint.Supplemental tracks retain the remaining tasks.
9 Conclusion
FlavourBench makes culinary reasoning measurable through executable ground truth and a continuous score. Its released artifacts support rerunning and auditing every result.
- FlavourBench makes culinary reasoning measurable without asking one language model to grade another.
- A versioned runtime enumerates candidate decisions, while paired uncertainty distinguishes resolved differences from apparent rank order.
- Every score, interval, contrast, and figure is reconstructable from public artifacts.
A Exact execution routes
The execution and reporting tables fix routes, family-score aggregation, and core design constants for the benchmark analysis.
- Frozen execution routes are listed by panel, with automatic fallback disabled.
- The equal-family mean of each 178-task family defines the FlavourBench Score using identical tasks for every model.
- Core design and inference constants are fixed before clean primary execution.