Source-linked AI summary
Watts If? A Benchmark for Counterfactual Reasoning About the AI Data Center Buildout
Pankaj Chouhan, Rajarshi Das, Aditya Dubey, Manminder Tomar, Sreekar Reddy, Samta Sharma, Satyapragnya Kar, PR Panigrahi
TL;DR
LLMs are being used to screen data-center commitments, but there is no test of whether they track what those choices rule out. DCBench-440 evaluates generative planning questions and source-grounded multiple-choice questions; every deployment scores higher on multiple choice, while none exceeds 4.0% exact match on action enumeration.
Problem
LLMs are used to help screen data-center commitments, but the paper identifies no test of whether they track what each commitment rules out.
Source: §1 · p. 1Method
DCBench-440 combines generative questions over five US deal archetypes with an independent multiple-choice arm grounded in site records, published equations, and stated scenarios.
Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4Results
Every deployment scores higher on multiple choice, but no deployment exceeds 4.0% exact match when listing every action available at a gate.
Source: §4 · pp. 4–5Takeaways & Limitations
Measuring omitted options is a precondition for trusting screening tools, and future work will weight omissions by the water and emissions quantities of foreclosed branches.
Source: §5 · p. 6The fixed test set can be optimized against, so the benchmark regenerates worlds.
Source: §5 · p. 6
Abstract
from arXiv · showhide
An AI data center is committed years before it draws a watt: a few discrete choices (grid or on-site gas, evaporative or dry cooling, which county to file in) fix its capital, water draw, and carbon path, and LLMs are now asked to help screen them with no test of whether they track what each commitment rules out. We introduce DCBench-440, with two arms reported separately: 200 generative questions over STRIPS worlds for five US deal archetypes, with search-verified gold answers and no tested shortcut above 12% on any question type, and 240 multiple-choice questions grounded in quoted site records, emission equations, and stated scenarios. Across open-weight deployments, accuracy spans 12.5-71.5% (generative) and 58.3-92.5% (multiple choice). One skill fails everywhere: asked to list every action available at a decision gate, no model exceeds 4.0% exact match. The strongest list 72-85% of the actions that would change the project's state but omit most permitted actions that change nothing; with those excused, the best exact match is still 24%. Code, data, and other materials are available upon request.
1 Introduction
Data-center commitments fix a site’s capital, water, carbon, and first-power path, yet LLMs are being used to screen these choices without a test of whether they track what each commitment rules out. DCBench-440 evaluates that gap with generative and multiple-choice arms and diagnoses a shared failure in listing every available action.
1 Introduction
- DCBench-440 targets omissions of available options, such as a permit prerequisite or a route foreclosed by an earlier choice, that a fluent answer can leave unchecked.The paper frames this as a distinct screening failure from an incorrect number.Source: §1 · p. 2
- The benchmark pairs generative questions over five US deal archetypes with an independent multiple-choice arm.The generative arm applies eight question types, and the paper evaluates 11 deployments of seven open-weight models.Source: §1 · p. 2
- Every evaluated model shares a failure in exact enumeration: listing every action available at a decision gate.The paper defines this as listing every step the project could take next.Source: §1 · p. 2
2 Background and related work
The paper situates its benchmark between planning evaluations, data-center siting studies, and assessments of AI’s environmental footprint. It uses state-based planning questions and source-grounded questions to test model reasoning rather than compute a site decision.
2 Background and related work
- A STRIPS task represents facts and states, with actions applying only when their preconditions hold; a no-op is allowed but leaves the state unchanged.The model answers questions about the described world, while a parser and symbolic validator grade its reply.Source: §2 · p. 2
- PlanBench and related studies score whole plans, while ACPBench Hard asks open-ended atomic questions; GPT-4o solved 25% when producing applicable sets [27, 28, 15, 16].Hybrid systems instead use symbolic planners or external verifiers, whereas this study measures the language model alone [18].Source: §2 · p. 2
- Prior data-center studies optimize or simulate siting decisions, while a review of 189 ML-assisted site-selection studies identifies missing benchmark datasets as a field challenge [26] [24].Work on AI’s environmental costs measures models’ own consumption, and data-center economics question sets test recall or use a model as judge [21].Source: §2 · p. 2
3 Two arms
The generative arm tests planning questions in structured worlds modeled on five US data-center deal archetypes, while the independent multiple-choice arm asks source-grounded questions across data-center topics. The arms use distinct content and grading, and are reported separately.
3.1 The generative arm: five deal archetypes
- Five US deal archetypes encode environmental, regulatory, and financial constraints as static facts that close off parts of the action space.One-shot technology forks and alternative routes vary which landmarks and routes are available.Source: §3.1 · pp. 2–3
- In the gulf_water example, absent dry-cooling viability makes the dry-cooling branch unreachable and determines answers to applicability, reachability, and landmark questions.Figure 1 shows the optimal plan and the eight generative-arm questions for this instance.Source: Figure 1 · p. 3, §3.1 · p. 3
- Worlds average 24,659 reachable states but remain small enough for exact search; no tested world-model-free strategy exceeds 12.0% on any question type.Gold answers come from breadth-first search, with questions dropped rather than guessed when search reaches its node cap.Source: §3.1 · pp. 3–4
3.2 The multiple-choice arm: source-grounded questions
- The multiple-choice arm uses two separately authored 120-question banks spanning six content buckets and eight skill labels.Contexts quote real-site records, published emission equations, vendor calculators, industry reports, or stated scenario assumptions; answers are graded by exact match.Source: §3.2 · p. 4
- The arms differ in content, skill definitions, prompting, and grading, so their accuracy gap is descriptive rather than evidence of a format effect.The longest-option shortcut scores 42.5% overall, and the paper also reports results on 138 questions where that option is wrong.Source: §3.2 · p. 4
4 Results
Across deployments, multiple-choice accuracy is higher than generative accuracy, but strong scores do not translate into reliable enumeration of available actions. The results also show that a state-blind list of grounded actions can outperform models on partial-credit scores.
Evaluation settings
- Eleven deployments of seven open-weight models were tested at temperature 0 without fine-tuning or tools; multiple-choice runs cover nine deployments.Groq Compound systems ran with server-side tools and are reported separately; some provider comparisons also differed in reasoning settings.Source: §4 · p. 4
Action enumeration
- No deployment exceeds 4.0% exact match on action enumeration; six GPT-OSS systems list 72–85% of state-changing actions but only 15–31% of no-ops.Even excluding no-ops from both sides, the best exact match is 24.0%.Source: §4 · p. 4, Figure 2 · p. 5
Partial-credit baseline
- A state-blind list of every grounded action scores F1 0.71 and Jaccard 0.56, above every model, motivating exact match, precision, and recall reporting.The best model scores F1 0.61 and Jaccard 0.47 on this comparison.Source: §4 · pp. 4–5
Per-arm accuracy
- 65.5–71.5% generative accuracy for six GPT-OSS deployments and 90.4–92.5% multiple-choice accuracy for five were not significantly different from each arm’s best.The reported per-arm comparisons use exact McNemar tests; Figure 2 shows deployment-level accuracy and 95% Wilson intervals.Source: §4 · p. 4, Figure 2 · p. 5
What the two arms do and do not show
- Every deployment scores higher on multiple choice, but differing content and grading mean the gap is not a format effect.Open-weight deployments above 90% on multiple choice score 0.0% on exact action enumeration.Source: §4 · p. 5
5 Limitations, impact, and pathway to climate impact
The benchmark uses small, synthetic worlds across five archetypes and measures planning rather than emissions or water quantities. The authors connect omitted options to the choices developers weigh, while leaving impact-weighted omissions for future work.
Scope
- The generative worlds are small for exact search, use synthetic metadata, and cover only five archetypes, a narrow base.A sixth archetype is paused for rework.Source: §5 · p. 5
Evaluation boundaries
Pathway to climate impact
- DCBench-440 scores neither emissions nor water quantities and does not claim that better planning by itself lowers emissions.The authors propose attaching water and emissions quantities to world branches so omissions can be weighted by what they foreclose.Source: §5 · pp. 5–6
Dual use and value beyond data centers
- The benchmark is technology-neutral, but a fixed test set can be optimized against, so the authors regenerate worlds; they also say the diagnosis applies to any agent that must enumerate options.Source: §5 · p. 6
Availability
- Code, data, and other materials are available from the authors upon request.Source: §5 · p. 6
A Screening funnel and stage-gate schematic
The schematic maps site screening to an ordered chain of commitments. Its domain design treats failed thresholds as permanent route closures and the financial investment decision as an irreversible one-shot choice.
A Screening funnel and stage-gate schematic
- The schematic is motivated by a reviewed U.S. origination pipeline, but the paper omits its confidential thresholds, company names, and deal-specific figures.The appendix presents the process only at the same generality as the domain design.Source: A Screening funnel and stage-gate… · p. 9
- After a site clears screening, it proceeds through ordered commitments, with the financial investment decision modeled as an irreversible one-shot fork.Capital is spent against a named risk and cannot be recovered by a later action, so an early gate can foreclose downstream options.Source: A Screening funnel and stage-gate… · p. 9, Figure 4 · p. 9
- Screening requires every static threshold to hold; one failed test permanently removes a route.These site-specific facts do not change through later actions and are encoded as static predicates in the generative-arm domains.Source: Figure 3 · p. 9, A Screening funnel and stage-gate… · p. 9
B Extended results
The appendix describes the result tables’ scopes and evaluation conditions, but the supplied passages do not include their cell values. The two benchmark arms are explicitly not a format comparison.
Extended results
- Table 2 reports accuracy with 95% Wilson intervals and paired exact McNemar tests against the best open-weight deployment within each arm.The caption warns that differences in content, skills, prompting, and grading mean the two arms are not a format comparison.Source: Table 2 · p. 10
- Table 3 reports generative-arm accuracy across question types, with 25 questions per type and 95% intervals up to ±19 points.It also defines the best world-model-free shortcut and an analytic shortcut bound used to select least-guessable questions.Source: Table 3 · p. 10
- Table 4 reports generative-arm accuracy by domain, with 40 questions per domain.Source: Table 4 · p. 11
- Table 7 distinguishes strict exact-match scoring for listing every applicable action from exact match after no-op actions are removed; 66% of gold actions are no-ops.It also reports precision, recall, F1, Jaccard overlap, and recall separately for state-changing and no-op actions.Source: Table 7 · p. 12
- The appendix classifies all 200 generative answers, reports inference settings and cost, and says each of five domains contributes 40 questions.Calls used temperature 0 and one attempt per question; generative prompts used two held-out tutorial examples, while multiple-choice prompts used none.Source: Table 8 · p. 12, Table 9 · p. 13, Table 10 · p. 13