CreativityBench: Evaluating Agent Creative Reasoning via Affordance-Based Tool Repurposing

Benchmark motivation

• CreativityBench operationalizes creative tool use as discovering unobvious but physically plausible affordances enabled by an object’s parts and attributes.The benchmark distinguishes grounded affordance discovery from random exploration, hallucinated actions, or coarse object-level plausibility.Source: §1 · p. 2

Affordance knowledge base

• The benchmark is built from a structured affordance knowledge base containing over 4K entities and 150K+ affordance annotations linking objects, parts, attributes, and uses.The knowledge base supports grounded task sampling, trajectory construction, training, and evaluation.Source: §1 · pp. 2–3

CreativityBench

• Using this knowledge base, the authors generate 14K diverse tasks by reverse-engineering creative solution trajectories that require non-obvious, physically grounded affordance reasoning.The task-generation process is designed to avoid surface-level object matching.Source: §1 · p. 3

Empirical analysis

• Models often identify plausible tools but fail to ground their use in specific parts or attributes, causing a performance drop of over 60%.Exact physical grounding is identified as a severe bottleneck in creative tool use.Source: §1 · p. 3
• Logical reasoning ability does not reliably predict creative affordance discovery: GPT-5-family models are outperformed by Qwen3-32B in novel tool discovery.The results indicate a dissociation between analytical reasoning and creativity.Source: §1 · p. 3
• Creative tool-use performance quickly saturates with model size, degrades for rare long-tail repurposing, and gains minimally from temperature changes, structured Chain-of-Thought, or interactive evaluation.These interventions can instead exacerbate hallucinations or premature commitment to incorrect hypotheses.Source: §1 · p. 3

2.1 Creativity in Language Models

• Existing creativity benchmarks span psychological assessments, object-repurposing tasks, simulated escape rooms, and multimodal or embodied settings, but often emphasize planning, reasoning, or perception rather than fine-grained physical affordance mechanisms.Psychological tests can be prompt-sensitive, costly, noisy, and difficult to scale; scenario-driven or prompt-generated tasks do not explicitly ground creativity in object components and their affordances.Source: §2.1 · pp. 3–4

2.2 Affordance and Physical Reasoning

• CreativityBench addresses these gaps by grounding affordances in a hierarchy linking entities, parts, physical and state attributes, and affordances, enabling evaluation of mechanism-based creative tool use under constraints.The benchmark tests whether models can identify non-obvious functional affordances and reason about the underlying physical mechanism enabling them.Source: §2.1 · p. 4, §2.2 · p. 4
• Physical reasoning benchmarks and robotics affordance models evaluate object attributes, canonical actions, or object–action relationships, but generally do not explain how structural and physical properties of components generate affordances.This limitation applies across PIQA, PROST, NEWTON, and affordance-based robotics and vision–language manipulation work.Source: §2.2 · p. 4
• SYNTHIA decomposes objects into parts and associated affordances, yet primarily represents conceptual part–affordance associations without explicitly modeling the physical attributes that enable them.For example, it does not explicitly encode how sharpness enables cutting.Source: §2.2 · p. 4

Experimental setup

• The comparison tests direct prompting against a scaffold covering tool inventory, part decomposition, physical properties, affordances, step justification, and constraint validation.GPT-4.1-mini generated solutions and GPT-5.2 judged them on 100 MacGyver creative tool-use tasks.Source: §3 · pp. 4–5

Evaluation

• Evaluation separates Correctness, Feasibility, Physical Grounding, Constraint Coverage, Tool Usage, and Creativity because task success alone cannot distinguish routine use from creative, feasible reinterpretation.Outputs receive absolute 1–5 scores and relative comparisons between prompting strategies.Source: §3 · p. 5

Preliminary results

• Structured CoT increases Feasibility from 3.44 to 3.52, Physical Grounding from 3.44 to 3.54, and Tool Usage from 4.22 to 4.31, but performs worse on Creative Reasoning.Relative evaluation likewise favors CoT for Physical Grounding (47% vs. 41%) and Tool Usage (38% vs. 28%).Source: §3 · p. 5, Figure 2 · p. 5

Implications

• The results indicate that structured reasoning is not the main limitation; models instead lack grounded affordance knowledge that can be flexibly recombined for creative reinterpretation.MacGyver is not explicitly affordance-structured, and LLM-as-judge evaluation complicates rigorous creativity measurement, motivating a part-level, attribute-grounded benchmark and affordance knowledge base.Source: §3 · p. 5

4 CreativityBench

• Creative tool use requires identifying actionable affordances enabled by an entity’s attributes and combining them toward a goal, rather than matching tasks to tools by semantic labels.The benchmark therefore treats affordances as its organizing principle and grounds them in object structure, materials, interfaces, constraints, and accessible resources.Source: §4 · pp. 5–6

4.1 Affordance Knowledge Base Construction

• The affordance knowledge base contains approximately 4K entities, 26K parts, 288K physical attributes, 125K state attributes, and 157K annotated affordances.It represents each entity through the hierarchy entity → part → attributes → affordances, with affordances annotated alongside conditions, recipients, failure modes, and typicality levels.Source: §4 · p. 6, §4.1 · p. 7

4.2 Benchmark Task Sampling

• CreativityBench contains approximately 14K approximately balanced tasks spanning diverse scenarios, affordance types, and distractor configurations.Tasks are reverse-engineered from sampled affordances, then verified against alternative parts and entities so the gold solution remains the most plausible.Source: §4.2 · pp. 9–10

5.1 Experimental Settings

• All models are evaluated on the full 14K tasks by selecting the relevant entity and part, filtering distractors, and specifying how the selection accomplishes the goal.Objective metrics are Entity Correct Rate and Gold Correct Rate; gold cases additionally receive LLM-as-Judge scores for constraint coverage, physical grounding, action feasibility, and overall correctness.Source: §5.1 · p. 10

5.2 Main Results

• Entity correctness reaches 0.5149, but gold correctness falls to 0.1910, showing that models often identify useful objects while missing the specific parts and attributes enabling correct affordances.The relative decrease exceeds 60%, and some models show a 0.35 entity-to-gold gap.Source: §5.2 · pp. 10–11
• Action feasibility averages 3.5860 versus 3.2003 for physical grounding and 3.0327 for prediction correctness, while recipient-constraint coverage is lowest at 2.8026.Use and environmental constraint coverage are higher at 3.1780 and 3.1860, indicating that plausible actions often omit target-object conditions or preparation steps.Source: §5.2 · p. 11
• Qwen3-32B achieves 0.2588 gold correctness and 0.6246 entity correctness, outperforming GPT-5.2’s 0.1819 gold correctness despite GPT-5.2 leading most reasoning-related metrics.This contrast separates creative affordance discovery from evaluating constraints and producing grounded reasoning after the correct direction is identified.Source: §5.2 · p. 11
• Scaling gains diminish beyond smaller model sizes: Qwen gold correctness rises from 0.1882 at 4B to 0.2483 at 14B, then only 0.2588 at 32B.GPT-5 Nano to Mini improves roughly 40% (0.1192 →0.1687), but Mini to GPT-5.2 gains about 7%, while Gemini gains about 9%.Source: §5.2 · pp. 11–12

6.1 How does Gold Commonality Affect Performance?

• Performance rises with gold-affordance commonality: larger clusters and emergency levels 3–5 are easier than rare clusters and levels 1–2.Rare, long-tail affordances consistently produce the lowest scores, supporting the validity of the commonality annotations.Source: §6.1 · p. 13

6.2 How does Distraction Severity Affect Performance?

• More distractors consistently reduce performance, but similar-affordance distractors score higher than dissimilar ones because they can cue the relevant affordance.With dissimilar distractors, performance steadily declines; with similar distractors, it can partially recover at nine entities.Source: Figure 7 · p. 13, §6.2 · p. 14

6.3 How do Inference Settings Affect Performance?

• Higher sampling temperature does not reliably improve creative tool use: it often degrades smaller-model performance by increasing hallucinated entities and parts, while larger models may improve mildly.The benchmark rewards grounded affordance reasoning rather than undirected output diversity.Source: Figure 9 · p. 14, §6.3 · p. 15
• Interactive evaluation substantially reduces performance because models inspect few entities and commit prematurely, whereas structured CoT yields only modest changes, typically within around 5% for Qwen models.Correct predictions usually involve inspecting the gold entity, while completely incorrect predictions have gold inspection rates below 20% for most models.Source: §6.3 · pp. 15–16

6.4 How are auxiliary metrics affected?

• Auxiliary metrics similarly show that rare affordances and dissimilar distractors hinder physical grounding, while similar distractors improve grounding and prediction correctness.Models attend more to use conditions for common affordances, indicating that rare affordances require conditional reasoning they often fail to capture.Source: §6.4 · pp. 16–17

6.5 Error Analysis

• Gold solutions win over 95% of judged wrong-tool cases, while action feasibility falls by nearly 0.6 on average and open-source models suffer larger grounding and constraint-coverage losses.Predictions with both the entity and part wrong are judged slightly worse than predictions with only the part wrong; these metrics are not all directly comparable with the main table.Source: §6.5 · pp. 18–19

6.6 Attribution Analysis

• Physical invalidity dominates failure attributions, with affordance mismatch the most common fine-grained cause among sampled model failures.The attribution taxonomy also includes practical infeasibility, risk or constraint mismatch, and comparative inferiority.Source: §6.6 · p. 19, Figure 11 · p. 20
• Physical invalidity dominates failures, particularly affordance mismatches caused by selecting objects whose geometry, materials, or mechanics cannot support the intended function.This indicates insufficient grounding in candidate tools’ physical properties rather than merely poor planning.Source: §6.6 · p. 20
• Hallucinated affordances are also frequent, as models optimize for abstract task functions while failing to verify whether objects realistically possess the required capabilities.Such predictions may sound plausible but depend on imagined affordances.Source: §6.6 · p. 20
• Practical infeasibility and risk or constraint mismatches remain substantial secondary causes, whereas comparative inferiority is relatively uncommon.Most failures therefore reflect physically or practically mismatched solutions rather than merely weaker alternatives.Source: §6.6 · p. 20

6.7 Human Study

• Humans achieved 0.146 gold correctness and 0.450 entity correctness, both below the best-performing model, while gold solutions received positive ratings for groundedness, feasibility, constraint reasonableness, and creativity.The study used 10 STEM-background annotators, and average gold persuasiveness achieved 63.0% inter-annotator agreement.Source: §6.7 · p. 21
• Human performance was less sensitive than model performance to cluster size, gold affordance level, and distractor count, whereas models were more affected by physical-grounding difficulty and uncommon correct affordances.Humans were limited more by navigating dense candidate spaces than by changes in individual design factors.Source: §6.7 · p. 21, Figure 12 · p. 21

Difference of Creativity and Hallucination

• CreativityBench studies structured creativity: novel tool uses must connect goals to non-obvious but physically plausible object affordances rather than arbitrary or hallucinated solutions.This grounding distinguishes the benchmark from creative writing, open-ended design, and research ideation while highlighting practical constrained problem-solving for agents.Source: §7 · p. 21

Toward Physical-Textual Dual Reasoning and Foresight Governance

• Models can propose plausible-sounding affordance uses yet miss their mechanical consequences, motivating physical-textual dual reasoning that predicts object states before execution.The proposed loop could filter candidates by feasibility, safety, and side effects, then support benchmarks for consequence prediction, risky-hypothesis rejection, and plan revision (Qian et al., 2026).Source: §7 · pp. 21–22

Enhancement of Model Creativity

• Current reinforcement-learning objectives may sharpen output distributions and suppress the structured diversity required for creative problem solving.Majority-vote pseudo-labeling in TTRL (Zuo et al., 2025) may reduce diversity despite improving analytical intelligence, while rewarding unlikeliness (He et al., 2025) remains untested for creative problem solving.Source: §7 · p. 22
• The affordance knowledge base could support thousands of grounded creative trajectories, enabling models to learn creative reasoning from data rather than outcome supervision alone.This positions the KB as both a benchmark resource and a potential training substrate for broader exploration.Source: §7 · p. 22

8 Conclusion and Future Work

• CreativityBench reveals that current language models struggle to ground creative solutions in the correct part, attribute, and physical mechanism, despite stronger reasoning, larger scale, or CoT.The findings characterize creative intelligence as distinct from analytical reasoning or action planning, requiring grounded comparison, flexible affordance recombination, and attention to physical constraints.Source: §8 · p. 22
• Future work should extend CreativityBench from static text to multimodal, interactive, and embodied environments while developing objectives that encourage grounded exploration and diverse, physically plausible affordance discovery.The benchmark is intended both as an evaluation testbed and as a resource for training grounded creative reasoning.Source: §8 · p. 22

A.1 Why CreativityBench Matters

• The benchmark evaluates whether models identify the object and physical property that make a non-obvious solution usable, rather than merely producing a plausible verbal plan.This targets a practical but under-measured capability for agents operating under constraints, limited resources, or unforeseen situations.Source: §A.1 · p. 26
• CreativityBench moves beyond object-level plausibility by requiring models to localize relevant parts, connect attributes, and recover the affordance mechanism enabling success.The largest performance drop at this finer grounding level suggests the benchmark exposes a previously under-tested weakness rather than repackaging commonsense tasks.Source: §A.1 · p. 26
• Its structured affordance knowledge base links entities, parts, attributes, and affordances, enabling scalable task construction with interpretable solution paths and systematic failure analysis.Failures can be diagnosed by whether models selected the wrong entity, part, attribute basis, or use condition, improving reproducibility over open-ended creativity evaluations.Source: §A.1 · p. 26
• For future models and embodied agents, the benchmark highlights that robust problem solving depends on recognizing latent affordances under constraints, not merely following canonical object functions.It provides a testbed for studying interactions among grounded knowledge, comparative search, and flexible recombination in creative problem solving.Source: §A.1 · p. 26

• The benchmark uses an explicit ontology and LLM assistance to scale structured construction, making task generation systematic, auditable, and extensible rather than unconstrained.The ontology links entities, parts, attributes, affordances, and usage conditions.Source: §A.2 · p. 26
• Tasks require models to connect goals, parts, attributes, and usage conditions into non-obvious, physically grounded, constraint-compatible functions beyond retrieving familiar object-use associations.This mechanism-sensitive requirement explains the gap between coarse entity selection and full gold selection.Source: §A.2 · p. 26
• Single-gold evaluation is a measurement choice: verification and cross-entity comparison filter tasks so one solution path is sufficiently supported for rigorous model comparison.This design clarifies measurement without claiming that real-world creativity has only one valid solution.Source: §A.2 · pp. 26–27
• Human performance on the strict gold metric is interpreted cautiously because textual descriptions remove perceptual shortcuts, turning the benchmark into a demanding test of symbolic affordance inspection.The setting therefore diagnoses grounded affordance identification rather than everyday embodied improvisation.Source: §A.2 · p. 27
• The focused household domain improves control over entity distributions, affordance granularity, and distractors, supporting internal validity while limiting claims to one important subtype of creative problem solving.The benchmark is presented as principled and scalable, not as a complete evaluation of creativity.Source: §A.2 · p. 27
• Across proprietary and open-source models, recurring structural weaknesses matter more than leaderboard ordering: entity-to-part accuracy drops, physical grounding lags action plausibility, common affordances dominate, and interventions help little.These consistent patterns support the benchmark’s diagnostic contribution and indicate that current models remain substantially weaker at this capability than general reasoning progress suggests.Source: §A.2 · p. 27

B.1 Core Prompt Details

• The directive prompt requires feasible, practical solutions using only available items, respecting physical constraints, optionally inventing tools from listed items, and explaining partial plans when completion is impossible.Its structured output includes solvability, solution steps, final solution, used tools, and constraint handling.Source: §B.1 · pp. 27–28
• The Chain-of-Thought prompt adds a required procedure that identifies task goals, inventories tools, derives part-level affordances, builds referenced action steps, validates constraints, and minimizes unnecessary actions.The output schema records tool parts, inferred physical properties, affordances, reasoning steps, constraint handling, and a creative-reasoning summary.Source: §B.1 · pp. 28–29
• Correctness is scored from 1–5 according to whether a candidate achieves the task goal, allowing solutions that differ from the ground truth but distinguishing partial from complete, logically consistent solutions.A score of 1 is completely wrong, 3 is partially correct or assumption-dependent, and 5 fully achieves the goal unambiguously.Source: §B.1 · p. 29
• Feasibility and physical grounding evaluate whether solutions are executable under constraints and whether they correctly use object properties, geometry, and mechanics.The rubrics range from constraint-violating or physically nonsensical solutions to plans that are realistically executable and exploit relevant attributes and principles such as leverage, friction, torque, and buoyancy.Source: §B.1 · pp. 29–30
• Constraint coverage and tool usage assess whether every stated constraint is explicitly handled and whether every tool is available, purposeful, and used without introducing phantom objects.Both dimensions use 1–5 rubrics, with top scores requiring complete constraint acknowledgment and perfect fidelity to the listed tools.Source: §B.1 · p. 31
• The prompts define creative reasoning as non-obvious tool repurposing or combination and evaluate solutions for correctness, feasibility, physical grounding, constraint coverage, tool usage, creativity, and overall quality.They require partonomies to identify essential, non-overlapping components covering the whole object, generate diverse and internally consistent physical attribute combinations, and annotate state attributes.Source: §B.1 · p. 32, §C.3 · pp. 36–38

B.2 Textual v.s. Visual Grounding in Creative Tool Use

• The text-versus-visual experiment uses a 2×2 GPT-4.1-mini ablation crossing Text-only versus Vision+Text inputs with Direct versus Affordance-CoT prompting.Text-only inputs specify goals, tools, and constraints, whereas vision-plus-text inputs require inferring object properties and visible constraints from images, isolating grounding, scaffolding, and interaction effects.Source: §B.2 · p. 33

C.1 Overview of Annotation Stages

• The annotation pipeline builds complete entity records through sequential entity grounding, part decomposition, physical and state characterization, affordance annotation, and consistency checking.It begins with eight indoor scenarios and retains entities with complete part coverage, producing datasets of entities, parts, attributes, and affordances.Source: §C.1 · p. 34
• Functional affordances are annotated per part and entity variant with use and environment conditions, attribute evidence, graded uses, recipient conditions, examples, and failure cases.State variants are composed before affordance annotation, enabling condition-aware and recipient-aware descriptions grounded in explicit physical and state attributes.Source: §C.1 · p. 35
• The annotation process uses strict stage-specific schemas and GPT-5.2 generation under human supervision, with approximately 98% automatic-judge pass rate and 95% human-review pass rate on a 0.1% sample.Quality is assessed for commonsense validity, groundedness, and factual consistency across the pipeline.Source: §C.1 · p. 35

C.3 Core Prompt Details

• Outputs must follow a JSON array format, with reasoning first, and include structured attribute annotations identifying whether each attribute is physical or state-derived and whether it is visually conveyed or requires text.The template also requires concrete example recipients that satisfy the recipient condition and reflect diverse real-world cases.Source: §C.3 · pp. 39–40
• Each affordance must specify preparation and environmental requirements, enabling attributes with visual-versus-text justification, the function, suitability level, recipient constraints, concrete recipients, and failure cases.Failure analysis covers access, environment, recipient compatibility, user skill or force, and other practical limitations.Source: §C.3 · pp. 39–40
• The prompt calibrates affordance levels by considering accessibility, detachment requirements, irreversibility or damage, practical likelihood, and whether the use is normal or emergency-only.Emergency ratings range from rarely suitable and difficult to access to highly suitable, easy, natural, and effective uses.Source: §C.3 · p. 39
• The prompt requires six diverse, non-overlapping affordances—one Normal 0 and one for each Emergency level 1–5—grounded only in the part’s provided attributes.Uses must be physically plausible, scenario-agnostic, practically meaningful, recipient-directed, and completable solely by the specified part rather than other entity components.Source: §C.3 · pp. 39–40

D.1 Overview of Sampling Stages

• The pipeline enforces a preferred entity-part-action gold affordance, controlled semantic distractors, and grounded first-person narratives with judge-checkable constraints.These properties define the benchmark task-generation target before sampling and assembly proceed.Source: §D.1 · p. 41
• Gold affordances are clustered and stratified by semantic compactness, cluster-size range, and normal-versus-emergency level, making benchmark composition controlled and analyzable.Clustering uses Text-Embedding-3-Large with complete-linkage hierarchical clustering, adaptive thresholds, and a hard diameter cap; each scenario yields roughly 3.2K–3.7K clusters.Source: §D.1 · p. 41
• Tasks hide the sampled entity, part, and mechanism while specifying a concrete recipient and condition, preserving a goal-centric rather than answer-leaking question.The task is instantiated as a first-person description asking what to use or how to use it.Source: §D.1 · p. 41, §D.3 · p. 43
• Gold candidates undergo intra-entity dominance checks, while near and far entity pools support distractors ranging from highly confusable to clearly different.Entities are filtered through inter-part comparisons and grouped into dissimilar, mixed, and similar-but-not-better difficulty tiers.Source: §D.1 · pp. 41–42
• Final assembly packages annotations, entities, scene items, first-person environments, and four-step solutions whose fields align with conditions and affordance mechanics for direct verification.Stage-specific JSON-constrained prompts enforce groundedness, comparability, and consistency; a sampled 0.1% quality assessment found 98% human-check success.Source: §D.1 · p. 42

D.2 Core Hyperparameters Details

• Core hyperparameter tables specify controls for gold sampling, clustering, distractor selection, final-task composition, inference settings, and the current generation scale.The supplied passages identify these tables and their scopes but do not provide cell values.Source: Table 12 · p. 42, Table 13 · p. 42, Table 14 · p. 43

D.3 Core Prompt Details

• The core prompts require concrete recipient-centered situations, forbid affordance or entity leakage, and judge target parts using attributes, conditions, comparative tradeoffs, and explicit difficulty scores.Final packaging produces scene items, an environment, and four grounded solution steps covering recipient preparation, tool setup, environmental setup, and affordance application.Source: §D.3 · pp. 43–44

E Experiment Details

• Most models use temperature 0 for deterministic evaluation, while GPT-5-Mini and GPT-5-Nano use default sampling because they lack adjustable temperature.All models receive a maximum output length of at least 16K tokens.Source: E Experiment Details · pp. 45–46
• The task prompt requires models to identify the best entity part, explain their reasoning, and return exact entity and part names plus detailed usage instructions in JSON.The prompt exposes the environment, available entities with full descriptions, and other scene items.Source: E Experiment Details · p. 46
• Evaluation reports gold correct and entity correct alongside six subjective metrics scaled from 0–2 to 1–5.Subjective metrics assess constraint coverage, physical grounding, and related properties of the generated “how to use” explanation, and are applied in the main table only to Gold Correct answers.Source: E Experiment Details · p. 46
• Gemini-3.1-Flash-Lite judges subjective quality by comparing each predicted instruction with the gold affordance and gold solution under a strict embodied-task rubric.The judge receives the gold affordance JSON, gold solution JSON, and predicted how_to_use field.Source: E Experiment Details · p. 46
• The rubric scores whether predicted instructions cover environment, tool-part use, and recipient conditions, each using 0/1/2 or NA when the corresponding gold condition is absent.Condition scores distinguish missing, partial, and well-covered prerequisites, with recipient assumptions also marked false when unmet.Source: E Experiment Details · pp. 46–47

F.1 Inference setting’s impact on performance

• Section 6.3 evaluates 10% of the dataset, retaining about 1.4K examples while reducing the substantial cost of full and multi-turn interactive evaluation.The sampling choice balances reliable analysis against experimental expense.Source: §F.1 · p. 47
• Temperature-sampling experiments exclude GPT-5 Mini and GPT-5 Nano because their sampling temperatures cannot be controlled.Source: §F.1 · p. 47
• Full-test results reproduce the main trends: temperature sampling helps GPT-5.2 and Llama-3-70B, smaller models generally decline, interactive mode substantially reduces performance, and CoT offers no clear benefit.These findings are reported in Figure 13 and its accompanying full-set analysis.Source: §F.1 · pp. 47–48, Figure 13 · p. 48
• The full 14K-test-set results provide additional evidence that the 10% sampled subset is representative of the broader dataset.The reported full-set trends are consistent with those in the main text.Source: §F.1 · pp. 47–48

F.2 Fine-grained analysis on auxiliary metrics

• The analysis covers gold affordance cluster size, gold affordance emergency level, distractor similarity, and distractor number.These complete auxiliary-metric results are presented in Figures 14–17.Source: §F.2 · p. 48

• A 10% sample of each model’s failures is judged by Gemini-3.1-Flash-Lite at temperature zero, using criteria aligned with the main-results analysis.The protocol is designed to produce deterministic judgments across models.Source: §F.3 · p. 48
• Each failure is assessed independently for constraint satisfaction, physical grounding, and action feasibility, then compared with the gold solution and its supporting rationale.The comparative judgment determines whether the gold is preferable or whether the prediction has meaningful merits, summarizing both sides’ pros and cons.Source: §F.3 · p. 48
• The comparison judgment rates predictions using scores 1 or 2: 1 indicates meaningful merit with close trade-offs, while 2 indicates a convincing prediction that should replace gold.It considers accessibility, side effects, willingness, commonness, safety, and whether the predicted use strengthens or weakens the substitution.Source: §F.3 · pp. 48, 52
• The fine-grained rubric scores environmental, use, and recipient conditions alongside attributes grounding and action feasibility, requiring concrete evidence and lowering scores when evidence is unclear.Condition fields use 0–2 or NA, while attributes grounding and action feasibility use 0–2; reasons must quote evidence from the predicted use and relevant information.Source: §F.3 · pp. 49–50

Judge reliability

• Gemini’s primary-category judgments overlap with Qwen3-32B in 83.3% of cases and with GPT-5-Mini in 90.91%, supporting cross-model consistency.The analysis uses a randomly sampled 10% subset and fixes the judging temperature at 0.0.Source: §F.4 · p. 52

Attribution procedure

• The judge receives the gold comparison reason and assigns exactly one primary contributing factor, adding further categories only when clearly appropriate.The categorization prompt requires JSON output with a primary category, an all-categories list, and a concise reason.Source: §F.4 · pp. 52–53

Attribution taxonomy

• The taxonomy distinguishes physical or functional invalidity from practical invalidity, including hallucinated affordances, affordance mismatch, performance mismatch, destructive workarounds, and process impracticality.These categories capture incorrect object models or mechanics as well as solutions that are inaccessible, complex, or damaging.Source: §F.4 · p. 53
• The taxonomy also covers safety or constraint violations and comparative inferiority, including workable solutions that are less preferable or arguably as good as the gold solution.These categories separate genuine risks and requirement mismatches from preference-sensitive judgments.Source: §F.4 · p. 53

Per-model attribution results

• Physical invalidity is the dominant failure reason across models, with similar fine-grained category distributions reported individually in Figures 18–27.The appendix provides per-model breakdowns for GPT-5.2, GPT-5-Mini, GPT-5-Nano, Gemini, Qwen, Llama, and Ministral variants.Source: §F.4 · p. 53, Figure 18 · p. 54, Figure 19 · p. 54