MacGyver: Are Large Language Models Creative Problem Solvers?

• MACGYVER contains 1,683 verbal problems requiring human-like creativity in physical reasoning and unconventional object use.The dataset targets functional fixedness and requires both divergent exploration of object uses and convergent goal-directed solving.Source: §1 · pp. 1–2
• In creative problem-solving, GPT-4 complements an arbitrary human in certain domain-specific settings, such as fixing a hole on the wall, but humans’ collective wisdom remains stronger overall.Source: §1 · p. 2
• LLMs commonly propose physically infeasible, unnecessary, or incorrect actions, hallucinate unavailable tools, or violate stated constraints.The benchmark explicitly includes unsolvable problems, for which agents are expected to identify infeasibility and justify it.Source: §1 · p. 2
• Iterative self-reflection and divergent-convergent prompting are proposed to improve LLM creativity and physical-related reasoning.The strategies respectively verify and revise solution steps, then explore possible tool uses before selecting a solution.Source: §1 · p. 2
• The work evaluates creative problem solving in real-world scenarios involving innovative object usage and physically feasible actions.Source: §1 · pp. 1–2

2 MACGYVER Dataset

• The dataset pipeline combines GPT-4 generation with human verification and pairs each problem with a human-provided or verified solution.The final dataset contains solvable problems with feasible step-by-step answers and unsolvable problems with explanations of infeasibility.Source: §2 · p. 2, §2.2 · p. 3

2.1 Progressive Problem Refinement for Dataset Creation

• Progressive refinement removes straightforward solution paths by adding object-property constraints, then expands the search space with distracting objects.Human assessments found that both refinement steps increase challenge and creativity requirements for GPT-4.Source: §2.1 · p. 2, Figure 2 · p. 3

2.2 Human Verification Process

• Human verifiers judge solvability, ambiguity, and whether efficient solving requires unconventional use of objects; agreement measured by Cohen’s Kappa is 0.67 and 0.77 for the two tasks.Source: §2.2 · p. 3
• 78% of the 1,683 created problems are solvable, 22% are unsolvable, 7% are discarded as ambiguous or contradictory, and 82% of solvable problems require unconventional tool use.Source: §2.2 · p. 3, Table 1 · p. 3

2.3 Diversity Control and Check

• The collection uses more than 50 location and activity tags and identifies 3,800 unique tools, with long-tailed affordance frequencies indicating broad diversity.Holding and covering are the most frequent affordance types, followed by tying or connecting and cleaning.Source: §2.3 · p. 3

3 Assessing the Task Difficulty

• Human evaluation distinguishes feasible efficient, feasible inefficient, infeasible, and incorrect solvability judgments for solvable problems, plus justified or unjustified judgments for unsolvable problems.This evaluation addresses the limits of automatic assessment for unconventional physical problem solving.Source: §3 · p. 4
• GPT-4 had an 18.9% likelihood of providing an efficient solution and a 37.5% likelihood of providing an infeasible solution.Human annotators evaluated answers on the full MACGYVER dataset using feasibility, efficiency, solvability status, and justification.Source: §3 · p. 4
• Figure 4 separates GPT-4 performance on 1,306 human-judged solvable problems from 377 human-judged unsolvable problems, including whether unsolvability is identified for the right reason.The figure’s categories distinguish correct solvability judgments from the quality of their justifications.Source: Figure 4 · p. 4
• GPT-4 often proposes solutions to inherently unsolvable problems and struggles to judge whether problem descriptions are concrete enough to resolve.The authors relate this overconfidence partly to RLHF’s emphasis on helpfulness and to ambiguity in problem descriptions.Source: §3 · p. 4

4 Benchmarking Humans and LLMs

• The benchmark samples 323 problems and compares PaLM2, Claude2, Llama2, GPT-3.5, GPT-4, and humans.Human participants supplied independent solutions, while machine responses used standardized prompting and sampling; GPT-4 additionally produced multiple responses for fair comparison.Source: §4 · p. 4, §4.1 · p. 4, §4.2 · p. 5

4.3 Human Evaluation

• Human annotators classify answers by feasibility, efficiency, solvability recognition, justification quality, and degrees of incorrectness.The six fine-grained categories are A–F, with A–C representing correct response types and D–F incorrect ones.Source: §4.3 · p. 5, §4.4 · p. 5

4.4.1 Performance with Single Effort

• Most recent LLMs achieve only a 35% to 42% chance of success, while an arbitrary average person reaches a 65.1% total correct rate.GPT-4 and Claude2 are the strongest tested LLMs but still underperform the human comparison.Source: §4.4.1 · p. 5, §4.4 · p. 5, Figure 5 · p. 5
• LLM families differ in solvability judgments: some propose solutions to unsolvable problems, while others conservatively reject constrained problems that remain solvable.Among Llama2 variants, larger models identify solvability better, but scale alone does not significantly unleash creative problem solving.Source: §4.4.1 · pp. 5–6

4.4.2 Performance with Multiple Efforts

• The best of four human answers clearly leads to near-perfect performance, while humans’ majority correctness is 79.3% versus 73.3% for GPT-4.The best-answer score serves as an upper bound based on the individual best response for each problem.Source: §4.4.2 · p. 6
• Humans can be slightly worse than GPT-4 at producing a correct solution, especially an inefficient one, but humans outperform GPT-4 overall because GPT-4 rarely identifies unsolvable problems correctly.Collective human responses combine complementary expertise and wisdom.Source: §4.4.2 · p. 6

5.1 Humans have higher variance than LLMs.

• Humans tend to either solve a problem nearly perfectly or fail totally, whereas GPT-4’s answers more often fall into partially wrong, inefficient, or mostly wrong categories.GPT-4’s broader information aggregation is paired with weaker understanding of tool affordances and action consequences.Source: §5.1 · p. 6

5.2 Humans possess better general everyday knowledge, but less domain-specifically.

• Humans outperform GPT-4 on familiar categories such as household and personal life, while GPT-4 performs better on domain-specific categories such as gardening, farming, and fishing.Everyday-category outliers where humans perform poorly include demonstrating refraction without a prism and making a sundial.Source: §5.2 · pp. 6–7, Figure 7 · p. 7
• The complementary strengths of humans and AI suggest that collaborative approaches may be effective for out-of-the-box problem solving.The proposed complementarity follows from humans’ everyday familiarity and GPT-4’s domain-specific advantages.Source: §5.2 · pp. 6–7, §5.1 · p. 6

6.1 Error Analysis for GPT-4

• Wrong tool usage accounts for 42.4% of GPT-4’s analyzed errors, followed by failure to achieve the goal at 17.7%.Using unavailable tools and violating constraints together account for 26.4% of errors.Source: §6.1 · p. 7
• GPT-4 is highly prone to proposing physically infeasible, unwanted, or wrong actions, with wrong tool usage accounting for 42.4% of errors and failures to realize consequences and tool affordances in context.Source: §6.1 · p. 7

6.2 Improving LLMs via Prompting

• Iterative step-wise reflection repeatedly checks each generated step for physical feasibility and affordance, then modifies the solution as needed.Divergent-convergent thinking first enumerates object affordances and their usefulness before generating goal-directed steps.Source: §6.2 · pp. 7–8, Figure 8 · p. 8
• For GPT-4, Self-Reflect reduces infeasible solutions by 9.7% versus 4.3% for Div-Conv Thinking, while Div-Conv improves efficient solutions by 6.5% versus 2.2%.Both methods reduce infeasible solutions, but their strongest benefits differ between feasibility checking and efficient solution generation.Source: §6.2 · p. 8, Figure 9 · p. 8
• Div-Conv Thinking improves feasibility and efficiency across GPT-4, Claude2, and Llama2-13b, whereas Self-Reflect does not improve Claude2 or Llama2-13b.The authors interpret this pattern as suggesting smaller models may lack GPT-4’s inherent capacity for self-reflection about physical consequences.Source: §6.2 · p. 8

• Everyday innovation is under-explored in AI creativity research despite representing creative activities ordinary people commonly perform.Source: §7 · p. 8
• The benchmark contributes a distinct everyday-creativity setting spanning household, training, and outdoor-sports problems, complementing existing reasoning and creativity domains.Source: §7 · p. 8
• MACGYVER addresses functional fixedness by requiring familiar objects to be used in novel ways; over 82% of solvable problems demand unconventional tool use.Source: §7 · pp. 8–9
• Prior physical-reasoning benchmarks examine physical concepts, attributes, or commonsense situations, whereas MACGYVER emphasizes affordances, unnecessary-tool elimination, and multi-object combinations.Source: §7 · p. 9

• Everyday innovation is less studied than mathematical, logical, or writing problems despite requiring multi-step planning, search-space pruning, and unconventional use of multiple tools.Source: §8 · p. 9
• The paper introduces MACGYVER as a quality- and diversity-controlled benchmark covering everyday innovation across household, training, and outdoor-sports domains.Source: §8 · p. 9
• The study compares LLM and human performance, finding complementary LLM capabilities in some domain-specific settings while highlighting failures in physically feasible action planning.Source: §8 · p. 9
• Two prompting methods improve LLM reasoning ability by addressing the identified feasibility and constraint-related failures.Source: §8 · p. 9

• Future embodied agents could interact with physical or simulated worlds and receive environmental feedback to reduce mistakes in fictional physical reasoning.Source: §9 · p. 9
• Future planning and reasoning strategies should enhance physical knowledge and spatial understanding while reducing hallucination.Source: §9 · p. 9
• Automatic evaluation methods are needed to assess claims and physical consequences in candidate solutions for this complex reasoning task.Source: §9 · p. 9

Limitations

• Human evaluation is relatively slow and costly because standardized automated metrics correlate weakly with human judgments of creative problem solving.BLEU-2/BLEU-3 has Pearson correlation 0.07, while BertScore has 0.12.Source: Limitations · p. 9
• The authors release 4,100 graded answer-annotation pairs to support future efficient and automatic evaluation frameworks.Source: Limitations · p. 10
• The dataset’s GPT-4-generated problems may inherit the underlying model’s biases and tendencies, including cultural nuances from predominantly English-speaking training data.Source: Limitations · p. 10

A.1 Comparing GPT-4 with Humans

• Humans find more efficient solutions for familiar tasks, whereas GPT-4 can solve specialized gardening problems that humans fail to answer.The examples illustrate complementary strengths rather than uniform superiority by either group.Source: §A.1 · p. 12, Figure 11 · p. 12
• Humans are more proficient with a broader set of presented tools, including magnifying glasses, rocks, calculators, and knives, while AI is stronger with mirrors, gloves, and scarves.The analysis scores tools based on attempted usage or justified non-use, not only tools appearing in proposed solutions.Source: §A.1 · p. 12, Table 4 · p. 13

A.2 Benchmark Results

• Across the benchmark, all benchmarked LLMs lagged behind human performance despite varying characteristics across the response aspects evaluated.Source: §A.2 · p. 12

A.3 Enhancing LLMs’ Problem Solving

• Divergent-convergent thinking benefits Claude2 and Llama2 across evaluation dimensions, whereas self-reflection does not reduce infeasible answers for either model.Llama2 often rejects and blindly revises its original answer, suggesting weaker faithful physical reasoning in these tests.Source: §A.3 · pp. 12–13, Figure 12 · p. 13

C.1 Human Verification Process

• Problems are verified for solvability and required unconventional object use by three human annotators, with Cohen’s Kappa scores of 0.67 and 0.77.The dataset pairs solvable problems with feasible step-by-step solutions and unsolvable problems with explanations of infeasibility.Source: §C.1 · p. 13, §C.2 · p. 13

C.2 Collecting Gold Solutions

• GPT-4-generated gold solutions enter the dataset only when all three verifiers agree; otherwise, human workers provide a solution or infeasibility justification.This combines LLM generation with human validation to reduce annotation effort while retaining verification.Source: §C.2 · pp. 13–14

C.3 Does the data collection pipeline result in progressively challenging problems?

• As problems are refined from iteration 1 to 3, feasible and efficient GPT-4 solutions decrease while infeasible answers increase.This indicates that iterative refinement makes the same type of problems increasingly challenging for GPT-4.Source: §C.3 · p. 14, Table 7 · p. 15

C.4 Diversity Control

• Diversity is controlled with more than 50 location and activity tags, batch generation, and semantic filtering of newly generated problems.The tags span indoor, neutral, and outdoor activities, while batch generation was found to produce higher diversity.Source: §C.4 · p. 14, Table 6 · p. 15
• The dataset contains more than 3,800 different tools, whose long-tailed distribution is treated as evidence of substantial tool diversity.Table 8 samples 16 frequently presented tools and records their affordances and frequencies.Source: §C.4 · p. 14, Table 8 · p. 16

D.1 Benchmark Setup

• Benchmarking collects human judgments and approximately four GPT-4 solutions per problem, matching the number of human responses across problems.GPT-4 responses use separate API calls, manually designed instructions, nucleus sampling, and top-one sequence selection.Source: §D.1 · p. 15

D.2 Analyzing Results

• Three MTurk workers annotate each answer with average Cohen’s Kappa agreement of 0.71, while efficiency disagreements are resolved conservatively using the worse label.Majority voting remains the rule for other cases, and the modification is reported to produce more accurate labels.Source: §D.2 · p. 15

D.3 The Prompts for Improving LLM’s Ability

• The appendix documents dedicated prompts for self-reflection and divergent-convergent thinking, alongside interfaces for data collection, human responses, and benchmark evaluation.The interfaces cover solvability judgments, step-by-step solutions, infeasibility explanations, and benchmark annotation.Source: §D.3 · p. 16, §D.4 · p. 16, Figure 15 · p. 17