PaperlayerView exact revision on arXiv

Source-linked AI summary

Advancing Creative Physical Intelligence in Large Multimodal Models

Cheng Qian, Hyeonjeong Ha, Jiayu Liu, Jeonghwan Kim, Emre Can Acikgoz, Bingxuan Li, Kunlun Zhu, Jiateng Liu, Aditi Tiwari, Zhenhailong Wang, Xiusi Chen, Mahdi Namazifar, Heng Ji

arXiv:2605.26396v2cs.AIcs.CLcs.LG

TL;DR

Current LMMs struggle to discover creative, physically feasible solutions through fine-grained visual grounding and sustained exploration. MM-CreativityBench evaluates this ability interactively, while affordance-grounded alignment trains evidence-based exploration and preference for grounded reasoning; training improves accuracy and interaction efficiency.

  • Problem

    It remains unclear whether LMMs can perform creative reasoning as an evidence-driven process grounded in object parts, geometry, materials, and physical affordances.

  • Method

    The paper introduces MM-CreativityBench and affordance-grounded alignment using structured exploration supervision, affordance knowledge, and preference optimization with hard negatives.

  • Results

    Training improves grounded accuracy and exploration efficiency: the strongest 4B variant raises gold correctness from 0.156 to 0.417 while reducing average turns from 18.92 to 6.21.

  • Takeaways & Limitations

    Grounded creativity depends on learning exploration and verification policies that inspect relevant parts, compare affordances, and reject visually unsupported reasoning.

Abstract

from arXiv · show

Large multimodal models (LMMs) have rapidly advanced in perception and reasoning; however, it remains unclear whether these capabilities generalize to discovering visually grounded solutions in open-ended environments, beyond pattern recognition. In such settings, intelligence requires more than answering well-posed questions: it involves identifying how elements in a scene can be repurposed in non-obvious yet physically feasible ways. This form of creative problem-solving is central to human intelligence, but remains largely untested in current benchmarks. To evaluate this ability, we introduce MM-CreativityBench, a benchmark for affordance-grounded creative tool use in visually rich, physically constrained environments. Each instance presents a scenario image with structured views of candidate entities and their parts, enabling fine-grained, interactive evaluation of how models iteratively inspect the scene, identify relevant affordances, and compose visually and physically grounded solutions. Our experiments show that current LMMs often fall short, not due to lack of generative capability, but because they do not sustain grounded exploration. Models often overlook relevant entities, under-examine critical parts, or hallucinate attributes not grounded in the image. Motivated by this failure mode, we propose affordance-grounded alignment, which casts creative tool use as a preference learning problem. Using Direct Preference Optimization, we encourage models to prefer attribute-affordance reasoning grounded in visual evidence over hallucinated alternatives. In addition, we incorporate supervision derived from an affordance knowledge base to guide broader entity exploration and multi-turn planning. Our results show consistent gains in selecting the correct entities and parts, while substantially reducing hallucination and grounding-related errors.

1 Introduction

The paper argues that creative tool use requires visually grounded, physically feasible affordance reasoning, which existing multimodal models and benchmarks inadequately test. It introduces MM-CreativityBench and affordance-grounded alignment to evaluate and improve evidence-driven exploration.

  • Motivation: Creative intelligence involves repurposing object parts for novel, useful functions under physical constraints.Examples include using rigid edges as scrapers or thin metal tips as levers.
  • Motivation: Existing LMMs often fail to transfer knowledge across functional similarity, physical affordance, and task context.Their reasoning may rely on word-level or pixel-level shortcuts rather than compositional understanding of physical properties.
  • Motivation: Current evaluations are largely text-centric and provide limited insight into perception-grounded creative reasoning in physical environments.Effective reasoning requires inspecting environments, identifying candidate objects, attending to relevant parts, and judging physical attributes.
  • Benchmark and approach: MM-CreativityBench evaluates constrained creative tool repurposing using scene, entity, and part-level visual evidence.Success requires identifying the correct object and part and justifying physical feasibility.
  • Findings: Less than 25% accuracy is achieved by even the strongest models, which commonly overlook relevant entities, neglect decisive parts, or hallucinate unsupported affordances.The reported bottleneck is sustaining grounded exploration linking perception, interaction, and physical plausibility.
  • Benchmark and approach: Affordance-grounded alignment uses stepwise supervision and preference optimization to improve performance, grounding, and hallucination reduction.The approach encourages evidence-based exploration and penalizes unsupported reasoning trajectories.

2 Related Work

Prior creativity benchmarks study open-ended generation, planning, and interaction, but generally underexplore fine-grained visual affordance grounding. Existing affordance reasoning work focuses mainly on canonical action feasibility rather than creative recombination.

  • Creativity in multimodal and language models: Creativity benchmarks for language and multimodal models commonly evaluate novelty, diversity, usefulness, planning, reasoning, or interaction.They include storytelling, design, ideation, tool use, object repurposing, and non-literal image understanding.
  • Creativity in multimodal and language models: These evaluations remain largely scenario-driven and underexplore how models derive novel solutions from object properties under visual grounding.They provide limited insight into fine-grained mechanisms of affordance-grounded creative tool use.
  • Affordance-grounded reasoning and alignment: Affordance reasoning research connects perception and action through physical commonsense benchmarks, robotics, embodied AI, and structured part-level representations.These approaches improve grounded perception, reasoning, manipulation, or planning.
  • Affordance-grounded reasoning and alignment: Existing affordance approaches primarily recognize canonical affordances or action feasibility rather than flexibly recombining object properties for creative tool use.The paper positions creative affordance-grounded reasoning as a distinct underexplored capability.

3 MM-CreativityBench

MM-CreativityBench constructs visually grounded creative tool-use tasks from part-level affordance structure and evaluates models through interactive scene, entity, and part inspection. Its design makes success depend on identifying the correct entity-part pair and grounding the proposed use in physical evidence.

  • Preliminary experiment: The preliminary benchmark probe uses 100 constrained creative tool-use tasks augmented with scenario images requiring visual identification of relevant objects and parts.The setup tests whether models can rely on visual evidence rather than task descriptions alone.
  • Preliminary experiment: Structured affordance-level prompting modestly improves Constraint Coverage, Tool Usage, and Creativity but yields only marginal Correctness gains.Feasibility and Physical Grounding remain limited or inconsistent.
  • Benchmark construction: The benchmark is built from a part-level affordance knowledge base containing object decompositions, physical and state attributes, and functional affordances.These annotations provide evidence for deciding whether a part supports a novel use.
  • Benchmark construction: Each task is constructed as an inverse grounding problem with a target entity-part-affordance triple and distractors selected to challenge fine-grained grounding.Distractors may share affordances or scene plausibility while lacking decisive physical or state attributes.
  • Visual grounding: MM-CreativityBench provides environment, entity, and zoomed-in part images so models can inspect progressively finer evidence.Object-level distractors may appear plausible even when the correct answer depends on a local part attribute.
  • Interactive evaluation: The interactive protocol represents trajectories through feedback, visual observations, reasoning, and structured actions for inspecting entities, inspecting parts, or answering.Each action determines which evidence the model receives next.
  • Training trajectory construction: Knowledge-guided exploration prioritizes candidate entities and pushes affordance-relevant parts for finer verification.The relevance function marks parts whose affordances are similar or relevant to the target affordance.

4 Experiment

The benchmark uses interactive inspection of scenario images, entities, and parts to test whether creative solutions are grounded in visual evidence. Results show that models often locate relevant entities but struggle with fine-grained part grounding and evidence integration; targeted SFT plus hard-negative DPO improves accuracy and exploration efficiency.

  • Evaluation protocol: The protocol lets models iteratively inspect entities or parts before explaining a creative, grounded final answer.Each step requires reasoning followed by an inspection action or final response.
  • Evaluation metrics: Gold Correct Rate requires selecting both the correct entity and part, while Entity Correct Rate requires only the correct entity.Entity Correct Rate is therefore expected to be no lower than Gold Correct Rate.
  • Base-model findings: Models often inspect useful evidence without converting it into correct answers, showing that exploration alone does not guarantee grounded reasoning.Qwen3-VL-32B achieves 0.240 final accuracy despite inspecting the gold entity in nearly all successful entity cases.
  • Base-model findings: Fine-grained part grounding is the main bottleneck: GPT-5.4 reaches 0.435 entity correctness but only 0.192 gold correctness.InternVL3.5-38B similarly reaches 0.426 entity correctness versus 0.156 gold correctness.
  • Base-model findings: More exploration is not necessarily better: Qwen3-VL-8B explores more than GPT-5.4 yet reaches the same 0.192 gold correctness.The number of interaction turns also exceeds explored entities and parts, indicating redundant exploration.
  • Training results: SFT plus hard-negative DPO raises 4B gold correctness from 0.156 to 0.417 while reducing average turns from 18.92 to 6.21.Training makes interaction more selective by gathering useful evidence earlier, avoiding revisits, and stopping when evidence is sufficient.
  • Affordance similarity: Performance often degrades as more entities share similar affordances, while average exploration turns remain largely stable.This points to weak fine-grained grounding and affordance disambiguation rather than insufficient exploration volume.

5 Analysis

Model difficulty rises when candidates share affordances, reflecting weak fine-grained grounding rather than insufficient search. Affordance-grounded training improves exploration efficiency, semantic focus, and physical affordance judgments.

  • Performance drops as affordance-similar distractors increase, while exploration remains stable, indicating that fine-grained grounding—not search effort—is the main bottleneck.
  • Higher-typicality affordances induce longer exploration but do not improve gold correctness, suggesting greater ambiguity among plausible candidates.
  • Visual information improves performance, with interactive access to recent task-relevant observations supporting grounded decisions.
  • Prompt-format changes preserve overall training trends, although pure-JSON prompting can increase exploration turns for trained models.
  • SFT+DPO reduces redundant exploration while increasing affordance-relevant similarity density, reaching part-density values of 0.594 for Qwen3-4B and 0.605 for Qwen3-8B.
  • For Qwen3-8B, SFT+DPO reduces Category A errors from 45.3% to 25.8% and affordance mismatch from 31.2% to 18.9%.

6 Discussion

The paper frames grounded creativity as an interactive evidence-gathering process rather than unconstrained novelty. It argues that training should reward exploration, verification, and mechanically consistent affordance reasoning.

  • MM-CreativityBench distinguishes physically grounded creative repurposing from unconstrained novelty, where hallucination may sometimes support exploration.
  • The results motivate training policies that inspect scenes, compare affordances, verify parts, and reject unsupported evidence rather than optimizing only final answers.

7 Conclusion

MM-CreativityBench evaluates visually grounded creative tool repurposing through interactive inspection of scenes, entities, and parts. Affordance-grounded alignment improves accuracy and exploration efficiency while targeting grounding failures.

  • MM-CreativityBench exposes grounding weaknesses by requiring interactive inspection of scenes, entities, and parts for creative tool repurposing.
  • Affordance-grounded alignment, especially with hard-negative preference signals, improves accuracy and exploration efficiency.

A.1 Why MM-CreativityBench Matters

MM-CreativityBench targets visually grounded creative tool repurposing, requiring models to search, inspect, compare, and revise hypotheses in structured visual environments. Its design also connects interpretable evaluation with targeted model improvement.

  • MM-CreativityBench measures whether models discover non-obvious, physically feasible object uses beyond recognition, visual question answering, and instruction following.
  • Its interactive protocol lets models inspect scenes, candidate entities, and zoomed-in parts before committing to a solution.
  • The benchmark isolates part-level grounding, where relevant object categories are insufficient without identifying the enabling part, attribute, or visual cue.
  • A structured affordance knowledge base supports systematic tasks, interpretable solution paths, multimodal augmentation, and diagnostic failure analysis.
  • Structured positive and contrastive negative trajectories improve exploration and attribute–affordance reasoning, making the benchmark useful for model improvement as well as evaluation.
  • The benchmark isolates capabilities relevant to adaptive agents that repurpose available objects in homes, laboratories, workshops, and other resource-limited settings.

A.2 Clarifications of Concerns

The benchmark is intentionally scoped to constrained, visually grounded tool repurposing rather than creativity in the abstract. Its controlled interface, generated scenes, single-gold targets, and affordance supervision are diagnostic choices rather than claims of full realism or algorithmic novelty.

  • MM-CreativityBench evaluates novel, physically feasible, visually evidenced object–part repurposing rather than creativity as an unrestricted ability.
  • Generated images provide controlled visual grounding by making candidate objects, inspectable parts, and affordance structure reproducible without replacing real-world embodied evaluation.
  • The benchmark prevents visual leakage by showing initial scenes rather than task execution and requiring object-specific evidence for the final answer.
  • Single-gold evaluation measures recovery of a verified solution path for consistent comparison, not the claim that alternative valid solutions cannot exist.
  • The inspection protocol abstracts segmentation and camera control to isolate whether models conduct evidence-driven creative reasoning in an inspectable environment.
  • SFT and DPO are established tools used to test whether alignment toward visually supported attribute–affordance reasoning improves a learnable failure mode.

B Preliminary Experiment

The preliminary experiment compares prompting strategies for multimodal creative tool use under explicit physical and tool-availability constraints. It evaluates practical solution quality through structured reasoning and pairwise judgments across multiple criteria.

  • The experiment compares two prompting strategies on 100 creative tool-use tasks sampled from MacGyver.
  • Scenario images are generated to let models perceive physical attributes while reasoning about creative repurposing in multimodal environments.
  • Outputs are evaluated on Correctness, Feasibility, Physical Grounding, Constraint Coverage, Tool Usage, and Creativity through pairwise comparison.
  • The solver prompt requires using only pictured items, respecting physical constraints, proposing executable steps, and avoiding unnecessary actions.
  • The required procedure progresses from defining the goal and inventorying tools to identifying part-level affordances, planning steps, and validating constraints.

C.2 Reverse Task Construction

MM-CreativityBench constructs tasks backward from verified entity–part–affordance relations, adds controlled distractors, and renders each task into an interactive hierarchy of environment, entity, and part views. This design turns creative tool use into an inverse grounding problem requiring evidence-based recovery of the correct object and part.

  • Reverse task construction: Each task begins with a verified target entity, decisive part, and supported affordance, then generates a realistic constrained problem around that relation.
  • Reverse task construction: GPT-5.4 proposes candidate task descriptions, while human annotators verify affordance necessity, uniqueness, distractor separability, and faithful visualization.
  • Reverse task construction: Distractors are selected from the same knowledge base and may be affordance-similar or scene-plausible, forcing comparison of parts and their physical attributes.
  • Formalization: The symbolic formulation T = (x, E, g) defines an inverse grounding problem in which the model recovers the gold entity and part from task needs and visual evidence.
  • Multimodal image construction: Interactive instances provide progressively finer environment, entity, and part images so models can inspect candidate objects and verify local physical cues.
  • Multimodal image construction: Textual clarification is restricted to ambiguous low-level attributes and is applied uniformly, preserving the need for visual inspection without revealing the target affordance or solution.

D Training Trajectory Construction Details

The trajectory construction models creative tool use as interactive, evidence-seeking exploration from scene entities to relevant parts and a grounded final answer. Positive trajectories use knowledge-guided structure, while preference data contrasts grounded responses with plausible but unsupported alternatives.

  • Trajectory formulation: Each task contains an environment image, scene entities with annotated parts, and interaction turns combining visual observations, reasoning, and structured actions.Actions include inspecting an entity, inspecting one of its parts, or answering with the selected entity, part, and intended use.
  • Trajectory formulation: The interaction protocol exposes scene, entity, or part-level evidence according to the preceding inspection action.Entity inspection returns the entity image and part list; part inspection returns a zoomed part image with optional attribute-level disambiguation.
  • Knowledge-guided exploration: An ordered exploration stack prioritizes entities and parts whose affordances are relevant to the target, while fine-grained verification distinguishes useful distractors from the gold solution.The resulting trajectory searches coarsely over entities, then verifies relevant parts using physical evidence.
  • Three-branch trajectory sampling: At each shared interaction context, three aligned branches provide a preferred grounded response and rejected alternatives for preference learning.The positive branch follows the knowledge base, exploration stack, and gold solution; negative branches differ in the information exposed to the teacher model.
  • Positive supervision: Positive responses must remain grounded in the current observation, consistent with the exploration stack, and explicit about attribute–affordance relationships.Positive trajectories support supervised fine-tuning, while SFT alone does not directly penalize plausible but incorrect reasoning because inference lacks hidden labels and gold rationales.

E.3 Overall Training Procedure

The training procedure combines supervised imitation of grounded exploration with turn-level preference optimization. Together, these objectives promote coarse-to-fine search and discourage fluent reasoning unsupported by physical evidence.

  • Two-stage training: SFT teaches a grounded exploration policy by imitating positive trajectories.The model learns systematic entity and part inspection from trajectories constructed as preferred responses.
  • Two-stage training: DPO sharpens the model’s decision boundary through aligned turn-level comparisons between preferred and rejected responses.The comparisons target decisions made during interactive exploration rather than only the final answer.
  • Two-stage training: The combined objectives train systematic coarse-to-fine exploration while avoiding fluent but physically unsupported attribute–affordance reasoning.This directly targets the benchmark’s main failure mode.

F Experiment Details

The evaluation presents models with a scene and entity names, then requires interactive inspection of entities and parts before a grounded answer. Prompts enforce exact actions, explicit comparison, and physically plausible justification.

  • Evaluation protocol: Models begin with the task, scenario, environment image, and entity names, but must inspect entities to discover their available parts.They can inspect one entity, inspect one part, or provide a final answer at each turn.
  • Evaluation protocol: The exploration instructions require multiple promising entities and parts to be compared using affordances, geometry, material cues, attachment, reachability, and constraints.Models are told not to stop at the first plausible option unless the choice is already decisive.
  • Action format: Each turn must return one JSON action, with exact formats for entity inspection, part inspection, and the final grounded answer.The final answer specifies the entity, part, and how the part should be used.
  • Inspection feedback: Inspection feedback supplies exact part names for entities and physical and state explanations for parts, while requiring continued grounding in visual evidence.The protocol warns against selecting a plausible part without comparing other candidates when uncertainty remains.

G.1 Error Analysis Details

The error-analysis procedure combines human annotation with GPT-5.4 categorization using a structured taxonomy of physical, practical, and constraint-related failures. Predictions are judged against the task, gold solution, gold image, and a supporting heuristic.

  • Annotation procedure: GPT-5.4 annotates remaining failure cases after manual annotation of 50 cases achieved 92% agreement with human labels.The annotations identify one primary failure reason and any additional contributing reasons.
  • Error taxonomy: The taxonomy groups errors into physical invalidity, practical infeasibility, and risk or constraint mismatch.Physical invalidity includes hallucinated affordances, affordance mismatch, and performance shortfall; practical infeasibility includes destructive workarounds and context or accessibility issues.
  • Judgment procedure: The categorization output requires reasoning, one major reason code and label, and one or more contributing reason codes and labels.The decision rule prioritizes concrete physical, practical, risk, or constraint problems.
  • Judgment procedure: Judges compare the predicted entity, part, and usage with the ground truth and use the gold image and change-reason heuristic as supporting evidence.The major reason must be selected exactly once and included among the contributing reason codes.

G.2 Case Study Details

The case studies show that successful creative tool use requires inspecting the right entities and parts, then matching their grounded physical affordances to the task. Baseline traces instead fixate on locally plausible properties such as softness or friction and miss required geometry, durability, or pressure-management features.

  • Case A: Wall Protection from a Metal Hook: The baseline instead chooses the microfiber_pile_surface by generalizing from softness and rejects the shower rod as rigid without inspecting its rubber end pads.Its proposed towel buffer is superficially plausible, but the trace does not examine the relevant shower-rod part.
  • Case A: Wall Protection from a Metal Hook: In wall protection, the successful trace inspects the shower rod’s non_slip_end_pads and selects them because their soft, pliable, high-friction rubber cushions pressure against the painted wall.The pad is described as EPDM rubber that is sturdy, stretchable, water-tolerant, and slightly tacky.
  • Case B: Wrapping Paper Straight-Edge and Backing Board: In wrapping-paper trimming, the successful trace selects the under-bed storage bin’s lid_panel because it supplies a flat, sturdy, non-slip backing surface for marking.This demonstrates part-level geometry reasoning: the solution provides both backing and a straight edge rather than only local friction.
  • Cross-case capability: The cases evaluate exact-part discrimination under safety and state constraints, including partially visible, blocked, wet, or hot conditions.The capability is framed as grounding material, friction, pressure distribution, contact-point geometry, and state information in the selected action.
Loading 2605.26396v2…