PaperlayerView exact revision on arXiv

Source-linked AI summary

LTD-Bench: Evaluating Large Language Models by Letting Them Draw

Liuhao Lin, Ke Li, Zihan Xu, Yuchen Shi, Yulei Qin, Yan Zhang, Xing Sun, Rongrong Ji

arXiv:2511.02347v1cs.CL

TL;DR

Current LLM evaluations provide limited intuitive insight into spatial capabilities. LTD-Bench assesses spatial imagination and perception through visual drawing and recognition tasks. Its experiments find substantial challenges in both abilities, including mapping between language and spatial concepts.

  • Problem

    Current evaluations offer limited insight into LLMs’ spatial and physical-world capabilities, leaving researchers and users without an intuitive account of what benchmark scores reveal.

    Source: §1 · p. 1

  • Method

    LTD-Bench evaluates spatial imagination and perception through complementary generation and recognition tasks across three difficulty levels, with models producing visual representations from text.

    Source: §1 · p. 2

  • Results

    Current LLMs face substantial challenges in spatial perception and imagination, often failing to establish robust bidirectional mappings between linguistic symbols and spatial concepts.

    Source: §6 · p. 11

  • Takeaways & Limitations

    LTD-Bench’s visual outputs offer a promising avenue for exploring model similarity.

    Source: §6 · p. 11

    The benchmark’s relatively small dataset and narrow focus on spatial perception and imagination limit the comprehensiveness and generalizability of its findings.

    Source: §5 · p. 10

Abstract

from arXiv · show

Current evaluation paradigms for large language models (LLMs) represent a critical blind spot in AI research--relying on opaque numerical metrics that conceal fundamental limitations in spatial reasoning while providing no intuitive understanding of model capabilities. This deficiency creates a dangerous disconnect between reported performance and practical abilities, particularly for applications requiring physical world understanding. We introduce LTD-Bench, a breakthrough benchmark that transforms LLM evaluation from abstract scores to directly observable visual outputs by requiring models to generate drawings through dot matrices or executable code. This approach makes spatial reasoning limitations immediately apparent even to non-experts, bridging the fundamental gap between statistical performance and intuitive assessment. LTD-Bench implements a comprehensive methodology with complementary generation tasks (testing spatial imagination) and recognition tasks (assessing spatial perception) across three progressively challenging difficulty levels, methodically evaluating both directions of the critical language-spatial mapping. Our extensive experiments with state-of-the-art models expose an alarming capability gap: even LLMs achieving impressive results on traditional benchmarks demonstrate profound deficiencies in establishing bidirectional mappings between language and spatial concept--a fundamental limitation that undermines their potential as genuine world models. Furthermore, LTD-Bench's visual outputs enable powerful diagnostic analysis, offering a potential approach to investigate model similarity.

1 Introduction

LTD-Bench replaces opaque score-based evaluation with observable drawings and probes both directions of language–spatial mapping. Its experiments expose substantial spatial-mapping limitations in otherwise strong LLMs and enable output-based model-similarity diagnostics.

1 Introduction

  • Traditional aggregate scores obscure what models can and cannot do, especially in physical-world understanding needed for robotics, autonomous systems, and design tools.The introduction identifies this as a gap between reported performance and intuitive understanding of model capabilities.Source: §1 · p. 1
  • LTD-Bench makes spatial reasoning visibly assessable by requiring models to produce visual artifacts from textual instructions, using dot matrices or executable code.The outputs allow direct human assessment rather than relying solely on abstract numerical scores.Source: §1 · pp. 2–3
  • Generation tasks test spatial imagination and recognition tasks test spatial perception across three levels, from character representation to complex real-world objects.Together, the tasks assess both language-to-spatial and spatial-to-language mapping.Source: §1 · pp. 2–3
  • Even LLMs strong on traditional reasoning tasks show profound deficiencies in mapping bidirectionally between language and spatial concepts.The paper argues this gap undermines their potential as genuine world models.Source: §1 · pp. 2–3
  • LTD-Bench’s visual outputs support diagnostic comparisons of generated-image styles to investigate similarities among models.The paper presents this as an analysis not well captured by traditional evaluation metrics.Source: §1 · pp. 2–3

2 Related Work and Discussion

The section argues that text-centered benchmarks miss LLMs’ spatial understanding and motivates LTD-Bench’s visual evaluation of spatial perception and imagination.

  • Existing benchmarks measure knowledge, factuality, mathematical reasoning, and code generation but lack intuitive visual assessment of bidirectional language–space mappings.The section characterizes these evaluations as confined to a text-to-symbol paradigm.Source: §2 · p. 3
  • Studies of congenitally blind individuals support text-based spatial cognition, and code-rendering results [Achiam et al., 2023][Gupta and Kembhavi, 2023] suggest latent LLM spatial reasoning, yet no benchmark systematically evaluates it.The cited work also argues that spatial reasoning relies on innate neural mechanisms rather than visual experience [Cooney et al., 2024].Source: §2 · p. 3
  • LTD-Bench makes spatial perception and imagination more directly interpretable by requiring generation of renderable dot matrices or Python drawing code.Source: §2 · pp. 3–4
  • For certain open-ended generation tasks, LTD-Bench uses GPT-4.1 to assess generated-image quality, enabling direct, interpretable evaluation of spatial reasoning.The section notes prior prompting-based evaluation work in NLP and multimodal domains [Zheng et al., 2023][Chiang and Lee, 2023][Liu et al., 2023][Fu et al., 2023][Zheng et al., 2025][Yu et al., 2023].Source: §2 · p. 4

3 LTD-Bench

LTD-Bench makes LLM spatial reasoning directly inspectable through visual outputs, addressing the gap between reported performance and observable language–spatial mapping. It evaluates spatial imagination and perception across three progressively challenging levels.

3.1 Design Principles and Problem Addressing

  • To expose gaps obscured by numerical benchmarks, LTD-Bench renders generations as inspectable images, tests both mapping directions, and uses three difficulty levels to pinpoint where models struggle.Source: §3.1 · p. 4

3.2 Benchmark Structure and Task Design

  • The framework comprises 183 distinct data distributed across three difficulty levels, each designed to assess specific aspects of spatial reasoning.Source: §3.2 · p. 5

3.2.1 Easy Level: Discrete Grid-Based Spatial Understanding

  • Easy tasks require models to translate text into 0–1 dot matrices or identify characters from matrices, enabling visual checks and unambiguous recognition answers.Source: §3.2.1 · pp. 5–6

3.2.2 Normal Level: Curve Composition in Infinite 2D Coordinate Space

  • Normal tasks require models to generate curve-only Python drawings of characters or identify characters from code that renders curve combinations.Source: §3.2.2 · p. 6

3.2.3 Hard Level: Real-world Object Drawing in Infinite 2D Space

4 Experiments

LTD-Bench experiments show that current LLMs perform poorly on spatial reasoning, with results varying across generation and recognition tasks. The section also explores generated-image style similarity as a possible way to assess model similarity.

4.1 Experiment Settings

  • Experiments evaluate seven advanced LLMs, score Easy and Normal generation with human and GPT-4.1 evaluation, Hard generation with GPT-4.1, and recognition by accuracy.A generation output whose code fails to execute receives a score of 0.Source: §4.1 · p. 7

4.2 Result Analyses

  • Only Deepseek-r1 exceeds 70% average accuracy, while GPT-4.1-mini exceeds 60% and Qwen2.5-72B-Instruct and Llama3.3-70B-Instruct achieve around 30%.Human experts solve Easy and Normal tasks with near-perfect accuracy even in text-only settings, while LLMs fall far short.Source: §4.2 · pp. 8–9, Table 3 · p. 9
  • Deepseek-r1 beats GPT-4.1-mini by over 25% in recognition accuracy but lags in generation; Llama3.3-70B distilled with Deepseek-r1 data gains 18.05% recognition accuracy but declines 2.91% on generation.The authors suggest reasoning may help spatial perception more than spatial imagination, and may lead to overthinking when models lack the abilities a task requires.Source: §4.2 · p. 9, Table 4 · p. 10, Table 5 · p. 10
  • GPT-4.1-mini and GPT-4o do not consistently outperform text-only Deepseek models; GPT-4.1-mini excels in generation, but Deepseek-r1 leads overall.The authors note that language-modeling differences may affect this comparison and call for further research on aligning visual and textual features in multimodal learning.Source: §4.2 · p. 9, Table 3 · p. 9, Table 4 · p. 10

4.3 Model Similarity

  • Over 50% of valid samples are stylistically closest between Qwen2.5-72B-Instruct and Qwen2.5-32B-Instruct, while only three samples are more similar to GPT-4.1-mini.The authors present stylistic similarity in open-ended generated images as a potential approach for investigating model similarity.Source: §4.3 · pp. 9–10

5 Limitations and Future Work

LTD-Bench’s current scope and preliminary model-similarity analysis limit the comprehensiveness of its findings. Future work will expand its tasks and develop more rigorous similarity methods.

  • The relatively small dataset and focus solely on spatial perception and imagination limit the benchmark’s comprehensiveness and the generalizability of its findings.Source: §5 · p. 10
  • Future work will expand the dataset and add a wider array of tasks for more thorough LLM evaluation.Source: §5 · p. 10
  • Model-similarity analysis is preliminary, using stylistic comparisons of generated images as a proxy; future work aims for more systematic, quantitative metrics and methods.Source: §5 · pp. 10–11

6 Conclusion

LTD-Bench enables intuitive, visual evaluation of LLMs by having them draw, targeting spatial perception and imagination. Experiments reveal substantial challenges in both, while the benchmark also offers a way to explore model similarity.

  • LTD-Bench enables intuitive, visual evaluation of LLMs by asking them to draw.Source: §6 · p. 11
  • Experiments show that current LLMs struggle with spatial perception and imagination, often failing to map linguistic symbols and spatial concepts robustly in both directions.Source: §6 · p. 11
  • LTD-Bench offers a promising avenue for exploring model similarity and informing future research.Source: §6 · p. 11

A Experiment details · A.1 Implementation details

The experiments used platform APIs with temperature set to zero, but repeated GPT-4.1 evaluations addressed variance in its outputs.

A.1 Implementation details

  • GPT-4.1-based automated evaluation found variance in GPT-4.1 outputs despite temperature being set to zero.The researchers therefore used GPT-4.1 to evaluate all models’ outputs five times.Source: §A.1 · p. 15
  • Open-source LLMs were accessed through Bailian, which hosts models identical to those on HuggingFace.Source: §A.1 · p. 15
  • GPT-4.1, GPT-4.1-mini, and GPT-4o were accessed through their official APIs.Source: §A.1 · p. 15
  • All models were run with temperature set to 0.Source: §A.1 · p. 15

A.2 More details of human evaluation

Human evaluation used independent annotators across three difficulty levels, drawing on varied technical backgrounds and averaging 10 annotators’ scores per level to reduce individual subjectivity.

A.2 More details of human evaluation

  • Each difficulty level was evaluated by 10 independent annotators, whose scores were averaged to determine the final result.Averaging was intended to mitigate individual subjectivity.Source: §A.2 · p. 15
  • Annotators represented varied technical backgrounds, including computer beginners, master’s students, and laboratory engineers.Source: §A.2 · p. 15

A.3 Additional evaluation methods experiment · B Prompt templates

The additional evaluation experiment finds that GPT-4.1 evaluation preserves model rankings despite slightly higher accuracy scores, while the prompt-template appendix documents task prompts and evaluation prompts across difficulty levels.

A.3 Additional evaluation methods experiment

  • GPT-4.1 evaluation produces slightly higher accuracy scores but preserves model rankings, supporting its use for large-scale automated evaluation alongside human evaluation as a more precise, resource-intensive reference.The comparison covers generation tasks at Easy and Normal levels.Source: Figure 2 · p. 15, §A.3 · p. 15

B Prompt templates

  • The prompt-template appendix presents prompts for Easy and Normal generation and recognition tasks, plus the Hard generation task.These templates are shown in Figures 3–7.Source: Figure 3 · p. 16, Figure 4 · p. 16, Figure 5 · p. 16
  • Separate evaluation prompts are provided for model outputs on the Easy- and Normal-level generation tasks.The prompts appear in Figures 8 and 9.Source: Figure 8 · p. 17, Figure 9 · p. 17
  • The Hard-level generation evaluation prompt instructs the evaluator to identify the object, assess drawing requirements, and then score the image.The supplied prompt excerpt ends after introducing the score range.Source: B Prompt templates · p. 18
  • Figure 10 presents the system and user prompts for evaluating model outputs on the Hard-level generation task.Source: Figure 10 · p. 18
  • Figure 11 provides a prompt for comparing the style similarity of images generated by three models on the Hard-level generation task.Source: Figure 11 · p. 18

C Case study

Failed generation examples across three difficulty levels expose errors in spatial orientation, language-to-space mapping, and spatial imagination, making model limitations visible.

Easy level

  • Easy-level dot matrices reveal mirrored > and J characters and problematic W renderings, indicating weak orientation understanding and spatial imagination.The mistaken counterparts are < and L.Source: Table 8 · p. 19, C Case study · p. 19

Case study

  • The failed cases provide intuitive visual evidence of models’ spatial shortcomings and offer insights for future research on how LLMs understand the world.The section uses failures across generation tasks at three difficulty levels to analyze current model limitations.Source: C Case study · pp. 19–20

Normal level

  • Normal-level failures include an upside-down W from QwQ-32B and other models’ chaotic, incorrect outputs, suggesting weak links between actions and spatial states.These outputs deviate substantially from the intended results.Source: C Case study · p. 19, Table 9 · p. 20

Hard level

  • Hard-level clock, airplane, and leaf drawings show that models struggle to translate language into spatial representations, even when they understand the requested concepts.Deepseek-r1, GPT-4o, and QwQ-32B make errors on spatially constrained clock tasks; airplane and leaf failures further expose limits in spatial imagination and mapping linguistic symbols to spatial entities.Source: C Case study · pp. 19–20, Table 10 · p. 21
Loading 2511.02347v1…