PaperlayerView exact revision on arXiv

Source-linked AI summary

Autoregressive Mosaics: Probing 2D Spatial Reasoning in Text-Only Language Models

Ashwin Nedungadi, Stefan Oehmcke, Stefan Lüdtke

arXiv:2608.30751v2cs.AIcs.CV

TL;DR

The paper asks whether text-only language models genuinely represent and compose 2D spatial layouts, rather than merely translating descriptions into code. AM-Bench separates these abilities with translation and underspecified layout tasks, then probes output media and internal states. Translation is reliable across models while layout varies, SVG improves layout scores, and generation tracks evolving geometry rather than a fixed plan.

  • Problem

    It is unclear whether image-generating text-only models internally represent 2D layouts or mainly translate spatial descriptions into code.

  • Method

    AM-Bench separates fully specified geometry-to-code translation from underspecified layout composition and probes output media and model activations.

  • Results

    Across eight models, translation is reliable but open-ended layout varies substantially; raw SVG improves layout scores across all models, while generation tracks evolving geometric state.

  • Takeaways & Limitations

    2D spatial performance in text-only language models depends on both the model and the output medium, not code-generation ability alone.

  • Takeaways & Limitations

    The study evaluates eight open-weight 8B–34B models, so generalization to larger or closed-weight models remains untested.

Abstract

from arXiv · show

Large language models (LLMs) trained only on text and code can sometimes generate programs that draw recognizable images. However, it is unclear whether this reflects an internal representation of 2D spatial layout or simply the ability to translate spatial descriptions into code. We introduce Autoregressive Mosaics (AM-Bench), a benchmark that separates these factors: First, a translation task gives a model a fully specified geometry of a picture in words as a prompt and asks for the code that produces it. Second, a layout task requires the model to compose an image from an underspecified prompt. Across eight open-weight text-and-code-only models, all models reliably translate specified geometry into code, but their open-ended layout performance differs substantially, indicating that these differences are not explained by code-generation ability alone. An output-medium ablation further shows that the interface or medium of expression that the model uses matters: replacing procedural code with raw SVG improves layout scores across all models. Finally, probing model activations shows that a coarse layout plan is present before generation, but reflects only the layout implied by the prompt. During generation, models track the evolving geometric state instead of executing an initially fixed plan. Overall, these results show that 2D spatial performance in text-only LLMs depends on both the model and the output medium, and is not explained by code-generation ability alone.

1. Introduction

AM-Bench separates spatial composition from spatial expression by contrasting fully specified translation with underspecified layout generation. Across eight text-and-code-only models, translation is reliable while open-ended layout varies, and output medium and generation dynamics further shape performance.

  • Benchmark motivation: AM-Bench represents images as autoregressive mosaics and separates translating specified geometry into code from composing underspecified layouts.Its pipeline includes the prompt, internal plan, program code, and rasterized geometry.
  • Benchmark motivation: An LLM-generated image alone cannot reveal whether poor performance reflects spatial composition, code expression, or constraints imposed by the output medium.Existing evaluations often entangle these factors.
  • Main findings: All eight open-weight models reliably solve translation, but their open-ended layout performance differs substantially.This indicates that code-generation ability alone does not explain layout differences.
  • Main findings: Replacing procedural canvas code with raw SVG improves layout scores across all models, showing that the output medium influences results.The benchmark evaluates expression, medium effects, and pre-generation spatial plans in three experiments.
  • Main findings: A coarse spatial representation appears before generation, but models track evolving geometric state rather than executing a fixed initial plan.The pre-generation representation reflects the layout implied by the prompt.

2. Related Work

AM-Bench builds on probing work showing that text-trained transformers can encode board-like structure without visual input. It also addresses prior programmatic-generation evaluations that do not cleanly separate executable code from spatial correctness, while introducing repeated-attempt consistency as a proxy for stable internal imagery.

  • Emergent structure: Othello-GPT and later chess studies showed that transformers trained only on move sequences can yield readable board representations.AM-Bench applies a related probing motivation to grid-structured visual layouts in models that never saw images.
  • Programmatic visual generation: Prior systems paired language models with renderers or refinement loops, adding learned components rather than isolating inherent spatial abilities.This motivates evaluating text-and-code-only models directly.
  • Programmatic visual generation: Existing benchmarks compare generated outputs with references or assess spatial reasoning, but they do not establish the same separation between code execution and spatial correctness.PRISM is described as finding a gap between code that runs and code that is spatially correct.
  • Consistency analysis: AM-Bench uses repeated attempts for one prompt to measure how tightly outputs cluster as a proxy for a stable internal spatial picture.This representation-level consistency analysis using DINO embeddings is presented as new for LLM-generated mosaics.

3. AM-Bench

AM-Bench uses constrained, low-resolution programs to test translation and open-ended layout separately, with symbolic geometric scoring for translation and VLM judgments for layout. Its custom canvas vocabulary also reduces concerns that SVG performance reflects memorized markup rather than zero-shot composition.

  • Autoregressive mosaics and canvas: An autoregressive mosaic is a 24 × 24 raster rendered from an LLM-written program using six constrained primitives.The allowed program includes arithmetic, iteration, and math, while rasterization uses deterministic line, circle, and polygon procedures.
  • Autoregressive mosaics and canvas: The custom primitive vocabulary is intended to reduce data-contamination concerns associated with models pretrained on public SVG markup.The paper notes that SVG evaluation can risk measuring retrieval of memorized training patterns.
  • Translation task: Translation prompts specify every shape’s position, size, and color in grid-based type-and-extent language, avoiding recognizable subject names.This isolates expressing supplied geometry from recalling familiar drawings.
  • Translation task: Translation scores assign generated primitives to reference parts by overlap and compute per-part mean exact-area IoU in normalized continuous coordinates.Exact polygon clipping makes PIoU resolution-independent, while an over-paint penalty discourages extra shapes.
  • Translation task: The translation set combines 145 sampled references, 100 harder controlled references, and 13 hand-crafted icons, with a preselected pass threshold of τtrans = 0.6.The threshold was chosen before scores were observed and was confirmed above chance.
  • Layout task: Layout prompts are underspecified, so outputs are judged holistically by two VLMs across five dimensions on a 0–5 scale with pass ≥3.0.Cross-model layout comparisons require a model’s median translation score to clear τtrans.

4. Experiments

Across eight text-and-code-only models, AM-Bench separates executable code translation from open-ended spatial composition, then tests how the output medium and internal representations affect layout performance.

  • Experiment setup: 13,920 translation attempts yielded a median PIoU of 1.000 for every model, while 98.1% passed the τtrans=0.6 threshold.This establishes a common level of stated-plan-to-code competence before comparing layout performance.
  • Experiment 1: Layout performance: All models passed translation, so subsequent layout-score differences isolate spatial composition rather than code-generation capability.Every model exceeded 88% validity across tiers, although trivial uniform-fill rates varied substantially.
  • Experiment 1: Layout performance: GLM-4 32B led every tier under both judges, while CodeLlama 34B ranked last; the judges agreed exactly on model-level ordering (Spearman ρ=1.0).Judge 2 was systematically about one point more lenient than Judge 1, but preserved the same ranking.
  • Experiment 1: Layout performance: T3 Compositional outscored T2 Iconic under both judges, reversing the expected difficulty gradient; the inversion held for five models under J1 and six under J2.The paper attributes this pattern to 24×24 resolution constraints that make fine iconic structure harder to approximate than compositional spatial language.
  • Experiment 2: Output medium: Every model improved when canvas code was replaced by raw SVG, showing that layout scores depend on the rendering medium.The ablation used the full 150-prompt Layout suite with identical judging and sampling protocols; the ranking remained similar across media.

5. Conclusion

AM-Bench shows that 2D layout performance depends on both model compositional ability and output medium, while internal layout representations remain generic and generation is autoregressive.

  • AM-Bench separates internal layout availability from the ability to express layouts through a required medium.
  • Median translation PIoU is 1.0 for all models, so translation is not the bottleneck.
  • Raw SVG raises every model’s layout score relative to the custom canvas medium, even after controlling for SVG’s higher rendering fidelity.
  • A coarse layout is decodable above a text baseline, but only the component shared across models is decodable; the model-specific component reaches R2 ≤0.005.
  • The study covers eight open-weight 8B–34B models, small mosaic outputs, VLM judging, and code rather than activation interventions.

A. Model Generation & Prompting

The appendix specifies two generation media, their prompts and primitives, and a shared judging framework for evaluating generated mosaics.

  • A.1. Medium specifications: The canvas medium uses a 24×24 raster, six drawing primitives, arithmetic, iteration, and math under sandbox restrictions.
  • A.2. System prompts: The experiment-specific prompt table maps generation, translation, probing, and judging paths while keeping probe inputs aligned with real generation inputs.
  • A.2.1. Canvas generation system prompt: Canvas generation requires only valid Python code, explicit background initialization, bounded coordinates, and concise code without imports or print statements.
  • A.2.1. Canvas generation system prompt: The canvas prompt defines RGB color formats, a recommended named palette, and row-column orientation with the top-left origin.
  • A.2.2. SVG generation system prompt (Exp. 2): SVG generation asks for a self-contained 512×512 SVG using standard shape elements, local references, explicit backgrounds, and concise markup.
  • A.2.2. SVG generation system prompt (Exp. 2): SVG prompts support reusable shapes and pattern definitions for repeated or tiled structures instead of listing every instance.
  • A.2.3. Judge system prompt (Exp. 1, both judges): Both media arms use a medium-neutral judge prompt, while judges share byte-identical prompts and differ only in their underlying VLMs.
  • B.1. Symbolic metric details: The rubric scores prompt fidelity, shape, color, spatial accuracy, and completeness on 0–5 scales; PIoU assigns parts by overlap ratios with separate over-paint penalties.

B.2. Translation stress-test details

The translation stress tests show that specified geometry remains reliably transcribable across increasingly difficult references, with rare failures linked more to generation uncertainty than picture complexity.

  • The appendix’s stress-test evidence is limited to a two-model pilot and was not rerun for the other six models.
  • Random placement averages PIoU 0.072, with 95% of attempts below 0.253, far below real scores and the passing threshold.
  • Color matching lowers scores by 0.033 for 14B and 0.036 for 7B, without changing the translation outcome.
  • The translation ladder varies four geometric difficulty axes across five rungs, while correct transcriptions exist by construction.
  • 14B scores 1.000 at every ladder rung, while 7B scores 1.000, 0.996, 1.000, 1.000, and 0.993.
  • Across 13 icons, mean score falls to 0.90, with the drop attributed to an outline-shape gap rather than reasoning.
  • Each unit increase in mean entropy raises miss odds to 4.8 for 14B and 2.6 for 7B, while picture complexity has odds ratios indistinguishable from 1.
  • Median PIoU remains 1.000 for both models across first-round, ladder, and pooled references.

C.2. Human-validation materials and results

Human validation supports the judges’ preference rankings, while moderate inter-annotator agreement and underpowered secondary comparisons limit stronger conclusions.

  • Agreement and judge validity: 70.6% of pairs received matching verdicts from both annotators, with Krippendorff’s α = 0.60.Of 350 pairs, 247 had agreement; disagreements were dropped rather than arbitrarily resolved.
  • Agreement and judge validity: 82.4% (61/74) of agreed, judge-separated pairs matched Judge 1, compared with 76.7% (46/60) for Judge 2.These rates were above chance and exceeded the corresponding single-annotator rates.
  • Score-gap validation: Agreed ties decreased as the judges’ absolute score gap increased, from 66.7% at [0, 0.5] to 34.1% at [2] [5].This pattern supports interpreting close judge scores as reflecting visual similarity rather than only judge noise.
  • Secondary comparisons: The cross-model 14B-over-7B win rate was 76.2% (95% CI [52.8, 91.8], n=21), whereas the cross-tier T2 win rate was 28.6% (CI [3.7, 71.0], n=7).Both intervals were wide, and the cross-tier interval included no preference.
  • Scope: Because agreement was moderate rather than strong, the validation supports above-chance judge alignment but not strong agreement among human raters.The authors identify a larger multiannotator run as future work.

D.1. Exp. 2: Medium Ablation Details

The experiments test whether activation decodability reflects a model-specific planned layout and whether models use that information during generation, with controls for target reliability and behavioral confounds.

  • Representational target: The continuous target is each probed model’s consensus occupancy across valid generations, rather than one binary reference layout.It is computed as the per-cell mean of 6×6 occupancy grids, with n ranging from 138–145 across models.
  • Stage 1: representational probing: Activation decodability exceeded the text baseline for all eight models, with mean ∆R2 = +0.164 and values from +0.130 to +0.205.All intervals excluded zero and all models remained significant after Benjamini-Hochberg correction; permutation controls were effectively zero.
  • Cross-model selectivity: Eight models decoded one another’s layouts about as well as their own, with a mean own-minus-others gap of +0.015.Own-source was best for only Gemma 2 9B, and GLM-4 9B had a negative gap of −0.007.
  • Component decomposition: The shared component decoded at R2 = 0.550 and 0.573, while the model-specific component reached only 0.005 and −0.005.The corresponding text baselines for the shared component were 0.289 and 0.293.
  • Stage 2: behavioral causal test: When the model’s own completed-picture statement was displaced, continued drawing rose from 43.8% to 84.6% for GLM-4 32B and from 2.5% to 41.2% for Gemma 2 27B.The changes were ∆= +0.415 and ∆= +0.387, respectively, both with pFDR = 0.0007.
  • Limitations: The behavioral intervention is limited by pixel-measure confounds and a weak 4 px manipulation whose destination overlaps its origin 64–69% of the time.The authors therefore emphasize the code-level headline measure and report targeting analyses as mixed.

E.2. DINO concept-consistency analysis

The DINO analysis measures whether repeated generations for the same prompt form coherent visual clusters, including global embedding coherence and a foreground-masked local-structure measure.

  • Analysis goal: The analysis uses DINO ViT-S/8 embeddings from 11,976 valid, non-trivial layout generations to assess repeated-attempt consistency by prompt.This consistency is treated as a proxy for drawing from a stable internal picture rather than improvising anew.
  • Preprocessing: Images are resized so each DINO patch corresponds exactly to one cell of the 24×24 canvas, yielding a 24×24 grid of 576 patches.Generated 512×512 images are reduced to 24×24 and then upscaled to 192×192.
  • Global coherence: Lower d_intra indicates tighter within-prompt embedding clusters, while Llama 3.1 8B and CodeLlama 34B have negative silhouette values at every tier.Table 7 reports intra-prompt coherence for all eight models.
  • Global coherence: GLM-4 32B and Qwen2.5-Coder 32B maintain the largest intra-versus-inter-prompt distance gap at every tier.For Llama 3.1 8B and CodeLlama 34B, intra-prompt distance approaches inter-prompt distance throughout.

E.3. Repeatability with DINO and the Entropy Confound

Repeatability largely reflects sampling determinism rather than spatial reliability. DINO analyses separate semantic from local structure, while illustrative attention patterns align with—but do not independently establish—quantitative consistency differences.

  • Repeatability: Raw DINO clustering of repeated GLM-4 32B generations mainly tracks sampling determinism, so visual repeatability is heavily confounded by randomness.Repeated attempts at the same prompt form small, separated clusters, but this does not establish spatial reliability.
  • Entropy confound: Spearman ρ = 0.93 (p < 0.001) links token entropy to local-structure distance across eight models, explaining most apparent consistency differences through sampling determinism.The entropy analysis uses teacher-forced values for Qwen2.5-Coder-14B and live generation logging for the other models.
  • Residual structure: A small entropy-adjusted residual remains: Gemma 2 27B is more locally consistent than predicted, whereas CodeLlama 34B is less consistent.The residuals are −0.014 and +0.013 respectively, several times smaller than the raw cross-model range and insufficient alone to explain layout-score gaps.
  • Consistency measures: Local structure is measured with foreground-masked DINO patch-key self-similarity, making it sensitive to spatial arrangement rather than whole-image semantic content.This contrasts with the global metric’s whole-image CLS embedding.
  • Attention maps: Attention maps show tighter focus for GLM-4 32B than CodeLlama 34B, but maximum-focused examples are illustrative rather than quantified evidence of typical behavior.The selected pattern is directionally consistent with quantitative coherence results, not independent confirmation.
  • Embedding structure: DINO embeddings separate T1 from T2/T3 more clearly than they separate T2 from T3, indicating visual complexity dominates the embedding geometry.This pattern appears in an approximately 2,000-image stratified subsample across all eight models.

F.2. Limitations, full detail

The paper’s limitations concern benchmark scope, incomplete controls, and non-uniform model coverage. Several analyses remain restricted to pilots or illustrative examples, while broader spatial competence is not measured.

  • Model coverage: The eight-model roster is broader than a single-family pilot but is not a controlled scaling-law sweep, and closed-weight models were not tested.Each family contributes exactly two availability-selected sizes; whether the reported patterns generalize across larger within-family scales remains untested.
  • Interpretive boundary: The translation gate rules out code-writing competence as an explanation for low layout scores but does not alone distinguish absent layout representations from representations not convertible into primitives.Exp. 3 addresses this distinction by probing decodability above a text baseline.
  • Potential confounds: Numeric tokenization could confound Layout performance because every primitive call uses explicit numeric coordinates.The translation task does not directly rule out this explanation.
  • Evaluation coverage: The translation stress-test ladder and 13-icon set were run only for the original two-model pilot, leaving the 145-reference result as the sole translation analysis covering all eight models.The ladder tests geometric difficulty but not periodicity.
  • Reference quality: Seventeen of 145 translation targets required simplification, producing lower pilot scores on those examples while adding a documented source of noise.The effect was not large enough to change any eight-model outcome.
  • Illustrative analysis: The DINO attention comparison selects the most spatially focused example from candidate generations, so it is not evidence about typical behavior across generations.Its qualitative pattern is only directionally consistent with quantitative coherence results.
  • Scope: AM-Bench measures allocentric 2D layout at mosaic granularity, not metric egocentric continuity, depth, or occlusion.It is framed as a necessary-condition probe for embodied spatial competence, not a sufficient one.
Loading 2608.30751v2…