PaperlayerView exact revision on arXiv

Source-linked AI summary

A Stevens's Power Law Check-up of GPT-5.5's Image-Based Visualization Reading

Kaichun Yang, Jian Chen

arXiv:2610.08365v1cs.CV

TL;DR

The paper addresses the difficulty of telling how models perceive visual encodings when task accuracy may reflect pattern matching rather than innate perceptual behavior. It measures GPT-5.5 with legend-free, reference-relative judgments and finds responses vary by visual channel, including inverse responses for three colormaps.

  • Problem

    Prior visualization evaluations often assumed models perceive images like humans, leaving innate perceptual behavior difficult to distinguish from pattern matching or guessing.

    Source: §1 · p. 1

  • Method

    The authors fit Stevens’s power law to GPT-5.5’s legend-free, reference-relative magnitude estimates across twelve visual variables.

    Source: §1 · pp. 1–2

  • Results

    Across twelve visual variables, GPT-5.5’s responses were approximately linear for some channels, mildly compressive for others, and inverse for three colormaps.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors conclude that psychophysical experiments can expose machine problem-solving gaps and that visual information is necessary.

    Source: §6 · p. 4

    The analysis covers one model and controlled synthetic stimuli that do not capture the complexity of real-world visualization images.

    Source: §5 · p. 4

Abstract

from arXiv · show

We adapt Stevens's power law to measure the innate ability of AI models to read visualizations, which can reveal the built-in perceptual mechanisms of algorithmic models. In our pilot study, models see no legend. A model first views a reference visual representation and estimates its magnitude, then estimates the magnitude of each subsequent image of the same representation relative to that reference. Our evaluation of twelve visual variables makes how algorithmic models read visual encodings measurable, comparable with human perception, and more interpretable to humans.

1 INTRODUCTION

The paper adapts Stevens’s power law to measure how GPT-5.5 estimates visual magnitudes, using legend-free, reference-relative judgments across visual variables. Its pilot finds approximately linear to mildly compressive responses for most variables, but inverse responses for three colormaps.

1 INTRODUCTION

  • Prior evaluations often assumed models perceive visualizations like humans, leaving innate perceptual behavior difficult to distinguish from pattern matching or guessing.The authors identify this as an open measurement problem despite prior work on accuracy and more recent sampling-based evaluations.Source: §1 · p. 1
  • The method asks GPT-5.5 to estimate a reference magnitude and then estimate target magnitudes relative to it, without a legend.The study fits Stevens’s power law to responses across twelve visual variables.Source: §1 · p. 1
  • Across twelve visual variables, GPT-5.5 scaled approximately linearly for length, area, angle, and line texture; mildly compressively for point texture, luminance, and shape; and inversely for three colormaps.The three colormaps are Greys, plasma, and jet.Source: §1 · p. 1, Figure 1 · p. 1
  • The authors present the method as a psychophysical way to measure model behavior and make GPT-5.5’s visual readings transparent to human observers.Their contributions include a twelve-channel account of perceptual scaling and errors.Source: §1 · p. 2

2 RELATED WORK

Related work motivates measuring model responses across controlled inputs rather than relying only on aggregate task performance. The paper situates Stevens’s law as a measure of perceptual scaling, complementary to methods that measure discrimination.

2 RELATED WORK

  • Aggregate benchmark scores can obscure variation across tasks and input conditions, motivating fine-grained evaluation signals and behavior-focused measures [4, 5].The passage describes ARC-AGI-2 as extending this direction through fine-grained signals and human-tested tasks.Source: §2 · p. 2
  • Sampling-based evaluations have examined graphical-perception sensitivity and stability, while this work models responses across controlled stimulus ranges and visual variables [11].The paper analyzes response functions, relative error, and comparisons with human observers.Source: §2 · p. 2
  • Stevens’s method measures the global relationship between physical stimulus intensity and perceived magnitude with a power function fitted separately for each variable.The account contrasts roughly linear perceived line length with strongly expansive electric-shock discomfort.Source: §2 · p. 2
  • Weber’s law measures just-noticeable differences, whereas Stevens’s law measures the form of the perceptual scale.The paper uses Stevens’s law to study how magnitude estimates compress or expand across each visual variable.Source: §2 · p. 2

3 PSYCHOPHYSICAL FRAMEWORK FOR QUANTIFYING AI OBSERVERS

The framework asks how power-law exponents fitted to model responses vary across visual variables. It operationalizes this question through reference-relative magnitude judgments, controlled sampling, and synthetic visual encodings.

3 PSYCHOPHYSICAL FRAMEWORK FOR QUANTIFYING AI OBSERVERS

  • The framework asks how Stevens’s power-law exponents fitted to MLLM responses differ across visual variables.It frames the model as an observer whose response behavior can be measured rather than presuming human-like perception.Source: §3 · p. 2

3.1 Task

  • In each conversation, the model estimates a reference image’s value, then estimates a target image of the same visual variable on the reference-established scale.Each variable appears in a separate image and prompt, and conversations share no history.Source: §3.1 · p. 3

3.2 Study Design on Data Sampling

  • The pilot sampled 48 reference-target trials per visual variable from eight non-identical, non-adjacent bin pairs across values 1–100.Six random pairs were sampled per bin pair, and stimuli smaller than 32 pixels were excluded.Source: §3.2 · p. 3, Figure 3 · p. 3

3.3 Visualization Images

  • The images used a centered visual variable on a white 1,000 × 1,000-pixel canvas, with color, luminance, and texture represented through fixed-size regions or patterned marks.Color used the Greys, plasma, and jet colormaps; luminance varied in CIELAB lightness, while texture used lines or dots.Source: §3.3 · p. 3
  • The remaining encodings represented angle with radial lines and arcs, area with black circles or squares, length with vertical segments, slope with a line in an L-shaped frame, and shape with spiked forms.Data values determined the corresponding angle, area, length, slope, or spike count.Source: §3.3 · p. 3

4 RESULTS

The results first fit scaling and compare errors across visual variables, then probe GPT-5.5’s implicit numerical mappings for three colormaps.

4.1 Analysis Method

  • The analysis fit an exponent from target-to-reference response ratios against stimulus ratios, and compared log absolute errors across variables using ANOVA and Tukey HSD.Slope was excluded from the power-law analysis but included in the error analysis; for angles, reported values were replaced with their circular complements when closer to the ground-truth angle.Source: §4.1 · p. 3

4.2 GPT-5.5 Stevens’s Power-Law Fits

  • Exponents ranged from 0.985 to 1.001 for length, area, angle, and line-based texture, indicating approximately linear scaling; all three colormaps had negative exponents.Point texture was compressive (α̂ = 0.941), while luminance and shape had the smallest positive exponents (α̂ ≈ 0.880); colormap exponents ranged from −0.754 for Greys to −0.531 for jet.Source: §4.2 · p. 4

4.3 Ranking Visual Variable by Log Absolute Error

  • Length had the lowest mean log absolute error, while all three colormaps had significantly higher errors than every non-colormap variable.The colormaps shared a Tukey HSD group; the analysis excluded trials with zero-valued reference responses.Source: §4.3 · p. 4, Figure 4 · p. 4

4.4 GPT-5.5’s implicit numerical mapping of colormaps

  • Without a legend or colormap name, GPT-5.5 read colors as hue angle or gray level rather than colormap position, reversing direction for jet and Greys.For plasma, this mapping yielded a negative fit despite preserved hue order.Source: Figure 5 · p. 4, §4.4 · p. 4

5 DISCUSSION

GPT-5.5 tends to assign higher values to lighter colormap colors, opposite the dark-is-more bias often expected of people. The discussion frames models as measurable observers while noting that the pilot’s scope is limited.

5 DISCUSSION

  • Treating an MLLM as an observer enables analysis of measurable responses without assuming innate perception of visual variables.The response function and innate maps may be connected to accuracy scores from broader image-based evaluations [2].Source: §5 · p. 4
  • GPT-5.5 tended to assign higher values to lighter colormap colors, opposite the dark-is-more bias often expected of people.One possible reason is that lighter colors have higher luminance values in color space.Source: §5 · p. 4
  • The analysis covers one model and controlled synthetic stimuli, not the complexity of real-world visualization images.The pilot also samples only a small subset of possible reference-target combinations.Source: §5 · p. 4

6 CONCLUSION

The conclusion presents legend-free magnitude estimates as a way to measure built-in sensory responses directly. It reports an opposite colormap tendency and argues that visual information and explicit context may matter for machine perception.

6 CONCLUSION

  • The method measures a model’s built-in sensory responses directly when it reports magnitudes for text-named encodings without an image legend.This offers a way to study those responses rather than infer them from task performance.Source: §6 · p. 4
  • The measured innate colormap responses were opposite to the colormaps used to generate the visualization images.The conclusion also states that language alone might not be enough and visual information is necessary.Source: §6 · p. 4
  • The authors argue that explicit context displays such as legends may be important for proper machine perception.The conclusion presents psychophysics experiments as a way to make machine problem-solving gaps transparent.Source: §6 · p. 4

AI USE STATEMENT

The statement describes AI assistance with plotting and experiment code, while distinguishing that work from the underlying data, analyses, and findings.

AI USE STATEMENT

  • GPT-5.6 Sol assisted with writing and debugging plotting code for Figures 1, 4, and 5.It was not used to generate or modify the underlying data, analyses, or research findings.Source: AI USE STATEMENT · p. 4
  • GPT-5.6 Sol and GPT-6 Astra helped debug and write experiment code and debug figure-style code.The authors reviewed all AI-generated content for accuracy.Source: AI USE STATEMENT · p. 4
  • The authors state that AI tools did not generate or modify the underlying data, analyses, or research findings.All AI-generated content was reviewed for accuracy.Source: AI USE STATEMENT · p. 4

A GPT-5.5 CONFIGURATION

Table 1 is identified as the GPT-5.5 configuration used in the pilot study.

A GPT-5.5 CONFIGURATION

  • Table 1 identifies the GPT-5.5 configuration used in the pilot study.The supplied caption does not specify configuration values.Source: Table 1 · p. 6

EXAMPLE STIMULI AND PROMPTS

Each example pairs a visual variable image with reference and target prompts, asking the model to assign values on a scale established by the reference. The prompts cover geometric, shape, texture, luminance, and color representations.

EXAMPLE STIMULI AND PROMPTS

  • Examples cover angles, circle and square areas, and line length or slope through dedicated reference and target prompts.Source: EXAMPLE STIMULI AND PROMPTS · p. 6
  • For each variable, the reference prompt establishes a numerical scale and the target prompt asks for a value using that reference.Prompts request a brief explanation and a final numerical answer in a specified format.Source: EXAMPLE STIMULI AND PROMPTS · p. 7
  • Other prompts ask models to assign values to shape, dotted or line texture, luminance, and colors in colored-square images.Source: EXAMPLE STIMULI AND PROMPTS · pp. 7–8

C DOMAIN OF THE VISUAL VARIABLES

Figure 6 presents examples for twelve visual variables, with sampled-bin examples arranged in rows and columns.

C DOMAIN OF THE VISUAL VARIABLES

Loading 2610.08365v1…