PaperlayerView exact revision on arXiv

Source-linked AI summary

Why VLMs Miss Small Objects, and When Zooming In Is Safe

Junzhe Shi, Yuan Gan, Shida Jiang

arXiv:2610.09313v1cs.CV

TL;DR

VLMs often miss small objects in large images, leaving limits and remedy conditions open. This paper models token density per object and content per call, and proves when decomposition is safe under monotonicity assumptions. No predicted ordering failed on controlled images; the post hoc tests found four failures, all for OpenAI models given more pixels than their default path.

  • Problem

    VLMs often fail to find many small objects in large images, and remedies have been evaluated on separate benchmarks, baselines, and model generations.

    Source: §1 · p. 1

  • Method

    The paper models tokens per object side and content per call, then proves decomposition safe under average monotonicity when views do not zoom out and overlap by one object.

    Source: §7 · p. 25

  • Results

    No predicted ordering failed on controlled images; post hoc, 61 of 89 orderings held significantly and 4 failed, all for OpenAI models given more pixels than their default path.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Better models can lower the tokens an object needs and carry more per call, but the cost of seeing the image remains.

    Source: §7 · p. 25

    The safety guarantees concern recall under monotonicity assumptions tested on a particular image distribution; they do not guarantee F1 or transfer to another distribution without testing.

    Source: Limitations · p. 25

Abstract

from arXiv · show

Vision-language models (VLMs) often miss small objects in large images. We ask three questions: what limits them, which of these limits better models can remove, and whether the classical way of handling large images, local decomposition, still has a future. We answer them with a theory built on two quantities of the image interface: S, the number of visual tokens across an object's side, and L, the content a call must cover. The limits: a W x H image sent whole within N tokens gives an object of side m at most m*sqrt(N/(WH)) tokens per side. Doubling the token budget N therefore raises the largest S a whole image can reach by only 41%, and seeing an image at the S an object needs costs at least order S^2 tokens whatever the model. If recognition improves gradually with S, any search strategy, zoom agents included, obeys a recall-cost frontier. What better models can change: the S an object needs and how much one call can carry, measured in bits per object found; the coverage cost remains. Decomposition: yes. Assuming only that more tokens per object and less content per call do not hurt on average, splitting an image cannot lower recall if no view zooms out relative to the whole image and views overlap by one object. Neither part of this condition can be dropped; we bound the cost of every such decomposition, and a simple rule approaches the bound as the image grows. We test the theory in about 177,000 requests on 797 images. On controlled images, none of 40 orderings predicted for 8 VLMs is violated. Checked after the fact on drawings, floor plans, natural and synthetic images, 61 of 89 implied orderings hold significantly and 4 fail, all for OpenAI models given more pixels than their default path. On construction drawings the rule never significantly lowered recall relative to the whole image and raised it by up to 0.28. Code and data: https://github.com/shijunzhe/vlm-small-objects

1 Introduction

The paper frames small-object detection through tokens per object side (S) and content per call (L), then uses these interface quantities to analyze model limits and safe image decomposition. Tests find that predicted orderings largely hold, while identifying exceptions for some higher-pixel OpenAI paths.

1 Introduction

  • The interface is described by S, tokens across a target side, and L, the content a call must cover and enumerate; remedies change these quantities in different ways.Tiling raises S and lowers L; high-resolution modes raise S but retain whole-image coverage; zoom tools can set both, while reasoning changes neither.Source: §1 · pp. 1–2
  • Under assumptions that more tokens per target and less content per call do not hurt on average, decomposition cannot lower expected recall if views do not zoom out and overlap by one object.The condition is exact, and a simple rule approaches the lower-bound cost as the image grows.Source: §1 · pp. 2–3
  • None of 40 predicted orderings failed on controlled images; post-hoc checks found 4 failures, while the safe rule never significantly lowered construction-drawing recall and raised it by up to +0.28.The 4 failures involved OpenAI models given more pixels than their default path; construction-drawing gains came at 11% to 106% of the recipe’s tokens.Source: §1 · p. 2
  • The 44 px tile recipe lowered recall in 19 of 21 cells, significantly in 12, when it zoomed out on large symbols.The theorem’s protection does not apply without its conditions.Source: §1 · p. 2
  • Recognition improves with S, and training moves that recognition curve: at the same token grid, Gemini 3.8 recalls +0.55 more of the same symbols than Gemini 2.5.The introduction also reports GPT-6.1 recalling 0.97 of symbols on 34 test drawings in one call.Source: §1 · pp. 2–3
  • Doubling a whole-image token budget raises the largest reachable S by only 41%, while seeing an image at a needed S costs at least order S^2 tokens.These coverage bounds follow from the interface, whatever the model.Source: §1 · p. 3
  • At fixed S, adding surrounding content reduces capacity; training can improve it, with recall on 96 isolated symbols rising from 0.57 to 1.00 for Claude Sonnet 4.6 to 5.5.For GPT-5.4 to GPT-5.6, recall rises from 0.85 to 0.99 on the same setup.Source: §1 · p. 3
  • Zoom agents reach 0.97 and 0.95 F1 on full sheets for GPT-5.6 and Claude Sonnet 5.5, respectively, by decomposing drawings themselves.Two of four agents count about right but fail to convert positions from crop to overview.Source: §1 · p. 3

2 Related Work

Prior work addresses small-object recognition through higher-resolution inputs, image decomposition, and zooming, while counting and drawing-specific methods offer related approaches. This paper organizes these remedies around interface quantities and tests their effects across several settings.

2 Related Work

  • Architectural remedies increase pixels through dynamic tiling or any-resolution encoders, while inference-time methods divide and recombine images or guide crop-and-zoom with attention, search, or learned policies.Prior work establishes that zooming helps; Liu et al. (2026) argue that its benefit mainly comes from removing background rather than enlarging the target.Source: §2 · p. 3
  • The paper asks what governs zooming gains by expressing configurations, including agent calls, through two pre-label interface quantities and experimentally separating their effects.In an agent ablation, overview crops without enlargement gave no significant gain; enlargement helped GPT-5.6 (+0.11), but not significantly Claude Sonnet 5.5.Source: §2 · p. 3
  • Counting research includes exemplar-based counters, image division, and benchmarks where accuracy falls as counts rise; overlays are also reported to help grounding and counting.This paper argues overlays cannot add Shannon information, although enlargement can add usable information, and measures per-call capacity directly.Source: §2 · p. 3
  • Engineering-drawing work includes dedicated datasets and methods, and recent benchmarks report VLM difficulty recognizing and counting blueprint symbols.With a clean exemplar, normalized cross-correlation template matching is a strong training-free baseline; drawings are the main testbed because they stress both interface quantities.Source: §2 · pp. 3–4
  • The paper distinguishes Shannon information from information usable by a computationally bounded predictor, which can change under processing.Overlays and interpolation add no Shannon information but, like enlargement, may affect how much a fixed model uses.Source: §2 · p. 4

3 Theory

This section develops a theory separating visual-token scale and call-coverage limits from assumptions about model behavior. It characterizes when image decomposition preserves detection, extends the analysis to local tasks and adaptive search, and bounds the associated information and coverage costs.

3.1 What the interface alone implies

  • The interface bounds whole-image resolution and decomposition coverage, while enlarging an input can help only when token geometry—not unresolved pixels—is the bottleneck.For fixed-grid interfaces, enlarging a whole image does not raise tokens per object; a smaller view can.Source: §3.1 · pp. 4–5
  • At least WHS*^2/(κ^2m^2) tokens are required to cover the drawing with views at scale S ≥ S*, regardless of their shapes, sizes, cores, or overlaps.The corresponding call lower bound is WHS*^2/(κ^2m^2N), with N = g^2 for fixed-grid interfaces.Source: §3.1 · p. 4, §3.2 · p. 6

3.2 What follows for any model that is monotone

  • Under (R) and (M), crops detect at least as many targets in expectation as the whole image when no view zooms out and the views cover every target interior to the whole-image call.The coverage condition holds when cores shrunk by half a target box cover the correspondingly shrunk drawing and each view contains its core expanded by that margin.Source: §3.2 · pp. 5–6
  • The safe-decomposition conditions are necessary: without adequate scale coverage, more tokens per target, or less content per call, a model satisfying the remaining assumptions can make decomposition miss targets the whole image detects.The tightness results show that neither (R) nor (M) can be dropped and that a target outside every qualifying kept core cannot be guaranteed protected.Source: §3.2 · p. 6
  • A tiled rule using S = max(S*, Sw), views overlapping by one target, and no further resizing preserves expected detections while approaching the lower cost bound as the drawing grows.Its cost ratio tends to the rounding factor χ, which is 1 for ideal whole-patch views and close to 1 when views span many patches.Source: §3.2 · pp. 6–7
  • If monotonicity is imperfect, the recall guarantee degrades according to coverage gaps, refinement and crop defects, and losses from merging duplicate hits, without a penalty that grows with tile count.Exact monotonicity, full coverage, and lossless merging recover the safe-decomposition guarantee.Source: §3.2 · pp. 7–8
  • Monotonicity fixes only the sign of decomposition’s recall gain, not its size; any gain from zero to the whole-image call’s miss rate is consistent with the assumptions.The amount of improvement is model-specific and must be measured.Source: §3.2 · p. 8
  • For F1, decomposition’s additional unmatched points can erase its hit gains; under the theorem’s conditions, expected-count F1 is at least the whole-image value when added unmatched points do not exceed added hits.Keeping full views as cores can retain false points near view edges and duplicate detections in overlaps.Source: §3.2 · pp. 7–8
  • For zoom tools, monotonicity orders a region cut from an overview, an enlarged crop, and native pixels from no worse to no worse to no worse; larger calls also offer no predicted gain in the tested designs.The ordering applies to targets interior to all compared calls and assumes a common core for the three region views. Packing tiles or adding drawing around a fixed region likewise predicts no gain when the larger call uses no greater scale.Source: §3.2 · pp. 8–9

3.3 Other local tasks

  • Under (R) and (M), decomposition of assignment tasks answers at least as many items correctly in expectation as a whole-image call, while enumeration-task results carry over with matching restricted to equal answers.For assignment tasks, each item receives exactly one answer, so wrong answers on background pixels or empty fields count alongside missed objects; the expected accuracy ratio is also no lower than for the whole image.Source: §3.3 · p. 9

3.4 The content of a call, in bits

  • For independent uniform target centres, expected recall is bounded by answer information in bits divided by Lbit, and any response-based policy must convey at least expected recall × Lbit bits.Lbit charges each hit by the log ratio of area per target to the tolerance disk; near full recall, a construction is within about 2.1 bits per target of the bound.Source: §3.4 · pp. 9–10
  • On controlled images, the best model’s whole-image answers carried at least about 422 bits, versus at least 438 bits needed for full recall without false points for 12 px symbols at recall 0.98.Both figures are lower bounds, so they indicate conveyed information rather than establishing that the model’s capacity binds.Source: §3.4 · p. 10

3.5 Adaptive zooming

  • Under a hard recognition threshold, any always-correct adaptive policy pays a coverage cost over eligible target locations; cropping and zooming therefore perform decomposition rather than escape it.An overview shown at or above the recognition threshold is itself a covering view whose tokens count; the stated comparison with tiling concerns cost only.Source: §3.5 · p. 10
  • The graded frontier is indicative rather than certified from measured detection curves, which provide only a lower bound on how much a call’s response can reveal about a target.The paper reports that recognition rises gradually rather than at a sharp threshold, making the hard-threshold bound weak for current models.Source: §3.5 · pp. 10–11
  • With graded recognition, any response-based search strategy faces a recall-cost frontier: finding fraction R̄ of targets across Ω costs about R̄|Ω|/(κ_f m²) tokens absent false points.The frontier covers adaptive and non-adaptive strategies, including tiles and zoom agents, and charges false points explicitly; if recognition rises more slowly than S² at small S, the bound can be vacuous.Source: §3.5 · p. 11

4 Experimental Setup

The experiments measure the theory’s open model parameters and test its predictions across construction drawings, floor plans, natural images, and controlled data. Conditions, matching rules, and registrations define how those comparisons are made.

4 Experimental Setup

  • The experiments test theory predictions and measure the model parameters left open by the theory: required S, call capacity, and coordinate interface.Table 1 summarizes experiments, runs, and the registrations fixing each experiment’s data, conditions, and hypotheses.Source: §4 · p. 11, Table 1 · p. 12
  • REDP-X40 contains 40 construction-drawing screenshots and 1,028 annotated instances; six images tune design choices, while the remaining 34 form the registered test set.The test set has 660 instances, and a post-run label audit found no missing labels.Source: §4 · p. 11
  • Conditions include whole-drawing and vendor high-resolution calls, tiled views, the Corollary 1 rule, zoom agents, and template matching.The zoom-agent variants separate lowering content per call, increasing scale without new information, and accessing native-crop information.Source: §4 · p. 13
  • Comparisons use one-to-one matching within half a symbol for strict F1; theory tests use hit recall, and significance is assessed with paired bootstrap intervals over drawings.The registered first-round and agent-control test families use Holm correction; other tests use unadjusted intervals.Source: §4 · p. 13
  • Every experiment was registered before its calls in six plans, with later-added conditions marked and deviations mapped in Appendix B.The plans cover the initial and newer-model rounds, synthetic sweeps and tools, assumption tests, and the safe rule.Source: §4 · p. 13
  • Experiments covered about 177,000 VLM requests on 797 images with about 28,600 annotated instances, across construction drawings, FloorPlanCAD, FSC-147, and controlled images.The controlled data include synthetic sweeps and an assumption battery; construction-drawing data include REDP-X40 and REDP-10.Source: §4 · pp. 11–13

5 Results

The results evaluate theory-implied orderings and how recognition and recall vary with token allocation, call coverage, and image decomposition strategies. Tests across controlled and held-out images show that gains from enlargement, tiling, and counting vary by model, object size, and image type.

5.1 The theory’s predictions, checked

  • None of 40 controlled-image orderings violated the theory, and 61 of 89 post-hoc orderings held significantly, while 4 violated it.Under the decomposition test, differences were at least −0.01 in all 40 model-data cells and significantly positive in 33 when each tile had S at least as large as the whole image; the condition matters because the 44 px recipe lowered S for some items. All four post-hoc violations involved OpenAI models given more pixels than their default path.Source: §5.1 · pp. 14, 16, Figure 2 · p. 15, Table 2 · p. 16

5.2 Tokens per symbol govern recognition

  • Recognition rose gradually with tokens per symbol S for every model, but the level varied: Gemini 2.5 reached S50 at 1.28, versus 0.31, 0.35, and 0.54 for Gemma 4, Claude Sonnet 4.6, and Qwen3-VL.With pixels fixed, Gemini 3.8’s recall for 8 px symbols rose from 0.44 to 0.71 across budgets; newer generations also changed recognition, and enlargement helped GPT-5.6 at 8 px (0.65 to 0.83) but not Claude Sonnet 5.5 (0.65 to 0.62). The drawing-level ceiling tracked F1, while tiles helped when that ceiling was low (+0.41 F1 for GPT-5.4); enlargement’s benefit depended on the recognizer.Source: §5.2 · pp. 16–18

5.3 Content per call governs capacity

  • At fixed S, recall generally fell as one call covered more targets or image area; with 96 targets, recall was 0.22 for Qwen3-VL, 0.48 for Gemini 3.8, and 0.57 for Claude Sonnet 4.6.For the same first block on FPC sheets, adding eight surrounding blocks lowered whole-image F1 by −0.22 for GPT-5.6, −0.16 for Claude Sonnet 5.5, and −0.34 for Gemini 3.8; tiles and GPT-5.6’s zoom agent stayed flat. Packing eight tiles lowered GPT-5.6 recall by −0.31 [−0.44, −0.16].Source: §5.3 · p. 18

5.4 Who splits: tiles, the rule, and zoom agents

  • On full sheets, GPT-5.6’s zoom agent reached 0.97 F1 versus 0.91 for matched-reasoning tiles (+0.06, Holm p=0.013); Corollary 1 never significantly lowered recall and raised it by up to +0.28.Tiles significantly beat whole drawings for every first-round model except GPT-5.4; the Corollary 1 rule used 1.6–3.5 times the lower bound Cmin. Zooming into the overview alone gave neither GPT-5.6 nor Claude Sonnet 5.5 a significant gain; enlarging crops raised GPT-5.6 F1 by +0.11, while native pixels added +0.01 (95% interval −0.003 to +0.03).Source: §5.4 · pp. 19–20

5.5 The coordinate interface is a further model parameter

  • Coordinate frame, scale, and output format changed scores substantially, and their behavior was not fixed even within a model.For Claude Sonnet 5.5, reading all answers in the documented resized frame gave F1 0.22 at eight packed tiles, versus 0.90 with a label-free per-answer frame rule. Gemini 3.8 and Qwen3-VL agents reported wrong positions despite counting well, scoring 0.63 and 0.21 versus 0.62 for Qwen3-VL tiles.Source: §5.5 · p. 21

5.6 Held-out drawings, public floor plans, and natural images

  • On 10 held-out REDP-10 drawings, tiles changed F1 by 0.00 [−0.06, +0.09] for GPT-5.6, −0.05 [−0.08, +0.02] for Claude Sonnet 5.5, and +0.23 [−0.004, +0.33] for Gemini 3.8.The configurations from Proposition 2 were applied unchanged; the supplied passage reports median S values of 1.25 for GPT-5.6 and 1.43 for Claude in whole-drawing calls, versus 0.92 for Gemini 3.8 and 2.5 in its tiles.Source: §5.6 · p. 21
  • On held-out drawings, only Gemini had a significant registered item-level tile gain (+0.08, [+0.08, +0.40]); zoom agents again outperformed tiles for GPT-5.6 and Claude Sonnet 5.5.The registered hypothesis of tile gains for every model was not supported, though the post-hoc geometric reading—gains only where S is low—matched the data. Template matching scored 0.38 (0.54 with its best REDP-10 threshold), and every VLM configuration scored above it by at least +0.30 [+0.14, +0.68]; zoom-agent advantages over tiles were GPT-5.6 +0.14 [+0.04, +0.19] and Claude Sonnet 5.5 +0.10 [+0.01, +0.14].Source: §5.6 · p. 21
  • On public floor plans, tiles raised F1 for Gemini 2.5 (+0.10 [+0.07, +0.13]) and Gemma 4 (+0.21 [+0.17, +0.24]), whose whole-image S was about one.For six models with whole-image S about two, the registered no-change expectation held only for Qwen3-VL (−0.03 [−0.07, +0.002]); Gemini 3.8 gained +0.08 [+0.05, +0.11], while GPT-5.4 (−0.10 [−0.14, −0.06]), Claude Sonnet 4.6 (−0.11 [−0.15, −0.08]), GPT-5.6 (−0.07 [−0.10, −0.03]), and Claude Sonnet 5.5 (−0.08 [−0.11, −0.04]) lost F1. The 44 px recipe shrank large-symbol images, lowering S for GPT-5.6 on 46% of items; zoom agents scored highest for both models that had one.Source: §5.6 · pp. 21–22
  • On FSC-147, upsampling helped GPT-5.6 and Claude Sonnet 5.5 on small objects, while tiles added gains on small, crowded images.Upsampling gains on small objects were GPT-5.6 +0.09 [+0.002, +0.17] and Claude Sonnet 5.5 +0.08 [+0.03, +0.12]; tiles added +0.14 and +0.20. Upsampling added little on large objects (+0.02 [+0.001, +0.04] and +0.02 [−0.003, +0.05]); GPT-5.6 also gained on medium (+0.06 [+0.03, +0.10]) and narrowly on large objects. Gemini 3.8 showed no upsampling effect but gained from tiles in every stratum, including large objects (+0.18 [+0.11, +0.27]); its fixed grid raised S and lowered content per call.Source: §5.6 · p. 22
  • CountGD and GeCo2 reached 0.90 and 0.86 F1 on FSC-147, but at most 0.25 on REDP-X40, so trained counters did not replace decomposed VLMs on drawings.Thresholds were not tuned per drawing; OWLv2 scored lower on REDP-X40.Source: §5.6 · p. 22

6 The Three Answers in Detail

The limits on small-object recognition divide between interface constraints that models cannot change and model parameters that training can improve. Decomposition remains useful when one call cannot both resolve targets and cover the image, while practical pipelines should preserve native detail and verify the coordinate interface.

6.1 What limits VLMs, and which limits the interface sets

  • A native-resolution zoom tool with reasoning reached F1 of 0.98 for GPT-5.6 and 0.97 for Claude Sonnet 5.5 on REDP-X40, while GPT-5.6’s high-resolution mode reached 0.33 with reasoning.By comparison, GPT-5.6’s default 1,600 px call reached 0.89 F1 with reasoning; the high-resolution path showed a failure of monotonicity.Source: §6.1 · p. 22

6.2 What better models can change: where the next models should improve

  • Halving the tokens per target side needed for recognition cuts complete-search cost fourfold, while doubling a whole-image token budget raises its maximum S by only 41%.The recognition threshold responds to training: Gemini 3.8 recalled 0.55 more of the same symbols than Gemini 2.5 at the same token grid.Source: §6.2 · p. 23
  • On a fixed block, adding eight surrounding blocks changed GPT-5.6’s F1 by −0.22; at equal S, cropping away content raised its recall by +0.39.With 96 isolated symbols in one call, recall rose from 0.57 for Claude Sonnet 4.6 to 1.00 for Claude Sonnet 5.5; the drawing-sheet content effect remains.Source: §6.2 · p. 23
  • The four observed safety-guarantee violations occurred in OpenAI models given more pixels than their default path, and GPT-5.6’s high-resolution mode lowered F1 from 0.89 to 0.33.The authors propose consistency across crops and scales and nested-pair tests; frame, scale, and effective resolution can also vary by vendor, mode, or answer.Source: §6.2 · p. 24

6.3 Does decomposition, the classical idea, have a future?

  • Decomposition remains necessary when one call cannot pay the coverage cost and carry the image content; newer models can perform it through tools or within their architecture.On REDP-X40, GPT-6.1 recalled 0.97 of symbols in one call and the safe rule added only +0.03; on full FPC-sheets it recalled 0.63, and the rule did no better.Source: §6.3 · p. 24

6.4 Practice today

  • For exhaustive, auditable work, send native detail in tiles or native-file crops, verify coordinate frames, and retain per-item locations for review.The authors also recommend parsing vector sources where available; their measurement battery checks budget, token size, frame, scale, recognition threshold, per-call capacity, and agent coordinate conversion.Source: §6.4 · pp. 24–25

7 Conclusion

The paper separates interface-imposed limits from model parameters and derives when image decomposition can preserve recall. Its tests support the predicted orderings on controlled images, while better models can reduce—but not remove—the cost of seeing large images at sufficient resolution.

Conclusion

  • Under tested monotonicity assumptions, decomposition is provably safe when no view zooms out relative to the whole image and views overlap by one object.Neither condition can be dropped, and a simple rule approaches the lowest cost of such decomposition as image size grows.Source: §7 · p. 25
  • None of the predicted orderings failed on controlled images; in other experiments, the only failures came from OpenAI models given more pixels than their default path.Better models can lower the tokens an object needs and carry more content per call, but the cost of seeing a large image at sufficient resolution remains.Source: §7 · p. 25
  • Until one call can pay the coverage cost and carry the content, decomposition persists, increasingly performed by models themselves.The paper identifies fewer tokens per object, greater per-call capacity, crop-and-scale monotonicity, and explicit frame and scale as areas for progress.Source: §7 · p. 25

Limitations

The safety results rely on monotonicity assumptions assessed on average over an image distribution, and the guarantee does not automatically transfer to another distribution. Some measured limits and effects are specific to the model interface and evaluation setting.

Scope of the assumptions

  • A guarantee established for one image distribution does not carry over to another without testing there.The safety guarantees hold for models satisfying the monotonicity assumptions, which current models may violate or satisfy only weakly.Source: Limitations · p. 25
  • For the four observed failures in OpenAI models given more pixels than their default path, the guarantees cover the default path only.The guarantees concern recall; the paper gives no F1 guarantee because false outputs are not monotone under cropping.Source: Limitations · p. 25

Deviations from the plan

  • Several registered expectations failed, and the authors report post-result changes to the Claude Sonnet 5.5 packed condition and the FSC-147 prompt.The FSC-147 test was repeated with a domain-neutral prompt after the drawing prompt led GPT-5.6 to report single objects.Source: Limitations · p. 25

Evaluation scope

  • The study evaluated commercial model versions available in September and October 2026, and those models can change without notice.Tiled and propose-and-verify pipelines cost 13 to 146 calls per drawing; the zoom agents used about 11 to 29 thousand input tokens per drawing for GPT-5.6 and Claude.Source: Limitations · p. 25

Ethics Statement

The released REDP-X40 drawings are partial screenshots without project-identifying information; REDP-10 drawings are used for evaluation but not released. The paper reports no human subjects and recommends review safeguards for systems used in cost estimation.

Ethics Statement

  • REDP-X40 drawings are released as partial screenshots without project-identifying information, while REDP-10 drawings are evaluation-only and not released.Source: Ethics Statement · p. 26
  • The work evaluates commercial and open models through their public interfaces and involves no human subjects.Source: Ethics Statement · p. 26
  • Because cost-estimation counts can cause financial errors, the authors recommend retaining per-item locations so results can be checked.Source: Ethics Statement · p. 26

A Proofs

This section proves geometric and information-theoretic bounds on target-finding and decomposition costs, and establishes when decomposition preserves recall. It also bounds cell-based localization and verification performance, including F1, and examines how the conditions and proposer quality affect these guarantees.

Localization and information bounds

  • Under token-quantized position, localization success is at most φ(S/2), equal to πS^2/4 for S ≤ 1 and 1 for S ≥ 1.The ceiling is attained by reporting the center of the token cell containing the target.Source: A Proofs · p. 28

Decomposition guarantees

  • Under crop and refinement monotonicity, a decomposition whose views cover the whole-image interior preserves or improves expected detections there.The proof partitions the whole-image interior across views and applies the two monotonicity conditions on each part.Source: A Proofs · p. 29

Limits of the assumptions

  • The decomposition guarantee depends on its assumptions: examples satisfying the other conditions show that dropping refinement or crop monotonicity can make decomposition miss targets found by the whole-image call.A separate threshold model satisfying the assumptions detects targets below r_w with the whole-image call but below r_d with the decomposition, under the stated coverage condition.Source: A Proofs · pp. 29, 31, 32

Decomposition guarantees

  • A tiled-view rule covers every whole-image interior target and never significantly lowers recall; its cost bound is attainable, with χ = 1 for the stated view geometries.The construction tiles the interior using overlapping views; equality in the cost ratio holds for the specified square-target or fixed-grid settings.Source: A Proofs · p. 30
  • With coverage fraction ψ and merge loss bounded by ζ times the whole-image interior count, the decomposition’s expected detections obey the theorem’s defect bound, attained by a construction when ζ = 0.The proof requires no independence assumption.Source: A Proofs · p. 31

Localization and information bounds

  • The information needed to report detections is bounded below by E[Rec]Lbit, while a propose-and-verify system’s F1 is bounded by proposer coverage and verifier acceptance.No verifier can recover an instance the proposer missed; both F1 bounds coincide at 2q/(1 + q) when α1 = 1 and α0 = 0.Source: A Proofs · pp. 28, 32

Proposition 8

  • A policy that is correct with probability at least 1 − δ must incur expected cost at least (1 − 2δ)|D|S_rec^2/m^2.The proof shows that, on average, views reaching S_rec must cover at least (1 − 2δ)|D| of possible target centres; overlapping views only overcount cost.Source: A Proofs · p. 33
  • This version fits the gradual recognition curves of Section 5.2 better than a hard threshold.Source: A Proofs · p. 33

Proposition 10

  • For isolated instances, one report per occupied cell has expected recall bounded by φ(S/2), while k reports per cell give min(1, kφ(S/2)).The cell-centre estimate maximizes localization probability; a Monte Carlo check matched the closed form within 0.002 at every tested S.Source: A Proofs · pp. 33–34

Proposition 11

  • The verifier's F1 lies between 2α1u/(α1u + α0v + n) and 2u/(u + n), with both bounds coinciding when α1 = 1 and α0 = 0.Here u is the maximum-matching size of all candidates, and the lower bound applies directly to the paper's maximum-matching metric.Source: A Proofs · p. 34

B Pre-registered Hypotheses, Outcomes, and Deviations

Registered comparisons found mixed support across model generations, token budgets, and image-viewing conditions. The section also records deviations from preregistered procedures and analyses.

B Pre-registered Hypotheses, Outcomes, and Deviations

  • Four of 20 cells exceeded the quantized-observer ceiling by more than 0.05, so the ceiling hypothesis was not supported, as expected.The 20-cell comparison was reported as a second-round result.Source: B Pre-registered Hypotheses, Outcomes,… · p. 34
  • At the same geometry, GPT-5.6 improved by +0.17 [+0.08, +0.27], while GPT-5.5 changed by −0.07 [−0.16, +0.03].The larger-budget hypothesis was supported for Claude Sonnet 5.5 (+0.53 [+0.37, +0.68]) and Gemini 3.8 (+0.52 [+0.33, +0.70]) on sheets, with the registered pixel-reading exception for Gemini.Source: B Pre-registered Hypotheses, Outcomes,… · p. 34
  • On sheets, tiles beat the whole drawing for Claude Sonnet 5.5 (+0.24 [+0.06, +0.43]) and Gemini 3.8 (+0.36 [+0.18, +0.55]); GPT-5.6's +0.13 [−0.09, +0.37] was inconclusive.The registered tile hypothesis was supported for two of three models.Source: B Pre-registered Hypotheses, Outcomes,… · p. 34
  • Native-resolution sheets fell below both the default whole-drawing condition and tiles: native-minus-default was −0.28, while tiles-minus-native was +0.41.Because the native condition also fell below the default, the tile comparison does not isolate capacity and indicates nonmonotonicity in the high-resolution path.Source: B Pre-registered Hypotheses, Outcomes,… · p. 34
  • Propose-and-verify met the template-matching comparison within 0.05, with a spread across eight models of 0.08; the per-drawing dose response was supported for new models in the registered pixel reading.For that dose response, ρ ranged from 0.50 to 0.62 with all intervals above zero; Gemini 3.8's native-reading estimate was 0.33 [−0.05, 0.66].Source: B Pre-registered Hypotheses, Outcomes,… · p. 34
  • Recorded deviations included 20 rather than 16 agent zooms, a lowest-accepted-effort run for GPT-6.1 Sol, and added or adjusted analyses and reading procedures.These included recovering Qwen3-VL list-valued points after early rounds, registering additional experiment sets, and adding some analyses after calls or results were available.Source: B Pre-registered Hypotheses, Outcomes,… · pp. 34–36
  • The 44 px tile recipe was capped at 12,000 px for FloorPlanCAD after one 6 px reference would have enlarged a 3,000 px sheet to 22,000 px; only that item was affected.The expected tile recovery for large symbols was a heuristic, not a theoretical consequence, and was unsupported for Qwen3-VL on FPC-sheets (−0.09 [−0.16, −0.03]).Source: B Pre-registered Hypotheses, Outcomes,… · p. 36
  • The safe rule did not lower recall below the whole-image call in all 8 cells; 4 were significantly positive, and its median token cost was 2.0 times Cmin.The identity check against Corollary 1's cost bound held for every drawing.Source: B Pre-registered Hypotheses, Outcomes,… · p. 36

C Measured Input Geometry

The measured input geometry exposed undocumented vendor behavior and coordinate-frame differences, while checks on the tuning drawings informed experimental choices.

C Measured Input Geometry

  • GPT-5.5 and GPT-5.6 accepted at least 8,192 patches at default, although documentation lists 2,500 for high; Gemini 2.5 ignored HIGH when a request contained two images.The study measured billed input tokens using blank white images of ten sizes.Source: C Measured Input Geometry · p. 36
  • On a 1600×1000 coordinate probe, Gemini 2.5 Flash erred by 24 to 203 px, about one token, while GPT-5.5, GPT-5.6, Claude Sonnet 5.5, and Gemini 3.8 stayed within about 1 px.Claude Sonnet 4.6 answered in its internally resized frame and erred by 0.3 to 2.3 px after rescaling.Source: C Measured Input Geometry · p. 36
  • Resolving the target in one call required a median of 9,198 image tokens on 14 test sheets (maximum 36,520) and 1,156 tokens on views.A 10,000-token budget sufficed for 57% of sheets, and 40,000 tokens for all of them.Source: C Measured Input Geometry · p. 36

D Mechanism Checks on the Tuning Set

  • On six tuning drawings, padded 384 px tiles raised F1 from 0.34 to 0.61 for GPT-5.4 and from 0.15 to 0.60 for Claude Sonnet 4.6.The same pixels were placed on a white 768 px canvas; this check informed the choice not to send tiles below about 768 px.Source: D Mechanism Checks on the Tuning Set · p. 36
  • On tuning tiles, telling Claude most contained no target cut false positives from 0.79 to 0.26 per true instance but lowered recall from 0.79 to 0.71; rulers raised Gemini 2.5 F1 from 0.31 to 0.40.Whitening half of tiles and querying the halves separately raised tuning F1 from 0.66 to 0.77 and from 0.58 to 0.68; on the test set, the effect shrank to +0.03 and +0.09.Source: D Mechanism Checks on the Tuning Set · p. 36

E Controlled Stimuli

The controlled stimuli use synthetic drawings to vary visual resolution, call capacity, image area, clutter, and upsampling, with measured interface geometry used to characterize scale.

E Controlled Stimuli

  • Each synthetic image placed n copies of a target template and n distractors on a white canvas, with jittered-grid spacing and thin grey line segments.Seeds were fixed per image, and points were matched within m/2.Source: E Controlled Stimuli · p. 37
  • The sweeps varied target size, target count, canvas area, line clutter, and twofold Lanczos upsampling, using six images per level.For example, the resolution sweep used m values from 8 to 64 px, while the capacity sweep varied n from 2 to 96.Source: E Controlled Stimuli · p. 37
  • S was computed per model from measured interface geometry, and S50 was defined as the midpoint of a logistic fit of per-image recall on log S.Intervals for S50 resampled images over 1,000 draws.Source: E Controlled Stimuli · p. 37
  • For controlled stimuli and the mosaic, the reported robust reading recovered some malformed JSON centres and selected Claude Sonnet 5.5's frame per answer on inputs above its tier.The frame choice used only the image; strict parsing and the documented frame were used for the registered reading.Source: E Controlled Stimuli · p. 37

F FloorPlanCAD Details

The FloorPlanCAD analysis registers its construction and scoring choices in advance, then compares results across nested drawing areas. Its all-block scores reveal a reference-matching confound, so it also reports a post hoc first-block analysis.

F FloorPlanCAD Details

  • FPC-sheets contains 60 items at each of three nested areas, with one class per source drawing selected by the stated instance-count criteria.FloorPlanCAD blocks cut from larger drawings truncate edge-crossing instances, so predictions and annotations within m/2 of a block edge are ignored.Source: F FloorPlanCAD Details · p. 37
  • Median whole-image S on FPC-300 ranged from 1.00 for Gemini 2.5 to 2.23 for both Claude models.Across FPC-sheets, median S fell as area increased for every listed model, including GPT-5.6 from 3.0 at k = 1 to 1.62 at k = 3.Source: F FloorPlanCAD Details · p. 37
  • On the first block, tiles changed by at most 0.08 for every model, while whole-image calls declined for every model; all intervals for tiles reached zero and whole-image intervals were below zero.The registered all-block analysis found every method declined with area, including template matching (0.54 to 0.18), because instances in other blocks differed more from the first-block reference.Source: F FloorPlanCAD Details · p. 37

G Beyond Drawings: FSC-147

The FSC-147 test compares a drawing-oriented prompt with a neutral exemplar prompt while keeping other settings fixed. The drawing wording often led GPT-5.6 to report just one natural-image object, with low F1 even for large objects.

G Beyond Drawings: FSC-147

  • The drawing prompt often led GPT-5.6 to report a single object on natural images, with low F1 even for large objects.The registered neutral prompt instead asks for every object of the same kind as the exemplar and changes no other setting.Source: G Beyond Drawings: FSC-147 · p. 38
  • The neutral prompt asks for every object of the same kind as the exemplar, changing no other setting.It was registered before its run.Source: G Beyond Drawings: FSC-147 · p. 38
  • Predicted points are matched to annotated dots within m/2.Source: G Beyond Drawings: FSC-147 · p. 38

H Sensitivity and Error Analysis

The sensitivity analysis checks whether key comparisons survive altered scoring tolerances and repeated runs, and separates error types. Most reported directions persist, with a specific tolerance-related exception and a model-specific verifier trade-off.

H Sensitivity and Error Analysis

  • At twice the tolerance, GPT-5.4 tiles scored 0.80, overtaking Gemma 4 at 0.76 and coming within 0.07 of template matching.This was the sole comparison-direction exception; Claude Sonnet 4.6's whole-drawing score rose from 0.36 to 0.50.Source: H Sensitivity and Error Analysis · p. 38
  • 0.35 spurious reports per instance for GPT-5.6 with propose-and-verify exceeded 0.15 with tiles, while whole-drawing calls missed 0.35 of instances and added 0.24 spurious reports per instance.Whole-drawing counts were closer to the truth than positions; propose-and-verify removed nearly all displaced reports and most spurious reports for most models.Source: H Sensitivity and Error Analysis · p. 38

I Cost

The cost subsection lists calls and tokens per drawing for each pipeline. The accompanying prompts specify how tiles, verification crops, and whole drawings are presented to models.

I Cost

  • Table 15 lists the calls and tokens per drawing for each pipeline.Source: I Cost · p. 38

J Prompts

  • The tile prompt presents one target-symbol reference followed by a drawing tile and asks for instances centered inside its blue rectangle.The area outside the rectangle is context only; the response uses a count and center list.Source: J Prompts · p. 38
  • Verification presents a target-symbol crop and a same-scale drawing crop marked with one candidate location, asking whether the candidate matches.Rotated or slightly rescaled instances may match if they retain the reference's distinctive shape.Source: J Prompts · p. 38
  • The whole-drawing prompt supplies the legend crop and drawing with image size, then requests every instance's pixel center and a total count.Source: J Prompts · p. 38

K Further Results

Further results probe safe tiling, tolerance changes, model-specific configurations, candidate verification, and trained counters. They also examine how these findings vary across datasets and operating conditions.

Label audit

  • The label audit found no missing labels among locations flagged for review, with a one-sided 95% upper bound of 5% of flagged locations.Because the annotator also created the labels, the check bounds omissions but not systematic disagreement about what counts as an instance.Source: K Further Results · p. 38

Unmatched points under safe tiling

  • Safe tiling raised F1 for all 8 models by +0.04 to +0.32, significantly for 4; 7 models added fewer unmatched points than hits.Gemma 4 added 5.7 unmatched points per additional hit, within its affordable rate ρw = 49, because whole-image recall was 3%.Source: K Further Results · p. 38

The ceiling against log S

  • On REDP-X40, log S explains as much variance as the ceiling for GPT-5.4 (R2 0.62 vs 0.62) and more for Claude Sonnet 4.6 (0.52 vs 0.39).The ceiling adds significant variance only for Gemma 4 (+0.11, p = 0.001); on FPC-sheets it adds variance beyond log S in 1 of 10 settings (at most 0.02 R2), while on FPC-300 both explain little (R2 ≤0.15).Source: K Further Results · pp. 38–39

First-round configurations

  • With model-specific tiles, Gemma 4 moves from third to first in the model ranking.Table 16 reports first-round strict F1 on 34 test drawings under two pixel conventions.Source: K Further Results · p. 39, Table 16 · p. 44

Propose and verify

  • Propose-and-verify scores ranged from 0.83 to 0.91, with model spread falling from 0.34 for model-specific tiles to 0.08.Candidates contained true instances at a ceiling of 0.98; verifiers accepted 0.81–0.93 of true candidates and 0.03–0.14 of false ones. Current tiles were within 0.03 of or above this method.Source: K Further Results · p. 39

Trained counters in detail

  • On FSC-147, CountGD and GeCo2 reached 0.90 and 0.86 F1, above GPT-5.6’s 0.75 on the same whole images; on REDP-X40 their scores were 0.10 and 0.23.The authors attribute the drawing results to domain differences: thin line instances among look-alike symbols differ from the textured objects these counters were trained on.Source: K Further Results · p. 39

Agent tokens

  • Per drawing, GPT-5.6’s agent used 29.3k input and 2.9k output tokens, versus 16.8k and 0.5k for its tiles.Claude’s agent used 11.3k input and 1.4k output tokens, about half its tiles’ 20.6k and 2.3k.Source: K Further Results · p. 39
Loading 2610.09313v1…