Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench

• Existing fine-grained evaluations emphasize text reading, chart reasoning, or artificial low-level vision tests, while instrument-reading studies cover specific types rather than diverse instruments and designs.Instrument reading requires fine-grained visual perception, light quantitative reasoning, and basic arithmetic.Source: §1 · p. 1
• MeasureBench spans 26 instrument types and four readout designs, with 2,442 pairs: 1,272 human-annotated real-world images and 1,170 synthetic images for 39 instruments.Source: §1 · pp. 1–2
• The automated synthesis pipeline combines a controllable 2D renderer for layout, fonts, and geometry with a 3D Blender renderer for realistic lighting, materials, reflections, and occlusions.It scales across instrument types and variations and generates precisely labeled sketch or photorealistic images with randomized readings for 39 instruments.Source: §1 · p. 2
• 30.2% real-world and 26.3% synthetic accuracy were achieved by the best model, underscoring that current VLMs still struggle with instrument reading.Source: §1 · p. 2
• Across 18 contemporary VLMs, unit recognition exceeds 90% accuracy, but models struggle to map scales to numbers and misread pointers, adjacent ticks, or scale-value matches.Source: §1 · p. 2
• Preliminary reinforcement finetuning with synthetic data significantly improves the synthetic subset, but is less promising on real-world images and highlights the need for better visual representations for generalization.Source: §1 · p. 2

2.1 Overview of MeasureBench

• MeasureBench contains 2,442 questions: 1,272 real-world images and 1,170 synthetic images, spanning 26 real-world instrument types and 16 synthetic types.The benchmark emphasizes dial and linear instruments because digital devices primarily test OCR capabilities and composite instruments are comparatively rare in practice.Source: §2.1 · p. 3

2.2 Evaluation metrics

• Readings are graded by interval matching rather than exact-value matching, with full correctness requiring a correct value and, when specified, a correct unit for the same candidate.A prediction is value-correct when it falls within a candidate's numeric interval; candidate scoring prefers full correctness, then value correctness, with tie-breaking by relative error and interval width.Source: §2.2 · pp. 3–4

2.3 Real-world subset curation

• The real-world subset combines permissively licensed search images, team-contributed photos, and purchased images, with independent labeling, verification, and adjudication.Ten annotators labeled images according to their professional backgrounds; an additional independent review checked annotation correctness, including numeric intervals and units.Source: §2.3 · p. 4

2.4 Data synthesis framework

• The synthesis framework uses a shared generator interface and 2D programmatic and 3D physical-rendering back ends to generate 1,170 images across 39 appearances and 16 instrument types.It randomizes scales, readings, ranges and units, materials, lighting, backgrounds, and camera pose while enforcing semantic validity; 2D generators are code-drafted by LLMs and verified, while 3D rendering uses Blender assets to produce photorealistic images.Source: §2.4 · pp. 4–5

3.1 Main results

• 30.2% overall accuracy on real images and 26.3% on synthetic images is the best result, achieved by Gemini 2.5 Pro; value reading is substantially harder than unit recognition.For Gemini 2.5 Pro on real images, unit accuracy is 96.2% versus 30.7% value accuracy. Digital displays are easier, while dials and linear instruments typically score 10–32%, and composite instruments are the most challenging.Source: §3.1 · pp. 5–7

3.2 To think, or not to think?

• On MeasureBench, enabling thinking yields very little improvement and sometimes degrades performance; it increases average reasoning-token use modestly—by a few hundreds up to 1k-2k tokens—while accuracy gains remain limited.Source: §3.2 · p. 7

3.3 Prompt sensitivity analysis

• Naming the instrument moderately improves unit accuracy but does not materially change total accuracy, while asking for more precise answers has no consistent effect.The authors therefore retain the original simple prompts for the main benchmark evaluation.Source: §3.3 · p. 7, Table 3 · p. 8

3.4 Case studies

• Models generally identify the instrument, indicator, unit, and tick spacing, but small localization or indicator-interpretation errors can substantially change the reading.Some correct final answers also arise from error cancellation, so scoring only the final number can overstate reasoning correctness.Source: §3.4 · p. 8

3.5 Impact of detailed instructional prompts

• Detailed, instrument-specific reading instructions yield very limited gains on real-world images, and adding in-context examples does not seem to help.The results suggest the bottleneck is fine-grained visual perception rather than missing procedural knowledge.Source: §3.5 · p. 9, Table 4 · p. 9

3.6 Generalization on general benchmarks

• GRPO training on synthetic measurement data preserves Qwen2.5-VL-7B’s performance on general-purpose benchmarks, with no observed degradation.The evaluation tests for potential negative transfer from reinforcement fine-tuning on synthetic data.Source: §3.6 · p. 9, Table 5 · p. 9

3.7 Comparison with supervised fine-tuning

• SFT improves in-domain synthetic performance but degrades real-world accuracy, whereas GRPO improves performance on both subsets.The comparison uses the same synthetic dataset for Qwen2.5-VL-7B; SFT tests direct-answer and rationale-plus-answer formats.Source: §3.7 · p. 9, Table 6 · p. 9

3.8 Would earlier special-purpose systems work?

• Earlier gauge-reading systems generalize poorly to MeasureBench, failing on components such as gauge detection, OCR, and pointer segmentation.The evaluated systems include specialized pipelines from Shu et al. (2023) and Reitsma et al. (2024); their pointer-value detection and OCR-based unit recognition perform worse than general-purpose VLMs on the benchmark subset.Source: §3.8 · p. 10, Table 7 · p. 10

4 Can training with synthetic data help?

• The authors train Qwen2.5-VL-3B and Qwen2.5-VL-7B with GRPO on 3,900 image–question pairs, generated as 100 samples for each of 39 instruments.The task format supports reinforcement learning by assigning a positive reward to correct readings.Source: §4 · p. 10

4.1 Reward design

• The study uses a rule-based reward aligned with the evaluation method for training with reinforcement finetuning.A soft reward variant for predictions close to the ground-truth interval did not produce much difference.Source: §4.1 · p. 10, §4.2 · p. 10

4.2 Results and analysis

• RFT gains on Qwen2.5-VL-7B occur primarily for simpler instruments, while composite reading designs remain challenging.Top gains include single-needle dials, digital readouts, and linear gauges with sparse ticks; Figure 7 covers the top 10 categories with at least 20 samples.Source: §4.2 · p. 10, Figure 7 · p. 11
• Synthetic-data RFT raises Qwen2.5-VL-3B synthetic accuracy from 8.4% to 31.5% and Qwen2.5-VL-7B real-world accuracy from 14.6% to 19.7%.The authors describe the in-domain increase as more than threefold and the OOD real-world gain as notable.Source: §4.2 · p. 10, Table 8 · p. 11
• After RFT, Qwen2.5-VL-7B’s spikes at round multiples of ten are reduced, and its prediction distribution becomes smoother and closer to Gemini-2.5-Pro’s.Before RFT, the spikes indicate a language-model prior favoring round values over visual evidence.Source: §4.2 · p. 11, Figure 8 · p. 11
• The results suggest potential for further data curation, while raising whether improved architectures and visual encoding are needed to reason from detailed visual cues and generalize to unseen instruments.The authors frame these as an open question for future work.Source: §4.2 · p. 11

VLMs & VLM Benchmarks

• VLM benchmarks span holistic multimodal competence, academic problem solving, visual mathematical reasoning, and fine-grained visual understanding.Examples include MMBench, MMMU (Yue et al., 2024; 2025), MathVision (Wang et al., 2024), and CV-Bench.Source: §5 · p. 11
• Instrument reading remains underexplored despite benchmarks that expose persistent VLM weaknesses in low-level perception, shape, geometry, and spatial reasoning.SalBench, BlindTest, SRBench, and VisOnlyQA report these difficulties, while instrument reading additionally requires localized perception and light numerical computation.Source: §5 · pp. 11–12

Measuring Instruments Reading

• Instrument reading combines fine-grained perception, text reading, and visuospatial reasoning to infer values and units from scales and pointers.Typical pipelines detect or segment scales and pointers, rectify geometry, and use OCR or tick-interval estimation.Source: §5 · p. 12
• Traditional instrument-reading methods target specific device families and generalize poorly across device types, design variations, viewpoints, glare, occlusion, and unit ambiguity.Prior work addresses rulers (Pan et al., 2025), clocks (Yang et al., 2022; Saxena et al., 2025), water meters, and pressure gauges (Reitsma et al., 2024).Source: §5 · p. 12
• Existing VLM instrument-reading evaluations cover limited device diversity and seldom assess calibrated numeric error, unit normalization, tolerance bands, or robustness stressors.GPT-4o (OpenAI, 2024) reports preliminary ability on industrial gauges, while CAD2DMD-SET (Valente et al., 2025) evaluates VLMs on digital measurement devices.Source: §5 · p. 12

6 Conclusions and Discussion

• MeasureBench combines real-world and synthetic subsets to benchmark VLM instrument reading.Its synthetic generation pipeline provides controlled benchmark data and training augmentation.Source: §6 · p. 12
• Current VLMs struggle with fine-grained visual cues and precise visual–numeric correspondences, causing value-estimation and unit-normalization errors.Source: §6 · p. 12
• The synthetic generation pipeline enables controlled benchmarks and serves as training-data augmentation.Source: §6 · p. 12
• Preliminary GRPO reinforcement finetuning results suggest small amounts of targeted synthetic data yield measurable gains that transfer to real-world settings, but only moderately.Source: §6 · p. 12
• The authors identify comprehensive training-data curation and improved visual representation modeling as directions for developing VLM fine-grained understanding, geometric alignment, and spatial reasoning.Source: §6 · p. 12

A.1 Training details

• RFT uses the verl framework and GRPO with a format reward that trains the model to output its thinking process inside <think>...</think> tags.The training data comprise 3,900 synthesis-dataset images.Source: §A.1 · p. 15
• Training runs for 15 epochs on 8×H100 GPUs, with a global batch size of 128, learning rate of 1 × 10−6, and rollout number of 8.Source: §A.1 · p. 15

A.2 Soft-margin Reward

• Soft-margin reward yields performance comparable to hard-margin reward after RFT on real-world and synthetic subsets.It gives partial credit to near-miss numeric predictions while leaving the formatting term unchanged; exact answers retain evaluation-consistent rewards.Source: §A.2 · pp. 15–16, Table 9 · p. 16

B Examples of synthesized images

• Figure 9 shows synthesized instrument images rendered with offline libraries for 2D and Blender for more realistic 3D images.Generators are designed to render instruments with similar appearances and randomized readouts.Source: B Examples of synthesized images · p. 16, Figure 9 · p. 16

C 3D modeling with Blender; C.1 Asset retrieval

• The 3D asset pipeline combines BlenderKit assets with Sketchfab models for underrepresented categories, including pointer-based and linear-scale instruments.The modeling workflow uses Blender v4.2 and repositories containing models, HDRs, and materials.Source: C 3D modeling with Blender · p. 17, §C.1 · p. 17

C.2 Model normalization

• Pointer-based models were reset to zero, while linear-scale models were geometrically adjusted so pointer movement matched the instrument’s measurement scale.For pointer models, the pointer was rotated to zero and transformations were reset; for linear scales, minimum-maximum mappings were determined.Source: §C.2 · p. 17

C.3 Contextual scene augmentation

• Context objects, such as fruit on a scale or a pen beside a ruler, made nonzero readings visually consistent and grounded measurement queries.The additions improved scene realism and reduced ambiguity for vision-language evaluation.Source: §C.3 · pp. 17–18

C.4 Pointer rotation control

• Blender’s Python API controlled pointers using target-time angles for clocks and model-specific linear mappings for other dials.Rotation axes varied with model orientation, and the approach generalized to linear or semi-linear dial mappings.Source: §C.4 · p. 18

C.5 Camera alignment; C.6 Lighting and environment HDRs

• Camera alignment and lighting were tuned to improve dial legibility, using HDR environments for realistic reflections or direct lights for neutral backgrounds.Camera distance and viewing angle varied by instrument size; lighting strategy depended on dataset requirements.Source: §C.5 · p. 18, §C.6 · p. 18

C.7 Rendering execution

• Rendering scripts ran in Blender’s Scripting panel or externally through Python, enabling automated rendering across instrument categories.Figure 10 illustrates examples of the described 3D model augmentation strategies.Source: Figure 10 · p. 17, §C.7 · p. 18

D Extended Analysis

• The section analyzes model behavior on specific challenging cases and reports statistical distributions of numerical outputs.Source: D Extended Analysis · p. 19

D.1 Reading of complex measuring instruments

• All three evaluated models describe the electricity-meter reading procedure correctly but misread almost all five pointers, producing incorrect final readings.The example illustrates how fine-grained perception or interpretation errors can propagate to the final result.Source: Figure 11 · p. 19, §D.1 · p. 19
• The electricity meter has five dials with pointers that must be read from left to right, with adjacent dials rotating in opposite directions.The reading depends on localizing each pointer and interpreting its position within the dial sequence.Source: §D.1 · p. 19
• For a pointer between two numbers, record the lower number, except between 9 and 0, where the recorded number is 9.Source: §D.1 · p. 19
• Errors in fine-grained visual perception or reading interpretation can propagate through a complex instrument reading to an incorrect final result.Composite readouts, multiple dials, and multiple pointers remain highly challenging for current VLMs.Source: §D.1 · p. 19

D.2 Example of correct “guessing”

• The ammeter’s pointer indicates 20 A, but GPT-5 estimates 26 A while GPT-5-Mini gives the correct 20 A despite clear uncertainty.Source: §D.2 · p. 19
• GPT-5-Mini’s correct ammeter answer is described as essentially a guess, showing that correctness alone need not indicate accurate visual understanding.Both GPT-5 and GPT-5-Mini provide detailed reasoning before their final answers.Source: §D.2 · p. 19

E The “10:10” phenomenon

• Real-world collection deliberately excludes “10:10” images, and synthesis samples clock times uniformly, leaving few “10:10” ground truths in the benchmark.Source: E The “10:10” phenomenon · p. 19
• Qwen2.5-VL-72B-Instruct answers “10:10” for 72.88% of real-world clock images and 50.74% of synthetic clock images.Source: Table 10 · p. 20, E The “10:10” phenomenon · p. 20
• RFT-trained Qwen2.5-VL-7B predicts “10:10” on 3.39% of real-world clock images, versus 23.73% for the original model.The RFT model was trained on synthetic data with uniformly distributed clock times.Source: E The “10:10” phenomenon · p. 21
• GPT-5 predicts “10:10” in more than 20% of its answers on real-world clock images.Source: E The “10:10” phenomenon · p. 21

F Spikes distributions at integers

• InternVL3.5 and Qwen2.5-VL outputs show significant spikes at multiples of ten, with similar patterns within each model series.Source: Figure 13 · p. 21, Figure 14 · p. 21, F Spikes distributions at integers · p. 21
• Thinking mode does not mitigate the spikes, which may reflect training data that represents round numbers more frequently.Source: F Spikes distributions at integers · p. 21