PaperlayerView exact revision on arXiv

Source-linked AI summary

RealityBench: Simulated Environments to Evaluate a Vision Language Model's Ability to Document Human Physical Work

Andrew Perrault, Anmol Takiar, Dominik Winecki, Arnab Nandi

arXiv:2610.14397v2cs.CVcs.AIcs.CL

TL;DR

Physical-work VLM capabilities are difficult to benchmark because field data is often proprietary and highly variable. RealityBench models eleven work environments in Blender and evaluates acceptability and documentation; text-in-frame tasks are largely solved, but measurement and scene judgments remain weak, while rankings transfer on three matched real-image questions.

  • Problem

    Field-work data is often proprietary and variable, leaving VLM capabilities in unconstrained physical-work settings difficult to benchmark systematically.

    Source: §1 · p. 2, §3 · p. 4

  • Method

    RealityBench renders eleven physical-work environments in Blender and scores proof-of-work acceptability and task-relevant documentation against programmatic scene-based references.

    Source: §1 · p. 3, §4 · p. 5

  • Results

    Models largely solve documentation when answers are text in frame but remain weak at measurements and small-part or spatial judgments; rendered-frame rankings transfer on all three matched real-image documentation questions.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    The benchmark indicates that scene perception, especially measurements and small-part or spatial relations, remains a weakness, while acceptability judgments are difficult to assess given rater disagreement.

    Source: §8 · p. 10

    The evaluation uses single frames, covers eleven environments from two kinds of work, and tests general-purpose models zero-shot rather than specialist readers or detectors.

    Source: A LIMITATIONS · p. 15

Abstract

from arXiv · show

Augmenting worker capabilities in the physical world through guidance has been a decades-long goal for AI agents. One key supporting capability is to understand what the worker is doing, seeing through the human's camera or smart glasses, and assist them in documentation. Despite advances in vision language models, their capabilities in these settings remain uncertain and are challenging to benchmark because most physical work data is proprietary, and involves too much variability to study in a principled manner. Unlike structured settings such as kitchens and warehouses that have predictable lighting, geometry, and objects, field-facing tasks such as rooftop solar installation and vehicle repair are unconstrained and high variance. Here, we model real-world physical work in Blender across eleven environments, incorporating variants in both image acceptability and documentation. The acceptability task asks whether a particular view would be suitable proof-of-work and documentation asks whether an agent can understand the task-relevant state of the environment from that view. Across 26 vision-language models, we find that top models largely solve documentation when the answer is text in frame, but are weak at reading measurements and at judging small parts and spatial relations, such as whether a tape measure leans. Acceptability is harder to pin down because human raters disagree with each other. Measured against a scorer that agrees with the raters' majority about as often as a single rater does, the best models cannot be distinguished from a single non-author rater, at a cost within the range of a human pass. In the three environments where we can trace acceptability errors, most wrong rejections come from misreading the scene rather than the requirement. On the three documentation questions for which we have real images, model rankings on rendered frames transfer to real images.

1 INTRODUCTION

RealityBench evaluates whether vision-language models can judge proof-of-work views and extract task-relevant information, addressing the difficulty of benchmarking variable, proprietary field-work data. Across the benchmark, documentation is strongest for text in frame, while measurement and scene-reading remain difficult; rankings transfer to real images on three matched documentation questions.

1 INTRODUCTION

  • RealityBench renders eleven environments in Blender and evaluates 26 models on 1,378 frames, with public access and programmatic task scoring.The benchmark addresses restricted access to field data and uncertainty about what a model should infer from each image.Source: §1 · pp. 2–3
  • Across 22 models, rendered-frame rankings transfer to real photographs on all three matched documentation questions (Spearman ρ ≥0.63 per question, p ≤0.002).The real-image evaluation uses a public tape-measure set and two proprietary vehicle-photo sets.Source: §1 · p. 2
  • Documentation is strong when answers are text in frame, but weak for measurements and judgments about small parts or spatial relations.Examples include reading a tape at its crossing, deciding whether a tape leans, and detecting a lag bolt.Source: §1 · p. 2
  • On acceptability, the best models score about as well as a single non-author rater (F1 0.61 against 0.59).Human raters disagree with one another, complicating interpretation of acceptability scores.Source: §1 · p. 2
  • In three environments with traceable errors, most wrong rejections arise from misreading the scene rather than the requirement.Models judge 75% of upright tape measures as tilted.Source: §1 · p. 3

2 RELATED WORK

Prior work studies egocentric task labels, synthetic benchmarks, image quality, specification scoring, and fine-grained perception. The paper distinguishes its focus on whether worker-submitted views document completed work and whether rendered-task model rankings match those on real photographs.

2 RELATED WORK

  • Egocentric and procedural datasets label work steps, mistakes, or proficiency, but the authors found no public dataset judging worker-submitted proof-of-work photos for acceptability.Examples include Ego4D and Ego-Exo4D (Grauman et al., 2022; 2024), Assembly101, IndustReal, and CaptainCook4D (Sener et al., 2022; Schoonbeek et al., 2024; Peddi et al., 2024).Source: §2 · p. 3
  • Prior synthetic benchmarks and simulators provide rendered data or diagnostic test beds, but the authors know of no prior study comparing rendered and real-photo VLM rankings on the same task.The passage cites CLEVR (Johnson et al., 2017), Super-CLEVR (Li et al., 2023), and embodied simulators including ProcTHOR and Habitat (Deitke et al., 2022; Savva et al., 2019).Source: §2 · p. 3
  • Image-quality methods judge perceptual quality independently of content, while task-defined acceptability has appeared in document capture and insurance imaging.Prior VLM approaches include CLIP-IQA (Wang et al., 2023a), Q-Bench (Wu et al., 2024a), Q-Align, and DepictQA (Wu et al., 2024b; You et al., 2024).Source: §2 · p. 3
  • Specification-scoring approaches range from prompt decomposition and yes/no VQA to detector pipelines and generated verifiers; RealityBench uses hand-written programs over privileged scene state.The authors validate their scorers against human ratings and avoid a VLM reference scorer, citing inconsistent absolute scoring by multimodal LLM judges (Chen et al., 2024b).Source: §2 · pp. 3–4
  • Prior fine-grained perception studies report difficulty with orientation, small details, metric questions, and non-semantic strings such as serial numbers.The passage also notes that MeasureBench (Lin et al., 2026) finds frontier VLMs struggle to read instruments.Source: §2 · p. 4

3 ENVIRONMENTS

RealityBench models field-facing work in Blender with camera-rendered frames and paired acceptability and documentation tasks. Its eleven environments cover rooftop solar installation and vehicle repair, with tasks tied to the work’s concrete documentation needs.

3 ENVIRONMENTS

  • The benchmark contains eleven Blender environments across rooftop solar installation and vehicle repair, with a shared phone-camera model and varied focus, blur, and image-processing conditions.Frames include focused targets, autofocus locked elsewhere, or hand shake; rendering also applies sensor noise, white balance, tone mapping, and other camera effects.Source: §3 · p. 4

3.1 ROOFTOP SOLAR INSTALLATION

  • Six solar environments depict a house roof and array, while the seventh places the task at the home load center.The solar tasks follow installation documentation, from attachment hardware and wire adequacy to tape gap, slope, equipment labels, horizon, and breaker directory.Source: §3.1 · pp. 4–5

3.2 VEHICLE REPAIR

  • Vehicle-repair scenes cover whole-car views, close damage views, license plates, and repair-order boards photographed with the identified car.The documentation questions ask for vehicle face and body style, damage type and severity, plate characters and state, and the RO number and VIN.Source: §3.2 · p. 5

4 BUILDING THE BENCHMARK SET AND ACCEPTABILITY SCORER

The benchmark pairs environment-specific acceptability scorers with documentation answers checked against scene ground truth. Human ratings inform scorer tuning, while ambiguous requirement ranges are excluded from acceptability evaluation.

4 BUILDING THE BENCHMARK SET AND ACCEPTABILITY SCORER

  • The benchmark begins with 100 camera-sampled frames per scene plus positive sampling for seven environments, and reports API cost and answer-grammar match rates as additional metrics.Twenty-nine ro_board frames were withdrawn because of a board-texture defect; three self-hosted models are not priced.Source: §4 · pp. 5–6
  • Each environment’s acceptability scorer intersects terms derived from scene state and rendered-image properties, while documentation answers are checked against scene ground truth.Scene-derived terms include framing, occlusion, distance, and view angle; image-derived terms include target sharpness and text contrast.Source: §4 · p. 5
  • Six raters judged 30 random frames per environment, and the authors tuned the scorer on 60% of the ratings before evaluating agreement on held-out frames.The held-out majority reference included 95 frames after ties were excluded.Source: §4 · p. 5
  • The tuned scorer agrees with held-out human majority on 82% of frames (95% CI 75–90%), and the final benchmark retains 1,378 frames.It accepts 4 of 62 frames the majority rejects and rejects 13 of 33 majority-accepted frames; individual raters agree with the other raters’ majority on 76–92%.Source: §4 · pp. 5–6
  • Ambiguous requirement ranges are marked gray and excluded from scoring; this removes 43 frames from the acceptability set.For damage_view, distances from 1.5 to 3 m are gray for the “about arm’s length” requirement.Source: §4 · p. 6

5 REAL-WORLD DATA

The paper evaluates three documentation tasks on real-world images, using task-specific ground truth to compare model answers with the scene. The tasks cover tape-measure reading, visible car faces, and damage severity.

5 REAL-WORLD DATA

  • 133 tape readings across 84 photos, 35 items, and 12 scenes were checked against caliper or careful human measurements, with answers scored within 1/8 inch.The photos show a tape measure held against everyday objects.Source: §5 · p. 6
  • 117 real car frames were labeled for the visible sides and thirds, with an answer correct if it named any labeled part of the car.Source: §5 · p. 6
  • For damage severity, one author rated 63 photographs of damaged panels using the documentation task’s severity scale.Source: §5 · p. 6

6 EXPERIMENTS

The experiments evaluate 26 vision-language models on acceptability and documentation, comparing performance and cost and testing real-image transfer. Documentation leaders outperform individual models through routing, while acceptability remains comparable to a single human rater.

6 EXPERIMENTS

  • The evaluation covered 26 models: 23 via provider APIs and three small open-weight models served by the authors; Figure 2 compares acceptability and documentation against set cost.Documentation is scored on scorer-acceptable frames, and the figure also compares synthetic and real photographs on three matched questions.Source: §6 · pp. 6–7, Figure 2 · p. 7

6.1 RESULTS

  • Gemini 3.8 Flash led documentation at 0.84 ($7.16), while GPT-6 Astra led acceptability at 0.61 ($30.05); only GPT-6 Astra and Claude Opus 5.5 were in both top groups.The synthetic-data rankings for the two tracks were in approximate agreement (Spearman ρ = 0.69, p < 0.001).Source: §6.1 · p. 7
  • Routing each environment to its best model reached documentation score 0.86 ($17.90), above every single model, but acceptability F1 was 0.60 ($20.60).The majority of all models reached F1 0.60 and documentation 0.83 at $322.Source: §6.1 · p. 8
  • Among moderately priced models, GLM-5.3 Flash led acceptability at 0.57 ($1.30), while Gemma 4 31B led documentation at 0.79 ($0.70); the top unpriced models scored F1 0.43 and 0.42.LFM2.5-VL-3B and Gemma 4 E4B scored documentation 0.68 and 0.69, respectively.Source: §6.1 · p. 8
  • On three matched questions, synthetic and real-image task difficulties were broadly consistent, and model rankings transferred with Spearman ρ of 0.63–0.79 (p ≤0.002).Tape reading was hardest on both, followed by damage severity and then naming the visible face of a car.Source: §6.1 · p. 8
  • The strongest models could not be distinguished from a single non-author rater, whose F1 against the scorer was 0.59 (95% CI 0.50–0.67).A human pass over all 1,378 items was estimated to cost $11–165, compared with $16.36 for Claude Opus 5.5 and $30.05 for GPT-6 Astra.Source: §6.1 · p. 8

7 ANALYSIS

Documentation performance varies by field: text in view is often easy, while measurements and judgments about small parts or spatial relations are harder. Analysis links errors to scene interpretation, viewing conditions, and task-specific visibility.

7 ANALYSIS

  • Eight of ten reading fields scored 0.90–1.00 on average, but tape reading was hardest; small-part and relation judgments were also difficult.The number at a tape crossing was within 1/8 inch 11% of the time on average, while whether a tape leaned scored 0.55 on average.Source: §7 · p. 8
  • In tape_gap, models judged tape angle relative to the image rather than the module; camera roll and limited roof reference were associated with more tilt errors.The tape was within 5° of upright on every scorer-acceptable frame, yet models called it tilted 75% of the time; the rate rose with camera roll.Source: §7 · pp. 8–9
  • For damage severity, real frames were not restricted to scorer-acceptable images, so the synthetic comparison used only a sharpness-and-visibility condition.Source: §7 · p. 8
  • Across tape_gap, slope_gauge, and rail_attachment, reported defects triggered rejection on 89–99.6% of frames, while scene misreadings accounted for most wrong rejections.Misreadings accounted for 97% of wrong rejections in tape_gap, 74% in slope_gauge, and 63% in rail_attachment.Source: §7 · p. 9
  • For seven of nine models, defect reports did not change beyond run-to-run noise when the verdict was asked last or omitted.The two cheapest models reported fewer defects without a verdict prompt; for them, asking for the verdict first worsened documentation answers.Source: §7 · p. 9
  • Documentation averaged 0.71 on scorer-acceptable frames and 0.50 on the rest, with every model scoring higher on acceptable frames by 0.14–0.26.Tape_gap was an exception: its score was −0.17 lower on acceptable frames, where rejected frames often showed a leaning tape.Source: §7 · p. 9

8 CONCLUSION

RealityBench pairs proof-of-work acceptability judgments with documentation of task-relevant state across eleven simulated environments. The conclusion reports that text reading is largely solved, while measurement and small-part or spatial-relation judgments remain weak; real-image rankings transfer on the three matched documentation questions.

8 CONCLUSION

  • RealityBench comprises eleven rendered rooftop-solar and vehicle-repair environments with programmatic scoring for proof-of-work acceptability and task-relevant state documentation.The environments are simulators rather than image sets and can be extended to new tasks and closed-loop guidance.Source: §8 · p. 9
  • The conclusion reports strong text-in-frame reading but persistent weaknesses in measurements, small parts, and spatial relations; acceptability is harder to assess because raters disagree.In three environments with traced errors, most wrong rejections come from scene misreading; rankings transfer to real photographs on all three matched documentation questions.Source: §8 · p. 10

AI USE STATEMENT

  • The authors report using generative AI across research design, implementation, data handling, interpretation, coding, figures, drafting, literature review, and editing, with required and recommended disclosures distinguished.They state that authors reviewed AI-assisted work and verified retained interpretations and checked model runs and analysis scripts.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • Acceptability ratings came from four authors and two colleagues; one author also labeled vehicle photographs for face visibility and damage severity.No personal data was collected from raters, so no ethics review was sought.Source: ETHICS STATEMENT · p. 10
  • Vehicle-damage photographs were used with permission, only to score model answers, and were not released or reproduced; tape-measure photographs were taken by an author and released.Vehicle photographs were sent only to providers whose data-retention terms met the authors’ requirements.Source: ETHICS STATEMENT · p. 10
  • The benchmark evaluates third-party model capabilities, running API models at public list prices and three open-weight models on rented GPUs.Third-party 3D assets are CC0 or CC-BY and their licenses and attributions are listed in the released dataset’s attribution file.Source: ETHICS STATEMENT · p. 10
  • The paper identifies a potential harm: photograph-based work judgments could pressure workers or automate rejection of legitimate work.The authors argue that better capability evaluation matters for societal decisions about AI use in human work.Source: ETHICS STATEMENT · p. 10

A LIMITATIONS

The evaluation is bounded by single-frame testing, a scorer and human-comparison setup with limited validation, and narrow real-world and error-tracing coverage.

A LIMITATIONS

  • The benchmark evaluates single frames, although workers would naturally produce video.Its eleven environments cover only rooftop solar installation and vehicle repair, and it tests general-purpose models zero-shot at provider defaults rather than specialist readers or detectors.Source: A LIMITATIONS · p. 15
  • The scorer is one reading of each requirement, validated against six raters on 30 frames per environment, and it errs on the strict side.The human comparison uses non-expert raters, and its cost is only a rough range.Source: A LIMITATIONS · p. 15
  • Error tracing covers wrong rejections only in the three environments where acceptability and documentation share a field.The real-world comparison covers three documentation questions and no acceptability task.Source: A LIMITATIONS · p. 15
  • Model versions and prices change quickly, limiting the durability of the reported model and cost comparisons.Source: A LIMITATIONS · p. 15

B GLOSSARY

The glossary describes solar-installation parts and benchmark tables, while the prompt examples show how common instructions and environment-specific task fields are presented to models.

B GLOSSARY

  • A solar module is clamped to rails, each attached to the roof by L-feet lag-bolted through flashing into rafters.Flashing slips under shingles at each attachment to prevent leakage at the lag-bolt hole; only its lower part is exposed.Source: B GLOSSARY · p. 15
  • Table 1 summarizes environments, tasks, and scorers; capture tasks also require sharpness at the target, and its acceptability fraction uses random field-pose frames.Table 2 separates scoring primitives computed from the placed scene from those read from the rendered frame.Source: Table 1 · p. 16, Table 2 · p. 17

C PROMPT EXAMPLES

  • Prompts share a frame, technician, requirement, question, and answer keys; the example asks models to judge adequacy and answer documentation fields even when the photo is inadequate.The horizon-peak requirement is set-level, so its frame is judged as belonging to such a set; the tape-gap example asks for a reading, estimated gap, and tilt judgment.Source: C PROMPT EXAMPLES · pp. 17–18

D EVALUATION DETAILS

The evaluation details specify frame sampling, scoring rules, prompt-order checks, and the reporting conventions for model performance and cost.

D EVALUATION DETAILS

  • Frames failing render sanity checks are excluded; seven environments with rare acceptable random poses receive 50 acceptable positive-stratum frames to estimate misses.False-accept rates use random frames only; the sampling passage also reports withdrawn ro_board frames after a board-texture defect.Source: D EVALUATION DETAILS · pp. 19–20
  • Each frame is one query; acceptability is scored against the scorer’s verdict and documentation against field ground truth, using exact-match normalization or task-specific scoring and tolerances.The breaker-directory score matches rows by name and amperage, in any order, with extra rows unpenalized; tape reading, gap, and gauge tolerances are 1/8 inch, 1/4 inch, and 0.2 degrees.Source: D EVALUATION DETAILS · p. 20
  • Gemini 3.1 Flash Lite reported fewer defects with the verdict last and about half as many without a verdict across three environments; Gemma 4 31B reported fewer missing parts without a verdict in rail_attachment.The ablation re-asked 150 items under four prompt variants and compared paired defect claims and rerun noise.Source: D EVALUATION DETAILS · p. 20, Table 4 · p. 20
  • Table 5 reports false-accept and miss rates, acceptability precision, recall and F1, documentation score, answer format, set cost, and mean thinking tokens.F1 and documentation intervals use stratified bootstrap intervals over items; replies cut by output caps are also listed.Source: D EVALUATION DETAILS · p. 20, Table 5 · p. 21
  • Set cost is calculated using each model developer’s list API price, read from its pricing page on the shown date, multiplied by API-reported tokens; prompt tokens use input rate and thinking tokens output rate.Table 7 lists prices and source dates; Table 11 compares token prices with set cost.Source: D EVALUATION DETAILS · p. 21, Table 7 · p. 23, Table 11 · p. 26
  • Table 6 reports paired model differences in F1 and documentation score; only pairs whose paired 95% interval excludes zero are marked as distinguishable.Per-environment acceptability and false-accept results, per-field documentation scores, and pooled misreading/rejection statistics are reported in Tables 8–10 and 12.Source: Table 6 · p. 22, Table 8 · p. 24, Table 9 · p. 24

E HUMAN RATINGS

Human acceptability ratings and scorer agreement are reported by environment; on three matched documentation questions, model rankings on synthetic frames correlate with rankings on real photos.

E HUMAN RATINGS

  • The frozen scorer agreed with raters’ majority on 82% of 95 held-out frames with at least three ratings, excluding ties and unscored frames.Table 13 also reports raw agreement across rater pairs and Krippendorff’s alpha, which corrects for chance and allows skipped frames.Source: Table 13 · p. 27, E HUMAN RATINGS · p. 27

F REAL-PHOTO DETAILS

  • Across 22 models, synthetic-to-real correlations were tape ρ = 0.68 (p = 0.001), r = 0.82 (p < 0.001); face ρ = 0.79 (p < 0.001), r = 0.85 (p < 0.001); severity ρ = 0.63 (p = 0.002), r = 0.58 (p = 0.005).Table 14 compares matched questions; real-photo measures use different criteria and sample counts for tape readings, visible faces, and damage severity.Source: F REAL-PHOTO DETAILS · p. 27, Table 14 · p. 28
Loading 2610.14397v2…