PaperlayerView exact revision on arXiv

Source-linked AI summary

Embodied-BenchForge: A Closed-Loop Agentic Workflow for Embodied Benchmark Construction

Baoyang Jiang, Fengchun Zhang, Leyuan Wang, Haotian Li, Yida Wang, Zhe Ji, Jinshan Lai, Xi Ren, Danyang Li, Zheng Yang, Jianwei Hu, Qiang Ma

arXiv:2609.13082v1cs.AI

TL;DR

Embodied benchmark construction remains labor-intensive, and existing automated approaches often cover isolated stages or predefined settings while dependent artifacts may pass defects downstream unchecked. Embodied-BenchForge combines typed, reusable skill workflows with artifact-specific verification and provenance-guided repair. Its benchmarks distinguish capabilities across observation-based understanding and closed-loop execution, including a substantial gap between them.

  • Problem

    Embodied benchmark construction remains labor-intensive, and automated methods often cover selected operations or predefined settings while dependent artifacts can propagate defects without artifact-specific checks.

    Source: §1 · pp. 1–2

  • Method

    Embodied-BenchForge composes typed, reusable skills into workflows and applies artifact-specific contracts with provenance-guided local re-execution or rollback.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    The benchmarks distinguish model capabilities in observation-based understanding and closed-loop execution, revealing a substantial gap between them.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Embodied-BenchForge provides a scalable and continually updatable framework for grounded, executable, and traceable embodied benchmark construction.

    Source: §6 · p. 10

Abstract

from arXiv · show

Agentic systems offer a promising way to automate embodied benchmark construction, but existing approaches typically cover isolated stages or remain specialized to predefined environments and task families. More importantly, multi-step construction produces dependent intermediate artifacts that are often passed downstream without artifact-specific verification, allowing local defects to propagate into the final benchmark. We present Embodied-BenchForge, an agentic framework that transforms user-specified evaluation intents into complete embodied benchmark artifacts. It formulates construction as Closed-Loop Benchmark Synthesis, integrating forward artifact synthesis with backward verification and repair. Skill-Orchestrated Artifact Synthesis composes typed and reusable skills into executable workflows, while an artifact dependency graph records intermediate outputs and their dependencies. Requirement-Guided Verification and Repair applies artifact-specific contracts throughout construction and uses provenance to trigger local re-execution or upstream rollback when verification fails. Embodied-BenchForge constructs six benchmarks covering diverse embodied scenarios in the Offline EQA Track, together with one interactive benchmark containing 220 executable tasks in the Interactive Embodied Track. Evaluations of representative MLLMs and embodied agents show that the benchmarks distinguish model capabilities in both observation-based understanding and closed-loop execution. Quality assessment and ablations validate benchmark quality and the effectiveness of verification and repair, while repair and skill-reuse analyses demonstrate efficient localized recovery and cross-benchmark reusability.

1 INTRODUCTION

Embodied benchmark construction remains labor-intensive, with automated approaches often limited to selected operations or predefined settings, and dependent artifacts can pass defects downstream without specific checks. Embodied-BenchForge addresses these gaps with skill-based synthesis and provenance-guided verification and repair, then evaluates the resulting benchmarks across observation-based and interactive tasks.

1 INTRODUCTION

  • Existing construction is labor-intensive, and automated methods often cover selected operations or remain specialized to predefined settings and task families.Heterogeneous resources and outputs still require substantial manual configuration; unchecked defects in dependent artifacts can propagate downstream.Source: §1 · pp. 1–2
  • Embodied-BenchForge composes typed, reusable skills into workflows and uses artifact-specific contracts and provenance to trigger local re-execution or upstream rollback.The framework represents construction as Closed-Loop Benchmark Synthesis, combining forward artifact synthesis with backward verification and repair.Source: §1 · p. 2
  • The benchmarks distinguish model capabilities in observation-based understanding and closed-loop execution, revealing a substantial gap between them.The suite includes six Offline EQA benchmarks and one Interactive Embodied benchmark with 220 executable tasks.Source: §1 · p. 2
  • The paper evaluates artifact quality, construction efficiency, repair behavior, skill reuse, and model performance to assess benchmark reliability and framework reusability.Its stated contributions include construction across heterogeneous resources and both observation-based and interactive task settings.Source: §1 · p. 2

2 RELATED WORK

Prior work spans embodied task benchmarks and automated construction methods, but often remains tied to specific resources, environments, or task families. Embodied-BenchForge is positioned as constructing complete benchmarks from evaluation intents while combining reusable skills with verification and repair.

2 RELATED WORK

  • Embodied benchmarks cover scene understanding, navigation, manipulation, planning, and interaction, while OpenEQA evaluates question answering over recorded experience and active exploration.Examples named include ALFRED, LIBERO, EmbodiedScan, and EmbodiedBench.Source: §2 · p. 2
  • Table 1 compares construction mechanisms across end-to-end synthesis, reusable modules, heterogeneous resources, executable artifacts, and process-level verification and repair.Its comparison focuses on mechanisms rather than performance; partial support in the final column means verification without selective artifact reconstruction.Source: Table 1 · p. 3
  • Scene-generation, demonstration-expansion, and language-model task-design methods reduce generation costs but generally operate within predefined simulators or task families.The passage names ProcTHOR, RoboCasa, MimicGen, Eureka, and RoboGen.Source: §2 · p. 3
  • Automated benchmark systems use staged workflows, requirements, search, filtering, psychometric analysis, or dependency checks, while A2Eval selects subsets of existing embodied benchmarks.Embodied benchmark construction additionally requires consistency among observations, privileged states, grounded evidence, feasible actions, goals, and executable verifiers.Source: §2 · p. 3

3 EMBODIED-BENCHFORGE

Embodied-BenchForge builds benchmarks through typed, reusable skills and a forward construction path coupled to requirement-guided verification. Failed artifacts are diagnosed and locally repaired or rolled back, while track-specific checks govern which artifacts proceed to packaging.

3 EMBODIED-BENCHFORGE

  • The workflow transforms evaluation intents and embodied resources into benchmark artifacts, then sends verified outputs to reporting and failed outputs back for repair or rollback.Its forward path includes intent blueprinting, data collection, evidence and state structuring, benchmark synthesis, and evaluation reporting.Source: §3 · p. 3, Figure 1 · p. 4
  • Artifacts carry a type, content, metadata and verification state, and construction provenance; typed skills encode inputs, outputs, preconditions, and executable backends.Hierarchical skill decomposition and compatible skill composition form workflows bound to available resources and target artifact types.Source: §3 · p. 4
  • The forward path structures resources into evidence or environment states, then synthesizes questions, answers, instructions, interfaces, goals, verifiers, and scoring protocols.Intent blueprinting establishes capability targets, resource requirements, task specifications, and output schemas.Source: §3 · pp. 4–5
  • After grey-batch synthesis, OE-Track checks invalid items and image dependence, whereas IE-Track checks task validity, completion, and initial-state restoration before model evaluation and IRT diagnosis.The workflow begins with a verified evidence pool, template–evidence binding, and candidate-item synthesis before these track-specific gates.Source: Figure 2 · p. 5, §3 · pp. 5–6
  • Artifact-specific contracts check structural, grounding, semantic, and execution requirements before release, with criteria specialized to blueprints, evidence records, Offline EQA items, and interactive tasks.Interactive items must include restorable initial states, feasible instructions, valid action spaces, consistent goals, and executable terminal verifiers.Source: §3 · p. 5
  • Failure diagnosis uses recorded inputs, outputs, resources, templates, parameters, and verification results to repair an artifact, re-execute a step, or roll back upstream.Only affected downstream artifacts are reconstructed; unrelated verified artifacts are retained, and repaired artifacts return to the contracts and quality gates.Source: §3 · p. 6
  • Table 2 reports Offline EQA capability dimensions, question types, and expert templates, and IE-Track tasks, scenes, and difficulty levels across four interactive room types.The listed scene types are kitchen, living room, bedroom, and bathroom.Source: Table 2 · p. 6

4 CONSTRUCTED BENCHMARK SUITE

The constructed suite pairs six Offline EQA benchmarks spanning embodied settings with an Interactive Embodied benchmark of 220 executable tasks. Together, the tracks cover grounded question answering and closed-loop task completion.

4 CONSTRUCTED BENCHMARK SUITE

  • The OE-Track contains six benchmarks spanning household robots, mobile robots, autonomous vehicles, robotic arms, UAVs, and quadruped robots.Instances combine embodied observations with grounded questions, reference answers, and executable scoring protocols.Source: §4 · p. 6
  • The IE-Track contains 220 instruction-conditioned tasks with restorable initial states, action interfaces, goal conditions, and executable terminal verifiers.The tasks use shared AI2-THOR scene pools and retain interaction trajectories for subsequent EQA.Source: §4 · p. 6
  • Together, the tracks extend evaluation from observation-based reasoning to executable interaction and closed-loop task completion.Source: §4 · p. 6

5 EXPERIMENTS

The experiments assess benchmark quality, construction efficiency, evaluation utility, and construction mechanisms. Results show discriminative performance across both tracks, high module reuse, and gains from verification and repair, alongside distinct error patterns across tracks.

5 EXPERIMENTS

  • The experiments assess quality, repair and system components, benchmark utility, and scalability; they evaluate 11 API and local models and measure quality with judge/human ratings and efficiency with cost, time, and throughput.Quality assessment uses four LLM/VLM judges and ten human annotators.Source: §5 · pp. 6–7

5.1 CONSTRUCTION QUALITY AND EFFICIENCY

  • Across six OE-Track benchmarks, Overall quality scores range from 88.34 to 90.75; target-signal dependency and scoring consistency remain harder to control automatically.Scores are averaged over four judge models.Source: §5.1 · p. 7, Table 3 · p. 7
  • A 10,000-item OE-Track benchmark takes 38–160 minutes to construct, averaging 86.0 minutes, while 20 executable IE-Track tasks take 1.5 hours to construct and validate.OE-Track orchestration averages 11.45M tokens, or 1,145 tokens per accepted item, at 116.3 items per minute.Source: §5.1 · p. 7

5.2 BENCHMARK EVALUATION UTILITY

  • Humans average 85.89 on OE-Track, above GPT-5.5’s 57.67; on IE-Track, GPT-5.5 leads evaluated models with 83.18% SR, 73.64% EQA accuracy, and 61.36% SRL.Smaller models can retain moderate EQA accuracy despite near-zero task success, separating trajectory understanding from executable competence.Source: §5.2 · pp. 7–8, Table 4 · p. 8, Table 5 · p. 8

5.3 CONSTRUCTION MECHANISMS AND ABLATION

  • Across construction settings, 91.6% of invoked modules are reused; four simulator-based OE benchmarks reuse 97.5%, while SpatialUAV and IE-Track require additional specialized skills.SpatialUAV and SpatialQuadruped add real-image or dataset-specific processing; IE-Track adds interaction-specific skills.Source: §5.3 · p. 8
  • Removing verification and repair reduces human quality by 23.25 points and valid rate from 93.7% to 62.4%, the largest degradation among the tested system components.Removing source grounding and evidence construction drops human scores by 19.16 and 17.20 points, respectively.Source: §5.3 · pp. 8–9
  • On SpatialUAV, final-stage inspection yields 10.4% valid items and 14.6% high-discrimination items; without provenance-guided repair, token cost rises from 2.62M to 3.93M.Removing requirement contracts decreases quality by 10.47 points and valid rate by 31.5 percentage points.Source: §5.3 · p. 9
  • Offline failures center on answer binding, multi-view consistency, and temporal reasoning; interactive errors center on navigation, repeated execution, recovery, and action ordering.Trajectory-EQA errors primarily involve temporal memory and spatial relations.Source: §5.3 · p. 9, Figure 3 · p. 9

6 CONCLUSION

This section summarizes Embodied-BenchForge’s closed-loop construction and provenance-guided quality control across its offline and interactive tracks. It documents offline repair and rejection cases alongside the interactive track’s execution protocol and verification scope.

6 CONCLUSION

  • Embodied-BenchForge combines forward synthesis with backward verification and repair, producing six Offline EQA settings and an Interactive Embodied benchmark with 220 executable tasks.The conclusion reports strong artifact quality, model discrimination, measurement reliability, and construction efficiency, and a gap between embodied understanding and closed-loop interaction.Source: §6 · pp. 9–10

A.1 DEFINITIONS AND SHARED CLOSED-LOOP PROTOCOL

  • Artifact-specific contracts and track-specific quality gates check requirements throughout construction; deterministic or execution-based failures cannot be overridden by model-assisted checks.Artifacts proceed downstream only after mandatory predicates pass; model-assisted checks address semantic relations that structure or execution alone cannot reliably establish.Source: §A.1 · p. 13, §A.1.2 · p. 13

A.1.3 PROVENANCE-GUIDED REPAIR PROTOCOL

  • Provenance traces identify the earliest causal defect and limit reconstruction to its downstream dependents, retaining verified artifacts and applying the same acceptance criteria after repair.The full process uses 2.62M tokens, compared with 3.93M when provenance-guided repair is removed.Source: §A.1.3 · pp. 14–15

A.2.1 REPRESENTATIVE BEFORE-AND-AFTER REPAIRS

  • The offline case set reports five released items after re-verification and one rejected candidate whose visual evidence remained insufficient.Cases demonstrate local field repair, template re-binding, candidate re-synthesis, skill re-execution, and upstream rollback.Source: §A.2.1 · p. 17, §A.2.2 · p. 18, §A.2.3 · p. 19

A.2 OFFLINE EQA TRACK: VERIFICATION AND REPAIR CASES

  • In SHT, local field repair corrects the answer while retaining evidence and binding; in SHN, template re-binding removes an answer-revealing option cue.In SA, candidate re-synthesis replaces a question that depended on hidden task metadata with one answerable from the registered frame sequence.Source: §A.2.2 · p. 18, §A.2.3 · p. 19, §A.2.4 · p. 20

A.2.5 SD (CARLA): SKILL RE-EXECUTION

  • In the CARLA case, skill re-execution creates model-visible RGB anchors so the route can be determined relative to the registered vehicle heading.The repaired item passes verification while other CARLA records are retained.Source: §A.2.5 · p. 21

A.2.6 SQ (TARTANGROUND): UPSTREAM ROLLBACK

  • In the TartanGround case, upstream rollback aligns the terminal image, pose, and rendered arrow to restore traceability and answer derivability.The repair passes verification; the registered displacement is southwest and the terminal heading is closest to north.Source: §A.2.6 · p. 22

A.2.7 SU (UAV): FINAL REJECTION

  • The UAV candidate was rejected because neither target met the 0.25 evidence-admission threshold, and hidden depth could not replace reliable model-visible evidence; only final rejection was re-executed.The source image and unrelated verified UAV evidence were retained.Source: §A.2.7 · p. 23

A.3 INTERACTIVE EMBODIED TRACK: EXECUTION AND VERIFICATION

  • The interactive track specifies benchmark composition, execution protocol, metrics, task packages, GPT-5.5 records, and verification cases; its protocol applies uniformly, while examples separate task artifacts from execution trajectories.Source: §A.3 · p. 24

A.3.1 TRACK COMPOSITION AND EVALUATION PROTOCOL

IE-Track evaluates 220 independent task episodes under shared conditions, combining executable task success with trajectory-grounded EQA and episode-level efficiency.

A.3.1 TRACK COMPOSITION AND EVALUATION PROTOCOL

  • 220 independent task instances span 28 AI2-THOR scenes across four household scene types, and every evaluated model receives the same tasks and protocol.The protocol also holds initial states, action interface, and stopping rules constant.Source: §A.3.1 · p. 24, Table 11 · p. 24
  • An episode succeeds only when all required goal conditions are met; partial progress does not count, and reaching 150 steps without completion is a timeout failure.Invalid actions are recorded but do not immediately stop execution.Source: §A.3.1 · p. 24
  • Trajectory-grounded EQA uses the full interaction history for questions about initial, earlier, or final states, with accuracy computed as a question-level micro average.The action interface supports object-directed navigation and manipulation actions, among other operations.Source: §A.3.1 · p. 24
  • SRL measures joint task success and trajectory-EQA correctness, while Avg. Steps averages executed steps across successful, failed, and timeout episodes.Metrics are computed from saved episode-level execution records without manual correction or score imputation.Source: §A.3.1 · p. 25

A.3.2 REPRESENTATIVE TASK PACKAGES AND GPT-5.5 EXECUTIONS

The section presents representative task packages and GPT-5.5 executions, then shows how verification detects construction defects and routes repairs through revalidation.

A.3.2 REPRESENTATIVE TASK PACKAGES AND GPT-5.5 EXECUTIONS

  • Task packages define initial state, available actions, goals, and an executable verifier, whereas execution records capture GPT-5.5 actions, outcomes, and trajectory-grounded EQA responses.The examples show that successful closed-loop execution and trajectory understanding provide complementary evaluation signals.Source: §A.3.2 · p. 25
  • 7 actions complete the slice-and-place task and 60 actions complete all four wash-and-stow goals; both episodes pass the terminal verifier.The retained trajectories yield 10/15 and 9/15 correct EQA answers, respectively.Source: Table 12 · p. 25, §A.3.2 · p. 25, Figure 5 · p. 27
  • The single-subtask package requires slicing a tomato and placing a slice on the countertop; the multi-subtask package sets four object-state and placement goals.The complete packages also store simulator state and compiled verification information.Source: §A.3.2 · p. 25, Figure 4 · p. 26, Figure 6 · p. 28

A.3.3 REPRESENTATIVE VERIFICATION AND REPAIR CASES

  • Two recurring construction failures appear among five audited candidates: instruction–goal inconsistency and underspecified placeholder instructions.The examples show a mismatched PDDL goal and a placeholder instruction whose operation target appears only in the hidden goal.Source: §A.3.3 · pp. 25–26, Table 13 · p. 27, Table 14 · p. 30
  • When provenance indicates a shared origin, repeated defects are handled at the common template or synthesis operation rather than only in isolated task files.Source: §A.3.3 · p. 26
  • Repaired candidates re-enter validity, completion, and initial-state checks before affected verifiers, trajectories, model records, EQA results, and aggregate metrics are regenerated.Admission follows successful revalidation.Source: §A.3.3 · p. 26

B RESOURCE COVERAGE AND CROSS-RESOURCE ADAPTATION

Embodied-BenchForge applies a shared construction workflow across heterogeneous embodied resources, using localized adapters and specialized skills where resource interfaces differ. The appendix describes this reuse across resource coverage, two adaptation cases, and the skill taxonomy.

B RESOURCE COVERAGE AND CROSS-RESOURCE ADAPTATION

  • The first four simulator- or dataset-based runs reuse more than 97% of invoked skill definitions, while SpatialUAV, SpatialQuadruped, and IE-Track reuse 82.9%, 87.8%, and 81.8%, respectively.The evaluated settings span simulator-derived and real-world resources, static and sequential observations, and offline question answering and online execution; the authors characterize adaptation as localized rather than requiring a redesigned workflow.Source: B RESOURCE COVERAGE AND CROSS-RESOURCE… · pp. 31–32
  • SpatialUAV uses unlabeled real-world aerial images to construct 10,000 grounded questions across recognition, counting, localization, scale, depth, geometry, and matching.The case uses visual and geometric evidence without privileged simulator states or pre-existing benchmark annotations.Source: B RESOURCE COVERAGE AND CROSS-RESOURCE… · p. 32
  • SpatialQuadruped repurposes TartanGround trajectories and synchronized multimodal observations for language-grounded spatial questions, retaining 87.8% skill reuse.Its questions cover semantic visibility, localization, depth ordering, cross-view consistency, trajectory structure, and motion-orientation change.Source: B RESOURCE COVERAGE AND CROSS-RESOURCE… · p. 32

C.1 REPRESENTATIVE SKILL TAXONOMY

  • The skill-library appendix groups 64 registered skill/card definitions into representative functional families covering construction, verification, repair, and track-specific diagnosis.The table summarizes functions rather than individual card names; the complete inventory is in the accompanying code snapshot.Source: §C.1 · p. 33, C SKILL TAXONOMY AND REUSE ACCOUNTING · p. 33

C.2 SKILL-REUSE COUNTING PROTOCOL

The appendix specifies how skill reuse is counted and how construction cost and throughput are accounted for. Its protocol distinguishes per-run skill usage from pooled reuse, and excludes target-model execution from reported construction time.

C.2 SKILL-REUSE COUNTING PROTOCOL

  • Each construction run counts distinct invoked skill definitions, with retries, re-executions, and repair-time invocations counted once; a definition is reused if shared with another setting.Specific skills or adapters are used only in the current setting.Source: §C.2 · p. 33
  • 91.6% of invoked skills are reused across all seven construction runs, based on 263 reused incidences among 287 benchmark–skill incidences.The 287 total counts incidences across runs, not globally unique definitions; the four simulator-based OE benchmarks have a pooled reuse rate of 97.5%.Source: §C.2 · p. 33

D.1 TOKEN AND TIME ACCOUNTING

  • Construction cost is reported by five stages, from Intent Blueprinting through Verification & Evaluation Reporting, with verification during S1–S4 charged to the artifact-producing stage.Token accounting includes construction-side calls and IRT processing but excludes target-model inference and four-judge quality assessment; wall-clock time excludes target-model execution but includes result aggregation.Source: §D.1 · p. 35, D CONSTRUCTION COST AND EFFICIENCY… · p. 35
  • Approximately 1.5 hours per 20-task IE-Track batch includes task synthesis, state initialization, verifier construction, and script-based feasibility checks, but excludes VLM-agent execution.The throughput protocol uses accepted items divided by construction time, pooling totals across benchmarks when applicable.Source: §D.1 · p. 35

D.2 CONSTRUCTION COST AND STAGE-WISE BREAKDOWN

The appendix reports normalized construction costs for the six Offline EQA benchmarks and details experimental models, evaluation protocols, and benchmark composition. Costs are compared at a common output size of 10,000 items per benchmark.

D.2 CONSTRUCTION COST AND STAGE-WISE BREAKDOWN

  • Across six construction runs normalized to 10,000 final items per benchmark, Embodied-BenchForge averaged 11.45M tokens and 86.0 minutes; pooled throughput was 116.3 items per minute across 60,000 items and 516 minutes.Source: §D.2 · p. 35, Table 18 · p. 36
  • Benchmark Synthesis is the largest cost stage at 52.8%–62.3% of total cost, while Verification & Evaluation Reporting contributes 4.2%–5.6%.SpatialUAV’s evidence-structuring stage takes 65 minutes; its unannotated aerial images require additional cleaning, annotation, and evidence organization.Source: Table 19 · p. 36, §D.2 · p. 37

E.1.1 CONSTRUCTION MODELS AND RUNTIME SETTINGS

  • Workflow coordination, artifact synthesis, process verification, and quality assessment are assigned distinct model roles, and evaluated models receive the same prompts and interfaces within each track.Experiments used a server with four NVIDIA H100 80 GB GPUs and 512 GB of host memory.Source: §E.1.1 · p. 38

E.2 HUMAN AND JUDGE EVALUATION DETAILS

  • Ten embodied-intelligence or benchmark-development researchers independently assessed sampled items, while four model judges used shared scoring inputs and schemas.Judge–Human agreement is computed as the sample-level Spearman correlation between aggregated judge and human scores.Source: §E.2 · p. 39

E.3 OFFLINE EQA BENCHMARK COMPOSITION

  • Each of the six Offline EQA benchmarks contains 10,000 items, and its capability dimensions and question types are independent distributions rather than rowwise mappings.Table 22 orders capabilities by local identifier and question types by decreasing frequency.Source: Table 22 · p. 40, §E.3 · p. 40

E.4 LLM-AS-JUDGE BENCHMARK QUALITY ASSESSMENT

LLM-as-Judge assessment finds consistently high quality scores across the six Offline EQA benchmarks, with lower scores on skill challenge and target-signal dependency. The appendix also supplies capability-level model results to complement the main aggregate results.

E.4 LLM-AS-JUDGE BENCHMARK QUALITY ASSESSMENT

  • The quality assessment weights evidence grounding, spatial consistency, and answer consistency more heavily because they directly affect answerability and scoring reliability.The eight assessment dimensions are intent alignment, usability, answer consistency, evidence grounding, spatial consistency, affordance validity, target-signal dependency, and skill challenge.Source: §E.4 · p. 41, Table 23 · p. 42
  • Overall quality scores range from 88.34 to 90.75 across the six benchmarks, with SpatialArm highest; scores are consistently strong in intent alignment, usability, evidence grounding, and spatial consistency.The authors identify skill challenge and target-signal dependency as areas with room for more discriminative and evidence-dependent items.Source: §E.4 · pp. 41–42, Table 24 · p. 43

E.5 PER-CAPABILITY MODEL PERFORMANCE

  • Tables 25–30 report capability-level results for six Offline EQA benchmarks, showing which spatial, temporal, grounding, and embodied reasoning skills remain difficult for different model families.The AVG column summarizes performance across capability dimensions within each benchmark.Source: §E.5 · p. 44

E.6 IRT DIAGNOSTIC PROTOCOL AND STABILITY ANALYSIS

The IRT diagnostic supplements artifact-validity checks by measuring item discrimination across models and evaluating estimate stability. It also characterizes where the six benchmarks provide measurement information across model ability levels.

Analytical objective and protocol

  • The four-stage diagnostic builds model–item response matrices, fits a regularized 2PL model, flags highly discriminative or potentially reversed items, and assesses estimate stability.IRT is supplementary measurement evidence, not a replacement for benchmark-validity checks.Source: §E.6 · p. 44

Discrimination criterion and audit rule

  • The fixed High-Disc. threshold ai ≥0.7 represents a 2.01 odds multiplier for a one-standard-deviation ability increase, not a significance test.The same threshold, scale, and optimization settings are applied across all six OE-Track benchmarks and Table 8 ablation variants; negative slopes prompt inspection, while ReverseRisk routes items for review without changing benchmark scores.Source: §E.6 · p. 44, Table 31 · p. 48

Bootstrap stability and classical discrimination analysis

  • Bootstrap replicates resample the same 11 model identities across benchmarks; a classical robustness check compares the three highest- and lowest-scoring models on each item.The reported 95% interval uses the 2.5th and 97.5th percentiles, and ρS measures agreement between the 2PL and classical item rankings.Source: §E.6 · p. 44

Item-discrimination results

  • More than 35% of items are highly discriminative in all six benchmarks, exceeding 40% in four; SpatialUAV reaches 48.8%, versus 49.4% for its Table 8 Full Process Quality Control result.All mean top–bottom score gaps are positive, and the six rank correlations range from 0.845 to 0.929.Source: §E.6 · p. 44

Benchmark information analysis

  • Figure 8 compares absolute benchmark information in panel (a) and normalized information profiles in panel (b); all peaks fall between θ = −0.75 and −0.10, with information retained across [−2, 2].Higher total information indicates more precise model separation at that ability level.Source: §E.6 · pp. 45–46, Figure 8 · p. 49
Loading 2609.13082v1…