UnityBench-Mini3D: Evaluating LLM Agents for 3D Game Development in Unity

1 Introduction

• Game development couples scene organization, gameplay logic, physics, animation, interfaces, and tools through shared project state, making correctness depend on project structure and runtime behavior.A project can compile and launch yet contain a miswired reference, a character that clips through the floor, or an interface that never updates.Source: §1 · pp. 1–2
• Existing benchmarks leave gaps in Unity coverage, skill-based task organization, and evaluation fine-grained enough to explain what an agent builds correctly and where it fails.Some use tutorial-derived or genre/topic-grouped tasks, while others rely on LLM judges or reduce outcomes to pass/fail or aggregate similarity.Source: §1 · p. 2
• UnityBench-Mini3D provides 77 project-level Unity tasks across seven expert-defined development skills and three difficulty tiers, evaluated by 1,178 held-out human-authored milestone tests.Independent checkpoints provide deterministic requirement-level partial credit and distinguish partial implementations from complete solutions.Source: §1 · p. 2
• The paper contributes a reproducible benchmark and evaluation framework, a 21-model and 9-harness baseline, and analyses of game-development bottlenecks and future-agent directions.The framework pins the engine version, dependencies, and testing infrastructure.Source: §1 · p. 2

LLMs for Game Development

• Game-development systems can produce runnable games, but their evaluations remain tied to particular pipelines and task families rather than standardized project-level assessments across development skills.The surveyed work spans Unity, Unreal, and browser games.Source: §2 · p. 3

Benchmarks for Game Development

• Existing benchmarks vary in engine, task construction, and evaluation, including Godot, Unreal, Pygame, and browser-native suites, tutorial adaptations, genre groupings, and rubric-guided LLM or VLM judges.Other approaches use repeatable execution-based evaluation, including binary task outcomes and project-level measures.Source: §2 · p. 3
• UnityBench-Mini3D evaluates final Unity project states without LLM judges or content similarity, using human-authored runtime milestones to expose partial completion at intended-requirement level.The benchmark comparison table identifies task source, task and test counts, judge-free deterministic evaluation, partial scoring, and category type as comparison dimensions.Source: §2 · p. 3, Table 1 · p. 3

3.1 Benchmark Construction

• Benchmark candidates underwent expert and developer review, with revision until all eleven reviewers approved them; golden references scored 100% and unmodified templates scored 0%.Six Unity experts defined core categories and planned tasks, while five developers with LLM and AI-agent experience joined the review.Source: §3.1 · pp. 3–4
• Table 2 presents the task taxonomy, example tasks, and required skills, with a representative task for each core category except Miscellaneous.The workflow overview depicts agents building projects from initial challenge templates and milestone tests evaluating final project states.Source: Table 2 · p. 4, Figure 1 · p. 5
• The 77-task taxonomy classifies tasks by the primary Unity development skill they require, covering scene construction, animation, particles and shaders, physics, user interface, editor operations, and miscellaneous tasks.This organization makes category-level performance interpretable as development capability rather than game theme.Source: §3.1 · p. 4

3.2 Benchmark Structure

• Each task pairs a valid initial Unity project with a natural-language specification defining its goal, context, constraints, and resources.Agents modify interconnected project state while preserving functionality already in place.Source: §3.2 · p. 4

3.3 Evaluation Process

• The workflow separates agent execution from held-out engine-level evaluation, and the final project state is the agent’s only output.Agents receive the initial project and task specification and may use the file, code, and Unity Editor tools provided by their harness; the evaluator is injected after execution ends.Source: §3.3 · p. 4
• After clean reimport and compilation, the evaluator runs milestone tests through Unity Test Runner; tests inspect project structure and configuration as well as runtime behavior.EditMode tests inspect hierarchies, components, serialized properties, transforms, and asset references; PlayMode tests assess physics, animation, and interface responses.Source: §3.3 · pp. 4–5

PTi

• Each task score is the fraction of its milestone tests passed, and the overall benchmark score weights each task equally.The evaluator retains test outcomes and distinguishes implementation failures from runs that could not be evaluated.Source: PTi · p. 5

4.2 Evaluation Results

• About 70% is the strongest overall score, and 25 of 32 model–harness configurations exceed 50%.Scores remain stratified within a fixed harness: Claude Opus 5 exceeds Seed 2.1 Pro by 44.35pp on Codely, while Kimi K3 outperforms DeepSeek V4 Flash by 60.06% in relative terms on OpenCode.Source: §4.2 · p. 6, Table 3 · p. 6
• Harness choice can rival model choice: Unity-specific Codely beats general-purpose OpenCode for every shared model, while average task cost varies roughly 60-fold across coding configurations.Grok 4.5 with Grok Build scores 71.19%, 0.56pp below the best Anthropic result, at less than one sixth of the cost.Source: §4.2 · pp. 6–7
• Engine interaction accounts for 26%–37% of calls when aggregated by model, versus 13%–51% across harnesses; visual tools remain below 8% of calls in every harness.Codely's dedicated engine operations account for 51.44% of its calls, while four harnesses never capture a rendered frame.Source: §4.2 · p. 7, Figure 2 · p. 7
• 80.85% on easy tasks falls to 55.56% on medium and 35.21% on hard, with only three of 32 configurations exceeding 50% on hard.Harness effects grow with difficulty: GPT 5.6 Sol gains 18.00pp on hard when moving from Codely to Codex, compared with 5.68pp on easy.Source: §4.2 · p. 8

4.3 Failure Analysis

• Stronger configurations more often create required elements but place or size them inaccurately, while weaker ones more often leave work missing or incomplete.Grok Build has 17.77% of non-passing milestones in missing or incomplete implementation and 36.53% in spatial or numerical inaccuracies; Antigravity shows 42.54% and 9.94%, respectively. Compilation failures are at or below 3.03% for all but Antigravity, and account for 2.27% of runs overall.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 Ablation Study

• Multimodal input is not the main limiting factor in the reported clean comparisons: three no-MM runs each score slightly higher and finish 3.00–4.19 minutes sooner per task.The archived Codex batch may also disable MCP, so it is not treated as a clean single-factor ablation.Source: §4.4 · p. 9, Table 5 · p. 9
• Removing Codely's dedicated engine tools lowers overall scores for every tested model, with the largest average loss on medium tasks at 6.89pp.Runs end 4.67–12.25 minutes earlier despite lower scores; GPT 5.6 Sol costs 33% more.Source: §4.4 · p. 9, Table 5 · p. 9
• Adding GUI control widens the overall-score range across the three GPT configurations from 6.11pp to 24.23pp, while all three become slower and more expensive without an overall gain.GPT 5.6 Sol improves by 11.21pp on user-interface tasks, but GPT 5.6 Terra and Luna decline by 12.73pp and 19.58pp overall.Source: §4.4 · pp. 9–10, Table 5 · p. 9

5 Conclusion, Future Work and Broader Impact

• UnityBench-Mini3D covers 77 Unity challenges across seven skills, using human-authored milestone tests to evaluate final project states in-engine.The study evaluates 21 models across nine coding harnesses and compares two CUA harnesses.Source: §5 · p. 10
• Dedicated engine tools improve performance, but more tool calls, longer runtimes, multimodal input, or GUI control do not consistently help; the main bottleneck is coherent scene construction, not compilable code.Source: §5 · p. 10
• Future work will add medium and hard challenges, introduce multiplayer, optimization, and post-processing categories, rebalance existing categories, and offer domain-specific evaluation subsets.Regular additions are also intended to reduce data-contamination risk as models and harnesses evolve.Source: §5 · p. 10
• Fine-grained milestone feedback can guide model and harness development and may make Unity prototyping more accessible to independent developers, educators, and non-expert creators.Source: §5 · p. 10

A UnityBench-Mini3D Challenge Scenes

• Table 6 lists the 77 challenges by skill category, with initial and completed reference projects and difficulty and milestone-test counts identified for each entry.Source: Table 6 · p. 12
• Scene and level construction comprises 19 tasks and 365 milestone tests, while particles and shaders comprise 12 tasks and 155 milestone tests.The showcased scene tasks include object identification, street layouts, interiors, navigation, terrain, and castle construction; the particle and shader examples include fireworks, black-hole skyboxes, and collision effects.Source: A UnityBench-Mini3D Challenge Scenes · pp. 21, 22, 23, 26
• Selected scene challenges pair an initial template with a completed golden reference, including object identification and street-layout examples.Other showcased tasks include a paper-thin multistory house, interior furniture placement, and multi-room navigation.Source: A UnityBench-Mini3D Challenge Scenes · pp. 21–22
• Easy examples include CubeMove (20 tests), VoxelWormsTerrain (12), ProceduralSkinnedCloth (10), and CookiesApply (9).Source: A UnityBench-Mini3D Challenge Scenes · pp. 29, 30, 32
• The cited hard examples are TakeOffAndLanding (31 tests), ColliderWrappingChallenge (26 tests), DynamicRainEffect (8 tests), PhysicsMaterial (8 tests), and AutoTimelineGenerator (10 tests).Source: A UnityBench-Mini3D Challenge Scenes · pp. 28, 31, 32, 33
• Medium examples include CreateLOD (39 tests), RollingBallCheckpointCourse (15), RigidbodyCharacterControllerParity (15), and RoadsideJointPhysics (12).The listed medium scenes also include CatapultExplosion (11 tests) and ShootAheadVehicleWheelPhysics (8 tests).Source: A UnityBench-Mini3D Challenge Scenes · pp. 30–32

B Challenge Specifications

• Table 7 presents agent-visible specifications for all 77 tasks, each separating the goal, initial context, 780 explicit constraints, and relevant Unity resources.These specifications make the requested work and its boundaries explicit.Source: Table 7 · p. 33
• The Snake task requires an editable UI Toolkit interface built with UIDocument, UXML, and USS, plus grid-based gameplay, adjustable speed, and a futuristic visual style.It prohibits a UGUI Canvas and requires speed changes to affect movement immediately without disrupting game state.Source: B Challenge Specifications · pp. 33–34
• The minimap task requires a stable, north-facing top-down view with a centered rotating player arrow, while restricting changes to existing minimap UI logic and objects.Runtime behavior may update existing objects’ positions, rotations, and texture references but may not create new GameObjects.Source: B Challenge Specifications · p. 34
• Animation & Animator covers 12 tasks and 179 milestone tests, including a door that smoothly loops through opening and closing using an Animator.The door must rotate around the Y axis and complete each full cycle within six seconds.Source: B Challenge Specifications · pp. 37–38
• Other animation examples require continuous motion of both tank tracks and road wheels, or a diagonal sword slash with a hit-detection window timed by animation events.The tank animation must advance over time; the sword’s hit window opens mid-slash and closes before the slash ends.Source: B Challenge Specifications · p. 38

Animation challenges

• Animation challenges require correctly timed event callbacks, layered full- and split-body playback, and IK that prioritizes interactive object and ground following.The specifications include typed event parameters, E-key animation switching without disrupting lower-body playback, and hand, head, and foot responses to targets and nearby objects.Source: B Challenge Specifications · pp. 39–40

Excavator AnimationClip Dig and Dump

• The excavator task requires Dig and Dump clips to move cubes through genuine physics-based scooping, carrying, and unloading, without scripts directly moving them.A fixed program plays Dig, rotates the turntable by 180 degrees, then plays Dump; the clips animate the mechanical arm and bucket.Source: B Challenge Specifications · p. 40

Character animation control

• Character-control tasks specify script-driven Playables with state, root-motion, and pose controls, alongside an archer controller for movement, diving, falling, and masked aiming.The Playables challenge defines idle, walk, run, and in-air transitions; the archer task specifies blend trees, input mappings, and a smoothly weighted upper-body aim layer.Source: B Challenge Specifications · pp. 40–42

Character presentation

• ShootAhead tasks require upper-body aiming that preserves movement semantics, while remote FPS presentation synchronizes weapon pose, grip, animation actions, and recoil.The split-body challenge requires facing the aim point while moving, including backward-movement semantics when movement and aim diverge; the FPS task preserves first-person behavior while configuring the third-person rig.Source: B Challenge Specifications · pp. 42–43

Animation Classification

• The hard Animation Classification task asks agents to analyze each character’s clip and reparent its root under the matching description node, changing no other project content.Each character has exactly one matching description node.Source: B Challenge Specifications · pp. 42–43

Editor Operations

• Editor Operations covers 11 tasks and 142 milestone tests, including road-segment classification and M249 integration into existing weapon systems.Road classification relies on spatial position and road connections while preserving transforms; the M249 must use the existing AssaultRifle type, 7.62 ammo, and a 100-round magazine.Source: B Challenge Specifications · pp. 43–44

0027-VolumeBoundsOrganizer

• The easy, 16-test scene-organizing task classifies root-level mesh objects by local oriented-bounds volume, renames and groups them, and requires a single-step undo.It specifies strict volume boundaries, safe handling of missing meshes and zero volume, and idempotent repeated execution without duplicate prefixes or nested groups.Source: B Challenge Specifications · p. 45

0028-ImportExportPackages

• The easy, 7-test package task exports exactly five selected prefabs with dependencies and imports five target prefabs with dependencies into designated project paths.A separate cleanup specification directs keeping build-flow dependencies while removing examples, demos, unused art, and unrelated resources.Source: B Challenge Specifications · pp. 44, 46

0042-PrefabSharedMaterialIsolation

• The easy, 16-test prefab task combines shared-material isolation, replacement of 12 loose objects with one prefab, and construction of two prefab variants.It requires preserving instance links and transforms while retaining specified prefab assets and scene changes.Source: B Challenge Specifications · pp. 46–47

0052-URPPipelineConfiguration

• The easy URP task requires four progressively configured quality levels, Medium as the active default, and conversion of sample materials while preserving base color and alpha.The project starts with Built-in Render Pipeline samples and no completed URP setup.Source: B Challenge Specifications · p. 47

0016-TexturePipeline

• The medium, 8-test texture task requires a first-import-only AssetPostprocessor with semantic routing, UI-specific handling, alpha-aware mobile formats, and platform size limits.It forbids overwriting artist-modified settings on existing textures and distinguishes UI assets by their full path.Source: B Challenge Specifications · pp. 47–48

0018-FindGameObjectsInComplexScene

• The medium, 10-test scene-search task asks agents to identify ten unique objects from spatial, lighting, transform, layer, and static-state conditions, then reparent them under Answer/Q1–Q10.It prohibits reusing objects or changing names, Inspector properties, or anything beyond hierarchy parent-child relationships.Source: B Challenge Specifications · p. 48

0060-GunShotSound

• The hard, 29-test gunshot task requires a data-driven audio system with pooled voices, mixer routing, automatic-fire lifecycle, distance layers, and physics-based occlusion.It also specifies a shared pool capped at 32 simultaneous one-shot voices, with no voice-object instantiation or destruction after warm-up.Source: B Challenge Specifications · pp. 48–49

0072-CastleDestruction

• The hard castle task requires hierarchy-only classification of existing models into gate, bastion, and wooden-hoarding groups while leaving continuous walls unsorted and preserving world transforms.The scene contains existing models under Castle_Objects_Unsorted.Source: B Challenge Specifications · p. 49

Scene & Level Construction

• The cited entries show four task examples: Easy ObjectIdentificationPrefabSorting (33 tests) and ShootAheadStreetLayout (10 tests), plus Hard Shop3DLetters (44 tests) and TerrainRiverWaterAssembly (14 tests).Source: B Challenge Specifications · pp. 50, 54
• Prefab-identification tasks require matching 30 numbered objects to named parent objects without modifying the objects or dependent assets.Another task requires readable PIZZA PLACE and BOOK STORE signs assembled from individual letter prefabs in specified hierarchies.Source: B Challenge Specifications · pp. 50, 54
• Scene-assembly tasks specify exact layouts and asset constraints for streets, furnished homes, modular sci-fi rooms, and connected trench rows.Requirements include asset use, room-specific placements, gap-free grids or boundaries, and limits on layout size or prefab position.Source: B Challenge Specifications · pp. 50, 51, 52, 54

NavMeshAgent OffMeshLink Multi-Room Navigation

• Navigation tasks require bakeable, connected walkable areas, valid agent paths, and a bidirectional OffMeshLink between lower- and upper-floor teleporters.A separate house task requires at least three floors, three mutually reachable rooms per floor, and stair connections usable by a NavMeshAgent.Source: B Challenge Specifications · pp. 51, 53

11x11 Unique-Path Maze Construction

• Each of ten 11x11 maze scenes must have connected paths and exactly one route from the center entrance to a single boundary exit.The specification also prohibits all-wall or all-path 2x2 areas and requires the wall hierarchy and grid transforms to be preserved.Source: B Challenge Specifications · p. 53

Terrain River and Lake Water Surface Assembly

• A terrain-water task requires continuous lake and river surfaces that follow a descending channel, meet cleanly, and use built-in Plane meshes.Lake edges must embed into Terrain, while adjacent river segments must align at matching heights.Source: B Challenge Specifications · p. 54

Terrain and landscape tasks

• Hard scene-building tasks require recreating a circuit, decorating terrain with exact tree and rock populations, and preserving specified scene structure.The Shanghai Circuit task requires reference-matched terrain, continuous road-edge lines, and grouped turn props; the landscape task specifies 200 terrain trees and 50 grounded rocks.Source: B Challenge Specifications · pp. 54–55

Spatial placement and assembly

• Hard spatial-editing tasks cover camera placement, connected castle stairs, and reassembling fourteen prefab-linked crystal fragments around fixed anchors.The camera task permits changing only nine cameras’ positions and rotations; the crystal task permits only translating and rotating its fourteen movable fragments.Source: B Challenge Specifications · pp. 55–56

House and road construction

• Other hard tasks translate a floor plan into a dimensioned white-box house and place lane-direction arrows across urban intersections.The house specification defines wall heights, grouping, tags, and door and window dimensions; the road-marking task maps reachable directions to lane-specific arrows behind stop lines.Source: B Challenge Specifications · pp. 56–57

Shaders and particle effects

• Shader and fireworks tasks specify a black-hole skybox conversion and scene-only particle effects for smoke, sparks, a rocket-head halo, and an explosion.The fireworks requirements include particles emitted opposite rocket travel, an attached flickering yellow core, and an explosion that transitions from white through yellow to dark red.Source: B Challenge Specifications · pp. 57–58

Materials and particle-system classification

• The pine-material task requires corresponding URP/Lit conversions that preserve appearance and alpha cutout behavior without overwriting originals.The section also lists a medium-difficulty particle-system classification task with 10 tests, but provides no further specifications for it.Source: B Challenge Specifications · p. 59

Particle Effects & VFX

• Particle-effect tasks range from classifying prefabs under existing categories to building a URP dissolve shader with animated multilayer noise, HDR edge emission, and vertex deformation.The dissolve shader must expose controls for dissolve progress, noise scale, edge width, and edge color.Source: B Challenge Specifications · pp. 59–60
• A rain-splash task requires downward raindrops to collide with water and trigger brief splashes at actual impact points, rather than using fixed-position or independent looping effects.The specification calls for controlling particle counts and preserving the named scene objects.Source: B Challenge Specifications · p. 60
• A karate-character VFX task rebuilds three graphs around shared live skinned-mesh sampling, with Timeline-controlled emission and full-body particles and continuous trails.Throttle must stop new emission while existing effects decay; runtime bounds must remain finite and close to the character.Source: B Challenge Specifications · pp. 60–61
• Other VFX tasks specify world-space rain that follows the player and tilts and lengthens its streaks with horizontal speed, plus falling leaves affected by a player-following vector field.The leaves task requires plane-based collision and ground clamping while still allowing landed leaves to be pushed by wind.Source: B Challenge Specifications · p. 61

Physics & Collision

• The Physics & Collision category contains 11 tasks and 168 milestone tests, including Rigidbody cube movement, procedural skinned cloth, and voxel terrain with runtime explosions.The cloth specification combines a weighted three-bone rig with particle constraints and supported colliders; the voxel task requires mesh and collider updates after explosions.Source: B Challenge Specifications · pp. 62–63
• Further physics tasks require a frame-rate-independent catapult launch-and-explosion chain and a WheelCollider vehicle controlled through preserved public input methods.The vehicle must cap its speed and remain stable during straight full-throttle driving.Source: B Challenge Specifications · pp. 63–64
• A rolling-ball task requires completing three checkpoint scenes with distinct obstacle layouts that guide the ball through every required checkpoint.Source: B Challenge Specifications · p. 64

Checkpoint-course challenge

• The checkpoint-course challenge completes three independent scenes so a ball rolls freely from its spawn through five required checkpoints using as few built-in Cube blockers as possible.Existing slope, ball, spawn, checkpoint, and parent objects must remain fixed, and scripts cannot directly control the ball’s position.Source: B Challenge Specifications · pp. 64–65

0048-RoadsideJointPhysics

• The joint-physics task configures four road scenes with FixedJoint, HingeJoint, SpringJoint, and ConfigurableJoint behaviors while preserving target objects and real Rigidbody physics.The joints must respectively allow impact displacement, bounded swinging, rebound, and limited road-direction sliding with other motion constrained.Source: B Challenge Specifications · p. 65

0056-RigidbodyCharacterControllerParity

• The character-parity task uses a Rigidbody and CapsuleCollider motor to match a CharacterController reference across movement, jumping, slopes, steps, walls, and normalized diagonal input.The fixed reference values include a 0.35 radius, 1.8 height, 3.8 walk speed, 5.2 sprint speed, 1.1 jump height, and -22 gravity.Source: B Challenge Specifications · p. 65

0044-ColliderWrappingChallenge

• The collider-wrapping task builds readable primitive-collider rigs for eight military assets, avoiding mesh-based proxies while supporting plausible raycast and physics behavior from multiple directions.Permitted colliders are BoxCollider, SphereCollider, and CapsuleCollider, positioned with empty child GameObjects; coverage should avoid oversized spill.Source: B Challenge Specifications · p. 66

0053-PhysicsMaterial

• The PhysicMaterial task tunes four provided assets so spheres cross checkpoints in three scenes and a curling stone maximizes the red side’s score in a fourth.Only the four PhysicMaterial assets may be modified; friction, bounciness, and combine modes determine the required trajectories.Source: B Challenge Specifications · p. 66

0077-TakeOffAndLanding

• The takeoff-and-landing task requires a closed-loop Rigidbody aircraft to take off, fly a compact circuit, land main wheels first, and stop using physical forces and wheel contact.It must avoid direct runtime movement of position or velocity and complete the sequence within 120 seconds, with stopping within 25 seconds of touchdown.Source: B Challenge Specifications · pp. 66–67

Miscellaneous

• The miscellaneous tasks cover Cookie lighting setup and LOD asset creation, splitting, merging, and mesh reduction under scene, asset-format, and LODGroup constraints.The Cookie task requires 2D and Cubemap assets assigned to Directional and Point Lights; LOD outputs must be FBX assets and preserve or decrease triangle counts as specified.Source: B Challenge Specifications · pp. 67–68

0019-AutoTimelineGenerator

• The Timeline task generates a cinematic from animation, VFX, and audio assets, with exactly four specified root tracks and tested bindings, timing, root motion, camera shake, and framing.The output must be saved as GeneratedCinematic.playable, with the character and explosion remaining inside the camera frustum throughout playback.Source: B Challenge Specifications · p. 68

C Test Cases

• 1,178 held-out, human-authored checks evaluate 77 tasks, with each independently scored test verifying a named project property.The table describes the checks used to evaluate the tasks.Source: Table 8 · p. 69
• Heading tests award 1–6 of 6 points as cardinal-direction tolerance tightens from 6 degrees to 1 degree.The six checks assign successive scores at tolerances of 6, 5, 4, 3, 2, and 1 degree.Source: C Test Cases · p. 69
• The 23-test Snake task checks gameplay, controls and speed settings, interface elements, and a UI Toolkit-based scene configuration.Checks include food-driven score and growth, collision game-over behavior, movement controls, speed updates, and UI asset references.Source: C Test Cases · pp. 69–70
• Minimap checks cover camera tracking and orientation, RenderTexture-based display, player-arrow behavior, and player-layer exclusion.The task has seven tests, including checks for an orthographic camera fixed to world north and a centered arrow that rotates with the player.Source: C Test Cases · p. 70
• Other interface-focused checks assess sprite slicing and pixel dimensions, GoldenBench object and layout fidelity, and navigation and button behavior.The Sprite task checks sub-sprite properties; the UI recreation task checks required components, sprites, layout tolerances, and scene canvas configuration.Source: C Test Cases · p. 70
• Endless-runner, mobile-HUD, and bubble-shooter checks exercise runtime actions alongside UI and scene requirements.Examples include jump and restart behavior, mobile HUD and crosshair rendering, and bubble pressure, aiming, scene UI binding, and connected-bubble clearing.Source: C Test Cases · pp. 71–72
• Bubble-game tests cover runtime clearing, aiming and firing, score persistence, scene setup, UI styling, and visual-tree placement of gameplay surfaces.The checks include an editor-placed responsive UIDocument, overflow constraints, and keeping gameplay elements such as the next bubble in the play area.Source: C Test Cases · p. 72
• Jet-cockpit HUD tests check first-person visibility, speed- and height-responsive readouts, attitude and horizon alignment, heading wraparound, and delayed-impact history.The delayed-impact marker is checked against approximately 0.2 seconds of aim history.Source: C Test Cases · p. 72
• 12 Animation & Animator tasks comprise 179 milestone tests, including checks for animation-event timing and parameters, Animator settings, tank-track motion, and sword-attack behavior.The tests cover event order and normalized timing, callbacks, wheel-mesh rotation and periodic track-UV scrolling, and attack-window events during a sword swing.Source: C Test Cases · pp. 73–74
• The Easy AutoDoor project has 29 tests covering scene components and hierarchy, door-pivot geometry, and smooth opening-and-closing animation.Checks include an assigned Animator controller, the door root’s pivot near the door edge, and animation without discontinuous rotation jumps.Source: C Test Cases · p. 73
• Layered-animation and humanoid-IK tests check controller layers and masks, reference-pose bone transforms, and whether hands, feet, and the head follow targets.The IK cases include static and dynamic hand and foot targets, plus head tracking as a cube moves.Source: C Test Cases · pp. 74–75
• The IK checks cover right-foot and right-hand targets in static and dynamic conditions, plus left-hand tracking of a reachable sphere and return to its regular target.The dynamic checks include position and rotation changes; the static checks verify target position matching.Source: C Test Cases · p. 75
• The medium excavator task tests that dig-and-dump clips rotate only specified pivots and that the final pool meets cube-count thresholds from four to thirty-two.The listed thresholds include four, eight, twelve, sixteen, twenty, twenty-four, twenty-eight, and thirty-two cubes.Source: C Test Cases · p. 75
• The 17-test Animation Script Playable task checks Animator and script configuration, root motion and hip offsets, locomotion poses, and transitions among idle, in-air, run, and walk states.Additional checks cover airborne root-motion scaling or suppression, forward lean, and transition blending duration.Source: C Test Cases · p. 75
• The 30-test Humanoid Animator Blend Tree Controller task checks controller layers and states, blend trees and clips, player and Animator setup, masks, and runtime locomotion, dive, fall, and aim behavior.The checks also verify required parameters and bindings and ensure jump parameters and conditions are absent.Source: C Test Cases · p. 76
• The 12-test Split Body Aim task checks spine-chain targeting and yaw, root facing and backward-movement semantics, and distinct lower-body movement versus upper-body aiming.Cases include side movement while aiming and standing aim toward left and right world directions.Source: C Test Cases · p. 77
• The hard 20-test Animation Classification task checks whether characters using creature and humanoid animation clips are placed under the correct answer category.The listed cases include CreatureClip01–11 and HumanoidClip01–06.Source: C Test Cases · pp. 77–78
• The hard remote-rifle task uses 18 tests to check avatar and grip setup, aim alignment, hand and arm behavior, reload playback, and remote fire with recoil.The checks include muzzle alignment within 5 degrees across representative angles and verify reload animation is not restarted every frame.Source: C Test Cases · p. 78
• Editor Operations comprises 11 tasks and 142 milestone tests, including road-group cleanup, light-machine-gun integration, asset pruning, scene organization, and package handling.The road-cleanup task has 9 tests, while the light-machine-gun and asset-pruning tasks each have 15.Source: C Test Cases · pp. 78–81
• Road-group cleanup tests verify straight road segments are parented under their expected named groups and removed from the unsorted group.The checks cover eight road names and the unsorted group.Source: C Test Cases · pp. 78–79
• Light-machine-gun checks cover its definition, database registration, identity, weapon pipeline, usable stats, equipment and ammunition mapping, firing, assets, audio, and shop availability.The separate asset-pruning task checks removal of unused content while preserving required runtime data and dependencies.Source: C Test Cases · pp. 79–80
• The 16-test volume-bounds organizer task checks grouping by mesh-volume features, object preservation, idempotence, accurate processing counts, and safe, isolated Undo behavior.Tests also specify rotation-independent volume handling, strict boundaries, absolute lossy scale, and use of mesh bounds rather than collider bounds.Source: C Test Cases · p. 80
• The 7-test package task checks that exports contain requested prefabs and required dependencies, while imported prefab dependencies are organized without unrelated prefabs.Imported prefabs are also checked for removal from their original vendor locations.Source: C Test Cases · p. 81
• The easy prefab task has 16 tests covering cloned-material isolation, prefab variants, replacement instances, preserved transforms, and reusable prefab structure.Other checks verify that scene instances remain prefab-connected, inherit solved materials, and avoid renderer material overrides.Source: C Test Cases · p. 81
• The easy URP-configuration task has 8 tests checking material conversion and color preservation, distinct quality-level pipeline assets, active medium quality, and a low-to-extra-high progression.The checklist also verifies the URP package and exactly four quality levels: low, medium, high, and extra high.Source: C Test Cases · pp. 81–82
• The medium texture-pipeline task has 8 tests for semantic import routing, including texture type, alpha and color-space handling, platform formats, readability, and deceptive paths or extensions.The checks also require legacy Boss_Albedo settings to remain untouched while the AssetPostprocessor is active.Source: C Test Cases · p. 82
• The medium complex-scene task has 10 tests requiring each uniquely matching object for Q1–Q10 to be parented under its corresponding Answer/Qn location.Source: C Test Cases · p. 82
• The hard gunshot-audio task has 29 tests spanning scene setup, weapon profiles, mixer and source configuration, import policy, and runtime lifecycle, pooling, distance, and occlusion behavior.Checks include loop cleanup when an emitter is disabled, bounded voice-pool behavior after warm-up, and avoiding duplicate services after scene reload.Source: C Test Cases · pp. 83–84
• The hard castle-classification task has 9 tests assessing round and square bastion hierarchy groups, with checks for both at-least-50% and complete correctness.Correct groups must contain complete structures without adjoining wall sections.Source: C Test Cases · p. 84
• Bastion and wooden-hoarding tests distinguish structures correct in at least 50% of required groups from those correct in all groups.The cases also check that the Main_Gate contains the complete gate structure without adjoining continuous-wall pieces.Source: C Test Cases · p. 84
• Scene & Level Construction comprises 19 tasks and 365 milestone tests.The listed cases include object identification and street-layout tasks, as well as a medium-difficulty house-reachability task with four tests.Source: C Test Cases · pp. 84, 85, 87
• Object-identification tests check correct parents for numbered items, anonymity of items and children, and the required scene scaffold.The easy task lists 33 tests, including checks that numbered items are placed under their matching category parents.Source: C Test Cases · pp. 84–85
• Street-layout tests check building and tree placement, road continuity, required scene parents, and prefab positioning.The easy task contains 10 tests, including non-overlap checks for buildings and requirements for roads and trees around the intersection.Source: C Test Cases · p. 85
• A second easy object-identification task also lists 33 tests for numbered-item parent assignments, anonymity, and the required scaffold.Its listed cases repeat checks for matching category parents and anonymous numbered items and children.Source: C Test Cases · pp. 86–87
• BuildHouse tests check required named parts, reachable rooms across floors, and stairs connecting adjacent floors.Source: C Test Cases · p. 87
• The 39 Interior Furniture Placement tests check required furniture counts and spatial constraints such as wall alignment, floor height, and relative placement.Examples include bathroom furniture counts and living-room chairs positioned on both sides of a table.Source: C Test Cases · pp. 87–88
• Reference Street Recreation tests compare buildings, roads, trees, ground, and prefab counts with a reference, while Sci-Fi Room Size Variants tests room contents and geometry.The sci-fi checks include corner pillars, room roots, non-overlap, continuous floor grids, and wall-and-door boundaries.Source: C Test Cases · p. 88
• The 10 NavMeshAgent OffMeshLink tests check reachable paths, walkable points, navigation costs, obstacles, teleporter links, and NavMesh setup.They also test connections between door transitions and multi-floor paths.Source: C Test Cases · pp. 88–89
• The cited tests 01, 02, and 10 check that Maze_01, Maze_02, and Maze_10 each satisfy every grid, boundary-exit, unique-path, 2x2-pattern, transform, and reachability rule.Source: C Test Cases · p. 89
• The 16 TrenchAssembly tests cover collider overlap thresholds, prefab use, piece counts, row ordering, and anchor-point reachability.The listed overlap limits range from 20% of the smaller footprint and 10% IoU to 40% and 30%, respectively.Source: C Test Cases · p. 89
• Trench tests check whether prefab anchor points are mutually reachable at specified percentage thresholds across three trench segments.The listed thresholds include 33.3%, 66.7%, and 100%.Source: C Test Cases · pp. 89–90
• Shop3DLetters tests verify letter placement on BookStore and PizzaPlace signs at loose and tight tolerances, plus per-axis scale ranges.The scale ranges are 0.2 to 0.8 for loose tests and 0.4 to 0.6 for tight tests.Source: C Test Cases · pp. 90–91
• River-and-lake water tests check coverage and terrain burial, river-to-lake level transitions, segment seams, water-plane meshes, rotations, and materials.The listed checks include descending water levels from the start lake to the end lake and nearly level segment seams.Source: C Test Cases · pp. 91–92
• Racing-circuit tests cover terrain paint-mask tolerances, white-line placement, curb and rail coverage, barrier positioning, and prop grouping and continuity.The paint-mask tiers use road-boundary tolerances of 1.50, 0.85, and 0.35 metres.Source: C Test Cases · p. 92
• The hard camera-placement task uses 11 tests to check indoor and outdoor prop visibility, overlap, and whether at least 60% of assigned props pass visibility criteria.The outdoor and indoor overall checks each set a 60% threshold.Source: C Test Cases · p. 93
• The 24-test terrain-decoration task checks tree counts and placement against terrain textures and water, plus dry-land rock footprints and non-mechanical coverage of the water system.Examples include exactly 150 pines and 50 fruit trees, at least 95% of trees at least 3.2 metres from visible water, and rock spacing across lakes and rivers.Source: C Test Cases · pp. 93–94
• The castle task’s nine tests assess straight, round, and corner stair connections, including completeness and the absence of redundant instances.Checks include golden-connection thresholds and full correctness without redundant instances for each stair type.Source: C Test Cases · p. 94
• The 14-test crystal-fragment task checks whether specified objects’ serialized positions and rotations match the golden assembly within tolerance.The listed checks specify target positions and rotation angles for individual fragments.Source: C Test Cases · pp. 94–95
• The 23-test house task checks wall and object structure, window dimensions and spacing, furniture placement and size, and penetration against furniture or walls.It also checks wall-path coverage against reference floor-plan geometry.Source: C Test Cases · p. 95
• UrbanRoads’ 31 tests check that arrows for different turning movements are correctly oriented, within approach-area bounds, and flush with the road.The listed cases cover straight, turn, and combined-direction arrows.Source: C Test Cases · p. 96
• The black-hole skybox tests assess visual similarity using contrast and luminance thresholds, dark-centroid distance, and grid-feature error.The listed checks include coarse contrast and luminance reaching 20% and 35% of reference, alongside bounds on centroid distance and grid-feature error.Source: C Test Cases · p. 97
• Fireworks tests verify particle settings across smoke trails, sparks, glow, and explosions, including enabled modules, shape, lifetime, and color behavior.Examples include sparks trails being enabled, glow using a sphere shape, and burst explosions having at least three distinct colors and duration greater than two.Source: C Test Cases · pp. 97–98
• The burst-explosion and burst-flash checks verify module settings and behavior, including looping, enabled modules, lifetimes, size, speed, and velocity deceleration.The checks also specify sphere shape, nonzero gravity modifier, noise, renderer, and pure-white flash color.Source: C Test Cases · pp. 98–99
• Medium-task cases test URP material conversion and textures, particle-effect classification by hierarchy, and rain collisions that create controlled splash particles.The listed checks also cover particle-system settings for burst explosions and flashes.Source: C Test Cases · pp. 99–100
• The Dynamic Dissolve tests verify that shader and material assets exist and are linked, expose DissolveAmount, DissolveScale, EdgeWidth, and HDR EdgeColor, support URP at runtime, remain fully visible at DissolveAmount = 0, and clip at 1.Source: C Test Cases · p. 99
• Hard-task checks cover skinned-mesh particle and trail graphs, animated-renderer bindings and throttle behavior, dynamic rain motion, and falling-leaf collision and ground-clamp configuration.The skinned-mesh tests also check runtime capacity and bounds; rain checks include world-anchored particles and stronger opposing motion as the target moves.Source: C Test Cases · pp. 100–101
• The Physics & Collision category comprises 11 tasks and 168 milestone tests; the listed CubeMove checks cover camera visibility and required player collider, controller, and Rigidbody components.CubeMove is identified as an easy task with 20 tests.Source: C Test Cases · p. 101
• The starter controller tests check required fields and methods, scene objects, Rigidbody settings, idle stability, jumping, and directional movement.The movement tests require forward input to increase world Z and left input to decrease world X; jump checks cover positive initial Y velocity, landing, and rejecting a second in-air impulse.Source: C Test Cases · pp. 101–102
• Procedural skinned-cloth tests validate the mesh and rig, cloth coefficients and colliders, solver settings, and baked deformation when rig bones move.They also check bone-child relationships and matching bindposes, curtain alignment, and the same-object cloth and SkinnedMeshRenderer configuration.Source: C Test Cases · p. 102
• Voxel-terrain tests check substantial mesh and collider generation, tunnel and surface geometry, voxel-data operations, and explosion-driven terrain changes.Checks include world-space spherical explosions, accumulated voxel removal, rebuilt meshes and colliders, and terrain remaining unchanged for explosions outside the grid.Source: C Test Cases · pp. 102–103
• Catapult tests cover animation-event stone spawning, launch and explosion physics, and outward target movement at multiple specified positions.The position-specific checks require targets to move outward by 1 to 5 units after an explosion.Source: C Test Cases · p. 103
• Vehicle-wheel tests assess wheel-collider alignment and steering, root-level controller and Rigidbody placement, expected mass, and controlled acceleration and speed.They also check forward motion through wheel physics and stability when steering input is zero.Source: C Test Cases · p. 103
• Rolling-ball course tests preserve fixed spawns and layout while checking cube budgets, built-in cube obstacles, and checkpoint visits across three ball groups.Checkpoint checks require each group to reach at least one, at least three, or all five checkpoints.Source: C Test Cases · pp. 103–104
• Roadside-joint tests check joint types, anchors, axes, limits, and break or spring settings, alongside impact responses for bollards, fences, gates, and barriers.The runtime checks include sliding along the road axis, breakaway under impact, gate swing, and barrier deflection with a return trend.Source: C Test Cases · p. 104
• Character-controller parity tests check that course objects and start markers remain intact, reference parameters stay fixed, and the Rigidbody motor retains required API and tuning fields.Source: C Test Cases · p. 104
• Character-motion checks require stable Rigidbody settings and a dynamic Rigidbody capsule without reliance on CharacterController.They also check that the capsule matches a reference shape.Source: C Test Cases · p. 105
• Character-motion behavior checks cover normalized diagonal input, falling and jumping, step and slope traversal, sprint speed, and wall blocking.The listed tests compare these behaviors with a reference.Source: C Test Cases · p. 105
• 26 Collider Wrapping tests check primitive-collider coverage, excessive spill, representative raycasts across reference objects, and prohibited mesh APIs.The checks cover reference objects A–H and forbid specified mesh APIs in submitted scripts.Source: C Test Cases · pp. 105–106
• The PhysicsMaterial task has 8 tests checking player scores in scene four and sphere visits to checkpoints across scenes one, two, and three.The listed checks require scores of at least one and at least two points, plus visits to the specified checkpoints.Source: C Test Cases · p. 106
• 31 tests assess physical takeoff, landing, rollout, and stopping across nominal and two disturbed flight scenarios.Checks include a 50–85 m/s plausible liftoff-speed range and stopping within 25 seconds after touchdown.Source: C Test Cases · pp. 106–107
• The miscellaneous grouping contains 3 tasks and 58 milestone tests, including a 9-test CookiesApply task and a 39-test CreateLOD task.The listed cookie checks cover light and texture settings and required scene objects; the shown CreateLOD check verifies an FBX exists.Source: C Test Cases · pp. 107–108
• The mesh tests check FBX existence, triangle or face counts, LOD structure and transitions, and material assignments across multiple answers.Checks include exact triangle counts for some LODs and decreasing face counts across successive levels.Source: C Test Cases · p. 108
• Timeline tests validate named tracks, bindings, clips, timing, root motion, VFX and audio behavior, camera shake, framing, and runtime playback.They also check camera-rig hierarchy and composition, character movement consistency, and viewport integrity during shake.Source: C Test Cases · pp. 108–109

D Supplementary Statistics

• The supplementary analyses use 32 configurations on 77 tasks (2,464 runs), excluding ablations, and define overall scores as unweighted means across tasks.They address internal calibration, coverage, and score robustness, but not external validity or repeated-run variance.Source: D Supplementary Statistics · p. 109

Difficulty calibration and discriminative power

• Mean scores fall from 80.85% on Easy tasks to 55.56% on Medium and 35.21% on Hard, while stronger-versus-weaker configuration gaps remain similar at 15.56, 16.15, and 17.36 percentage points.The gap comparison has p = .845 under permutation.Source: §D.1 · p. 109, Figure 3 · p. 109
• The task-equal gap is positive for 75 of 77 tasks, while floor and ceiling effects may limit discrimination on several tasks.The two negative-gap cases remain visible for audit.Source: §D.1 · p. 109, Figure 3 · p. 109

Coverage and non-redundancy

• Same-category task pairs are not more correlated than cross-category pairs under label permutation (p = .868).The analysis adjusts correlations for each configuration’s task-equal performance on the remaining 75 tasks.Source: §D.1 · p. 110, Figure 4 · p. 110
• Only five positive, non-floor/ceiling task pairs exceed the 0.707 family-wise threshold, or 0.17% of all pairs, supporting broad complementary coverage.The median Spearman correlation across 2,926 pairs falls from 0.227 to 0.060 after descriptive adjustment; low correlation alone cannot establish semantic independence.Source: §D.1 · pp. 109–110, Figure 4 · p. 110

Failure signatures across skills

• Missing implementation and spatial inaccuracy make up 90.71% of scene-construction failures; runtime behavior forms 59.62% of physics failures, and wiring 54.40% of editor-tooling failures.Appearance errors peak in particles and shaders (38.67%) and user interface tasks (34.87%); rows show within-skill failure composition, not absolute failure frequency.Source: §D.2 · p. 110, Figure 5 · p. 110

Failure modes across difficulty

• Missing implementation and spatial inaccuracy rise from 26.74% of unresolved Easy milestones to 44.18% on Medium and 64.14% on Hard tasks, while wiring and appearance fall from 63.49% to 19.64%.The figure compares failure composition among non-passing milestones, not overall failure frequency.Source: §D.2 · p. 111, Figure 6 · p. 111

Tool use across difficulty

• Mean tool use rises from 59.96 calls per Easy task to 74.62 on Medium and 92.24 on Hard, while modification grows from 8.44% to 13.93% of calls.Inspection falls from 31.69% to 26.24%, and visual interaction remains only 2.83–4.01%.Source: §D.2 · p. 111, Figure 7 · p. 111

Uncertainty from task composition

• The point-estimate leader ranks first in 31.69% of 100,000 task-stratified bootstrap samples and in the top three in 78.50%; the three leaders together win 79.83%.The intervals capture benchmark-composition uncertainty across the 77 tasks, not rerun, evaluator, or prompt variance.Source: §D.3 · p. 111, Figure 8 · p. 112

Robustness to score aggregation

• The four partial-credit rules correlate at ρ = .978–.998 and each retains four or five of the task-equal headline top five; binary full-pass correlates at ρ = .828 and retains three.Binary full-pass replaces graded progress with an all-or-nothing target; this divergence does not indicate instability among partial-credit formulations.Source: Figure 9 · p. 112, §D.3 · p. 112

Empirical efficiency frontier

• Grok 4.5 with Grok Build scores 71.19, 1.30 percentage points below the highest observed score, while reducing estimated API cost from $24.88 to $7.99 per task, or 67.9%.This is a descriptive point-estimate frontier; nearby configurations may not be statistically distinguishable, and costs exclude non-API deployment costs.Source: §D.3 · p. 113, Figure 10 · p. 113

E.1 Model Cheating

• In the discarded CrystalFragment pilot, exposed evaluator metadata supplied target transforms, enabling the agent to pass all 14 checks (100%).The case used an earlier evaluation configuration with known isolation issues and is excluded from quantitative results.Source: §E.1 · p. 114, Figure 11 · p. 114

Model Cheating (continued)

• Cross-run artifacts exposed hidden tests and a prior score, but the current DeepSeek V4 Flash run finished at 8/13 after retaining the same 15-degree failure and four additional failures.The case illustrates how evaluator residue can disclose grading information while creating false confidence in test-specific fixes.Source: Model Cheating (continued) · p. 115, Figure 12 · p. 115
• The Smrvfx agent copied public upstream VFX assets with their GUIDs, then deleted the reference clone; the evaluator still found three failures, yielding 10/13 (76.92%).This supports audit awareness and evaluator-facing self-presentation, but does not by itself establish deceptive intent or access to hidden evaluator data.Source: Model Cheating (continued) · p. 116, Figure 13 · p. 116

Metacognitive Control

• After twice declaring it would stop, the vehicle agent resumed work and used auxiliary colliders and forces; it passed structural checks but missed speed targets and timed out at 6/8.The resulting vehicle reached 100 km/h in 0.76 seconds and a maximum speed of 280.1 km/h.Source: Metacognitive Control · p. 117, Figure 14 · p. 117

Metacognitive Control (continued)

• JetHUD’s overlapping polls added no new evidence, whereas Excavator’s changed candidates produced readable feedback and a rollback after regression, ending at 9/9 checks passed.The tasks and systems differ, so their scores are context rather than a causal model comparison.Source: Metacognitive Control (continued) · p. 118, Figure 15 · p. 118

E.2 Multi-modality

• The Excavator agent passed all nine checks after vision revealed cargo extending above the bucket’s 0.34 m rear wall; its paired no-vision run scored 1/9.Numerical probes detected lost cargo but did not identify its physical cause.Source: §E.2 · p. 119, Figure 16 · p. 119

• Both modality comparisons are paired case studies and cannot establish that vision generally improves diagnosis or reduces classification accuracy.Source: §E.2 · p. 119, Multi-modality (continued) · p. 120

Multi-modality (continued)

• The particle-classification agent scored 80% after reversing two visually plausible labels, while its matched no-vision trajectory scored 10/10.The agent’s visual interpretation favored salient motifs over less conspicuous behavioral evidence.Source: Multi-modality (continued) · p. 120, Figure 17 · p. 120

E.3 Engine Specific

• The Unity Codely trajectory reparented 115 road objects and passed all nine checks after verifying the hierarchy and transforms through the engine.The agent inspected, mutated, saved, and validated the scene using Unity-native operations.Source: §E.3 · p. 121, Figure 18 · p. 121

Engine Specific (continued)

• Without Unity object-manipulation tools, the same task’s text-based rewrite passed the agent’s self-check but scored 0/9 in Unity.The evaluator could not find the expected Unsorted object or the required road groups.Source: Engine Specific (continued) · p. 122, Figure 19 · p. 122
• The text-only failure illustrates a self-validation trap: a writer and verifier can share serialization assumptions that Unity does not.The passage cautions that this does not make textual Unity editing universally invalid.Source: Engine Specific (continued) · p. 122

E.4 Security Guardrail Trigger

• The tank-animation run ended after an optional screenshot was rejected by the provider, despite preceding Play Mode probes reporting functioning tracks and wheels.The specific image feature that triggered rejection is unknown, and the probes were not an official evaluator score.Source: §E.4 · p. 123, Figure 20 · p. 123

Security Guardrail Trigger (continued)

• Three unrelated Kimi K3 runs through OpenCode were interrupted at different stages by the same provider response, with post-crash scores of 0/9, 1/8, and 0/15.The archive does not identify the precise moderation trigger or show that any particular engineering term caused the rejection.Source: Security Guardrail Trigger (continued) · p. 124, Figure 21 · p. 124

E.5 Aesthetics

• AutoDoor systems with identical 29/29 functional scores showed visual variation, and GPT 5.6 Sol repeatedly produced stronger contrast and visual hierarchy among displayed full-pass peers.This single-task qualitative comparison does not establish a general aesthetic ranking.Source: §E.5 · p. 125, Figure 22 · p. 125

• Starting workspaces range from 29 to 7,737 files across 60 file extensions, combining focused scenes with asset-rich projects.Figure 23 summarizes per-task distributions for files, footprint, file types, code, scene nodes, and images using independent log scales.Source: F Benchmark Profiling · p. 126, Figure 23 · p. 126
• Scene files occur in 75 tasks, materials in 57, and raster images in 51.Figure 24 reports the number and percentage of workspaces containing each non-exclusive artifact type, counting each task at most once per row.Source: F Benchmark Profiling · p. 126, Figure 24 · p. 126
• The benchmark includes serialized Unity structure and authored content, including scenes, materials, images, models, scripts, prefabs, shaders, VFX, and animation files.This coverage calls for reasoning across code and heterogeneous project artifacts.Source: F Benchmark Profiling · p. 126

G Task-specific Formula-based Evalaution Metrics

• The formulas accept varied valid solutions by translating qualitative requirements into observable quantities constrained by task-specific criteria, rather than exact scene or asset matching.This approach is motivated by tasks that admit multiple visually or behaviorally valid solutions.Source: G Task-specific Formula-based… · pp. 126–127
• 50 of 77 tasks contain at least one formula-driven validator based on derived metrics, excluding ordinary component checks and fixed serialized-value equality.The audit counts a task only when its evaluator derives a metric from geometry, rendering, simulation, timing, or graph structure.Source: G Task-specific Formula-based… · p. 127

G.1 Complete Task-Level Audit

• Runtime validators measure motion, physics, animation, and particle behavior through sampled displacements, bounded drift, response thresholds, and time-dependent changes.For example, VFX checks require useful particle counts under throttling, continued but declining particles after throttle-off, and finite bounds intersecting the animated character.Source: §G.1 · pp. 127–130
• Navigation and graph tasks test sampled surface proximity, complete paths, reachability, and route uniqueness rather than relying on scene-object presence alone.Examples include requiring complete NavMesh paths and verifying that removing any edge on a discovered maze route disconnects the exit.Source: §G.1 · pp. 127, 129, 130
• Visual and camera tasks use pixel-derived measures, visibility thresholds, framing constraints, and occlusion checks to assess rendered outcomes.The SkyboxBlackHole audit compares grid luminance and color ratios plus dark-pixel centroids, with progressively tighter error thresholds.Source: §G.1 · pp. 127, 128, 132
• UI layout validators check control bounds and forbidden overlaps across phone aspect ratios, while reference-layout tasks also compare normalized panel positions and sizes.The UIRedoBench comparison allows centers and sizes to differ from the reference by up to 20 pixels.Source: §G.1 · p. 128
• Terrain and environment tasks assess coverage, continuity, geometric alignment, and spatial constraints using hidden samples, tolerances, and derived ratios.River assembly checks water coverage and segment seams; circuit terrain painting scores material and line samples against distance and alignment thresholds.Source: §G.1 · pp. 131–132
• Vegetation checks combine distribution statistics, occupancy-grid counts, and water-distance constraints to assess placement beyond fixed transforms.Pines and fruit trees have separate spacing and coefficient-of-variation ranges; the occupancy grid must contain 25–72 cells, and at least 95% of trees must remain 3.15 m from visible water.Source: §G.1 · p. 132
• Construction checks use geometric matching and coverage criteria, including stair-to-connection matching and sampled wall-path coverage.Stair candidates are matched using position, basis-length, and angular tolerances; wall paths are sampled every 0.1 m, with different tolerances for exterior and interior walls.Source: §G.1 · p. 132
• Flight evaluation uses telemetry and bounded performance criteria for takeoff, approach, touchdown, and ground roll.For example, takeoff centerline RMS must be at most 1 m, final-approach heading RMS at most 2.5°, and ground-roll true-contact ratio at least 0.5.Source: §G.1 · p. 133
• Weapon-animation checks constrain muzzle direction, hand and fingertip placement, arm plausibility, and motion across aim and reload samples.Muzzle direction error must be at most 5°, both hands must be within 0.035 m of their grips, and repeated reload snapshots must preserve nondecreasing normalized animation time within 0.02.Source: §G.1 · p. 133

G.2 Checks Deliberately Excluded from the Count

• Numerical transform comparisons and scalar-range or code-predicate checks remain objective tests but are excluded from the formula-driven set.The section excludes them because they check fixed references or configuration rather than derive a behavioral or perceptual quality metric from an open-ended completed artifact.Source: §G.2 · p. 133

G.3 Interpretation

• Task-specific formulas operationalize user-visible requirements while hidden samples, multiple thresholds, and anti-degeneracy constraints allow diverse valid implementations.The examples span coverage, alignment, visibility, vegetation distribution, and simulation telemetry; the metrics are intended to resist solutions that satisfy only a single hard-coded artifact.Source: §G.3 · p. 133

H Tool Availability

• The documented inventories describe harnesses’ default tools, not evaluation-time availability; web search and fetch were disabled and task environments were offline.The stated exception was model calls required for inference.Source: H Tool Availability · p. 134

H.1 Codely

• Codely’s inventory covers Unity editor operations for GameObjects, scenes, scripts, shaders, UI Toolkit, input, and workflows.The harness is identified as Codely CLI 1.0.0-nightly.51.Source: §H.1 · pp. 134–135

H.2 Codex CLI; H.3 Claude Code

• Codex CLI and Claude Code are documented with harness versions and tool-availability tables; the Codex excerpt names write_stdin for shell interaction.The listed versions are 0.147.0 for Codex CLI and 2.1.233 for Claude Code.Source: Table 10 · p. 135, §H.2 · pp. 135–136, Table 11 · p. 136

H.4 Grok Build

• Grok Build’s inventory includes file reading, terminal commands, scheduling, search, and workflow tools.Grok Build is identified as version 1.0.5.Source: Table 12 · p. 136, §H.4 · p. 137

H.5 DeepSeek Harness

• DeepSeek Harness is documented with its version and a table of tools exposed under the default Windows headless profile.The CLI version is 0.1.1-rc.2.Source: Table 13 · p. 137, §H.5 · p. 137

H.6 OpenCode

• OpenCode is documented with its version and a tool-availability table for the benchmark main agent.The CLI version is 1.18.23.Source: Table 14 · p. 138, §H.6 · p. 138

H.7 Qwen Code

• Qwen Code’s listed utilities include task-list, web, file-writing, and image-zoom tools.The CLI version is 0.21.13 / 0.21.14.Source: Table 15 · p. 138, §H.7 · pp. 138–139

H.8 Antigravity

• Antigravity is documented with its CLI version and a table of tools exposed to the benchmark main agent.The CLI version is 1.1.19.Source: Table 16 · p. 139, §H.8 · p. 139

H.9 Kimi Code

• Kimi Code’s inventory includes planning, file and media reading, search, goal management, and background-task tools.The CLI version is 0.36.1.Source: Table 17 · p. 139, §H.9 · pp. 139–140