ArtifactArena: Evaluating Models by What They Build in the Physical World

• Static question-answer benchmarks are effectively saturated, while functional-code tests retain predefined objectives and symbolic arenas lack an objective physical verifier.Source: §1 · p. 2
• In MuJoCo, models build complete robot bodies and controllers to compete under simple rules, requiring joint reasoning about geometry, materials, mass distribution, actuation, control, and strategy.Source: §1 · pp. 2–3
• Frontier models zero-shot co-design complex robot morphologies and controllers; verifier feedback further enhances this capability, while open-ended agentic search lags behind refinement.Source: Figure 2 · p. 3, §1 · p. 4
• Three compute-matched harnesses compare artifact-generation strategies, with Elo rankings measuring competitive physical design and control in simulation, not direct real-world robotic capability.Source: §1 · p. 4
• Observed designs include wedge, hammer, and pusher strategies; strong agents create probes and adversarial opponents, but self-tests can diverge from tournament performance.Source: §1 · p. 4
• The released simulator, harnesses, and tournament infrastructure support community submissions and a continuing evaluation horizon.Source: §1 · pp. 3–4

2.1 The Last Bot Standing Game

• Robots compete head-to-head on a circular platform, with wins determined by pushing an opponent off or pinning it under arena rules.Qualification includes static checks and three matches against a passive block; an artifact that loses the best of three forfeits the tournament.Source: §2.1 · p. 6

2.2 Robot Artifact Design Space

• Robot morphologies use constrained MJCF kinematic trees, while controllers issue actuator commands every 0.01 seconds from observations including positions, contacts, and history.Hardware may use up to 2,000 bodies, geometries, and actuators; persistent controller state resets between matches.Source: §2.2 · pp. 6–7

2 Methods

• The evaluation separates artifact construction from performance ranking, holding the simulator, physical parameters, and game rules fixed across evaluations.Artifacts combine morphology, controller, and design rationale, encoding coupled design decisions.Source: §2 · p. 4, Figure 4 · p. 7

2.3 Test-time generation strategies

• The three harnesses vary model control over the design loop: Sampling produces independent artifacts, VGH uses iterative feedback, and DLH provides a sandbox for engineering.Each harness receives a budget of 10 API calls; VGH and DLH use round-robin selection to choose a final artifact.Source: §2.3 · pp. 7–8

2.4 Tournament

• The tournament selects three qualified artifacts per model–harness pair for a cross-model round robin, then scores each model’s strongest artifact in a separate Champion Round.Bradley–Terry Elo ratings are used; the Champion Round uses 11 unique seeds and 22 matches.Source: §2.4 · p. 8
• Champion rankings were robust across disjoint seed sets: Spearman correlation was ρ = 0.984, with 201 of 210 model orderings preserved and the top three unchanged.The reported bootstrap intervals are a lower bound on full-pipeline uncertainty because within-run finalist selection is not resampled.Source: §2.4 · p. 8

3.1 ArtifactArena Leaderboard

• VGH achieves the highest reported Elo for 15 of 21 models, and GPT-6 Astra’s VGH artifact wins 91% of cross-model matches among top artifacts.The three highest Elo positions belong to GPT-6 Astra (1539), Claude Fable 5.1 (1310), and Gemini 3.8 Flash (1223).Source: §3.1 · p. 9
• Under the same API-call budget, DLH trails VGH for most models and yields no top-seven artifact, while no harness leads for every model.VGH has the highest Elo for 15 models, but SH is strongest for four named models and DLH for two.Source: §3.1 · pp. 9–10

3.2 Emergent Creativity in ArtifactArena

• Models co-design varied hardware and controllers, including powered scoops or plows in 208 of 509 sampled robots and distinct behavioral modes in 63% of refinement-run controllers.The hardware examples include flippers, spinners or drums, clamps, and movable weights; controller modes respond to opponents and arena rules.Source: §3.2 · p. 11
• Among 5,250 zero-shot samples, 4,384 artifacts are built and 2,242 qualify, with qualification rates ranging from 2% to 91% across named models.The samples span 21 models; 29% of robots include a custom mesh, and some use tendons, passive springs, ball joints, or slide joints.Source: §3.2 · p. 11

3.3 Remarkable capabilities that surprised us

• Controllers anticipate being flipped, estimate physical conditions, and exploit game rules, while Design Lab runs alternate building with probes against a model’s own design.Examples include reversing drive or weapons when inverted, inferring wheel load from contact impulses, and testing a wedge against probe opponents.Source: §3.3 · pp. 11–12

3.4 What makes some models creative?

• Top models decompose broad goals into strategies and verifiable subgoals, such as lifting an opponent before pushing it toward the edge.Claude Fable 5.1 translates the strategy into controller state machines that adjust lift angle and limit speed near the edge.Source: §3.4 · p. 13
• Models can retain a strategy while changing its physical implementation: Kimi K3 replaced skid-steer slabs with a center-pivot wedge and improved qualification to 3-0.Its first five implementations won two of fifteen qualification rounds; iteration 6 changed the morphology and controller.Source: §3.4 · p. 13

3.5 Limitations

• ArtifactArena’s main limitation is the simulation-to-reality gap: success in MuJoCo measures simulated physical design and control, not direct real-world robotic capability.The authors state that constructing a robot in the simulator does not mean it will work in the real world.Source: §3.5 · pp. 13–14

Prescribed objectives and executable verification

• ArtifactArena extends executable competition to physically simulated robots whose morphology and controller are both designed by models.Related benchmarks include prescribed task objectives, pairwise human preferences, and multi-round software engineering or competing codebases.Source: §4 · p. 14

Physical reasoning and functional construction

• Unlike construction benchmarks for transport, support, lift, or standardized machine programs, ArtifactArena evaluates functioning robots through direct physical competition.The related construction benchmarks cited are BuildArena and BesiegeField.Source: §4 · p. 14

Co-design and evolving competition

• Prior work co-optimizes bodies and behavior or evolves environments and solutions together; ArtifactArena’s new opponents expose weaknesses and raise the bar for submissions.The passage cites competitive evolution, Evolution Gym, RoboMoRe, competitive reinforcement learning, and Enhanced POET.Source: §4 · p. 14

5 AI Statement

• Generative AI assisted with methods implementation, coding, experiments, analysis, interpretation, figure creation, and grammar correction.The authors state they reviewed AI-assisted contributions and checked code, analyses, and figures against outputs.Source: §5 · p. 18
• The evaluated models generated a synthetic dataset of robot morphologies and controllers, but AI tools did not generate the core research ideas or substantive manuscript content.The authors take responsibility for the final text, methods, results, claims, and AI-assisted artifacts.Source: §5 · p. 18

6.1 Fresh-seed robustness

• Spearman ρ = 0.984 (95% bootstrap interval [0.948, 0.990]); 201 of 210 pairwise model orderings were preserved across seed sets, and the top three models remained unchanged.Seven orderings reversed and two became ties.Source: §6.1 · p. 18

6.2 Ambitious Designs

• Several ambitious designs never won a match, including a 188 kg drum spinner that backflipped its own chassis and a self-righting pendulum design.Other attempts included a 0.68 m aluminum flywheel and nine of GPT-6 Astra’s sixteen sphere-drive lifters.Source: §6.2 · p. 18

6.3 Artifact Morphology Analysis

• Among powered mechanisms, 208 sampled robots had scoops or plows, 31 had flippers, 21 had spinners or drums, ten had clamps, and three had movable weights.Ten robots carried weapons at both ends.Source: §6.3 · p. 19
• Some attacks combine lifting and pushing: Gravemouth Leviathan pairs a wedge with an intake drum, while OblivionScoop adds a powered jaw to its scoop.Other robots use front and rear scoops to attack from either direction.Source: §6.3 · p. 19
• Movable weights serve different tactics: Shiftplow shifts ballast for pushing or escape, while Wedgewarden Atlas raises its weight to target the inactivity tie-break.Centerline Kraken moves a 553 kg core along two axes for what its rationale calls “controllable leverage.”Source: §6.3 · p. 19
• Defensive and recovery designs include skirts and skids, exposed wheels intended to work when inverted, a dome intended to shed climbers, and a flipper for attack and self-righting.These are described design intentions, not evidence that each mechanism succeeded in matches.Source: §6.3 · p. 19
• During refinement, robots with skirts increased from 29% to 51% and those with domed roofs from 3% to 7%; Groundhog added a second ramp and a 320 kg swinging weight.Every revision from the fourth onward qualified 3–0, and revision nine was selected with a 38–16 record in the run’s selection round robin.Source: §6.3 · p. 19
• Of 16 sampled GPT-6 Astra sphere-drive lifters, nine failed qualification, mostly due to size violations or inactivity.Omnidirectional Traction Thief exceeded the allowed dimensions during movement and lost all three qualification rounds.Source: §6.3 · p. 20

Decision structure

• In refinement runs, 63% of controllers use distinct behavior modes with explicit switching conditions; the controller corpus contains 209 distinct mode names.Examples range from chase and flank to holdout and activity_win.Source: §6.4 · p. 20

Responding to the opponent

• Controllers inspect opponent hardware and behavior to choose an approach; GPT-6 Astra checks for hinges above 1,800 N·m and calculates interception points.GLM-5.3 classifies opponents and, against an unreachable flying opponent, patrols the center while relying on its inactivity timer.Source: §6.4 · p. 20

Using the game rules

• Some controllers use inactivity rules strategically: Claude Fable 5.1 waits or retreats based on who stopped first, while Kimi K3 maintains a pin when the opponent’s timer expires first.Movement timing can make holding position preferable to releasing an opponent or continuing to push.Source: §6.4 · p. 20

Controlling grip and avoiding the edge

• Controllers account for physical limits by limiting motor torque to available wheel grip, calculating edge clearance from the robot’s outline, or braking near the edge.Kimi K3 aims to let the opponent’s momentum carry it off while keeping its own robot on the platform.Source: §6.4 · p. 20

Responding to inversion

• Some controllers change commands when overturned: Gemini 3.1 Pro reverses its drum, while GPT-5.6 Sol reverses wheel commands.These behaviors complement hardware intended to remain useful in more than one orientation.Source: §6.4 · p. 20

Elo calculation

• A 400-point Elo gap corresponds to 10:1 odds in the Bradley–Terry win-probability model.The probability uses the logistic function applied to the difference in BT log-strengths.Source: §6.5 · p. 21
• Each game is scored as a win (1), draw (1/2), or loss (0), with both starting sides included.Source: §6.5 · p. 21
• Because BT strengths are identifiable only up to an additive constant, ratings use an independent prior centered at 1000, fixing each fit’s mean rating at 1000.Source: §6.5 · p. 21

6.6 Tournament results

• Figure 8 presents champion-round head-to-head scores, with rows against columns and red or blue indicating advantage or disadvantage; compare all-harness matchups with within-harness panels.Source: §6.6 · p. 22, Figure 8 · p. 22

6.7 Dichotomous Key

• Figure 9 is identified as a dichotomous key of SH-generated bots.Source: Figure 9 · p. 23

6.8 Physical implementation across revisions

• Revision examples show hardware changes addressing observed problems: Kimi K3 repositioned drive weight to improve pivoting, and Claude Fable 5.1 repositioned wheels and ramp after size violations.GPT-5.4 replaced a fixed tungsten slab with sliding ballast to shift mass for pushing or edge recovery.Source: §6.8 · p. 24
• Of 216 built robots that did not qualify, 125 drove themselves off the platform, 60 triggered the inactivity rule, and 29 exceeded size limits during motion.Source: §6.8 · p. 24

7.1 Detailed Robot Design Constraints

• Robot structure, mesh geometry, materials, and initial-pose size are constrained; exceeding the size box during a match causes a loss.The root must have exactly one free joint, and every jointed body needs geometry; meshes must be defined in submitted XML and meet mesh validity requirements.Source: §7.1 · pp. 24–25
• Only motor actuators are allowed, while internal passive elements and arena-controlled contact settings limit how robots can generate motion and traction.Internal joint and tendon damping and friction loss are capped, and spring stiffness is unbounded subject to unloaded initial pose; material friction governs traction.Source: §7.1 · pp. 24–25
• Submissions must pass MuJoCo and controller checks, and controller failures or malformed returns forfeit the tournament.Controllers run in a restricted Python sandbox; a controller exception counts as a loss.Source: §7.1 · p. 25

7.2 Design Lab Harness Details

• Each Design Lab turn rebuilds the prompt from durable state, while a fresh isolated workspace limits access to the stationary block as the built-in opponent.Models can chain tool calls within the default 10-turn budget; the supplied tool groups include workspace, arena, analysis, and library.Source: §7.2 · p. 25
• The run’s artifact is selected from saved bots through a three-seed-per-side round robin ranked by Bradley–Terry strength.The selected artifact need not be the final draft; unsaved drafts and manual submissions cannot override the selection.Source: §7.2 · p. 25

7.3 Material Library

• During validation, geometry mass is computed from volume and material density, and geometry friction is overwritten by the material’s coefficient.Invalid or omitted material assignments default to foam; user-specified mass and density do not determine final physics.Source: §7.3 · p. 26

• Simulation experiments use a server with 446 CPU cores.Source: §8 · p. 26
• LLMs are accessed through API calls to third-party model providers.Source: §8 · p. 26

• The evaluation includes 21 LLMs across nine families.Source: §9 · p. 26
• Each model’s YAML configuration specifies its API identifier, sampling parameters, and reasoning or thinking flags.Source: §9 · p. 26
• Configurations are loaded at runtime and forwarded through LiteLLM, while DSPy is used for prompts.Source: §9 · p. 26

10 Simulation & Harness Code

• The browser runs live matches with the same C physics engine and Python match code used on the cluster.For recorded matches, it downloads stored trajectories and uses forward kinematics to recover body poses for rendering.Source: §10 · p. 26
• Browser and native matches are checked for model, state, trajectory, repeatability, winner, and termination agreement, but full-match bit-for-bit equality is not achievable across compilers.The paper therefore defines parity as outcome agreement plus bounded trajectory divergence, and labels site replays as reconstructions.Source: §10 · p. 27
• Stored games are replayed by recompiling the composed model and rendering poses recovered from each saved generalized-coordinate frame.Source: §10 · p. 27
• Users can play released artifacts under tournament rules and build bots in a 3D editor that recompiles each edit into a live, physically settled preview.The editor links XML parts to the 3D view and supports edit-history navigation and file downloads.Source: §10 · p. 27
• The release includes the harness, verifier, WebAssembly build materials, Pyodide bundle, parity suite, and web viewer.Source: §10 · p. 27

11 Model Prompt

• Models must design robots to maximize win rate and return mutually consistent, self-contained MJCF XML and Python controllers; the rules define the environment but not the robot design.The prompt asks for concise explanations, justifications, and calculations alongside the implemented artifacts.Source: §11 · p. 28
• Robots face round-robin 1v1 matches on a 15 m circular platform, lose by falling, inactivity, controller errors, size violations, or instability, and are ranked by Bradley-Terry ratings.Matches last up to 300 seconds; draws count as half a win, and starting positions are randomized.Source: §11 · pp. 28–29
• The MuJoCo environment uses 9.81 m/s^2 gravity, calls controllers every 0.01 seconds, and enforces full gravity with no gravcomp.The root body needs a freejoint, and the simulator runs 40 physics steps between controller calls.Source: §11 · p. 28
• The requested planning sections explain the robot concept, hardware and controller choices, risks, mass and dimensions, actuator sizing, and links between mechanisms and strategies.Models must also explain controller observations, actuator commands, handling of starting positions and opponents, edge proximity, and recovery.Source: §11 · p. 28
• Robot designs must satisfy a 25–800 kg total-mass range and remain within 2.44 x 2.44 x 3.05 m during matches.The prompt requires geometry and motor mass accounting, and the size limit is checked in the robot's root-body frame at every control step.Source: §11 · pp. 28, 30
• The controller must implement `policy_step(obs) -> dict[str, float]`, with state, including globals, persisting between calls within a match and starting fresh for every qualification and tournament match.Source: §11 · pp. 29, 41
• Qualification requires avoiding loss in three 20-second rounds against a randomized, motorless 342 kg block; failure to qualify forfeits the tournament.The block is a 1.2 x 1.2 x 0.25 m slab, can translate, rotate, and topple, and is exempt from inactivity loss.Source: §11 · p. 29
• Geoms attach rigidly to bodies for visualization, collision, and mass calculations; supported types exclude mesh, and every geom needs a unique name and palette material.Materials determine density and sliding friction.Source: §11 · p. 32
• Primitive geom sizes specify radii or half-lengths, and geom mass is volume multiplied by material density.For example, a box with size="1 1 1" measures 2 x 2 x 2 m.Source: §11 · p. 32
• Meshes are defined inline with vertices and triangular faces; closed, outward-facing surfaces can supply SDF collision geometry that preserves concave features.SDF dimensions use mesh scale, not geom size; mesh assets add no mass, while each geom using one derives mass from enclosed volume and material density.Source: §11 · p. 32
• Geom orientation is relative to its parent body and can use Euler rotations or xyaxes; geom orientation and joint axis are independent and must be aligned for intended motion.For cylinders and capsules, the local Z-axis passes through the centers of their ends.Source: §11 · p. 33
• Child bodies are welded to parents without joints; hinge and slide joints set motion axes and may be limited by ranges, while a free joint is reserved for the robot root.Hinge limits are radians and slide limits are meters.Source: §11 · pp. 33–34
• Internal joints and tendons can use damping and friction loss for passive resistance and springs for restoring forces, without additional mass cost.Spring stiffness must be finite, and springs must start unloaded in the XML initial pose.Source: §11 · p. 34
• Only named motor actuators are allowed, and they must drive parts within the robot; commands range from -1 to 1, while higher absolute gear increases force or torque and adds motor mass.Motors have no arena-imposed speed, power, energy, or thermal limits.Source: §11 · pp. 34–35
• Tendons transmit force through spatial cables routed via robot sites or fixed joint-coordinate combinations; they must stay within the robot, and sites and tendons add no mass or collision geometry.Spatial tendons can wrap around sphere or cylinder geoms; physical hardware must be represented with material geoms.Source: §11 · p. 35
• Assembly guidance defines body positions relative to parents and explains that parent motion carries child bodies along, while motors may drive spatial tendons.Tendon actuation applies gear times motor command to tendon length coordinates; negative force shortens a spatial tendon.Source: §11 · pp. 35–36
• Robot geoms must use arena-provided materials, whose density determines geometry mass and whose friction affects physics; explicit mass and custom or external assets are disallowed.Inline mesh definitions referenced by SDF geoms may be included and must be preserved.Source: §11 · p. 36
• The controller receives `obs` as a Python dictionary with documented top-level keys, types, shapes, units, and meanings; all listed keys are present on every call.The `fields` and `controller_environment` labels organize the specification but are not keys in `obs`.Source: §11 · p. 37
• Additional observations provide match timing, static robot and arena properties, occupancy and edge-distance grids, and rolling observation and action histories.The arena grid uses static bounding-radius circles, while history queues retain up to N observations or actions and are empty at the start.Source: §11 · pp. 38–39
• Detailed physical-state records include robot mass and center-of-mass data, opponent proximity, current contacts, and accumulated contact impulses.Positions and velocities use world XYZ coordinates unless marked local; values are detached snapshots, while definitions stay constant within a match and motion updates each call.Source: §11 · pp. 39–40
• The controller environment preloads a limited set of Python libraries and forbids imports beyond them, file or process access, and specified dynamic-execution and introspection functions.Forbidden examples include `open`, `exec`, `eval`, `os`, `subprocess`, and `socket`.Source: §11 · pp. 39–40
• The prompt defines structured records for robot components and physical feedback, and gives a minimal controller example that returns bounded commands for every actuator.The records include body and geom state, motor transmission and control limits, and contact forces and torques; the example is syntax-only, not a navigation or combat policy.Source: §11 · pp. 40–41

DESIGN-SPACE EXPLORATION

• The prompt makes tournament win rate—not simplicity, familiarity, or implementation safety—the design objective, while urging calculated risks and disciplined use of complexity.A reliable bot that draws or loses to stronger designs is described as a failed design.Source: §11 · pp. 41–42
• The prompt encourages purpose-built morphologies, including custom SDF geometry and intentional surfaces, when they can improve attack, movement, defense, stability, or recovery.It calls for exploring diverse body organizations, locomotion, support geometry, articulation, and force application before choosing an architecture.Source: §11 · pp. 41–42
• Candidates should be assessed using physics and observations for feasibility, failure modes, mass distribution, actuator forces, contact mechanics, stability, spawn robustness, and inactivity-rule constraints.The prompt says to compare designs by expected tournament win rate and accept calculated risk when potential advantage is substantial.Source: §11 · p. 42

CONTROLLER-SPACE EXPLORATION

• Controller exploration should compare ways to use observations, history, and actuator feedback for estimation, planning, adaptation, recovery, and opponent response, then stress-test diverse match scenarios.The prompt explicitly allows sophisticated or stateful controllers when they offer a credible advantage.Source: §11 · p. 42

11.1 Observation Schema

• The observation schema includes robot and opponent positions, orientations, and velocities, with angular-velocity conventions specified in the prompt.Positions use world coordinates; angular velocities are expressed about each robot's root-body local axes.Source: §11.1 · p. 43
• The schema reports actuator velocities keyed by motor name, with definitions that vary by transmission type, including hinge, slide, tendon, ball, slider-crank, and relative-site motors.Opponent actuator velocities use the same conventions.Source: §11.1 · p. 43
• Observations also provide robot-to-opponent distance, signed edge distances, opponent contact and force, and ground-contact status.Edge distance is positive inside the platform boundary and negative outside; ground contact includes platform sides.Source: §11.1 · p. 43
• The match timestep t counts completed control steps from 0 at the first policy_step call; tipping stability is normalized tilt, and inactivity loss triggers at 10 seconds.is_tipping is the angle in radians between the root body’s +Z axis and world +Z, divided by pi/2 and capped at 1 (0 upright; 1 tilted 90 degrees or more). The inactivity timer measures the longest continuous period ending now during which every pair of recorded center-of-mass positions is less than 0.5 m apart in XYZ; the opponent timer is measured likewise and is always 0 for the qualification block.Source: §11.1 · pp. 43–44
• The schema supplies top-down arena occupancy and mass grids, a signed edge-distance grid, and static robot bounding radii.The occupancy grids use circular masks around robot centers of mass, and the arena grid marks outside, free, opponent, and self cells.Source: §11.1 · p. 44
• Rolling histories retain up to 20 observations and actions by default, while action history records commands before motor-specific control-range limits are applied.Observation history includes positions, velocities, distances, yaw, edge distances, timestep, actuator velocity, and game context.Source: §11.1 · p. 44
• The prompt exposes detailed physical state, including robot masses and center-of-mass velocities, opponent proximity, solver contacts, and accumulated contact impulses.Opponent proximity returns geom pairs within 2 meters, with a maximum of 32 pairs.Source: §11.1 · pp. 44–45
• The controller environment preloads math, NumPy, typing, random, and collections, while forbidding file and process access and several dynamic-execution builtins.All Python builtins are available except the listed forbidden functions, including open, exec, eval, and compile.Source: §11.1 · p. 45
• The Design Lab Harness provides cheap static verification, simulation-based qualification against the block, and run_match verification without requiring qualification.The latter allows two unqualified bots to fight.Source: Table 1 · p. 46
• The validation pipeline’s material palette uses density to determine geometry mass from volume and friction coefficient to determine sliding contact friction.Source: Table 2 · p. 47
• The cross-model tournament table reports evaluated models, their reasoning or thinking configurations, and the per-call output budgets used.The listed output budgets match each provider’s documented maximum for that model.Source: Table 3 · p. 47