PaperlayerView exact revision on arXiv

Source-linked AI summary

AnyGameRenderer: Entity-Consistent Generative Rendering Across Games and an Entity-Level Benchmark

Ruicong Liu, Liangyang Ouyang, Fengbo Lan, Zhixiang Wang, Kaipeng Zhang

arXiv:2610.31466v1cs.CV

TL;DR

Agent-made games can be playable yet still rely on programmer-art visuals. AnyGameRenderer turns crude gameplay video into styled rendering while preserving entity correspondence; on its held-out benchmark, temporal match improves over the zero-shot base.

  • Problem

    Coding agents can build playable games quickly, but their outputs often use gray boxes, colored shapes, and flat semantic blobs instead of polished visuals.

    Source: §1 · p. 1

  • Method

    AnyGameRenderer uses a video-to-video renderer with paired crude and styled clips, a caption that maps crude shapes to entities, and an optional style anchor.

    Source: §3.2 · p. 5, §4.1 · p. 6

  • Results

    Temporal match rises from 50.9% to 62.9% over the zero-shot base, while missed count changes fall from 34.7% to 21.3%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Changing the prompt or style anchor lets a game switch visual style without modeling, texturing, or animation.

    Source: §1 · p. 2

Abstract

from arXiv · show

AI coding agents can now build small playable games in minutes, but the results usually look like programmer art: gray blockouts, colored geometric primitives, and flat semantic blobs. We present AnyGameRenderer, a video-to-video generative renderer that turns such footage into high-quality visuals with characters, motion detail, and effects. Throughout, it keeps the layout, camera, and every entity's position and timing intact. Our AnyGameRenderer generalizes across a wide range of genres that agents build, such as tower defense, auto-battlers, top-down survivors, turn-based battles, kart racing, and crowd runners. AnyGameRenderer takes plain video as its game-state input, with no complex representations such as engine G-buffers, per-object class labels, or low-poly proxies. Gray blockouts, colored primitives, semantic maps, and wireframes all work as input. It also preserves entity correspondence: group counts stay correct, and entities appear and disappear on the right frame. We implement AnyGameRenderer by LoRA fine-tuning MiniMax-H3 ref2va, a 33B reference-video-conditioned video generation model. Training uses $\sim$10k paired clips from three sources: (i) multi-stream rendering data from 38 agent-made games exported from GEAR, an AI game-creation tool; (ii) two-stream rendering data from Unreal Engine, with programmatically controlled crowds, count changes, and entity scales; and (iii) real gameplay recordings, with pairs reconstructed via 3D-box detection. To evaluate the entity correspondence, we propose an entity-level benchmark of 59 held-out cases with 753 entities, scored against ground-truth boxes. Compared with the base model, AnyGameRenderer better preserves entity correspondence: temporal match rises from 50.9\% to 62.9\%, missed count changes fall from 34.7\% to 21.3\%, and fewer entities are hallucinated or left unrendered.

1 Introduction

AnyGameRenderer turns crude gameplay video into styled visuals while preserving the input’s layout, camera, and entity timing. It targets the genres and input formats of agent-made games, and improves entity correspondence over a zero-shot base model.

1 Introduction

  • It renders the same gameplay in a prompted style while preserving layout, camera, and every entity’s position and timing.The input can be paired with a text prompt and optionally a style anchor image.Source: §1 · pp. 1–2
  • A single checkpoint generalizes across agent-made genres such as tower defense, auto-battlers, turn-based battles, kart racing, and crowd runners.Other examples include top-down survivors, arena survival, co-op games, and shooters.Source: §1 · p. 3
  • Group counts stay correct and entities appear or disappear on the right frame, addressing failures such as unrendered blocks and characters absent from the input appearing in the output.The paper also notes that fewer small entities are lost.Source: §1 · p. 3
  • The renderer accepts plain video—including blockouts, colored primitives, semantic maps, wireframes, and coarse figures—instead of requiring complex intermediate representations.The paper contrasts this with systems that require class labels, state-augmented 3D boxes, or low-poly engine proxies.Source: §1 · p. 3
  • Training pairs crude and styled videos from GEAR games, Unreal Engine scenarios, and real gameplay recordings, yielding 10,330 clips.The Unreal Engine data includes scripted count changes and entity sizes, while real-game inputs are reconstructed with 3D-box detection.Source: §1 · p. 3
  • Temporal match rises from 50.9% to 62.9%, while position match rises from 57.1% to 61.9% and unrendered and hallucination rates fall from 4.1% to 2.9% and 52.1% to 46.0%.These comparisons are against the zero-shot base on an entity-level benchmark of 59 held-out cases with 753 entities.Source: §1 · p. 3

2 Related Work

Related work spans AI-driven game creation, interactive world models, and generative renderers that keep the engine in charge of game state. AnyGameRenderer differs through genre coverage, plain-video inputs, and an entity-level focus on correspondence.

2.1 AI-driven game creation

  • Coding-agent game systems produce working game logic, but their visuals are usually simple shapes and placeholder assets rather than commercial-quality art.GEAR turns a short prompt into a playable game in minutes; other approaches divide development into collaborating roles or iterate designer, builder, player, and reviewer roles.Source: §2.1 · p. 4

2.2 Interactive world models

  • Interactive world models generate frames from player actions, but exact entity counts, scripted events, and rules that hold for minutes remain difficult to control.Long-horizon memory reduces drift but does not make the world programmable.Source: §2.2 · p. 4

2.3 Generative rendering for games

  • Generative renderers keep the engine in charge of game state, but prior systems often require engine-exact inputs or fixed representations; AnyGameRenderer instead targets broad genres, plain video, and entity-level correspondence.The paper argues that frame-level metrics can overlook missing small entities or late-arriving groups.Source: §2.3 · p. 4

3 Method

AnyGameRenderer treats game rendering as reference-conditioned video editing: crude gameplay video fixes the structure, while a caption and optional anchor guide its styled rendering. Its caption schema specifies what must correspond, and its training recipe uses paired data and LoRA fine-tuning.

3.1 Task setting

  • The renderer edits a crude gameplay clip into the target style while matching its length, resolution, camera, layout, timing, entity positions, sizes, headings, and appearances.The input is a plain reference video; the prompt describes the target style and what each crude entity represents.Source: §3.1 · p. 4, §3 · p. 4
  • The crude clip fixes structure, while a six-segment caption and optional anchor image specify style; changing the prompt or anchor allows a new style.During training, the anchor is the target’s first frame; at inference it is a style reference or can be omitted.Source: §3.1 · p. 5, Figure 2 · p. 5

3.2 Caption schema: the correspondence contract

  • The correspondence contract names real entities and their crude representations, declares position, size, timing, and head count as preserved, and uses positive captions with deliberate noise.Captions are generated from engine metadata and detection results and may list characters absent from a clip, teaching the model to infer existence from video.Source: §3.2 · pp. 5–6

3.3 Training recipe

  • Training resamples sources to 124-frame clips at 24 fps and fits a rank-128 LoRA on all transformer blocks; three epochs over 10,330 clips take about 48 hours.For unseen text-only styles without an anchor, the paper recommends a LoRA multiplier of 0.7 to recover zero-shot visual quality while keeping correspondence gains.Source: §3.3 · p. 6

4 Data

The training corpus pairs crude and styled videos from agent-made games, controlled Unreal scenes, and reconstructed real gameplay. Its sources target genre and input diversity, entity correspondence, and realistic scenes.

4.1 GEAR games

  • GEAR contributes 3,221 clips from 38 agent-made games, with five aligned passes; training pairs the primitive pass with the target render.The passes also include coarse, semantic-map, and depth renderings, plus an anchor image, caption, and 3D boxes.Source: §4.1 · p. 6

4.2 Unreal Engine

  • Unreal scenes are scripted around zero-shot correspondence failures, including same-color groups of 2–20 that spawn, leave, or despawn during clips.Scenes also vary character entry and exit timing across character types.Source: §4.2 · pp. 6–7
  • Unreal cases vary entity size from 20–600 pixels and camera views (35% follow, 30% top-down, 20% isometric, 15% side); clips require a visible protagonist in at least 95% of frames.Source: §4.2 · p. 7

4.3 Real gameplay recordings

  • Real-gameplay pairs use detected 3D boxes rendered as varied proxies over checkerboard ground, with the original recording as target; this source contributes 3,199 clips.It adds realistic camera work and vehicle and pedestrian motion.Source: §4.3 · p. 7

4 Data

  • The dataset combines three paired-video sources into 10,330 124-frame clips (about 14.8 hours), with crude and styled views frame-aligned under the same camera and layout.Source: §4 · p. 6, §4.3 · p. 7

5 Experiments

Experiments evaluate entity correspondence on held-out cases and test generalization across genres. AnyGameRenderer improves the reported entity-sync metrics, though gains vary by data source and genre.

5.1 Evaluation protocol

  • The benchmark contains 59 held-out cases and 753 annotated entities from Unreal Engine, real recordings, and GEAR; evaluation tracks entity timing, position, unrendered inputs, and hallucinations.Input entities are matched to generated-video tracks using segmentation, tracking, and a Hungarian matching procedure; entities present for fewer than 12 frames are ignored.Source: §5.1 · pp. 7–8

5.2 Entity correspondence

  • Temporal match rises from 50.9% to 62.9%, position match from 57.1% to 61.9%, unrendered rate falls from 4.1% to 2.9%, and hallucination rate falls from 52.1% to 46.0%.On-time appearance rises from 55.9% to 67.6%, entities that never appear fall from 26.0% to 20.2%, and missed count changes fall from 34.7% to 21.3%.Source: Table 2 · p. 8, §5.2 · p. 9
  • In seven held-out cases, AnyGameRenderer fixes unrendered crude shapes, viewpoint changes, and caption-named characters absent from the input.The comparison uses the same seed, steps, and caption; the GT column in rows 5–6 is an in-domain layout reference.Source: Figure 4 · p. 9, §5.2 · p. 9
  • Gains differ by source: Unreal Engine temporal match reaches 96.6% against a 98.5% GT floor, while real recordings improve from 46.5% to 51.1%.No entity is missed in the Unreal cases; real-recording missed count changes stay at 20.0%.Source: §5.2 · p. 9

5.3 Generalization across genres

  • A single checkpoint improves temporal match in all five benchmark genres, with the largest gains on crowd and third-person games; top-down remains hardest.Unrendered rate falls in every genre except crowd, position match improves most on vehicle games, and hallucination rate rises slightly on top-down and third-person games.Source: §5.3 · p. 9, Figure 5 · p. 10

6 Conclusion

AnyGameRenderer is a single-LoRA video renderer trained on paired data from GEAR, Unreal Engine, and real gameplay recordings. On a held-out benchmark, it improves entity timing and count-change correspondence while handling diverse input video and game genres.

6 Conclusion

  • Temporal match rises from 50.9% to 62.9%, and missed count changes fall from 34.7% to 21.3% on the held-out benchmark.The checkpoint handles a wide range of genres and keeps entities in sync with the input.Source: §6 · p. 10
  • The renderer turns programmer art into high-quality visuals using a single LoRA trained on paired GEAR, Unreal Engine, and real-gameplay data.The paired data combines GEAR multi-stream rendering, Unreal Engine two-stream rendering, and real gameplay recordings.Source: §6 · p. 10
  • It accepts plain video inputs ranging from gray blockouts and colored primitives to semantic maps and wireframes.The conclusion reports that the resulting checkpoint handles a wide range of genres.Source: §6 · p. 10

Related papers

Loading 2610.31466v1…