PaperlayerView exact revision on arXiv

Source-linked AI summary

Advances in Agentic Computer Graphics

Hengyu Liu, Jihyeon Je, Ruiqi Wang, Mingxuan Wu, Haven Feng, Qixing Huang, Georgios Pavlakos, Hao Zhang, Jiajun Wu, Maneesh Agrawala, Leonidas Guibas, R. Kenny Jones, Shangzhe Wu, Guandao Yang

arXiv:2611.01860v1cs.CV

TL;DR

Rapidly growing agentic graphics research and inconsistent terminology make systematic comparison difficult. The report introduces a shared taxonomy and an agentic literature system to analyze papers at scale. Across tasks, it finds recurring patterns in planning, verification, and refinement, alongside diverse harness designs and predominantly off-the-shelf base models.

  • Problem

    Rapid growth and inconsistent terminology across agentic graphics papers make it difficult to synthesize methods, compare designs, and assess broader research trends.

    Source: §1 · p. 2

  • Method

    The report defines a four-dimension taxonomy and builds an agentic system to discover papers, annotate designs, and retain structured records with supporting evidence.

    Source: §1 · p. 2, §4.1 · p. 9

  • Results

    Across tasks, systems show recurring patterns in planning, verification, and iterative refinement; most use off-the-shelf base models, while harness designs are more diverse.

    Source: §10 · p. 21

  • Takeaways & Limitations

    The system supports systematic analysis of existing work and emerging trends by applying the taxonomy to a growing literature and retaining records for later updates.

    Source: §1 · p. 2, §4.2.2 · p. 10

    Automated checks detect missing or inconsistent entries but do not verify whether taxonomy classifications are correct.

    Source: §4.1 · p. 9

Abstract

from arXiv · show

Computer graphics is built on code. Large language model (LLM) agents can now generate graphics code and iteratively refine it through various feedback. This capability opens the door to a potentially new paradigm of computer graphics, with graphics algorithms increasingly built around agentic systems to automate progressively complex tasks. Over the past years, research on agentic graphics has expanded rapidly across a diverse range of tasks. This rapid growth makes it increasingly difficult for manual literature reviews to keep pace with new developments. At the same time, similar ideas are often described using different terminology across these tasks, further complicating efforts to develop a systematic understanding of the field. In this report, we address these challenges through two contributions. First, we introduce a unified taxonomy for characterizing agentic systems across diverse computer graphics tasks. Second, we develop an agentic literature system that can assist human researchers in analyzing existing and newly published papers at scale. Building on these contributions, we analyze emerging research trends and discuss open challenges in coordinating agents and tools across graphics tasks, designing interfaces and representations for human-agent collaboration, sustaining long-horizon planning and refinement, and reliably evaluating interactive graphics systems.

1. Introduction

LLM agents can act in graphics environments, inspect results, and use feedback to refine graphics tasks. Rapid growth and inconsistent terminology make it harder to synthesize the field, motivating a unified taxonomy and an agentic literature survey system.

1. Introduction

  • Agentic graphics systems execute operations in graphics environments, inspect results, and use feedback to guide subsequent actions.Demonstrated tasks include 3D asset creation, iterative editing, articulated animation, and dynamic scene construction.Source: §1 · pp. 1–2
  • The literature is growing rapidly, while functionally similar components are described with different terms, complicating comparisons and assessment of field-wide progress.These challenges motivate a unified taxonomy and an agentic literature system for analyzing existing and new work.Source: §1 · p. 2
  • The report introduces a four-dimension taxonomy—specification, base model, harness, and graphics interface and tooling—and a literature system that annotates papers with it.The system records design labels, insights, limitations, and supporting evidence, and retains records for later review and updates.Source: §1 · p. 2

2. Background and Scope

The report scopes agentic graphics to LLM-based systems that act in graphics environments and use feedback to select or revise actions. It organizes the literature around a shared design space and distinguishes agentic workflows from single-pass generation and other out-of-scope methods.

2. Background and Scope

  • Agentic systems are defined by LLM action decisions, graphics-environment operations, and feedback that influences subsequent actions.The scope imposes no restrictions on representation, platform, training strategy, or automation level, and includes diverse tools, feedback sources, and 2D and 3D artifacts.Source: §2 · p. 2
  • The report excludes methods without a tool-using learned model, single-pass LLM generation, and feedback used only for training or final evaluation.Predefined stages may still qualify when the model uses prior-stage results to choose actions.Source: §2 · p. 2
  • The survey uses a shared design space to compare system choices and how representations and operations shape construction, inspection, and revision across graphics tasks.Application sections examine visual, geometric, and physical requirements.Source: §2 · p. 3

2.1. Background

  • Traditional graphics workflows combine modeling, rendering or state inspection, and revisions using representations and operations that agentic systems build on.They may also incorporate procedural methods, learned generation, and inference of graphics programs.Source: §2.1 · p. 3
  • Specialized networks that generate graphics programs fall outside the report’s scope because they do not use a general-purpose LLM to choose graphics operations.Their program representations can nevertheless support inspection and editing in agentic workflows.Source: §2.1 · p. 3
  • Single-pass LLM outputs for graphics descriptions, meshes, SVG, or CAD programs lack iterative refinement based on execution feedback.Agentic systems use that feedback to guide later construction or revision.Source: §2.1 · p. 3
  • Language-model agents connect task instructions and environmental observations to tool actions through a software harness that returns results to the model.Retained observations and feedback can support plan revision and error correction without necessarily updating model weights.Source: §2.1 · p. 3

3. Design Space of Agentic Graphics Systems

The taxonomy describes agentic graphics systems through four dimensions, linking task requirements and model decisions to harness coordination and the graphics resources used to execute actions.

3. Design Space of Agentic Graphics Systems

  • The taxonomy’s four dimensions are specification, base model, harness, and graphics interface and tooling.Specification describes task, inputs, and outputs; the base model decides how to proceed; the harness coordinates planning, execution, feedback, and memory; and the interface and tooling provide operations, resources, and representations.Source: §3 · p. 3
  • Icons mark design choices, and a system may combine multiple options for one choice, such as rendered images and execution diagnostics as feedback.The same notation supports comparison across application tables and connects those comparisons to the taxonomy’s definitions.Source: §3 · p. 3

3.1. Specification

Specification describes the task, user-supplied information and existing content, and expected output. The taxonomy distinguishes these choices across 2D and 3D tasks, input forms, and output representations.

3.1. Specification

  • Specification characterizes requirements through the target task, input form, and expected output.Tasks span 2D graphics, static 3D graphics, and dynamic 3D graphics.Source: §3.1 · p. 3
  • Inputs can include language, images, video, existing 3D representations, or documents.These provide intent, visual or temporal references, editable content, or source material for graphics authoring.Source: §3.1 · pp. 3–4
  • Outputs may be executable programs, structured specifications, editable graphics artifacts, or rendered images and videos.The figure presents specification as one dimension in a four-dimension design space.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2. Base Model

The base model makes decisions from requirements and harness-supplied context, returning plans, code, or tool calls. The taxonomy distinguishes models by type and training.

3.2. Base Model

  • Base models are characterized by model type and training, and return plans, code, or tool calls to the harness.The harness coordinates execution through graphics tools.Source: §3.2 · p. 4
  • LLMs process text, code, and tool reports; vision–language models additionally interpret images, while hybrid systems combine distinct backbones across roles.Source: §3.2 · p. 4
  • Training categories distinguish off-the-shelf models, supervised fine-tuning on task-specific examples, and reinforcement-learning post-training to maximize task rewards.Source: §3.2 · p. 4

3.3. Harness

The harness connects the base model to graphics tools and returns execution feedback and relevant memory, allowing subsequent actions to respond to results. Its design covers planning and acting, feedback, and memory.

3.3. Harness

  • The harness routes model decisions to graphics tools and returns observations, assessments, and memory so later actions can respond to execution results.Source: §3.3 · p. 4

3.3.1. Plan & Act

  • Planning ranges from fixed task steps through hybrid stages to agent-determined steps, while workflows may be sequential, parallel, or graph-structured.Source: §3.3.1 · p. 5
  • Update strategies include full regeneration, local patches, rollback, replanning, and selecting among candidate updates.Source: §3.3.1 · p. 5

3.3.2. Feedback Mechanism

  • A verifier assesses graphics results, and the harness combines that assessment with environmental observations into a signal that guides the next action.Source: §3.3.2 · p. 5
  • Verification may use an agent’s self-assessment, a separate critic, programmatic constraints, environment behavior, or human judgments.Source: §3.3.2 · p. 5
  • Feedback signals include verifier assessments—verdicts, scores, rankings, or diagnoses—and observations of visual, structural, or runtime state.Source: §3.3.2 · p. 5

3.3.3. Memory

  • Memory connects task inputs, tool records, and feedback to later decisions; its dimensions are retention time and access scope.Source: §3.3.3 · pp. 5–6
  • Working memory serves the current model call, short-term memory spans calls within a run, and long-term memory supports reuse across tasks without changing model weights.Source: §3.3.3 · pp. 5–6

3.4. Graphics Interface & Tooling

Graphics interface and tooling choices determine how agents express operations, what environments and capabilities they use, and which content representations they can edit or observe.

3.4.1. Action Type

  • Agents express operations through code or scripts, structured specifications, predefined tool or function calls, and GUI actions, which differ in what the agent specifies versus the execution environment.Source: §3.4 · p. 6, §3.4.1 · p. 6
  • Code offers control over operation order and parameter dependencies but may contain errors; schemas constrain edits to expressible operations, and tool calls limit direct control to exposed operations and parameters.Source: §3.4.1 · p. 6
  • GUI actions provide access to existing workflows but require locating interface elements and tracking state, whose selection, mode, or dialog can affect an action’s effect.Source: §3.4.1 · p. 6

3.4.2. Graphics Software

  • Graphics environments are agent-native, agent-adapted, or general-purpose; source availability, API coverage, and documentation access are separate properties.Source: §3.4.2 · pp. 6–7

3.4.3. Tools

  • Tools include graphics operations, perception models, and validators, which create or modify state, analyze visual inputs or generate assets, and assess or solve against requirements.Source: §3.4.3 · p. 7

3.4.4. Resources

  • Resources include reusable assets, examples and experience, and documentation and rules; they may be supplied in advance or retrieved on demand, and fixed or updated.Source: §3.4.4 · p. 7

3.4.5. Representation

  • Representations include code, geometry, dynamic state, and scene structure, retaining different information for coordinated edits, spatial queries, simulation, and relational checks.Source: §3.4.5 · pp. 7–8
  • Pixels expose sampled appearance but not topology, occluded geometry, or simulation variables; learned state can encode appearance without semantic boundaries or collision surfaces.Source: §3.4.5 · p. 8

4. Agentic Literature Survey System

The system discovers papers, annotates them with evidence and taxonomy labels, and curates records for later runs. Applied to a manually verified corpus, it surfaces cross-task patterns and changes over time in observations, updates, and resources.

4.1. System Design

  • Each run uses a taxonomy, candidate papers, previous records, a date cutoff, and a reading budget to discover papers, annotate evidence, and curate knowledge.Source: §4 · p. 8, §4.1 · p. 9
  • Curation checks for missing fields and taxonomy consistency, but does not verify that classifications are correct.Source: §4.1 · p. 9

4.2.1. Search Results

  • The system was run to compile 350 papers published between 2024 and September 2026, followed by manual relevance verification.It records taxonomy labels, design insights, limitations, and source locations; examples compare SubDGuide’s local repairs constrained by its initial control mesh with PLAYGROUND’s fixed accepted groups.Source: §4.2.1 · p. 9

4.2.2. Search Analysis

  • The system’s taxonomy-based analysis is presented as a way to examine common patterns and emerging trends across the literature.Source: §4.2.2 · p. 10
  • Visual feedback appears in 77.0% of the 304 works using observation, and 58.2% combine multiple observation categories.The figure also compares observation-category task distributions.Source: §4.2.2 · p. 10, Figure 5 · p. 10
  • CAD generation more often uses full regeneration, whereas scene generation more often uses local patch updates; scene systems may also roll back revisions when critic scores decline.Source: §4.2.2 · p. 10, Figure 5 · p. 10
  • Scene generation most frequently uses asset libraries, whereas CAD generation most frequently uses documentation and rules.Source: §4.2.2 · p. 10, Figure 5 · p. 10
  • Local revision rose from 50.0% to 83.0%, while documentation and rules reached 85.3% of records in the latest period.The analysis presents temporal trends in update strategies and resources.Source: §4.2.2 · p. 10, Figure 5 · p. 10

5. Datasets and Benchmarks for Agentic Graphics

Agentic-graphics evaluation spans both spatial understanding and the correctness of generated artifacts, with newer benchmarks also examining workflows and interaction. The benchmarks cover static and dynamic outputs across multiple graphics tasks.

Evaluation of graphics artifacts

  • The benchmark landscape includes datasets for static CAD, Blender objects, and scenes, as well as dynamic articulated objects, scenes, and motion graphics.Figure 6 presents examples across both output types.Source: Figure 6 · p. 11
  • Benchmarks assess interactive agents’ ability to interpret goals, reason about scene state, use graphics tools, and produce usable artifacts.They evaluate artifact properties such as material, artistic quality, semantics, geometric validity, and preference.Source: §5 · p. 11

Spatial understanding in agents

  • Spatial-understanding benchmarks test spatial relations, distances, directions, multi-view reasoning, 3D grounding, and geometric reasoning as foundations for graphics interaction.Some also assess semantic understanding and multi-step planning in exploratory 3D environments.Source: §5 · p. 11

Evaluation of graphics artifacts

  • Workflow benchmarks evaluate sequential edits and iterative refinement, including long CAD-operation sequences and code–render–inspect loops.Video-to-3D benchmarks also assess perceptual similarity and preservation of spatio-temporal information.Source: §5 · p. 11

Evaluation of agentic interaction and coordination

  • Interaction benchmarks extend evaluation from final outcomes to action trajectories that reveal how agents plan, coordinate, and execute tasks.Examples include GUI tasks, interactive reconstruction settings, and mechanical CAD workflows.Source: §5 · pp. 11–12

6. Application: 2D Graphics

Agentic 2D graphics covers structured visual artifacts ranging from vectors and sketches to figures, charts, and presentations. Across these tasks, systems use structured representations and feedback to construct or revise components while addressing task-specific constraints.

SVG and vector graphics

  • Vector-graphics systems generate editable artifacts from language, images, or references, using rendered feedback and semantic grouping for localized edits and animation.The shared challenge is to ground structured representations in rendered evidence while retaining component-level editability.Source: §6 · p. 12

Visual design and editing

  • Visual-design systems span Photoshop tool use, web-based editable representations, collaborative sketching, and text-driven layout or scene construction.SketchAgent generates sequential strokes from a text prompt, with options for editing or animation.Source: §6 · p. 12, Figure 7 · p. 12

Scientific figure and diagram generation

  • Scientific-figure systems generate methodology diagrams, plots, and editable illustrations, while diagram tasks require realizing specified spatial relationships and geometric constraints.PaperBanana orchestrates specialized agents for academic illustrations; related work also targets text-to-diagram generation and editing.Source: §6 · pp. 12–13, Figure 8 · p. 12

Data visualization and plot generation

  • Data-visualization systems interpret requests, execute candidate code, and use feedback to correct errors, while other approaches specialize code generation across programming languages.Evaluation must consider whether charts faithfully represent data and meet visual requirements, not just whether code executes.Source: §6 · p. 13

Document and presentation graphics

  • Document-design systems coordinate layout and visual hierarchy within pages and across slide decks, with methods supporting controllable diversity, personalization, and iterative revision.Structured slide representations and semantic abstraction aim to enable local edits while maintaining global consistency.Source: §6 · pp. 13–14

7. Application: Modeling Static 3D Assets

Static 3D modeling encompasses object construction and editing, engineering CAD, and scene generation, reconstruction, and editing. Across these tasks, systems use task-specific observations, constraints, and feedback to guide revisions.

7.1. Object Modeling

  • Object-modeling systems construct geometry through iterative code refinement, incremental placement checks, or explicit part and assembly relations.Material methods build or revise procedural graphs and textures using visual feedback, while asset-editing systems use candidate selection, localized brush actions, and asset-specific constraints to preserve unchanged content.Source: §7.1 · pp. 14–15

7.2. CAD Modeling

  • CAD systems address specification-driven generation, reference-based reconstruction, and editing by grounding revisions in requirement checks, geometric measurements, reference comparisons, or inspected entities.For image-based reconstruction, depth and hidden structure remain ambiguous even when geometric checks pass.Source: §7.2 · pp. 15–16

7.3. Scene Modeling

  • Scene-generation systems organize placements through scene graphs, constraints, solvers, or room-and-surface stages, with visual review and memory supporting iterative correction.SceneSmith uses designer, critic, and orchestrator interactions; reconstruction systems inspect views or geometric measurements before targeted edits, while editing systems combine placement constraints with visual assessment and scene history.Source: §7.3 · pp. 16–17, Figure 14 · p. 16

8. Application: Modeling Dynamic Assets

Dynamic asset modeling covers articulation, animation, and simulation, with systems linking observed behavior to editable structures and controls. The section distinguishes matching visible motion from recovering the physical mechanism that produces it.

Articulation

  • Articulation systems construct parts and joints in executable programs, then use construction tests, visual references, or physical feedback to assess the mechanism.Artisan adds collision and stability checks, with independent review checking that repairs preserve intended motion.Source: §8 · pp. 17–18, Figure 17 · p. 18

Animation

  • Animation systems represent temporal intent through coordinated sequence tracks, persistent object states, joint trajectories, or emitter configurations, then map visual diagnoses to revisions.ArtiMo uses URDF constraints and bounded parameter updates; ParticleGen uses localized patches and selective rollback.Source: §8 · pp. 18–19, Figure 18 · p. 18

Simulation

  • Simulation systems use diagnostic interactions and runtime checks to distinguish errors in geometry, physical properties, numerical settings, or imposed interactions.DiagGen reruns the diagnosed generation stage and its downstream dependencies before retesting; CoDimRecon supports rod, shell, and solid representations, while its material settings are not calibrated from real-world measurements.Source: §8 · p. 19, Figure 19 · p. 19

Evaluation and open challenges

  • Strong static reconstruction does not ensure accurate temporal behavior, and a close trajectory match can still leave the underlying mechanism underconstrained.4DCodeBench identifies changed initial conditions, external forces, and longer-horizon prediction as future tests of generalization beyond the reference sequence.Source: §8 · p. 19

9. Open Problems

The report identifies open problems spanning agent and tool coordination, human-agent interaction, planning, and evaluation. It also highlights unresolved questions about creative attribution and human control.

Multi-agent collaboration and cross-task generalization

  • Agents must decompose complex tasks, coordinate specialized expertise, and integrate results across tools and domains with potentially incompatible representations and assumptions.The report also calls for knowledge sharing, skill transfer, and adaptation to unfamiliar requirements rather than manually designed pipelines.Source: §9 · p. 19

Action spaces and tool orchestration

  • Choosing and switching among code, GUI, API, and domain-specific tools while balancing reliability, editability, and cost remains an open challenge.These interfaces offer complementary capabilities: code supports precise, reproducible modifications, while GUIs expose functionality that may not be programmatically available.Source: §9 · pp. 19–20

Agent-oriented interfaces and representations

  • Agent-oriented languages and interfaces could clarify action exchange and composition, but must also translate machine-oriented representations into forms designers can inspect and edit.The report cites AIDL [JZH*25] as a CAD example that makes high-level structure explicit and offloads geometric constraints to a solver.Source: §9 · p. 20

Long-horizon planning, memory, and efficiency

  • Long-horizon workflows require agents to track project state, decisions, dependencies, and intermediate outputs, while learning from failures and reusing successful strategies.Without these capabilities, early mistakes can propagate and make later corrections increasingly costly; adaptation must also balance design quality and computational costs.Source: §9 · p. 20

Transparency and Interpretability

  • Transparent workflows should expose plans, assumptions, intermediate artifacts, and edit histories for inspection and intervention without overwhelming users with low-level details.Users need to understand both what changed and why.Source: §9 · p. 20

Human interaction and intent alignment

  • Ambiguous or subjective requests require agents to manage assumptions, ask for clarification when needed, and incorporate fine-grained human feedback.The report also notes work on building reusable operations through a user-centric shared vocabulary [JWA26].Source: §9 · p. 20

Reliable evaluation and validation

  • Reliable evaluation is constrained by multidimensional artifact quality, run-to-run variation, proprietary models, and the cost of expert human inspection.The report calls for stronger benchmarks and scalable protocols assessing both artifacts and agent behavior.Source: §9 · p. 20

Creativity and attribution

  • As agents contribute to ideation and refinement, research must clarify creative decision-making, attribution, provenance, accountability, and meaningful human control.These questions are particularly relevant when outputs incorporate user materials, retrieved references, or assets from multiple tools.Source: §9 · pp. 20–21

10. Conclusion

The survey defines and categorizes agentic graphics systems, automates literature discovery and annotation, and reviews benchmarks and systems across graphics tasks. It finds recurring patterns in planning, verification, and refinement, and presents an adaptive taxonomy as a basis for organizing and comparing the growing field.

10. Conclusion

  • The survey’s taxonomy organizes systems by user input, base model, harness, and graphics interface and tooling, alongside an agentic system for discovering and annotating literature.It also evaluates benchmarks and systems across 2D graphics, static 3D assets, and dynamic 3D assets, then discusses open problems.Source: §10 · p. 21
  • Across tasks, systems show recurring patterns in planning, verification, and iterative refinement, while harness designs vary more than base-model choices.Most use off-the-shelf base models; code and structured specifications dominate graphics interaction, while tool calls and GUI actions depend on available interfaces.Source: §10 · p. 21
  • The taxonomy is intended as an adaptive framework whose categories can evolve as tools, interfaces, and agentic mechanisms emerge.The survey presents it as a common vocabulary and foundation for understanding approaches, identifying trends, and guiding development.Source: §10 · p. 21

Related papers

Loading 2611.01860v1…