PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Visual Generation: From Generative Models to Agentic Control

Yinming Huang, Shuyuan Tu, Xi Yan, Jiahao Zhan, Zihan Yang, Zhen Xing, Hui Zhang, Tiehua Zhang, Yu-Gang Jiang, Zuxuan Wu

arXiv:2609.06758v1cs.CV

TL;DR

Existing visual-generation work lacks a consistent criterion for identifying when a system becomes agentic, because planning, tool use, collaboration, and training method do not specify controller scope. The paper introduces a controller-centered L0–L4 hierarchy based on the maximum causal reach of generation decisions and applies it across visual-generation tasks, finding rapid growth in within-trajectory feedback but comparatively rare persistent cross-task experience.

  • Problem

    Existing work uses inconsistent criteria for agenticity, so controller decision-making scope must be separated from generator progress, implementation topology, and learning procedure.

  • Method

    The paper classifies systems by the latest future generation decision their controller can change, from fixed support through conditioning, execution, outcome adaptation, and experience adaptation.

  • Results

    The structured corpus shows rapid growth in within-trajectory feedback, while persistent cross-task experience remains comparatively rare.

  • Takeaways & Limitations

    The framework makes controller capability comparable across modalities and separates it from model size, tool count, role topology, training procedure, and output quality.

  • Takeaways & Limitations

    Conditioning failures require different diagnostics, and a single final-image preference score cannot identify which interface failed.

Abstract

from arXiv · show

Visual generation is evolving from generative models used through a single invocation into agentic control processes that can plan, select tools, inspect intermediate synthesized outputs, revise failures, and reuse prior experience. In most existing systems, the controller is an LLM or VLM, while visual generation models serve as tools or executors. However, existing work lacks a consistent criterion for determining when a generation system becomes agentic. Planning depth, tool use, multi-role collaboration, and reinforcement learning are often treated as evidence of agenticity, even though none of them necessarily determines which generation decisions the controller can make. We organize the field according to what the controller can directly control in the generation process. At L1 Conditioning Control, the controller prepares the input to a predetermined generator but does not control which visual operation is executed. At L2 Execution Control, it selects and invokes actual generation, editing, rendering, or other content-modifying operations. At L3 Outcome-Adaptive Control, it observes an intermediate outcome and uses that observation to change a subsequent operation within the current task. At L4 Experience-Adaptive Control, it retains experience from completed tasks and uses that experience to change decisions on future tasks. L0 Fixed Support separately denotes generators, editors, evaluators, reward models, benchmarks, and fixed pipelines without a deployed controller that makes generation-level decisions. These levels describe a progressively broader decision-making scope rather than model size, system complexity, output quality, tool or role count, or training method. Applying this framework across image, video, editing, 3D, world, slide, and user-interface generation reveals how controller capabilities have evolved and how their mechanisms are distributed across levels.

L2 Execution Control

L2 Execution Control marks the transition from preparing inputs for a fixed executor to selecting and invoking visual operations. The framework separates this capability from task type, implementation mechanisms, and model architecture.

  • L2 Execution Control: L2 controllers choose and invoke generators, editors, programs, or workflows instead of only preparing input for a fixed executor.Examples include callable visual tools and executable node graphs whose components and data flow determine the generation route.
  • Level transitions: L2 extends control beyond conditioning by selecting operations, while L3 further uses observed results to redirect later generation actions.Evaluators can diagnose compositional, object-binding, temporal, or perceptual defects, but L3 systems close the loop by mapping observations to subsequent actions.
  • Inclusion boundary: The framework includes a visual generator or editor controlled by a generation-level decision process, excluding standalone supporting modules and fixed pipelines.The controller may collaborate with other agents or call external tools, but the system must contain a primary visual generator or editor.
  • Capability criterion: The framework defines agenticity by the maximum temporal and causal reach of controller decisions, spanning L0 Fixed Support through L4 Experience-Adaptive Control.L0 marks the inclusion boundary, while L1–L4 classify progressively broader controller capabilities.
  • Task and mechanism axes: Task type does not determine controller capability because systems for different visual tasks can expose different decision-making scopes.The task axis includes image, video, slide, user-interface, 3D, world, and interactive-simulation generation.

1) Agents:

The paper treats an agent as a goal-directed controller that selects actions from observations and maintains state so later generation decisions can adapt. Agentic visual generation separates this controller from the visual generator while allowing distributed roles, tools, or a unified model.

  • 1) Agents:: An agent selects actions from observations and maintains enough state to adapt later decisions.Relevant visual-generation decisions include prompt revision, layout construction, model routing, reference selection, editing, verification, memory update, and stopping.
  • 1) Agents:: Agenticity depends on an action space, an observation process, and feedback that changes a later generation decision, not on controller architecture.A controller may be a language model, multimodal model, learned policy, search procedure, or team of specialized roles.
  • 1) Agents:: The generator creates or updates synthesized visual content, while the controller maps goals, state, and observations to generation-level actions.This separation applies to images, videos, slides, user interfaces, 3D scenes, and interactive environments.
  • 1) Agents:: An agentic visual generation system contains a visual generator or editor and a decision process that controls generation over one or more steps.The controller follows a policy over actions, and state transitions may include generated content, critiques, or retrieved references.
  • 1) Agents:: Controllers may collaborate with specialized agents and external tools, whose outputs provide conditions, executable actions, evidence, or feedback.Tools include retrievers, generators, editors, renderers, simulators, browser runtimes, verifiers, and reward models.
  • 1) Agents:: A single learned model qualifies as a controller only when it demonstrates control decisions over generation conditions or subsequent refinement.UI2CodeN is described as generating content, inspecting rendered state, and deciding whether to refine it.

III. A HIERARCHY OF CONTROLLER DECISION-MAKING SCOPE

The hierarchy classifies agenticity by the latest future generation decision a controller can causally change, rather than by architecture, complexity, or training method. Its levels extend from conditioning a predetermined executor to adapting decisions across tasks, while remaining comparable across visual modalities.

  • Controller-Capability Axis: Agenticity is ordered by the deepest point in a generation trajectory where the controller can causally change a future decision.The framework treats temporal reach along the causal generation trajectory as the organizing principle.
  • Controller-Capability Axis: Systems receive the highest demonstrated level, while lower capabilities remain visible as a capability path.For example, L1+L2+L3 records specification construction, operation invocation, and outcome-dependent revision but is labeled L3.
  • Decision Procedure: L0 marks fixed supporting components, whereas L1–L4 distinguish conditioning, execution, within-task outcome adaptation, and cross-task experience reuse.The assignment procedure tests levels from L4 downward and assigns L0 when no controller-level decision criterion holds.
  • Decision Procedure: Architecture, planning, tool use, feedback, memory, and reinforcement learning do not determine agenticity unless they change the controller’s inference-time causal reach.A planner may terminate in one specification, and reinforcement learning may optimize policies with different action spaces and levels.
  • Cross-Modal Applicability: The hierarchy makes one comparison legible across modalities: how late can evidence still change a future generation decision?The same causal test can classify a storyboard planner and spatial-layout planner as L1, or an image router and 3D asset selector as L2.

1) Capability Evolution Over Time:

The corpus shows rapid growth in outcome-adaptive systems after 2025, while persistent cross-task experience reuse remains uncommon. Across modalities, controller organization is dominated by single-controller systems, with task distributions and architectural roles analyzed separately.

  • Capability evolution: After 2025, corpus growth is dominated by L3 Outcome-Adaptive Control rather than lower-level conditioning control.Only four reviewed records are assigned to L0 Fixed Support.
  • Implications: Persistent experience reuse remains substantially less mature than within-task correction, making the transition from L3 to L4 an open research challenge.
  • Capability evolution: L3 is prevalent across modalities, but its implementation differs because editing and interface systems inspect rendered states while video, 3D, and world systems require temporal or viewpoint consistency.
  • Task distribution: Image generation has the largest absolute corpus count and contains most L1 controllers based on prompt construction, layout planning, and retrieval.
  • Controller organization: Single-controller systems dominate every controller level; multi-role organizations occur mainly from L1 through L3, while unified multimodal policies remain less common.

A. Generation and Retrieval Components

L0 Fixed Support contains generators, retrieval, training, evaluators, benchmarks, and fixed pipelines that support generation without deciding which generation action occurs next. The boundary is determined by deployed decision scope, not by component complexity or offline process sophistication.

  • L0 Fixed Support: L0 components execute supplied conditions through predetermined paths, establishing what can be executed but not which operation should be selected.
  • Generation and retrieval: Retrieval remains L0 when neither retrieved evidence nor generated output changes a later generation action; it becomes L1 or higher only when a controller changes the specification or route.
  • Training support: Offline data construction and generator training may use tools, multiple roles, rewards, or reinforcement learning without changing the resulting inference system’s decision-making scope.
  • Evaluation support: Evaluators and benchmarks measure or organize judgments, but neither establishes L1–L4 scope unless another mechanism maps evaluation to a later generation action.
  • Boundary test: The counterfactual boundary test asks whether the deployed system can choose among materially different generation actions given the same goal and state.

B. Spatial and Geometric Specifications

L1 systems construct generator-facing specifications before execution, including spatial, temporal, camera, compositional, document, and interface structures. Richer conditions reduce ambiguity but commit decisions before generated outcomes can redirect execution.

  • Spatial specifications: Spatial specifications encode object arrangements, regions, scenes, or poses before rendering, so omissions or invalid geometry remain specification errors rather than outcome-driven repairs.
  • Retrieved evidence: Retrieval-based L1 systems first select evidence and then ground it in the generator-facing condition before rendering, rather than repairing generated outputs afterward.
  • Temporal and camera specifications: Temporal and camera specifications commit content evolution and viewpoints before the predetermined video or scene executor produces an outcome.
  • Structured specifications: Compositional, narrative, document, and interface specifications establish modality-specific structure before rendered feedback can redirect execution.
  • Trade-offs and limitations: Richer conditions reduce ambiguity and expose constraints before sampling, but they also commit the system more strongly to pre-generation decisions.
  • Trade-offs and limitations: A single final-image preference score cannot identify whether prompt, spatial, or retrieval conditioning caused the failure, and useful comparisons should include transfer across generator families.

VI. L2: EXECUTION CONTROL

L2 Execution Control lets controllers select, invoke, or compose executable visual operations rather than only prepare a fixed generator’s input. Systems span model routing, image and video operations, document and interface actions, and 3D or world construction, while remaining open-loop with respect to outcomes.

  • Definition and action space: L2 changes the controlled object from a specification to an executable choice selected before observing its outcome.
  • Model and tool operations: Model and tool routing systems select callable generation, editing, search, or retrieval operations according to the request and action budget.
  • Workflow construction: Workflow-construction systems compose ordered node graphs or programs before generated outcomes can redirect the route.
  • Image operations: Image controllers execute ordered edit programs or element-level actions over pixels, vectors, layers, or graph nodes.
  • Video operations: Video controllers operate over clips, shots, timelines, or synchronized audiovisual structures without depending on inspection of the resulting clip.
  • Document, interface, and 3D/world operations: Document, interface, 3D, CAD, and world systems select operations over structured objects, geometry, assets, scenes, or engine workspaces before outcome-based redirection.

F. Discussion: Cross-Domain Execution and the Open-Loop Boundary

L3 is defined by feedback changing a later generation-level action, not by planning length, role count, or executable state. Across domains, systems use perceptual, structured, physical, and human feedback to diagnose and revise visual outputs, while L4 extends adaptation beyond the current task.

  • Open-Loop Boundary: L3 begins when an observation returned by execution affects the next generation-level action.This criterion separates outcome-adaptive control from open-loop execution.
  • Cross-Domain Execution: Action abstraction determines how L2 controllers represent and invoke generation operations across image, video, document, interface, and 3D tasks.Flat tools, workflow graphs, code, domain-specific languages, and structured content models offer different execution guarantees and coordination requirements.
  • Outcome-Adaptive Control: L3 controllers map observed perceptual, structured, physical, or human-review outcomes to later generation actions such as repair, rerouting, stopping, or state preservation.These feedback sources define the decisive observation rather than separate response categories.
  • Cross-Domain Execution: Image, video, 3D, document, and interface systems apply rendered or structured feedback to localized edits, regeneration, workflow changes, and code revisions.The systems span multiple visual modalities while remaining classified by the observation that changes the next action.

C. Physical and Constraint Feedback

Physical feedback closes an outcome-adaptive loop when simulations, geometric tests, reachability checks, or hard rules determine whether the next action should repair or proceed. Human review is likewise feedback only when in-episode judgment causally changes a later generation action.

  • Physical Feedback: Physical feedback uses simulation, geometric relations, reachability tests, or hard rules as accept-or-repair conditions for the next action.A renderer or engine may implement the check, but the violated constraint must change subsequent control.
  • Physical Feedback: Simulated dynamics and world-constraint systems close loops around motion, collision, occlusion, stability, connectivity, navigability, and physical behavior.These checks distinguish physical correctness from merely plausible appearance.
  • Human Review: In-episode human review counts as feedback only when explicit judgment changes a later generation action; terminal preference studies remain evaluation.Examples include prompt changes, concept approval, and supervisor redirection of revision.
  • Discussion: L3 can improve tasks through conditional computation, but every additional action creates another failure opportunity.The discussion therefore compares diagnosis-action coupling, state preservation, feedback sources, and the within-task boundary.
  • Diagnosis-Action Coupling: Localized edits and executable representations clarify diagnosis-action coupling but require more precise critics and can silently damage valid components.Global prompt errors may degrade an entire sample, while code edits can break previously valid structure.
  • Episode Boundary: Reliable within-task feedback does not provide cross-task adaptation unless experience survives the episode boundary.History discarded at task completion cannot help a later independent goal.

C. Reusable Procedures and Skills

L4 stores experience from completed trajectories and uses it to change control on later independent tasks through procedures, executable workflows, policies, or model updates. These carriers differ in validation, reversibility, and the risks of transferring stale or harmful experience.

  • Reusable Procedures and Skills: Procedural memory abstracts completed trajectories into model-readable operational knowledge that must be interpreted before execution.GenEvolve, EvoDiagram, ManimAgent, SceneCraft, and SimWorld Studio exemplify reusable procedures and skills.
  • Executable Workflows: Executable persistence stores validated and versionable graphs, programs, middleware, or control harnesses for later workflows.COMFYCLAW promotes workflows after staged verification, while FigAgent, AutoDesign, VideoWeaver, and AVA-Encoder evolve reusable executable objects.
  • Policy and Model Updates: Policy and model updates write completed experience into later behavior without retrieving a separate memory item or workflow.Persistent updates include diffusion behavior, editing and evaluation components, video policies, clarification policies, and symbolic concepts.
  • Recursive Self-Improvement: Recursive self-improvement remains an L4 form because its causal reach crosses episode boundaries, not because it constitutes a separate hierarchy level.Changing the mechanism that proposes or evaluates later updates distinguishes RSI from fixed improvement loops.
  • Evaluation and Reversibility: Cross-task evaluation must report forward transfer, negative transfer, stale-experience sensitivity, provenance, and rollback because experience can preserve mistakes as well as useful behavior.A self-evolution label is insufficient without behavioral evidence on later tasks.
  • Training and Control: Training method does not determine controller level: a fixed generator may remain L0, while supervised routing or an untrained repair loop may implement L2 or L3 decisions.The hierarchy classifies what the controller changes, whereas training describes how capabilities are learned.

B. Supervised Fine-Tuning

Supervised fine-tuning learns from demonstrated targets or action trajectories, but agentic controller training also requires records of observations, failures, rewards, and costs. Evaluation must separate controller improvement from stronger generators, search budgets, or exploitable evaluators.

  • Supervised Fine-Tuning: Supervised fine-tuning updates a generator or controller to imitate labeled outputs and action trajectories rather than assigning scalar credit to sampled alternatives.The paper distinguishes generator or condition SFT from executable controller-trajectory SFT.
  • Supervised Fine-Tuning: Generator and condition SFT initializes behavior within a predetermined interface, whereas controller-trajectory SFT can learn interleaved reasoning, operations, and observations.GenAgent and ToolArtist use multimodal trajectories to initialize tool use, reflection, and search behavior.
  • Reward and Feedback: Terminal rewards capture holistic output appeal but cannot identify which tool call or revision caused improvement.Process and verifiable rewards instead assess intermediate validity, constraint satisfaction, diagnosis-edit consistency, preservation, and action worth.
  • Reward and Feedback: The cost-aware objective combines goal evidence, process evidence, and action cost, with α and β weighting goal and process terms and λ penalizing computation, tools, or interaction.The components should be reported separately so gains cannot be attributed only to weighting.
  • Optimization Scope: Optimization may target generators, conditions, workflow controllers, or persistent future-task behavior, and these targets are distinct from whether interaction is offline or online.Persistence determines whether updates end with the current request or alter later independent-task control.
  • Failure Analysis: Incomplete trajectories, narrow demonstrations, weak rewards, and unconstrained optimization each create distinct failure points in controller training.Reporting all four stages helps distinguish improved decisions from stronger generators, larger search budgets, or evaluator overfitting.
  • Evaluation: Level-conditioned evaluation matches generators, tools, budgets, and evaluators across L1–L4 to isolate the causal value of broader controller decision-making scope.L0 supplies the fixed-executor baseline.

A. L0 Fixed Support: What Can the Fixed Executor Already Do?

L0 establishes fixed-executor performance before controller interventions, separating task validity, quality, preference, trajectory behavior, efficiency, robustness, and attribution. Later levels require matched comparisons that isolate added decision scope rather than differences in tools or computation.

  • Fixed-executor baseline: L0 reports hard-constraint success, synthesized-content quality, human preference, executability, and resource cost without letting aggregation hide hard failures.Requirement-specific checks precede quality and preference measures; one-number aggregation requires disclosed weighting.
  • Trajectory evaluation: Controller evaluation additionally examines decision validity, including constraint coverage, dependency consistency, argument correctness, evidence relevance, redundancy, and diagnosis-to-repair correspondence.Persistent controllers must also show that memory is relevant rather than stale or unrelated.
  • Trajectory evaluation: Efficiency compares terminal quality against trajectory cost, reporting Pareto frontiers and fixed-budget success rather than equating best-of-many sampling with a single call.Cost includes calls, candidates, editing operations, tokens, latency, accelerator time, and monetary expense.
  • Trajectory evaluation: Robustness tests whether a controller detects an injected failure and returns to a valid route under the same remaining budget.Failure injection targets the state available at the evaluated level, such as an unavailable tool for L2 or critic error for L3.
  • Trajectory evaluation: Faithful attribution uses interventions to test whether planned entities match rendered regions, reported actions match tool calls, and diagnoses predict successful repairs.Counterfactual removal or alteration of cited components tests whether the claimed controller contribution matters.
  • Level-conditioned comparisons: L1 and L2 comparisons hold relevant executors or operation sets fixed while varying condition policies or routing decisions.L1 varies the condition for the same predetermined executor; L2 compares learned routing with fixed, random, frequency-based, or oracle routes over the same tools.

E. L3 Outcome-Adaptive Control: Does Feedback Cause a Successful Repair?

L3 evaluation must show that observing an intermediate outcome causes a targeted later repair, not merely a higher score after repeated sampling. Matched loops, independent evaluation, failure recovery, and logged interventions make feedback-driven attribution testable.

  • Selection bias: A higher final score is insufficient because repeated sampling, best-of-many selection, or a shared evaluator can mimic successful repair.Selection bias is addressed by separating controller, stopping, and held-out evaluators and auditing hard-requirement failures.
  • Matched feedback loops: L3 requires paired open-loop and closed-loop executions with the same initial goal and budget, while only the closed loop observes intermediate renders.Generated candidates and evaluator scores should be logged at every step.
  • Selection bias: Complete L3 reports measure non-target preservation, improvement per accepted revision, and recovery after injected critic or tool failure.The protocol also measures reward-model agreement before and after optimization to detect searched-for evaluator blind spots.
  • Cross-level attribution: Cross-level experiments add one class of controller decisions at a time while preserving budgets and linking each gain to a logged intervention.Matched-call and matched-cost settings distinguish decision efficiency from practical resource use.
  • Cross-level attribution: Counterfactual tests turn plans, critiques, and memories into falsifiable components by removing, replacing, or forcing the claimed intervention under matched conditions.The relevant comparison is whether the targeted repair outperforms an unrelated edit or omission of the credited component.
  • Benchmark interpretation: Modality-specific benchmarks provide observations, but the intervention and its causal reach determine the evaluated controller level.Content metrics establish task success, matched counterfactuals attribute gains, and cost and robustness tests assess practical persistence.

A. L1 to L2: From Declarative to Executable Control

Executable control expands the system from specification toward tool selection, feedback-driven revision, and persistent state, while introducing provenance, budget, credit-assignment, and memory challenges. A possible generator-as-controller regime could shorten visual-state feedback loops, but it remains agentic only when evolving state changes later actions and future tasks.

  • Executable control: L2 control selects and invokes generators, editors, programs, or workflows rather than only preparing input for a fixed executor.Executable control also expands the safety and provenance surface through retrieval, routing, asset reuse, and code execution.
  • Executable control: Useful intervention requires estimating expected quality gain, latency, monetary cost, privacy risk, and tool availability under a matched-tool comparison.Otherwise an apparent L2 gain may arise entirely from access to a stronger generator.
  • Outcome-adaptive control: Outcome-adaptive control requires feedback that maps a diagnosed failure to an actionable region, object, frame interval, or code component.Independent held-out evaluators help prevent repeated generation from raising a selector score without repairing the stated defect.
  • Outcome-adaptive control: Human co-creation extends the same transition when user feedback changes later actions while preserving accepted work.Video timelines and rendered-interface comparisons motivate interfaces exposing plans, masks, timelines, layouts, candidates, uncertainty, and rollback.
  • Experience-adaptive control: L4 begins only when a completed trajectory changes control on a later independent task; extensive within-task state can still support only L3.Evaluation must address stale experience, provenance, timestamps, transfer, negative transfer, and forgetting.
  • Generator as controller: A generator-as-controller could act on shared visual state for localized revision, preserve accepted content, allocate computation to unresolved regions, and reduce language-mediated feedback exchanges.The proposed regime remains subject to evidence that evolving visual state changes later actions and retained experience changes later tasks.
  • Conclusion: Agentic visual generation changes the unit of computation from one conditional sample to a goal-directed trajectory involving planning, tool coordination, intermediate judgment, feedback, and memory.The framework therefore evaluates control by the reach of generation-level decisions rather than modality, model count, or output quality.
Loading 2609.06758v1…