PaperlayerView exact revision on arXiv

Source-linked AI summary

MaLiang-Harness: A Programmable Path to Image and Video Generation

Haoyu Zhao, Zihao Zhang, Xudong Wang, Jiaxi Gu, Zuxuan Wu, Yu-Gang Jiang, Shuicheng Yan

arXiv:2609.34309v1cs.CV

TL;DR

Executable visual programs can run correctly while failing to meet requested visual requirements. MaLiang-Harness organizes MLLM-driven image and video generation as persistent construction, inspection, and revision, tying programs and verification to a shared revision reference. Across its benchmarks, models differ in visual outcomes, and GPT-6-Astra achieves 100% generation success on both.

  • Problem

    Executable code can run correctly yet produce images or videos that fail to satisfy the user's visual intent.

    Source: §1 · p. 3

  • Method

    MaLiang-Harness preserves program and task state, traces edits to rendered evidence, and supports revision-aware restoration and verification.

    Source: §1 · p. 3

  • Results

    Models differ in visual quality, generation success, and computational cost, while GPT-6-Astra achieves 100% generation success on both benchmarks.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    Similar general-capability scores can correspond to substantially different visual outcomes, motivating direct evaluation of visual program generation.

    Source: §4.3 · pp. 12–13, §5 · p. 14

    PEG, TGP, and REV support inspection and recovery but do not guarantee effective corrections; refinement can stall when the token budget is exhausted.

    Source: §4.3 · p. 13

Abstract

from arXiv · show

Executable programs offer explicit control over how images and videos are constructed, but generating runnable code is only the beginning of visual creation. A program can execute correctly while violating the requested composition, appearance, or motion. We define this discrepancy as the Program-to-Visual (P2V) gap and introduce MaLiang-Harness, a unified framework for organizing MLLM-driven visual generation into a persistent process of construction, inspection, and revision. Its central design is to make the evolving visual program, its construction history, and its verification share a common revision reference. We define the Persistent Executable Generation (PEG) state as preserving programs and task context. Traceable Generation Process (TGP) connects edits to rendered evidence, and Revision-aware Editing and Verification (REV) supports restoration and checks the current revision before completion. Together, these mechanisms coordinate planning, execution, and visual feedback across rendering backends. We evaluate 11 powerful closed-source MLLMs on MaLiang-IBench and four on MaLiang-VBench, measuring generation success, visual quality, and computational cost. GPT-6-Astra achieves 100% generation success on both benchmarks, with 96.0% of image tasks and 76.9% of video tasks meeting all quality thresholds. The comparison also reveals a mismatch between general capability scores and visual generation performance, with similarly scored models differing substantially in their ability to satisfy visual requirements. MaLiang-Harness provides a systematic basis for studying how MLLMs translate executable code into visual outcomes, exposing both the potential of programmable generation and the limitations of general benchmarks as predictors of this ability. The project is available at https://github.com/gulucaptain/MaLiang-Harness.

1 Introduction

Programmable visual generation can produce executable code that still misses a user’s visual intent, motivating a stateful process of construction, inspection, and revision. MaLiang-Harness organizes that process around persistent artwork state, traceable visual evidence, and revision-aware verification.

1 Introduction

  • A program can execute correctly yet fail to satisfy the requested composition, appearance, or motion, defining the Program-to-Visual (P2V) gap.Addressing the gap calls for visual reasoning over rendered results and continued assessment as revisions can affect previously satisfied requirements.Source: §1 · pp. 3–4
  • The framework connects creative intent to executable visual representations and supports image and video creation through a shared interface for state management, rendering, and revision-aware verification.It supports multiple rendering backends, with optional image assets when appearance is difficult to express through code alone.Source: §1 · p. 3
  • MaLiang-Harness maintains the evolving artwork and task context, records how operations affect rendered results, and ties restoration and verification to specific revisions.PEG maintains programs and context; TGP connects operations to visual results; REV supports restoration and renewed assessment of committed updates.Source: §1 · pp. 3–4
  • 100% generation success on both benchmarks; GPT-6-Astra outputs meet every quality threshold on 96.0% of image tasks and 76.9% of video tasks.The evaluation covers 11 MLLMs on MaLiang-IBench and four on MaLiang-VBench.Source: §1 · pp. 3–4
  • The evaluation compares models across visual quality, generation success, and computational cost, with rankings differing across those dimensions.Similar general-capability scores can correspond to substantially different visual generation outcomes.Source: §1 · p. 4

2 Related Work

Related work spans learned image and video synthesis, executable visual representations, and agent harnesses. MaLiang-Harness positions its contribution as persistent image and video creation with inspectable construction and revision across rendering backends.

Image and Video Generation

Executable Visual Representations

Agent Harnesses

  • Prior agent harnesses address code-based execution, robot actions, or refinement from verified executions [Ning et al., 2026, Chen et al., 2026, Xu et al., 2026b].MaLiang-Harness manages executable artwork through interfaces and feedback, linking visual construction to edit history and revision-specific evidence.Source: §2 · p. 5

3 MaLiang-Harness

MaLiang-Harness organizes MLLM-driven image and video generation around executable visual programs, rendered feedback, and revision-aware state. Its interface connects planning, rendering, construction history, and verification through shared PEG revisions.

3.1 Unified Programmatic Visual Generation Interface

  • The interface lets an MLLM plan visual content, select an executable representation and backend, generate code, and revise it using rendered discrepancies rather than execution success alone.Rendering backends retain native executable representations within a common protocol for state inspection, editing, rendering, and requirement review.Source: §3.1 · pp. 5–6, §3 · p. 5
  • PEG preserves the visual program, associated assets, output specification, scene attributes, task requirements, plan, and revision index as a common reference for planning, execution, and revision.Edits commit as new revisions after input validation and a check that the source revision is current; earlier snapshots remain available for restoration.Source: §3.1 · p. 6

3.2 Traceable Generation Process

  • TGP records generation operations, their inputs and results, and source and resulting revisions, linking program changes to revision-specific rendered evidence.Rendering and inspection can retain the same revision index, and evidence records include outputs and sampled timestamps to support diagnosis of visual discrepancies.Source: §3.2 · p. 7

3.3 Revision-aware Editing and Verification

  • REV permits restoration of historical content as a new revision while preserving task requirements, and requires current-revision evidence and approved checkpoints before delivery.Temporal reviews use ordered samples at three or more timestamps but do not establish continuity between frames; export checks also verify files, hashes, format, dimensions, and video timing.Source: §3.3 · pp. 7–8

4 Experiments

MaLiang-Harness is evaluated across image and video tasks, where execution success does not ensure visual quality. The comparisons and discussion also examine model capability, refinement strategies, and the construction history behind generated results.

4.1 Experimental Setup

  • The evaluation uses 50 image prompts and 13 video prompts, tests 11 image models and four video models, and assesses success, cost, and quality.Image quality covers alignment, aesthetics, and composition; video adds motion coherence.Source: §4.1 · p. 8

4.2 Comparisons

  • GPT-6-Astra achieves 100% generation success on MaLiang-IBench, with 48 images meeting all quality criteria; on MaLiang-VBench, 10 of 13 successful generations meet all four thresholds.The tables distinguish generation success and cost from visual quality counts.Source: §4.2 · pp. 8–9, Table 3 · p. 9, Table 2 · p. 9
  • Successful execution does not guarantee quality: GPT-5.6-Sol produces 43 qualifying images at 3.28 minutes each, while Astra produces 48 at 3.70 minutes each.The time estimates include failures and retries.Source: §4.2 · p. 9
  • Qualitative examples show model differences in composition, surface detail, and realization of prompted action sequences across selected image and video tasks.Image examples include greenhouse and robot scenes; video examples contrast scene detail and action progression.Source: §4.2 · pp. 9–10, Figure 6 · p. 10

4.3 Discussion

  • A path-tracing backend supports material and lighting effects, but local geometry edits can introduce artifacts, and controlled comparisons are still needed to establish its contribution to realism.Increasing the final sample count from 512 to 1024 reduces visible noise.Source: §4.3 · p. 11, Figure 7 · p. 11
  • Finer brushwork preserves the illustration’s overall composition but does not produce photorealism and can reduce grass coverage and distinctive surface textures.The discussion identifies coordinated shape, shading, and texture as still needed for realistic depiction.Source: §4.3 · pp. 11–12, Figure 8 · p. 12
  • General capability scores only partly predict visual outcomes: GPT-5.6-Luna and GPT-6-Luna both score 37, yet their drawing quality pass rates are 44% and 88%.Across 11 models, the measures have a positive rank correlation of Spearman ρ = 0.65.Source: §4.3 · pp. 12–13, Figure 9 · p. 12
  • Construction traces reveal how individual elements were produced and which requirements remain unresolved, while refinement may stall despite state preservation and inspection tools.The discussion distinguishes execution termination limits from successful task completion.Source: §4.3 · p. 13, Figure 10 · p. 13

5 Conclusion

MaLiang-Harness addresses the gap between correct executable programs and visual requirement satisfaction by organizing creation as persistent construction, inspection, and refinement. Evaluations show substantial differences across models, while photorealism and stalled refinement remain challenges.

5 Conclusion

  • MaLiang-Harness organizes MLLM-driven visual creation by maintaining executable state, recording construction histories, and linking verification to the current revision.The framework is motivated by the Program-to-Visual gap between program-level correctness and visual requirement satisfaction.Source: §5 · pp. 13–14
  • Evaluation of 11 MLLMs on MaLiang-IBench and four on MaLiang-VBench reveals substantial differences in generation success, visual quality, and computational cost.Models with similar general-capability scores can produce markedly different visual outcomes.Source: §5 · p. 14
  • Photorealism and stalled refinement remain challenges; future work will explore richer rendering backends and progress-aware refinement strategies.The paper concludes that general-capability benchmarks do not fully predict visual program generation performance.Source: §5 · p. 14

6 Appendix

The appendix documents procedural construction, evaluation and aggregation rules, comparative-plot conventions, and prompts for qualitative image and video examples. It also provides video-comparison prompts for a looping café animation and a doodle whitewater ride built around a real watering can.

6.1 Procedural Construction Example

  • The botanical portrait program builds four layers—background, foliage, portrait, and foreground flowers—with separate seeded generators and no image assets.Algorithm 2 describes operations within a fixed PEG revision rather than the full planning and refinement loop.Source: §6.1 · p. 16

6.2 Evaluation Protocol and Reporting Details

  • Image tasks comprise 50 prompts and video tasks 13; success requires decodable outputs passing completion checks, while quality scores use successful outputs only.Images are scored 1–5 for alignment, aesthetics, and composition; videos also receive motion-coherence scores. Threshold counts use the full task set, and All Criteria requires passing every threshold.Source: §6.2 · p. 16
  • Image and video cost comparisons reflect different run configurations and aggregation rules, so identically named cost columns are not always directly comparable.Image benchmark runs combine batches, retries, and historical runs; video results use one recorded attempt per model and task, with costs averaged over 13 attempts.Source: §6.2 · p. 16

6.3 Comparative Plots and Table Conventions

  • Quality-table colors show higher or lower GPT results relative to GPT-5.6-Luna for images and GPT-5.6-Sol for videos; threshold counts cover the full task set, unlike means over successful subsets.DeepSeek and Kimi use a separate gray scale because their quality results cover fewer successful tasks.Source: §6.3 · p. 18
  • The image-quality pass rate correlates with the Artificial Analysis Intelligence Index at ρ = 0.65 across 11 models, or ρ = 0.63 excluding DeepSeek-V4-Pro.The authors characterize this as an association, not an isolated capability effect; the public DeepSeek checkpoint may not match the evaluated checkpoint.Source: §6.3 · p. 18

6.4 Prompts for the Qualitative Image Comparisons

  • The image prompts span constrained mixed-style illustrations, architectural boards, character compositions, and a layout-control diagram with explicit spatial specifications.The additional diagram prompt specifies normalized coordinates, dimensions, colors, and exactly 15 non-overlapping rectangles, interpreted through SVG or Canvas code.Source: §6.4 · pp. 18–20

6.5 Prompts for the Qualitative Video Comparisons

  • The video prompts specify pixel-art scenes with ordered actions, from a kitten’s star-delivery journey across a desk to the final illuminated miniature city.The appendix notes that the 12-frame quality review differs from the five uniformly sampled frames shown in the comparison figure.Source: §6.5 · pp. 20–21
  • The video prompts call for continuous, spatially legible action, stable camera movement, triggered object interactions, and structures assembled or unfolded rather than faded in.They also prohibit unstable pixel grids, antialiasing, motion blur, gradients, realistic water, blurred glowing edges, and game-interface elements.Source: §6.5 · p. 22
  • The café prompt specifies a 10-second, asset-free JavaScript Canvas animation in which a tiny dragon’s actions trigger escalating events before it returns to its sleeping pose.The sequence moves from a sneeze and ignited coffee beans to chasing, latte art, and a steam-triggered return to the cup; the final pose must match frame 0.Source: §6.5 · p. 22
  • The café combines a semi-realistic still life with illustrated fantasy overlays, a muted palette, and small ambient details such as moving crumbs and swaying menus.Thick dark-brown outlines are reserved for animated fantasy elements, while furniture and countertop use softer realistic shading.Source: §6.5 · p. 22
  • A second prompt turns a real watering can into the source of a doodle river ride, as cats draw channels and docks, launch boats, and navigate rapids, waterfalls, and whirlpools.The scene escalates with splashes, leaping fish, bouncing boats, a near-fall, and a cat surfing on a leaf.Source: §6.5 · p. 22

Related papers

Loading 2609.34309v1…