Source-linked AI summary
AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design
Yaxin Luo, Haobin Jiang, Jialv Zou, Xu Huang, Wenhao Yan, Haodong Li, Zhengrong Yue, Jing Li, Xiaofu Chen, Xiaohan Zhao, Jiacheng Liu, Jiacheng Cui, Zhiqiang Shen, Xiaotong Li
TL;DR
Multimodal design systems lack a way to turn feedback from individual attempts into reusable design knowledge. AutoDesign recursively optimizes a design harness using rollout evaluation, achieving the top PosterBench score of 78.32 and the highest human preference.
Problem
Existing multimodal design systems treat human-aligned feedback as transient rather than reusable design knowledge.
Method
AutoDesign uses nested design-harness and meta-harness loops to recursively improve an executable paper-to-poster system from human-aligned evaluation and rollout feedback.
Results
AutoDesign achieves the top PosterBench score of 78.32, surpasses Claude Design by 7.45 points, and receives the highest human preference estimate.
Takeaways & Limitations
AutoDesign turns recurring design failures into persistent improvements for future paper-to-poster generation while keeping model weights fixed.
Takeaways & Limitations
AutoDesign is presently validated for academic paper-to-poster generation, while reliable expansion to other media requires medium-specific data, evaluators, rendering gates, and objectives.
Abstract
from arXiv · showhide
Transforming multimodal sources into condensed and structured media outputs can be fundamentally conceptualized as a long-horizon agentic process centered on a model-harness system. While an ideal harness system should align with human design priors and accumulate reusable experience through empirical exploration to drive recursive self-improvement, existing paradigms remain static and fall short of this capability. In this paper, we present AutoDesign, a framework that aligns with human design priors, where a meta-harness optimizer guides a code agent to recursively improve harness based on rollout feedback. To instantiate and evaluate this framework, we focus on the academic paper-to-poster generation task and introduce PosterBench, comprising a 100-paper Main Track spanning five disciplines and PosterBench-mini, a shared 10-paper subset for controlled evaluation. On the PosterBench Main Track, AutoDesign achieves the highest score of 78.32, surpassing the closed-source commercial system Claude Design by 7.45 points. Across seven controlled code-agent-model configurations, integrating the learned DesignHarness consistently improves performance, increasing the average PosterBench Score from 54.99 to 67.39 (+12.4%). In a fully autonomous long-horizon loop, it executes 253 tool calls and 11 editing turns within 40 minutes for under $3, reaching average conference-poster quality in human evaluation. A system-blind human study further demonstrates that AutoDesign achieves the highest human preference among evaluated systems.
1 Introduction
The introduction presents AutoDesign as a recursively improving meta-harness for human-aligned multimodal design, instantiated through paper-to-poster generation and evaluated with PosterBench. It reports improved benchmark quality, competitive main-track performance, low-cost deployment, and highest human preference.
- Method: AutoDesign recursively optimizes the design harness itself using evaluation grounded in human preferences, rather than optimizing only an individual artifact.The outer loop proposes, evaluates, and accepts or rejects harness-component updates, while accepted updates accumulate into DesignHarness.
- Application: DesignHarness ingests academic papers, generates and revises posters under critic feedback, and finalizes readable, visually coherent outputs while preserving traceable evidence.Paper-to-poster generation must condense long, multimodal scientific sources into a single legible poster.
- Benchmark: PosterBench is introduced to jointly measure source fidelity, dense scientific communication, and rendered usability, addressing gaps in prior paper-to-poster benchmarks.The prior benchmarks covered only subsets such as layout, extraction, faithfulness, or visual quality.
- Results: Attaching DesignHarness to seven Code Agents raises average PosterBench Score from 54.99 to 67.39, a gain of 12.40 points.The improvement is reported on PosterBench-mini.
- Results: 78.32 PosterBench Main Track score lets AutoDesign surpass the closed-source Claude Design by 7.45 points under the same Claude Code and Claude 4.8 configuration; it also reaches 64.0% Bradley–Terry preference across 933 valid pairwise judgments.The 64.0% estimate has a 95% interval of 55.2–77.8%.
2 Meta-Harness Formulation
The section formalizes a design harness as the fixed-model system that converts multimodal sources into human-facing artifacts, and a meta-harness as the system that improves that harness. Optimization targets expected artifact quality while keeping the underlying model parameters fixed.
- 2.1 Design Harness Definition: A design harness H surrounds a fixed LLM or MLLM πθ to transform multimodal input x and context c into a human-facing artifact y.This formulation follows work treating harnesses as optimization targets distinct from model weights (Lee et al., 2026b; Ren et al., 2026).
- 2.1 Design Harness Definition: The harness produces y through trajectory τ, recording intermediate actions, states, and revisions during artifact creation.
- 2.1 Design Harness Definition: Its five components are Context and Memory, Tools and Specifications, Execution Runtime, Orchestration, and Evaluation and Feedback.They respectively support source and state management, artifact authoring, rendering and validation, workflow control, and critique-driven revision.
- 2.1 Design Harness Definition: The decomposition is a high-level abstraction whose concrete components are instantiated and iteratively improved by the meta-harness.
- 2.2 Meta-Harness Definition: A meta-harness operates on a design harness, transforming harness requirements and implementation into an optimized harness HT from an optional initial harness H0.This definition is associated with Lee et al. (2026b), Zhang et al. (2026a), and Lin et al. (2026).
- 2.2 Meta-Harness Definition: Its optimization objective is the expected quality of artifacts produced by H over task distribution ptask, evaluated by Rmeta(y, x, c).
- 2.2 Meta-Harness Definition: During optimization, θ remains fixed, so improvement acts on the surrounding scaffold rather than the underlying model πθ.This preserves the model-versus-scaffold distinction described by Ren et al. (2026).
3 Meta-Harness Learning Loop
AutoDesign organizes design-harness execution and meta-harness optimization as nested feedback loops. The inner loop revises artifacts through designer–critic interaction, while the outer loop uses rollout evidence to propose, evaluate, and selectively accept harness updates.
- Nested feedback loops: The framework nests artifact generation under a fixed design harness with cross-task harness optimization, forming inner and outer feedback loops.The inner loop records execution trajectories, which the outer loop analyzes to improve the harness across tasks.
- Inner-loop design harness: The initial design harness consists of a designer that generates and revises artifacts and a critic that evaluates them and supplies feedback for subsequent revisions.At the first step, the empty initial artifact and feedback allow the designer to produce a draft from the multimodal source and design requirements alone.
- Outer-loop optimization: Each outer-loop iteration performs rollout, evaluation, update proposal, and acceptance, with the evaluator combining rule-based checks and VLM judgments across seven quality dimensions.The optimization-time evaluator is distinct from the frozen PosterBench protocol used for final system comparison.
- Update proposal: The coding-agent optimizer analyzes trajectories, scores, and persistent history, then modifies exactly one harness component per iteration to keep credit assignment interpretable.It uses planner and code-editor roles, parallel subagents, structured failure analysis, and an update plan before implementing changes.
- Acceptance and persistent optimization: A proposed harness is accepted only when training performance improves without development performance declining, guarding against overfitting while persistent records support comparison, reproducibility, and rollback.The optimization record stores checkpoints, trajectories, scores, selected components, plans, code changes, and decisions; optional human guidance can redirect stalled search.
4 The Optimized DesignHarness
DesignHarness is a multimodal harness whose optimized architecture combines provenance-aware source ingestion, iterative artifact generation and revision, validation and visual critique, and finalization. It supports outputs including academic posters, presentation slides, videos, and web pages.
- Source ingestion: The ingestion stage builds a structured, provenance-aware context by extracting metadata, outlines, key passages, figures, and tables with source locations.These materials are organized into a content brief and medium-specific artifact plan for generation and revision.
- Artifact generation: The designer coding agent iteratively generates editable HTML artifacts conditioned on the current artifact, feedback, and retained ingested context.The source-grounded context persists across inner-loop refinement steps.
- Validation and critique: Each candidate undergoes deterministic blocking checks for unsafe or missing assets, broken provenance, severe layout failures, and typographic or layout violations.Candidates that fail are rendered for additional inspection.
- Validation and critique: Rule-based validation and critic-VLM assessment of design compliance, layout, readability, and aesthetics are consolidated into a repair signal for the next revision.This feedback-to-revision loop connects structured checks with visual critique.
- Finalization: At most K = 12 refinement attempts are allowed, after which a passing candidate is finalized through rendering adjustments, mathematical typesetting, and asset inlining.The loop terminates immediately when all blocking checks pass; otherwise, the harness falls back after exhausting the attempt budget.
5 Evaluation Protocol
PosterBench evaluates paper-to-poster systems on a 100-paper, five-discipline Main Track and a shared 10-paper subset using common inputs, rendering, and a frozen evaluator. AutoDesign leads the benchmark and human preference comparisons, while harness attachment improves scores across configurations.
- PosterBench Main Track: 78.32: AutoDesign achieves the highest PosterBench Main Track score, exceeding Claude Design by 7.45 points and OpenDesign by 8.87 points under matched Claude Code and Claude 4.8.Table 1 covers the 100-paper evaluation set and compares design agents, coding agents, and a handcrafted workflow.
- PosterBench-mini Main Track: 81.46: AutoDesign with Codex exceeds the native Codex baseline by 5.59 points, while its Claude Code configuration reaches 74.56 versus 69.55 for the standalone baseline.These results are reported on PosterBench-mini, the shared 10-paper subset used for controlled ablations.
- Harness Contribution: 5.01–19.56 points: attaching the design harness improves PosterBench Score across all seven completed model–code-agent configurations.The largest gain is 19.56 points for DeepSeek V4 Pro with Claude Code; the native Codex–GPT-5.5 configuration improves from 75.87 to 81.46.
- Human Evaluation: 64.0%: AutoDesign has the highest Bradley–Terry human-preference estimate, with tie-adjusted preferences of 61.3% against Claude Code, 63.1% against OpenDesign, and 67.6% against Claude Design.The study includes 933 ranking judgments and three skips, with a 95% interval of 55.2–77.8% for AutoDesign’s estimate.
- Benchmark–Human Alignment: 74.4%: benchmark-preferred posters agree with human decisions when their PosterBench Score gap is at least 20 points, versus 51.9% for 0–3-point gaps.PosterBench combines rule-based spatial, OCR, numeric-grounding, and render-integrity checks with rubric-guided VLM judges, and its score correlates positively with human preference (r = 0.34).
6 Future Directions
AutoDesign’s design pattern extends beyond paper-to-poster generation toward diverse multimodal outputs and a multimodal-in, multimodal-out agentic design system. Future work also connects harness adaptation with model post-training and model–harness co-evolution.
- General multimodal design: Figure 12 envisions a multimodal-in, multimodal-out system that integrates papers, visual evidence, code, data, and human guidance to create medium-specific outputs iteratively.
- Cross-medium generalization: The optimized DesignHarness already extends paper-to-poster generation to slide decks, webpages, and conference videos.These pilot outputs suggest the agentic design pattern is not tied to a single input or output medium.
- Model–harness co-evolution: Recent work connects continual harness adaptation, self-improving harnesses, and model–harness co-evolution, with harness optimization complementing model post-training.Long-horizon trajectories and repair outcomes provide execution-time supervision, while the model supplies reasoning and coding capabilities (Karten et al., 2026; Zhang et al., 2026a; Lee et al., 2026a).
7 Related Work
Prior work spans multimodal generation for posters and other document-derived media, response-level refinement, and agents that accumulate persistent experience. AutoDesign extends the latter direction to evolve a DesignHarness for source-grounded, editable academic artifacts.
- Multimodal design generation: Academic-poster systems study layout, source coverage, text–image alignment, multimodal generation, specialized agents, and visual refinement.These systems include SciPostLayout, deep submodular extraction, Paper2Poster, P2P, PosterGen, PosterForest, and Any2Poster (Pang et al., 2025; Sun et al., 2026; Zhang et al., 2025b; Choi et al., 2026; Vinaykumar et al., 2026).
- Multimodal design generation: Related systems also generate slides, webpages, and narrated videos from papers or other documents.The cited systems include Fu et al. (2022), Zheng et al. (2025), Ge et al. (2025), Yang et al. (2025), Chen et al. (2025), and Zhu et al. (2025).
- Refinement and persistent adaptation: Critics, render diagnostics, and regeneration policies improve current multimodal outputs, while persistent agents retain reflections, executable skills, or other experience.Self-Refine revises the current answer using feedback (Madaan et al., 2023); AutoDesign applies persistent evolution by developing a DesignHarness for source-grounded, editable artifacts.
- Refinement and persistent adaptation: Persistent system updates differ in how they separate learning signals from final evaluation across task-local histories, development splits, and related adaptation settings.RHI (Lee et al., 2026a) keeps its evaluation prompt on the evaluator side while retaining pairwise history as a task-local signal; Recursive Self-Evolving Agents use an independent development split, alongside Continual Harness, Adaptive Auto-Harness, and Live-SWE-agent (Nguyen et al., 2026; Karten et al., …).
8 Conclusion
AutoDesign converts recurring design failures into improvements for future multimodal outputs through persistent learning. Its MetaHarnessOptimizer updates a DesignHarness component while keeping model weights fixed, accumulating design priors and producing editable paper-to-poster outputs.
- AutoDesign turns recurring design failures into improvements to the system that generates future multimodal outputs.
- The MetaHarnessOptimizer aggregates trajectories, source and rendering diagnostics, evaluator feedback, and reference posters to update one DesignHarness component at a time.
- Keeping model weights fixed makes paper-to-poster generation a persistent learning process that accumulates design priors and produces editable outputs for direct use or local revision.
A Supplementary Experimental Materials
The appendix makes the experimental interface inspectable by documenting the controlled comparisons, shared generation instruction, frozen scoring protocol, released records, and matched visual evidence.
- The appendix documents the controlled comparisons and shared generation instruction used in the experiments.
- It specifies a frozen scoring protocol and releases the experimental records.
- Matched visual evidence complements the documented experimental materials.
Appendix guide
Appendix A.1 documents the benchmark inputs and comparison matrix, including source packages and the factors fixed or varied across reported tracks. It also provides the shared system-instruction excerpt and task prompt used for comparisons.
- A.1 Benchmark Inputs and Comparison Matrix: Appendix A.1 specifies benchmark inputs and the comparison matrix for reported tracks.It identifies the source packages used in the benchmark.
- A.1 Benchmark Inputs and Comparison Matrix: The comparison matrix records which factors are held fixed and which factor is varied in each reported track.This structure clarifies the conditions of every comparison.
- A.1 Benchmark Inputs and Comparison Matrix: The appendix includes the system-instruction excerpt and shared task prompt used across compared systems.These materials define the common prompting context for the comparisons.
A.2 AutoDesign Generation Interface
This section describes the staged capabilities accumulated in the optimized design harness and their mapping to a five-component abstraction.
- A.2 AutoDesign Generation Interface: The optimized design harness accumulates capabilities in stages and maps them to a five-component abstraction.
A.3 DesignHarness Evolution … A.6 Additional AutoDesign Poster Demonstrations
The appendix specifies AutoDesign’s controlled generation interface, evolving DesignHarness architecture, PosterBench scoring and released records, system-blind human evaluation, and additional qualitative poster demonstrations. Together, these materials define reproducible inputs, source-grounded artifact requirements, auditable evaluation, and representative outputs.
- A.1 Benchmark Inputs and Comparison Matrix: AutoDesign’s comparison protocol gives every system the same source package and requires one editable poster artifact, with the paper authoritative for claims, numbers, and visual evidence.Table 5 distinguishes complete-system main tracks from controlled tracks that vary only the named factor.
- A.3 DesignHarness Evolution: DesignHarness evolves from planning and critique into a coding-agent system that combines source grounding, specialist support, editable HTML, repair and validation gates, and image-native evaluation.Figure 14 presents this as an implementation-level architecture summary rather than a chronology of individual meta-harness iterations.
- A.4 PosterBench Evaluation Interface: PosterBench evaluates rendered artifacts with seven manually specified, frozen dimensions, separating completed-system scoring from Rmeta feedback used during harness optimization.The benchmark aggregates weighted dimension scores, applies record-level ceilings, and averages capped poster scores across cases.
- A.4 PosterBench Evaluation Interface: PosterBench combines programmatic checks for render integrity, occupancy, OCR readability, and source grounding with blinded VLM judgments conditioned on the rendered poster and compact source context.For sufficiently large batches, style-homogeneity checks may reduce only professional aesthetics; PosterBench-mini is exempt because it has 10 posters.
- A.5 Released Records and Human Evaluation: Released benchmark records identify configuration, source case and discipline, evaluation status, aggregate score, and seven dimension scores, enabling auditing and reaggregation of reported rows.The archive distinguishes fresh evaluations from reaggregated records.
- A.5 Released Records and Human Evaluation: The human study uses system-blind pairwise comparisons covering all six unordered system pairs per paper, with balanced presentation and Bradley–Terry analysis over retained non-skip decisions.The complete roster provides equal paper coverage and a connected comparison graph; ties count as half-wins and uncertainty uses 2,000 crossed paper–reviewer bootstrap resamples.
- A.6 Additional AutoDesign Poster Demonstrations: Additional materials provide matched three-system poster comparisons and four AutoDesign demonstrations rendered from the LongCat-Next, NeRF, Attention Is All You Need, and DDPM papers.The matched comparisons use identical source papers and a common generation prompt.