PaperlayerView exact revision on arXiv

Source-linked AI summary

Code Aesthetics with Agentic Reward Feedback

Bang Xiao, Lingjie Jiang, Shaohan Huang, Tengchao Lv, Yupan Huang, Xun Wu, Lei Cui, Furu Wei

arXiv:2510.23272v1cs.CL

TL;DR

LLMs remain weak at visually oriented coding, where code aesthetics are underexplored. The paper combines an aesthetics-focused dataset and benchmark with multi-agent reward feedback for training; its AesCoder models improve on both benchmarks and rival substantially larger models.

  • Problem

    LLMs remain inadequate at visually oriented coding tasks, leaving the aesthetic dimension of generated code underexplored.

    Source: §1 · p. 2

  • Method

    The paper introduces AesCode-358K and OpenDesign, then uses GRPO-AR with execution, static-aesthetics, and interactive-aesthetics reward feedback.

    Source: §1 · p. 2, §5.2 · p. 6

  • Results

    AesCoder achieves state-of-the-art results on PandasPlotBench and OpenDesign, surpasses GPT-4o and GPT-4.1, and rivals substantially larger models.

    Source: §7.2 · p. 9, §9 · p. 11

  • Takeaways & Limitations

    The authors conclude that multi-agent reward feedback can align coding LLMs with functional correctness and human-perceived aesthetics.

    Source: §9 · p. 11

    Web-agent failures can lead to interaction scores below their true values, potentially causing incorrect judgments.

    Source: §4.3 · p. 6

Abstract

from arXiv · show

Large Language Models (LLMs) have become valuable assistants for developers in code-related tasks. While LLMs excel at traditional programming tasks such as code generation and bug fixing, they struggle with visually-oriented coding tasks, often producing suboptimal aesthetics. In this paper, we introduce a new pipeline to enhance the aesthetic quality of LLM-generated code. We first construct AesCode-358K, a large-scale instruction-tuning dataset focused on code aesthetics. Next, we propose agentic reward feedback, a multi-agent system that evaluates executability, static aesthetics, and interactive aesthetics. Building on this, we develop GRPO-AR, which integrates these signals into the GRPO algorithm for joint optimization of functionality and code aesthetics. Finally, we develop OpenDesign, a benchmark for assessing code aesthetics. Experimental results show that combining supervised fine-tuning on AesCode-358K with reinforcement learning using agentic reward feedback significantly improves performance on OpenDesign and also enhances results on existing benchmarks such as PandasPlotBench. Notably, our AesCoder-4B surpasses GPT-4o and GPT-4.1, and achieves performance comparable to large open-source models with 480B-685B parameters, underscoring the effectiveness of our approach.

1 Introduction

LLMs remain weak at visually oriented coding, where code quality depends on rendered results as well as functionality. The paper introduces an aesthetics-focused dataset, benchmark, and multi-agent reward approach, and reports gains on existing and new benchmarks.

1 Introduction

  • Visually oriented coding tasks tie code correctness or quality to visual output, yet LLMs remain inadequate at them and code aesthetics are underexplored.The paper notes issues such as overlapping elements, inconsistent color schemes, and disorganized structures.Source: §1 · p. 2
  • The paper frames code aesthetics as the appeal of a visually oriented program’s execution result and proposes agents that assess execution, static visuals, and interaction.The execution agent checks code first; if it passes, the static and interactive aesthetics agents assess the result in parallel.Source: §1 · p. 2
  • AesCode-358K covers Python-based plot generation and webpage design, while OpenDesign assesses webpage visual and interactive aesthetics.The paper combines supervised fine-tuning with GRPO-AR reinforcement learning and reports improved performance on OpenDesign and PandasPlotBench.Source: §1 · p. 2
  • The paper’s stated contributions are the code aesthetics concept, the AesCode-358K dataset, the OpenDesign benchmark, and agentic reward feedback combined with GRPO.The benchmark is described as the first specifically designed to assess webpage design aesthetics.Source: §1 · p. 2

2 Related Works

Related work covers aesthetic evaluation of generated text and images, and reward systems used in reinforcement learning. The paper identifies static-only assessment and reliance on a single reward source as limitations relevant to its focus.

Related work on AI-generated content aesthetics

  • Textual-aesthetics research addresses cleaner layouts and coherence in LLM output [JHWW24], while image-aesthetics research assesses and improves image quality [WHW24].Source: §2 · p. 3
  • Text- and image-aesthetics methods rely on static images and may not assess webpages that require interaction [JHWW24] [WHW24].The passage presents interactive evaluation as increasingly feasible with more mature AI agents [AAA+23] [HLG+24].Source: §2 · p. 3

Reward systems in reinforcement learning

  • RL reward sources include model-based feedback [WXX+24] and rule-based feedback [SWZ+24] [XGR+25].Source: §2 · p. 3
  • The paper notes that relying on a single reward source in complex tasks can induce biased behavior and misdirect optimization.It also cites work combining human-preference rewards with verifiable signals [PQW+25].Source: §2 · p. 3

3 The AesCode-358K Dataset

AesCode-358K is a supervised instruction-tuning dataset for Python plot generation and webpage design. Its construction combines plot-code quality checks with a multi-step process for generating and filtering webpage examples.

3 The AesCode-358K Dataset

  • AesCode-358K targets two areas where visual outcomes and implementation style matter: Python-based plot generation and webpage design.Source: §3 · p. 3

3.1 Python-Based Plot Data Construction

  • The plot-data process regenerates code adapted from VisCode-200K because some snippets are non-executable or have aesthetic flaws such as chart–legend overlap.The regenerated code is quality-controlled with library restrictions and notebook runtime checks; filtering yields 158K plot examples.Source: §3.1 · p. 3

3.2 Webpage Design Data Construction

  • Webpage instructions are generated from keywords across five categories, then clustered to retain representative samples before HTML generation.GPT-5 and Qwen3-Coder-480B-A35B-Instruct-FP8 generate HTML; executable pages are rendered with Playwright and Selenium, and GPT-5 selects the higher-scoring output.Source: §3.2 · pp. 3–4

4 Agentic Reward Framework

The agentic reward framework evaluates code execution, static webpage aesthetics, and interactive aesthetics. Specialized agents provide these assessments, which are integrated into a weighted reward; the authors also note that web-agent failures can affect scores.

4 Agentic Reward Framework

  • Existing coding rewards focus on textual signals and lack visual assessment, while the framework evaluates executability, static aesthetics, and interactive aesthetics.Interactive aesthetics also guards against optimizing static appearance at the expense of functionality.Source: §4 · pp. 4–5

4.1 Execution Agent

  • The execution agent checks whether generated HTML passes basic syntax validation using HTMLHint and assigns a binary execution reward.Source: §4.1 · p. 5

4.2 Static Aesthetics Agent

  • The static aesthetics agent assesses full-page screenshots across instruction alignment, visual elements, and layout and cohesion.GPT-5 scores the dimensions and provides rationales; the agent returns the aggregated score.Source: §4.2 · p. 5

4.3 Interactive Aesthetics Agent

  • The interactive aesthetics agent plans and performs webpage interactions, judging responses from before-and-after screenshots and aggregating action outcomes.Its offline evaluation excludes interactions requiring internet access.Source: §4.3 · pp. 5–6
  • Web-agent failures can assign zero to an interaction and produce scores below the true values, though failures may also reveal non-standard design aspects.Source: §4.3 · p. 6

4.4 Reward Aggregation

  • The framework computes the overall reward as a weighted sum of execution, static-aesthetics, and interactive-aesthetics rewards.Each agent’s reward is multiplied by its assigned weight.Source: §4.4 · p. 6

5 AesCoder Training

The training pipeline pairs supervised fine-tuning on AesCode-358K with GRPO-AR reinforcement learning intended to improve webpage-design generalization.

5.1 Stage I: Supervised Fine-Tuning on AesCode-358K

  • Stage I fine-tunes Qwen3-4B-Instruct-2507 and Qwen2.5-Coder-7B-Instruct on AesCode-358K to test dataset generalizability and establish an RL foundation.Source: §5.1 · p. 6

5.2 Stage II: Reinforcement Learning with Agentic Reward Feedback

  • After supervised fine-tuning, limited generalization beyond training data, especially in webpage design, motivates RL for unseen scenarios.Source: §5.2 · p. 6
  • To avoid overlap with AesCode-358K, RL uses 20K WebSight v0.2 examples with GPT-4o-generated user instructions because source descriptions are homogeneous.Source: §5.2 · p. 6
  • GRPO-AR samples output groups per RL prompt and rewards each output on execution, static aesthetics, and interactive aesthetics.Source: §5.2 · p. 6

6 The OpenDesign Benchmark

OpenDesign automates webpage-aesthetics assessment across static and interactive dimensions. Its model rankings closely agree with Design Arena, and its LLM judgments align with human preferences.

6 The OpenDesign Benchmark

  • OpenDesign provides efficient automated assessment for 840 real-world webpage cases, addressing the time-consuming voting used by Design Arena.Source: §6 · p. 7

6.1 Evaluation Mechanism

  • For each case, the benchmark scores static aesthetics from a rendered image and interactive aesthetics from the prompt and HTML, then averages results across cases.Source: §6.1 · p. 7

6.2 Reliability Analysis of OpenDesign

  • OpenDesign rankings agree with Design Arena: Spearman = 0.98 and Kendall = 0.91; top-3 overlap is 66.7% and top-5 overlap is 80.0%.Source: §6.2 · pp. 7–8
  • Human-human agreement was 68.7% and GPT-human agreement was 80.9%, comparable to MT-Bench results (66% and 70%).Source: §6.2 · p. 8

7 Experiments and Results

AesCoder improves plotting-code reliability and webpage aesthetics, outperforming GPT-4o and GPT-4.1 while remaining competitive with substantially larger systems. Analyses report gains over simpler reward scoring and DPO/RFT, with increasing reward scores across model families.

7.1 Experimental Setup

  • Plot generation was evaluated on PandasPlotBench using head-descriptor and vis-mode cases, with executability, GPT-4o score, and good rate measured.Good rate is the share of scores above 75; OpenDesign evaluates webpage-design ability.Source: §7.1 · p. 8

7.2 Main Results

  • AesCoder models achieved lower PandasPlotBench error rates and higher reliability than their respective baselines.The supplied passage does not include the specific values.Source: §7.2 · p. 8
  • AesCoder-4B surpasses GPT-4o and GPT-4.1 on PandasPlotBench and OpenDesign, while GPT-5 and Claude Sonnet 4 retain a slight overall advantage.Across OpenDesign, AesCoder also improves static and interactive aesthetics and matches or outperforms open-source models with 30B–685B parameters.Source: §7.2 · pp. 8–9
  • Human evaluation showed that the evaluated models consistently outperformed the strong open-source baselines GLM-4-32B-0414 and Qwen3-Coder-30B-A3B-Instruct, further validating the results.Source: §7.2 · p. 9

7.3 Analysis

  • Both Qwen2.5-Coder-7B-Instruct-SFT and Qwen3-4B-Instruct-2507-SFT showed steadily increasing reward scores during reinforcement learning.The paper describes this trend as evidence of stable, informative feedback across model families and sizes.Source: §7.3 · p. 9
  • The controlled variant using only direct textual reward-model scores underperformed the full agentic reward method.The full method evaluates execution, static aesthetics, and interactive aesthetics; the authors say these perspectives provide richer, more reliable feedback.Source: §7.3 · pp. 9–10
  • Figure 4 compares AesCoder-4B with baseline models across General Website, Data Visualization, 3D Design, Game Dev, and UI Component cases.The figure caption identifies the five categories from top to bottom.Source: Figure 4 · p. 10
  • GRPO-AR surpassed DPO and RFT on OpenDesign across static and interactive aesthetics.The comparison used the Stage I checkpoint and the same training data for all methods.Source: §7.3 · p. 11

8 Case Study

The case study compares AesCoder-4B with state-of-the-art models across five OpenDesign webpage categories. AesCoder-4B performs at least on par across all five.

8 Case Study

  • In qualitative OpenDesign case studies using five representative cases from its five categories, AesCoder-4B achieved results superior to or on par with Claude Sonnet 4 and DeepSeek-R1-0528 across all five web design task categories.Source: §8 · p. 11

9 Conclusion

The paper presents code aesthetics, the AesCode-358K dataset, OpenDesign, and GRPO-AR as a pipeline for improving executability, static design, and interactivity. Its conclusion reports state-of-the-art results on PandasPlotBench and OpenDesign, rivaling much larger models.

9 Conclusion

  • AesCoder models achieve state-of-the-art results on PandasPlotBench and OpenDesign while rivaling much larger models.The conclusion attributes the alignment of code generation with functional correctness and human-perceived aesthetics to multi-agent reward feedback.Source: §9 · p. 11

B.1 Keyword Corpus and Instruction Generation

  • The webpage keyword corpus covers General Website, 3D Design, Data Visualization, Game Dev, and UI Component, with 9K seed keywords for General Website and 2.5K for each other category.Table 3 summarizes the keyword distribution.Source: §B.1 · p. 16, Table 3 · p. 16
  • Generating 20 instructions for each keyword produced 400,000 webpage-design instructions for further processing.Source: §B.1 · p. 16

B.2 Semantic Analysis and Deduplication

Instruction embeddings showed substantial overlap across categories before filtering. K-means filtering retained cluster-center-nearest samples and produced a visualization with clearer category boundaries and less overlap.

B.2 Semantic Analysis and Deduplication

  • The raw instruction embeddings showed significant overlap across categories and several dense clusters.The visualization sampled 2,000 instructions per category and used t-SNE with perplexity 30 and 1,000 maximum iterations.Source: §B.2 · p. 16
  • K-means clustering with K = 200K retained the sample nearest each cluster center, reducing the instruction dataset to 200K samples.The refined embedding visualization shows clearer class boundaries and reduced overlap.Source: §B.2 · p. 16, Figure 5 · p. 16

C OpenDesign Benchmark Categories

OpenDesign's category distribution is introduced as 840 cases, while the appendix describes DPO and RFT data construction and training settings. The settings cover supervised fine-tuning, reinforcement learning, and limits on interactive training.

C OpenDesign Benchmark Categories

  • OpenDesign's category-distribution table covers 840 cases, but the supplied caption gives no category breakdown.Source: Table 4 · p. 17

D Implementation Details for DPO and RFT

E Training Settings.

  • Stage I trains for 3 epochs with AdamW, 10% linear warmup, cosine learning-rate decay, a maximum learning rate of 1e−5, batch size 128, and an 8k-token sequence limit.Source: E Training Settings · p. 17
  • Stage II uses VeRL with a constant 3×10−6 learning rate, batch size 64, micro-batch size 8, 64 prompts, and 8 responses per prompt; KL coefficient is 0.001 and ϵ is 0.5.Source: E Training Settings · p. 17
  • Agentic reward weights are 0.1 for executability, 0.8 for static aesthetics, and 0.1 for interactive aesthetics; interactive elements are limited to 3 during training given GUI-agent success rates.Source: E Training Settings · p. 18

F Human Evaluation

This section presents human evaluation of AesCoder and its assessment criteria, including prompts for pairwise, pointwise, and interactive webpage evaluation and a strict rubric covering instruction alignment, visual aesthetics, and structural usability.

F Human Evaluation

  • AesCoder’s human-preference win rate exceeds 55% against mid- to large-scale open-source models and is near 50% against proprietary models.The evaluation used 100 randomly sampled OpenDesign test cases, producing 100 HTML pairs. The open-source comparisons were against GLM-4-32B-0414 and Qwen3-Coder-30B-A3B-Instruct; proprietary comparisons were against Claude Sonnet 4 and GPT-5.Source: Table 5 · p. 18, F Human Evaluation · p. 18

G Agentic Reward Training Curve

Prompt Template for pairwise Evaluation

  • The pairwise evaluator scores instruction alignment, aesthetics and readability, and structural integrity and responsiveness for each webpage, then returns an overall comparison.The criteria are weighted 40, 30, and 30 points, respectively; the prompt calls for discriminative judgments and feedback explaining each evaluation.Source: Prompt Template for pairwise Evaluation · pp. 19–20

Prompt Template for Pointwise Evaluation

  • The pointwise evaluator scores instruction alignment, aesthetics and readability, and structural integrity and cohesion, using a 0–100 scale and professional standards.The criteria allocate 40, 30, and 30 points, and the prompt reserves high scores for nearly flawless work while requiring justification for high or low scores.Source: Prompt Template for Pointwise Evaluation · p. 21

Prompt Template for Interactive Aesthetics Agent

  • The interactive-aesthetics agent plans webpage interactions, tests them offline one action at a time, and rewards only correct, expected feedback.It compares webpage states before and after interactions, including typing followed by applying a search, and avoids repeating actions when the page remains unchanged.Source: Prompt Template for Interactive… · pp. 23–24

Prompt Template for Ablation without Agentic Reward

  • The ablation evaluator weights instructional alignment, visual design and aesthetics, and structural coherence and usability at 20, 50, and 30 points.Its scoring prompt strongly favors polished, detailed design and penalizes placeholders, broken elements, and structural flaws.Source: Prompt Template for Ablation without… · pp. 25–26
  • Structural scores range from 20–30 for flawless organization to 0–9 for severe usability problems that make a page difficult to use or navigate.The intermediate bands are 15–19 for minor issues and 10–14 for major structural problems.Source: Prompt Template for Ablation without… · p. 26
  • Scores above 90 are reserved for flawless pages, while noticeable flaws can trigger scores of 10–30 and unprofessional or hard-to-use designs can receive 0–9.The rubric calls for severe penalties even for a single misplaced element or other imperfection.Source: Prompt Template for Ablation without… · pp. 26–27
  • The rubric assigns up to 20 points for Instructional Alignment, 50 for Visual Design and Aesthetics, and 30 for Structural Coherence and Usability, totaling 100.The output format also requests brief feedback justifying strengths and weaknesses.Source: Prompt Template for Ablation without… · pp. 26–27

H.5 Prompt for keyword

  • The keyword-generation prompt requests 300 diverse, unique website summaries of 10–30 words across varied domains.It requires each summary to describe a distinct website purpose, functionality, or use case and specifies JSON-array output.Source: §H.5 · p. 28

Prompt Template for Data Rewriting in RL

  • The prompt asks users to choose a website category, describe its content, purpose, features, and interactions, then give a 40–60-word HTML-generation instruction without specifying layout or visual details.The listed categories are general website, game development, data visualization, UI component, and 3D design.Source: Prompt Template for Data Rewriting in RL · p. 29

Related papers

Loading 2510.23272v1…