PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Frontier-Quality Declarative UI Generation at Small-Model Cost

Yingxiang Yang, Weihang Xiao, Ben Bullough, Tushar Deshpande, Niresh Agarwal

arXiv:2609.04184v1cs.HC

TL;DR

The paper asks whether small models can generate declarative UIs at deployable quality and cost. It compares SFT data strategies, model sizes, and catalog sizes across two UI domains. Augmented training strategies outperform the Full-catalog baseline, with a 4B student approaching teacher quality at much lower per-query cost.

  • Problem

    Whether small models can meet quality needs for declarative UI generation remains unclear, as untuned small models have weak A2UI validity and frontier models raise latency and cost concerns.

    Source: §1 · p. 1

  • Method

    The study compares SFT data-construction strategies, student model sizes, and component-catalog sizes for catalog-conditioned A2UI generation across task- and cloud-management UIs.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    A 4B SFT student recovers ∼98% of teacher semantic and ∼97% of teacher visual quality, while both augmented strategies Pareto-dominate Full-catalog and catalog richness scales quality monotonically.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper recommends Perturbed-catalog at 4B for general dynamic-UI generation, 0.8B for hard latency needs on mostly in-distribution queries with simpler catalogs, and Constrained-GT when binding rate is paramount.

    Source: §6 · p. 8

    Evaluation is single-turn across two domains and relies on automatic judges rather than usability testing.

    Source: §5 · p. 8, §6 · p. 8

Abstract

from arXiv · show

Declarative UI protocols such as A2UI let applications generate interactive UIs by selecting pre-built components from a catalog and binding their props to application data, rather than emitting frontend code from scratch. This contract is attractive for production systems because of safety and consistency. An open question is: can low-latency and low-cost small models achieve the required quality for A2UI-based UI generation? To answer this, we systematically study three controllable design choices for catalog-conditioned A2UI generation: supervised fine-tuning (SFT) data construction method, model size, and component-catalog size. Across two React/TypeScript domains and four base checkpoints spanning two model families (Qwen 3.5 0.8B/2B/4B; SmolLM 3B), we find: (i) a 4B fine-tuned student recovers ~98% of teacher semantic quality and ~97% of teacher visual quality at more than an order of magnitude lower cost than frontier API calls; (ii) both augmented strategies (Perturbed-catalog and Constrained-GT) Pareto-dominate the unaugmented Full-catalog baseline, while specializing on different axes; (iii) even small models can handle and benefit from relatively large component catalog size. We distill these results into practitioner-facing trade-offs and deployment recommendations across the three design choices.

1 Introduction

The paper asks whether small models can generate declarative UIs at deployable quality, given quality and cost concerns with frontier models and weak validity from untuned small models. It studies SFT data construction, model size, and catalog size, finding strong quality and reliability gains from augmented training.

Motivation

  • The open question is whether low-cost, low-latency small models can meet declarative UI quality needs, as untuned Qwen 3.5 4B and 2B parse valid A2UI on 62% and 3% of inputs.The paper frames this against frontier models’ quality alongside concerns about their latency and cost.Source: §1 · p. 1

Our study

  • The study evaluates three SFT strategies, Qwen students at 0.8B, 2B, and 4B, and catalogs from ∼10 to ∼100 components, with a SmolLM 3B replication on a cloud-management UI.The primary task-management domain uses an 86-component React/TypeScript catalog and ∼1,500 utterance-and-data pairs.Source: §1 · pp. 1–2

Main findings

  • A 4B SFT student reaches 3.45/5 semantic and 3.09/5 visual quality, recovering ∼98% and ∼97% of teacher quality, respectively, at over an order of magnitude lower per-query cost than SOTA commercial APIs.Both augmented strategies Pareto-dominate Full-catalog, with Perturbed-catalog favoring average quality and Constrained-GT favoring data-binding and path-resolution reliability.Source: §1 · p. 2

2 Related Work

Prior UI-generation work has focused mainly on frontend code and framework-specific UI workflows, while declarative protocols constrain output to catalog components and data bindings. Small-model SFT for this declarative regime lacks a public benchmark characterization.

LLM-driven UI generation

  • Raw frontend code generation is the most-studied regime, while framework-specific UI code and broader generation, editing, and repair workflows feature in recent benchmarks.Examples include pix2code and DesignBench [21].Source: §2 · p. 2
  • Declarative JSON protocols restrict models to selecting approved catalog components and binding props to data paths, which a client renderer turns into a native UI without arbitrary code execution.The paper studies this regime through A2UI.Source: §2 · p. 2
  • No public benchmark had characterized small-model SFT for declarative UI generation, the gap this study addresses.The stated benchmark gap is specific to the declarative regime.Source: §2 · p. 2

Evaluating generative UI

  • Generative UI evaluation commonly combines parse/render success, visual similarity, and judge-based quality scoring, while declarative protocols also require checks for catalog membership and executable bindings.This study adopts structured-validity checks alongside semantic and visual judgment of rendered outputs.Source: §2 · p. 2
  • OOD evaluation tests whether models can select and bind components added after SFT, rather than only handle paraphrased requests.Source: §2 · p. 2

3 Methodology

The methodology builds catalog-conditioned A2UI training data from application components, requests, and data, then compares three SFT constructions and evaluates contract adherence and rendered quality. It also checks judge calibration against another judge family and human ratings.

3 Methodology

  • A2UI outputs must satisfy syntactic, catalog, and data-binding validity; the controlled pipeline proceeds from catalog extraction and teacher targets through SFT to rendered evaluation.Each data-binding path must resolve against the provided data blob; Figure 1 depicts the stages and semantic, visual, and binding evaluation.Source: §3 · p. 2, Figure 1 · p. 3

3.1 SFT Strategies

  • Perturbed-catalog shuffles catalogs, drops unused components, and renames 20% of names; Constrained-GT retains the full catalog but uses simpler, component-limited targets with milder renaming.Full-catalog is the canonical baseline, showing the full 86-component catalog in prompts; the strategies share training queries.Source: §3.1 · p. 3

3.2 Training and Inference Configuration

  • Students receive LoRA SFT from Qwen 3.5 base checkpoints using ∼1,015 filtered examples, while evaluation uses unified-test examples and expanded-catalog OOD sets.Runs share the split, requests, and data blobs, differing in prompt or target construction; decoded outputs are evaluated against the aligned input data blob.Source: §3.2 · p. 4

3.3 Metrics and Judges

  • Evaluation measures parse/render success, catalog adherence, path-binding correctness, and semantic and visual quality of rendered outputs.Path validity is checked by JSON-path traversal; semantic and visual judges score request satisfaction and rendered layout quality, respectively.Source: §3.3 · p. 4

3.4 Judge Calibration

  • Cross-family judge rankings correlate at 0.98 for semantic and 0.96 for visual quality, supporting broad relative comparisons but not small cell-level differences.Qwen3-VL semantic scores average +1.31 higher, so it serves as a rank-consistency check.Source: §3.4 · p. 5, Table 1 · p. 5
  • Visual judge–human correlation is 0.48 versus 0.46 for human–human agreement, while task-satisfaction correlations are 0.25 versus 0.35.Human agreement on component appropriateness is itself low at 0.18; Table 2 reports the comparison for 50 items and three annotators.Source: §3.4 · p. 5, Table 2 · p. 5

4 Results

Across the reported experiments, augmented SFT strategies outperform Full-catalog on most measured axes, while Perturbed-catalog and Constrained-GT specialize differently. The results also examine model and catalog size and replicate the strategy comparison on another model and UI domain.

4.1 RQ1: Which SFT Strategy Wins, and on Which Axis?

  • Augmented strategies achieve 92.6–93.3% ID and 87.7–98.1% OOD path resolution, versus 50.8% ID and 56.4% OOD for Full-catalog; Perturbed-catalog leads judged semantic quality while Constrained-GT leads binding and OOD visual score.Constrained-GT reaches 98.1% OOD path resolution versus 87.7% for Perturbed-catalog, while Perturbed-catalog has the lowest OOD bad-component rate (0.7%).Source: Table 3 · p. 5, §4.1 · p. 5
  • Full-catalog’s five-seed variation (±0.05 semantic and ±0.03 visual ID; ±0.05 semantic and ±0.02 visual OOD) is below reported strategy gaps; paired tests confirm selected differences.For example, Perturbed-catalog exceeds Full-catalog by +0.45 semantic ID.Source: §4.1 · p. 5

4.2 RQ2: Model Size, Cost, and Quality Trade-Offs

  • The 4B Perturbed-catalog student recovers ∼98% of teacher semantic and ∼97% of teacher visual quality at more than an order of magnitude lower per-query cost than Claude API calls.It sits on the cost–quality Pareto front.Source: Figure 3 · p. 6, §4.2 · p. 6
  • At ∼8.1 s, 4B Perturbed-catalog is faster than Haiku, Kimi, Sonnet, and Opus; the 0.8B student is the fastest deployable point at ∼3.4 s.Sonnet and Opus reach higher absolute quality but take 19–23 seconds per request.Source: §4.2 · p. 6, Figure 4 · p. 7

4.3 RQ3: How Does Catalog Size Shape Student Quality?

  • From 10 to 86 components, semantic quality rises at every student size (0.8B 2.29→2.97; 2B 2.50→3.16; 4B 2.68→3.47), and visual quality also improves.Even the 0.8B model reaches 2.97 semantic and visual quality at C86.Source: Table 6 · p. 7, §4.3 · p. 7

4.4 RQ4: Can We Replicate the Findings on Other Student Models/Domains?

  • On the cloud-management console with SmolLM 3B, augmented recipes again outperform Full-catalog on the main axes, and Constrained-GT reaches 98.6% OOD path resolution; model and domain change jointly.Constrained-GT’s semantic result depends on whether the test workload fits its component budget: task-management page-level queries can exceed it.Source: Table 7 · p. 7, §4.4 · p. 7

5 Discussion and Limitations

The authors argue that the evaluated contracts and mechanisms may transfer to catalog-conditioned protocols, while A2UI-specific costs and untested alternatives bound that transfer. Evidence remains limited to single-turn evaluation in two domains and automatic-judge calibration rather than usability testing.

  • The evaluated contracts and training mechanisms are generic to catalog-conditioned protocols, where perturbation prompts catalog reading and catalog richness expands the output vocabulary.The contracts are syntactic validity, catalog membership, and resolvable bindings.Source: §5 · pp. 7–8
  • A2UI's two-message envelope, path syntax, and catalog prompt-token cost affect absolute cost and latency, but not reported rankings.Source: §5 · p. 8
  • Other protocols remain untested, and grammar-constrained decoding or renderer-side binding resolution could change the measured advantages.Source: §5 · p. 8
  • The evaluation uses single-turn request-response pairs, while deployments often involve multi-turn updates.Each target is also a single teacher generation, so students inherit that teacher's layout idioms.Source: §5 · p. 8
  • Two React/TypeScript domains cannot establish generality across consumer, data-visualization, or forms-heavy interfaces.The studied domains differ in catalog size and page character.Source: §5 · p. 8
  • Judge calibration covers 50 items rated by three team members, and the study is not a usability evaluation.The authors say task-based user testing would be needed to show that surfaces are usable.Source: §5 · p. 8

6 Conclusion

Across two domains and two base models, augmented data construction produces more robust and higher-quality UI than Full-catalog. The authors recommend choices based on catalog richness and whether general quality, latency, or binding reliability is the priority.

  • Across two base models and two UI domains, augmented data construction consistently produces more robust and higher-quality UI than the Full-catalog baseline.Source: §6 · p. 8
  • Student quality scales monotonically with catalog size, so richer component catalogs are preferable whenever available.Source: §6 · p. 8
  • Use Perturbed-catalog at 4B for general generation, 0.8B for hard latency budgets with mostly in-distribution simpler-catalog queries, and Constrained-GT when dynamic-catalog binding is paramount.Source: §6 · p. 8
  • These recommendations rest on single-turn evaluation over two domains and automatic judges rather than usability testing.Source: §6 · p. 8
Loading 2609.04184v1…