Source-linked AI summary
MiniAppBench: Evaluating the Shift from Text to Interactive HTML Responses in LLM-Powered Assistants
Zuhao Zhang, Chengyue Yu, Yuante Li, Chenyi Zhuang, Linjian Mo, Shuai Li
TL;DR
Existing benchmarks do not adequately evaluate LLMs generating interactive HTML applications that follow real-world principles and customized user intent. The paper introduces MiniAppBench and MiniAppEval to benchmark and agentically assess these applications, finding that current LLMs still struggle while MiniAppEval aligns closely with human judgment. The authors position the framework as a reliable method for assessing next-generation interactive systems.
Problem
Existing benchmarks focus on algorithmic correctness or static layout and do not capture whether generated applications embody real-world principles and customized interaction.
Method
MiniAppBench provides 500 real-world, principle-driven interaction tasks, while MiniAppEval uses browser-based human-like exploratory testing across Intention, Static, and Dynamic dimensions.
Results
Current LLMs still struggle to generate high-quality MiniApps, while MiniAppEval aligns closely with human judgments.
Takeaways & Limitations
MiniAppBench and MiniAppEval provide a benchmark and evaluation method for more faithful assessment of interactive application generation.
Takeaways & Limitations
The evaluation uses Gemini-3-Pro-Preview as its driver model, selected for highest agreement with expert annotations among five candidate judges.
Abstract
from arXiv · showhide
With the rapid advancement of Large Language Models (LLMs) in code generation, human-AI interaction is evolving from static text responses to dynamic, interactive HTML-based applications, which we term MiniApps. These applications require models to not only render visual interfaces but also construct customized interaction logic that adheres to real-world principles. However, existing benchmarks primarily focus on algorithmic correctness or static layout reconstruction, failing to capture the capabilities required for this new paradigm. To address this gap, we introduce MiniAppBench, the first comprehensive benchmark designed to evaluate principle-driven, interactive application generation. Sourced from a real-world application with 10M+ generations, MiniAppBench distills 500 tasks across six domains (e.g., Games, Science, and Tools). Furthermore, to tackle the challenge of evaluating open-ended interactions where no single ground truth exists, we propose MiniAppEval, an agentic evaluation framework. Leveraging browser automation, it performs human-like exploratory testing to systematically assess applications across three dimensions: Intention, Static, and Dynamic. Our experiments reveal that current LLMs still face significant challenges in generating high-quality MiniApps, while MiniAppEval demonstrates high alignment with human judgment, establishing a reliable standard for future research. Our homepage is available in miniappbench.github.io.
1 Introduction
MiniApps recast human–LLM interaction as executable, interactive HTML artifacts that must embody real-world principles and customized behavior. MiniAppBench and MiniAppEval address gaps in evaluating whether generated applications satisfy these requirements.
- Motivation: MiniApps shift human–LLM interaction from static text toward rendered HTML responses that externalize knowledge as dynamic, user-facing artifacts.The paper presents HTML, CSS, and JavaScript as components of an executable world model.
- MiniApps: MiniApps require fidelity to real-world principles and customized interaction aligned with user intent.Their structure and behavior are synthesized rather than limited to static content or predefined CRUD workflows.
- Evaluation Gap: Existing code and web-generation benchmarks emphasize algorithmic correctness, visual fidelity, or static layout, leaving principle adherence and interactive reasoning insufficiently evaluated.These benchmarks do not verify whether applications construct the real-world principles implied by user queries.
- MiniAppBench: MiniAppBench distills tens of millions of real-world user queries into 500 balanced, principle-driven, interaction-intensive tasks.The benchmark is designed to test whether LLMs align executable code with implicit user reasoning.
- MiniAppEval: MiniAppEval uses Playwright-based human-like exploratory testing to assess Intention, Static, and Dynamic dimensions without relying on rigid assertions or template matching.Its scoring combines static inspection with dynamic evidence and evaluates functional satisfaction of the user’s request.
- Findings: Experiments find that current LLMs struggle to reliably construct high-quality MiniApps, while MiniAppEval aligns closely with human judgment.The framework is presented as enabling more faithful assessment of next-generation interactive systems.
2 Related Work
Prior work evaluates code, real-world reasoning, and web generation through largely static, passive, or reference-dependent settings. These approaches do not adequately test MiniApps as open-ended interactive systems with customized behavior.
- Code Generation: Traditional code benchmarks assess function-level algorithmic reasoning and functional correctness rather than code as an interactive medium for human–LLM interaction.They do not capture whether models externalize knowledge, reason about real-world principles, or support customized interaction.
- Principle Reasoning and Embodied AI: Principle-reasoning benchmarks use passive textual inference, while embodied-AI frameworks place agents in predefined, immutable environments.Neither setting directly evaluates generating customized interactive applications.
- Web Generation: Early web-generation benchmarks focus on visual-to-code translation, pixel-level fidelity, and static layout reconstruction.These approaches treat web generation primarily as image captioning, translation, or structural alignment with reference designs.
- Web Evaluation: Static code analysis, screenshot comparison, and predefined interaction scripts overlook interaction processes and cannot adapt to open-ended state spaces.Consequently, they are ill-equipped to determine whether an application functions as a consistent dynamic system.
- Agent-Based Evaluation: Reference-dependent evaluators compare outputs with implementations or rank pairs, which is ill-suited to MiniApps admitting multiple equally valid realizations.Customized and open-ended generation weakens the assumption that a single reference or pairwise preference is an adequate criterion.
3 MINIAPPBENCH
MINIAPPBENCH is a 500-task benchmark for evaluating MiniApp generation beyond static layouts, emphasizing real-world principles, customized interaction, and open-ended task fulfillment. Its structured dataset spans domains, subclasses, and balanced difficulty levels, with flexible references supporting fine-grained evaluation.
- Benchmark scope: MINIAPPBENCH comprises 500 tasks evaluating LLMs’ ability to develop MiniApps that adhere to real-world principles and customized interaction.The benchmark is distilled from tens of millions of real user queries collected on a large-scale production platform.
- Data representation: Each task is represented with a user query, domain and subclass taxonomy, structured evaluation reference, and Easy, Mid, or Hard difficulty label.The reference specifies verifiable constraints across Intention, Static, and Dynamic dimensions while remaining a flexible inspection guide.
- Data representation: The benchmark preserves open-ended generation by validating artifacts that functionally satisfy user intent rather than matching a rigid template.Its evaluation reference guides inspection across Intention, Static, and Dynamic dimensions.
- Dataset construction: 1,123 high-quality seed queries were retained after LLM and human filtering, forming the foundation for benchmark construction.The filtering stage reported Fleiss’ κ = 0.75 for LLM agreement and Fleiss’ κ = 0.87 for human agreement.
- Dataset construction: 77 tasks (15.4%) are directly drawn from real online queries, while 423 tasks (84.6%) are LLM-augmented variants preserving the original topic and intent.Augmented tasks expand difficulty and domain coverage while remaining grounded in genuine user seeds.
- Dataset construction: The final dataset uses a balanced difficulty distribution of 30% Easy, 40% Medium, and 30% Hard.This distribution supports fair cross-model comparisons while maintaining challenge and diversity.
4 Agentic Evaluation Methodology
MINIAPPEVAL addresses the difficulty of evaluating open-ended MiniApps by combining browser-based exploratory interaction with static and dynamic evidence. It scores applications across Intention, Static, and Dynamic dimensions, and is designed to assess customized requirements beyond rigid scripts or reference comparisons.
- Evaluation pipeline: MINIAPPEVAL uses an LLM-powered agent and Playwright browser automation to interact with MiniApps and record full interaction trajectories.The agent observes the application, performs actions such as clicking and typing, and records a reproducible trajectory.
- Evaluation dimensions: The framework evaluates each application across Intention, Static, and Dynamic dimensions using structured scores and detailed rationales.Intention measures fulfillment of the user goal, Static checks implementation structure and syntax, and Dynamic evaluates runtime behavior through interaction trajectories.
- Scoring: MINIAPPEVAL combines static analysis with dynamic evidence while treating the evaluation reference as a flexible guide rather than a rigid oracle.Final judgment is based on whether the MiniApp functionally satisfies the user’s request.
- Scoring: MINIAPPEVAL directly evaluates open-ended user requirements, making it suitable for highly customized applications.This distinguishes it from assertion-based or comparative benchmarks.
- Advantages: Static analysis verifies implementation logic against real-world principles, while Playwright programmatic control improves execution efficiency.The LLM-powered evaluator can also leverage broad domain knowledge on specialized tasks.
5 Experiments
Experiments evaluate MiniAppBench under controlled conditions, analyze model performance and cost, and test MiniAppEval against human judgments. Results show substantial generation difficulty, scaling and domain effects, and strong evaluator–human agreement.
- Experimental Setup: Evaluations use deterministic seeds, fixed rendering settings, multiple resolutions, identical prompts, and a unified decoding protocol.
- Main Results: GPT-5.2 achieved the highest average pass rate at 45.46%, while the overall mean across models was 17.05%.
- Main Results: Closed-source models consistently outperform open-source models across difficulty levels, producing a more discriminative benchmark gap.
- Main Results: Pass rates decrease as task difficulty increases, although smaller models such as Qwen3-32B can handle some tasks effectively.
- Main Results: Visualization and Lifestyle exceed 30% pass rates, whereas complex domains requiring specialized knowledge and intricate engineering remain difficult.
- Main Results: Qwen3-235B-A22B reaches 2.88% versus Qwen3-32B at 0.66%, while GLM-4.7 reaches 18.31% versus GLM-4.5-Air at 7.09%.
- Main Results: Performance correlates strongly with token consumption at 0.8433 and moderately with inference time at 0.7387.
- Evaluator Validation: MINIAPPEVAL achieves an average F1 of 92.4% against human judgments, with expert inter-rater reliability of κ = 0.89.
6 Conclusion
The paper introduces MiniAppBench for principle-driven interactive application generation and MiniAppEval for comprehensive automated assessment. Experiments indicate that current LLMs struggle with high-quality MiniApps, while MiniAppEval aligns closely with human judgments.
- MiniAppBench evaluates principle-driven interactive application generation, addressing gaps left by prior benchmarks.
- MiniAppEval is an agentic, browser-based framework for comprehensive and automated MiniApp assessment.
- Current LLMs struggle to generate high-quality MiniApps, whereas MiniAppEval aligns closely with human judgments.
Impact Statement
The impact statement positions MiniAppBench within a broader shift toward dynamic human–AI interaction and describes its benchmark construction and coverage. The benchmark uses categorized, balanced tasks and a representative expert-annotated subset.
- Impact Statement: The work supports a potential transition from static text responses to collaborative creation of dynamic, interactive applications.
- Benchmark Construction: The taxonomy contains six domains: Science, Games, Tools, Humanities, Lifestyle, and Visualization.
- Benchmark Construction: The dataset is rebalanced to avoid excessive concentration in highly frequent categories while retaining sufficient representation.
- Benchmark Construction: The benchmark contains 500 tasks distributed across domains and difficulty levels, with manual review before inclusion.
- Benchmark Validation: The expert-annotated 200-item subset closely matches the full benchmark in domain distribution, difficulty distribution, and representative-model pass rates.
A.3 Dataset Composition and Augmentation
The dataset combines real-user queries with LLM-augmented variants and screens them for customized interaction and executable real-world principles. Records preserve task metadata and three-dimensional evaluation references.
- Dataset composition: 77 queries are direct real-user queries, while 423 are LLM-augmented variants designed to reduce simplicity and domain skew while preserving topic and intent.
- Customized interaction: Queries are retained when they require interaction logic beyond static presentation or generic CRUD workflows.Screening considers interaction mechanics, runtime behavior, and user exploration patterns.
- Real-world principles: Each retained query must involve at least one real-world principle that constrains MiniApp behavior, such as physical laws, temporal constraints, or domain conventions.
- Executable representation: Principles are retained only when they can be represented, executed, rendered, and verified within an offline, self-contained HTML/CSS/JavaScript interface.The decomposition maps state to HTML and variables, rules to JavaScript, and outcomes to visual encodings or UI feedback.
- Data format: Each dataset record contains an index, class, subclass, query, difficulty level, and serialized evaluation reference covering intention, static, and dynamic dimensions.
B.4 Temperature Robustness
Judge temperature has limited effect under threshold 0.8, with temperature 1 achieving the best or near-best F1 across representative generation models.
- Temperature 1 achieves the best or near-best F1 across representative generation models at threshold 0.8.Temperatures 0.5, 0.8, 1.0, and 1.2 were evaluated.
B.5 Evaluation Stability
The evaluation framework uses browser automation, structural and source-code observations, and multi-step actions to assess generated web applications. Its repeated evaluations show high stability and per-query agreement.
- Stability: Across five runs, pass-rate standard deviations remain low and per-query agreement is high, with Fleiss’ κ > 0.90 in all settings.
- Evaluation environment: Playwright-based browser automation enables controlled interaction with web applications through clicks, keyboard input, form submissions, and other GUI actions.
- Observation space: Agents observe both DOM snapshots and source code, combining structural page information with implementation details when available.
- Evaluation pipeline: The pipeline supports automated generation, artifact preparation, launching, and evaluation through a one-click workflow using an OpenAI-compatible API endpoint.
- Scoring: Each evaluation returns intention, static, and dynamic scores in [0, 1] with natural-language reasons used for pass/fail decisions and error analysis.
B.10 Time, Token Consumption, and Step Analysis
Trajectory analysis links evaluation cost to interaction length and context growth. Token use rises mildly with step progression, timing is long-tailed, and prompt tokens dominate completion tokens.
- Token usage increases mildly with step progression, largely because prompt context accumulates across interactions.
- Per-step time intervals show substantial variance and a long-tailed distribution.Figure 7 includes time interval versus step on a log scale.
- Prompt tokens dominate the overall token budget, while completion tokens account for only a small fraction.Figure 7 includes prompt-versus-completion token comparisons.
- These patterns indicate that evaluation cost is primarily driven by interaction length and context growth.
C.1 Experimental Design
The evaluation uses a two-stage double-blind protocol that separates objective page description from consistency judgment. It assesses generated pages through structured visual and interactive descriptions, then scores requirement satisfaction from those descriptions.
- Two-stage evaluation: The workflow consists of two sequential stages: blind page description followed by consistency scoring.Stage 1 receives webpage code and a URL without the user query; Stage 2 uses the description with the query and optional reference.
- Stage 1: Blind Description: Stage 1 produces a structured description of layout, visual elements, and interactive controls without judging requirement satisfaction.The description format records page summaries, layout structure, visual elements, and controls such as sliders and buttons.
- Stage 2: Consistency Scoring: Stage 2 scores consistency from the Stage 1 description rather than direct page access.The evaluator reports a score, reasoning, satisfied requirements, missing requirements, mismatch points, and confidence.
C.2 Experimental Results
Experiments compare double-blind and standard evaluation on graphical queries generated by three models. Double-blind evaluation improves overall agreement and negative-sample accuracy, but is stricter on positive samples.
- Overall results: 84.24% average accuracy was achieved by double-blind evaluation versus 80.00% under the standard setting.The comparison used 55 graphical queries and outputs generated by Gemini-3-Pro, GPT-5.2, and Claude-Opus-4.5.
- Negative samples: 96.33% versus 77.06% accuracy was achieved on manually labeled negative samples under double-blind versus standard evaluation.The result indicates improved sensitivity to failure cases and reduced reliance on contextual assumptions from the original query.
- Positive samples: 60.7% versus 87.27% accuracy was achieved on positive samples under double-blind versus standard evaluation.The authors interpret this lower positive-sample accuracy as evidence of a stricter evaluation standard.
- Interpretation: The results suggest that double-blind evaluation improves agreement with manually verified labels while reducing contextual evaluation bias.The two-stage separation prevents Stage 2 evaluators from accessing the original webpage and encourages judgments grounded in structured observations.
D Prompts
The prompts constrain MiniApp generation toward genuinely interactive, runnable, self-contained web applications. They also specify evaluation-oriented requirements while withholding exact reference-construction prompts to preserve benchmark robustness.
- D.1 Prompts for Generating MINIAPPS: The React edition requires a runnable React, TypeScript, and Vite project whose files follow a specified directory structure.The prompt restricts which files are generated and specifies configuration requirements for Vite, TypeScript, PostCSS, and autoprefixer.
- D.1 Prompts for Generating MINIAPPS: The generation prompts impose technical constraints such as avoiding fonts.googleapis.com and using accessible alternatives when fonts are needed.The restriction is motivated by inaccessibility in Chinese networks.
- D.1 Prompts for Generating MINIAPPS: The prompts require pure frontend, self-contained functionality without backend APIs or external services, using browser-native storage and media capabilities when needed.They also require mock-data options and prohibit external data services and AI or LLM calls.
- D.1 Prompts for Generating MINIAPPS: The prompts define interactive applications as systems where users change variables and observe corresponding result or interface-state changes.They explicitly exclude simple content folding, navigation switching, and pagination as sufficient interactivity.
- D.1 Prompts for Generating MINIAPPS: The HTML-generation prompts require a complete, independently runnable page with semantic HTML and responsive desktop and mobile design.CSS and JavaScript may be inline or supplied completely as external files, and the page must run directly in modern browsers.
- D.4 Prompts for Building Evaluation Reference: The evaluation-reference prompts are withheld because disclosure could encourage benchmark-targeted optimization and reduce evaluation reliability.The authors may share additional details later under controlled disclosure settings that preserve benchmark integrity.