PaperlayerView exact revision on arXiv

Source-linked AI summary

Interaction-as-a-Judge

Wayne Chi, I-Hung Hsu, Li Chen, Wei-Lin Chiang, Ion Stoica, Anastasios Nikolas Angelopoulos, Ameet Talwalkar, Chris Donahue, Cho-Jui Hsieh

arXiv:2610.17895v2cs.LGcs.AIcs.SE

TL;DR

Current LLM evaluations often inspect code or static artifacts, while interactive applications are commonly assessed through direct interaction. This paper studies Interaction-as-a-Judge, in which computer-use agents interact with applications to produce evaluation signals. Those signals improve preference prediction and requirement verification, and are also used to improve generated applications.

  • Problem

    Current automated LLM judges often rely on static or unimodal inspection, while interactive applications can be difficult to assess from code alone.

    Source: §1 · p. 1

  • Method

    Interaction-as-a-Judge uses computer-use agents’ interaction traces to evaluate applications and provide signals for generation, across preference and correctness tasks.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Interaction-as-a-Judge raises preference-judge accuracy from 65.6% to 69.1% and requirement-verification accuracy from 77.0% to 81.6%, while also improving generation outcomes.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Interaction-based evaluation signals can support generation improvements through iterative repair and reward-model training.

    Source: §1 · p. 3, §7 · p. 12

    Computer-use cost limited most experiments to GPT 5.6 Luna as the computer-use agent.

    Source: §7 · p. 12

Abstract

from arXiv · show

Coding agents are tasked with producing sophisticated interactive applications such as websites, desktop applications, and games. While humans assess these applications interactively, current automated LLM evaluations primarily focus on code analysis and unit tests. This mismatch can cause LLM judges to miss failures that are immediately apparent to human users. We study Interaction-as-a-Judge, an evaluation paradigm where computer-use agents (CUAs) directly interact with an application. We demonstrate that Interaction-as-a-Judge can improve both evaluation and generation of interactive applications. To this end, we collect and release two complementary datasets: $6,858$ pairs of applications with human preferences and corresponding agent traces, and $222$ applications with human-verified requirement labels. Relative to standard non-interactive approaches, we find that Interaction-as-a-Judge improves evaluation in terms of predicting human preferences ($65.6\% \to 69.1\%$) and requirement verification ($77.0\% \to 81.6\%$). We also observe scaling benefits: across both preference alignment and requirement verification, higher-quality interactions improve performance. Finally, through both inference- and training-time experiments, we demonstrate that Interaction-as-a-Judge can also improve the quality of generated artifacts. At inference time, Interaction-as-a-Judge repairs $75.5\%$ of broken requirements within five attempts, compared to $55.4\%$ for code-only self-feedback. At training time, using Interaction-as-a-Judge as a reward signal improves win-rate over the base model by $6.7$pp while reducing both generated code length by $35.7\%$ and build failure rates by $7.2$pp compared to a code-only reward signal.

1. Introduction

Interaction-as-a-Judge uses computer-use agents’ interaction traces to evaluate interactive applications and provide feedback for improving them. The paper studies its effects on evaluation and generation, introducing datasets and reporting gains across both.

1. Introduction

  • Static inspection can miss behavioral and experiential failures in interactive applications, motivating evaluation through direct interaction.An application may compile while animations, transitions, or interaction feedback fail to behave as intended.Source: §1 · p. 1
  • Interaction-as-a-Judge uses computer-use agents to produce multimodal interaction traces that can evaluate an application or provide feedback for improvement.The paper systematically studies how interaction strategy and agent capability affect preference prediction and correctness verification, and applies the resulting signals to inference-time scaling and reinforcement learning.Source: §1 · pp. 1–2
  • ArenaPreferences and ArenaCorrectness provide complementary datasets for studying human preferences and correctness verification.The datasets include 6,858 preference-labeled application pairs with agent traces and 67 game-development traces annotated with 412 requirements across game versions.Source: §1 · pp. 2–3
  • 65.6% to 69.1% in preference-judge accuracy and 77.0% to 81.6% in requirement verification show gains over non-interactive or code-only approaches.Interaction traces also improve trained reward-model accuracy from 68.2% to 70.2%.Source: §1 · p. 2
  • Interaction-as-a-Judge repairs 75.5% of initially broken requirements within five attempts, versus 55.4% with code-only feedback.Interaction-only reward-model training also yields a 6.7pp higher win rate against the base model, reduces build failures by 7.2 percentage points, and cuts code-context growth by 35.7% versus code-only reward training.Source: §1 · p. 3

2. Tasks and Datasets

The paper studies preference prediction and correctness verification as complementary evaluation tasks: applications may satisfy requirements yet differ in usability or visual quality. It introduces datasets for both tasks, with human preference labels and requirement-status labels.

2. Tasks and Datasets

  • Preference and correctness judgments address different qualities: an application can meet functional requirements yet differ substantially in usability or visual quality.The reverse can also occur: a visually preferred application may be functionally broken.Source: §2 · p. 3

2.1. Preferences

  • Pairwise preference judges compare two candidate outputs for a shared prompt; the paper focuses on this task but does not restrict the approach to it.The dataset’s seven categories include games, simulations, reference-based designs, content-creation tools, brand sites, consumer products, and data applications.Source: §2.1 · pp. 3–4
  • ArenaPreferences contains 6,858 pairwise comparisons across seven web-development categories, with prompts, agent traces, final websites, and human preference labels.Its collection filters remove ties and require successful builds, React websites, at least 5% character difference, and user interactions lasting 60 seconds to 10 minutes; PII is de-identified.Source: §2.1 · pp. 3–4

2.2. Correctness

  • Correctness tasks include verifying explicit requirements and discovering requirements before judging whether they work.Game development is the dataset focus because games are highly interactive and challenging software tasks.Source: §2.2 · p. 4
  • ArenaCorrectness contains 67 game-development traces, 828 user turns, and 412 requirements labeled across 222 game versions.Requirements are drawn from user complaints and feedback; annotators verify broken labels against the relevant game version and working labels against reported fixes or complaints not revisited.Source: §2.2 · p. 4

3. Interaction-as-a-Judge

Interaction-as-a-Judge uses computer-use traces to improve downstream evaluation accuracy; its interaction and judging components may be shared or separate systems.

3. Interaction-as-a-Judge

  • Interaction-as-a-Judge combines computer-use interaction traces with a judge to improve downstream evaluation accuracy.The interaction and judge components may be the same LLM, different LLMs, or a mixture of LLMs and other methods.Source: §3 · p. 4

3.1. Interaction

  • Applications are modeled through states, actions, and observations, with standard inputs producing traces of observations, reasoning, and actions.Observations may be screenshots or DOM representations; reasoning may be absent.Source: §3.1 · pp. 4–5
  • Iterative computer-use adapts after each environment transition, but requires a model query at every interaction step.The policy chooses actions from the prior trace and can use new observations to respond to state changes.Source: §3.1 · p. 5
  • One-shot computer-use predicts an entire action sequence from the initial observation, reducing inference cost but preventing adaptation to unexpected outcomes.When application code is available, it can also be provided for planning.Source: §3.1 · pp. 5–6
  • Heuristic computer-use selects actions uniformly from candidate interactions without model inference, but is less effective when meaningful actions are difficult to enumerate.The described policy restricts candidates to actions likely to change the environment.Source: §3.1 · p. 6

3.2. Judging

  • Judging uses off-the-shelf LLMs for preference prediction or correctness verification, or trained reward models for preference prediction.The reward model adds a scalar reward head and uses a Bradley–Terry loss over prompts, winner and loser code, and their interaction traces; sufficient in-domain training can improve accuracy and reduce inference costs.Source: §3.2 · p. 6

4. Interaction-as-a-Judge for Evaluation

The evaluation experiments compare interactive judges with static-evidence approaches for preferences and requirements. They report improved preference alignment and broken-requirement verification, with results also varying by model capability and interaction quality.

4.1. Human Preferences

  • 65.6%→69.1%: Gemini 3 Flash judge accuracy rises from code-only to iterative Interaction-as-a-Judge.The trained reward model reaches 70.8%, outperforming the off-the-shelf LLM judge at 69.1%.Source: Figure 3 · p. 7, §4.1 · pp. 7–8

4. Interaction-as-a-Judge for Evaluation

  • Preference comparisons include code-only, code-and-screenshot, and interactive judges, using full code capped at 32k tokens or only the core App.tsx file.The interaction traces in the main experiments contain 30 actions.Source: §4 · p. 6, §4.1 · p. 7

4.2. Correctness Verification

  • 71.1% versus 63.0%: Interaction-as-a-Judge verifies broken requirements more accurately than a code-only Agent-as-a-Judge.For working requirements, the approaches perform comparably at 92.1% and 91.0%, respectively.Source: Figure 4 · p. 8, §4.2 · p. 9

4.1. Human Preferences

  • 69.0% versus 69.7%: iterative computer-use without code performs slightly worse than a code reward model trained on the full code context.Interaction-only reward-model training is also comparable to a code reward model, and interactive judgments remain robust when code context is reduced.Source: §4.1 · p. 8
  • Iterative computer-use performs best across experiments; higher-quality interaction at inference can compensate for lower-quality training traces, but the reverse degrades performance.The comparison swaps heuristic and iterative traces at training and inference time.Source: §4.1 · p. 8

4.2. Correctness Verification

  • Requirement discovery omits the original requirements list and asks judges to propose broken requirements, a task that remains challenging for both approaches.Source: §4.2 · pp. 8–9, Figure 4 · p. 8
  • Broken-requirement verification rises from 65.2% with GPT 5.6 Luna to 68.0% with Terra and 71.1% with Sol.Source: §4.2 · p. 9

5. Interaction-as-a-Judge for Generation

Interaction-as-a-Judge uses interaction for iterative requirement repair and for reinforcement-learning reward signals. The section reports stronger repair results and improved generation measures compared with code-only approaches.

5.1. Inference-time Scaling

  • Interaction-as-a-Judge repaired 75.5% of initially broken requirements, compared with 55.4% for the code-only Agent-as-a-Judge, a 20.1-percentage-point improvement.The improvement persisted when evaluating with Sol as a code-only Agent-as-a-Judge; Interaction-as-a-Judge also outperformed the code-only judge within its first iteration.Source: Figure 5 · p. 9, §5.1 · p. 10

5.2. Training

  • Training used Qwen-3.5-9B with DAPO and heuristic computer-use on 14k prompts from six categories, excluding games.The 600-example test set was sampled from ArenaPreferences; failures were out of distribution for both reward models, so build or render failures received penalties.Source: §5.2 · p. 10
  • Interaction-only reward training produced 35.7% less code and raised build success from 81.3% to 88.5% versus code-only reward training.Among websites that built and rendered successfully, its win-rate gain over Qwen-3.5-9B was 6.72pp.Source: §5.2 · p. 10, Figure 6 · p. 10
  • The win-rate evaluation was conditioned on successful builds and renders, so it understates the benefit of interaction-based reward training.An interaction-only reward model also permits reinforcement learning where application access is available but code access is not.Source: §5.2 · p. 11

6. Related Work

The related work spans LLM-based judging, computer-use evaluation, and interaction feedback for generation. The paper distinguishes its systematic study of interaction as the central focus from prior work that uses it as one pipeline component.

Related work

  • The paper studies interaction strategies and model capabilities across preference alignment, correctness verification, inference-time scaling, and post-training.It describes this as the first systematic study spanning these dimensions.Source: §6 · p. 11

LLM-as-a-Judge and Agent-as-a-Judge

  • Interaction-as-a-Judge extends LLM- and Agent-as-a-Judge by conditioning judgments on traces from a computer-use agent operating the application.Earlier approaches use language-model judgments or inspect project files and development trajectories; interaction adds evidence of behavior during operation.Source: §6 · p. 11

Interactive evaluation

  • Prior interactive evaluation uses screenshots, replayed gameplay, or agent-generated test cases, while the effects of computer-use capability and interaction method remain unclear.WebDevJudge reports that its standalone GUI-agent workflow underperforms its strongest static judges due to planning and execution errors.Source: §6 · p. 11

Interaction feedback and reward learning

  • Prior generation methods use screenshot feedback or one-shot computer-use assessments, whereas this work trains a reward model on human preferences and interaction traces for DAPO.The cited approaches use model-generated assessments of appearance or functionality as reward signals.Source: §6 · pp. 11–12

7. Conclusion

The conclusion presents interaction traces as evidence for judging preferences and correctness, and as rewards for generation. The appendix passages specify the browser setup, interaction tools, and category-specific heuristic policies.

7. Conclusion

  • Computer-use cost remains a bottleneck; pricing mostly limited experiments to GPT 5.6 Luna, while better and more affordable agents may improve capabilities.The authors leave simultaneous improvement of generation and computer-use models as potential future work.Source: §7 · p. 12
  • The paper studies interaction traces for preference and correctness evaluation, releases ArenaPreferences and ArenaCorrectness, and uses interaction-only rewards for reinforcement learning.It also reports improved preference alignment and requirement verification, plus iterative self-feedback for generation.Source: §7 · p. 12

A.1. Browser Environment and Observations

  • Preference agents operate candidates in separate Chromium contexts with a 1280 × 800 viewport; the main preference experiments allow 30 interaction steps per candidate.Game timing is paused between actions and advances in increments, while other web applications run in real time and settle after actions.Source: §A.1 · p. 15

A.2. Action Space

  • Iterative and one-shot preference agents share the same tools, but iterative agents choose after observing results while one-shot agents execute a preplanned sequence.The action space includes pixel positions and durations, and dispatched observation requests also consume a step.Source: §A.2 · p. 15, Table 1 · p. 16

A.3. Heuristic Interaction; A.4. Games

  • For non-games, the heuristic samples visible enabled controls, with a fallback to pixel and keyboard actions when none are found.For games, it samples from a 12-action space and uses a harness that freezes play between interactions, advancing actions for up to 500 ms.Source: §A.3 · p. 15, §A.4 · p. 16

B. Dataset Details and Evaluation Protocol

The dataset passages describe ArenaPreferences sampling and category coverage, ArenaCorrectness labels and versions, and the summary-statistics table’s dataset scope.

Dataset details

  • ArenaPreferences battles were selected from July 2026, exclude specified countries, require successful upstream builds, and require more than 5% character difference between candidates.Both source records must declare the react-vite-tailwind template.Source: B. Dataset Details and Evaluation… · p. 17
  • ArenaPreferences contains 6,858 battles across seven categories, with training and validation pools of 20,160 and 4,460 battles, respectively.The training pool precedes the evaluation pool in time.Source: B. Dataset Details and Evaluation… · p. 17
  • ArenaCorrectness includes 222 game versions and 412 requirements: 323 labeled broken and 89 working.Versions represent application states after user turns, with repaired variants included for requirements users later confirmed fixed.Source: B. Dataset Details and Evaluation… · p. 17
  • Table 2 summarizes ArenaPreferences per website across 13,716 websites and ArenaCorrectness across 67 traces and 222 game versions.Lines of code exclude lockfiles.Source: Table 2 · p. 17, B. Dataset Details and Evaluation… · p. 17

C. Category Level Results

Across category comparisons, iterative interaction significantly outperformed code+screenshot in half of the comparisons. Simulations and games showed the largest average gains and the most consistent significant improvements.

C. Category Level Results

  • 14 of 28 category comparisons significantly favored iterative interaction over code+screenshot; one significantly favored code+screenshot and 13 showed no significant difference.The comparison used a two-sided paired test with p < 0.05.Source: C. Category Level Results · p. 18
  • +2.81 and +1.74 percentage points were the largest average iterative-interaction gains over code+screenshot, in simulations and games, respectively.Significant gains occurred in four of four approaches for simulations and three of four for games.Source: C. Category Level Results · p. 18
  • Figures 7–10 show per-category preference or reward-model agreement, with confidence intervals over battles or pooled seed–battle outcomes.Overall panels weight categories by battle counts rather than equally.Source: C. Category Level Results · p. 18, Figure 7 · p. 19, Figure 8 · p. 19

D. Reward-Model Training

The reward-model experiments use a scalar reward head and a shared optimization recipe across evidence conditions. Additional experiments vary model size, source-code availability, trace text, and training generators under a separate data-construction and training recipe.

D. Reward-Model Training

  • The reward model fine-tunes Qwen3.5-9B with a scalar reward head at a trailing classification token, training the language backbone and reward head while freezing the vision encoder.The optimization uses pairwise Bradley–Terry loss.Source: D. Reward-Model Training · p. 20
  • The shared recipe uses a 32,768-token sequence limit, global batch size 32, and a frozen vision encoder across evidence conditions.The table identifies the recipe as shared while source-code and visual inputs vary.Source: D. Reward-Model Training · p. 20, Table 3 · p. 20
  • Full-code inputs allow 100,000 source characters before the sequence limit, while App.tsx inputs use the main application file or a largest-file fallback.The full-code pipeline removes lockfiles and orders selected files from smaller to larger, placing the main application file last.Source: D. Reward-Model Training · p. 20

E. Additional Reward Modeling Experiments

  • Additional reward-model studies vary model size, source-code availability, trace text, and training generators using a separate data-construction and training recipe.Their traces come from GPT 5.4 Mini rather than GPT 5.6 Luna.Source: E. Additional Reward Modeling… · p. 21

E.1. Shared Setup and Evaluation Populations

Across three model sizes, interaction-trained reward models outperform code-only models, while trace-text and visual-resolution studies probe which parts of interaction evidence accompany the gains. Agreement rises with longer video prefixes, with much of the increase already present after four frames.

E.1. Shared Setup and Evaluation Populations

  • 1.39, 1.55, and 1.61 percentage points are the Interaction-as-a-Judge advantages over code-only at 4B, 9B, and 27B, respectively.The 4B and 9B means are similar, while 27B is higher; interaction models also exceed screenshot models at each size.Source: §E.1 · p. 21, Table 5 · p. 21

E.2. Actions and Reasoning as Judge Input

  • +0.03 points without video and −0.03 points with video were the mean-agreement changes from adding trace text.These results provide little evidence that recorded actions and reasoning alone explain the improvement in this campaign.Source: §E.2 · p. 21, Table 6 · p. 22

E.3. Spatial Resolution

  • 68.25% to 69.29% was the mean-agreement increase when the video spatial budget rose from 128 to 960 tokens per block.Screenshot agreement stayed between 64.85% and 65.27%; video configurations retained action and reasoning text, so the comparison does not isolate motion.Source: §E.3 · p. 22, Figure 11 · p. 22

E.4. Trace-prefix Length

  • 64.37% with the initial frame rose to 69.29% with the full 26-frame prefix, with much of the gain already present at four frames (67.84%).Later observations contributed smaller gains.Source: Figure 11 · p. 22, §E.4 · p. 23

F.1. Cross-policy Evaluation on the Preference Benchmark

Cross-policy tests find higher agreement with iterative evaluation traces in the reported comparisons, while a separate transfer study finds lower agreement when trained models receive recordings from another pipeline. These results concern selected exploratory evaluations and gaming recording comparisons, respectively.

F.1. Cross-policy Evaluation on the Preference Benchmark

  • 68.85% versus 67.95% aggregate agreement resulted when iterative rather than heuristic traces were supplied to the heuristic-trained model.For the iterative-trained model, replacing iterative traces with heuristic traces reduced agreement from 70.07% to 67.29%; iterative evidence improved every reported gaming row.Source: §F.1 · p. 24, Table 7 · p. 24

F.2. Transfer between Gaming Recording Pipelines

  • 3.12, 2.73, and 2.73 points of mean agreement were lost when recordings were replaced for the 4B, 9B, and 27B models, respectively.The fixed-model comparison uses gaming battles with identical prompts and labels.Source: §F.2 · p. 24, Table 8 · p. 25

G. Reinforcement-Learning Configuration and Evaluation

This section documents the reinforcement-learning setup and evaluation procedures for computer-use agents, including DAPO’s training schedule and failure-aware reward computation, agent prompts, and iterative and one-shot protocols for testing applications and games.

G. Reinforcement-Learning Configuration and Evaluation

  • Training follows standard DAPO settings for 35 steps, equivalent to one epoch.Table 9 describes the code-only and standard interaction-only reward arms’ policy-training hyperparameters, including group-normalized advantages and DAPO-style asymmetric clipping and length penalties.Source: G. Reinforcement-Learning… · p. 26, Table 9 · p. 26

G.1. Reward Computation and Failure Handling

  • Build and render failures receive graded penalties, while successful applications receive the scalar reward-model score minus the configured DAPO overlength penalty.Penalties are −5 for no submitted build, −4.5 for a build failure, −4 for an environment crash, and −3.5 for a render failure; the code-only arm retains the same build and render checks.Source: §G.1 · p. 26

H. Computer-use Agent Prompts

H.1. Iterative Computer-use

  • Iterative prompts direct agents to inspect screenshots, use harness actions to test real application workflows or game mechanics, and continue until features are exercised or the step limit is reached.For applications, agents are told to vary actions across features and verify responses; for games, they should test controls, mechanics, and states rather than stopping after one round.Source: §H.1 · pp. 27, 29, 30
  • Iterative computer-use agents test game controls and mechanics by acting, inspecting updated screenshots, and continuing only while functionality remains to exercise.The step budget is a ceiling, not a target; agents can finish early when they have fully tested the game.Source: §H.1 · p. 32

H.2. One-shot Computer-use

  • One-shot agents use the source code and initial screenshot to plan a deliberate, contiguous sequence of up to the action limit, predicting how each action changes app state.The action limit is a ceiling, and the planned list should include only actions needed to exercise the app, without filler waits.Source: §H.2 · pp. 32–33

Related papers

Loading 2610.17895v2…