PaperlayerView exact revision on arXiv

Source-linked AI summary

Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models

Xilin Wei, Shengyuan Ding, Xinyu Fang, Tiancheng Gu, Yuchi Wang, Zhixun Li, Ziyu Liu, Yuhang Zang, Weixiang Yu, Cheng Lu, Chonggao Sun, Tong Zhou, Dahua Lin, Hang Zhang, Shuai Bai, Dayiheng Liu

arXiv:2610.35138v1cs.AI

TL;DR

Coding agents need to translate multimodal real-user requests into working revisions, but existing benchmarks have limited coverage of compound session-derived edits across localized and repository changes. Co-Design Bench pairs runnable applications with hidden criteria and evaluates revisions using source and browser evidence with adversarial review. Across ten endpoints, criterion coverage exceeds complete task success, with lower completion on repository edits spanning more change types.

  • Problem

    Existing benchmarks have limited coverage of compound coding-session revisions spanning localized multimodal edits and coordinated repository changes.

    Source: §1 · p. 1

  • Method

    Co-Design Bench pairs real-session revision requests and runnable applications with hidden expert criteria, then evaluates criteria using source and browser evidence and a separate verifier.

    Source: §1 · p. 2

  • Results

    The strongest of ten endpoints passes 69.9% of criteria per task but fully satisfies only 26.4% of tasks, and all ten complete fewer criteria on repository edits spanning three or four change types than on those spanning one or two.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Co-Design Bench provides a common setting for measuring progress toward complete delivery by coding agents on real-user application revisions.

    Source: §6 · p. 9

Abstract

from arXiv · show

Evaluating human-AI co-design requires testing whether coding agents can translate real users' multimodal requests into working application revisions. We introduce Co-Design Bench, a reference-free benchmark of 273 tasks curated from 37,463 deduplicated queries across 26,314 real user coding sessions. Its two tracks, Multimodal Editing (ME) and Repository Editing (RE), cover localized edits with optional image, audio, video, or document inputs and coordinated repository revisions. Tasks span three capability dimensions: Grounded Understanding, UI Transformation, and Functional Implementation. The tasks pair normalized requests and runnable pre-edit applications with 2,331 hidden expert-authored acceptance criteria. Our judge-then-refute protocol grades criteria using source and browser evidence, challenges positive verdicts with a separate LLM verifier, and audits visual and interaction claims without gold code or target screenshots. Under a shared agentic harness, the strongest of ten model endpoints passes an average of 69.9% of criteria per task but fully satisfies only 26.4% of tasks. All ten endpoints achieve lower criterion completion on repository edits spanning three or four change types than on those spanning one or two. Co-Design Bench provides a testbed for studying the gap between partial requirement satisfaction and complete task success under agentic execution for real-user application revisions.

1 Introduction

Coding agents must turn multimodal user requests into working revisions of existing applications, a task involving interacting interface, behavior, and state requirements. Co-Design Bench evaluates these revisions from real coding sessions and finds a gap between criterion coverage and complete task success.

1 Introduction

  • Existing benchmarks leave limited coverage for compound revisions from coding sessions spanning localized multimodal edits and coordinated repository changes.Such requests can couple interacting requirements, so success depends on edited-application behavior across them.Source: §1 · p. 1
  • On 150 WebCompass-edit tasks, qwen3.7-plus rises from 73.6% single-shot to 95.1% agentic completion, while opus-4-8 reaches 99.4% agentic completion.The feature rubric, judge, and render pipeline are held fixed; the figure cautions that cross-benchmark bars use different task pools and protocols.Source: §1 · p. 1, Figure 1 · p. 2
  • Co-Design Bench contains 273 real-session tasks and 2,331 hidden expert criteria across localized Multimodal Editing and coordinated Repository Editing.Tasks pair normalized requests and available context or attachments with runnable pre-edit applications.Source: §1 · p. 2
  • Its judge-then-refute protocol grades hidden criteria with source and browser evidence, then has a separate verifier challenge PASS and PARTIAL verdicts.Visual and interaction claims require screenshots and recorded browser actions, respectively; evaluation aborts if required evidence is missing.Source: §1 · p. 2
  • The strongest of ten endpoints passes 69.9% of criteria per task but fully satisfies only 26.4% of tasks under a shared agentic harness.All ten endpoints achieve lower criterion completion on repository edits spanning three or four change types than on those spanning one or two.Source: §1 · p. 2
  • The benchmark combines real-user revisions, reference-free evidence-based evaluation, and analyses of criterion satisfaction versus complete task success.Its contributions also include analyses of input modality and edit breadth.Source: §1 · p. 2

2 Related Work

Prior web-coding benchmarks cover reconstruction, predefined edits, version-history revisions, and repository implementation requests. Co-Design Bench focuses on bounded, real-session revisions that span localized multimodal edits and coordinated repository changes.

Multimodal web generation and editing

  • Reconstruction benchmarks compare generated pages with references, while generation benchmarks add browser or sandbox tests and IWR-Bench assesses interaction-video reconstruction [Si et al., 2024; Liang et al., 2024; Lai et al., 2025; Chen et al., 2025; Lu et al., 2025; Zhu et al., 2025].Source: §2 · p. 3
  • Editing benchmarks use version-history pairs, synthesized repairs, predefined operations, or conversational multimodal feedback, whereas Co-Design Bench retains a runnable app and session context [Xiao et al., 2025; Li et al., 2026; Lei et al., 2026; Wu et al., 2026].Source: §2 · p. 3

Repository-level coding

  • Repository benchmarks assess issue fixes, visual tests, or freelance requests; Co-Design Bench connects coordinated component, route, state, and data-flow changes with localized edits on a common scale [Jimenez et al., 2023; Yang et al., 2024b; Miserendino et al., 2025].For application revisions, its judge-then-refute protocol combines source and browser evidence with a separate verifier, while evidence retention makes positive credit traceable but cannot eliminate judgment error.Source: §2 · p. 3

3 Co-Design Bench

Co-Design Bench evaluates real-user revisions from runnable starting applications through two tracks: localized multimodal edits and coordinated repository changes. It pairs normalized requests with hidden criteria and curates tasks whose starting environments can be executed and validated.

3 Co-Design Bench

  • The benchmark covers 273 tasks and 2,331 expert-authored criteria across localized edits grounded in multimodal evidence and coordinated repository changes spanning structure, behavior, and content.Source: §3 · pp. 3–4

3.1 Task Formulation

  • Each task pairs a runnable pre-edit baseline and environment with a normalized request, optional attachments and history, and hidden criteria that permit any implementation meeting task-specific constraints.Solvers can inspect, edit, build, and execute the application, while acceptance concerns its final state.Source: §3.1 · p. 4

3.2 Data Source and Curation

  • The corpus retains 37,463 candidates from 39,325 production queries across 26,314 sessions; unrecoverable environments and non-actionable requests are filtered before experts author criteria.Candidates are ranked by structural difficulty, verifiability, and historical failure signals, then consolidated into bounded session intervals; execution does not simulate further user feedback.Source: §3.2 · p. 4

3.3 Multimodal Editing (ME)

  • The 100 ME tasks cover localized visual, structural, and content edits with optional image, audio, video, or document inputs, and include 553 criteria averaging 5.5 per task.Rubrics allow different CSS and DOM implementations; some requests require a value to remain consistent across a source document, an editable row, and dependent summaries.Source: §3.3 · p. 5

3.4 Repository Editing (RE)

  • The 173 RE tasks require coordinated changes across repository components, routes, state, data flow, or content; 166 require multiple edit types.The track contains 1,778 criteria averaging 10.3 per task, and validation checks builds, runtime launch, route reachability, and browser rendering.Source: §3.4 · p. 5

4 Evaluation Protocol

The protocol evaluates the final application against hidden criteria using source and browser evidence, then challenges positive judgments and audits required evidence. Scores distinguish task-level completion from full delivery, while parallel checks diagnose failures for supplementary attribution analysis.

4.1 Evidence and Adversarial Verification

  • Candidates are built and rendered, then graded against hidden criteria using source diffs, rendered pages, and browser traces matched to the claim.Figure 3 depicts the path from artifact to evidence-backed verdict.Source: §4.1 · p. 5, Figure 3 · p. 6
  • A separate verifier challenges PASS and PARTIAL judgments, while evidence audits require screenshots for visual claims and browser-action logs for interaction claims.Audit violations abort evaluation before score aggregation.Source: §4.1 · p. 5

4.2 Scoring and Failure Attribution

  • Task normalization prevents longer rubrics from dominating; Strict Completion, Graded Completion, and Full-Pass Rate distinguish fully satisfied requirements, partial credit, and all-criteria delivery.Graded Completion assigns PASS 1, PARTIAL 0.5, and FAIL 0; each endpoint contributes 100 ME and 173 RE tasks, with combined scores weighting all 273 equally.Source: §4.2 · p. 6
  • For low-scoring cases, baseline and solvability checks distinguish model limitations, environmental blockage, mixed causes, and starting-artifact defects without changing raw scores.Attribution proceeds only when an independently rendered baseline passes the startability gate.Source: §4.2 · p. 6

5 Experiments

Across ten endpoints, criterion coverage substantially exceeds complete task delivery, while performance varies across editing settings and task strata. The analyses also compare execution profiles, task-group outcomes, and robustness across execution regimes, evaluators, and harnesses.

5.1 Experimental Setup

  • The fixed evaluation covers ten endpoints on 273 tasks, reporting strict, graded, and full-pass metrics alongside execution effort and tool use.The shared harness fixes the Claude Code setup, prompts, interfaces, and staging procedures across endpoints.Source: §5.1 · pp. 6–7, Table 2 · p. 7

5.2 Main Results

  • Endpoint rankings vary between tracks, and execution profiles differ, but observed tool-use patterns do not isolate causal effects on completion.ME video tasks have a 44-turn median versus 27 for text-only tasks; task-content differences limit causal interpretation of the profiles.Source: §5.2 · p. 7
  • The strongest endpoint averages 69.9% strict completion across tasks but fully passes only 26.4%; gpt-6-astra leads strict completion in both tracks.On ME, it scores 0.754 strict completion and passes 41.0% of tasks; on RE, 0.667 and 17.9%, respectively.Source: §5.2 · p. 7, Table 2 · p. 7

5.3 Task-Level Analysis

  • Across modalities, pooled strict completion ranges from 0.584 to 0.652; documents score highest in strict completion but have a lower full-pass rate than text-only tasks.Document model–task pairs fully pass at 18.2%, compared with 37.0% for text-only tasks.Source: §5.3 · p. 8, Figure 4 · p. 8
  • On repository edits, tasks with one or two edit types score 0.654 strict completion versus 0.591 for three or four types, a 6.3-point gap across every endpoint.Pooled full-pass rates are similar at 9.1% and 9.7%, while zero-pass rates rise from 0.9% to 5.8%.Source: §5.3 · p. 8, Figure 4 · p. 8
  • Functional Implementation has the highest pooled strict completion (0.619) but the lowest full-pass rate (10.3%) among the three capability groups.Capability groups overlap strongly with track membership, so these results describe performance rather than isolated abilities.Source: §5.3 · pp. 8–9, Table 3 · p. 8

5.4 Ablation and Robustness Studies

  • Ablations compare single-shot with agentic execution, vary the evaluator, and test identical solvers across three harnesses.The reported studies find robust relative rankings across evaluator swaps but strong model-by-harness interactions across scaffolds.Source: §5.4 · p. 9

6 Conclusion

Co-Design Bench evaluates real-user application revisions using runnable starting states and hidden acceptance criteria across localized multimodal edits and coordinated repository changes. Across ten endpoints, criterion completion exceeds full-task success, and the benchmark supports analysis of unmet requirements and future solvers.

6 Conclusion

  • Co-Design Bench covers 273 real-user revisions with runnable starting states and 2,331 hidden acceptance criteria across two editing tracks.The tracks span localized multimodal edits and coordinated repository changes, evaluated with source evidence, browser execution, and adversarial review.Source: §6 · p. 9
  • The strongest model reaches 69.9% mean strict completion but fully satisfies only 26.4% of tasks across ten endpoints.Track-specific rankings, edit-breadth differences, and harness sensitivity identify settings where reliable delivery remains difficult.Source: §6 · p. 9
  • Runnable baselines and criterion-level evidence support analysis of failures in request grounding, change propagation, and final-state verification.The benchmark provides a common setting for measuring progress toward complete delivery while retaining task context and execution evidence.Source: §6 · p. 9

A Effect of Solver Execution on WebCompass-edit

The paired WebCompass-edit comparison finds higher completion with agentic execution, while the benchmark results show that criterion completion and full-task success can diverge. Completion also varies with task grouping, evaluator, and harness conditions.

A Effect of Solver Execution on WebCompass-edit

  • Agentic execution raises mean graded completion from 73.6% to 95.1% on 150 tasks, a 21.5-point gain; all-pass tasks rise from 49 to 104.The corresponding full-pass rates are 32.7% and 69.3%; 46 tasks still retain at least one feature below PASS.Source: A Effect of Solver Execution on… · p. 13, Table 4 · p. 13

B.1 Combined Completion

  • gpt-6-astra leads all three combined metrics, while Qwen3.8-27B ranks fourth overall and second on RE.The combined metrics weight all 273 tasks equally; per-track results distinguish overall completion from track-specific strengths.Source: §B.1 · p. 13, Table 5 · p. 13

B.2 Modality and Edit Scope

  • Document tasks have the highest pooled Graded Completion, while tasks without attachments have the highest Full-Pass Rate; broader RE edits have lower Strict Completion and more zero-pass outcomes.The RE comparison is between tasks spanning three or four edit types and those spanning one or two; the groups differ in requests and contexts, so this is descriptive, not causal.Source: §B.2 · p. 14, Table 6 · p. 14

C Evaluator Sensitivity Check

  • Across four evaluator configurations, kimi-k3 stays ahead of Qwen3.8-27B on ME and behind it on RE, while absolute graded scores shift by at most 0.053.This judge-only sensitivity check supports the reported cross-track comparison but does not establish evaluator invariance.Source: C Evaluator Sensitivity Check · p. 14, Table 7 · p. 14

D Implementation and Resource Accounting

  • Runs face three-hour and 120-turn caps; scoring uses the final workspace, and telemetry records 135,370 tool calls.Earlier behavior earns credit only if it remains present in the final state; timeouts are recorded as infrastructure failures.Source: D Implementation and Resource Accounting · pp. 14–15

E Representative Benchmark Cases

Representative cases show how Co-Design Bench checks localized edits and coordinated repository revisions against dependent updates, interactions, and preservation constraints.

E Representative Benchmark Cases

  • Each case pairs a runnable page and relevant detail with a summarized request and three selected acceptance criteria; full rubrics may contain more criteria.The checks cover requested changes, dependent updates, and explicit preservation constraints; quantitative results use the full task pool and rubrics.Source: E Representative Benchmark Cases · p. 15, Figure 5 · p. 15, Figure 6 · p. 16
  • ME examples test reconciling receipt values across dependent summaries, propagating supplied counts while preserving metadata, and transferring video evidence to one product card without disturbing neighboring content.The common pattern is to establish authoritative evidence, verify requested dependent updates, and check that unrelated content remains stable.Source: E Representative Benchmark Cases · p. 15, Figure 5 · p. 15
  • RE examples combine navigation, content, responsive and accessibility requirements; resource-limited clip editing; or persistent themes and a motion-aware mega menu.Their checks distinguish visible content, executable interaction, and baseline behavior that must remain intact.Source: E Representative Benchmark Cases · p. 16, Figure 6 · p. 16

F.1 Dataset Composition

The benchmark spans varied starting artifacts and overlapping capabilities, while its category analyses and tool-use audits describe evaluation context rather than isolated causal effects.

F.1 Dataset Composition

  • ME includes 79 single-page HTML applications (79%) and 21 bundled multi-file application trees; 171 RE tasks (98.8%) use single-page React applications.A single-page application can still require coordinated edits across several repository files.Source: §F.1 · p. 16
  • ME groups tasks by primary attachment modality, while RE edit types overlap; Table 8 summarizes task counts, criterion density, reference materials, and artifact metadata.Among RE tasks with capture artifacts, the mean is 5.1 captures per task.Source: §F.1 · p. 16, Figure 7 · p. 17, Table 8 · p. 17

F.2 Capability Taxonomy and Multi-Label Coverage

  • 265 of 273 tasks (97.1%) have multiple capability labels, and 174 (63.7%) span more than one high-level dimension.The taxonomy has three dimensions, nine categories, and thirty fine-grained capabilities; primary labels partition tasks while additional labels capture overlap.Source: §F.2 · pp. 17–18, Figure 8 · p. 18

F.3 Category-Level Completion Breakdown

  • Motion and Adaptation has only four tasks, and Grounded Understanding aligns mostly with ME while Functional Implementation aligns mostly with RE.The category results therefore combine capability requirements with editing context.Source: §F.3 · p. 18, Figure 9 · p. 19

G Tool Use and External Perception Models

  • Tool-activity summaries count attachment, extracted-frame, and browser-screenshot reads alongside workspace edits, but do not attribute success to any tool family.Media-processing categories include ffmpeg, OpenCV, and faster-whisper; helper scripts count when their execution is recorded.Source: G Tool Use and External Perception… · p. 18, Table 10 · p. 19

G.1 External Pretrained Models

  • Selected trajectories show external models producing audio-class scores, audio–text similarities, transcripts, and descriptions; the examples are not prevalence estimates.A documented OWLv2 attempt raised a post-processing exception despite a zero shell exit code, before a corrected call returned detection boxes.Source: §G.1 · p. 19, Table 11 · p. 20

H Cross-Harness Sensitivity

Harness comparisons show solver-specific performance orderings and changing relative model scores. The results support reporting the solver and harness together, without attributing differences to individual tool practices.

H Cross-Harness Sensitivity

  • On ME, harness ordering reverses between kimi-k3 and Qwen3.8-27B (Spearman ρ = −1.00); on RE, the correlation is +0.50.Across the three tested harnesses, one solver’s ordering does not reliably predict the other’s.Source: H Cross-Harness Sensitivity · p. 20, Table 12 · p. 20
  • The kimi-k3 minus Qwen3.8-27B gap ranges from +0.027 to +0.127 on ME and from −0.040 to +0.122 on RE.These ranges span 10.0 percentage points on ME and 16.2 on RE.Source: H Cross-Harness Sensitivity · p. 20, Table 12 · p. 20
  • Because the comparison swaps entire harnesses, it cannot identify which individual tool practices account for score differences.The authors recommend reporting model and harness together and holding the harness fixed for direct model comparisons.Source: H Cross-Harness Sensitivity · p. 20

I Data Curation Details

The appendix details how session queries are recovered, filtered, ranked, and consolidated into benchmark tasks. Candidate selection prioritizes coordination difficulty, verifiability, and historical failure signals.

I Data Curation Details

  • The source corpus contains 39,325 query records from 26,314 sessions; deduplication leaves 37,463 candidates, and selected queries are consolidated in chronological session intervals.The figure notation uses Q = q, I = (X, K), and a manifest bundled with B specifies environment E.Source: I Data Curation Details · pp. 20–21
  • Recovered applications pass integrity, dependency, build, container-launch, and headless Playwright checks; candidates needing inaccessible private APIs, authenticated backends, or proprietary SDKs are excluded.Faulty repositories may undergo scripted repair, agentic repair, or structural pruning before the same validation gate.Source: I Data Curation Details · p. 21
  • Deduplication removes exact duplicates and near-duplicates above 0.80 tokenized Jaccard similarity, while deterministic filters reject blocked, unbounded, non-actionable, or non-editing turns.Normalization decodes attachments and replaces local image paths with packaged asset references.Source: I Data Curation Details · p. 21
  • Structural difficulty scores range from localized cosmetic edits at d = 1 to multi-component refactoring and complex edge cases at d = 4–5.Verifiability scores reflect static diffs, rendered visual properties, or dynamic post-interaction transitions; failure signals rise with reformulations or rollback.Source: I Data Curation Details · p. 21
  • Candidates are ranked by structural difficulty, verifiability, and historical failure signals, then sessions with question_total_score ≥10 proceed to consolidation.The selector keeps at most three eligible queries per session, with r(q) > 1.Source: I Data Curation Details · p. 21

J Evaluation Implementation Details

Evaluation runs candidate applications in an isolated browser environment and grades hidden criteria against source and execution evidence. Conditional failure attribution separately probes low-scoring cases without changing raw scores.

J Evaluation Implementation Details

  • The evaluator grades hidden criteria using source diffs, DOM trees, screenshots, reachability logs, and—when active—Playwright actions and network telemetry.The candidate is mirrored into an isolated workspace, built or served, and launched in headless Chrome.Source: J Evaluation Implementation Details · p. 21
  • Visual claims require successful renders and screenshots, while interaction claims require recorded user actions and resulting state transitions.Source handlers or internal state cannot substitute for executed interactions, and missing required evidence invalidates dependent claims.Source: J Evaluation Implementation Details · p. 21
  • Strict Completion counts only PASS, while Graded Completion also gives half credit to PARTIAL; raw and environment-adjusted scores are retained separately.A zero strict score can therefore coexist with partial credit.Source: J Evaluation Implementation Details · p. 22
  • Two frontend experts reviewed 200 task–model outputs, checking criterion verdicts against requirements, rubrics, generated interfaces, and required browser interactions.The audit trail links requests and rubrics to baselines, workspaces, evidence, verdicts, and refutation traces.Source: J Evaluation Implementation Details · p. 22

J.1 Conditional Failure Attribution

  • For low-scoring cases with a startable baseline, a solvability probe tests a minimal patch; attribution leaves raw scores and denominators unchanged.Supplementary environment-adjusted scores grant credit only when infrastructure failures are established.Source: §J.1 · p. 22

Related papers

Loading 2610.35138v1…