Source-linked AI summary
VibeJam: A User Study Platform for Web Development with Agents
Nishant Balepur, Connor Baumler, Valerie Chen, Eunsol Choi, Rachel Rudinger, Jordan Boyd-Graber
TL;DR
Coding-agent research relies heavily on offline benchmarks and lacks setup-free platforms for online studies that reflect real user workflows. VibeJam addresses this gap with a customizable browser-based system for website development, and studies found positive usability judgments and higher-quality websites from junior users than agents alone.
Problem
Offline benchmarks dominate coding-agent evaluation, but online assessment that mirrors how users work with agents remains limited by a lack of suitable public, setup-free systems.
Method
VibeJam is an open-source, customizable browser-based platform combining agent, chat, plan, diff-review, preview, and study-evaluation workflows for website development.
Results
Five experienced AI programmers rated VibeJam as fun, easy to use, and similar to commercial tools, while 13 junior programmers produced websites of higher quality than agents alone.
Takeaways & Limitations
VibeJam supports user-centered evaluations of coding agents and makes studies of long-horizon execution, code quality, and AI safety more feasible.
Abstract
from arXiv · showhide
Programming with AI is increasingly agentic, users prompt LLMs to directly edit their code and review the changes, with adoption growing especially for web development tasks. Despite this growth, most NLP work uses offline evaluation and lacks support for online studies, losing insights into how programmers truly use coding agents. We release VibeJam, a browser-based user study platform for users to collaborate with AI agents to develop websites. VibeJam enables agent customization and uses the open-source Aider agent by default, and to mirror downstream use, we add diff review, chat and plan modes, and live website previews. In a pilot study with 55 released, game-based website creation tasks, five experienced AI programmers rate our system as fun, simple, and resembling commercial tools, while 13 junior students use VibeJam to make websites of higher quality than agents in the same task. We open-source VibeJam to spur extensions and support studies on how coding agents can help users.
1 Introduction: Agentic Coding’s Boom
VibeJam addresses the lack of online, user-centered evaluation for agentic coding by providing a customizable browser-based platform that mirrors commercial workflows. Its pilot and realistic studies suggest the system is usable and that junior programmers can produce higher-quality websites than agents alone.
- Offline benchmarks dominate coding-agent evaluation, while existing interfaces lack public, setup-free support for online user studies.This limits evaluation that mirrors how users work with agents in deployment.
- VibeJam is a customizable, open-source platform for agentic coding studies that requires no participant setup.It was initially designed for web development with features such as multiple AI modes and model customization.
- VibeJam mirrors downstream coding-agent use through agent, chatbot, and plan modes, execution traces, diff review, previews, and post-task questions.Researchers can also designate users as annotators to rate submissions.
- Five experienced AI programmers rated VibeJam as fun, easy to use, and similar to commercial tools in the pilot study.The pilot evaluated the platform as a user-study environment.
- 13 junior programmers created 22 game-based websites in VibeJam that surpassed the quality of agents working alone on the same task.These outcomes support VibeJam’s use for user-centered evaluations of coding agents.
- VibeJam is released openly to support extensions and studies of long-horizon execution, code quality, and AI safety.The platform is intended to make these user-in-the-loop research directions more feasible.
2 A Formative Study to Ground VibeJam
A formative study with AI-coding researchers shaped VibeJam’s scope and interface priorities. Researchers emphasized website, debugging, refactoring, agentic editing, review, chat, plan modes, and customization, motivating a focused web-development platform.
- Eight NLP and coding researchers from distinct universities rated the importance of file structure, AI interaction, and configurability choices after reviewing an initial VibeJam demo.Their feedback was used to guide the implementation.
- Website, debugging, and refactoring tasks received high importance ratings, whereas function completion did not.The authors identify this as a gap because current interfaces mainly support function completion.
- The authors scoped VibeJam to one task type and chose web development because it is common among coding-agent requests and highly adopted on GitHub.The cited passages motivate web development as the initial domain rather than full workflow coverage.
- Researchers rated agentic edits as more important than tab autocomplete and highlighted chat, plan, review, and summary features.These priorities motivated an agentic coding system beyond existing autocomplete-oriented UIs.
- All researchers wanted customization, especially selection of the LLM backing agents and definition of custom evaluation metrics.This feedback directly informed VibeJam’s configurable design.
- Overall feedback was positive, and no researcher identified an existing system with similar features.One researcher also suggested potential use for arena-style rankings.
3 VibeJam: An Agentic Coding System
VibeJam is a browser-based agentic coding system for creating websites through coordinated task, AI interaction, review, debugging, and study-evaluation features. Users can work with configurable agent, chat, and plan modes while researchers collect interaction and submission data.
- 3.1 Tasks: VibeJam tasks ask users to produce websites from descriptions, with researchers specifying initial files and task instructions.The platform supports both test-case recreation and creative website-design tasks.
- 3.2 Writing Code with AI Agents: VibeJam provides configurable agent, chat, and plan modes for editing code, answering code questions, and preparing specifications for complex requests.Researchers can configure each AI through an API endpoint to compare designs.
- 3.2.1 Writing Code with AI Agents: AIDER is the default open-source agent, using targeted diff patches and conversation summarization to manage code edits and long context.Users can also clear conversation history to manage context length.
- 3.2.2 Reviewing AI-Generated Code: Chat Mode returns explanations or code snippets without directly editing files, while Plan Mode supports complex requests through a user-adjustable specification.Chat responses include syntax-highlighted snippets for readability.
- 3.2.1 Writing Code with AI Agents: Plan Mode lets users iteratively generate and edit a specification before sending it to the AIDER agent for execution.Users can build from the plan or clear it and restart.
- 3.2.2 Reviewing AI-Generated Code: Users review agent changes as Git diffs, accepting or rejecting edits at file or codebase level before further prompting or manual editing.The reviewed changes are also reflected in website previews and test cases.
3.3 Testing and Debugging Code
VibeJam supports debugging by combining a live website preview with a console that relays console.log() output. Website code runs in a sandboxed iframe without access to the parent interface.
- VibeJam includes a Preview tab for viewing the user’s current website during debugging.The preview helps users identify and resolve software issues.
- A console relays console.log() statements, providing functionality equivalent to Inspect Element in Chrome DevTools.This gives users runtime feedback alongside the live site preview.
- Website code is sandboxed in an iframe without access to the parent interface to mitigate risks from arbitrary user code.The sandbox constrains the code’s access to the surrounding application.
3.4 Code Submission
VibeJam supports structured code submission and evaluation, allowing researchers to collect user responses, annotate websites, define custom metrics, and support repeatable studies.
- Code Submission: Researchers can collect multiple-choice, multiple-select, and free-response answers after users submit code.Questions can be fixed across users or tailored per user through question-generation tools.
- Evaluation: Annotators can inspect task submissions and test users’ websites through live previews.
- Evaluation: Researchers can define custom evaluation metrics such as creativity and code quality for website tasks.The same evaluation infrastructure could also support peer-voting gamification.
- Infrastructure: Setup scripts install dependencies, launch the system, and load JSON task data.Accounts connect activity across sessions, while logs capture prompts, outputs, code decisions, edits, responses, and ratings.
- Task Suite: VibeJam curates 55 game-based website tasks, including replication and open-ended activities designed to be understandable and fun.
4 VibeJam in Practice: A Test Case Suite
Two studies test whether VibeJam resembles realistic coding-agent use and whether users improve website outcomes relative to agent-only baselines. Experienced programmers found the system usable and engaging, while junior users produced stronger game websites than agents alone.
- 4.1 Pilot: Five experienced AI programmers rated VibeJam simple, effective, and similar to commercial tools after creating websites with AIDER.Their study used GPT-4.1 and three game-recreation tasks, with seven Likert questions spanning website perceptions, AI interactions, and experience.
- 4.1 Pilot: Users reported higher-quality websites made more quickly with AI, while finding review and website testing easy.The agent did not follow every prompt and struggled with complex multi-step requests.
- 4.2 In-the-Wild Evaluation: 13 junior programmers created 22 game websites and prompted AIDER 12.85 ± 3.25 times per game.The tasks included platformer, puzzle, and reaction-time games, indicating substantial user engagement.
- 4.2 In-the-Wild Evaluation: User+AIDER teams were compared with agent-only baselines using one prompt or seven additional prompts to control prompting effort.Both baselines used the same AIDER configuration as the user study.
- 4.2 In-the-Wild Evaluation: Even the lowest-scoring user+AIDER teams surpassed both agent-only baselines on enjoyment, creativity, and style.Task adherence was the exception, with most websites receiving perfect scores; author agreement was fair at Cohen’s κ = 0.23.
5 Related Work: Agentic Coding Tools
Related systems support user studies through commercial tools, IDE forks, or online interfaces, but each category limits control, onboarding simplicity, openness, agent modernity, or task scope.
- Commercial Systems: Studies of commercial systems reveal current coding practices but give researchers little agent control and require annotation for data collection.
- IDE Forks: IDE-fork platforms support customized studies but require setup from every participant.VibeJam can instead be hosted completely online to simplify onboarding.
- Online Interfaces: Online interfaces closest to VibeJam may be closed-source, use older agents, or focus on function completion.These limitations distinguish VibeJam’s open, modern-agent, user-study focus.
6 Conclusion: Ways to Jam in VibeJam
VibeJam is presented as an open platform for online, user-centered coding-agent research. The paper highlights future applications in long-horizon execution, code quality, monitorability, and configurable agent design.
- Conclusion: Online studies and researcher surveys establish VibeJam as a system for AI-assisted coding workflows and future user-in-the-loop evaluations.
- Long-Horizon Execution: VibeJam could collect expert traces of task decomposition, accepted changes, and useful follow-up ideas for long-horizon programming tasks.These traces are motivated by the need for intermediate feedback during longer tasks.
- Rewarding Code Quality: VibeJam’s edits and human code judgments could provide training signals for reward models targeting code quality.The paper notes that task-completion rewards may incentivize fallbacks and hard-coded paths that hinder downstream use.
- Monitorability: VibeJam could support monitorability research by training code-review skills, testing review scaffolds, and detecting failures from execution traces.
- Agent Design: Custom-agent experiments in VibeJam could compare language models, planning and clarification, and long-context handling.
A.1 Case Study: Comprehension Questions
VibeJam tests whether LLM-generated, personalized questions can assess users’ comprehension of websites created in agentic workflows. In preliminary results, users answered feature questions more accurately than JavaScript questions, while overall accuracy exceeded chance but remained incomplete.
- 27 multi-select comprehension questions were generated with GPT-5.2 from websites produced in VibeJam.The questions tested which features the websites contained or did not contain.
- Users achieved 0.76 mean accuracy, exceeding the 0.50 random-chance baseline.
- Feature questions reached 0.875 accuracy, compared with 0.625 for JavaScript questions.
- The preliminary results suggest users may not fully comprehend code in agentic workflows.
- Automatic question generation and the questions’ ability to reveal coding behaviors make personalized questions a promising research direction.
A.2 Prompts
VibeJam’s prompts configure multiple agent interactions, including summaries, follow-up ideas, chat, planning, plan summarization, and personalized comprehension-question generation. They constrain outputs by task, project context, requested format, and implementation scope.
- Summary and idea generation: The summary-and-idea prompt asks the agent to summarize code changes and propose three feasible follow-up improvements with probabilities.Ideas must fit the project’s file types and stack and avoid custom assets, external services, or out-of-scope changes.
- Summary and idea generation: Summary instructions require first-person, file-specific, concise descriptions of edits, while idea instructions limit each suggestion to ten words.
- Chat Mode: Chat Mode answers questions using the current codebase and may provide brief code snippets, but it must not apply edits or emit patch blocks.When users want automatic edits, the prompt recommends switching to Agent mode.
- Plan Mode: Plan Mode creates or revises implementation-focused markdown plans using the user request, prior plan, and current project-file context.The plans are intended for later execution by a coding agent and therefore emphasize clear, actionable implementation steps.
- Plan summarization: The plan-summarization prompt reports only additions, removals, or changes between plans in concise first-person plain text.It also hints that users can continue adjusting the plan or click Build to execute it.
- Question generation: The question-generation prompt asks for five existing and five plausible nonexistent features from the user’s HTML, CSS, and JavaScript.It returns the results in separate real_features and fake_features JSON lists.