FlowCheck: Helping End-Users Specify and Verify Intent in Vibe-Coded Web Apps

• Silent behavioral failures are difficult to detect because apps may compile, render normally, and show plausible feedback while violating user intent; existing tests require users to encode expected results as executable oracles.LLM-based debugging is probabilistic, while traditional static analysis is designed for programmers rather than vibe coders.Source: §1 · p. 1
• FlowCheck lets users express constraints over visible interface elements, actions, and effects, then compiles them into CodeQL queries checking control- and data-flow relationships.Constraints cover outcomes such as updating a component, deriving values from inputs, persisting state across reloads, or triggering an API call.Source: §1 · p. 2
• A formative study across four apps, three coding models, and four iterations identified 23 distinct silent failures, 48% of which were fully expressible and checkable with static constraints.Source: §1 · p. 2
• 100% detection accuracy across 30 injected failures in four web apps exceeded the best LLM baseline, which detected at most 26/30 (87%).The apps had 14–21 constraints each.Source: §1 · p. 2
• Agents can author valid FlowCheck constraints, and using them to check code raised agents’ bug-catch rate on a test app, offering an early sign the language is learnable and usable by LLMs.Source: §1 · p. 2

2.1 Evaluating LLM-Generated Applications

• FlowCheck targets silent failures where plausible interfaces do not deliver information to expected states or elements, grounding correctness in user expectations unlike prior semantic failures.Coding agents struggle as requirements grow across front-end development [31] and backend correctness constraints [7] [33].Source: §2.1 · p. 2
• Existing benchmarks use predefined oracles or assess repository patches, while vibe-coding benchmarks use browser agents or LLM judgments; FlowCheck targets interface expectations with deterministic checks [9] [30] [10] [1] [31] [19].Its formative study characterizes failures, and FlowCheck checks the expected information flows users state.Source: §2.1 · p. 2
• Test generators such as EvoSuite and TestPilot [26] may derive assertions from existing code, whereas FlowCheck constraints provide an external oracle for intended behavior.When implementations contain silent failures, generated oracles may reflect the faulty behavior rather than detect it.Source: §2.1 · p. 2

2.2 Specifications and LLM Debugging

• Unlike specification tools using formal code abstractions [16] [27] [35] [37], FlowCheck uses a less expressive language of visible actions, elements, and flows that users can inspect and LLMs can generate.Constraints can be authored and displayed within the application interface.Source: §2.2 · p. 3
• LLM debugging uses self-revision [5], traces or semantic context [34], and tool-using agents [3], but repairs can be inconsistent and feedback limited [23]; FlowCheck leaves visible, independently checkable constraints with deterministic analysis.Source: §2.2 · p. 3

2.3 End-User Web Testing and Programming

• Dynamic web testers check interface states or DOM invariants, while Quickstrom uses runtime temporal logic; FlowCheck statically checks flows without execution, excluding runtime-only properties.Examples of excluded properties include external API results and dynamically generated code.Source: §2.3 · p. 3
• End-user tools expose program state and support assertions, artifact selection, questions, or trigger-action rules; FlowCheck instead begins from interface-visible intent and mechanically checks data- and control-flow constraints [21].Prior work identifies difficulty stating intent and evaluating generated code, and has addressed the abstraction gap by making code legible or visualized.Source: §2.3 · p. 3

3 Formative Study: Vibe Coding Failures

• 23 distinct failures appeared across four applications and three coding systems, and every application and system exhibited at least one.Source: §3 · p. 3

3.1 Methodology

• The study compared Claude Opus 4.7, DeepSeek V3, and Lovable across four scripted stages, manually checking each saved version against prewritten expectations.Prompts were fixed across systems, and applications were requested in plain JavaScript without frameworks.Source: §3.1 · p. 3
• The four applications covered speaker scheduling, task boards transformed into dependency-aware whiteboards, shopping, and image galleries.Source: §3.1 · p. 3

3.2 Observed Failures

• Observed failures included incorrect visible outputs, dropped constraints, missing action effects, and broken relationships after feature changes or refactoring.Examples included thumbnails opening the wrong image, promo totals ignoring cart contents, lost speaker-slot and task-state restrictions, and archive or swipe actions that did not work as advertised.Source: §3.2 · p. 3
• After task boards became whiteboards, all three systems hid task status, while DeepSeek’s dependency editor rewrote an existing edge when a second was added.Adding A→B and then A→C silently changed the graph to A→C→B.Source: §3.2 · p. 4
• Persistence failures either associated saved favorites with the wrong images or left Lovable’s speaker list and schedule unavailable after reload.Favorites were keyed by position rather than image identity; the scheduler saved neither list nor schedule despite the prompt’s “my speaker list” wording.Source: §3.2 · p. 4
• Public vibe-coding transcripts showed similar failures, including missing database tables for search and history interfaces and a game stuck loading when an asynchronous result never reached the interface.Source: §3.2 · p. 4

3.2.1 Summary of Findings.

• 11 of the 23 failures shared missing or incorrect relationships between user actions and observable effects, motivating an interface-grounded constraint language.These relationships span event handlers, control flow, DOM updates, and storage, making them non-local and difficult to reason about from code fragments.Source: §3.2.1 · p. 4

System Overview

• Users select interface elements, actions, and storage state, then specify expected flows while interacting with the app normally, without inspecting code.Constraints can also be generated by an agent.Source: §4 · p. 4
• The workflow preprocesses code to add element identifiers, authors constraints through the running app or an agent, and compiles them into queries that return pass/fail results.The paper assigns these steps to Sections 6.1, 6.2 and 6.4, and 5.5, respectively.Source: §4 · p. 4
• CodeQL models source code as relational representations of syntax, control flow, and data flow; queries over this database and preprocessed HTML identify matches that violate expected behavior.HTML is preprocessed separately to identify interface elements.Source: §4 · p. 4

5.1 Design Principles

• The design restricts constraints to observable elements and single-action cause-and-effect statements, and to properties that static analysis can check.Checks requiring execution, such as confirming an API’s actual value or detecting race conditions, are out of scope.Source: §5.1 · p. 4

5.2 Abstractions

• Constraints express whether an expected event must always occur or must never occur after a condition, corresponding to all-paths and no-paths checks.The abstractions use readable and writable components, reads and writes, and optional value or source requirements; components include UI, database, and API elements.Source: §5.2 · pp. 4–5

5.3 Grammar

• The grammar combines user or system actions, reads and writes, API calls, persistence, and nested logic, with identifiers classified as UI, storage, or API components.Semantic analysis rejects user actions or bare guards on the event side and multiple actions on the condition side.Source: §5.3 · p. 5

5.4 Expressiveness

• 29 of 34 constraint cases are statically checkable: 21 fully and 8 partially, with exact-value and timing checks left for future dynamic analysis.The cases span UI, API, and database constraints; partial checks verify control- and dataflow paths but defer exact-value checking to runtime.Source: §5.4 · p. 5

5.5 From Constraints to CodeQL Queries

• Each constraint compiles into CodeQL primitive queries whose results are combined according to the constraint’s probability and logical structure.P=1 checks that the event occurs on every path, while P=0 checks reachability and passes when the event cannot be reached; writes, calls, and persistence use corresponding primitives. In the promo example, the constraint fails because some paths do not write the total.Source: §5.5 · pp. 6–7

5.6 Why We Chose This Design

• FlowCheck favors user-visible expectations over formalisms that require non-developers to specify DOM state and internal code names.The paper notes that Hoare logic and Linear Temporal Logic can be more expressive but less accessible to non-developers.Source: §5.6 · p. 7

6.1 Preprocessing

• Preprocessing assigns missing elements unique cv_nnnn identifiers, giving UI elements static references across HTML and JavaScript.FlowCheck parses HTML with BeautifulSoup and uses regular expressions to find programmatically created JavaScript elements and append missing IDs.Source: §6.1 · pp. 7–8

6.2 User Interface

• Users author constraints through an overlay by selecting visible components and detected APIs or storage, then expressing action-to-update relationships with optional logic operators.FlowCheck records these actions, converts them to its constraint language, and sends them to the parser; users need not write constraints or identify internal variables.Source: §6.2 · p. 8

6.3 Analysis

• FlowCheck parses constraints into an AST, checks their semantics, selects primitive analyses, and executes corresponding CodeQL queries to return pass/fail results and failed queries.Semantic rules require valid identifiers, an action on the condition side, a valid effect on the event side, and probability 0 or 1.Source: §6.3 · p. 8

6.4 Authoring Constraints

• Constraints can also be generated by an LLM or agent given the code, app description, and language guidance, and combined with user-authored constraints for broader coverage.The guidance includes the grammar, constraint types, and examples; the paper reports that models can author valid constraints and use them to catch more bugs.Source: §6.4 · p. 8

7.1.1 Test Applications.

• The evaluation used four JavaScript web apps modeled on Amazon, Twitter, Airbnb, and Slack, with IDs on all elements and local storage to keep them within static-analysis scope.The apps were iteratively extended with features that created opportunities to specify constraints.Source: §7.1.1 · p. 9

7.1.2 Ground Truth and Injected Violations.

• Researchers specified 14–21 constraints per app, then added 7–8 subtle violations per modified app, breaking about half the constraints and creating 30 violations overall.The bugs included missing branch writes, missing switch defaults, wrong-source writes, and disconnected components; apps ranged from roughly 300 to 900 lines of code.Source: §7.1.2 · p. 9

7.1.3 Methods.

• The LLM comparison tested Claude Opus 4.7, DeepSeek V3, and Gemini Pro with three increasingly detailed prompts, using one run per model-prompt combination.The prompts progressed from a general bug question to a feature list, then a feature list plus an edge-case checklist.Source: §7.1.3 · p. 9

7.2 Results

• FlowCheck caught all 30 injected violations; the best LLM pair, Claude P3, caught 26/30, and models struggled with bugs spanning branches or handlers.Figure 6 compares FlowCheck with Claude, DeepSeek, and Gemini across three prompt variants.Source: Figure 6 · p. 9, §7.2 · p. 9, §7.1.2 · p. 9

7.2.1 What Models Catch.

• Models reliably caught localized bugs, including incorrect literals, missing writes on a conditional branch, wrong data sources, and partially persisted values.These bugs were common in Airbnb and did not require control-flow analysis or reasoning across handlers.Source: §7.2.1 · p. 9

7.2.2 What Models Miss.

• Models did not reliably detect universal-quantifier violations requiring all branches to be checked or cross-handler flows whose bugs emerge only when handlers interact.More detailed prompts did not consistently help; models sometimes trusted code and justified a broken branch as intentional.Source: §7.2.2 · pp. 9–10

8.2 Future Work

• The implementation checks P=1 and P=0; exact probability values and timing bugs require runtime data, motivating a proposed hybrid of static analysis and runtime tracing.The paper notes that runtime traces introduce variance because results depend on browser-agent paths.Source: §8.2 · p. 10

7.2.3 Agents and our Constraint Language.

• With FlowCheck constraints, models found more gold bugs than with their best prompt: Claude rose from 5 to 6, DeepSeek from 3 to 5, and Gemini from 3 to 4 out of 7.For Q1, each model generated 30–60, 50–86, and 15–20 constraints per run for Claude, DeepSeek, and Gemini, respectively; validity rates were 34–44%, 41–65%, and 45–100%. Framework-rendered interfaces and drag-and-drop interactions can also be difficult to check statically.Source: §8.2 · p. 10, §7.2.3 · p. 10

8.1 Limitations

• Because CodeQL analyzes code without running it, FlowCheck cannot link JavaScript to runtime-created DOM elements, such as elements built through strings.For example, IDs inside a string assigned to `container.innerHTML` are hidden from the analyzer.Source: §8.1 · p. 10
• A planned user study will assess FlowCheck’s language usability, users’ interpretation of analysis results, and integration into their coding workflow.Source: §8.1 · p. 10
• FlowCheck checks constraints on single code versions; future work would track and evolve constraints as apps grow through iterative vibe coding.The proposed next step is to let coding agents continuously generate, check, and refine constraints and code throughout development.Source: §8.1 · p. 10

9 Conclusion

• FlowCheck caught all 30/30 injected violations, compared to at most 26/30 for the strongest LLM baseline.The evaluation also found fewer violations caught by the other LLM baselines.Source: §9 · p. 10
• FlowCheck parses each constraint, classifies its type, maps it to CodeQL queries, and executes those queries against the code.Source: §9 · p. 10
• Users of any programming background can point at a web app to express and deterministically verify expected behavior without reading or understanding its code.Source: §9 · p. 10

A.1 Constraint.g4 File

• The grammar layers Boolean expressions through OR, XOR, AND, and NOT, and includes a P(...) probability-expression form.Atomic logic forms include write events, user actions, system events, persistence events, guards, and Boolean literals.Source: §A.1 · p. 12
• The grammar provides three w() forms for checking existence, a value derived from an expression, or values from an exact set of write events.The exact-set form accepts a list of source items or an empty set.Source: §A.1 · pp. 12–13
• Other atom forms represent reads, API results, API calls with optional expressions, and persistence checks.The grammar also includes A(identifier) as an atom.Source: §A.1 · p. 13
• Expressions support comparisons and range membership, arithmetic with addition, subtraction, multiplication, and division, and forms for lengths, statuses, function calls, and literals.Expression atoms include reads of identifiers or API results, their lengths, status(identifier), and f(expr).Source: §A.1 · p. 13
• Comparison operators are =, !=, <, >, <=, and >=; ranges are numeric intervals or D, and literals include numbers, strings, null, and Booleans.The lexer recognizes NOT, AND, OR, XOR, and in in their specified symbolic or textual forms.Source: §A.1 · p. 13
• Identifiers may begin with a dot so class selectors such as .wishlist-heart parse as one token and bind downstream to dynamic per-instance elements.The grammar also defines numeric and quoted-string tokens and skips whitespace.Source: §A.1 · p. 13

A.2 Parts of CodeQL Primitives

• The appendix points readers to GitHub for the full CodeQL queries.The passage provides the FlowCheck repository address.Source: §A.2 · p. 13
• The isElementRef predicate identifies elements by id through direct document.getElementById calls or references to cached results.The query checks both the direct call and a variable initialized by document.getElementById(id).Source: §A.2 · p. 13
• FlowCheck treats property assignments and calls to selected DOM-mutation, classList, and style methods as writes to an element.The listed DOM methods include appendChild, append, prepend, insertBefore, replaceChild, replaceChildren, insertAdjacentElement, removeChild, remove, and setAttribute; style methods are omitted for space.Source: §A.2 · pp. 13–14

A.3 LLM Rationalizations of Injected Bugs

• With the most detailed prompt (P3), models often located an injected bug but dismissed it rather than flagging it.Examples included treating quantity paths as unreachable or product-specific and an enterprise branch as dead code.Source: §A.3 · p. 14
• The examples of model rationalization came from the Amazon app: quantity bounds were treated as making paths unreachable or product-specific, and USER_TIER as making an enterprise branch dead code.The cited explanations concerned qty=0, a qty=99 cap, and USER_TIER initialized to 'free'.Source: §A.3 · p. 14
• The paper records receipt on 2026-07-11 and acceptance on 2026-08-14.Source: §A.3 · p. 14