PaperlayerView exact revision on arXiv

Source-linked AI summary

ABRA: Agent-Bound UI for Human-Agent Communication beyond Chat with Reusable Views

Seon Gyeom Kim, Hannah Kim

arXiv:2610.30780v1cs.HCcs.MA

TL;DR

Agent outputs are increasingly rich, but user input remains largely chat-based, limiting direct expression of intent tied to rendered representations. Agent-Bound UI separates reusable interactions from task-specific meanings and publishes bindings as shared state; the evaluated agents effectively operated over those published semantics.

  • Problem

    Rich agent representations are rarely used to express user intent, while chat remains dominant even when intent is tied to a rendered representation.

    Source: §1 · pp. 1–2

  • Method

    Agent-Bound UI separates reusable interaction structures from task-specific meanings, which agents bind and publish as shared canvas state.

    Source: §1 · p. 2, §3 · p. 3, §3.3 · p. 4

  • Results

    The evaluated LLM agents effectively operated over published interaction semantics, providing a foundation for richer human-agent interaction and cross-agent collaboration.

    Source: §7 · p. 7

  • Takeaways & Limitations

    Published task-specific bindings let humans and other agents inspect interaction meanings and use shared surfaces across tasks.

    Source: §3.3 · p. 4

    The paper has not evaluated how users interpret published semantics or how agent-bound interactions affect user experience and task outcomes compared with chat.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Agents can create rich representations, yet these representations are rarely used as a medium for expressing user intent. We present Agent-Bound UI, an interaction model that separates reusable interaction structures from task-specific meanings. Agents bind task-specific meanings to interactions in reusable views and publish the resulting bindings as shared state, enabling humans and agents to understand and act on shared surfaces whose meanings vary across tasks. We implement ABRA, a framework-agnostic prototype of Agent-Bound UI, in which users and agents collaborate through chat and shared interactive canvases. We demonstrate ABRA through a representative use case and evaluate four LLM-backed agents on canvas authoring, owner-side consequence resolution, and cross-agent invocation. The code and demo video are available at https://github.com/megagonlabs/abra.

1 Introduction

Agent outputs have become richer and more structured, while user input remains largely chat-based, creating an asymmetry when intent is tied to rendered representations. Agent-Bound UI addresses this by letting agents bind and publish task-specific meanings on reusable views, and ABRA demonstrates the model in a shared-canvas prototype.

1 Introduction

  • Rich agent representations contrast with chat-dominant user input, leaving a representation–interaction asymmetry between structured outputs and natural-language input.When intent is difficult to verbalize or closely tied to a rendered representation, text-only interaction can be inefficient or ambiguous.Source: §1 · pp. 1–2
  • Tasks involving recurring representations but task-varying interaction meanings call for separating reusable interaction structures from task-specific semantics and publishing bindings as shared state.Observable bindings let participants inspect interaction meanings, including when another agent uses a surface created by the first.Source: §1 · p. 2
  • Agent-Bound UI turns reusable interactive views into task-specific communication surfaces by allowing agents to bind and publish interaction semantics.The approach uses representations already rendered on screen as surfaces users can manipulate, grounding interaction in the objects agents produced.Source: §1 · p. 2
  • ABRA is an open, framework-agnostic prototype for users and multiple agents to collaborate over shared interactive canvases, demonstrated through a case study and experiments.The paper presents the prototype alongside the Agent-Bound UI interaction model.Source: §1 · p. 2

2 Related Works

Prior interfaces either generate task-specific interactions, assemble interfaces from reusable components, or target particular workflows; Agent-Bound UI instead separates reusable interaction structure from task-specific meaning.

2.1 Generative and Composable Interfaces

  • Generative interfaces encode manipulation meanings in generated code, while composable interfaces reuse components but often require task-specific components or logic for richer representation-level interactions.Agent-Bound UI keeps interaction structure reusable and places adaptation in task-specific semantic bindings.Source: §2.1 · p. 2

2.2 Task-Specific Interaction Surfaces

  • Task-specific systems support rich manipulation but fix both the interaction surface and its meanings to a target application.Here, “generative” refers to interfaces generated as a medium for human-agent communication, excluding generated applications that are themselves task deliverables.Source: §2.2 · pp. 2–3

2.3 Agent–UI Communication Protocols

  • Agent–UI protocols transport state, messages, and events but do not define interaction meanings; Agent-Bound UI supplies that layer independently of transport.Bindings and events are ordinary messages and state, so canvases can use these protocols subject to their capabilities.Source: §2.3 · p. 3

3 Agent-Bound UI

Agent-Bound UI pairs reusable interaction structures with task-specific semantic bindings. Publishing those bindings as shared canvas state lets humans and agents inspect and use the same interactions.

3 Agent-Bound UI

  • Agent-Bound UI pairs a reusable interaction structure with a task-specific semantic binding, distinguishing user gestures from the actions communicated to agents.A view supplies a reusable interaction vocabulary over a representation; a canvas instantiates views with task-specific content and interaction semantics.Source: §3 · p. 3

3.1 A Reusable Action Vocabulary

  • Views provide stable, reusable action vocabularies over elements, and bounded composable controls can join that vocabulary and acquire task-specific meanings.The same operations can support different tasks because the vocabulary describes actions rather than task meanings.Source: §3.1 · p. 3

3.2 Semantic Binding and Action Handling

  • Agents bind view elements to task objects, enable applicable actions, and assign task-specific meanings; local effects stay view-side while task-reasoning consequences are resolved by agents.Bindings can cover existing representations, such as editing cells or connecting graph elements, as well as composable controls.Source: §3.2 · p. 3, Figure 2 · p. 4

3.3 Published Interaction Semantics

  • Published canvas state exposes element-to-referent bindings, enabled actions, and their task-specific meanings so humans and other agents can inspect them.A second agent can interpret or invoke a first agent’s published actions without a private channel between them.Source: §3.3 · p. 4

4 ABRA System

ABRA realizes Agent-Bound UI through reusable view definitions, task-specific canvas specifications, structured agent tool calls, and a shared canvas hub.

4.1 Realization of Agent-Bound UI

  • ABRA implements Agent-Bound UI with reusable view definitions, task-specific canvas specifications, and an agent interface for shared canvas state.The prototype pairs chat-based natural-language interaction with direct manipulation on agent-rendered views.Source: §4.1 · p. 4

4.2 View and Canvas Specification

  • View definitions specify element schemas, supported actions, and metadata, while canvas specifications select views and provide task content, referent bindings, enabled actions, and semantic bindings.The prototype supports graph, table, text, and display views; actions use predefined local-effect or agent-resolved consequence modes.Source: §4.2 · p. 4, Table 1 · p. 5

4.3 Agent Tool Interface

  • Structured tool calls let agents create, inspect, and operate on shared canvases through a lifecycle that includes reading, updating, firing actions, and closing canvases.For actions requiring agent resolution, the owner agent handles the consequence and the resulting state is shared with participants.Source: §4.3 · pp. 4–5

4.4 Prototype Architecture

  • A Node.js canvas hub maintains and synchronizes shared states, while React views and Python agent adapters provide client-specific access.The prototype integrates agents built with different SDKs, which inspect published states and interact through the tools.Source: §4.4 · p. 5

5 Use Case

ABRA’s World Cup analysis use case shows users and agents coordinating through evolving shared canvases rather than relying solely on chat. Users refine representations directly, while agents interpret edits and enforce task-specific constraints; agents also reuse published information across canvases.

5 Use Case

  • Three specialized agents progressively build a shared workspace with a knockout-stage table, bracket, player gallery, and explanatory notes.Analyst maintains the table, Curator the bracket and gallery, and Researcher the notes.Source: §5 · p. 5
  • The canvases function as active coordination surfaces where users and agents communicate by manipulating shared representations.The interaction model is not tied to the described deployment choice; other agent environments can realize it.Source: §5 · p. 5
  • The user refines the workspace by editing table entries and connecting tournament-graph nodes, with agents interpreting those manipulations in context.An abbreviation prompts a proposed column-wide convention; an invalid tournament connection is rejected by the owner agent.Source: §5 · p. 5
  • Shared referent bindings let agents reuse player statistics across canvases while keeping teams and players consistently interpreted.Curator asks Analyst to extend the existing table, then reuses the published information in its own canvases.Source: §5 · p. 5

6 Experiments

The experiments assess whether four LLM-backed agents can author canvases, resolve consequences, and invoke another agent’s published interactions. Structural success was generally high, but results also expose gaps in specification following and semantic grounding.

6 Experiments

  • The evaluation targets whether agents can author canvases, resolve consequences on canvases they own, and invoke semantics published by other agents.These capabilities are evaluated in isolation rather than through end-to-end task performance.Source: §6 · pp. 5–6

6.1 Tasks

  • The tasks cover canvas creation, owner-side consequence resolution, and cross-agent invocation, with 45 authoring cases and 33 cases for each other task.Cases vary across canvas configurations or actions and three domains.Source: §6.1 · p. 6

6.2 Experimental Setup

  • Each case ran three times per model with a fresh agent instance, and each agent had one response turn to submit canvas operations.The one-turn setup tests grounding and execution from the provided context without iterative interaction.Source: §6.2 · p. 6
  • The evaluation compares Claude Opus 4.8, Haiku 4.5, GPT-5.4, and GPT-5.4-mini using structural checks and an LLM-based evaluator.The four agents come from two model families.Source: §6.2 · p. 6

6.3 Results

  • 88.9%–97.8% of create_canvas runs and 98%–100% of update_canvas runs passed structural checks; fire_action pass rates ranged from 66.7% to 100%.GPT-5.4-mini had the lowest fire_action pass rate at 66.7%; 24 of its 33 failures used update_canvas instead of fire_action.Source: Table 2 · p. 6, §6.3 · p. 6
  • View recall ranged from 56% to 78%, while exact all-or-nothing view selection ranged from 33% to 53%, indicating that requests often received only some expected views.GPT-5.4-mini had the lowest exact selection score, 33%.Source: §6.3 · p. 6, Table 3 · p. 7
  • Among structurally valid fire_action runs, judged performance was near ceiling, and agents generally selected the correct published binding and grounded the correct element.Overall, the results indicate reliable executable canvas operations alongside remaining challenges in open-ended specification following and semantic grounding.Source: §6.3 · p. 7

7 Conclusion

The paper presents ABRA as a prototype of Agent-Bound UI for shared interactive canvases. Its demonstration and experiments show evaluated LLM agents operating over published interaction semantics, supporting a foundation for richer human–agent interaction and cross-agent collaboration.

7 Conclusion

  • ABRA is a prototype of the Agent-Bound UI interaction model for shared interactive canvases.The authors report that the demonstration and experiments show evaluated LLM agents can effectively operate over published interaction semantics.Source: §7 · p. 7

Limitations

The paper identifies unresolved questions about reusable interaction vocabularies and about users’ interpretation of published semantics and their effects on interaction outcomes.

Limitations

  • Finding reusable interaction vocabularies with broad task coverage remains challenging.Future work will investigate discovering common views and actions from existing systems and evaluating coverage across diverse task benchmarks.Source: Limitations · p. 7
  • The study has not evaluated how effectively users interpret published semantics or how those semantics affect interaction experience and task outcomes.Future user studies will examine presentation and compare agent-bound interactions with chat-based interaction.Source: Limitations · p. 7

A Evaluation Details

Evaluation combines deterministic checks with a separate-model LLM evaluation. Deterministic checks cover contract-level constraints and task-specific outcomes, while the LLM evaluator handles open-ended criteria.

A Evaluation Details

  • Evaluation combines deterministic checks with a separate-model LLM evaluator for criteria that require different assessment methods.Deterministic checks cover contract-level constraints and task-specific expected outcomes; the LLM evaluator assesses open-ended criteria.Source: A Evaluation Details · p. 9
  • For create_canvas, checks verify schema, output format, expected outcomes, and whether generated views and action types match the specification.The checks also assess expected outcomes for update_canvas and fire_action.Source: A Evaluation Details · p. 9
  • For update_canvas and fire_action, checks compare the revised canvas content or fired action type with the expected outcome.Source: A Evaluation Details · p. 9

A.1 Judge Prompts

The judge prompts distinguish fixed view structure from task-specific canvas content and meanings. They assess canvas creation and cross-agent action invocation against the relevant task context and ownership rules.

A.1 Judge Prompts

  • Judges assess the agent-authored canvas against a fixed view specification that defines element types, content shape, and available actions.The authored portion includes selected views, task content, published action meanings, and optional controls or presentation settings.Source: §A.1 · p. 9
  • Ownership rules reserve consequence resolution to the canvas owner; collaborative mode allows locked co-authoring without changing published meanings.In exclusive mode, non-owners only fire published actions.Source: §A.1 · p. 9
  • A create_canvas prompt checks whether an agent instantiates expected views and supplies task content, while allowing extra appropriate views.The prompt presents the canvas and expected views alongside criteria to judge.Source: §A.1 · pp. 9–10
  • A fire_action prompt evaluates whether a non-owner agent's published-action invocation addresses the stated problem for the canvas owner to carry out.The prompt supplies the canvas, published action meanings, and the composed action, path, and value for assessment.Source: §A.1 · p. 10

A.2 Evaluation Criteria

Table 4 summarizes the evaluation criteria for canvas authoring and cross-agent invocation.

A.2 Evaluation Criteria

  • Table 4 summarizes criteria for create_canvas authoring and fire_action cross-agent invocation.Source: §A.2 · p. 10
Loading 2610.30780v1…