Source-linked AI summary
Empty Commitments: When Agents Promise What Their Runtime Cannot Deliver
Jiaqi Tang, Lan Wei, Bingyu Shen, Boyang Li
TL;DR
Agents may receive requests that require action beyond the current conversation, but whether they can fulfill them depends on runtime affordances. This paper defines and measures future-directed commitments against those affordances; its central conclusion is that an agent’s promise about the future is only as good as its runtime.
Problem
Prior work examined whether agents later break promises whose actions were possible, leaving open whether a promise was possible at all given the agent’s runtime.
Source: §1 · p. 1Method
The paper defines commitments against deployment affordances, classifies response outcomes, and measures them across five setups with inferred or stated runtime information.
Source: §2 · pp. 1–2, §3 · p. 2Results
The paper concludes that future promises are only as good as the agent’s runtime and defines three promise failures decidable from one turn and deployment configuration.
Source: §5 · p. 2Takeaways & Limitations
The taxonomy treats refusing every request as no fix, distinguishing over-refusal from failures and desirable responses.
Source: §2 · p. 2The protocol is single-turn with mocked tools, and its LLM-based labels need human validation.
Source: §5 · p. 2
Abstract
from arXiv · showhide
A chatbot that says "I will remind you tomorrow" will not run again until the user writes. We call such a promise an empty commitment: a promise of an action after the current turn that nothing in the agent's tools or runtime can carry out. Unlike a broken promise, its emptiness follows from the agent's configuration alone; no later trajectory is needed. We define empty commitments on top of commitment semantics, with three failure types, an anchoring condition for promises that a tool could make real, and a response-level outcome taxonomy. We then describe a measurement protocol: follow-up requests run in five setups that add one persistence affordance at a time, with the environment either left implicit or stated.
1 Introduction
Requests such as reminders and callbacks depend on what an agent’s deployment can persist or do after a turn. The paper asks whether a promised future action is possible at all and introduces a way to identify and measure promises that the runtime cannot fulfill.
1 Introduction
- A future-facing request may require a scheduler, memory store, ticketing tool, or an agent that runs again without a new user message.The model is rarely told what persists after the turn ends.Source: §1 · p. 1
- A promise presupposes that its speaker can perform the promised act, a condition a tool-less agent fails when it promises a reminder.A false action claim is a related case: saying a reminder is set when none exists.Source: §1 · p. 1
- The paper asks whether the promised act was possible at all, rather than whether an agent later performs a possible act.An empty commitment can be identified when it is made; nothing the agent does later can redeem it.Source: §1 · p. 1
- The paper contributes a configuration-and-one-turn definition, an outcome taxonomy separating failures, desirable responses, and over-refusal, and a measurement protocol.Source: §1 · p. 1
2 Empty Commitments
The paper treats future-directed commitments as actions whose feasibility depends on an agent’s affordances. It distinguishes empty commitments from unanchored promises and other response outcomes.
2 Empty Commitments
- A commitment is modeled as x owing y action α when φ holds within window δ, unless released by ρ; this section focuses on actions after the turn ends.Promises triggered by the user’s next message are set aside, and affordances include tools and four persistence properties.Source: §2 · p. 1
- A commitment is felicitous when an execution path under the agent’s affordances performs α when φ holds within δ; otherwise, it is empty.Source: §2 · pp. 1–2
- A feasible commitment is anchored when the agent calls an enabling tool in the same turn; without that call, it is unanchored.The passage gives create_reminder as an example of an enabling tool.Source: §2 · p. 2
- The outcome taxonomy distinguishes FALSE_CLAIM, EMPTY, and UNANCHORED failures from ANCHORED, DONE_NOW, DEFERRAL, OVER_REFUSAL, and OTHER responses.The first three are failures; ANCHORED, DONE_NOW, and DEFERRAL are desirable, while OVER_REFUSAL is a cost.Source: §2 · p. 2
3 Measurement Protocol
The protocol varies persistence affordances and whether runtime facts are stated, then classifies responses and reports outcome and failure rates. Its requests span future-oriented tasks, with doability used to distinguish refusal outcomes.
3 Measurement Protocol
- Requests cover reminders, monitoring an external state, callbacks and escalation, deferred delivery, and cross-session memory.A doability matrix determines whether declining counts as DEFERRAL or another outcome.Source: §3 · p. 2
- Five setups are tested in inferred and stated variants; C2–C4 each add one affordance to C1 rather than accumulating changes, and all tools are mocked.The stated variant adds runtime facts without advice; two personas yield 20 conditions.Source: Table 3 · p. 2, §3 · p. 2
- A setup-blind detector extracts future-directed commitments, deterministic rules assess feasibility and anchoring, and a judge supplies booleans used to assign outcomes.Source: §3 · p. 2
- The primary result is the outcome distribution per setup and variant, alongside empty-commitment, unanchored, and over-refusal rates.The rates use different denominators: all future-directed commitments, feasible commitments needing an enabling call, and doable requests, respectively.Source: §3 · p. 2
4 Open Questions
The section frames four open questions about response outcomes, persistence tools, runtime disclosure, over-refusal, and tool-less agents’ public logs.
4 Open Questions
- When persistence is unstated, the paper asks how responses distribute across outcomes and which failure types dominate.Source: §4 · p. 2
- When an enabling tool exists, the paper asks how often models promise without calling it.Source: §4 · p. 2
- The paper asks whether stating the environment reduces failures and how much over-refusal it causes.Source: §4 · p. 2
- The paper asks how common these failures were in public logs of tool-less agents (Zhao et al., 2024).Source: §4 · p. 2
5 Conclusion
The paper defines three promise-related outcomes that can be decided from one turn and the deployment configuration. Its measurement protocol is single-turn and uses mocked tools, while its LLM-based labels still need human validation.
- The paper defines empty commitments, false action claims, and unanchored promises as outcomes decidable from one turn and the deployment configuration.Source: §5 · p. 2
- The measurement protocol is single-turn and uses mocked tools.Source: §5 · p. 2
- The LLM-based labels need human validation, and a multi-model evaluation is under way.Source: §5 · p. 2