Dialog2API: Task-Oriented Dialogue with API Description and Example Programs

Motivation

• Free-form program synthesis targets complex and compositional goals that closed intent-and-slot schemas may not represent, such as replacing ordered items.Semantic parsing can represent compositional goals but typically depends on extensive training data and degrades on uncovered request forms.Source: §1 · pp. 1–2

Paradigm

• Dialog2API lets developers provide API documents and optional annotated example dialogues while the model synthesizes programs, manages policy, and generates responses.The model interacts with users and an execution environment through API calls, reducing the developer effort required to build domain-specific systems.Source: §1 · p. 2

Evaluation

• Execution-based evaluation measures successful dialogue turns by comparing environment states rather than requiring one exact program implementation.This accommodates multiple API compositions that achieve the same goal and is intended to better reflect user experience.Source: §1 · p. 2

Limitations

• Dialog2API trades stronger functionality and more natural dialogue experience for lower dialogue-policy interpretability and bot-response controllability.The paper notes that explaining policy and validating response correctness and safety may require additional mechanisms.Source: §1 · p. 2, Table 1 · p. 2

Approach

• The proposed dialogue state is a stack of programs, with each program representing a sub-goal and moving among drafting, final, and abandoned states.Final programs execute immediately, execution outcomes are attached to the stack, and updated programs condition subsequent response generation.Source: §1 · p. 2

Results

• 33%: the best in-context-learning baseline reaches 33% on the proposed execution-based metric, while API documents remain helpful even when example dialogues are available.The study constructs a dataset of 100 AWS S3 conversations with turn-level program annotations.Source: §1 · pp. 2–3

2 Dialog2API

• Dialog2API uses API documents, dialogue history, and optional annotated example dialogues to generate programs and responses for each user turn.The API document describes available functionality and usage, while examples show how APIs are triggered in conversational context.Source: §2 · p. 3
• Each program receives a goal ID and status, with drafting programs revisable, final programs immediately executed, and abandoned programs marking relinquished goals.Execution results, including errors, are observed by the model; final programs cannot be revised.Source: §2 · p. 3
• Multiple programs may represent sub-goals within one user turn, execute sequentially, and use unrestricted executable programming styles rather than exact code matching.Programs need only be executable, so naming conventions and programming styles may differ across dialogues.Source: §2 · p. 3
• For each turn, the model predicts programs from API definitions, dialogue history, and the user utterance, then generates a response using the predicted programs and execution outcomes.Each predicted program includes its goal ID, status, and code.Source: §2 · p. 3

2.1 Evaluation Metrics

• EMR evaluates program prediction by comparing deterministic environment signatures, including program return values, and measuring the ratio of successful dialogue turns.A turn is successful when a program executes and the resulting environment signature matches the ground truth.Source: §2.1 · pp. 3–4
• EMR is an automatic approximation that can under-estimate success under different dialogue flows or over-estimate it when wrong responses do not disrupt later program prediction.BLEU measures generated-response quality, while code edit distance is reported as a supplementary teacher-forced metric.Source: §2.1 · p. 4

3 Approaching Dialog2API with a Stack of Programs

• A program stack records each goal’s latest program, status, execution result, and temporal order, giving the model a compact representation of dialogue progress.Programs may be drafting, final, or abandoned; the most recently created or revised program is placed on top.Source: §3 · p. 4

3.1 Role of Drafting Programs

• Drafting programs expose a user’s goal and missing information before execution, allowing the model to finalize a program from fewer recent turns.Placeholders such as “?1” mark parameters or code blocks that remain to be completed.Source: §3.1 · pp. 4–5

3.2 Program Design Details

• Programs are standard Python code describing developer-controlled user goals, and one utterance may produce multiple programs when it contains multiple goals.Programs return JSON data, while execution returns values or errors to the model; equivalent implementations need not share a consistent coding style.Source: §3.2 · p. 5
• Goal revision requires regenerating the full updated program, permitting changes beyond placeholder substitution, including added or rewritten code before execution.A revised program can replace a placeholder, add code, and change its status to final.Source: §3.2 · p. 5
• Conversational exceptions are represented through program execution, with errors recorded in execution results while response behavior is learned from example dialogues.The framework distinguishes program-initiated errors from conversational exceptions such as out-of-scope requests.Source: §3.2 · pp. 5–6

Dataset scope

• 100 dialogues cover 10 AWS S3 APIs and use a 46/54 training/testing split with no overlapping goals between splits.The execution environment reuses boto3, and environment signatures are computed from snapshots of S3 buckets and objects.Source: §4 · p. 6
• The dataset supports deterministic environment changes through API calls and covers around 70% of S3 API use cases found through GitHub search.The coverage estimate is based on GitHub search hits for S3 API usage with boto3.Source: §4 · p. 6

Dialogue collection

• The collection process creates 50 goals and gathers two dialogues per goal through professional data linguists.Goals are described with canonical utterances, including file-manipulation tasks.Source: §4 · p. 6
• User-side annotators plan dialogues from canonical goals without seeing agent guidelines or supported APIs, allowing conversations to diverge from the initial goal.Divergence can result from different planning or unexpected agent responses.Source: §4 · p. 6
• Agent-side annotators classify conversational exceptions, assess program completable-ness, request missing information in API-defined order, and produce guided responses.Final execution outcomes may be predicted or obtained by coding and running the actual program; dialogue collection averages six minutes.Source: §4 · p. 6

Program annotations

• Program annotation verifies final programs first, annotates drafting programs afterward, and executes final programs to produce ground-truth environment signatures.The signatures follow the procedure described for representing environment changes.Source: §4 · p. 6

5.1 In-context Learning Baseline

• Dialogue-state representations feed the language models the program stack and recent turns for program prediction, while response generation additionally uses the current program state.The experiments also study whether including API definitions in context affects performance.Source: §5.1 · pp. 6–7
• The baselines use GPT-Neox, Bloom, and Codex with greedy search, five examples for example-based prompts, and three prompt settings involving API documents and examples.The settings are API document, in-domain examples, and API document combined with in-domain examples.Source: §5.2 · p. 7, §5.1 · p. 7

5.2 Baseline Results

• Neox and Bloom generate programs close to ground truth but often make minor mistakes, which can sharply reduce EMR because success is measured across dialogue interactions.The execution-based metric penalizes an error occurring midway through a dialogue.Source: §5.2 · p. 7
• 33% EMR is achieved by Codex in the best setting, while API documents and in-domain examples improve in-context learning performance.With Codex Davinci, API-document-only prompting reaches 8.4% EMR, and adding the API document to in-domain examples increases EMR by 3%.Source: §5.2 · pp. 7–8

5.3 Qualitative Analysis

• A Codex Davinci dialogue corrects an initial extension hallucination after user revision, but an incorrect confirmation message demonstrates response-generation errors.The final correct program matches the ground-truth environment signature with EMR 1.0, while the response error lowers BLEU.Source: §5.3 · p. 8
• Because evaluation supplies the ground-truth user utterance at each turn, response errors can make later inputs contextually mismatched and overestimate EMR.This caveat concerns the end-to-end test-dialogue procedure.Source: §5.3 · p. 8

Related Work

• Unlike Andreas et al. (2020), Dialog2API evaluates functional chatbots generated from API documents and few example dialogues rather than modeling dialogue primarily as dataflow synthesis.The paper identifies these as two major differences from the closest related work.Source: §6 · p. 8
• Dialog2API represents dialogue state with a program stack instead of the dataflow representation used by Andreas et al. (2020).The stack stores the latest program versions and supports dialogue revisions.Source: §6 · p. 8
• Free-form programming codes allow more flexible goal descriptions, while revising the full updated program makes exact-match evaluation difficult.The paper addresses this difficulty with an execution-based metric.Source: §6 · pp. 8–9
• Compared with semantic parsing works such as Yu et al. (2019) and Gupta et al. (2018), Dialog2API targets few-shot generation of programs with unseen structures and API combinations.The model must also analyze dialogue context and generate responses for interaction.Source: §6 · p. 9

• The paradigm is intended to expand functionality and support dialogue unrestricted by the underlying schema through programs combining APIs.Source: §7 · p. 9
• Execution match ratio evaluates program prediction by matching the environment signature and reflects conversational user experience.Source: §7 · p. 9
• 33% EMR was achieved by the best existing large language model, while 22% of test dialogues were perfectly generated on the S3 dataset.Source: §7 · p. 9
• Dialog2API represents dialogue state with a stack of programs and conditions generation on API documents, the program stack, and dialogue history.Source: §7 · p. 9
• API documents and example dialogues improve prediction of more correct programs.Source: §7 · p. 9

• The conventional examples separate bucket creation, listing, and copying into schema-specific turns and responses.Source: A Comparing dialogue experience of… · p. 11
• Dialog2API combines APIs in a drafted program to filter and copy all files containing “sea” in their filenames.Source: A Comparing dialogue experience of… · p. 11
• Traditional semantic parsing requires requests to follow the underlying schema and functions closely for correct parsing.Source: A Comparing dialogue experience of… · p. 11
• Slot-filling dialogue restricts flow to predefined intents and slots, whereas Dialog2API supports progressive clarification and compound file-copying goals.Source: A Comparing dialogue experience of… · p. 11

• The environment signature is a checksum summarizing the environment state and program return value.Source: B Computing environment signature · pp. 11–12
• S3 signatures include bucket names, all object keys in each bucket, and the content of each object.Source: B Computing environment signature · p. 11

• OutOfScopeRequestError handles requests outside the target domain, while AmbiguousRequestError handles unclear in-domain requests.Source: C Conversational Exceptions · p. 12
• ChitChat handles conversational requests, while FAQ handles in-domain questions that cannot be solved by a program.Source: C Conversational Exceptions · p. 12
• OverSpecificationError handles requests containing constraints for which suitable programs cannot be generated.Source: C Conversational Exceptions · p. 12
• EndDialog handles user requests to end the dialogue, such as thanking the system.Source: C Conversational Exceptions · p. 12

• The dialogue begins with a drafted program that lists objects under the land_animals/mammals prefix and filters results to .txt files.Source: D Example Dialogue with Program… · p. 12
• When the user asks the system to find a standard bucket, it raises OutOfScopeRequestError and responds that it cannot search for a bucket.Source: D Example Dialogue with Program… · p. 12
• The system revises the earlier goal after listing available buckets, then returns three matching mammal text-file paths from zoology-bucket.Source: D Example Dialogue with Program… · p. 12
• It composes a follow-up program that sorts the matching files by size in descending order, returning bat.txt at 1551 bytes, pika.txt at 878 bytes, and deer.txt at 402 bytes.Source: D Example Dialogue with Program… · p. 12
• After the user signals completion, the dialogue ends by executing an EndDialog program and producing a closing response.Source: D Example Dialogue with Program… · pp. 12–13

E In-context Learning Details

• The experiments prompt the language model separately for program prediction and response generation, using the same prompting format and retrieval strategy.Source: E In-context Learning Details · p. 13

E.1 Prompt Example for Program Prediction

• The program-prediction prompt combines API definitions with stacked drafting and finalized goals, allowing parameters to be filled across turns and programs to compose multiple S3 operations.Source: §E.1 · p. 13
• The prediction examples include multi-step deletion workflows that list objects, construct deletion requests, and delete a bucket after confirmation.Source: §E.1 · pp. 13–14

E.2 Prompt Example for Response Generation

• The response-generation prompt pairs executed programs and results with natural-language outputs that report object lists, empty results, invalid bucket names, and missing objects.Source: §E.2 · pp. 14–15
• The examples also show responses reporting a successful broader-prefix listing and translating a false object-membership result into a concise negative answer.Source: §E.2 · pp. 14–15