Source-linked AI summary
Iris: A Conversational Agent for Complex Tasks
Ethan Fast, Binbin Chen, Julia Mendelsohn, Jonathan Bassen, Michael Bernstein
TL;DR
Existing conversational agents are largely limited to standalone commands, leaving complex tasks that require command combination unsupported. Iris combines commands through a conversational architecture with an automata-based DSL and conversational type system. It is designed for data science and enables users to weave lower-level commands into complex tasks.
Problem
Today’s conversational agents use standalone commands that restrict users to tasks explicitly supported by the system.
Method
Iris uses an automata-based DSL and conversational type system to dynamically compose and sequence commands through natural language.
Results
Iris enables users to execute complex commands without predefined support by weaving together multiple lower-level commands.
Takeaways & Limitations
Iris supports human conversational behaviors such as dependent questions and command sequencing that are largely missing from today’s systems.
Takeaways & Limitations
Iris does not attempt to cover all human language, and command interpretation may become more difficult as its supported command set grows.
Abstract
from arXiv · showhide
Today's conversational agents are restricted to simple standalone commands. In this paper, we present Iris, an agent that draws on human conversational strategies to combine commands, allowing it to perform more complex tasks that it has not been explicitly designed to support: for example, composing one command to "plot a histogram" with another to first "log-transform the data". To enable this complexity, we introduce a domain specific language that transforms commands into automata that Iris can compose, sequence, and execute dynamically by interacting with a user through natural language, as well as a conversational type system that manages what kinds of commands can be combined. We have designed Iris to help users with data science tasks, a domain that requires support for command combination. In evaluation, we find that data scientists complete a predictive modeling task significantly faster (2.6 times speedup) with Iris than a modern non-conversational programming environment. Iris supports the same kinds of commands as today's agents, but empowers users to weave together these commands to accomplish complex goals.
INTRODUCTION
Today’s conversational agents are limited to standalone commands, while Iris combines commands to support complex data-science tasks beyond predefined system support.
- INTRODUCTION: Standalone-command agents handle simple goals but restrict users to tasks the system was explicitly designed to support.
- INTRODUCTION: Human conversational strategies combine commands through nesting and references to previous commands, enabling more complex interactions.
- INTRODUCTION: Iris lets users compose or sequence commands, such as plotting log-transformed data or applying a t-test across extracted columns.
- INTRODUCTION: Iris uses a DSL that transforms commands into an automata-based programming model for dynamic transformations.
- INTRODUCTION: Its conversational type system infers which commands can combine and gracefully handles type mismatches.
- INTRODUCTION: Iris targets increased expressivity and complexity without attempting to cover all human language or linguistic complexity.
CONVERSATION ANALYSIS THEORY
Iris adapts concepts from conversation analysis to model command combination, clarification, dependent questions, and references to prior command outputs.
- CONVERSATION ANALYSIS THEORY: Conversation analysis models dialogue with typed adjacency pairs, which Iris uses to constrain responses that execute commands.
- CONVERSATION ANALYSIS THEORY: Insert expansions nest conversations to resolve issues before continuing, making them especially relevant to conversational agents.
- CONVERSATION ANALYSIS THEORY: Iris supports dependent questions by letting a new command generate the value needed to resolve an earlier command request.
- CONVERSATION ANALYSIS THEORY: Iris treats values produced by previous commands as anaphoric expressions, supporting sequencing through named variables and pronoun coreference.
- CONVERSATION ANALYSIS THEORY: The architecture draws on insert expansions, dependent questions, and anaphora to support conversational behaviors missing from many current systems.
TODAY’S CONVERSATIONAL AGENTS
Compared with existing agents, Iris broadens command composition and sequencing beyond interactions that have been hard-coded for particular commands.
- TODAY’S CONVERSATIONAL AGENTS: Existing agents execute standalone commands, extract arguments, and resolve missing arguments through questions.
- TODAY’S CONVERSATIONAL AGENTS: Iris supports composition by allowing a response to an argument request to initiate a new command whose result resolves the original request.
- TODAY’S CONVERSATIONAL AGENTS: Iris supports sequencing across its commands, such as taking the mean of a column returned by a previous command.
- TODAY’S CONVERSATIONAL AGENTS: Current systems generally support composition and sequencing only when those interactions are hard-coded for specific commands.
RELATED WORK
Iris extends natural-language interfaces and dialogue systems with structured, modular command combination informed by theories of ambiguity and human conversation.
- RELATED WORK: Iris maps natural language to commands and manages ambiguity through interaction patterns such as clarification requests.
- RELATED WORK: Iris draws on data-science tools that use natural-language command descriptions and support code organization or reuse.
- RELATED WORK: Iris is a modular, structured dialogue system with a conversational type system and a concrete set of supported tasks.
- RELATED WORK: Unlike systems that hard-code individual interactions, Iris aims to enable general command combination through conversation.
- RELATED WORK: The system is situated within dialogue research contrasting modular versus end-to-end and structured versus unstructured systems.
SCENARIO
Iris demonstrates command composition, sequencing, and type-guided interaction in a data-science analysis of dogmatism in social media. The scenario culminates in statistically significant linguistic differences and exportable Python code.
- Scenario: Iris replicates a dogmatism analysis by combining statistical, text-analysis, and plotting commands over social-media data.The scenario uses statistical commands from scipy.stats and sklearn alongside LIWC-based text analysis.
- Composing Commands: Composition lets Iris use a data-selection command to supply the array argument for quartile computation.The system extracts the score column from dogmatism_data and reports quartile ranges.
- Type System Guardrails: Iris’s type system converts a Collection into the Array required by quartile computation and guides the user when the request provides the wrong type.It offers a column-selection conversion rather than failing on the mismatched input.
- Saving and Sequencing Commands: Sequencing lets users save command results, reuse them in later analyses, and reference prior outputs through names or pronouns.The scenario saves dogmatic and non-dogmatic collections, applies LIWC, and runs Mann–Whitney tests on the resulting category scores.
- Displaying Plots and Non-textual Data: The final plot shows dogmatic associations with swearing, negative sentiment, and sexual language, while non-dogmatic associations involve first-person pronouns and past tense.Iris generates the odds-ratio plot from final_stats, and the conversation can be exported as Python code for replication.
- Architecture: The architecture represents commands as automata that support composition and sequencing, enabling complex statistical tasks while providing conversational guardrails.The user can combine commands recursively or run them in series while the type system performs conversions automatically.
A Programming Model for Conversation
Iris represents conversational commands as automata generated by a domain-specific language, allowing commands to modify, compose, and sequence their execution dynamically. A conversational type system resolves arguments, checks compatibility, and guides conversions.
- A Programming Model for Conversation: The DSL solves the need for runtime-modifiable conversation state machines by transforming commands into an automata-based programming model.Its automata can dynamically modify transitions and maintain scoped bindings for nested conversations.
- Conversational DSL: Iris automata model conversational states that can receive user input, pass it as arguments, and transition between computation steps.The DSL also provides scopes and dynamic transitions for composed commands.
- The Structure of Iris Commands: Iris commands wrap Python functions with conversational metadata describing triggers, argument types, extraction rules, and return-value explanations.Commands inherit from IrisCommand and are transformed into automata that extract, request, and execute arguments.
- Composition: Composition transfers control from a parent automaton to a child command and then binds the child’s result back to the parent argument.New scopes prevent nested argument bindings from overwriting one another.
- The Conversational Type System: The conversational type system resolves missing arguments, dynamically checks command compatibility, and converts nonmatching values when possible.Iris supports types including integers, strings, arrays, collections, models, metrics, and plots.
A Statistical Model of User Requests
Iris maps user language to commands, executes their automata, and supports nested or sequential requests through conversation. It also maintains runtime representations that can expose and export composed interactions.
- A Statistical Model of User Requests: A multiclass logistic regression model predicts commands from user language and updates its training examples as users enter new requests.The architecture can later replace this classifier with a more sophisticated model.
- Executing Requests via Iris Commands: Iris executes each selected command’s automata to extract arguments, ask clarification questions, and run the requested operation.Requests enter through a command-prediction model before the command automata handle interaction.
- Composing Command Execution: When a response cannot be parsed as the requested value type, Iris can interpret it as a nested command and compose that command with the parent.The child runs in a separate scope, returns its result, and supplies the parent’s pending argument.
- Sequencing Command Execution: Sequencing stores command results in history variables so later requests can refer to them with pronouns such as “this,” “those,” or “that.”Commands can also save results under named variables for future conversations.
- Transforming Conversations into Programs: Iris incrementally builds an abstract syntax tree of the conversation and compiles it into Python code when users request an export.The interface presents these interactions through a chat window with hints, metadata, variables, and embedded non-textual outputs.
EVALUATION
The evaluation compared Iris with sklearn in Jupyter on a predictive modeling task. Participants completed the task 2.57 times faster with Iris, while also using varied interaction styles and reporting benefits alongside interface challenges.
- Method: Eight trained computer scientists completed a flower-species modeling task using both Iris and sklearn in randomized order.The task involved building and cross-validating a model, then examining model coefficients.
- Results: 2.57 times faster on average, participants completed the predictive modeling task with Iris than with sklearn in Jupyter (p<0.01).All participants completed the task correctly in both conditions.
- Interaction patterns: Participants used both full natural-language queries and faster keyword-style requests, with keywords still supporting command composition and sequencing.Full-query users instead emphasized automatic argument extraction.
- Advantages to Conversation: Less experienced sklearn users valued Iris’s structural guidance for threading API components together during modeling.Participants described Iris as walking them through dependencies such as choosing a model before cross-validation.
- Advantages to Conversation: Participants also reported faster access to familiar functions and appreciation for Iris’s ability to perform complex operations.One experienced user valued avoiding memorization of function names, while another highlighted the system’s sophistication.
- Challenges for Conversation: Iris’s expressivity created more opportunities for mistakes, although its conversational type system prevented many incompatible-command and wrong-input errors.Wrong command selection remained a vocabulary problem, with three pilot participants triggering incorrect commands.
LIMITATIONS AND FUTURE WORK
The paper identifies limitations involving scalability, workflow reuse, composition strategy, and task scope. Future work targets more robust language interpretation, reusable workflows, richer parsing, and longer-running pipelines.
- LIMITATIONS AND FUTURE WORK: As Iris expands beyond 95 manually supported commands, interpreting user language and acquiring enough examples for new commands may become more difficult.The current dataset contains roughly five examples per command.
- LIMITATIONS AND FUTURE WORK: Saving conversation-derived ASTs could support reusable natural-language workflows, but deciding which parameters are arguments versus constants remains a usability challenge.The authors also propose mining higher-level commands from many saved ASTs.
- LIMITATIONS AND FUTURE WORK: Iris composes commands through conversational turn-taking, whereas semantic parsing can learn composition within individual utterances but requires large training datasets.Future work considers using Iris conversations as a source of training data for semantic parsing and other NLP methods.
- LIMITATIONS AND FUTURE WORK: Iris supports exploratory interactive analysis but is not currently designed for long-running pipelines that require extended setup and debugging.The authors connect future workflow creation and saving to the possibility of supporting such pipelines.
CONCLUSION
Iris shows how conversational agents can combine commands using human conversational strategies, extending assistance to tasks beyond their explicit programming. The paper presents this approach through a data science and machine learning agent.
- CONCLUSION: Iris combines commands to assist with tasks that conversational agents were not explicitly programmed to support.The approach draws on human conversational strategies and is demonstrated in data science and machine learning.