Source-linked AI summary
Bel Esprit: Multi-Agent Framework for Building AI Model Pipelines
Yunsu Kim, AhmedElmogtaba Abdelaziz, Thiago Castro Ferreira, Mohamed Al-Badrashiny, Hassan Sawaf
TL;DR
Complex tasks can require multiple AI models, so Bel Esprit uses a multi-agent framework to construct pipelines from user requirements. Its framework configurations achieve overall gains of +9.5% EM and -28.1% GED, while the paper identifies remaining limits in handling highly ambiguous queries and the available function pool.
Problem
A single AI model is often insufficient for complex tasks, motivating pipeline construction that connects models to solve them.
Source: §1 · p. 1Method
Results
+9.5% EM and -28.1% GED overall were achieved by incrementally incorporating Mentalist, Builder, and Inspector into a baseline pipeline builder.
Source: §5.4 · p. 6Takeaways & Limitations
Node substitutions are generally more significant errors than node insertions or deletions in pipeline generation.
Source: §6 · p. 7The system still struggles with highly ambiguous queries missing critical input or output requirements, and pipeline building and matchmaking use a predefined function pool.
Source: Limitations · p. 8
Abstract
from arXiv · showhide
As the demand for artificial intelligence (AI) grows to address complex real-world tasks, single models are often insufficient, requiring the integration of multiple models into pipelines. This paper introduces Bel Esprit, a conversational agent designed to construct AI model pipelines based on user-defined requirements. Bel Esprit employs a multi-agent framework where subagents collaborate to clarify requirements, build, validate, and populate pipelines with appropriate models. We demonstrate the effectiveness of this framework in generating pipelines from ambiguous user queries, using both human-curated and synthetic data. A detailed error analysis highlights ongoing challenges in pipeline construction. Bel Esprit is available for a free trial at https://belesprit.aixplain.com.
1 Introduction
Complex tasks with multiple inputs or outputs can exceed the capabilities of a single AI model, motivating pipelines that connect models and automate intermediate steps. Bel Esprit is presented as a conversational assistant for building such pipelines, with contributions spanning task formalization, a multi-agent framework, and evaluation.
1 Introduction
- Pipelines connect different models to handle complex tasks and automate intermediate steps between tasks.The paper notes applications including multimodal content moderation, multilingual video dubbing, speech translation, and voice conversion.Source: §1 · p. 1
- Bel Esprit is a conversational assistant that constructs pipeline solutions from diverse AI models.Source: §1 · p. 1
- The paper formalizes model-pipeline building as graph generation involving scientific reasoning.Source: §1 · p. 1
- Its multi-agent framework is designed to improve pipeline quality and alignment with user intent.Source: §1 · p. 1
- The authors establish an evaluation scheme for pipeline building with a data-preparation protocol and automatic metrics.Source: §1 · p. 1
2 Related Work
Prior automated machine-learning efforts mainly target training or tuning single models, while workflow-generation methods often focus on general agents or simple utility ordering. Bel Esprit instead assembles existing AI functions into pipelines for complex tasks and extends workflow generation across modalities.
2 Related Work
- Bel Esprit assembles off-the-shelf models into pipelines for complex tasks rather than training a single model for an atomic task.Source: §2 · p. 1
- Existing workflow-generation methods largely focus on prompts for a few general agents or ordering simple utility functions, with evaluations limited to classical reasoning tasks.The cited task examples include math, coding, and question answering.Source: §2 · pp. 1–2
- Bel Esprit integrates more than 70 AI functions across modalities, adds tools for missing functionalities, and uses conversational clarification and graph verification for reliability.Generated pipelines can also serve as tools within agents, reducing redundant planning and accelerating recurring tasks.Source: §2 · p. 2
3 Task Definition
The paper frames pipeline generation as structured prediction: a user query describing a computational task maps to a graph of AI functions. Graph nodes represent inputs, outputs, or functions, and edges represent data flow; pipeline building is also cast as deductive reasoning over function premises.
3 Task Definition
- Pipeline generation maps a user query describing a computational task to a pipeline of AI functions.The output is a graph whose nodes represent inputs, outputs, or functions, and whose edges denote data flow.Source: §3 · p. 2
- Three special node types extend pipeline functionality: Router directs data by modality, Decision branches on input values, and Script runs Python code for arbitrary tasks.Source: §3 · p. 2
- Pipeline generation can be viewed as deductive reasoning, using AI functions as premises that convey knowledge from specific inputs to outputs.The goal is to find a reasoning path comprising multiple premises that supports the user query as a comprehensive conclusion.Source: §3 · p. 2
4 Framework
Bel Esprit turns user requirements into model-pipeline graphs through a sequence of agents that clarify the request, build and inspect a graph, then select or create implementations for its functions.
4.1 Mentalist
- Mentalist clarifies missing requirements, extracts structured input/output details, and matches attached files to the appropriate input nodes.Attachment matching uses conversational context, especially when multiple inputs share a modality.Source: §4.1.1 · p. 3, §4.1.2 · p. 3, §4.1.3 · p. 3
4.2 Builder
- Builder generates the pipeline graph from the refined query and extracted specification, using guided prompts and example pipelines.It generates one branch at a time; branches represent paths to outputs and can reuse nodes from earlier branches.Source: §4.2 · p. 3, §4.2.1 · pp. 3–4, Figure 5 · p. 4
4.3 Inspector
- Inspector checks graph structure and semantic alignment, sending errors back for revision before the pipeline proceeds.Structural violations include invalid connections; semantic checks compare each branch’s summarized path with the requirements.Source: §4.3 · p. 4, §4.3.1 · p. 4, §4.3.2 · p. 4
4.3.1 Syntax
- Structural repairs range from removing duplicate output nodes to rebuilding a graph when a node receives incompatible input.For example, audio routed directly to a text-input translation node requires locating or creating a transformation node.Source: §4.3.1 · p. 4
4.3.2 Semantics
- Semantic inspection can catch a structurally valid branch that omits a required translation between English transcription and French speech synthesis.The Inspector checks whether each branch fulfills the user’s requirements, not only whether its connections satisfy graph constraints.Source: §4.3.2 · p. 4
4.4 Matchmaker
- Matchmaker selects models according to user preferences or defaults, and uses generic LLM nodes or generated scripts when suitable models are unavailable.Generic nodes address tasks such as domain mixing or creative writing; scripts handle simpler functions such as counting words or extracting PDF text.Source: §4.4 · pp. 4–5, §4.4.1 · p. 5, §4.4.2 · p. 5
5 Experiments
The experiments evaluate Bel Esprit on manually created and synthetically expanded query-pipeline pairs using exact-match and graph-edit-distance metrics. Incrementally adding framework components improves overall performance, while query ambiguity and weaker Builder models remain challenges.
5.1 Data
- The curated dataset contains 441 pipelines: five AI solution engineers created 82 realistic tasks and corresponding pipelines, while the LLM-generated set began with 500 entries, of which 359 remained after human review.For synthetic entries, ambiguity was intentionally introduced into the LLM prompt when synthesizing an initial user query.Source: Structured synthesis with human… · p. 5, §5.1 · p. 5
5.2 Metrics
- Exact Match requires an identical pipeline graph under the study’s node and edge matching rules, while GED counts the edits needed to transform a generated graph into its reference.GED captures incremental differences that Exact Match treats as non-matches.Source: §5.2 · p. 5
5.4 Results
- +9.5% EM and -28.1% GED overall as Mentalist, Builder, and Inspector components are incrementally added to the baseline.GPT-4o outperforms open-source Builder alternatives, and Llama 3.1 8B achieves below 3% EM.Source: Table 2 · p. 6, §5.4 · p. 6
5.3 Models
- The reported framework uses GPT-4o for Mentalist’s query clarifier and Builder, and Llama 3.1 70B for other LLM-assisted framework tasks.Prompt similarity is computed with all-MiniLM-L6-v2.Source: §5.3 · p. 6
5.4 Results
- For GPT-4o, component contributions appear in GED improvements, whereas they are less consistent for weaker models and EM misses nuanced improvements.Semantic inspection occasionally confuses weaker Builders, leading to unnecessary graph repetitions and sporadic performance drops.Source: §5.4 · p. 6
5.5 Qualitative Example
- In the qualitative example, Mentalist clarifies an unknown input language and chain-of-branches reuses an ASR node across French and German output paths.Inspector identifies missing language information and adds edges to the MT nodes, producing a complete and correct pipeline.Source: §5.5 · pp. 6–7
- Figure 11 reports GED over increasing query ambiguity.Source: Figure 11 · p. 7
6 Analysis
Pipeline generation becomes harder as queries grow more ambiguous or reference pipelines grow larger, though clarification and branch decomposition help address these challenges. Most errors are node substitutions; edge errors can be difficult to resolve.
6 Analysis
- The Mentalist significantly improves performance on ambiguous queries by clarifying missing information and specifying inputs and outputs.Performance becomes increasingly challenging as query ambiguity rises.Source: §6 · p. 7
- Larger reference pipelines are more challenging to construct, but chain-of-branches helps by splitting them into manageable subgraphs.The analysis includes pipelines with simultaneous processing of the same input across multiple paths.Source: §6 · p. 7
- Most pipeline errors are node substitutions, often caused by parameter mismatches or incorrect node types.Figures 13 and 14 show the distribution of required edits and causes of node substitution errors, respectively.Source: §6 · p. 7, Figure 13 · p. 7, Figure 14 · p. 7
- Node insertions often reflect unmet query requirements, while deletions can reflect repeated functions across paths; both are generally less significant than substitutions.Relocating a misplaced node also requires corresponding edge insertions and deletions.Source: §6 · p. 7
- Missing edges for functions requiring multiple inputs are detectable by Inspector, but resolving them requires locating the correct data source through semantic understanding of the graph and query.Source: §6 · p. 7
7 Conclusion
The paper frames AI pipeline generation as a new task and presents Bel Esprit, a multi-agent framework for constructing and validating pipelines. Future work includes retrieval-augmented generation and generating autonomous agents beyond static pipelines.
7 Conclusion
- The paper introduces generating AI solution pipelines from user queries as a novel task.Source: §7 · p. 7
- Bel Esprit uses Mentalist, Builder, and Inspector for query clarification, stepwise pipeline construction, and validation.Source: §7 · p. 7
- Future work includes using retrieval-augmented generation with valid pipelines and extending generation beyond static pipelines to autonomous agents.Source: §7 · p. 7
Limitations
The system has limitations in handling highly ambiguous queries, its predefined AI-function pool, and verification of script-node code.
Limitations
- Bel Esprit still struggles with highly ambiguous queries when critical input or output requirements are missing.Source: Limitations · p. 8
- Pipeline building and matchmaking use a predefined AI-function pool, and expanding it with parameter details increases prompt length and computational costs.Generic nodes partly address this constraint but are currently limited to text-to-text functions.Source: Limitations · p. 8
- Inspector does not verify generated script-node code, so each script needs custom test cases that are not yet automated.Source: Limitations · p. 8
A List of AI Functions
The paper treats the AI functions listed in Table 3 as possible pipeline nodes.
A List of AI Functions
- AI functions listed in Table 3 are treated as possible nodes in a pipeline.Source: A List of AI Functions · p. 10
B Graph Constraints
The graph constraints define valid pipeline structure at both the node and edge levels, including required connectivity and modality consistency.
Nodes
- Nodes must satisfy structural rules for inputs, outputs, and routers, including router branching across different modalities.Function names must be predefined, and function nodes must use predefined parameters and include all required inputs.Source: Nodes · p. 10
Edges
- Edges must connect existing parameters with matching modalities, while ensuring input and output connectivity and reachability from an input node.Input parameters have only one incoming edge; non-output-node output parameters need at least one outgoing edge.Source: Edges · p. 10
C Query-Pipeline Dataset
The query-pipeline dataset spans practical application domains and varied task modalities. Its reference pipelines cover all types of transformations between the four modalities.
Domain Coverage
- The dataset covers educational and research, business intelligence, accessibility, information management, and safety applications.Examples include customer-feedback analysis, multilingual audiobook creation, corpus generation, and sensitive-information redaction.Source: C Query-Pipeline Dataset · pp. 10–11
- Example queries include assessing multilingual document readability and suggesting simplifications, categorizing customer feedback, and redacting sensitive information across text, audio, and video.Other examples request multilingual audiobook conversion and generation of an Arabic sentiment-analysis corpus from English text.Source: C Query-Pipeline Dataset · p. 11
Modality Coverage
- The dataset distinguishes basic text transformations in image/video and speech/audio tasks from multimodal functions that process multiple modalities together.The described multimodal example processes both image and text inputs.Source: C Query-Pipeline Dataset · p. 11
- Reference pipelines cover all types of transformations between the four modalities.Figure 17 presents the frequency of modality conversions required by dataset queries.Source: C Query-Pipeline Dataset · p. 11, Figure 17 · p. 11