Source-linked AI summary
Beyond Quacking: Deep Integration of Language Models and RAG into DuckDB
Anas Dorbani, Sunny Yasser, Jimmy Lin, Amine Mhedhbi
TL;DR
Knowledge-intensive analytical pipelines combine structured and unstructured data but require costly orchestration and low-level LLM management. FlockMTL integrates LLM and RAG operations into DuckDB through SQL functions, resource objects, and automatic optimizations. Its batching can deliver up to 7× speedup for chat-completion functions and 48× for embedding functions on the Kaggle Bank Review dataset.
Problem
Knowledge-intensive analytical pipelines require heterogeneous data integration and substantial manual management of models, prompts, context, caching, and execution decisions.
Method
FlockMTL extends DuckDB with composable scalar and aggregate LLM functions, reusable MODEL and PROMPT objects, hybrid search, and automatic batching and prompt optimizations.
Results
Up to 7× speedup for chat-completion map functions and 48× for embedding-based functions was achieved through batching on the Kaggle Bank Review dataset.
Takeaways & Limitations
FlockMTL enables users to build semantic and analytical data applications in SQL while reducing the implementation burden of LLM pipelines.
Abstract
from arXiv · showhide
Knowledge-intensive analytical applications retrieve context from both structured tabular data and unstructured, text-free documents for effective decision-making. Large language models (LLMs) have made it significantly easier to prototype such retrieval and reasoning data pipelines. However, implementing these pipelines efficiently still demands significant effort and has several challenges. This often involves orchestrating heterogeneous data systems, managing data movement, and handling low-level implementation details, e.g., LLM context management. To address these challenges, we introduce FlockMTL: an extension for DBMSs that deeply integrates LLM capabilities and retrieval-augmented generation (RAG). FlockMTL includes model-driven scalar and aggregate functions, enabling chained predictions through tuple-level mappings and reductions. Drawing inspiration from the relational model, FlockMTL incorporates: (i) cost-based optimizations, which seamlessly apply techniques such as batching and caching; and (ii) resource independence, enabled through novel SQL DDL abstractions: PROMPT and MODEL, introduced as first-class schema objects alongside TABLE. FlockMTL streamlines the development of knowledge-intensive analytical applications, and its optimizations ease the implementation burden.
1 INTRODUCTION
Knowledge-intensive applications combine structured and unstructured data with semantic analysis, but current pipelines require substantial low-level orchestration. FlockMTL addresses this burden by integrating LLM operations, resource abstractions, and optimizations into DuckDB.
- Current LLM pipelines require engineers to manage model selection, prompts, context, caching, and optimizations, while adapting to quality, latency, cost, and scale changes.
- FlockMTL integrates semantic operations into SQL through scalar and aggregate functions, including classification, summarization, reranking, and hybrid-search support.
- MODEL and PROMPT are first-class schema objects, allowing model and prompt updates without changing application logic.
- FlockMTL automatically handles context management, batching, caching, and deduplication so developers can focus on higher-level application logic.
2 SYSTEM OVERVIEW
FlockMTL extends DuckDB with reusable model and prompt resources, composable LLM functions, hybrid retrieval, and automatic inference optimizations. Its SQL interface supports chained semantic analysis and retrieval pipelines while hiding key execution details.
- 2.1 Models and Prompts: MODELs and PROMPTs are reusable, local or global resources whose versioned updates remain inspectable while the latest version applies by default.
- 2.2 Functions: FlockMTL chains filtering, summarization, and structured extraction through composable scalar functions applied to research-paper tuples.
- 2.2 Functions: Its hybrid-search pipeline combines vector similarity and BM25 retrieval, fuses their results, and reranks the top ten passages with an LLM.
- 2.3 Optimizations: FlockMTL automatically batches tuples to fit the LLM context window and reduces batch size iteratively after context-limit errors.
- 2.3 Optimizations: 7× chat-completion speedup and 48× embedding-function speedup were achieved on the Kaggle Bank Review dataset with batching.
3 DEMONSTRATION
The demonstration lets users query tabular datasets through natural language, inspect generated SQL and execution plans, and examine FlockMTL’s optimization choices. It illustrates both SQL-based semantic applications and reduced developer burden.
- Users issue natural-language requests over Kaggle datasets, while ASK generates SQL augmented with FlockMTL functions and displays the results.
- Plan inspection exposes standard SQL operations alongside FlockMTL functions, meta-prompts, serialization formats, and automatically selected batch sizes.
- Users can alter batch size, serialization format, and prompt templates to compare performance, structure, and output trade-offs.
- The hands-on demonstration highlights SQL accessibility for semantic and analytical applications and the value of optimizations in reducing developer burden.