Reducing hallucination in structured outputs via Retrieval-Augmented Generation

Patrice Béchard; Orlando Marquez Ayala

Source-linked AI summary

Reducing hallucination in structured outputs via Retrieval-Augmented Generation

Patrice Béchard, Orlando Marquez Ayala

arXiv:2404.08189v1Apr 12, 2024cs.LGcs.AIcs.CLcs.IR

TL;DR

The paper tackles hallucinations and limited generalization in natural-language workflow generation, which can undermine adoption of real-world GenAI systems. It applies RAG to structured JSON workflow generation and reports reduced hallucination, improved out-of-domain generalization, and resource-efficient deployment with a small retriever and LLM.

Problem
LLM-based workflow generation can hallucinate, especially for out-of-distribution enterprise inputs, while per-enterprise fine-tuning may be prohibitively expensive.
Method
The system retrieves relevant workflow steps and tables, appends them to the user query, and fine-tunes an LLM to generate the workflow as JSON.
Results
RAG reduces hallucinated steps and tables from as high as 21% without a retriever to below 7.5% and 4.5%, respectively, across StarCoderBase LLMs.
Takeaways & Limitations
A very small retriever can be coupled with a small LLM for structured generation in limited-resource settings without loss of performance.
Takeaways & Limitations
The system’s internal labeling data is mostly IT-domain data, so its behavior in other domains requires assessment.

Abstract

from arXiv · show

A common and fundamental limitation of Generative AI (GenAI) is its propensity to hallucinate. While large language models (LLM) have taken the world by storm, without eliminating or at least reducing hallucinations, real-world GenAI systems may face challenges in user adoption. In the process of deploying an enterprise application that produces workflows based on natural language requirements, we devised a system leveraging Retrieval Augmented Generation (RAG) to greatly improve the quality of the structured output that represents such workflows. Thanks to our implementation of RAG, our proposed system significantly reduces hallucinations in the output and improves the generalization of our LLM in out-of-domain settings. In addition, we show that using a small, well-trained retriever encoder can reduce the size of the accompanying LLM, thereby making deployments of LLM-based systems less resource-intensive.

1 Introduction

The paper addresses hallucination in natural-language workflow generation, where untrustworthy or out-of-distribution outputs can hinder enterprise use. It proposes RAG to improve trustworthiness while supporting smaller, resource-efficient deployments.

Natural-language workflow generation can lower the specialized-knowledge barrier for automating repetitive enterprise processes.
Naive LLM use can produce untrustworthy outputs, particularly when workflow requirements are out of distribution.
RAG is applied to workflow generation to reduce hallucination and improve the trustworthiness of JSON workflow outputs.
Fine-tuning separately for each enterprise may be prohibitively expensive, while deployment constraints favor the smallest LLM that can perform the task.
The paper demonstrates that RAG reduces hallucination, improves results, and permits a smaller LLM paired with a very small retriever without performance loss.

2 Related Work

Prior work addresses hallucination and structured-output generation through retrieval and constrained decoding. This paper’s setting differs because natural-language queries retrieve structured JSON objects, requiring semantic alignment across modalities.

Retrieval-Augmented Generation: RAG retrieves relevant information before generation so the model bases its output on specific data sources.This approach has been used to limit false or outdated information in question answering and summarization.
Dense Retrieval: Dense retrieval represents queries and documents in a shared multidimensional semantic space, often outperforming lexical methods such as TF-IDF and BM25.The cited applications primarily involve unstructured queries and documents.
Structured Retrieval: This work addresses retrieval between unstructured natural-language queries and structured JSON objects containing steps and tables.Its retrieval training is similar to SANTA, which aligns code and text semantics.
Structured Output: Structured-output tasks require both syntactically valid outputs and entities or field values drawn from a permitted lexicon.Examples include text-to-code, text-to-SQL, and if-then program synthesis.
Large Language Models: Code LLMs have made structured generation more accessible by generating code snippets from instructions and supporting tasks requiring reasoning.These models are trained on large source-code datasets.
Guided Generation: Guided generation can enforce grammar-valid steps, but it does not supply knowledge about which steps a natural-language query requires.The paper presents this as complementary to hallucination-reduction techniques based on retrieval.
System Architecture: Figure 2 depicts a high-level architecture in which the user query feeds both the retriever and the LLM before structured JSON generation.The figure is described as an architecture diagram rather than an evaluation result.

3 Methodology

The method trains a domain-specific retriever to map natural-language workflow requirements to existing steps and database tables, then supplies those suggestions to a separately fine-tuned LLM. The LLM generates JSON workflows from this augmented prompt using greedy decoding and a concise training format.

Inference Pipeline: The system indexes steps and tables, retrieves suggestions for each user request, appends them to the query, and generates a workflow as JSON via greedy decoding.Figure 2 presents this high-level pipeline.
Training Strategy: The retriever and LLM are trained separately: the retriever aligns natural language with JSON objects, while the LLM is fine-tuned with retrieved outputs in its prompt.This design was chosen for simplicity rather than end-to-end joint training.
Problem Setting: Hallucination risk concentrates in step names and database table names because workflows may draw from tens of thousands of customer-specific steps and trigger-dependent tables.The retriever therefore maps natural language to existing step and table names.
Retriever Architecture: A siamese transformer with mean pooling and normalized embeddings encodes the query, step, and table into fixed-length vectors.The retriever defines separate embeddings for queries, steps, and tables.
Training Data: Training pairs map each query to relevant workflow steps and, when required by the trigger, to one or more tables; queries may map to zero tables.A workflow with four steps produces four positive pairs, while a daily trigger example maps to an empty table list.
Retriever Training: Contrastive training minimizes distance for positive query-step or query-table pairs and separates negative pairs using cosine-based distance.Negative pairs are sampled with random, BM25-based, or ANCE-based strategies.
Retrieval: At inference, FAISS indexes support cosine-similarity retrieval of the maximum K steps and tables associated with a natural-language requirement.The incoming query is embedded with the retriever before lookup.
RAG Prompting: The LLM receives retrieved JSON objects in its input, enabling it to copy relevant objects during structured-output generation.Suggested steps and tables precede the user query in the training prompt.

4 Experiments

The experiments use internally collected and generated workflow data, including out-of-domain splits, and evaluate both retrieval and end-to-end generation. They also examine model-size trade-offs under production infrastructure constraints.

4.1 Datasets: The dataset combines around 4,000 deployed workflows with around 1,000 deterministic samples for incremental workflow construction.These data support retriever training and system evaluation.
4.1 Datasets: The internal datasets are mostly IT-domain data, so five additional real-user workflow splits assess out-of-domain behavior.The authors identify domain concentration as a limitation of the labeling approach.
4.1 Datasets: The percentage of steps absent from training ranges from less than 10% to more than 70% across out-of-domain splits.This variation motivates retrieval and deployment-specific index customization.
4.2 Metrics: End-to-end evaluation uses Trigger Exact Match, Bag of Steps, Hallucinated Tables, and Hallucinated Steps.The hallucination metrics measure invented tables or steps, and lower is better for this metric family.
4.2 Metrics: Retriever evaluation uses Recall@15 for steps and Recall@10 for tables, checking whether retrieved items cover the workflow JSON contents.The retriever returns top-K items from separate step and table indices.
4.3 Models: Models are compared across StarCoderBase sizes from 1B to 15.5B, alongside CodeLlama-7B and Mistral-7B-v0.1, under a 7B deployment limit.The retriever uses a 110M-parameter all-mpnet-base-v21 encoder and is compared with larger off-the-shelf GTR-T5 models.

5 Results

RAG improves workflow-generation quality by reducing hallucinations while retaining strong structured-output performance, including in out-of-domain settings. Fine-tuned small retrievers support smaller, more deployable LLM systems, though retrieval and structural-generation errors remain.

5.1 Retriever encoder: Fine-tuning the retriever, rather than scaling off-the-shelf encoders, was crucial for significantly improving step and table retrieval.Scaling GTR-T5 encoder size did not yield significant improvements on both retrieval metrics.
5.1 Retriever encoder: 110M-parameter all-mpnet-base-v2 yielded the best performance after fine-tuning across all negative sampling strategies.The small encoder was selected for deployment considerations.
5.2 Retrieval-Augmented Generation: Hallucinated steps fell below 7.5% and hallucinated tables below 4.5% with retrieval, compared with up to 21% without a retriever on Human Eval.All models produced valid JSON documents following the expected schema after fine-tuning.
5.2 Retrieval-Augmented Generation: RAG produced more consistent improvements as StarCoderBase model size increased, and the 3B RAG model outperformed larger recent 7B alternatives across all reported metrics.CodeLlama-7B and Mistral-7Bv0.1 were worse even than the smaller StarCoderBase-3B when also fine-tuned with RAG.
5.2 Retrieval-Augmented Generation: A 3B RAG fine-tuned model was competitive with the 15.5B non-RAG model on Trigger EM and Bag of Steps while keeping hallucination low.The 7B model offered the best trade-off because the performance difference from 15.5B was marginal.
5.3 OOD evaluation: On average, RAG made all out-of-domain metrics similar to the in-domain Human Eval results across five OOD splits.The evaluation used a weighted average based on the number of samples per split.

6 Conclusion

The paper proposes Retrieval-Augmented Generation for structured workflow output, reducing hallucination and supporting deployment with a small retriever and small LLM in resource-limited settings.

RAG reduces hallucination and supports generalization in structured workflow generation.The paper presents this as an approach for deploying retrieval-augmented LLMs in structured output tasks.
A very small retriever can be coupled with a small LLM for limited-resource deployment.The authors identify improving retriever–LLM synergy through joint training or architectural changes as future work.

Ethical Considerations

The system reduces hallucination but does not claim to eliminate harm from hallucinated workflow content, so users are prompted to review flagged output.

The approach does not eliminate the risk of harm caused by hallucination.
Post-processing identifies generated steps that do not exist and urges users to fix them before continuing.

A Training details for LLM and retriever

The LLMs and retriever were fine-tuned with specified AdamW-based training configurations, including fixed schedules, batch sizes, and LoRA for the LLMs.

LLMs used AdamW with learning rate 5e −4, 5,000 training steps, cosine scheduling, and effective batch size 32.The LLMs used 100 warmup steps and gradient accumulation when needed.
LLM fine-tuning used LoRA with r = 16, α = 16, and dropout rate 0.05.
The retriever used AdamW with learning rate 2e −5, batch size 128, and 10 training epochs.Retriever fine-tuning used the SentenceTransformers framework.

B Differences in generation with and without suggestions

Suggestions change generation behavior: they increase step-name diversity, while their absence leads to more invented names; table-name behavior is more conservative when suggestions are provided.

Step names: Without suggestions, the RAG-fine-tuned StarCoderBase-7B generates significantly fewer unique step names.Suggestions allow the model to copy retrieved names, increasing generation diversity.
Step names: Without suggestions, a greater percentage of unique step names are invented.The model is more conservative about generating steps without suggestions, relying on steps seen during training.
Step names: Even with suggestions, some splits contain more than 10% hallucinated unique step names despite an overall hallucination rate below 2%.The text attributes remaining cases to retrieval misses or the LLM not using suggestions.
Table names: With table-name suggestions, the model generates fewer unique table names, possibly because the data contains less table diversity.Without suggestions, a greater percentage of unique table names are invented.

C Sample perfect output and errors

Figure 4 presents three workflow-generation cases: one where the retriever and LLM succeed, one where retrieval causes a step substitution, and one where the LLM misinterprets flow logic.

The first example shows the LLM following the requested workflow structure and using the expected steps.The retriever suggests only post_incident_details because the remaining steps are considered common.
In the second example, retrieval omits send_slack_message, leading the LLM to use send_notification instead.The resulting workflow remains partly correct but is lower quality because it does not match the user's expectation.
In the final example, the LLM fails to apply TRY and CATCH flow logic implied by the word Try in the query.The generated workflow therefore does not reflect the requested logic.