Source-linked AI summary
Learning to Reason and Use Tools through Unsupervised Fine-Tuning in Task-Oriented Dialog Systems
Markel Ferro, Oier Lopez de Lacalle
TL;DR
TOD systems often rely on static knowledge, causing factual inconsistencies when tasks require dynamic information retrieval. This paper adapts ReAct and develops an unsupervised trajectory-based fine-tuning loop, finding stronger baseline performance and enabling an 8B model to match or surpass a 70B in-context system.
Problem
LLM-based TOD systems lack dynamic access to real-time information and therefore can generate factually inconsistent responses during task completion.
Method
The method generates reasoning trajectories with in-context learning, filters them using an LLM judge, and iteratively fine-tunes models on the retained data.
Results
On SIMMC, the approach outperforms baseline LLMs, remains more robust in high-object-density scenes, and lets Llama 3 8B match or surpass a Llama 3 70B in-context system.
Takeaways & Limitations
ReAct grounding and self-improvement support factual dynamic retrieval, efficient smaller-model deployment, and cross-domain generalization within the evaluated setting.
Takeaways & Limitations
Experiments are conducted exclusively on SIMMC 2.1, so broader evaluation across task-oriented dialogue datasets remains necessary.
Abstract
from arXiv · showhide
Current dialogue systems struggle with dynamic information retrieval, often leading to hallucinations and lower response accuracy. We address this by adapting the ReAct framework for Task-Oriented Dialogue, enabling Large Language Models (LLMs) to access external knowledge and produce factual responses. Mainly, we propose an unsupervised fine-tuning pipeline that harvests reasoning trajectories via in-context learning inference. High-quality samples are filtered using an LLM-based judge to construct a robust training set. This is enhanced by a unsupervised self-improvement loop, where improved checkpoints generate increasingly better trajectories for subsequent fine-tuning iterations. Experiments on the SIMMC dataset demonstrate that ReAct-based systems outperform baselines due to superior reasoning and tool use. Notably, our fine-tuned 8B model surpasses a 70B in-context system. Finally, we present an error analysis, impact of scene complexity, and cross-domain generalization.
1 Introduction
End-to-end TOD systems can hallucinate because they lack dynamic access to current external information. This work adapts ReAct and unsupervised fine-tuning to improve reasoning, tool use, robustness, and transfer.
- LLM-based TOD systems can produce factually inconsistent responses because they rely on static training knowledge and lack dynamic information access.
- ReAct guides TOD agents through iterative thoughts, actions, and observations, using external tools to decompose tasks and retrieve information.
- The proposed pipeline generates reasoning trajectories through in-context inference and filters them with an LLM judge before fine-tuning.
- Explicit reasoning and tool use improve access to external knowledge compared with approaches limited to input-context information.
- 8B-parameter models can match or surpass 70B-parameter counterparts through unsupervised iterative self-improvement, reducing computational and resource requirements.
- ReAct-based methods degrade less rapidly than prompting-based methods as scene complexity and object count increase.
2 Related Work
Related work identifies weaknesses in both TOD evaluation and agentic dialogue systems. Existing metrics can misjudge multi-object responses, while applying ReAct to constrained TOD settings requires careful adaptation.
- Agentic ReAct applications in structured TOD are non-trivial because models may imitate few-shot examples and generate inconsistent reasoning.
- Traditional overlap metrics and TOD success metrics can inadequately assess conversational quality and object-specific correctness.
- String-matching evaluation may mark responses as successful when attributes are present but refer to the wrong object, especially in multi-object responses.
- LLM judges offer an alternative to costly human evaluation by assessing generated outputs with language understanding.
- Open-source judges such as Prometheus 2 support controllable, user-defined evaluation criteria and both scoring and pairwise ranking.
- SIMMC and MultiWOZ are prominent TOD datasets, but models trained on them may struggle to adapt to new tasks.
3 ReAct as Dialogue System
The system adapts ReAct to SIMMC response generation by retrieving relevant scene-object information with specialized tools. It combines visual and non-visual metadata through iterative tool use before producing a final response.
- The study focuses on Automatic Response Generation in SIMMC 2.1 without access to ground-truth annotations.
- Each dialogue occurs in a multi-object scene, requiring accurate grounding in relevant visual and textual object metadata.
- The adapted ReAct agent actively retrieves relevant object information with custom tools, limiting unnecessary access to unrelated information.
- Look[] returns visual metadata for all scene objects, while Search[query] retrieves non-visual metadata satisfying specified constraints.
- Finish[answer] terminates tool use and specifies the response returned to the user.
- For an inexpensive-shoes query, the agent visualizes shoes, searches for cheap items, combines both observations, and reports relevant prices.
- The initial in-context behavior is followed by a separate unsupervised fine-tuning pipeline.
4 Unsupervised Adaptation
The unsupervised adaptation pipeline creates training data from model-generated reasoning trajectories, filters low-quality examples, and fine-tunes the model without human supervision. Iterative self-improvement expands the trajectory set as the model improves.
- The pipeline addresses scarce environment-specific reasoning data through generation, filtering, and fine-tuning.
- In-context inference generates thought–action–observation trajectories, assuming correct final responses reflect valid reasoning patterns.
- An LLM-based judge automatically selects high-quality trajectories for fine-tuning without human intervention.
- Only trajectories receiving a perfect score of 5 on the judge’s Likert scale are retained, with the threshold treated as tunable.
- The first fine-tuned model, ReAct FT, is trained on the filtered dataset using LoRA and validation-based checkpoint selection.
- Iterative fine-tuning lets an improving Llama 8B model generate better trajectories and enrich the training set until no further judge-detected gains appear.
5 Experimental setting
The experiments use SIMMC 2.1 response generation across fashion and furniture scenes, comparing ReAct systems with baselines under manual and LLM-assisted evaluation. The setup examines scene complexity, unsupervised trajectory filtering, and generalization constraints.
- SIMMC 2.1 contains 11,244 task-oriented dialogues and 117,236 utterances, with models conditioned on two preceding dialogue turns.
- Fashion scenes average 30 objects and furniture scenes 8.9 objects, while each object has 11 versus 7 attributes, respectively.The paper defines scene complexity through object density and metadata richness, making fashion the more challenging domain.
- The dataset’s visual scene information is converted into textual metadata so unimodal models can use object information.
- The study compares Blind and All-in-context baselines, both using Llama 3.3 70B Instruct.Blind excludes object information, whereas All-in-context supplies all object metadata in the input.
- Direct comparison with state-of-the-art models is problematic because their dataset-specific training procedures can produce overfitting and weak out-of-distribution generalization.
- Prometheus 2 correlates more strongly with manual annotations than BLEU4 but tends to overestimate baseline performance, increasing false positives.Accordingly, Prometheus is primarily used to filter trajectories in the unsupervised self-improvement pipeline, with manual checks for preference overfitting.
6 Experimental Results
Manual evaluation across fashion and furniture shows that ReAct systems outperform the baselines, while fine-tuning improves over in-context learning. Smaller 8B systems can be competitive with or surpass larger 70B systems, although scene complexity affects performance.
- ReAct 70B systems outperform the baselines, and fine-tuned models generally surpass their corresponding in-context learning systems.
- 65% vs. 60%: ReAct FT 8B significantly outperforms ReAct ICL 70B in the fashion domain despite having fewer parameters.
- USI ReAct 8B surpasses ReAct ICL 70B in both domains and is comparable to ReAct FT 70B in fashion while improving furniture performance by +8%.Its fashion result is within -1% of ReAct FT 70B, while its smaller size offers reduced inference cost but requires more training.
- Systems using metadata perform better in furniture than fashion, and the ReAct ICL 70B advantage over All-in-context is smaller in furniture.The paper attributes this gap to the greater number of objects in fashion scenes and their resulting task complexity.
7 Analysis
The analysis examines intent-specific accuracy, self-improvement, scene complexity, trajectory filtering, and cross-domain transfer. Results show that iterative refinement, clean reasoning trajectories, and ReAct-based retrieval improve robustness, while evaluation scope and trajectory-quality assessment remain constrained.
- 7.1 Intent Analysis: USI ReAct 8B achieves the best overall results across most intents, especially improving INFORM performance, but its ASK:GET accuracy decreases.ReAct ICL 70B gains on single-object reasoning, while ReAct FT 70B improves nearly all intent types.
- 7.2 Evolution of Self-Improving: Prometheus validation scores rise from 2.77 to 3.67 after five self-improvement iterations, while score-5 trajectories increase in both domains.Fashion trajectories increase from 5,492 to 11,073 and furniture trajectories from 2,342 to 4,408.
- 7.2 Evolution of Self-Improving: The first iteration increases filtered instances by 55% in fashion and 43% in furniture, whereas the sixth fine-tuning step performs worse than the fifth.Subsequent iterations produce smaller gains, motivating termination after five iterations.
- 7.3 Effect of Scene Complexity: As scene object counts increase, the All-in-context baseline degrades substantially, while ReAct ICL 70B remains more robust under complex conditions.The analysis uses synthetic objects to create scenes containing 20 to 150 total objects.
- 7.4 Filtering noisy triplets: Filtered trajectories improve fine-tuned systems, with ReAct FT 8B improving by up to 7 points after noisy triplets are removed.The filtering heuristic removes malformed actions and repeated Look[] calls that yield identical results.
- 7 Analysis: Cross-domain evaluation shows a substantially smaller performance drop than the BART baseline, although the Prometheus analysis is limited to the fashion validation partition.A fashion-trained model achieves higher scores on furniture, while BART models show a pronounced decline.
8 Conclusions
The paper adapts ReAct to task-oriented dialogue and combines dynamic information retrieval with unsupervised fine-tuning on reasoning trajectories. On SIMMC, the approach outperforms in-context baselines, remains more robust in complex scenes, and enables an 8B model to match or surpass a 70B in-context system.
- ReAct adaptation grounds task-oriented dialogue models through dynamic information retrieval and leverages reasoning trajectories without human feedback.
- The approach outperforms baseline LLMs relying solely on in-context information and is more robust under high object density.
- A self-improvement pipeline enables a Llama 3 8B model to match or surpass a Llama 3 70B in-context system.
- Filtering inefficient reasoning steps improves inference efficiency and overall performance.
- Fine-tuning on reasoning trajectories supports strong cross-domain generalization.
9 Limitations
The evaluation is limited to SIMMC 2.1, excludes direct visual input, and relies mainly on manual assessment of a few hundred examples. The authors identify broader datasets, visual integration, and scalable automatic evaluation as future needs.
- Experiments use exclusively the SIMMC 2.1 dataset, limiting assessment across a broader range of task-oriented dialogue datasets.
- The method uses textual and structured metadata but omits direct visual input despite SIMMC 2.1’s multimodal nature.
- Results rely primarily on manual evaluation, restricting assessment to a few hundred examples.
- The authors call for reliable automatic evaluation methods to analyze performance at larger scale.
A Prompts
The appendix presents the prompts used for baseline, ReAct, and Prometheus evaluation systems. These materials distinguish prompt variants across baseline and ReAct domains and include the operational tool instructions and evaluation rubric.
- A.1 Baselines prompts: Figure 4 presents the blind baseline prompt.
- A.1 Baselines prompts: Figure 5 presents the all-in-context baseline prompt.
- A.1 Baselines prompts: The baseline prompt begins with previous dialogue turns and the current user question.
- A.2 ReAct systems prompts: Figure 6 presents the ReAct ICL 70B fashion prompt.
- A.2 ReAct systems prompts: Figure 7 presents the ReAct ICL 70B furniture prompt.
- A.2 ReAct systems prompts: The ReAct prompt includes operators, a Look[] action for visible items, and Finish[answer] for returning the response.
- A.3 Prometheus Evaluation prompt: Figure 8 presents the Prometheus Evaluation prompt.
- A.3 Prometheus Evaluation prompt: The evaluation prompt rates whether answers provide accurate information about relevant objects.