Learning to Reason and Use Tools through Unsupervised Fine-Tuning in Task-Oriented Dialog Systems

• The unsupervised pipeline generates reasoning trajectories through in-context learning inference, filters them with an LLM-based judge, and fine-tunes on selected instances.Source: §1 · p. 2
• Manual SIMMC evaluation covers fashion and furniture, synthetic fake objects for increased scene complexity, and the effects of inefficient trajectories on performance and computational efficiency.Source: §1 · p. 2
• Explicit reasoning and tool use improve access to external knowledge compared with approaches limited to input context, using iterative thoughts, actions, and observations.Source: §1 · p. 2
• 8B-parameter models match or even surpass 70B-parameter counterparts through iterative self-improvement, with reduced computational cost and resource requirements.Source: §1 · p. 2
• In high-complexity scenes with many objects, prompting-based methods degrade faster than the ReAct-based approach.Source: §1 · p. 2
• Fine-tuning on reasoning trajectories mitigates in-domain overfitting and improves generalization to unseen domains.Source: §1 · p. 2

TOD systems with agents

• ReAct gives LLMs access to external information, but applying agentic frameworks to structured task-oriented dialogue can prompt imitation of few-shot examples and inconsistent reasoning.Related work also describes systems for reasoning, planning, and tool use, including multi-agent architectures for managing real-world dialogue complexity.Source: §2 · p. 3

Evaluation of TOD systems

• BLEU, ROUGE, Inform Rate, and Success Rate can misjudge dialogue quality because overlap and string matching may miss whether an attribute refers to the correct object.This problem is exacerbated in multi-object responses, and metrics focused on final outcomes can make systems appear successful while remaining unreliable in real-world scenarios.Source: §2 · p. 3

LLM judges

• LLM judges aim to reduce costly human evaluation; GPT-4 is used for alignment with human judgments, while Prometheus 2 offers controllable direct assessment and pairwise ranking.Prometheus 2 supports evaluation against custom, user-defined criteria.Source: §2 · pp. 3–4

Task-oriented Dialogue Datasets

• SIMMC and MultiWOZ are prominent multi-turn task-dialogue datasets, yet models trained on them may struggle to adapt to new tasks, motivating dynamic adaptation strategies.Their tasks range from simulated police interactions to clothing purchases.Source: §2 · p. 4

Problem formulation

• The study focuses on SIMMC 2.1 Automatic Response Generation, where the assistant uses dialogue history, the current query, and scene-object metadata without ground-truth annotations.The task requires identifying relevant objects and grounding responses in their visual and textual metadata.Source: §3 · p. 4

ReAct

• The adapted ReAct agent generates responses through iterative thought, action, and observation steps, using custom retrieval tools to access relevant object information.The agent receives an instruction prompt, dialogue history, and user query; restricting retrieval to relevant information helps avoid unnecessary access to unrelated data.Source: §3 · p. 4
• Look[] exposes visual metadata, Search[query] retrieves objects matching non-visual metadata constraints, and Finish[answer] returns the final response and ends tool use.For example, Search[customerRating>3] retrieves objects rated above 3.Source: §3 · p. 4
• In the inexpensive-shoes example, the system uses Look[] to inspect shoes, Search[] to find cheap items, then Finish[] to report relevant prices.Prompts were adapted to the task and refined using preliminary experiments; initial behavior comes from in-context learning, and the following section introduces unsupervised fine-tuning.Source: §3 · pp. 4–5

4 Unsupervised Adaptation

• The three-stage pipeline generates thought–action–observation trajectories through in-context learning, filters for quality, and fine-tunes on trajectories whose final responses are correct.The approach assumes correct final responses indicate valid reasoning patterns.Source: §4 · p. 5
• Prometheus 2 evaluates final-response factual accuracy, and only trajectories scoring a perfect 5 on its Likert scale are retained.The threshold was set through preliminary experiments and can be tuned for the task and dataset.Source: §4 · p. 5
• Filtered trajectories are used for supervised fine-tuning with LoRA; the best validation-split checkpoint, selected using Prometheus, is ReAct FT.Filtering and fine-tuning are performed without human supervision, with LoRA used to accommodate resource constraints.Source: §4 · p. 5
• USI ReAct 8B iteratively fine-tunes the Llama 8B base model on an expanding set of high-quality trajectories until Prometheus detects no further performance gains.As the model improves, it generates better trajectories than those from the initial ReAct ICL 70B stage.Source: §4 · p. 5

Dataset and hyperparameters

• SIMMC 2.1 provides 11,244 dialogues and 117,236 utterances; models generate responses using scene metadata and the two preceding dialogue turns as context.Visual scene information is converted into textual metadata for unimodal models.Source: §5 · p. 6
• Fashion scenes average 30 objects with 11 attributes each, versus 8.9 objects and 7.2 attributes in furniture, making fashion more context-intensive.The paper defines scene complexity through object density and metadata richness, which increase context-window saturation.Source: §5 · p. 6
• ReAct systems use Llama 3.3 70B Instruct or Llama 3.1 8B Instruct, with five manually curated few-shot trajectories for ICL and rank-32 LoRA for fine-tuning.The LoRA configuration accommodates resource constraints.Source: §5 · p. 6

Baselines

• The Blind baseline omits object information, while All-in-context supplies all object metadata; both use Llama 3.3 70B Instruct.Direct comparison with state-of-the-art models is problematic because their training procedures often overfit specific datasets.Source: §5 · p. 6

Evaluation

• Final evaluation uses blind manual annotation of 100 randomly selected instances per domain because lexical-overlap metrics can reward hallucinated details or penalize correct paraphrases.For example, a response giving a price of $30 instead of $300 may still receive a high metric score.Source: §5 · p. 6
• Prometheus 2 correlates more strongly with manual annotations than BLEU4 for filtering, but tends to overestimate baselines and can produce false positives.Its use is limited mainly to filtering in the self-improvement pipeline, where manual evaluation checked for overfitting to its preferences.Source: §5 · p. 6

• All systems were judged correct or incorrect through blind manual assessment on the same evaluation instances to ensure comparability.The system results are presented in Table 1.Source: §6 · p. 7
• ReAct FT 8B achieves 65% versus ReAct ICL 70B’s 60% in fashion, and fine-tuned models generally surpass their in-context counterparts.Source: Table 1 · p. 7, §6 · p. 7
• USI ReAct 8B surpasses ReAct ICL 70B in both domains, is 1% below ReAct FT 70B in fashion, and improves furniture performance by 8%.Its smaller parameter size offers reduced inference computational cost despite higher training requirements.Source: §6 · p. 7
• Systems using metadata perform better in furniture than fashion, while the All-in-context versus ReAct ICL 70B gap is smaller in furniture because fashion scenes contain more objects.The passages attribute the furniture advantage to its simpler nature and the greater number of objects in fashion scenes to increased task complexity.Source: §6 · p. 7

7.1 Intent Analysis

• USI ReAct 8B achieved the best results across most intents and improved INFORM performance, but its accuracy fell on ASK:GET.ReAct FT 70B improved nearly all intent types, while All-in-context particularly helped multi-object queries and ReAct ICL 70B gained on single-object reasoning tasks.Source: Table 2 · p. 8, §7.1 · p. 8

7.2 Evolution of Self-Improving

• Prometheus validation scores rose from 2.77 to 3.67 over five iterations, while score-5 trajectories increased in both fashion and furniture.Fashion trajectories grew from 5,492 to 11,073 and furniture trajectories from 2,342 to 4,408. The first iteration increased filtered instances by 55% in fashion and 43% in furniture; gains then diminished, and training stopped after performance worsened at iteration six. Prometheus scoring used only the fashion validation partition, limiting the scope of that analysis.Source: §7.2 · p. 8, Table 3 · p. 9

7.3 Effect of Scene Complexity

• As scene object counts increased, All-in-context performance degraded substantially, while ReAct ICL 70B was more robust.The analysis augmented fashion-scene metadata with synthetic objects, varying scene totals from 20 to 150.Source: Figure 3 · p. 9, §7.3 · p. 9

7.4 Filtering noisy triplets

• Filtering noisy triplets improved most fine-tuned models, with ReAct FT 8B improving by up to 7 points.The heuristic removes malformed actions and repeated consecutive Look[] calls; cleaned trajectories also reduce generated triplets, while holistic trajectory judging may still reward long or inefficient reasoning.Source: §7.4 · pp. 9–10, Table 4 · p. 9, Table 5 · p. 10

7.5 Generalization of Fine-tuned ReAct

• Cross-domain evaluation showed a substantially smaller performance drop for the fine-tuned ReAct approach than for BART, and fashion-trained ReAct scored higher on furniture.BART models declined markedly on out-of-domain instances.Source: Table 6 · p. 10, §7.5 · p. 10

8 Conclusions

• Adapting ReAct to task-oriented dialogue grounds models in their environment through dynamic information retrieval, mitigating hallucinations without relying on human feedback for unsupervised fine-tuning.Source: §8 · p. 10
• On SIMMC, the self-improvement pipeline enables a Llama 3 8B model to match or surpass a Llama 3 70B in-context system and improves robustness in complex scenarios.The approach also outperforms baseline LLMs that rely solely on in-context information.Source: §8 · p. 10
• Filtering inefficient reasoning steps improves inference efficiency and overall performance, while fine-tuning on reasoning trajectories fosters strong cross-domain generalization.Source: §8 · p. 10

Dataset coverage

• Experiments use only SIMMC 2.1, so evaluation on more task-oriented dialogue datasets is needed to assess robustness more comprehensively.SIMMC 2.1 is multimodal and spans multiple domains, but the paper calls for broader dataset evaluation.Source: §9 · p. 10

Visual input

• The method uses textual and structured metadata but not direct visual input, making integration of visual signals an important future direction.This limitation remains despite SIMMC 2.1’s multimodal nature.Source: §9 · p. 10

Evaluation

• Manual evaluation covers only a few hundred examples, motivating reliable automatic evaluation for scalable analysis across larger example sets.Source: §9 · p. 10

A.1 Baselines prompts

• The appendix includes separate blind-baseline and all-in-context-baseline prompts.Source: Figure 4 · p. 14, Figure 5 · p. 14

A.2 ReAct systems prompts

• ReAct ICL 70B prompts are shown for both fashion and furniture.Source: Figure 6 · p. 14, Figure 7 · p. 14
• The furniture ReAct prompt frames the model as a shop seller answering client questions through interleaved Thought, Action, and Observation steps.Source: §A.2 · p. 14
• The prompt specifies Search, Look, and Finish actions, catalogue fields and operators for Search, and visible-item attributes returned by Look.Search must use the listed catalogue parameters; Look reveals position, corner_pos, type, color, and prefab_path, while Finish returns an answer to the client.Source: §A.2 · pp. 14–15
• Few-shot examples are included, with an instruction not to reuse information from example conversations and to ask again for needed information.Source: §A.2 · p. 15

A.3 Prometheus Evaluation prompt

• The Prometheus evaluation material includes a prompt and a rubric fragment describing accurate information about relevant objects as the top-score criterion shown.The excerpt also refers to incorrect information or information about irrelevant objects, but does not specify the intervening score levels.Source: Figure 8 · p. 15, §A.3 · p. 15