PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Study on Zero-Data Bootstrapping for Conversational Recommender Systems

Rohan Surana, Junda Wu, Zhouhang Xie, Yu Xia, Nathan Kallus, Julian McAuley

arXiv:2504.15476v2cs.IR

TL;DR

CRS typically requires costly, scarce domain-specific dialogue data, motivating bootstrapping from non-conversational signals. The paper systematically studies active selection, teacher-LLM generation, and target-model fine-tuning, finding that domain-grounded synthetic data is effective across evaluated settings. The evaluation is limited to the movie domain, with cross-domain transfer left for future work and possible synthetic-data artifacts.

  • Problem

    CRS typically requires large domain-specific conversational datasets that are costly and scarce, while non-conversational bootstrapping has not been systematically investigated.

  • Method

    The study formalizes zero-data CRS and evaluates active selection of non-conversational signals, teacher-LLM conversation generation, and target-model fine-tuning across experimental settings.

  • Results

    Domain-grounded synthetic data outperforms zero-shot and naïve synthetic baselines, while active selection improves data efficiency across models, benchmarks, and fine-tuning paradigms.

  • Takeaways & Limitations

    Systematically selected and transformed non-conversational signals provide a practical path to building CRS without conversational training data.

  • Takeaways & Limitations

    The evaluation is limited to the movie domain, so cross-domain generalization remains future work; generated data may also contain artifacts.

Abstract

from arXiv · show

Conversational Recommender Systems (CRS) typically require domain-specific dialogue data, which is costly, scarce, and often unavailable in new domains. We conduct a systematic empirical study of zero-data CRS bootstrapping: generating synthetic conversational supervision from non-conversational signals---item reviews, metadata, and user-item interactions---without any in-domain dialogue corpus. We compare two information-theoretic selection strategies, Jensen-Shannon diversity and Fisher information, across domain signals, model architectures, datasets, and fine-tuning paradigms. Our results show that domain-grounded synthetic data consistently outperforms zero-shot prompting and naive synthetic baselines; active selection improves data efficiency over random sampling; metadata and collaborative filtering signals each improve selection quality; and, in low-resource settings, synthetic data can outperform scarce real dialogues while further complementing them. These findings establish non-conversational domain signals as a viable path toward building CRS without conversational training data. The code is available at https://anonymous.4open.science/r/zero_data_crs/ .

1 Introduction

CRS development is constrained by scarce, costly conversational data, while direct LLM deployment has practical limitations. This study formalizes zero-data bootstrapping from non-conversational signals and systematically evaluates selection and domain-signal choices.

  • Domain-specific conversational datasets are costly and scarce because of annotation costs, privacy concerns, and domain-specific constraints.
  • Direct LLM deployment faces scalability, cost, interpretability, and privacy challenges, while smaller models still require limited conversational data for fine-tuning.
  • Existing CRS augmentation methods generally assume an existing conversational corpus or external knowledge graph, leaving non-conversational bootstrapping unstudied.
  • The study formalizes zero-data CRS and evaluates active selection, synthetic conversation generation, and target-model fine-tuning across signals, budgets, architectures, and paradigms.
  • The controlled framework isolates how selection strategies and domain signals affect synthetic-data quality and downstream recommendation performance.
  • Zero-data CRS operates without an in-domain conversational corpus or external knowledge graph, using non-conversational signals such as reviews, metadata, and interactions.

2 Zero-Data CRS Bootstrapping

Zero-data CRS bootstrapping derives conversational supervision from non-conversational signals through a staged pipeline. It selects seed samples, generates synthetic conversations, and fine-tunes a target CRS.

  • Zero-data CRS assumes no in-domain conversational dataset and instead uses item reviews, metadata, and user-item collaborative interactions.
  • The pipeline selects informative samples from seed data, generates synthetic conversations with a teacher LLM, and fine-tunes a target model on them.
  • The pipeline is organized into selection, synthetic conversation generation, and fine-tuning stages.

4 end

The study represents non-conversational seed items with target-model embeddings and applies two label-free information-theoretic selection criteria. Reviews, metadata, and collaborative signals provide complementary inputs for selection and downstream evaluation.

  • Selection uses target-model last-layer hidden states to represent seed samples before synthetic conversations and training targets exist.
  • JS diversity prioritizes distributional coverage, while Fisher information prioritizes parameter-level informativeness under a last-layer approximation.
  • JS selection favors high-entropy boundary samples and novelty relative to the already selected set.
  • Fisher selection uses incremental information gain, with a greedy log-determinant objective favoring parameter-relevant directions beyond those already selected.
  • The two criteria represent complementary notions of utility: distributional representativeness and parameter-aligned informativeness.
  • Reviews encode semantic opinions, metadata supplies structural context, and collaborative features encode user preference patterns.
  • Each signal type is concatenated with the seed input before embedding extraction, enabling comparisons of individual signals and combinations.

10 end

After active selection, a teacher LLM converts selected items and reviews into synthetic query–recommendation examples. The target model is then adapted using supervised fine-tuning with LoRA or Full-SFT.

  • For each selected item, sampled style templates and reviews condition a teacher LLM to generate synthetic conversational queries.
  • Each synthetic query receives 20 pseudo-target recommendations, and the resulting query–recommendation pairs form the fine-tuning dataset.
  • The target language model is adapted with supervised fine-tuning using either LoRA or Full-SFT.

3 Experimental Setup

The study evaluates zero-data CRS bootstrapping on ReDial and INSPIRED using multiple model architectures, fine-tuning paradigms, baselines, and active-selection signals.

  • 3.1 Benchmarks and Seed Data: Evaluation uses ReDial and INSPIRED as CRS benchmarks at different resource scales.Seed data comes from Amazon Reviews ’23 in the Movies & TV category.
  • 3.2 Models and Fine-Tuning: The study examines Llama3.2-3B-Instruct, Qwen2.5-1.5B-Instruct, and Qwen3-4B to study model-capacity effects.Qwen3-4B is included for RQ1 only.
  • 3.2 Models and Fine-Tuning: Fine-tuning compares LoRA parameter-efficient adaptation with Full-SFT, which optimizes all model parameters.GPT-4o serves as the teacher LLM for synthetic-data generation.
  • 3.3 Baselines and Selection Conditions: Zero-data baselines include zero-shot prompting, naïve GPT-generated data, popularity ranking, and NBCRS.These baselines operate without an in-domain conversational corpus.
  • 3.3 Baselines and Selection Conditions: Existing CRS augmentation methods are excluded because they require an in-domain conversational corpus and are inapplicable to the zero-data setting.The study therefore focuses on synthetic supervision generated from non-conversational signals.
  • 3.3 Baselines and Selection Conditions: Active selection compares random sampling, semantic-only JS/Fisher selection, metadata-augmented variants, and collaborative-filtering variants.Performance is evaluated with Recall@k and NDCG@k; RQ4 combines synthetic and in-domain data.

4 Main Results

Across benchmarks, architectures, budgets, and signal combinations, domain-grounded synthetic supervision and information-theoretic selection outperform naïve or frequency-based alternatives. Synthetic data is especially valuable when real conversational data is scarce, while combinations with real data can add further gains.

  • 4.1 RQ1: Domain-Grounded Synthetic Data Enables Zero-Data CRS: Domain-grounded synthetic data consistently outperforms zero-shot prompting and naïve GPT-generated training across backbones and benchmarks.On INSPIRED with Qwen2.5-1.5B, SFT achieves +207.8% Recall@1 over zero-shot versus +18.8% for GPT-generated data.
  • 4.1 RQ1: Domain-Grounded Synthetic Data Enables Zero-Data CRS: Qwen2.5-1.5B gains +41.8% (LoRA) and +40.6% (SFT) Recall@1 on ReDial, versus +18.3% and +9.0% for Qwen3-4B.The reported relative gains are larger for the smaller model.
  • 4.1 RQ1: Domain-Grounded Synthetic Data Enables Zero-Data CRS: Full-SFT generally outperforms LoRA, including +207.8% versus +82.8% Recall@1 for Qwen2.5-1.5B on INSPIRED.For Llama3-3B, LoRA sometimes underperforms zero-shot while Full-SFT consistently improves.
  • 4.1 RQ1: Domain-Grounded Synthetic Data Enables Zero-Data CRS: NBCRS Recall@5 on ReDial improves from 0.39 with raw seed data to 11.52 with synthesized conversational data.This extends the reported bootstrapping gains beyond LLM-based CRS.
  • 4.2 RQ2: Active Selection Improves Data Efficiency: JS and Fisher selection outperform random and popularity-based selection at comparable budgets, reducing teacher-LLM calls while improving performance.Their relative advantages vary by dataset and budget, reflecting different notions of sample informativeness.
  • 4.3 RQ3: Domain Signals Improve Active Selection: Metadata-aware selection improves Recall@1 and Recall@5 over semantic-only and random baselines, especially at larger budgets.The cumulative advantage suggests structural signals sustain informative selection over longer horizons.
  • 4.3 RQ3: Domain Signals Improve Active Selection: Collaborative-filtering signals add gains orthogonal to semantic features by enriching selection with user-preference structure.Collaborative-aware methods show noticeable accuracy gains at larger budgets.
  • 4.4 RQ4: Synthetic and Real Data: On INSPIRED with approximately 1k dialogues, synthetic-only training outperforms training on the original INSPIRED set.Combining synthetic and real data further improves INSPIRED results, whereas synthetic augmentation is less consistently helpful on ReDial.

5 Related Work

CRS research addresses interactive recommendation and data scarcity through model integration, language models, prompting, and synthetic augmentation. Existing augmentation generally assumes in-domain conversations, while this work studies zero-data bootstrapping and active selection from non-conversational signals.

  • CRS supports interactive recommendation, often through mixed initiative between users and systems.
  • Recent CRS methods integrate recommendation and conversation modules, incorporate language models, or prompt and fine-tune LLMs for recommendation.
  • Existing augmentation methods simulate conversations, generate narrative queries, or enrich dialogues, but operate on top of in-domain conversational data.
  • LLM-REDIAL verbalizes user histories into a multidomain synthetic corpus, whereas this study examines signal and selection choices under a fixed teacher budget.
  • Prior active-learning work selects informative sessions, student-error examples, or queries guided by explanations and teacher corrections.

6 Conclusion

The paper presents a systematic study of zero-data CRS bootstrapping with synthetic conversational supervision derived from non-conversational domain signals. Across models, benchmarks, and fine-tuning paradigms, domain grounding and active selection improve the usefulness and efficiency of this approach.

  • The study examines zero-data CRS bootstrapping: generating conversational supervision from non-conversational domain signals via LLMs.
  • Domain-grounded synthetic data outperforms zero-shot and naïve synthetic baselines across models, benchmarks, and fine-tuning paradigms.
  • Active selection improves data efficiency, while metadata and collaborative signals strengthen selection.
  • Synthetic data is especially effective when real dialogues are scarce.
  • Non-conversational signals, when systematically selected and transformed, offer a practical path to building CRS without conversational training data.

Limitations

The evaluation is limited to the movie domain, and broader transfer is not established. Applying the pipeline elsewhere would require domain-specific resources and benchmarks, while synthetic-data quality remains sensitive to hallucinations, stylistic artifacts, and prompt design.

  • The experiments evaluate exclusively on the movie domain.
  • Transfer to music or ecommerce would require domain-specific seed corpora, generation templates, and evaluation benchmarks.
  • Cross-domain generalization is left to future work.
  • LLM-generated synthetic data may contain hallucinations or stylistic artifacts that propagate to downstream models.
  • Generated-dialogue quality depends on prompt design, an optimization direction not explored here.

D.1 Human Evaluation

Human evaluation finds generally strong quality across naturalness, coherence, relevance, and diversity, with relevance the weakest dimension. Format and catalog checks show that catalog overlap is not a reliable proxy for downstream supervision usefulness.

  • Human Evaluation: Two graduate students rated 100 synthetic query-recommendation pairs on naturalness, coherence, relevance, and diversity using a 1–5 scale.Agreement within one point was 83–90% across dimensions.
  • Human Evaluation: 84% of the 400 example-dimension scores were at least 3.5, and all four dimensions averaged close to 4.
  • Human Evaluation: Relevance was the lowest-scoring dimension, consistent with occasional loosely related items in 20-item lists.
  • Title and Format Validation: Both generated corpora almost always followed the requested output format.
  • Title and Format Validation: The naïve corpus had higher catalog matching but weaker downstream performance, indicating catalog overlap rewards popular titles rather than useful supervision.

E.2 Experiments

The experiments examine active selection for conversational recommendation and illustrate prompt-based generation of synthetic queries and recommendation responses. Figure 7 compares JS and Fisher selection with random and popularity-based sampling across budgets on ReDial and INSPIRED.

  • Active selection strategies, JS and Fisher, consistently outperform random sampling and popularity-based selection across budgets on ReDial and INSPIRED.The comparison uses Llama3-3B performance in Figure 7.
  • The movie-recommendation prompt instructs a system to produce 20 recommendations from a provided conversation without extra sentences.
  • The review-to-query prompt converts sentiments, issues, or features from multiple product reviews into one distinct query matching example-query style.
  • Synthetic movie-query examples span requests for whimsical ensemble adventures, dialogue-driven films, and stories about friendships formed during journeys.The examples pair conversational requests with recommendation lists and draw on Reddit movie-recommendation templates.
Loading 2504.15476v2…