ChatHome: Development and Evaluation of a Domain-Specific Language Model for Home Renovation

• ChatHome targets home renovation, a relatively unexplored domain requiring knowledge spanning aesthetics, functionality, architecture, spatial design, and human-centered considerations.Source: §1 · p. 1
• Mainstream models such as ChatGPT can fail to generate high-fidelity, accurate domain-specific content, motivating a specialized home-renovation model.Source: §1 · p. 1
• ChatHome combines post-pretraining on professional articles, standards, and web content with instruction tuning on home-renovation question-answer pairs.Source: §1 · p. 1
• The study evaluates whether domain adaptation improves specialized performance while tracking changes in general capabilities.Source: §1 · p. 1
• The paper introduces ChatHome and a domain dataset, then uses experiments on universal and domain datasets to assess the model.Source: §1 · p. 2

• Domain-specific tasks contain complex concepts, technical terminology, and entity relationships that can induce severe hallucination without targeted guidance.Source: §2 · p. 2
• Domain adaptation has been studied in medical, financial, and legal settings, using either parameter updates or retrieval-based knowledge bases.Source: §2 · p. 2
• Existing specialization strategies include training from scratch on domain data, direct fine-tuning on domain instructions, and domain pretraining followed by instruction fine-tuning.Source: §2 · p. 2

3.1 Pre-training Corpus

• The pretraining corpus combines home-renovation sources with general text to add domain knowledge while preserving general knowledge.Source: §3.1 · p. 2
• Domain sources include national construction standards, recent books, and approximately 30,000 articles from home-renovation websites.Source: §3.1 · p. 2
• Text extraction, quality filtering, and article- and sentence-level deduplication yield approximately 26.6M domain tokens and 276.6M general tokens.Source: §3.1 · p. 2

3.2 SFT Corpus

• The SFT corpus contains approximately 25k instruction examples drawn from home-decoration books and website articles to support domain adaptation.Source: §3.2 · pp. 2–3
• GPT-4 generates one-turn and multi-turn designer-client dialogues, with relevant articles used to orient multi-turn content and mitigate hallucinations.Source: §3.2 · p. 3
• The instruction data are summarized through a word cloud, while preprocessing is documented through a workflow figure.Source: §3.2 · p. 3, Figure 2 · p. 3, Figure 1 · p. 3

4.1 Baseline Models

• The Baichuan-13B-Base and Baichuan-13B-Chat baselines provide pretrained and instruction-tuned starting points for home-renovation adaptation.Baichuan-13B-Base has 13 billion parameters and 1.4 trillion training tokens, while Baichuan-13B-Chat is instruction fine-tuned.Source: §4.1 · p. 3

4.2 Experiments Setups

• The setup compares identical instruction-tuning experiments with and without DAPT while testing domain-to-general data ratios of 1:0, 1:1, 1:2, 1:5, and 1:10.The experiments address catastrophic forgetting by balancing domain-specific and general data; DAPT and SFT differ in maximum sequence length.Source: §4.2 · pp. 3–4, §4.3 · p. 4

4.3 Metrics

• Evaluation combines C-Eval and CMMLU for general capabilities with EvalHome, a 113-question multiple-choice benchmark spanning fundamentals, expertise, and innovative design.The reported C-Eval results use the development set and the strongest overall average from zero- or few-shot settings.Source: §4.3 · p. 4, Table 2 · p. 4

4.4 Results and Analysis

• A 1:5 ratio produced the best performance on the current base model and home-renovation data, with the strongest general-capability retention after DAPT.At 1:5, average CEval and CMMLU scores decreased from the base model by 2.57 and 3.08 points, respectively.Source: §4.4 · pp. 4–5, Table 3 · p. 5
• Adding more general instruction data lowered general-capability scores, possibly because C-Eval and CMMLU measure specific knowledge not covered by that instruction data.This interpretation is tied to the current benchmarks and general instruction dataset.Source: §4.4 · p. 5
• Instruction tuning after DAPT reached 59.29 and 55.75 on EvalHome, while direct tuning of Baichuan-13B-Chat reached 60.17; post-DAPT tuning did not significantly surpass direct adaptation.The authors speculate that the instruction-aligned base model already contained substantial decoration-related data.Source: §4.4 · pp. 5–6, Table 4 · p. 5
• MIP combining domain pre-training and domain instruction data achieved 69.03 on EvalHome and outperformed the other models on both EvalHome and the two general benchmarks.This experiment used no general corpus because of training resource and time constraints.Source: §4.4 · p. 6

• ChatHome was evaluated for domain-specific capabilities and general performance using various base models and data ratios.Source: §5 · p. 6
• The model still exhibits hallucination defects, while assessing its professional capabilities remains an ongoing improvement area.Source: §5 · p. 6
• Future work will explore knowledge bases and reinforcement learning to enhance ChatHome’s performance.Source: §5 · p. 6

• The appendix includes prompts for data generation and detailed evaluation scores on C-Eval and CMMLU.Source: A Appendix · p. 7, Table 5 · p. 9, Table 6 · p. 11
• Figure 3 presents an example of EvalHome with English translations below the corresponding Chinese text.Source: Figure 3 · p. 7
• Figure 4 presents an example of one-turn dialogue with English translations below the corresponding Chinese text.Source: Figure 4 · p. 8
• Figure 5 presents an example of multi-turn dialogue.Source: Figure 5 · p. 10