ExpertPrompting: Instructing Large Language Models to be Distinguished Experts

• Prompt crafting can change whether the same intent receives a comprehensive response or a less helpful one, making output quality and user satisfaction prompt-dependent.Source: §1 · p. 1
• For each instruction, ExpertPrompting envisions a fitting distinguished expert identity and conditions the LLM’s answer on that identity.Source: §1 · p. 1
• It generates detailed, instruction-specific identities through In-Context Learning with several instruction–expert exemplars, without sophisticated templates or iterative processes.The authors describe the method as generalizable across domains and genres.Source: §1 · p. 2
• ExpertLLaMA outperforms Alpaca, Vicuna, and LLaMA-GPT4, and approximately reaches 96% of original ChatGPT’s capability.Alpaca uses the same instructions but different answers, while LLaMA-GPT4 uses GPT-4 as its language model.Source: §1 · p. 2
• GPT-based evaluation found improved answer quality in expert data generated by applying ExpertPrompting to GPT-3.5 on 52k Alpaca instructions.Source: §1 · p. 2

2 Method

• ExpertPrompting pairs each instruction with a tailored expert identity and prompts the LLM to produce an augmented answer.The augmented answer is expected to improve on the answer produced without the identity.Source: §2 · p. 2, §2.2 · p. 2

2.1 Expert Identity

• Expert identities should be instruction-specific, detailed enough to guide the LLM, and generated automatically; the authors use in-context examples to synthesize them (Brown et al., 2020; Xu et al., 2023).They manually revise three identity descriptions for sampled instructions as exemplars, prepend them to the prompt, and report that GPT-3.5 can generate satisfying descriptions.Source: §2.1 · p. 2

2.2 Expert Prompting

• The augmented-answer prompt combines the original instruction with its expert identity, and postprocessing removes recognizable identity-related introductions that sometimes appear in answers.The authors illustrate the augmentation with an atomic-structure instruction and a physicist identity.Source: §2.2 · p. 2

2.3 ExpertLLaMA

• Using the Alpaca instruction set, the authors generate standard and expert answers with GPT-3.5-turbo, then train ExpertLLaMA on the expert answers using LLaMA 7B (Touvron et al., 2023).They release the assistant and expert answers for future research.Source: §2.3 · p. 3

3.1 Experimental Setting

• Evaluation used GPT-4 comparisons, shuffled answer order, sampled 500 of 52k training examples, and tested models on GPT-4-synthesized Vicuna80 across varied question categories.Model evaluation compared ExpertLLaMA trained on expert answers with LLaMA-GPT-3.5 trained on vanilla answers.Source: §3.1 · p. 3

3.2 Baselines

• Baselines included Alpaca, LLaMA-GPT4, GPT-3.5 vanilla and Static DESC models, Vicuna, ChatGPT, and Bard; Static DESC adds a fixed expert-role description.The GPT-3.5 baselines used the same 52k instructions, with Static DESC adding a fixed description to the prompt.Source: §3.2 · pp. 3–4

3.3 Data Eval

• GPT-4 preferred ExpertPrompting answers 48.5% versus vanilla answers 23%, and expert answers were significantly longer despite no request for longer answers.The length comparison used GPT-3.5-Turbo answers for the 52k Alpaca instructions.Source: Figure 3 · p. 4, §3.3 · pp. 4–5, Table 1 · p. 4

3.4 Model Eval

• Approximately 96% of ChatGPT capability was achieved by ExpertLLaMA, which outperformed the compared open-source assistants and was inferior only to ChatGPT.The authors note that the 96% conclusion needs more rigorous validation.Source: Figure 4 · p. 4, Figure 5 · p. 4, §3.4 · p. 5

4 Conclusion

• ExpertPrompting uses GPT-3.5 to produce instruction-following data for training ExpertLLaMA, an open-source chat assistant.The prompting strategy is automatic, generalized, and simple to implement.Source: §4 · p. 5
• 96% of ChatGPT’s capability was achieved by ExpertLLaMA, which outperformed existing open-source chat assistants in GPT-4-based evaluation.The evaluation also found that ExpertPrompting produces higher-quality answers.Source: §4 · p. 5
• The authors plan to enlarge the instruction-data scale beyond 52k Alpaca to further improve ExpertLLaMA.Source: §4 · p. 5

• Figure 6 shows the In-Context Learning template used to produce expert identities.Source: Figure 6 · p. 7
• Figure 7 presents the ExpertPrompting template.Source: Figure 7 · p. 8
• Figure 8 presents the GPT-4-based automatic evaluation template, adapted from Chiang et al. (2023).Source: Figure 8 · p. 8