LAB: Large-Scale Alignment for ChatBots

• LLM training typically proceeds from self-supervised pretraining to supervised alignment, with instruction tuning followed by preference tuning.Instruction tuning maximizes the likelihood of task-specific ground-truth responses; preference tuning uses human feedback to rank responses.Source: §1 · p. 1
• Instruction tuning and preference tuning use a small fraction of pretraining's data and compute, while conventional machine learning would train directly on desired tasks using as much data as possible.The authors note that this imbalance relies on pretraining capturing enough language and knowledge for limited supervised training to shape instruction-following behavior.Source: §1 · pp. 1–2
• LAB combines taxonomy-guided synthetic data generation and quality assurance with multi-phase tuning, aiming to add instruction-following abilities without catastrophic forgetting.The authors report competitive performance against models using human annotations and/or GPT-4-generated synthetic data.Source: §1 · p. 2
• High-quality human-generated task-specific instruction data is costly to procure and often closely guarded, even in ostensibly open model-building efforts.LAB is proposed to address alignment-tuning scalability without substantial human curation or proprietary LLMs such as GPT-4.Source: §1 · p. 2

• Human-data approaches such as OpenAI's alignment work and LLaMA 2 (Touvron et al., 2023) require substantial annotation effort, while templating can reduce cost but yield less natural or shorter responses (Wei et al.).The passage describes screening annotators for inter-annotator agreement and coordinating large groups to collect data.Source: §2 · p. 2
• Synthetic-data methods include Self-Instruct, teacher-generated student data, iterative complexity growth, and progressive training for reasoning and response style (Mitra et al., 2023).Source: §2 · p. 2
• GLAN uses a human-curated taxonomy but depends on GPT-4 and cannot generate data for domains outside its teacher's support, unlike LAB's open-source Mixtral teacher.The passage notes potential commercial-use concerns because proprietary-model terms may forbid using generated data to improve other models.Source: §2 · p. 2

3.1 TAXONOMY

• The taxonomy organizes tasks hierarchically into knowledge, foundational skills, and compositional skills, with leaf-node examples and room to add tasks.Knowledge is organized by document type and domain; foundational skills use public datasets, while compositional skills combine knowledge and foundational abilities.Source: §3.1 · pp. 3–4

3.2 TAXONOMY-DRIVEN SYNTHETIC DATA GENERATOR

• Taxonomy-driven sampling targets teacher-model coverage around individual taxonomy leaves, addressing random sampling that can favor dominant modes and miss long-tail tasks.Focused leaf-node prompts produce distinct input modes and encourage generation for each task.Source: §3.2 · pp. 4–5, Figure 2 · p. 4

3.2.1 SKILL GENERATION

• Skills-SDG uses Mixtral-7x8B to expand leaf-node examples into skill data, while knowledge-SDG grounds generation in documents rather than the teacher’s internal knowledge.For skills, the teacher generates and evaluates instructions, generates responses, then filters instruction-response pairs; knowledge-SDG also checks responses for faithfulness to source documents.Source: §3.2 · p. 5, §3.2.1 · pp. 5–6, §3.2.2 · p. 6

3.2.2 KNOWLEDGE-GENERATION

• Knowledge-SDG supplies documents, manuals, or books alongside curator examples so generated instruction data can cover domains absent from the teacher’s training.The teacher evaluates whether generated responses are grounded in and faithful to the source documents.Source: §3.2.2 · p. 6

3.3 MULTI-PHASE TRAINING

• Training proceeds from knowledge and foundational skills to compositional skills, using a replay buffer from the first phase during the second.Knowledge tuning trains on short-response samples before long-response samples; the best knowledge-phase checkpoint then starts skills tuning.Source: §3.3 · pp. 6–7
• The authors report that short-then-long knowledge tuning improves performance and that training knowledge and foundational skills before compositional skills yields better benchmark performance.MMLU selects checkpoints during knowledge tuning, while MT-bench is used during skills tuning.Source: §3.3 · p. 7
• Training uses small learning rates, an extended warm-up, and an effective batch size of 3840; the authors report cosine decay can destabilize later phases.The specified learning rates are 2 × 10−5 for Llama-based models and 1×10−6 for Mistral-based models.Source: §3.3 · p. 7

4 RESULTS

• The LAB models were trained on 1.2 million synthetic samples: 617k knowledge-based and 588k skill-based.LLAMA-2-13B and MISTRAL-7B were the base models, with MIXTRAL-8X7B-INSTRUCT-V0.1 as teacher.Source: §4 · p. 7
• LABRADORITE-13B and MERLINITE-7B outperform the strongest fine-tuned models on their respective base models on MT-Bench, with superior performance reported on the remaining metrics.The authors report using open-weights MIXTRAL-8X7B-INSTRUCT-V0.1 as the teacher, at orders of magnitude less cost than GPT-4 teachers.Source: Table 3 · p. 8, MISTRAL-7B · pp. 8–9, §4 · p. 8

LLAMA-2-13B

• The Llama-2 comparisons include LLAMA-2-13B-CHAT and WIZARDLM-13B-V1.2; the Mistral comparisons include MISTRAL-7B-INSTRUCT-V0.2 and ZEPHYR-7B-BETA.WIZARDLM-13B-V1.2 and ZEPHYR-7B-BETA are identified as the highest-MT-Bench models among those using their respective base models.Source: LLAMA-2-13B · p. 8, MISTRAL-7B · p. 8

MISTRAL-7B

• Evaluation covered MT-Bench, MMLU, ARC, HellaSwag, Winogrande, and GSM8k, using settings consistent with the LMSYS Chatbot Arena Leaderboard.The specified settings were 1-turn and 2-turn average for MT-Bench, 5-shot for MMLU, 25-shot for ARC, 10-shot for HellaSwag, 5-shot for Winogrande, and 5-shot strict for GSM8k.Source: MISTRAL-7B · p. 8