Aligning Large Language Models with Human: A Survey

• LLMs may misunderstand instructions, generate biased content, or invent facts despite fluent text generation.Source: §1 · p. 1
• Alignment research addresses costly data collection, resource-intensive supervised fine-tuning, unstable reinforcement learning, and incomplete benchmark coverage.Source: §1 · pp. 1–2
• The survey reviews high-quality data collection using NLP benchmarks, human annotators, and strong LLMs.Source: §1 · pp. 2–3
• Training research targets efficiency and stability through parameter-efficient methods, ranking-based preference signals, and language-based feedback.Source: §1 · p. 2
• Human-centric benchmarks and automatic protocols, including LLMs for evaluation, support more comprehensive assessment of aligned LLMs.Source: §1 · pp. 2–3
• The survey distills current alignment technologies and identifies future research avenues for navigating alignment with human values and expectations.Source: §1 · p. 3

• Instructions are represented as I_k = (x_k, y_k), where x_k is the input and y_k is the corresponding response.Source: §2 · p. 3
• Instruction data can come from human-generated examples or strong LLMs and can be combined into diverse training collections.Source: §2 · p. 3

2.1.1 NLP Benchmarks

• NLP benchmarks convert diverse dialogue, reasoning, and coding tasks into natural-language instructions for multitask learning and unseen-task generalization.Source: §2.1.1 · p. 3
• Figure 2 illustrates how an NLI benchmark example can be reformulated as an instruction.Source: Figure 2 · p. 3, §2.1.1 · p. 3
• Unnatural Instructions use LLMs to generate varied templates or instances at scale, while FLAN-style instructions improve aligned models’ reasoning capabilities.Source: §2.1.1 · p. 3

2.1.2 Hand-crafted Instructions

• Benchmark-derived instructions are efficient to construct but often remain narrow because many source datasets target specific skills rather than complex real-world interaction.Source: §2.1.2 · p. 3
• Manual annotation frameworks broaden instruction coverage through datasets such as Dolly-15k, OpenAssistant, translated Chinese instructions, and ShareGPT conversations.Source: §2.1.2 · pp. 3–4
• OpenAssistant combines prompts, dialogue replies, and ranked dialogue quality, enabling both supervised fine-tuning and human-preference alignment.Source: §2.1.2 · p. 4
• ShareGPT and user-generated resources such as Stack Overflow, Quora, Zhihu, and Wikipedia provide human-written prompts that can seed synthetic multi-turn dialogues.Source: §2.1.2 · p. 4

2.2 Instructions From Strong LLMs

• Synthetic instruction generation is scalable but must address the challenge of producing diverse and high-quality instructions through effective prompting.Source: §2.2 · p. 4

2.2.1 Self-Instruction

• Self-instruction uses in-context learning to generate instructions from an existing pool, filters them, obtains responses, and iterates until the target volume is reached.Source: Figure 3 · p. 4, §2.2.1 · p. 4
• Synthetic inputs can lack diversity; strategies use task-specific generation, external information, metadata, and Evol-Instruct rewriting to increase coverage, complexity, and factuality.Source: §2.2.1 · pp. 4–5
• High-quality outputs are encouraged through reasoning prompts, hand-crafted principles, and role-playing conditions that elicit reasoning, ethical behavior, or expert-style responses.Source: §2.2.1 · p. 5
• Difficulty monitoring retains instructions when a student model’s response falls short of a teacher model’s response under LLM-based pairwise evaluation.Source: §2.2.1 · pp. 5–6

2.2.2 Multi-turn Instructions

• Multi-turn instruction data is generated through self-chatting, role-playing, revision-based dialogue, and real-world knowledge sources to support conversational alignment.Source: §2.2.2 · p. 6

2.2.3 Multilingual Instructions

• Multilingual construction trades off cultural preservation against output quality through post-answering or post-translating, with mixed-language dialogue offering another adaptation strategy.Source: §2.2.3 · p. 6

Instruction Implications

• Adding more instructions benefits standard NLP tasks but has negligible influence on Math, Code, CoT, and Brainstorming.The reported contrast indicates that instruction quantity does not uniformly improve complex reasoning capabilities.Source: §2.3 · p. 6
• Approximately 50% programming instructions can preserve general conversational performance while enhancing LLM reasoning.This finding supports targeted augmentation of instruction mixtures rather than indiscriminate expansion.Source: §2.3 · p. 6
• Composite instruction sets improve overall performance because different instruction types affect different LLM capabilities.CoT and coding instructions are especially important for reasoning, while combining instruction types supports broader performance.Source: §2.3 · p. 7

Instruction Quantity

• LLaMA requires approximately 8K instructions for a high IFS score, while 6K high-quality instructions can suffice for human-preference alignment.More instructions may induce a semantic shift in foundational models, motivating quality-focused selection.Source: §2.3 · p. 7
• The LLM trained on the top 9K instructions notably outperforms LLMs trained on the complete set of 52K Alpaca instructions.Source: §2.3 · p. 7

3.1 Online Human Preference Training

• RLHF combines instruction SFT, a reward model trained on ranked responses, and PPO policy optimization with KL regularization.The three-stage pipeline collects instructions, learns response quality from comparisons, and optimizes the policy using reward-model scores.Source: §3.1 · p. 7
• PPO-based preference training is difficult to implement and stabilize, motivating alternatives such as RAFT that select top-ranked generated samples for SFT.RAFT samples responses, ranks them with an existing reward model, and applies only the highest-ranked examples; the procedure can also update offline datasets.Source: §3.1 · pp. 7–8

3.2.1 Ranking-based Approach

• DPO directly optimizes ranked response pairs while implicitly targeting the RLHF objective with KL regularization, and PRO extends ranking-based preference training to ordered response sets.These methods avoid the online PPO procedure by incorporating preference rankings directly into fine-tuning objectives.Source: §3.2.1 · p. 8
• Ranking strategies and regularization choices vary with model scale: rank loss with KL performs best in one study, whereas RRHF finds SFT more effective than KL for preventing over-fitting.The cited comparison used BERTScore-simulated preferences and models no larger than 2B, while RRHF further optimized LLaMA-7B.Source: §3.2.1 · p. 8

3.2.2 Language-based Approach

• Language-based methods encode response quality through prefixes or revision instructions, allowing models to learn from high- and low-quality outputs through SFT.CoH concatenates instructions, outputs, and feedback, applies loss only to model outputs, and uses position feedback during inference; revision-dialogue training supports self-correction.Source: Figure 4 · p. 8, §3.2.2 · p. 9

3.3 Parameter-Effective Training

• Parameter-efficient fine-tuning freezes most LLM parameters and trains only a limited set of additional or low-rank parameters.Prefix and prompt tuning add trainable tokens, while LoRA updates low-rank matrices; QLoRA quantizes the backbone to 4-bit precision and reduces memory usage.Source: §3.3 · pp. 9–10

Trade-offs For Parameter-efficient Training

• LoRA can underfit relative to full fine-tuning, while using a larger LLM may outperform using a larger instruction dataset at lower training cost.This trade-off is reported under the same set of training instructions.Source: Trade-offs For Parameter-efficient… · p. 10

4.1 Evaluation Benchmarks

• Closed-set benchmarks test predefined skills and knowledge, whereas open-set benchmarks assess flexible responses without standardized answers.Closed-set tasks include knowledge, reasoning, and coding; open-set tasks use chatting questions or topics with flexible outputs.Source: §4.1 · p. 10, §4.1.1 · pp. 10–11, §4.1.2 · p. 11

4.1.1 Closed-set Benchmarks

• MMLU covers 57 subjects from elementary to advanced professional levels, while GSM8K, Maths, CSQA, StrategyQA, and coding benchmarks target specialized reasoning and programming abilities.HumanEval, HumanEval+, MBPP, and DS-1000 automatically verify generated code against test cases.Source: §4.1.1 · pp. 10–11

4.1.2 Open-ended Benchmarks

• Open-ended evaluation evolved from small shared-instruction comparisons toward Win Rate, multi-turn testing, and fine-grained skill assessment.AlpacaEval reports Win Rate against text-davinci-003, MT-Bench uses 80 multi-turn instances, and FLASK evaluates 1,700 instances across 12 alignment skills.Source: §4.1.2 · p. 11

4.2.1 Human-based Evaluation

• Human evaluation uses ordinal labels, pairwise comparisons, and Elo ratings, but ordinal judgments can suffer from subjectivity and poor inter-rater reliability.Elo updates relative skill scores from pairwise comparison outcomes.Source: §4.2.1 · p. 11

4.2.2 LLMs-based Evaluation

• LLM evaluators can score or compare outputs without references and support task-specific, factuality, and multidimensional evaluation, but their judgments contain systematic biases.Reported biases include positional, length, factuality-related, and self-enhancement preferences; response swapping, few-shot examples, chain-of-thought, references, and multidimensional Elo are proposed mitigations.Source: §4.2.2 · pp. 11–12
• Evaluation-specific models can reduce reliance on closed-source evaluators, with PandaLM matching GPT-3.5 and GPT-4 on human-annotated meta-evaluation despite being smaller.PandaLM fine-tunes LLaMA-7B on around 300K synthetic evaluation instructions generated from GPT-3.5.Source: §4.2.2 · p. 12

5 Challenges and Future Directions

• Instruction datasets vary widely, leaving quality control, training order, and mixture design unresolved and limiting fair comparison across alignment methods.FLAN and programming instructions improve reasoning, while ShareGPT performs consistently across broad benchmarks.Source: §5 · pp. 12–13

LLMs Alignment for non-English Languages

• Most alignment research is English-dominated, so performance and transfer to other languages, especially low-resource languages, remain unclear.The survey calls for multilingual evaluation and methods that transfer alignment effects across languages.Source: LLMs Alignment for non-English Languages · p. 13
• Because SFT does not explicitly incorporate human preference, the survey calls for resource-constrained comparisons of alignment methods under fixed data and training budgets.Example constraints include a maximum of 10K instructions and five hours of training, supporting more effective and environmentally friendly solutions.Source: LLMs Alignment for non-English Languages · p. 13
• ShareGPT illustrates human-in-the-loop alignment because people freely determine what models should generate, and it performs consistently across a wide range of NLP tasks.The survey proposes exploring additional human-in-the-loop data-generation solutions.Source: LLMs Alignment for non-English Languages · p. 13

Human-LLM Joint Evaluation Framework

• A joint human–LLM evaluation framework could assign tasks according to each evaluator’s strengths to preserve both efficiency and evaluation quality.The survey notes that state-of-the-art LLMs can show similar or superior evaluation capability on various NLP tasks.Source: Human-LLM Joint Evaluation Framework · p. 14

6 Conclusion

• The survey provides an up-to-date review of recent LLM alignment technologies and organizes them into instruction collection, training, and evaluation.It also points out promising future directions for the field.Source: §6 · p. 14
• The authors intend the survey to provide perspectives and inspire further research on improving LLM alignment.Source: §6 · p. 14

• Table 2 compares example outputs from original LLaMA and a Chinese tokenizer.Source: Table 2 · p. 19
• The example is attributed to Cui et al. (2023b).Source: Table 2 · p. 19
• The table belongs to the appendix section.Source: Table 2 · p. 19

A.1 Training Language-Specific LLMs

• Chinese tokenization produces more compact, semantically meaningful representations than default LLaMA tokenization.This is especially useful for long and complex Chinese phrases.Source: §A.1 · p. 19
• English-dominated LLaMA has fewer than 1,000 Chinese characters in its vocabulary, so Chinese characters often use byte-based fallback representations.This significantly increases input length and decreases inference efficiency.Source: §A.1 · p. 19
• Cui et al. (2023b) add 20K Chinese words and phrases to LLaMA’s vocabulary before two-stage Chinese pretraining.The first stage trains only input word embeddings, while the second adds LoRA and trains embeddings, self-attentive heads, and LoRA parameters.Source: §A.1 · p. 19
• The second stage reduces training resources by using LoRA while jointly adapting selected parameters.Source: §A.1 · p. 19