Source-linked AI summary
A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, Jie Zhou, Siming Chen, Tao Gui, Qi Zhang, Xuanjing Huang
TL;DR
The paper examines the underexplored evolution of GPT-3 and GPT-3.5 capabilities across NLU tasks. It evaluates six models on 21 datasets under zero-shot and few-shot settings, finding that capability does not improve universally and that RLHF can compromise some task performance while robustness remains limited.
Problem
Comprehensive evidence is limited on how GPT series capabilities evolve and how their training strategies affect NLU performance.
Method
The study evaluates six GPT-3 and GPT-3.5 models on nine NLU tasks using 21 datasets, comparing performance and robustness across zero-shot and few-shot settings.
Results
GPT capability does not improve consistently across NLU tasks; RLHF enhances human-like responses but can compromise task performance, while robustness shows no significant enhancement.
Takeaways & Limitations
Training strategy and task characteristics should be considered when balancing task-solving ability with user-friendly responses and robustness.
Takeaways & Limitations
Some models were tested without the full dataset because of OpenAI API access-rate limits, and GPT-4 could not be evaluated because its API was unavailable.
Abstract
from arXiv · showhide
GPT series models, such as GPT-3, CodeX, InstructGPT, ChatGPT, and so on, have gained considerable attention due to their exceptional natural language processing capabilities. However, despite the abundance of research on the difference in capabilities between GPT series models and fine-tuned models, there has been limited attention given to the evolution of GPT series models' capabilities over time. To conduct a comprehensive analysis of the capabilities of GPT series models, we select six representative models, comprising two GPT-3 series models (i.e., davinci and text-davinci-001) and four GPT-3.5 series models (i.e., code-davinci-002, text-davinci-002, text-davinci-003, and gpt-3.5-turbo). We evaluate their performance on nine natural language understanding (NLU) tasks using 21 datasets. In particular, we compare the performance and robustness of different models for each task under zero-shot and few-shot scenarios. Our extensive experiments reveal that the overall ability of GPT series models on NLU tasks does not increase gradually as the models evolve, especially with the introduction of the RLHF training strategy. While this strategy enhances the models' ability to generate human-like responses, it also compromises their ability to solve some tasks. Furthermore, our findings indicate that there is still room for improvement in areas such as model robustness.
1 Introduction
The paper addresses the limited comprehensive evidence on how GPT-3 and GPT-3.5 capabilities evolve across NLU tasks and training strategies. Its findings show uneven task performance, prompt sensitivity, and limited robustness gains rather than consistent improvement.
- The study evaluates six GPT-3 and GPT-3.5 models across nine NLU tasks using 21 datasets.It compares model performance, training-strategy effects, and zero-shot versus few-shot scenarios.
- All models are prompt-sensitive in both zero-shot and few-shot settings, with sensitivity varying by model and task.The paper identifies this variation as requiring further investigation.
- Different models lead on different zero-shot tasks: code-davinci-002 on ABSA, MRC, and SC; text-davinci-003 on POS, RE, and SM; and gpt-3.5-turbo on NLI and wSC.Gpt-3.5-turbo also has difficulty following instructions in POS, similarly to text-davinci-001.
- Few-shot prompting generally improves performance, but gains depend on the model, task, prompt design, and example selection.Few-shot scenarios do not always outperform zero-shot scenarios.
- Text-davinci-001 has the weakest overall capability on most tasks, while gpt-3.5-turbo and text-davinci-003 perform similarly on most tasks.Gpt-3.5-turbo is disadvantaged in MRC, POS, and RE, whereas text-davinci-001 remains moderate in MRC and SC.
- Alignment and RLHF can trade off task performance against human-like response generation rather than producing universal NLU improvements.Text-davinci-002 may underperform code-davinci-002 on some tasks, while text-davinci-003 is comparable to or worse than text-davinci-002 on several tasks.
2 Background
GPT-3 and GPT-3.5 evolved through different model families and training strategies, while TextFlint provides a robustness-evaluation platform used extensively in this study.
- GPT-3 and GPT-3.5 Series Models: GPT-3 and GPT-3.5 comprise OpenAI language-model series whose evolution includes supervised fine-tuning, code training, and RLHF-based optimization.Davinci underlies GPT-3; code-davinci-002 became the GPT-3.5 base, followed by text-davinci-002, text-davinci-003, and gpt-3.5-turbo.
- TextFlint: TextFlint is a multilingual platform for evaluating NLP-task robustness through general and task-specific transformations, adversarial attacks, subgroups, and combinations.It offers over 80 transformation methods for 12 NLP tasks, including 20 general and 60 domain-specific transformations.
- TextFlint: Sixteen of the 21 datasets used in this paper were collated by TextFlint.
3 Experiment Setup
The experiments compare six GPT-3 and GPT-3.5 models on nine NLU tasks using 21 datasets, transformation data, and zero-shot or few-shot prompts. Evaluation uses model-specific sample counts and selected task prompts, including label mappings for several structured prediction tasks.
- GPT Systems: The six models represent GPT-3 and GPT-3.5 evolution from davinci through instruction-tuned, code-trained, RLHF-trained, and chat-optimized variants.
- Evaluation Protocol: Evaluation sample sizes differ by model: davinci and code-davinci-002 use 100 samples, text-davinci-001 and text-davinci-002 use 1000, and the remaining models use entire datasets.Datasets with fewer than 1000 records are evaluated in full.
- Prompt Selection Strategies: The study compares zero-shot and few-shot scenarios using task-specific prompts gathered from existing sources and manually designed prompts.Three best-performing prompts per dataset are selected, and labels are mapped to phrases for relation extraction, named entity recognition, and part-of-speech tasks.
4 Experiments
Davinci can achieve good analyzability and performance on many datasets without examples, but its results depend strongly on prompt format and few-shot demonstrations. In particular, removing a terminal “Answer” cue severely reduces analyzability on most datasets.
- Performance Comparison on Different Datasets: Figure 3 compares analyzability rates across datasets in zero-shot and three-shot settings, ordering datasets by the three-shot-to-zero-shot ratio.The detailed results are listed in Appendix A.2.
- Performance Comparison on Different Datasets: Figure 4 compares davinci’s performance across datasets in zero-shot and three-shot settings using the same ratio-based ordering.The detailed results are listed in Appendix A.2.
- Performance Comparison on Different Datasets: Davinci achieves good analyzability and performance on many datasets, including MNLI-m, MNLI-mm, IMDB, and WSC273, in zero-shot evaluation.These results occur without supervised fine-tuning.
- Performance Comparison on Different Datasets: For datasets where zero-shot performance is not possible, davinci can learn from provided examples in the three-shot scenario.The cited examples include ACE 2005, WSJ, and PKU-SEGPOS.
- Instruction Comprehension of davinci: Removing “Answer” from zero-shot prompts severely reduces analyzability on most datasets, indicating weak instruction comprehension.The authors argue that including instructions in training is necessary for handling NLU problems.
4.2 Comparison Experiments
Model performance varies substantially across NLU tasks, datasets, prompting conditions, and robustness transformations. Later models can improve task performance, but these gains are uneven, and robustness often fails to improve correspondingly.
- Aspect-based Sentiment Analysis: Zero-shot ABSA performance is nearly identical across models, while code-davinci-002 is the most consistent and other variations are generally poor.The authors attribute the similar primary-task performance presumably to ABSA being simpler.
- Machine Reading Comprehension: Code-davinci-002 achieves the best zero-shot MRC performance on both F1 and EM, while chat-oriented outputs can lower gpt-3.5-turbo’s automatic scores.Gpt-3.5-turbo often generates complete sentences containing correct answers, which automatic metrics may score poorly.
- Machine Reading Comprehension: Code-davinci-002 outperforms the other four models by ten to twenty points in the three-shot MRC setting.Few-shot examples also help gpt-3.5-turbo produce answer words or phrases instead of complete sentences, improving its metrics.
- Named Entity Recognition: Text-davinci-003 and gpt-3.5-turbo generally lead zero-shot NER results, but the best model varies by dataset and prompting scenario.In few-shot evaluation, code-davinci-002 leads ACE2005, while its performance decreases when examples increase on CoNLL2003.
- Natural Language Inference: Gpt-3.5-turbo performs best in many zero-shot NLI cases, but text-davinci-003 becomes the overall best performer in few-shot evaluation.Gpt-3.5-turbo can nevertheless show poor robustness on the NumWord variation across all three NLI datasets.
- Part-of-Speech Tagging: Text-davinci-003 leads zero-shot POS performance, whereas code-davinci-002 or gpt-3.5-turbo lead in few-shot settings.Text-davinci-001 and gpt-3.5-turbo sometimes fail to produce the expected output format in zero-shot evaluation.
- Robustness: Models that produce expected answers show strong robustness on WSJ, while robustness does not significantly change across GPT generations overall.The cited robust models on WSJ are code-davinci-002, text-davinci-002, and text-davinci-003.
- Robustness: SwapAnt causes up to a 61.64% drop in average performance, while few-shot prompting can significantly improve performance and robustness in some tasks.The SwapAnt result indicates difficulty with semantic contradiction expressed through antonyms; in QQP, model performance differences decrease as examples increase.
5 Conclusion
The paper evaluates six GPT-3 and GPT-3.5 models on performance and robustness across 21 datasets and nine NLU tasks. Model evolution does not yield universal task improvements, and robustness shows no significant enhancement despite improved performance.
- Conclusion: The study evaluates six GPT-3 and GPT-3.5 models on 21 datasets spanning nine NLU tasks, measuring both performance and robustness.The evaluation covers the models’ capabilities across the two model series.
- Conclusion: GPT model evolution does not necessarily produce universal improvements across NLU tasks, with outcomes influenced by training strategy and task characteristics.The conclusion specifically notes this pattern across the evaluated models and tasks.
- Conclusion: Despite improved model performance, robustness does not show significant enhancement and warrants further investigation.The paper frames balancing task-solving ability with user-friendly responses and improving robustness as future directions.
6 Limitations
The study’s analysis is constrained by incomplete testing for some models and the unavailability of the GPT-4 API during the study. These constraints limit direct evaluation of GPT-4’s role in addressing earlier issues.
- Limitations: Some models were not tested on full datasets because OpenAI API access was rate-limited, although the authors state this did not affect overall trend analysis.This constrains dataset coverage for parts of the evaluation.
- Limitations: GPT-4 could not be evaluated because its API was unavailable during the study, leaving whether it addresses prior model issues unresolved.The authors identify this comparison as a critical area for future research.
A.1 Performance of Different Models in the Zero-shot Scenario
Table 22 reports zero-shot performance across different models. Except for davinci, a dash denotes that the non-analyzable rate exceeded the threshold and the task was counted as incomplete.
- Table 22 compares different models in the zero-shot scenario.
- For models other than davinci, “-” indicates a non-analyzable rate above the threshold.
- Results marked “-” are counted as not completing the specified task.
A.2 Analyzability Rate and Performance of davinci in All Datasets
Table 23 presents davinci’s analyzability rate and performance in zero-shot and few-shot settings. Evaluation results were manually calculated, with non-analyzable outputs treated as wrong answers.
- Table 23 reports davinci’s analyzability rate and performance across zero-shot and few-shot scenarios.
- The evaluation results were manually calculated.
- Non-analyzable results were counted as wrong answers.
A.3 Analyzability Comparison of davinci
Table 24 compares davinci’s analyzability in the zero-shot setting. Removing “Answer” from the prompt decreases the model’s analyzability.
- Table 24 compares davinci’s analyzability in the zero-shot scenario.
- “w/o ‘Answer’” denotes zero-shot prompts without “Answer” appended at the end.
- Adding “Answer” at the end of the zero-shot prompt increases davinci’s analyzability relative to omitting it.
B Prompts
The appendix specifies three prompts per dataset for 0-, 1-, and 3-shot scenarios, with detailed 0/1-shot templates covering the evaluated datasets and task-specific input placeholders or separators.
- Each dataset uses three prompts for the 0-, 1-, and 3-shot scenarios; 3-shot adds examples to the 1-shot prompt.
- Tables 29–34 provide 0/1-shot prompts for ACE 2005, CoNLL 2003, OntoNotes v5, HONOR, MSRANER, and OntoNote4NER, using text or sentence fields and, where specified, structured formats or separators.
- Tables 36–41 define prompts for MNLI-m, MNLI-mm, SNLI, WSJ, Daily547, and PKU-SEGPOS using premise, hypothesis, text, candidate, or sentence placeholders.
- Tables 42–45 provide prompts for IMDB, MRPC, QQP, and WSC273 using sentence, paired-sentence, question, and target placeholders.