Source-linked AI summary
Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on Consistency with Human Preferences
Yunjie Ji, Yan Gong, Yiping Peng, Chao Ni, Peiyan Sun, Dongyu Pan, Baochang Ma, Xiangang Li
TL;DR
Because prior research focused mainly on evaluating individual samples, ChatGPT's ability to rank multiple responses remained underexplored. The authors test it on diverse prompts using responses from five models, finding some consistency with human preferences. They discuss using ChatGPT-generated preferences to support reward-model training and model optimization.
Problem
Although ChatGPT had been used to evaluate individual samples, its ability to rank multiple samples had not been well explored.
Source: §1 · p. 1Method
The study tests ChatGPT ranking responses from five models across diverse prompts and compares its rankings with human preference annotations.
Source: §1 · p. 1Results
Takeaways & Limitations
The authors discuss using ChatGPT to generate initial preference annotations for bootstrapping reward models and to identify use cases where existing models perform poorly.
Source: §5 · pp. 4–5An SFT model trained to fit ChatGPT's preferences may never surpass ChatGPT.
Source: §5 · p. 4
Abstract
from arXiv · showhide
As a natural language assistant, ChatGPT is capable of performing various tasks, including but not limited to article generation, code completion, and data analysis. Furthermore, ChatGPT has consistently demonstrated a remarkable level of accuracy and reliability in terms of content evaluation, exhibiting the capability of mimicking human preferences. To further explore ChatGPT's potential in this regard, a study is conducted to assess its ability to rank content. In order to do so, a test set consisting of prompts is created, covering a wide range of use cases, and five models are utilized to generate corresponding responses. ChatGPT is then instructed to rank the responses generated by these models. The results on the test set show that ChatGPT's ranking preferences are consistent with human to a certain extent. This preliminary experimental finding implies that ChatGPT's zero-shot ranking capability could be used to reduce annotation pressure in a number of ranking tasks.
1 Introduction
ChatGPT had been shown to evaluate individual content samples, but its ability to rank multiple samples remained less explored. This study tests ranking across varied prompts and reports some consistency with human preferences.
1 Introduction
- Prior studies found ChatGPT could effectively evaluate content, while its ability to rank multiple samples remained underexplored.The cited work used carefully designed prompts to evaluate content from different perspectives.Source: §1 · p. 1
- The study tests ChatGPT's ranking on prompts spanning typical NLP tasks, brainstorming, article generation, and other open-ended tasks.Five models generate responses for each prompt, which ChatGPT ranks against human preference annotations.Source: §1 · p. 1
- ChatGPT's ranking preferences showed some consistency with human preferences, suggesting a possible source of initial preference annotations for ranking tasks.The authors give RLHF reward-model bootstrapping as an example application.Source: §1 · p. 1
2 Related Work
Prior work examined ChatGPT as an evaluator and found strong alignment with human judgments in translation and NLG evaluation. This work instead studies ranking generated content from different models.
2 Related Work
- ChatGPT evaluation studies reported strong results for translation and NLG, including high consistency with human-annotated outcomes.Kocmi and Federmann (2023) reported state-of-the-art accuracy in WMT22's Metrics shared task; Wang et al. (2023) reported the highest consistency with human-annotated NLG results.Source: §2 · p. 2
- Earlier research also used ChatGPT for data generation and annotation, including synthetic data and genre classification (Dai et al., 2023; Kuzman et al., 2023; Tang et al.).Source: §2 · p. 2
- Unlike the related evaluation work, this study focuses on ranking generated content from different models.Source: §2 · p. 2
3 Method
The method uses two prompt formats to elicit rankings: one scores responses individually, while the other ranks five responses together. Both ask ChatGPT to judge response quality against an instruction.
3 Method
- The one-by-one prompt asks ChatGPT to score a single model response from 1 to 5 against its instruction.A score of 1 is specified for responses that do not meet the instruction's requirements.Source: §3 · p. 2
- The formats differ in whether ChatGPT assesses one response at a time or ranks all five responses together.Source: §3 · p. 2
- The one-for-all prompt supplies an instruction and five potential responses, asking ChatGPT to return their ranked indices.The responses are ranked together in one request.Source: §3 · p. 2
4 Experiments
The experiments compare ChatGPT’s rankings with human preferences across varied prompts and model responses. They report substantial human consistency, with one-by-one prompting outperforming one-for-all and model comparisons varying across use cases.
4.1 Datasets
- The test prompts cover NLP tasks and open-ended instructions such as brainstorming and open QA, selected to evaluate ChatGPT as a comprehensive content-preference annotator.Table 1 is identified as the test-prompt statistics table, but its values are not supplied here.Source: §4.1 · pp. 2–3, Table 1 · p. 3
4.2 Models
- The study focuses on Chinese text and uses Bloomz-series baselines because their pretraining includes more Chinese and they received further Chinese-corpus multitask fine-tuning.The response set also includes Text-davinci-003, ChatGPT, and the authors’ 7-billion-parameter InstrLLM.Source: §4.2 · p. 3
4.3 Human preference labeling
- Human preference annotations come from 129 colleagues, with three annotators independently ranking each prompt’s responses and their rankings averaged; 30 experienced developers label code prompts.Ninety percent of the colleagues had a bachelor’s degree or higher.Source: §4.3 · p. 3
4.4 Metric
- Spearman correlation measures agreement between human and ChatGPT preferences, while top-1 consistency measures whether they select the same first-ranked model..Source: §4.4 · pp. 3–4
4.6 Model Comparison
- Across six use cases, ChatGPT scores highest among the five models in both preference views, while InstrLLM exceeds bloomz-176b-mt overall, especially in brainstorming, generation, and math.Human preferences rank the authors’ model worst on NLP tasks; ChatGPT’s evaluation is described as insufficiently attentive to details such as factualness and ethics.Source: Figure 3 · p. 4, Figure 2 · p. 4, §4.6 · p. 4
4.5 Main results
- One-by-one prompting is more consistent with human preferences than one-for-all, especially for brainstorming, generation, and open QA; it also wins on top-1 consistency across prompt types.The authors attribute the larger gap on some open-ended tasks mainly to response truncation during one-for-all prompting; they note room for improvement on other use cases.Source: §4.5 · p. 4
5 Discussion and Conclusion
ChatGPT’s rankings on diversified prompts show some consistency with human preferences, motivating its possible use in model training and optimization. The discussion identifies preference bootstrapping, targeted diagnosis, and a potential ceiling from continual fitting to ChatGPT’s preferences.
5 Discussion and Conclusion
- ChatGPT could provide initial preference annotations to bootstrap reward models, particularly in RLHF.The authors frame this as a possibility for helping train models.Source: §5 · p. 4
- An SFT model continually fit to ChatGPT’s preferences may never surpass ChatGPT.Source: §5 · p. 4
- ChatGPT’s zero-shot annotations could identify use cases where existing models perform poorly and guide targeted optimization.Source: §5 · p. 5
- ChatGPT shows some consistency with human preferences when ranking model responses on diversified test prompts.The authors discuss using this capability to train and optimize models.Source: §5 · p. 5