PaperlayerView exact revision on arXiv

Source-linked AI summary

#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models

Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, Jingren Zhou

arXiv:2308.07074v2cs.CLcs.AIcs.LG

TL;DR

SFT data is considered to benefit from diverse and complex queries, but these attributes have lacked quantitative analysis. The paper develops INSTAG to tag and measure query diversity and complexity, then selects training samples using those tags. TagLM models trained on the selected data outperform open-source models aligned with considerably more data on MT-BENCH.

  • Problem

    Query diversity and complexity are considered important to SFT, but their definitions remain unclear and they lack quantitative analysis.

    Source: §1 · p. 1

  • Method

    INSTAG assigns open-set fine-grained intention tags, uses tag coverage and average tag count to measure diversity and complexity, and selects diverse, complex queries for fine-tuning.

    Source: §1 · p. 2, §3.1 · p. 4, §3.4 · p. 6

  • Results

    TagLM-13b-v1.0 and TagLM-13b-v2.0 score 6.44 and 6.55 on MT-BENCH, respectively, surpassing models aligned with considerably more SFT data.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Tag-based analysis finds that model performance increases with more diverse and complex data, and the selected-data models outperform open-source models trained on considerably more data.

    Source: §3.4 · p. 7, §5 · p. 11

    Evaluation relies mainly on MT-BENCH, and the SFT dataset analysis is mainly English, so claims may not directly extend to multilingual scenarios.

    Source: LIMITATIONS · p. 11

Abstract

from arXiv · show

Foundation language models obtain the instruction-following ability through supervised fine-tuning (SFT). Diversity and complexity are considered critical factors of a successful SFT dataset, while their definitions remain obscure and lack quantitative analyses. In this work, we propose InsTag, an open-set fine-grained tagger, to tag samples within SFT datasets based on semantics and intentions and define instruction diversity and complexity regarding tags. We obtain 6.6K tags to describe comprehensive user queries. Then we analyze popular open-sourced SFT datasets and find that the model ability grows with more diverse and complex data. Based on this observation, we propose a data selector based on InsTag to select 6K diverse and complex samples from open-source datasets and fine-tune models on InsTag-selected data. The resulting models, TagLM, outperform open-source models based on considerably larger SFT data evaluated by MT-Bench, echoing the importance of query diversity and complexity. We open-source InsTag in https://github.com/OFA-Sys/InsTag.

1 INTRODUCTION

SFT alignment depends on training queries that cover diverse semantics and specialties, but query diversity and complexity have lacked clear quantitative analysis. The paper introduces INSTAG to measure these attributes and reports that models trained on selected data outperform open models trained on considerably more SFT data.

1 INTRODUCTION

  • SFT data is typically formatted as multi-turn human queries paired with expected chatbot responses, and is gathered through crowdsourcing or LLM distillation.The cited work argues for diversity across domains, tasks, and formats.Source: §1 · p. 1
  • Quantifying query diversity and complexity is significantly understudied, despite claims that these attributes matter for alignment SFT.The authors identify query formation as the main determinant of these data attributes.Source: §1 · pp. 1–2
  • INSTAG uses open-set fine-grained intention tags to quantify instruction diversity and complexity, analyzes open-source datasets, and selects diverse, complex training queries.The selector's selected-data models perform well on MT-BENCH.Source: §1 · p. 2
  • TagLM-13b-v1.0 and TagLM-13b-v2.0 score 6.44 and 6.55 on MT-BENCH, respectively, surpassing models aligned with considerably more SFT data.The models use LLaMA and LLaMA-2, respectively.Source: §1 · p. 2

2 RELATED WORKS

Prior alignment work has explored SFT, response ranking, and reinforcement learning, while studies of SFT data quality address response quality, task difficulty, query complexity, semantic diversity, and scale. The paper situates its contribution in the lack of quantitative measures for query diversity and complexity.

2 RELATED WORKS

  • Alignment research uses SFT, response ranking, and reinforcement learning, with SFT studies drawing data from demonstrations, proprietary-model logs, and generated or rewritten samples.Fine-tuned models on these datasets have shown instruction-following performance across real-world tasks.Source: §2 · p. 2
  • Prior work treats SFT data quality as involving response quality, task difficulty, query complexity, semantic diversity, and sample scale.Self-Instruct generates diverse queries, Evol-Instruct complexifies queries, and other work rewrites samples or constructs diverse conversations.Source: §2 · pp. 2–3
  • Existing work had not discussed how to quantify query diversity and complexity, motivating tag-based analysis of training data mixtures.The paper uses tags to quantify data attributes and study mixtures for alignment.Source: §2 · p. 3

3 INSTAG

INSTAG assigns open-set, fine-grained intention tags to queries and normalizes the resulting annotations. The authors evaluate tag quality and use the tags to compare the diversity and complexity of open-source SFT datasets.

3 INSTAG

  • INSTAG uses CHATGPT for open tagging, normalizes its annotations, evaluates tag quality, and analyzes open-source SFT datasets.The section presents fine-grained intention tags and an automatic tagging method.Source: §3 · pp. 3–4

3.1 OPEN-SET FINE-GRAINED TAGGING

  • Open-set fine-grained tagging leaves the ontology unrestricted, allowing tags to represent multifaceted user intentions that a fixed label set may not cover.CHATGPT receives few-shot examples, returns JSON tags, and briefly explains each tag; the raw outputs exceed 12,000 tags but include format and granularity noise.Source: §3.1 · p. 4

3.2 TAG NORMALIZATION

  • Normalization addresses lexical noise, excessive granularity, and spurious tag associations through frequency filtering, rule-based cleanup, semantic clustering, and association aggregation.The process yields 6,587 tags after filtering and aggregation, and mines 1,772 association rules for merging tag groups.Source: §3.2 · pp. 4–5

3.3 QUALITY EVALUATION

  • GPT-4 achieves 96.1% precision and 86.6% consistency in tag evaluation, while human majority votes indicate full correctness on both tasks.The evaluation uses 4,000 GPT-4-labeled cases and human annotations on 1% of cases; the reported agreement and counterfactual checks support the authors' assessment of tag quality.Source: §3.3 · pp. 5–6, Table 2 · p. 5

3.4 PRELIMINARY ANALYSIS

  • Diversity is measured by unique-tag coverage, while complexity is measured by the average number of tags assigned per query.These operational measures respectively capture the range of covered intentions and the number of intentions attributed to queries.Source: §3.4 · p. 6
  • Datasets with high diversity and complexity include ShareGPT, UltraChat, and OpenChat-v1, whose fine-tuned models rank strongly on AlpacaEval; math and code datasets show lower diversity but relatively high complexity.In Figure 2a, dot size represents dataset size and color represents model performance; tag recalls also show math and code datasets are distinctive, while WizardLM (Alpaca), WizardLM (ShareGPT), UltraChat, and ShareGPT cover other datasets.Source: Figure 2 · p. 6, §3.4 · p. 7

4 INSTAG FOR DATA SELECTION

The authors select and evaluate 6K-query SFT datasets using InsTag, then examine how tag-based complexity and diversity relate to MT-BENCH performance. TAGLM-13b-v1.0 outperforms open-source aligned models with far fewer samples, while the analyses find performance generally rises with greater complexity and diversity.

4.1 EXPERIMENT SETUP

  • The 6K-query pool combines four datasets selected for large tag recall and average tag counts; it contains 306,044 samples and 6,398 tags.The authors select a subset with average tag number 16.56 and 100% tag coverage using complexity-first diverse sampling.Source: §4.1 · pp. 7–8

4.2 RESULTS

  • 6.44 average MT-BENCH score: TAGLM-13b-v1.0 outperforms open-source aligned LLMs using 6K LLaMA samples; TAGLM-13b-v2.0 trails LLaMA-2-chat by 0.1.TAGLM-13b-v1.0 lags GPT-4 by 2.44 on MT-BENCH.Source: §4.2 · p. 8, Table 3 · p. 9
  • Across MT-BENCH subtasks, TAGLM-13b-v1 leads on stem and extraction, ranks second on math, coding, and writing, and falls short on roleplay and reasoning.The authors suggest some tasks may need few alignment samples, whereas reasoning and writing may benefit from larger-scale data.Source: §4.2 · pp. 8–9, Figure 3 · p. 9

4.3 DECOUPLED ANALYSIS

  • With tag coverage held at 100%, MT-BENCH performance generally rises as average tag numbers increase from 6.7 to 16.6, though fine-grained differences may be insignificant.All ten subsets outperform the random baseline, whose average tag number is around 4.5; the authors note tag counts may be inaccurate due to missed or filtered tags.Source: Figure 4 · p. 10, §4.3 · p. 10
  • With average tag number held at 5.0, increasing tag coverage generally corresponds to higher MT-BENCH scores, with a plateau around 50%–90%.Randomly sampled data at 71.9% coverage performs similarly to the subset at 70%; the authors note tags may differ in how much diversity they represent.Source: Figure 4 · p. 10, §4.3 · p. 10

5 CONCLUSION

The conclusion presents InsTag as a tool for analyzing SFT query distributions and selecting diverse, complex training data. The authors report that TAGLM models trained on 6,000 selected queries outperform open-source models trained on considerably more data.

5 CONCLUSION

  • InsTag is an open-set fine-grained tagger using modern chatbots’ instruction-following ability to annotate queries.The authors report that its tags support deeper analysis of query distributions in LLM alignment.Source: §5 · p. 11
  • A complexity-first diverse sampler selects 6,000 queries, and models fine-tuned on them outperform open-source models aligned with considerably more data.The conclusion also reports that model performance increases with greater query diversity and complexity.Source: §5 · p. 11
  • The authors identify comprehensive evaluation and tag-based self-instruct as potential extensions beyond the demonstrated data selection application.Source: §5 · p. 11

LIMITATIONS

The authors qualify their conclusions by noting that their evaluation relies mainly on MT-BENCH and their SFT-dataset analysis focuses mainly on English.

LIMITATIONS

  • The conclusions rely mainly on MT-BENCH, which may miss some effects caused by SFT data.Source: LIMITATIONS · p. 11
  • MT-BENCH scores show randomness in GPT-4 judgments; the authors provide random ablations and standard score variances to address this instability.Source: LIMITATIONS · p. 11
  • The SFT-dataset analysis focuses mainly on English, so the claims may not directly extend to multilingual scenarios.Source: LIMITATIONS · p. 11

A TAG REVIEW

The tag review highlights common tags and the prompt materials document ChatGPT annotation and evaluation procedures.

A TAG REVIEW

  • The sunburst plot highlights information-related, data-manipulation, and coding tags among those occurring most frequently.It shows the first two words of tags with frequencies above 2,000; segment size represents frequency.Source: A TAG REVIEW · p. 15

B PROMPT TEMPLATES FOR CHATGPT

  • The appendix presents ChatGPT prompts for annotation, precision evaluation, and consistency evaluation.These prompt templates are shown in Tables 4–6.Source: B PROMPT TEMPLATES FOR CHATGPT · p. 15

C DATASETS

The section catalogs 17 open-source SFT datasets used for intention tagging and introduces the LLM baselines for human alignment.

C DATASETS

  • The study applies INSTAG to 17 open-source SFT datasets, including collections for mathematical and code queries.The math collections include GSM8K and MATH; code collections include DMCC, MBPP, and DrRepair.Source: C DATASETS · pp. 15, 17
  • ShareGPT histories are reproduced because the exact dataset was not released, while OpenChat uses ShareGPT histories with GPT-4 responses.The study follows Vicuna preprocessing for the reproduced ShareGPT version and uses OpenChat v1.0 with 8,000 GPT-4 responses.Source: C DATASETS · pp. 15–16
  • The dataset descriptions span template-generated FLAN tasks, 15,000 human-generated Dolly pairs, multilingual crowdsourced OAssist conversations, and 1,000 curated Lima examples.UltraChat is described as a large-scale collection of multi-turn instructional conversations without human queries; Alpaca has 52,000 demonstrations generated from text-davinci-003, and WizardLM uses EVOL-INSTRUCT to augment query complexity.Source: C DATASETS · p. 16

D BASELINE LLMS

  • The baselines include Alpaca, WizardLM, Vicuna, OpenChat, Baize, and LLaMA-2 Chat, which differ in training data and alignment methods.LLaMA-2 Chat is based on LLaMA-2 and uses both SFT and RLHF; Baize uses 100K self-chat dialogues and also includes Alpaca data.Source: D BASELINE LLMS · p. 17

E SAMPLING ALGORITHM FOR DECOUPLED ANALYSIS

The paper presents separate sampling algorithms to analyze data complexity and diversity independently.

F DETAILED RESULTS

The section points to Table 7 for detailed MT-BENCH results, which also provide the source data for Figure 3.

G COUNTERFACTUAL EVALUATION

The evaluation probes tag precision and consistency with counterfactual cases, and reports human and GPT-4 judgments. The section also presents sampling algorithms and an illustrative INSTAG case.

G COUNTERFACTUAL EVALUATION

  • The precision counterfactuals replace tags with semantically similar alternatives, while consistency counterfactuals replace the original instructions with inconsistent ones.Source: G COUNTERFACTUAL EVALUATION · p. 17
  • The complexity-sampling algorithm sorts queries by tag count in descending order and samples subsets at different complexity levels.Source: G COUNTERFACTUAL EVALUATION · p. 18
  • Humans and GPT-4 recognize most counterfactual cases; humans perform better on tag precision, while GPT-4 performs better on consistency.The analysis reports low false positive rates and confidence in judgments on the original tagging results.Source: G COUNTERFACTUAL EVALUATION · p. 18
  • The evaluation uses GPT-4 on 4,000 tagging results and three human annotators on 1% of cases, with majority voting for human judgments.Counterfactual cases probe annotator judgment abilities.Source: Table 8 · p. 19

H CASE FROM INSTAG

I INSTAGGER: LOCAL TAGGER BY DISTILLATION

INSTAGGER distills high-performing chatbot tagging ability into a smaller model for settings where CHATGPT or GPT-4 API access is unavailable or infeasible. On the validation set, it achieves 31.8% exact-match and 73.4% semantic fuzzy-match tag-level F1.

I INSTAGGER: LOCAL TAGGER BY DISTILLATION

  • INSTAGGER distills high-performing chatbot tagging ability into a specialized smaller LLM for localized or large-scale deployments where CHATGPT or GPT-4 API access is unavailable or infeasible.The authors use INSTAG results on open-source SFT datasets to fine-tune a 7B LLaMA-2 model.Source: I INSTAGGER: LOCAL TAGGER BY… · p. 19
  • The tagging prompt requests user-intention tags and explanations in JSON format for each query.The template asks for a tag and explanation for each query and uses the resulting annotations in fine-tuning.Source: I INSTAGGER: LOCAL TAGGER BY… · p. 19
  • The fine-tuning set contains 773,511 samples, with 1,000 randomly sampled for validation; training uses batch size 512 for one epoch because longer training overfits.The authors concatenate queries and tagging results, including tag explanations, to support fine-tuning.Source: I INSTAGGER: LOCAL TAGGER BY… · pp. 19–20
  • A WizardLM example is described as capturing query semantics such as food and health and intentions such as recommendation and quantity.The example is from SFT data tagged by INSTAG.Source: Table 9 · p. 20
  • 31.8% exact-match F1 and 73.4% semantic fuzzy-match F1 were achieved on the validation set.The fuzzy-match metric counts a prediction as correct when its PHRASEBERT cosine similarity to any gold tag exceeds 0.8.Source: I INSTAGGER: LOCAL TAGGER BY… · p. 20
Loading 2308.07074v2…