PaperlayerView exact revision on arXiv

Source-linked AI summary

LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions

Chaochen Gao, Xing Wu, Zijia Lin, Debing Zhang, Songlin Hu

arXiv:2505.17134v2cs.CLcs.AI

TL;DR

Long-context instruction datasets remain proprietary, while human annotation is costly and template-based synthesis can limit diversity. LongMagpie elicits document-related queries from aligned models and generates instruction responses, then evaluates the resulting data on long-context benchmarks. The authors report leading long-context performance and competitive short-context performance when combined with p-Mix.

  • Problem

    Long-context instruction data remains proprietary, while human annotation is costly and template- or seed-based synthesis does not guarantee the diversity needed for effective long-context instruction.

    Source: §1 · pp. 1–2

  • Method

    LongMagpie uses aligned models’ learned document-query behavior to generate document-related queries and responses, with a multi-document extension and p-Mix for mixing long- and short-context instructions.

    Source: §1 · pp. 2–3

  • Results

    LongMagpie-trained models achieve leading performance on long-context tasks, and p-Mix maintains competitive short-context performance.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    LongMagpie is presented as an effective approach for making long-context capabilities more accessible to the broader research community.

    Source: §6 · p. 10

    LongMagpie inherits biases from its source instruction-tuned LLMs and inadequately covers tasks requiring long-form outputs, extended reasoning, or generation.

    Source: §7 · p. 10

Abstract

from arXiv · show

High-quality long-context instruction data is essential for aligning long-context large language models (LLMs). Despite the public release of models like Qwen and Llama, their long-context instruction data remains proprietary. Human annotation is costly and challenging, while template-based synthesis methods limit scale, diversity, and quality. We introduce LongMagpie, a self-synthesis framework that automatically generates large-scale long-context instruction data. Our key insight is that aligned long-context LLMs, when presented with a document followed by special tokens preceding a user turn, auto-regressively generate contextually relevant queries. By harvesting these document-query pairs and the model's responses, LongMagpie produces high-quality instructions without human effort. Experiments on HELMET, RULER, and Longbench v2 demonstrate that LongMagpie achieves leading performance on long-context tasks while maintaining competitive performance on short-context tasks, establishing it as a simple and effective approach for open, diverse, and scalable long-context instruction data synthesis.

1 Introduction

Long-context instruction data remains proprietary despite public model weights, while human annotation is costly and template- or seed-based synthesis can limit diversity. LongMagpie uses aligned models’ document-query behavior to generate instruction data, extends it to multi-document contexts, and pairs it with p-Mix; evaluations report leading long-context performance and competitive short-context performance with p-Mix.

1 Introduction

  • Long-context instruction datasets remain proprietary despite public releases of models such as Qwen and Llama, limiting open-source long-context model development.Human annotation is costly, and template- or seed-based synthesis does not guarantee diverse instructions; large-scale quality and diversity remain expensive and time-consuming.Source: §1 · pp. 1–2
  • LongMagpie exploits aligned LLMs’ tendency to generate document-relevant queries after a document and user-turn tokens, then generates responses without human annotation or complex prompting.The method draws on learned document-query patterns rather than explicit query-generation instructions.Source: §1 · p. 2
  • The multi-document extension combines query-response pairs with source and sampled documents to create more challenging contexts involving information across sources.The authors state that this increases context length and task difficulty without additional computational overhead.Source: §1 · p. 2, Figure 1 · p. 2
  • p-Mix probabilistically combines long- and short-context instructions to address short-task performance degradation from predominantly long-context training.It begins each training sequence with a short-context instruction and dynamically samples later segments.Source: §1 · pp. 2–3
  • Models trained on LongMagpie data achieve leading performance on HELMET, RULER, and Longbench v2, while p-Mix maintains competitive short-context performance.The authors describe this as a step toward making high-quality long-context instruction data accessible to the broader research community.Source: §1 · p. 3

2 Method

LongMagpie synthesizes instruction data by eliciting document-related queries from aligned long-context models, generating responses, and extending examples with sampled documents. p-Mix probabilistically mixes long- and short-context instructions in training sequences to balance capabilities.

2.1 Key Insight: Auto-Regressive Document-Query Generation

  • Aligned long-context LLMs can generate contextually relevant queries when given a document followed by tokens that typically precede a user query.The method uses learned document-query patterns rather than explicitly instructing the model to ask about the document.Source: §2.1 · p. 3

2.2 LongMagpie Pipeline

  • The method is organized as query-and-answer generation followed by extension to multi-document contexts.The overview frames these as the pipeline’s two main steps.Source: §2 · p. 3, §2.2 · p. 4

2.2.1 Query and Answer Generation

  • LongMagpie prepares diverse documents, elicits queries using pre-query tokens, and generates responses to form document-query-response training triplets.Multiple queries per document can be sampled with different parameters; generated queries are filtered by question-mark ending and a 1.5k-character limit.Source: §2.2.1 · p. 4

2.2.2 Multi-Document Extension

  • For multi-document examples, LongMagpie randomly samples documents, concatenates them with separators, and generates queries and responses requiring cross-document reasoning.Sampling zero added documents reduces to the single-document QA setting.Source: §2.2.2 · p. 4

2.3 p-Mix: Balancing Long-Context and Short-Context Capabilities

  • p-Mix starts each sequence with a short-context instruction, then probabilistically appends long- or short-context segments until approaching the target length.The long-context segment is selected with probability PL; otherwise another short-context sample is chosen.Source: §2.3 · p. 4

3 Experiments

The experiments compare LongMagpie-generated instructions with long- and short-instruction datasets under a shared training setup and evaluate long- and short-context performance. LongMagpie leads on long-context benchmarks, while p-Mix results remain competitive on short-context tasks.

3.1 Experimental Setup

  • LongMagpie data is generated with Qwen2.5-70B-Instruct from FineWeb-Edu documents, and comparisons include long-context and short-context instruction datasets.The comparison includes ChatQA and LongAlign as long-instruction datasets, plus Tulu, Magpie, and UltraChat as short-instruction datasets.Source: §3.1 · p. 5

3.1.1 Model Training

  • Training uses Llama-3-8B-NExtLong-512K-Base for 250 steps at 4M tokens per batch, totaling 1B tokens for each dataset.The same training configuration is applied across datasets for comparison.Source: §3.1.1 · p. 5

3.1.2 Evaluation Benchmarks

  • Evaluation covers HELMET, RULER, and LongBench-v2 for long contexts, plus seven widely used short-context datasets.The long-context benchmarks assess application-centered tasks, reasoning, and extremely long-context understanding.Source: §3.1.2 · p. 5

3.2 Main Results

  • 62.56 LongAVG: LongMagpie leads the Long Instruction Data group, beating ChatQA by +2.28 and LongAlign by +6.44 on LongAVG.Table 1 compares long- and short-context benchmarks; LongMagpie also tops HELMET (62.10), RULER (91.17), and LongBench-v2 (34.4).Source: Table 1 · p. 6, §3.2 · p. 6
  • With p-Mix, LongMagpie + UltraChat leads or ties mixed datasets on HELMET (62.11), RULER (89.70), and LongAVG (61.60), while ShortAVG is 64.10.The ShortAVG result is 0.33 below the overall best.Source: §3.2 · p. 6

4 Ablation Studies

The ablations examine multi-document context, mixing strategy, data volume, source-model size, query quality and diversity, and sample efficiency. Across these settings, p-Mix balances long- and short-context performance, while the analyses identify conditions associated with stronger long-context results and more efficient synthesis.

4.1 Impact of Different Multi-Document Settings

  • The multi-document strategy significantly improves long-context performance from 60.19 to 62.56; performance rises and then degrades as n increases, with the best result at n = 10.Source: §4.1 · p. 7

4.2 Impact of Different Mixing Strategy

  • LongAVG is 61.60 with p-Mix, with ShortAVG best at 64.10; alternative mixing strategies balance long- and short-context performance less effectively.LongAVG is only slightly below no mixing, while the short-context score is the best among the strategies compared.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 Impact of Different Data Size

  • +1.05 LongAVG points result when LongMagpie training data increases from 190k to 450k samples, alongside gains on HELMET, RULER, and Longbench v2.The reported benchmark gains are +0.81 on HELMET, +0.52 on RULER, and +1.8 on Longbench v2.Source: §4.3 · p. 7, Table 4 · p. 8

4.4 Impact of Different Source Model Size

  • LongAVG improves from 59.61 to 62.56 with Qwen-2.5-70B rather than Qwen-2.5-7B as the source model, with similar ShortAVG performance.The authors suggest the larger model’s long-context modeling ability may explain the improvement.Source: §4.4 · pp. 7–8, Table 5 · p. 8

4.5 Analysis of of LongMagpie Queries

  • LongMagpie has higher reward-model-scored data quality than ChatQA and LongAlign, and its queries show lower pairwise similarity than other datasets.The query-similarity distributions were aggregated over 30 repeated samples.Source: §4.5 · pp. 8–9, Figure 2 · p. 8
  • LongMagpie’s query embeddings are more dispersed than those of other datasets, reflecting greater query diversity.The comparison uses t-SNE visualizations of query embeddings.Source: Figure 3 · p. 8, §4.5 · p. 9

4.6 Sample Efficiency of LongMagpie

  • LongMagpie achieves superior long-context performance while averaging 1.6K tokens per instruction, versus 10–13× more tokens for ChatQA and LongAlign.The comparison is between long-context performance and tokens processed per instruction during synthesis.Source: Figure 3 · p. 8, §4.6 · p. 9

5 Related Work

Prior long-context synthesis includes continuation-oriented pretraining methods and instruction-oriented data pipelines. LongMagpie is positioned as an instruction-synthesis approach that extends Magpie’s seed-free generation paradigm to long-context document-query data.

5.1 Long-Context Data Synthesis

  • Continuation-oriented methods extend context through concatenated documents, using techniques such as clustering, retrieval, keyword matching, and hard-negative interleaving, but focus on pretraining rather than instruction tuning.LongMagpie instead generates instruction-following data using aligned models’ autoregressive capabilities.Source: §5.1 · p. 9
  • Existing instruction-oriented methods use templates, curated examples, QA blending, packing, or instruction selection, often relying on complex pipelines.LongMagpie is contrasted as eliminating seed questions and complex pipelines by eliciting document-relevant queries from aligned LLMs.Source: §5.1 · p. 9

5.2 Synthesis Methods for Short-Context Instruction Data

  • LongMagpie extends Magpie’s seed-free autoregressive instruction synthesis to long contexts through document-query relationship patterns, without specialized prompting.Magpie demonstrated diverse instruction generation without human annotation or seed examples.Source: §5.2 · pp. 9–10

6 Conclusion

LongMagpie automatically synthesizes large-scale long-context instruction data without human annotation or seed examples. Evaluations report leading long-context performance and competitive short-context capabilities when the data is combined with p-Mix.

6 Conclusion

  • LongMagpie automatically generates large-scale long-context instruction data without human annotation or seed examples.Source: §6 · p. 10
  • Models trained on LongMagpie data achieve leading performance on HELMET, RULER, and Longbench v2 long-context tasks.Source: §6 · p. 10
  • Combined with p-Mix, LongMagpie maintains competitive short-context capabilities, supporting its use as an approach to democratizing long-context capabilities.Source: §6 · p. 10

7 Limitations

LongMagpie inherits biases from its source instruction-tuned LLMs, and its current implementation inadequately covers tasks requiring long-form outputs.

7 Limitations

  • LongMagpie inherits biases from source instruction-tuned LLMs, which the authors say future work should detect and mitigate.Source: §7 · p. 10
  • The current implementation focuses on document-query relationships and inadequately covers tasks requiring long-form outputs, extended reasoning, or generation.Source: §7 · p. 10

A.1 Training Config

The training setup uses AdamW, long concatenated sequences with document masking, and memory- and speed-oriented tools; Table 6 is identified as the source of detailed configuration.

A.1 Training Config

  • AdamW uses β1 = 0.9 and β2 = 0.95.The setup follows ProLong for concatenating samples up to 64K tokens and applying document masking between independent sequences.Source: §A.1 · p. 15
  • Samples are concatenated up to 64K sequence length, with document masking preventing interactions between independent sequences.Source: §A.1 · p. 15
  • FlashAttention-2 and ZeRO are used to optimize memory usage and accelerate training.Table 6 is identified as the detailed training-configuration table.Source: §A.1 · p. 15, Table 6 · p. 15

A.2 Impact of Multi-Document Setting

Figure 4 reports that long-context task performance first improves and then declines as the number of documents increases.

A.2 Impact of Multi-Document Setting

A.3 Ablation Study on p-Mix Strategy Parameters

The p-Mix ablation varies the number of prepended short-context samples and the probability of selecting long-context samples; its construction starts with a short sample and probabilistically mixes in further samples within a length limit.

A.3 Ablation Study on p-Mix Strategy Parameters

  • The p-Mix ablation varies the number of prepended short-context samples (NS) and long-context selection probability (PL).The experiments use n = 10 for the multi-document context length parameter; Table 7 is identified as reporting detailed ablation results.Source: §A.3 · p. 15, Table 7 · p. 16
  • The ablation evaluates how p-Mix parameter configurations affect overall performance on long- and short-task benchmarks.Source: §A.3 · p. 15, Table 7 · p. 16
  • Hybrid sample construction begins with a randomly selected short-context sample, then probabilistically appends short- or long-context samples until the length limit is reached.The procedure uses PL for long-context selection and 1 − PL for short-context selection, stopping when the next sample would exceed Lmax.Source: §A.3 · p. 16
Loading 2505.17134v2…