Source-linked AI summary
CliqueParcel: An Approach For Batching LLM Prompts That Jointly Optimizes Efficiency And Faithfulness
Jiayi Liu, Tinghan Yang, Jennifer Neville
TL;DR
Post-LLM efficiency methods can sacrifice precision or answer completeness, creating the “discounted output” problem. CliqueParcel batches prompts from similar domains and is evaluated across eight datasets, where it outperforms standard prompting and other baselines while maintaining faithfulness and accuracy.
Problem
Post-LLM optimization often gains efficiency through precision loss or answers less complete than the originals, creating a “discounted output” problem.
Source: §1 · p. 1Method
CliqueParcel classifies prompts by clique domain, batches prompts from the same domain for LLM processing, and dispatches disassembled answers to users.
Source: §1 · p. 1Results
Across eight question-answering datasets, CliqueParcel consistently outperforms standard prompting and other baselines while enhancing efficiency and maintaining answer faithfulness and accuracy.
Source: §1 · p. 1Takeaways & Limitations
The paper frames inference efficiency and faithfulness as joint objectives for evaluating prompt-batching methods.
Source: §5 · p. 8The problem simplification assumes each prompt can be classified into a clique domain.
Source: §3.2 · p. 2
Abstract
from arXiv · showhide
Large language models (LLMs) have become pivotal in recent research. However, during the inference process, LLMs still require substantial resources. In this paper, we propose CliqueParcel, a method designed to improve the efficiency of LLMs via prompt batching. Existing strategies to optimize inference efficiency often compromise on output quality, leading to a discounted output problem. This issue might result in reduced accuracy or outputs that are less detailed. CliqueParcel is our answer to this challenge. While ensuring accuracy and minimizing deviations from the original outputs (i.e., faithfulness), our method significantly improves efficiency during inference. To lay the groundwork, we first redefine efficiency measurements by excluding the reduction in running time due to shorter lengths. Then, we provide a comprehensive trade-off between efficiency and faithfulness to clarify the nature of the 'discounted output' problem. Within the CliqueParcel framework, we suggest multiple batching sub-methods and discuss the specific scenarios in which they can be applied. During evaluation, CliqueParcel is tested on eight widely recognized datasets, which can be classified into three types: reading comprehension, open-source question-answering, and reasoning. Our experiments explore the performance of CliqueParcel, including efficiency, faithfulness, and the trade-off between them. This work provides novel insights into inference efficiency and demonstrates promising performance.
1 INTRODUCTION
The introduction frames post-LLM efficiency as a “discounted output” problem when gains entail precision loss or incomplete answers. It proposes CliqueParcel and reports improved efficiency while maintaining faithfulness across eight datasets.
1 INTRODUCTION
- Post-LLM optimizations often trade efficiency for precision or answer completeness, a problem the paper calls “discounted output.”The paper frames its challenge as improving efficiency while maintaining faithfulness.Source: §1 · p. 1
- CliqueParcel classifies prompts by clique domain, batches prompts from the same domain for LLM processing, then disassembles outputs and dispatches answers to users.The paper reports improved efficiency versus traditional prompt flow and other baselines while maintaining faithfulness.Source: §1 · p. 1
- Across eight question-answering datasets, CliqueParcel consistently outperforms standard prompting and other baselines while enhancing efficiency and maintaining answer faithfulness and accuracy.The evaluation covers SQuAD, HotpotQA, TREC, CSQA, GSM8K, MATH, ANLI, and MMLU.Source: §1 · p. 1
- The paper introduces an efficiency metric designed to mitigate extraneous-factor effects and treats efficiency and faithfulness as joint optimization objectives.It also proposes multiple batching methods and feasible application scenarios.Source: §1 · pp. 1–2
2 RELATED WORK
The related work reviews faithfulness definitions and overlap-based measures, then surveys post-training efficiency methods. It distinguishes model compression from prompt-based techniques and notes their shared output-completeness concern.
2.1 Faithfulness in Large Language Model
- Faithfulness is the extent to which generated responses reflect source information or adhere to a reference; BLEU and ROUGE measure output-reference overlap.The review describes these metrics as adapted from machine translation and summarization.Source: §2.1 · p. 2
2.2 Efficiency of post-training large language models
- Post-training efficiency work includes model compression methods and prompt-based techniques, which the review says often improve efficiency through incomplete outputs.Compression examples include quantization, distillation, and pruning; prompt methods include shortening or batching prompts.Source: §2.2 · p. 2
3 APPROACH
CliqueParcel batches prompts to improve efficiency while preserving answer faithfulness, which the paper defines in relation to original answers and accuracy. Its approach measures efficiency while accounting for token lengths, then groups prompts by clique domain before generating and dispatching answers.
3.2 Problem definition
- The problem is framed around batching prompts from multiple users, with challenges in choosing batching criteria, designing the batch function, and dispatching answers.The simplification assumes each prompt can be assigned a clique domain, then grouped by that domain.Source: §3.2 · p. 2
3.2.1 Faithfulness.
- Faithfulness combines semantic similarity to original answers, BLEU/ROUGE-based completeness, and ground-truth accuracy, capturing both resemblance and factual correctness.The paper treats completeness as protection against “discounted” answers.Source: §3.2.1 · p. 3
3.2.2 Efficiency.
- The weighted efficiency ratio excludes apparent gains from incomplete outputs; runtime depends on input and output lengths, output length has greater impact, and larger batches improve relative efficiency.The paper reports that output length affects running time more substantially than input length.Source: §3.2.2 · pp. 3–5
3.2.3 Trade-off between faithfulness and efficiency.
- A multi-objective satisfaction model [24] balances faithfulness and efficiency, which the paper notes can conflict.The method assigns weights to the two objectives before applying the model.Source: §3.2.3 · p. 5
3.3 CliqueParcel framework
- The framework batches prompts assigned to the same clique domain, sends the combined prompt to the LLM, and dispatches the corresponding answers to users.A numbered-itemization template and dispatch function are used to associate generated answers with their original prompts.Source: §3.3 · p. 5
- Clique functions include concept-based, random, semantic-similarity, and concept-plus-semantic-similarity grouping.The concept-plus-similarity variant first groups by concept, then uses semantic similarity to select smaller groups.Source: §3.3 · p. 5
- Other variants use average-length, maximum-difference, or random-plus-average-length grouping, with separate prompts serving as a control.The paper notes that different clique functions perform differently across data types.Source: §3.3 · p. 5
4 EXPERIMENTS
The experiments evaluate CliqueParcel’s efficiency and faithfulness trade-offs across eight datasets, comparing clique functions and prompting baselines. The results highlight efficiency gains, while showing that faithfulness and accuracy vary across methods and tasks.
4.1.1 Benchmarks.
- Across eight datasets, the evaluation covers reading comprehension, open-source question-answering, and reasoning tasks.The benchmarks include single- and multi-document reading comprehension, TREC classification, and commonsense, math, and logic reasoning.Source: §4.1.1 · pp. 5, 7
4.1.2 Baselines.
- The experiments compare CliqueParcel with separate few-shot prompting and the SSC and MDC batching baselines, using GPT 3.5 turbo and GPT 4 across ten runs.The study varies batch size, then selects each dataset’s maximum batch size when comparing clique functions.Source: §4.1.2 · p. 7
4.2 Task and efficiency performance
- Against second-place RpALC, RC improves efficiency by 10% for GPT 4 and 4.8% for GPT-3.5 on average; CC, RpALC, and CpSC score lowest.The paper relates the weaker results for those concepts and combinations to there being no discernible pattern for the LLM to exploit.Source: §4.2 · pp. 7–8
- Compared with the original method, CliqueParcel improves average efficiency by 207.25% on GPT 3.5 and 70.63% on GPT 4.RC also consistently outperforms other clique functions when output token length is considered.Source: Table 3 · p. 6, §4.2 · pp. 7–8
4.3 Efficiency and faithfulness trade-off
- In the OWA efficiency-faithfulness trade-off analysis, RC remains top-performing even though it is not the top performer in faithfulness alone.The analysis applies varying weights to efficiency and faithfulness.Source: Figure 3 · p. 6, §4.3 · p. 8
- Accuracy is no worse on nearly all clique functions for HotpotQA, SQuAD, CSQA, ANLI, GSM8K, and MMLU, but TREC drops 4.5% and weaker model-task pairs lose accuracy.The paper reports losses when the model itself performs worse on MATH with GPT 4 and GPT 3.5, and on GSM8K and MMLU with GPT 4.Source: §4.3 · p. 8, Table 5 · p. 8
5 CONCLUSION
The conclusion frames CliqueParcel as an approach to the discounted-output problem, optimizing efficiency and faithfulness together. It reports a 138.93% improvement over the default LLM framework and describes practical options for different tasks.
5 CONCLUSION
- CliqueParcel achieves a 138.93% improvement compared with the default framework in LLMs.The conclusion does not specify the metric for this improvement.Source: §5 · p. 8
- The paper introduces efficiency and faithfulness measurements as trade-off objectives and proposes CliqueParcel to improve efficiency while maintaining output faithfulness.It also proposes multiple clique functions for different scenarios and a practical parameter-selection strategy for various tasks.Source: §5 · p. 8
- The conclusion states that CliqueParcel can be integrated into pre-trained LLMs and evaluates it on eight datasets.Source: §5 · p. 8
A.1 The Dispersion Of Length
The appendix describes length distributions across the eight datasets, including which task types have longer prompts and which datasets show the greatest or least length dispersion.
A.1 The Dispersion Of Length
- Reading-comprehension datasets SQuAD and HotpotQA have longer input prompts, and context-dependent ANLI also has relatively long prompts.The passage attributes the reading-comprehension prompt lengths to the need for context.Source: §A.1 · p. 10
- Length dispersion is greatest in SQuAD and HotpotQA, followed by TREC, MMLU, and GSM8K; CSQA, MATH, and ANLI show the least variation.Figure 4 presents Relative Standard Deviation of lengths and illustrates the distribution of Z-scores.Source: §A.1 · p. 10, Figure 4 · p. 10
A.2 Qualitative analysis for Table 2
The qualitative analysis of Equation 6 supports the conclusion that output-token length has a much larger coefficient than input-token length. Figure 5a provides an auxiliary view of the Table 2 conclusion.
A.2 Qualitative analysis for Table 2
- The output-token length coefficient w2 is much greater than the input-token length coefficient w1.Figure 5a qualitatively analyzes Equation 6 and provides an auxiliary view of the Table 2 conclusion.Source: §A.2 · p. 10
A.3 Full results for efficiency and faithfulness trade-off
The section applies an OWA satisfaction model to examine the trade-off between efficiency and faithfulness. It reports that RC performs better under different objective weights.
A.3 Full results for efficiency and faithfulness trade-off
- RC performs better when different objective weights are applied.Figure 6 presents the trade-off between completeness and effectiveness with different weights.Source: §A.3 · p. 10
- The section uses an OWA satisfaction model to examine the efficiency-faithfulness trade-off.The model is cited as [24].Source: §A.3 · p. 10
- Different weights are assigned to the objectives before applying the model to compare performance.Source: §A.3 · p. 10
A.4 Batching size over running time
Larger batching sizes reduce running time, but efficiency gains also reflect shorter output lengths. The preferred batching size depends on input length and task characteristics.
A.4 Batching size over running time
- Increasing batching size significantly reduces running time.Figure 7 examines running time across batching sizes for the basic clique sequence.Source: §A.4 · p. 10, Figure 7 · p. 11
- Some efficiency-ratio improvement comes from shorter output token lengths, not batching size alone.Figure 8 examines gain/ratio across batching sizes.Source: §A.4 · p. 10, Figure 8 · p. 11
- Larger batching sizes increase output ratios, but ratios stabilize for short-answer datasets such as TREC and MATH.Reading-comprehension datasets SQuAD and HotpotQA exhibit lower output ratios.Source: §A.4 · p. 10
- Increasing batching size is advantageous for longer-input tasks, while shorter-input tasks require a more nuanced choice.For shorter inputs, the choice involves user preferences and trade-offs.Source: §A.4 · p. 10