PaperlayerView exact revision on arXiv

Source-linked AI summary

Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation

Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, Yu Wang

arXiv:2307.15337v3cs.CLcs.AI

TL;DR

LLM inference is slow, in part because sequential decoding generates tokens one by one. SoT has the model plan an answer skeleton and expand its points in parallel. Across 12 LLMs, it provides speed-ups of up to 2.39× and can improve answer quality in many cases.

  • Problem

    Sequential token-by-token decoding contributes to high LLM inference latency, which hinders interactive use.

    Source: §1 · p. 1

  • Method

    SoT prompts an LLM to produce an answer skeleton, then completes its points in parallel using batched decoding or parallel API calls.

    Source: §1 · p. 2

  • Results

    SoT provides speed-ups of up to 2.39× across 12 LLMs and can improve answer quality in many cases.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SoT illustrates a data-centric route to inference efficiency by letting LLMs organize their output content.

    Source: §1 · p. 2

    SoT currently ignores dependencies between answer points.

    Source: §6 · p. 9

Abstract

from arXiv · show

This work aims at decreasing the end-to-end generation latency of large language models (LLMs). One of the major causes of the high generation latency is the sequential decoding approach adopted by almost all state-of-the-art LLMs. In this work, motivated by the thinking and writing process of humans, we propose Skeleton-of-Thought (SoT), which first guides LLMs to generate the skeleton of the answer, and then conducts parallel API calls or batched decoding to complete the contents of each skeleton point in parallel. Not only does SoT provide considerable speed-ups across 12 LLMs, but it can also potentially improve the answer quality on several question categories. SoT is an initial attempt at data-centric optimization for inference efficiency, and showcases the potential of eliciting high-quality answers by explicitly planning the answer structure in language.

1 INTRODUCTION

LLM latency hinders interactive use, and sequential token generation cannot be parallelized. SoT plans answer structure first and expands points in parallel, speeding generation across 12 LLMs and sometimes improving answer quality.

1 INTRODUCTION

  • LLM inference is slow enough to hinder interactive use, and sequential decoding adds latency because tokens are generated one by one.The introduction identifies model size, attention, and sequential decoding as three causes of slow inference.Source: §1 · p. 1
  • SoT first prompts an LLM to produce an answer skeleton, then expands its points in parallel and aggregates them into the final answer.It can use batched decoding for open-source models or parallel API calls for API-based models.Source: §1 · p. 2, Figure 1 · p. 2
  • SoT is suited to long answers whose structure can be planned ahead, but not to short answers or step-by-step reasoning.SoT-R adds a router that triggers SoT only for suitable questions.Source: §1 · p. 2
  • SoT provides speed-ups of up to 2.39× across 12 LLMs and can improve answer quality in many cases.For the example question, latency fell from 22 to 12 seconds with Claude and from 43 to 16 seconds with Vicuna-33B V1.3.Source: §1 · p. 2
  • SoT takes a data-level approach to inference efficiency by having the LLM organize its output content.The paper contrasts this pathway with model- and system-level efforts.Source: §1 · p. 2

2 SKELETON-OF-THOUGHT (SOT)

SoT elicits a concise answer skeleton, expands each point in parallel, and joins those expansions into the final answer. For open-source models, batching can exploit decoding's weight-loading bottleneck; API parallelism trades latency for more requests and tokens.

2.1 METHOD

  • SoT first prompts the LLM for a concise skeleton and extracts its points, then requests an expansion for each point and concatenates the responses.The skeleton is generated from the question; each expansion request includes the question, skeleton, point index, and point skeleton.Source: §2.1 · p. 3
  • For API-only models, parallel point-expansion calls can reduce end-to-end latency at the cost of more API requests and tokens.Source: §2.1 · p. 3
  • For open-source models, batching point-expansion requests exploits decoding's weight-loading bottleneck, allowing roughly B× more tokens in the same time.The passage says increasing batch size does not much increase per-token latency because decoding is bottlenecked by weight loading rather than activation loading or computation.Source: §2.1 · pp. 3–4

3 SOT EVALUATION

Skeleton-of-Thought (SoT) is evaluated for efficiency and answer quality across models and question types. It often speeds generation, while quality varies by model, question category, and metric.

3.1.1 SPEED-UP BREAKDOWN: MODELS

  • SoT achieves more than 2× speed-up on 8 of 12 models, reaching up to 2.39×.Most models produce final SoT answers averaging 1–2× the length of normal answers.Source: §3.1.1 · p. 4, Figure 2 · p. 5
  • SoT speed varies with point count and response length; StableVicuna-13B often ignores the brevity instruction and therefore gains little acceleration.API-based models generate shorter point-expanding responses than open-source models.Source: §3.1.1 · p. 4

3.1.2 SPEED-UP BREAKDOWN: QUESTION CATEGORIES

  • For knowledge, generic, common-sense, roleplay, and counterfactual questions, SoT speeds answer generation by 1.89× to 2.33×.SoT also provides speed-ups across all question categories.Source: §3.1.2 · p. 5, Figure 2 · p. 5

3 SOT EVALUATION

  • Quality is assessed with FastChat and LLMZoo preference judgments; results cover Vicuna-80 and WizardLM and 12 models.The answer-pair evaluation is run with both presentation orders to reduce ordering bias.Source: §3 · p. 4, §3.2 · p. 5

3.2.1 OVERALL QUALITY

  • Across models and questions, SoT is not worse than normal generation in around 60% of cases, though strict-win estimates differ between FastChat and LLMZoo.The reported strict-win rates are 45.8% and 29.5%, respectively.Source: §3.2.1 · pp. 5–6, Figure 3 · p. 6

3.2.2 QUALITY BREAKDOWN: MODELS

  • Net win rates are low for OpenChat-13B, Vicuna-7B V1.1, Claude, and LLaMA2-Chat-13B, but high for Vicuna-13B V1.3, StableVicuna-13B, and UltraLM-13B.The two general metrics agree on these rankings; some models struggle with SoT prompts, while Claude's normal answers are already strong.Source: §3.2.2 · p. 6, Figure 4 · p. 6

3.2.3 QUALITY BREAKDOWN: QUESTION CATEGORIES

  • SoT performs relatively well on generic, common-sense, knowledge, roleplay, and counterfactual questions, but relatively poorly on writing, Fermi, math, and coding.It works well when answers can be divided into independently expandable points; step-by-step questions such as math are challenging.Source: §3.2.3 · p. 6, Figure 5 · p. 6

3.2.4 QUALITY BREAKDOWN: METRICS

  • SoT improves diversity and relevance on average, while hurting immersion and coherence.For coherence and immersion, SoT is no worse than normal generation around 60% of the time.Source: §3.2.4 · p. 7, Figure 6 · p. 7

4 SOT WITH ROUTER (SOT-R): ADAPATIVELY TRIGGERING SOT

SoT with router (SoT-R) selects SoT for suitable requests and normal generation otherwise. On Vicuna-80, it improves quality on categories poorly suited to SoT while retaining gains on categories where SoT performs well.

4 SOT WITH ROUTER (SOT-R): ADAPATIVELY TRIGGERING SOT

  • The router is introduced to apply SoT only when suitable, addressing its unsuitability for questions requiring step-by-step reasoning.The evaluation compares prompting and trained routers, as well as manual human routing.Source: §4 · p. 7, §4.3 · p. 7

4.1 PROMPTING ROUTER / 4.2 TRAINED ROUTER

  • The prompting router asks GPT-4 whether the desired answer is a list of independent points; the trained alternative fine-tunes a 120M-parameter RoBERTa classifier.The prompting router avoids model training, while its performance is sensitive to prompt design.Source: §4.1 · p. 7, §4.2 · p. 7

4.3.1 EVALUATION OF EFFICIENCY

  • SoT-R retains speed-ups above 1× for most models, though its speed-ups are lower than SoT because it skips some SoT calls and adds router latency.On Vicuna-80, the trained router yields slightly higher speed-ups for 7 of 12 models; on WizardLM, the prompting router is faster for all models.Source: §4.3.1 · p. 8, Figure 7 · p. 8

4.3.2 EVALUATION OF ANSWER QUALITY

  • On Vicuna-80, SoT-R improves answer quality on categories poorly suited to SoT by falling back to normal decoding, while retaining gains on categories where SoT works well.Prompting and trained routers can also surpass the human router on some questions, such as roleplay.Source: §4.3.2 · p. 8, Figure 8 · p. 8
  • All three routers show notable consistency on Vicuna-80, while WizardLM results diverge more and the trained router aligns more closely with human annotations.Source: §4.3.2 · p. 8

5 SOT IN THE CONTEXT OF LITERATURE

SoT shifts inference efficiency from model and system redesign to organizing answer content, using the LLM’s own planning ability to parallelize generation. It differs from token-parallel methods and extends prompting toward efficiency rather than answer quality alone.

Efficient LLM methods at model and system levels

  • SoT’s latency focus may complement throughput-oriented serving techniques, creating opportunities for improved latency-throughput trade-offs.The paper describes this as a potential synergy for future serving systems.Source: §5 · p. 8
  • SoT is a data-level technique that organizes output content, unlike model-level and system-level methods targeting model architecture, computation, scheduling, or serving.The paper notes that many system techniques target throughput, while SoT trades throughput for end-to-end latency and may complement them.Source: §5 · p. 8

Efficient LLM methods through parallel generation

  • Unlike speculative decoding and non-autoregressive generation, SoT prompts the LLM itself to plan content for parallel generation across answer segments.The compared approaches use assisting or specially modified models, or parallel sampling and refinement.Source: §5 · pp. 8–9

Prompting methods for LLMs

  • SoT applies prompting to inference efficiency rather than answer quality, unlike prompting work focused on reasoning quality and multimodal planning.The paper describes this as a first attempt to exploit prompting for efficiency.Source: §5 · p. 9

6 LIMITATIONS, FUTURE WORK, AND OPEN QUESTIONS

The discussion identifies limits in quality evaluation, point dependencies, workload fit, token costs, and the predictability of data-centric efficiency gains. It also describes the evaluated model sources.

6 LIMITATIONS, FUTURE WORK, AND OPEN QUESTIONS

  • Answer-quality evaluation is limited by the prompt set, potential GPT-4 judge bias, and the difficulty of evaluating LLM generations; human evaluation was not conducted because SoT’s answer pattern could bias judgments.The authors use LLM-based evaluation and explain why they did not conduct human evaluation.Source: §6 · p. 9
  • SoT ignores dependencies between points; the authors propose static and dynamic Graph-of-Thoughts as possible ways to represent them.The proposed dynamic structure could combine SoT’s efficiency and global thinking with reasoning methods’ logical and impromptu thinking strengths.Source: §6 · p. 9
  • SoT can reduce latency and improve GPU utilization under unsaturated workloads, while at saturation computation overhead matters even though answer-quality improvement may remain useful.The paper highlights edge-side single-user applications and centralized services with underutilized capacity as unsaturated cases.Source: §6 · p. 9
  • For API-based models, extra prefilling tokens may increase costs, and prompt tuning is suggested as a way to shorten SoT prompts.The concern follows from API charges tied to token usage.Source: §6 · p. 9
  • SoT’s acceleration ratio depends on the prompt, model, and question, making it less predictable and controllable than model- or system-level techniques and potentially hindering adoption.The authors frame data-centric efficiency optimization as an initial attempt with substantial room for further work.Source: §6 · p. 9

A MODEL DETAILS

B IMPLEMENTATION DETAILS OF SKELETON-OF-THOUGHT

The implementation uses concise skeleton and point-expansion prompts, adapts prompt formatting across models, and supports multi-round conversations. It routes questions to SoT or normal decoding and trains a router using annotated question-answer data.

B.1 PROMPT

  • The skeleton prompt requests a numbered list of 3–10 short points, while the expansion prompt asks for concise point content; partial answers and demonstrations help standardize prompt outputs.GPT-4 omits skeleton demonstrations, and demonstrations are not needed for reasonable point-expansion responses.Source: §B.1 · p. 19

B.2 SUPPORTING MULTI-ROUND CONVERSATION

  • For multi-round conversations, the question and final aggregated answer remain in history while SoT prompts are removed, avoiding added prefill cost in later rounds.The paper says this prevents SoT in one round from introducing additional prefill cost in future rounds.Source: §B.2 · p. 20

C.1 PROMPTING ROUTER

  • The prompting router selects SoT for independent answer points and falls back to normal decoding for dependent-point answers or answers that should not be lists.The router prompt labels these options A, B, and C, respectively.Source: §C.1 · p. 20, §B.1 · p. 20

C.2 TRAINED ROUTER

  • The trained router treats routing as sequence classification, fine-tunes RoBERTa on annotated LIMA questions, and applies it to Vicuna-80 and WizardLM.The LIMA training set contains 1,030 Q&As from Stack Exchange, wikiHow, and Pushshift Reddit.Source: §C.2 · p. 20, §C.2.1 · p. 20

C.2.1 ANNOTATION PROCESS

  • GPT-4 assists annotation by judging whether questions yield independently expandable points with enough detail; two authors independently annotate and resolve disagreements.The paper presents router confusion matrices for Vicuna-80 and WizardLM in Tables 3 and 4.Source: §C.2.1 · p. 21, Table 3 · p. 21, Table 4 · p. 21

C.2.2 TRAINING DETAILS

The router is trained with a false-positive-sensitive objective, and its agreement with other routers varies by dataset. The paper also describes a concurrent SoT-R execution option that could lower latency at the cost of token overhead.

C.2.2 TRAINING DETAILS

  • The trained router uses RoBERTa-base with 120M parameters, trained for two epochs with batch size 32 and sequences padded or truncated to 512 tokens.Training uses AdamW with weight decay 0.01, a warm-up over the first 1% of iterations to learning rate 5e-5, and linear decay.Source: §C.2.2 · p. 21
  • The RoBERTa router uses Tversky loss with α = 0.7 and β = 0.3 to penalize false positives more heavily, plus label smoothing with ϵ = 0.2.The authors treat false positives as more concerning than false negatives because they can degrade answer quality; fine-tuning completes in two minutes on an NVIDIA A100 GPU.Source: §C.2.2 · p. 21

C.3 ROUTER CONSISTENCY

  • On Vicuna-80 the three routers show notable agreement, while on WizardLM they diverge substantially and GPT-4 produces many false positives; RoBERTa aligns more closely with human annotations there.On Vicuna-80, the trained router has slightly more false negatives than the GPT-4 router relative to human annotations.Source: §C.3 · p. 21

C.4 CONCURRENT EXECUTION FOR SOT-R

  • SoT-R could run routing, normal generation, and SoT generation concurrently, aborting one generation after routing; the authors defer this option because it increases token overhead.Figure 9 depicts the SoT-R pipeline and a possible further latency reduction approach with token overhead.Source: §C.4 · p. 21, Figure 9 · p. 22

D.1 EFFICIENT LLMS

The related work spans model- and system-level efficiency methods, prompting for answer quality, and hierarchical text generation. SoT instead uses prompts for efficient generation through explicit, free-form planning with off-the-shelf LLMs.

D.1 EFFICIENT LLMS

  • Model-level efficiency methods modify or compress models, while system-level approaches target computation and throughput; SoT instead organizes content and can use throughput techniques for batched decoding.Existing speculative decoding and non-autoregressive methods address sequential decoding through assisting models or specially designed models and schemes, unlike SoT’s plan-based generation.Source: §D.1 · pp. 22–23

D.2 PROMPTING METHODS FOR LLMS

  • Prior prompting work chiefly targets capabilities and answer quality, whereas SoT is presented as an initial use of prompting to improve inference efficiency.The literature includes step-by-step reasoning, multiple reasoning paths, tool use, and prompts for complex tasks.Source: §D.2 · p. 24

D.3 HIERARCHICAL TEXT GENERATION

  • Unlike hierarchical generation methods aimed at answer quality, SoT uses interpretable language for explicit, free-form plans without new architectures or planning modules.The authors suggest hierarchical-generation research could inspire future SoT extensions to broader question types.Source: §D.3 · p. 24

E EFFICIENCY ANALYSIS

For local models, SoT batches point-expanding requests to exploit decoding’s low GPU utilization. Profiling and latency breakdowns show the potential and practical limits of this approach.

E EFFICIENCY ANALYSIS

  • Batched decoding keeps per-sequence token latency roughly stable as B rises, increasing GPU utilization nearly linearly and enabling ideal B× decoding speed-up, though practical gains are smaller.Prefilling overhead and imbalanced segment lengths reduce the practical speed-up; Figure 10 tests latency, GPU performance, and peak memory across batch sizes.Source: E EFFICIENCY ANALYSIS · p. 25, Figure 10 · p. 25
  • SoT’s peak memory overhead grows slowly with batch size because model weights dominate the tested memory footprint, allowing experiments on one GPU without other memory optimizations.This memory comparison assumes prefilling token lengths are not too large.Source: E EFFICIENCY ANALYSIS · p. 25, Figure 10 · p. 25

F EFFICIENCY PROFILING

  • Profiling measures prefilling and one-token decoding latency across lengths and batch sizes, then uses those latency tables to estimate request-generation latency.Decoding measurements cover batch sizes 1–16 and context lengths 1–1024; prefilling is profiled at ten-token intervals.Source: F EFFICIENCY PROFILING · p. 26

G.1.1 DETAILED STATISTICS OF TOKEN LENGTHS AND POINT NUMBERS

  • Figure 11 organizes token-length and point-count statistics by question category in rows and model in columns.Source: Figure 11 · p. 27

G.1.2 LATENCY BREAKDOWN: SOT STAGES AND PHASES

  • Although SoT adds prefilling in its skeleton and point-expanding stages, decoding dominates end-to-end latency, so the added prefilling has negligible impact on overall speed-up.Figure 12 presents the latency breakdown for SoT and normal generations on Vicuna-80.Source: §G.1.2 · pp. 27–28, Figure 12 · p. 28

G.1.3 EFFICIENCY EVALUATION ON NVIDIA RTX 3090

SoT speeds up generation in RTX 3090 and actual A100 batch tests, with results varying across models and question categories. Router comparisons also differ between Vicuna-80 and WizardLM.

G.1.3 EFFICIENCY EVALUATION ON NVIDIA RTX 3090

G.1.4 ACTUAL LATENCY TESTING

  • SoT achieves >2× speed-up on 6 of 9 open-source models, >1.7× on OpenChat-13B and Vicuna-33B V1.3, and no speed-up on StableVicuna-13B; across five high-quality-answer categories, generation speeds up 2.15×–2.50×.Figure 14 reports actual batch testing on the Vicuna-80 dataset.Source: §G.1.4 · p. 29, Figure 14 · p. 29

G.2.1 SPEED-UP BREAKDOWN: MODELS

  • On Vicuna-80, trained and GPT-4 prompting routers yield similar SoT-R speed-ups; on WizardLM, GPT-4 prompting usually achieves higher speed-ups than trained routing, especially on GPT-4 itself.Source: §G.2.1 · p. 29

G.2.2 SPEED-UP BREAKDOWN: CATEGORIES

H OVERHEAD OF SOT IN DIFFERENT SCENARIOS

SoT adds prefilling work, particularly for API models, and the authors identify ways to reduce that overhead. The answer analysis also describes model-dependent problems with following SoT prompts and completing expansions.

H OVERHEAD OF SOT IN DIFFERENT SCENARIOS

I.1.1 ANSWER ANALYSIS: DIFFERENT MODELS

  • Weak models sometimes mis-follow SoT prompts or omit needed expansion details, while Claude’s low net win rate is attributed mainly to already-strong normal answers.The authors also report that models with high net win rates are sufficiently able to understand SoT prompts and have more room to improve over normal generation.Source: §I.1.1 · p. 32, SoT’s answer (OpenChat-13B) (Orange… · p. 33

I.1.2 ANSWER ANALYSIS: DIFFERENT QUESTION CATEGORIES

This section examines SoT across question categories, using answer examples to describe where its point-based approach can help and where it encounters limitations. It covers dependence between answer steps, point expansion and skeleton presentation, and roleplay assessments by LLMZoo and FastChat metrics.

I.1.2 ANSWER ANALYSIS: DIFFERENT QUESTION CATEGORIES

  • Math exposes SoT’s dependence problem: even when a strong model gets the skeleton right, expanding steps independently can produce errors, while weaker models may generate an incoherent skeleton.The passage contrasts this with Chain-of-Thoughts, which emphasizes sequential thinking for solving math problems.Source: §I.1.2 · pp. 33–34
  • In Fermi reasoning, ChatGPT-3.5 loses track of prior assumptions, producing inconsistent calculations and refusing to answer, while normal generation produces a consistent solution.SoT may instead embed computation in the skeleton and fall back to sequential generation, a behavior the authors say needs better control.Source: §I.1.2 · pp. 34–35
  • Coding expansion often omits code or repeats it; SoT is poorly suited to tightly dependent short programs, but may help larger modular tasks with skeleton-defined interfaces and parallel implementations.The authors suggest coding-specific prompts, including asking for code snippets, as possible improvements.Source: §I.1.2 · p. 36, SoT’s answer (OpenChat-13B) (Orange… · p. 36

SoT’s answer (OpenChat-13B) (Orange texts are the skeleton.)

  • Writing answers are often less detailed, and including skeleton points in the final prose can reduce coherence; removing them or letting the model decide whether to include them could improve fluency.The authors suggest prompts requesting more detail as a possible remedy for reduced detail.Source: SoT’s answer (OpenChat-13B) (Orange… · pp. 36–37
  • Counterfactual, knowledge, common-sense, and generic questions benefit from SoT because their answers cover relatively independent points, enabling more comprehensive, readable, and structured responses.The passages describe these four categories as having relatively independent points that can be outlined and expanded.Source: SoT’s answer (OpenChat-13B) (Orange… · p. 37

Knowledge

  • The knowledge example contrasts single-use bottles’ pollution, emissions, resource use, chemical exposure, and microplastics with reusable bottles’ environmental and health benefits.The answer also mentions bacteria risks from improperly disposed single-use bottles and says regular cleaning lowers risk for reusable bottles.Source: SoT’s answer (OpenChat-13B) (Orange… · p. 39

Common sense

  • For common-sense advice about reading conversational interest, the example lists body language, engaged responses, eye contact, follow-up questions, and tone as cues.It cautions that cues are not definitive: communication styles differ, and directly asking may be needed to understand someone’s intentions.Source: SoT’s answer (OpenChat-13B) (Orange… · p. 39

Generic

  • The generic time-management examples recommend goals and priorities, scheduling, limiting distractions, breaks, delegation, and tools such as calendars or to-do lists.One answer also recommends using a timer and says improvement takes practice and perseverance.Source: SoT’s answer (OpenChat-13B) (Orange… · p. 40

Roleplay

  • For roleplay, LLMZoo and FastChat metrics usually favor SoT’s level of detail, though its answers could be better if skeleton points were removed.The example is a medieval-knight introduction at a royal banquet.Source: SoT’s answer (OpenChat-13B) (Orange… · p. 40

I.1.3 ANSWER ANALYSIS: DIFFERENT METRICS

This section analyzes SoT across distinct answer-quality metrics, including how its list structure relates to coherence and immersion and how its planning and point-expansion stages relate to diversity and relevance. It also covers quality across models and question categories, evaluation variants and comparisons, reporting that with ChatGPT-3.5 as judge, SoT is better than or equal to normal generation in around 80% of cases under general metrics.

I.1.3 ANSWER ANALYSIS: DIFFERENT METRICS

  • SoT’s point-list format can lower coherence and immersion: it may omit transitions, introductions, and conclusions in scripts, and make role-play answers feel less in-character.The examples are described as less fluent for a YouTube script and stiffer than a chef’s conversational presentation.Source: §I.1.3 · pp. 40–42
  • SoT tends to improve diversity by prompting multiple perspectives and relevance by limiting expansion to proposed points; examples show five explained reasons versus one aspect and less irrelevant content.The relevance example contrasts SoT’s structured explanations with a normal answer whose opening adds little and whose closing offers keywords without concrete explanations.Source: §I.1.3 · pp. 42–44

I.1.4 QUALITY BREAKDOWN: QUESTION CATEGORIES AND MODELS

  • Figure 22 presents per-model and per-category results evaluated with the FastChat metric and GPT-4 as judge.The supplied passage identifies the breakdown’s scope but gives no individual results.Source: §I.1.4 · p. 44, Figure 22 · p. 44

I.2 SKELETON-OF-THOUGHT WITH ROUTER

  • SoT-R achieves similar or better quality than SoT in both reported cases, with usually non-negative net win rates, indicating fallback to normal decoding on unsuitable categories.On WizardLM, the trained router outperforms the prompting router in most cases; routers sometimes outperform humans.Source: §I.2 · p. 44, Figure 23 · p. 44, Figure 24 · p. 45

I.3 QUALITY COMPARISON WITH LONGER NORMAL ANSWER

  • When overall answer lengths are similar, SoT’s quality is comparable to longer sequential answers for both ChatGPT-3.5 and LLaMA2-Chat-7B.The comparison added prompts requesting longer normal-generation answers to account for possible GPT-4 judge bias toward longer responses.Source: §I.3 · p. 46

I.4 CHATGPT-3.5 AS THE JUDGE

  • ChatGPT-3.5 evaluations yield takeaways similar to GPT-4’s, but readers are directed to GPT-4 results for a more accurate view because GPT-4-based evaluation usually aligns better with humans.The passage cites prior work for the alignment claim.Source: §I.4 · p. 46

I.4.1 OVERALL QUALITY

  • The two quality metrics differed on when SoT was strictly better (50.2% v.s. 12.4%), but both found it no worse than baseline in more than 76% of cases; FastChat’s rate exceeded 89% excluding unsuitable math and coding questions.The authors take this as evidence that SoT answers maintain good quality.Source: §I.4.1 · p. 47
  • SoT performs better than or equal to normal generation in around 80% of cases under FastChat and LLMZoo general metrics.Evaluation used ChatGPT-3.5 as the judge.Source: Figure 28 · p. 47

I.4.2 QUALITY BREAKDOWN: QUESTION CATEGORIES

  • Net win rates show SoT performs relatively well on generic, common-sense, knowledge, roleplay, and counterfactual questions, but relatively badly on writing, Fermi, math, and coding.LLMZoo tends to be more optimistic than FastChat, though their conclusions are consistent.Source: §I.4.2 · p. 47, Figure 29 · p. 47

I.4.3 QUALITY BREAKDOWN: MODELS

  • FastChat and LLMZoo differ in absolute net win rates but agree that OpenChat-13B, Vicuna-7B V1.1, Claude, and ChatGPT-3.5 score low, while Vicuna-13B V1.3, StableVicuna-13B, and UltraLM-13B score high.Net win rates are computed across models.Source: §I.4.3 · p. 47, Figure 30 · p. 48

I.4.4 QUALITY BREAKDOWN: QUESTION CATEGORIES AND MODELS

  • Per-model and per-category net win rates are reported in a breakdown with question categories as rows and models as columns.The results are shown in Fig. 31.Source: §I.4.4 · p. 47, Figure 31 · p. 48

I.4.5 QUALITY BREAKDOWN: METRICS

On average, SoT improves diversity and relevance while hurting immersion and coherence. The appendix also evaluates SoT-R together with 4-bit GPTQ quantization.

I.4.5 QUALITY BREAKDOWN: METRICS

  • SoT improves diversity and relevance on average while hurting immersion and coherence.Figure 32 reports these LLMZoo metrics, evaluated with ChatGPT-3.5 as judge.Source: §I.4.5 · p. 48, Figure 32 · p. 48

J COMBINING SOT-R WITH MODEL QUANTIZATION

J.1 SPEED-UPS OF SOT + QUANTIZATION ON QUANTIZED MODELS

SoT-R speeds up quantized and unquantized-model comparisons, while the appendix also reports where SoT is suitable, its memory overhead, and application considerations. More points do not necessarily produce greater speed-ups.

J.1 SPEED-UPS OF SOT + QUANTIZATION ON QUANTIZED MODELS

J.2 SPEED-UPS OF SOT + QUANTIZATION ON UNQUANTIZED MODELS

  • 1.54× to 2.07× speed-ups over unquantized-model normal generation; across five high-quality-answer categories, speed-ups are 1.33× to 3.41×.The category results use prompting and trained routers.Source: §J.2 · p. 49, Figure 35 · p. 50, Figure 36 · p. 50

K.1 NUMBER OF SUITABLE QUESTIONS

  • Human assessment found 37/80 Vicuna-80, 58/218 WizardLM, and 371/1030 LIMA questions suitable for SoT; trained and prompting routers judged similarly.Source: Figure 37 · p. 50, §K.1 · p. 50

K.2 PEAK MEMORY OVERHEAD

K.3 SPEED-UPS WITH DIFFERENT NUMBER OF POINTS

  • More skeleton points do not necessarily yield greater speed-ups because SoT cannot control total response length to match normal generation.Figure 40 shows speed-ups by point count for three models on Vicuna-80.Source: Figure 40 · p. 51, §K.3 · p. 51

L NOTES ON APPLICATION SCENARIOS

  • Reduced latency can help users check response structure or extract specific information without waiting for the full answer; lower latency may also benefit agent-agent interaction.Source: L NOTES ON APPLICATION SCENARIOS · p. 51
Loading 2307.15337v3…