Efficient Iterative Retrieval with Heterogeneous Batching

• Orthrus executes embedding and generation requests together in a shared model execution pass, co-scheduling tasks for fine-grained load balancing without additional model training.Source: §1 · pp. 1–2
• Embedding requires substantial computation while generation decode uses relatively little compute, so naïve batching makes lighter work wait and fixed batch ratios become brittle as output lengths vary.Source: §1 · p. 2
• Chunked embedding processes input tokens across scheduling iterations while incremental pooling preserves the cross-chunk state needed for the final embedding.Source: §1 · p. 2
• Orthrus adjusts batch composition using remaining token work in each queue rather than raw request counts.Source: §1 · p. 2
• 1.28×–4.52× higher throughput than GPU-level splits on four A100 GPUs; chunked and single-pass embeddings are equivalent up to floating-point reduction-order effects.Source: §1 · p. 2

2.1 Modern RAG Workflow

• Iterative retrieval alternates embedding and generation to refine queries dynamically, turning a single prompt into a sequence of mixed-workload requests.Source: §2.1 · p. 3

2.2 Computational Characteristics of Embedding and Generation

• Generation decode is memory-bound, whereas embedding is compute-bound; profiling found that neither pure workload fully exploits both GPU resources.Source: §2.2 · p. 3

2.3 Batching and Scheduling

• Existing iteration-level schedulers batch generation homogeneously; Orthrus extends chunked execution to embeddings and jointly schedules both request types while preserving within-type arrival order.Source: §2.3 · pp. 3–4

3.1 High-Level Workflow

• Orthrus uses one runner for embedding and generation, with schedule, forward, and emit stages in each iteration.The emit stage samples a generation token or incrementally pools an embedding into its final vector.Source: §3.1 · p. 4, §3 · p. 4, Figure 2 · p. 5

3.2 Batch Unit

• Batches combine embedding chunks, generation prefill chunks, and single-token generation decode items in a shared transformer pass.Items differ in input length and output handling; sorting them by type makes each output kernel operate on a contiguous hidden-state slice.Source: §3.2 · p. 4

3.3 Incremental Pooler

• Incremental pooling aggregates hidden states across embedding chunks and emits the pooled vector after the final chunk.Mean and weighted-mean pooling use a streaming formulation; CLS and last-token pooling retain the selected hidden state.Source: §3.3 · p. 4
• Incremental and single-pass pooling are algebraically equivalent for the described heads, though floating-point non-associativity may cause small numerical differences.Arbitrary deterministic heads can buffer hidden states, at an O(nd) memory cost.Source: §3.3 · pp. 4–5

4 Intra-Batch Scheduling

• IBS dynamically allocates capacity proportional to the workload mix and is designed to meet three requirements: proportional allocation, per-type FCFS, and no head-of-line blocking.Requests of the same type retain arrival order, while the two types may interleave.Source: §4 · p. 5, §4.1 · p. 5

4.2 Scheduler Design

• The scheduler first reserves capacity for in-flight generation decodes, then alternates between embedding and generation queues to fill the remaining budget.Embedding requests are admitted as chunks of at most C tokens, which can leave fewer than C tokens unused when token capacity is limiting and no further chunk fits.Source: §4.2 · p. 5
• Persistent queue credits make the scheduled embedding-to-generation ratio converge to the ratio of their remaining token counts, avoiding scheduler-induced starvation under stated assumptions.For fixed weights, allocation differs from the ideal proportional share by at most one scheduling opportunity; this does not bound queuing delay when offered load exceeds capacity.Source: §4.2 · pp. 5–6

5.1 Experimental Setup

• The evaluation compares dedicated, disaggregated, unified homogeneous, unified heterogeneous, and Orthrus configurations on up to four A100 40GB GPUs, using synthetic workloads and Iter-RetGen.The baselines vary model placement, batching, and scheduling; Orthrus combines same-GPU placement, heterogeneous batching, and IBS.Source: §5.1 · p. 6

5.2 High Throughput

• Across evaluated workload ratios, Orthrus achieves higher combined throughput than every static GPU partition and fully utilizes GPUs where Dedicated1:3 leaves the embedding GPU underused.The controlled workload tests use 512 clients over 180 seconds.Source: §5.2 · p. 7, Figure 3 · p. 7
• At four GPUs, Orthrus reaches 0.624 requests/s on Iter-RetGen, versus 0.548 for Unified-Homo, 0.603 for Disagg-PD, and 0.260 for Dedicated.It achieves the highest throughput across all tested GPU counts.Source: §5.2 · p. 7, Table 3 · p. 7

5.3 Low Latency with IBS

• On Iter-RetGen, Orthrus reduces four-GPU end-to-end p99 latency by 55.8% versus Dedicated (593.4s vs. 1342.0s).It has the lowest latency across all evaluated GPU counts.Source: §5.3 · p. 8, Table 4 · p. 8

5.4 High GPU Utilization

• Across three phases with 10%, 50%, and 90% embedding, Orthrus averages 79% GPU utilization versus 38% for Dedicated1:3 and completes each phase 43% faster.Static partitioning leaves the dedicated embedding GPU underused in generation-heavy phases and overloaded in embedding-heavy phases.Source: §5.4 · p. 8, Figure 5 · p. 9

5.5 Minimal Overhead

• At saturation (at least 64 clients), Orthrus matches dedicated vLLM generation throughput; its embedding throughput is comparable to or higher than the corresponding dedicated configuration.At lower concurrency, Orthrus trails dedicated generation throughput; the lower absolute throughput of both LoRA curves reflects adapter overhead.Source: §5.5 · p. 8, Figure 6 · p. 9

5.6 Generalization

• Across Mistral 7B, Qwen2 7B, and LLaMA3.1 8B, Orthrus has higher combined throughput than Dedicated1:3 at all tested request ratios and LoRA ranks up to 64.For Mistral 7B at a 1:9 embedding-to-generation ratio, embedding throughput falls from 5.10 to 4.20 requests/s while generation throughput rises from 29.56 to 39.10 requests/s.Source: §5.6 · p. 9
• On the 2WikiMultihopQA workload, Orthrus improves embedding and generation throughput by 22.9% and 20.2%, respectively, reduces both average latencies and embedding p95 latency, while generation p95 rises 3.8% and mean generation latency falls 17.9%.Source: §5.6 · p. 9

• Orthrus co-serves embedding and generation in a unified model runner through heterogeneous batching.Source: §6 · p. 9
• Chunked embedding, incremental pooling, and workload-aware scheduling are the design components used to handle mixed workloads.Source: §6 · p. 9
• The paper reports higher throughput and lower latency under mixed workloads, supporting efficient serving for knowledge-intensive LLM applications.Source: §6 · p. 9

Limitations

• Experiments use one model replica per GPU on up to four A100 GPUs, so gains beyond four GPUs are not established.More replicas may change load balancing and workload skew, affecting utilization and tail latency.Source: Limitations · p. 10
• Orthrus does not support a single model sharded across multiple GPUs; tensor or pipeline parallelism may interact differently with its batching and scheduling.The separate-model experiment covers only two models that fit together on one A40.Source: Limitations · p. 10
• The experiments use open-weight models with fewer than 10 billion parameters and do not establish the same benefits for larger models.Larger models may have different memory demands, compute-to-memory ratios, and batch-capacity constraints.Source: Limitations · p. 10

A.1 Pooling Equivalence

• The paired-output test uses 250 samples spanning a range of input lengths and compares chunked with single-pass embeddings.It evaluates mean, CLS, and weighted-mean pooling at chunk sizes 256, 512, and 1024.Source: §A.1 · p. 14
• The small embedding discrepancies reflect floating-point reduction-order effects rather than an algorithmic approximation.Source: §A.1 · p. 14
• Minimum cosine similarity exceeds 0.9999 in all nine pooling-head and chunk-size configurations, while mean and p99 relative L2 errors remain below 1%.The comparison covers mean, CLS, and weighted-mean pooling at chunk sizes 256, 512, and 1024.Source: Table 7 · p. 14, §A.1 · p. 14

A.2 Scheduling-Parameter Sensitivity

• Combined throughput ranges from 2.6167 to 2.6500 requests/s across chunk sizes 128–1024, with generation p99 at 4.08–4.16 s and embedding p99 at 83.58–98.77 ms.The sweep uses the 5:5 workload; the passage reports performance as insensitive to chunk size over this range.Source: Figure 7 · p. 14, §A.2 · p. 14
• Combined throughput varies by less than 1% across token budgets 2048, 4096, and 8192, while generation p99 is 4.06–4.12 s and embedding p99 is 82.95–109.35 ms.The sweep holds the 5:5 workload and chunk size fixed; the 256- and 512-token chunk settings tie for highest throughput, and 512 is selected for lower embedding p99.Source: Figure 7 · p. 14, §A.2 · p. 14
• The evaluated throughput and latency for embedding and generation remain stable across the tested chunk sizes and token budgets.Source: Figure 7 · p. 14

A.3 Standalone Chunked Embedding Ablation

• In the 80% generation, 20% embedding simulation, chunking raises average token-budget fill from 8.8% to 85.7%; generation throughput stays 93.3 requests/s and embedding throughput rises from 25.5 to 26.9 requests/s.The simulation compares atomic requests with embedding chunks of at most 64 tokens under a 256-token iteration budget.Source: §A.3 · pp. 14–15, Table 8 · p. 15
• On 250 mean-pooled OPT-1.3B inputs, chunking reduces mean latency by 9.8% (20.4 to 18.4 ms) and raises throughput by 10.9% (49.0 to 54.4 embeddings/s).Source: Table 9 · p. 15, §A.3 · p. 15

A.4 Latency Robustness

• At 90% generation, embedding p95 rises to 1.71 s, while generation TTFT p99 stays below 200 ms and embedding p50 remains 278.70–301.87 ms across tested shares.Embedding p95 is 322.59 and 327.35 ms at 10% and 50% generation, respectively.Source: Figure 8 · p. 15, §A.4 · p. 15
• With decode lengths from 256 to 4096 tokens, generation p99 rises from 7.06 to 79.68 s while embedding p99 stays between 53.09 and 59.92 ms.The central result is latency isolation; the passage cautions that throughput is not monotonic across settings.Source: Table 10 · p. 15, §A.4 · p. 15