From Generation to Scoring: Scaling LLM-Based Semantic Labeling at LinkedIn

1 INTRODUCTION

• Packed generation shares profile encoding but couples each answer to companion qualifications and earlier answers, while also decoding and parsing an answer list.The task is to label retrieved qualifications against profile information; stored predictions are reused downstream.Source: §1 · p. 1
• Semantic scoring uses single-candidate prompts and an answer-token score from one prefill pass; MIS batches scores while keeping candidate decisions independent.The recipe includes label-only fine-tuning and an answer-token readout whose decision threshold can be set without retraining.Source: §1 · pp. 1–2
• Reordering changes at most 0.01% of labels; matching agreement takes about 4.7× fewer training pairs, while the MIS-based stack raises throughput 1.24× offline and 1.9× in serving.The serving comparison uses a matched one-second 99th-percentile latency target.Source: §1 · pp. 1–2
• The paper adapts MIS for Ray Data batch inference and uses MIS support in SGLang for model serving, evaluating agreement, order sensitivity, throughput, and request sizing.The serving and offline paths use the same per-candidate decision contract for profile updates.Source: §1 · p. 2

2.1 Case study: profile qualification labeling

• The system retrieves ranked catalog qualifications in rounds, may stop when the accepted fraction falls below a threshold, and stores accepted IDs for downstream reuse.Retrieval and early stopping trade recall for cost; candidates outside the top K are never judged, and stopping may miss later positive labels.Source: §2.1 · p. 2, Figure 1 · p. 2

2.2 Workload shape and notation

• The mean query is about 27 times as long as the mean candidate item, so independent scoring repeats most input tokens without prefix reuse.Each judgment combines shared query text q with one candidate item ci.Source: §2.2 · p. 2

2.3 System requirements

• The system requirements are comparable agreement with reference labels, consistent decisions across candidate orders and paths, efficient offline throughput, and streaming throughput at a tail-latency target.The streaming metric is throughput at fixed tail latency, not end-to-end label freshness.Source: §2.3 · p. 3

2.4 Packed-generation baseline and its trade-off

• Packed generation encodes the profile once and amortizes request overhead, but conditions each answer on all qualifications and earlier answers, generates a formatted list, and returns hard decisions rather than independently thresholdable scores.Independent scoring preserves per-item decisions but requires N requests per round, each containing the same query.Source: §2.4 · p. 3

3 FROM GENERATION TO SEMANTIC SCORING

• Semantic scoring assigns one score to each query–item pair, and MIS later computes many such scores together without changing them.The recipe defines the prompt, trains the labeler, and converts answer-token logits into a score.Source: §3 · p. 3

3.1 Prompt design

• The prompt places instructions and candidate-invariant text in the shared query and candidate-dependent text in the item, ending at the answer position.Moving the query–item boundary requires revalidation; the compact prompt uses a single-token readout marker.Source: §3.1 · pp. 3–4

3.2 Model training

• Training regroups reference-labeled examples to one qualification each and masks all tokens except the answer label when minimizing its negative log-likelihood.Members are split before pairing, preserving the member-disjoint holdout; MIS is used at inference, not training.Source: §3.2 · p. 4

3.3 Score readout and validation

• A prefill pass supplies answer-position logits, which are renormalized over YES and NO; validation tuning selects the precision–recall operating point without retraining.A qualification is accepted when its score meets threshold τ; the default is 0.5, with ties assigned to YES.Source: §3.3 · p. 4

4.1 Mechanism

• Each item attends causally to the shared query and its own item prefix, while reused position IDs preserve the context of a separate [q; ci] prompt.Together, masking and reused positions yield independent scores up to floating-point differences, regardless of other items or their order.Source: §4.1 · pp. 4–5

4.2 Cost of scoring one query

• MIS consolidates N items into one sequence and one output, whereas independent scoring schedules N sequences and collects N outputs, even with perfect query-prefix reuse.A single-token generation baseline also uses one prefill pass, so the decode savings over packed generation come from removing the multi-token answer list.Source: §4.2 · p. 5

4.3 Cost model

• At N = 200, MIS uses about 32 prefill tokens per label, versus about 63 for packed generation and about 770 for independent requests without reuse.Packed generation uses about 2.0× the input tokens per label of MIS; MIS also removes answer-list decoding.Source: §4.3 · p. 5
• As N grows, the prefill-token reduction factor approaches 1 + ρreq, while compute follows the token ratio only up to constant factors.For items no longer than the query, attention work is O(Lq^2 + NLq L̄req); MIS computes query self-attention once and each item attends only to q and itself.Source: §4.3 · pp. 5–6

4.4 Request sizing

• A request’s aggregate length is Lq + the sum of item lengths, so it must fit the engine’s input-length limit and KV-cache budget.Splitting requests can reduce aggregate length but repeats q unless prefix caching reuses it; larger requests also leave less room for serving batches.Source: §4.4 · p. 6

5 BATCH AND MODEL-SERVING INTEGRATION

• Both paths replace only the labeler, leaving retrieval, rounds, early stopping, and the stored-label schema unchanged for downstream consumers.They share the checkpoint, answer tokens, and threshold, but use different item layouts: compact offline and chat in model serving.Source: §5 · p. 6
• Offline inference uses a Ray Data MIS scoring API that maps each input row to a shared query and its items, returning one score per item in input order.Separate autoscaling actor pools scale CPU tokenization and GPU inference independently; completed rows are persisted so interrupted jobs can resume.Source: §5 · p. 6
• Profile-update serving packs each round into one request to a prefill-only SGLang engine, which returns item-aligned scores without sampling or decoding.Servicer processes tokenize and pack requests off the engine event loop; the engine gathers scores on the GPU and frees the request’s KV cache after prefill.Source: §5 · p. 6

6.1 Setup

• The benchmark uses about 500K profiles and 100M query–item pairs; reference-label agreement is evaluated on a member-held-out set.The workload averages 744 query tokens and 28 item tokens; catalog gaps, retrieval misses, early-stopping omissions, and product outcomes are outside the evaluation.Source: §6.1 · p. 7

6.2 RQ1: Quality

• 86.3% agreement at 45K training pairs: packed generation needed about 210K pairs, or 4.7× as much data, to match semantic scoring.At 443K pairs, agreement was comparable: 88.0% for semantic scoring and 87.8% for packed generation; no significance claim was made.Source: §6.2 · p. 7, Figure 4 · p. 7
• At every training size, semantic scoring’s inconsistency rate stayed at or below 0.01%, while packed generation’s was 0.86% even at its largest size.The comparison uses labels changed across qualification reorderings; packed generation’s largest-size inconsistency was at least 86× higher.Source: Figure 5 · p. 7, §6.2 · p. 8
• Semantic scoring trains on one answer token per pair, avoiding answer-list length, formatting, and alignment targets, while repeating the profile increases training tokens per labeled pair.MIS recovers profile sharing at inference; the authors present this as a possible explanation for the quality and consistency gaps.Source: §6.2 · p. 8

6.3 RQ2: Throughput

• 1.24× offline and 1.9× in model serving: the MIS-based stack outperformed packed generation in throughput, with serving measured at a one-second p99 target.These are stack-level comparisons; the stacks also differ in engine, checkpoint, prompt format, and runtime tuning.Source: Table 3 · p. 8, §6.3 · p. 8
• The offline MIS run reached 6.3K labels/s per GPU versus 5.1K for packed generation after pipeline tuning, including a compact layout and two SGLang processes sharing the GPU.The initial MIS configuration reached only 4.4K labels/s; the reported configuration raised offline MIS throughput by 44%.Source: §6.3 · p. 8, Table 3 · p. 8

6.4 RQ3: Request size

• At N = 200, serving throughput reached about 4.8K labels/s per GPU, overtaking packed generation’s 2.5K between N = 20 and N = 50.Splitting 200 candidates into two N = 100 requests lowered throughput by about 14%, to 4.1K labels/s.Source: §6.4 · pp. 8–9, Figure 6 · p. 9

6.5 RQ4: A smaller decision model (exploratory)

• The 0.6B labeler reached about 10.9K labels/s per GPU in serving and an estimated 11.3K offline, but its quality metrics are unreported.The exploratory results make no general claim about the size–quality trade-off; the offline prompts also did not use the model’s own chat template.Source: §6.5 · p. 9

• Deployment implies about 20% fewer GPU-hours for the labeling stage of each offline run, with smaller end-to-end savings because other stages are not accelerated.Retrieval, prompt preparation, and output stages remain outside MIS acceleration.Source: §7 · p. 9
• On the streaming update path, deployment implies about 50% fewer provisioned GPUs at the same peak update rate and latency target, assuming capacity scales with per-GPU throughput.Source: §7 · p. 9
• Offline incremental runs and periodic refreshes retain the same schedule and completion window, and downstream consumers need no interface changes.Source: §7 · p. 9

8 RELATED WORK

• The paper’s contribution is combining prior answer-token readout, hard-label fine-tuning, and candidate-isolating MIS to move production labeling from generation to scoring, plus a new Ray Data batch API.The combined recipe serves both labeling paths; the Ray Data API is proposed upstream.Source: §8 · p. 9
• Qualification labelers follow a retrieve-then-label pattern, assigning independently thresholded support scores so one profile can receive multiple qualifications.Li et al. (2020) use taxonomy-based candidates and learned ranking; Xu et al. (2026) retrieve attribute candidates and classify values.Source: §8 · p. 9
• Pointwise ranking uses answer-token probabilities as relevance scores, while listwise methods generate an ordering after seeing candidates together; qualification decisions need to remain independent.The paper applies pointwise scoring to binary qualification judgments with a causal language model; Sun et al. (2023) is the listwise comparison.Source: §8 · pp. 9–10
• Prior MIS work shares query computation across independent item scores for live ranking; this paper instead thresholds scores into stored labels on batch and streaming paths.Li et al. (2026) introduced MIS in MixLM, and Borisyuk et al. (2026) used it in semantic search.Source: §8 · p. 10
• Prefix caching and shared-prefix attention reuse query computation but keep each candidate in a separate sequence, whereas MIS packs candidates together with attention-isolating masks.Parallel Context Windows also masks attention between segments, but extends the context of one generated answer rather than reading independent item scores.Source: §8 · p. 10

9 DISCUSSION AND LIMITATIONS

• Semantic scoring with MIS applies when many candidates share one context, decisions depend only on that context and candidate, and one answer position yields the score.The paper does not evaluate the listed candidate applications; it excludes listwise reranking and joint deduplication because the mask blocks their required cross-candidate interactions.Source: §9 · p. 10
• Quality is measured as agreement with reference judgments on retrieved candidates, not product-level outcomes; an absent label may mean a negative prediction or an unevaluated qualification.The catalog, retriever, and early-stopping rule remain unchanged.Source: §9 · p. 10
• Throughput gains are stack-level comparisons and cannot be attributed to MIS alone; smaller-model results are exploratory, leaving the smallest quality-preserving scorer unresolved.The stacks differ in engine, checkpoint, prompt format, and runtime tuning.Source: §9 · p. 10
• Prompt, checkpoint, answer-token mapping, and threshold form one versioned artifact across both paths, while request size trades query-work amortization against latency and batching headroom.MIS’s shared-context reuse within each request does not depend on cache residency or request routing.Source: §9 · p. 10

10 CONCLUSION

• At most 0.01% of labels change under reordering, and packed generation needs about 4.7× as many training pairs to match agreement with reference labels.The stack-level throughput increases are 1.24× in offline batch inference and 1.9× in model serving for profile updates.Source: §10 · p. 10
• The end-to-end recipe uses single-candidate prompts, label-only training, answer-token readout, and prefill-only execution with MIS.MIS-based scoring is implemented as a reusable Ray Data API proposed upstream for offline batch inference; model serving uses MIS in SGLang.Source: §10 · p. 10
• When decisions are independent, the recipe separates judging each candidate alone from the systems optimization that executes many judgments together.Source: §10 · p. 10