PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Dimensional Comparative Scale Construction for Efficient Personalized Subjective Judgment in High-Traffic Applications

Xianglong Shi, Shifeng Liu, Sirui Zhao, Shengming Yuan, Enhong Chen

arXiv:2609.33282v1cs.AI

TL;DR

Subjective intensity is hard to quantify, and judgments vary across individuals, complicating subjective assessment in high-traffic applications. The paper builds relative scales from case–person comparisons, uses sparse multi-judge Elo construction and BPO-trained SubJudge for efficient scoring, and reports competitive results with faster inference.

  • Problem

    Subjective intensity is difficult to quantify, and judgments vary substantially across individuals, although both matter in high-traffic applications.

    Source: §1 · pp. 1–2

  • Method

    The framework compares case–person pairs to build relative scales, then uses sparse multi-judge Elo construction and BPO to train SubJudge for personalized scoring.

    Source: §1 · pp. 2–3

  • Results

    9B models match or surpass evaluated frontier LLMs on multiple metrics across PLURIHARMS and iNews; on H100, SubJudge achieves approximately 1.29× to 261× speedup in mean inference latency over Qwen3.5-9B on PLURIHARMS.

    Source: §1 · p. 2, §4 · p. 10

  • Takeaways & Limitations

    Comparative supervision is reported as a practical basis for personalized subjective evaluation in high-traffic applications.

    Source: §4 · p. 10

Abstract

from arXiv · show

Subjective judgments are central to many high-traffic applications, but subjective intensity is difficult to quantify and perceptions vary substantially across individuals. To address these challenges, we propose a pairwise comparative framework for multi-dimensional scale construction. By comparing case-person pairs along case and profile dimensions, the framework constructs relative scales that capture both fine-grained intensity and individual variation. To support practical high-traffic deployment, we optimize both offline scale construction and online inference. For scale construction, we combine sparse Elo comparisons with multi-judge voting, cutting the comparison cost from $O(N^2)$ to $O(NK)$ for $N$ objects and a budget of $K$ opponents per object, while limiting reliance on any single judge. For inference, we propose SubJudge, a System One model for personalized scoring with Batchwise Preference Optimization (BPO). Using Bradley-Terry comparisons, BPO trains the model to learn relative orderings, and SubJudge reads a continuous score from digit-token probabilities at the first response position, requiring only one forward pass per criterion and reducing the inference complexity to $O(1)$. Experiments on PluriHarms and iNews show that our 9B models match or surpass the evaluated frontier LLMs on multiple metrics. On the H100 GPU, SubJudge achieves an approximately $1.29\times$ to $261\times$ speedup in mean inference latency over Qwen3.5-9B with different thinking budgets. The code is available at https://github.com/Longchentong/SubJudge.

1 INTRODUCTION

The introduction frames subjective judgment as difficult to quantify and variable across people, then proposes pairwise comparisons of case–person pairs to construct relative scales. Sparse multi-judge Elo construction and SubJudge target efficient use of those scales, and the reported evaluations show strong benchmark performance and faster inference.

1 INTRODUCTION

  • Subjective judgments are difficult to quantify and vary across people, with one PLURIHARMS request receiving ratings across the full 0–100 range despite an average of 51.16.The example concerns a request about covertly favoring applicants from prestigious universities.Source: §1 · pp. 1–2
  • Pairwise comparisons of case–person pairs capture fine-grained subjective intensity while preserving individual variation through case and profile dimensions.Comparing different cases for the same person reveals intensity differences; comparing different people for the same case reveals individual variation.Source: §1 · p. 2
  • Sparse Elo comparisons with multiple LLM judges reduce scale-construction comparisons from O(N^2) to O(NK), while reducing reliance on any single evaluator.Each object competes against K selected opponents, and multiple judges' votes are aggregated before Elo updates.Source: §1 · p. 2
  • SubJudge scoring models match or surpass evaluated frontier LLMs on multiple metrics across PLURIHARMS (Li et al., 2026) and iNews; on the same H100 GPU, they achieve approximately 1.29× to 261× speedup in mean inference latency over Qwen3.5-9B on PLURIHARMS.Source: §1 · p. 2
  • SubJudge uses Batchwise Preference Optimization to transfer ruler-induced rankings into a scoring model that requires one forward pass per evaluation criterion.The framework's overview depicts judges voting on case–person matches, Elo updates constructing a ruler, and BPO transferring preferences to SubJudge.Source: §1 · p. 3, Figure 2 · p. 3

2 METHOD

The method builds criterion-specific relative scales from case–person comparisons along case and profile dimensions, then trains SubJudge to score those pairs without reference comparisons at inference. Sparse multi-judge Elo construction and batchwise preference training address the costs of comparison and scoring.

2.1 MULTI-DIMENSIONAL SCALE CONSTRUCTION WITH MULTI-JUDGE ELO

  • Sparse comparisons cover two axes: different cases for the same person and different people for the same case, isolating content and person variation.Each target receives a budget of K opponents across the pools, subject to availability, with K ≪ N.Source: §2.1 · p. 3, §2 · p. 3
  • Majority voting aggregates each pair’s judge verdicts before Elo updates; ties contribute no winning votes, reducing reliance on any single judge.The same fixed schedule is replayed for all judges, so asynchronous response times do not determine Elo update order.Source: §2.1 · p. 4

2.2 TWO-STAGE POST-TRAINING FOR SUBJUDGE

  • SubJudge learns ruler-induced pairwise orderings with Batchwise Preference Optimization and scores a case–person pair in one forward pass per dimension, without reference comparisons.The model is initialized with digit-based supervision, then trained on batchwise preferences using unrounded ruler targets to retain fine distinctions.Source: §2.2 · pp. 4–5
  • With fixed numbers of dimensions and judges, sparse scheduling reduces scale-construction comparisons from O(N^2) to O(NK), while inference requires O(1) model evaluations with respect to ruler size.The scorer requires one forward pass per dimension and no judge queries or reference comparisons at inference.Source: §2.1 · p. 4, §2.2 · p. 5

3 EXPERIMENTAL RESULTS

Across PLURIHARMS and iNews, the evaluated 9B models achieve results comparable to or better than frontier LLM baselines on multiple metrics, with additional analyses examining training, judge panels, Elo budgets, and inference efficiency.

3.1 BENCHMARK RESULTS

  • On iNews, T+P Arousal, Dominance, and Valence MAEs are 0.7530, 0.6960, and 0.9361; Arousal exact-match accuracy is 42.31%.The Arousal MAE is 15.7% lower than GPT-5.6-sol with T+P inputs (0.8929).Source: §3.1 · p. 5, Table 2 · p. 7
  • On PLURIHARMS, Individual MAE is 0.1659 versus 0.168 for Kimi-K3 with thinking enabled; on iNews, T+P Arousal and Dominance MAEs are 0.7530 and 0.6960.The PLURIHARMS result is the lowest Individual MAE among evaluated models; the iNews comparisons are against GPT-5.6-sol (T+P) and GPT-5.4 (I+P), respectively.Source: §3.1 · p. 6, Table 1 · p. 6, Table 2 · p. 7

3.2 ABLATION AND ANALYSIS

  • Across four five-judge panels, PLURIHARMS Individual MAE ranges from 0.1659 to 0.1682 and Aggregated MAE from 0.2243 to 0.2278.Training on judge-derived scales also comes close to training directly on human ground-truth ratings.Source: §3.2 · p. 7, Table 3 · p. 8
  • Adding BPO after digit SFT lowers PLURIHARMS Individual and Aggregated MAEs from 0.1960 and 0.2341 to 0.1659 and 0.2278, the lowest point estimates across settings.The reductions are 15.3% and 2.7%, respectively; BPO-only yields 0.1788 and 0.2442.Source: §3.2 · p. 7, Table 4 · p. 8
  • In the binary safety replay, raw Accuracy peaks at 91.18% with 58 comparisons per item, while the defined saturation budget is 53.At evaluation subset sizes of 800 and above, the raw-best budget median is 58 with a 57–59 10th–90th percentile range; the saturation median remains 53.Source: §3.2 · p. 8, Figure 3 · p. 8
  • With I+P inputs, the full five-judge panel reaches 43.18% Test accuracy and 43.33% Personalization-test accuracy, improving over one judge by 0.86 and 1.46 points.The single-judge comparison is against GPT-5.6-sol.Source: §3.2 · p. 9, Table 5 · p. 9

3.3 COMPUTATIONAL EFFICIENCY

  • With thinking disabled, SubJudge’s mean latency is 0.503 seconds, yielding 1.29×, 17.53×, and 261.0× speedups over Qwen3.5-9B with thinking off, 256-token, and 4,096-token budgets.On one H100 80GB GPU, batch size 64 throughput is 4.095 valid requests/s; the gain over thinking-off is smaller because short responses leave input processing dominant.Source: §3.3 · p. 9, Table 6 · p. 9

4 CONCLUSION

The framework constructs multi-dimensional comparative scales that capture subjective intensity and individual variation, then transfers their orderings into SubJudge for personalized scoring. Experiments show that 9B models match or surpass evaluated frontier LLMs on multiple metrics across PLURIHARMS and iNews.

4 CONCLUSION

  • Sparse Elo updates and multi-judge voting make scale construction efficient while reducing reliance on a single evaluator.Comparisons between case–person pairs provide relative supervision along case and profile dimensions.Source: §4 · p. 9
  • BPO transfers comparison-derived orderings into SubJudge, which produces personalized scores with one forward pass per evaluation criterion.SubJudge reads continuous scores from digit-token probabilities at the first response position.Source: §4 · p. 9
  • 9B models match or surpass evaluated frontier LLMs on multiple metrics across PLURIHARMS and iNews.Source: §4 · p. 10

AI USE STATEMENT

The authors describe generative AI use in developing and implementing the work, while stating that they reviewed the AI-assisted work and take responsibility for its final content.

AI USE STATEMENT

  • Generative AI tools were used for synthetic data, theoretical or conceptual development, research methods, experiments, implementation, and translation.Source: AI USE STATEMENT · p. 10
  • The authors state that generative AI was not used for qualitative or thematic data analysis or interpretation of results.Source: AI USE STATEMENT · p. 10
  • Two authors verified and tested all LLM-generated code, and the authors take responsibility for the final content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The work studies subjective judgment for an open-weight language model using public benchmark data and does not recruit human participants or collect new user interactions.

ETHICS STATEMENT

REPRODUCIBILITY STATEMENT

The paper describes its release and implementation details alongside related work on comparative scales and modeling annotator-specific judgments. Its approach links judgments across cases and people before distilling comparisons into a scorer.

REPRODUCIBILITY STATEMENT

  • The reproducibility statement identifies appendices covering benchmark splits, comparison schedules, training, evaluation, and efficiency, and says code and comparison API datasets will be released.Source: REPRODUCIBILITY STATEMENT · p. 10

A RELATED WORK

  • Prior LLM-derived harmfulness scales use a single judge and Elo ratings directly, while QuRating trains a rater on one text dimension.The passage notes that separately constructed person-specific scales do not identify relative positions across people.Source: A RELATED WORK · p. 15
  • The framework compares different cases for the same person and different people for the same case, then distills the resulting orderings into SubJudge.Prior work models annotator variation, but many methods use absolute ratings that can be interpreted inconsistently across contexts and individuals.Source: A RELATED WORK · pp. 15–16

B IMPLEMENTATION DETAILS

  • The implementation describes five-judge Elo for iNews with screenshot and profile inputs, a text-and-profile variant, and separate scorers for each iNews dimension.Source: B IMPLEMENTATION DETAILS · p. 16

B.1 DATASETS AND DATA PREPARATION

The appendix documents the benchmark data and examples, then specifies how case–person comparisons are scheduled, judged, and aggregated into scales. It also describes how the examples illustrate variation across people, cases, and dimensions.

B.1 DATASETS AND DATA PREPARATION

  • iNews contains 2,899 UK news posts and 291 participants; PLURIHARMS contains 150 prompts rated by participants on a 0–100 harm scale.iNews targets Arousal, Dominance, and Valence on 1–7 scales; PLURIHARMS descriptors characterize the dataset rather than serving as scorer inputs.Source: §B.1 · pp. 16–17
  • The iNews exports overlap, and the evaluation preserves released split membership while keeping calibration records separate from student training and validation.The exports' union contains 11,758 distinct person–post pairs; calibration uses 600 records from 30 Test participants, none of which is an evaluated target.Source: §B.1 · p. 17

B.2 ILLUSTRATIVE EXAMPLES

  • The examples show that a participant’s Arousal ordering can reverse across posts, while ratings can be tied on one dimension and ordered on another.For PLURIHARMS, three participants rate Q1 near the benign endpoint, while Q2 ratings span 24–69; the examples are illustrative, not predictive-performance estimates.Source: §B.2 · pp. 18–19

B.3 PAIR CONSTRUCTION AND MULTI-JUDGE ANNOTATION

  • The iNews schedule contains 232,009 unique edges for 4,926 observed targets, with 30 same-person and 30 same-post opponents per target.PLURIHARMS uses 20 opponents per axis and 195,160 edges over 9,789 observed objects; unobserved combinations extend small opponent pools but are not student training examples.Source: §B.3 · p. 19
  • For iNews, GPT and Gemini judges assess Arousal, Dominance, and Valence independently, with invalid responses retried rather than counted as ties.The prompt compares posts for the same person and people for the same post, using persona and screenshot information.Source: §B.3 · pp. 19–20
  • Five-judge directional votes are aggregated before Elo replay; ties contribute to neither side, and equal directional counts produce a panel tie.The resulting Elo strengths are min-max normalized for digit targets, while BPO learns the continuous Elo ordering.Source: §B.3 · p. 20

B.4 TRAINING CONFIGURATION

The training configuration initializes scoring with digit-based supervised fine-tuning, then applies BPO to learn comparative orderings. The PLURIHARMS reference configuration instead trains BPO directly from the original backbone.

B.4 TRAINING CONFIGURATION

  • iNews students receive two epochs of digit SFT followed by six epochs of BPO, using Qwen3.5-9B and LoRA adapters.SFT supervises the first response digit; its adapter is merged before a new BPO adapter is trained.Source: §B.4 · p. 20
  • For iNews, Arousal and Valence compare 16 examples per GPU, while Dominance compares 64 examples within each four-GPU group.The eight GPUs form two Dominance groups; the ranking term is normalized over valid pairs and is zero when none exist, while scalar regularization remains active.Source: §B.4 · p. 20
  • The reference configuration skips digit SFT and trains BPO directly from the original backbone, selecting checkpoint-876 by validation pairwise accuracy, loss, then earliest step.It retains the ten-digit LM-head readout and compares examples locally within each GPU.Source: §B.4 · p. 21

B.5 INFERENCE AND EVALUATION PROTOCOL

The protocol reads continuous predictions from digit-token probabilities and maps scores to ordinal ratings using calibrated thresholds. It evaluates benchmark-specific errors and accuracy, while documenting baseline handling and output validity.

B.5 INFERENCE AND EVALUATION PROTOCOL

  • At the final valid prompt position, the scorer returns the probability-weighted expectation over digit tokens 0–9, requiring one forward pass per requested dimension.The output is not sampled as a textual response; all three ADV scores require three separately trained models.Source: §B.5 · p. 21
  • For iNews, continuous scores are mapped to seven ordered levels using six thresholds fitted on training folds and selected on validation folds within a separate 600-record calibration pool.A fixed threshold vector is applied to test examples for each input mode and criterion.Source: §B.5 · p. 21
  • With matched calibration data, SubJudge has the lowest MAE on all three iNews dimensions.Table 14 compares remapped Qwen3.5-9B answers and refitted SubJudge thresholds using the same calibration records and cross-validated selection.Source: §B.5 · p. 21, Table 14 · p. 22
  • PLURIHARMS Aggregated MAE compares shared prompt predictions with individual ratings; its median-based oracle reaches the minimum Aggregated MAE of 0.2057.Individual and aggregated predictions are evaluated separately, with human ratings scaled to 0–1.Source: §B.5 · p. 22
  • Invalid API outputs remain in metric denominators with a score of −1, whereas SubJudge’s digit-probability readout cannot produce invalid outputs.On the 579-record Test split, the only reported invalid outputs are two Qwen3.5-9B responses.Source: §B.5 · p. 22

B.6 COMPUTATIONAL RESOURCES AND REPRODUCIBILITY

The appendix reports computational accounting, split definitions, and iNews pairwise voting outcomes. Aggregating five judges yields fewer ties than any individual judge in each dimension, providing more strictly ordered comparisons for scale construction.

B.6 COMPUTATIONAL RESOURCES AND REPRODUCIBILITY

  • Recorded trainer runtimes are reported for iNews T+P training on eight H100 GPUs, excluding queueing, data staging, adapter merging, and separate test evaluation.The appendix also records artifact identities and hashes to support checks of supervision consistency and target-label exclusion from student inputs.Source: §B.6 · pp. 22–23, Table 15 · p. 23
  • Each five-judge iNews run supplies 1,160,045 successful pair judgments across three dimensions; judge calls and scale fitting occur offline, while online cost depends on model and input length.PLURIHARMS supplies 975,800 successful pair judgments; these counts exclude retries and are not billed-attempt counts.Source: §B.6 · p. 23

C.1 RESULTS ON THREE OTHER TEST SPLITS OF INEWS

  • The additional iNews tests cover personalization on new posts, generalization to unseen participants, and cold-start evaluation on unseen participants and posts.They evaluate 1,641, 1,676, and 498 records, respectively; Tables 16–18 identify the corresponding splits.Source: §C.1 · p. 23, Table 16 · p. 24, Table 17 · p. 25

C.2 PAIRWISE VOTING ON INEWS

  • 4.09%, 5.34%, and 1.68% of comparisons are panel ties for Arousal, Dominance, and Valence, respectively—below every individual judge’s corresponding tie rate.The five judges assess the same 232,009 comparisons per dimension; the panel yields 9,489 Arousal, 12,383 Dominance, and 3,889 Valence ties.Source: §C.2 · p. 23, Table 19 · p. 26

C.3 ABLATION ON ELO

The Elo ablations show stable pooled agreement across parameter and replay-order choices, while pairwise comparison outperforms direct scoring on iNews. Pair-type agreement varies, with cross pairs dominating the pooled result.

C.3 ABLATION ON ELO

  • Across Elo settings, pooled agreement ranged from 78.020% to 78.964%; default global shuffles yielded 78.756% ± 0.026 points, and initial ratings of 0, 1500, or 3000 gave identical category results.The ablation varied replay order, update step, scale parameter, and initial rating.Source: §C.3 · p. 27, Table 20 · p. 27
  • Agreement was 75.659% for intensity, 72.959% for individual variation, and 78.906% for cross pairs; cross pairs were 98.12% of non-ties, dominating pooled agreement of 78.819%.Cross-pair order is inferred indirectly through the connected Elo graph because those pairs have no direct comparison edges.Source: §C.3 · p. 27, Table 21 · p. 27

C.4 PAIRWISE COMPARISON VS. DIRECT SCORING

  • Five-judge pairwise scales exceeded direct scoring by 2.0–5.6 points across all pairs and 1.7–4.9 points for pairs with one-point human-rating gaps.The comparison covers GPT-5.4 and Gemini-3.5-Flash on 4,897 records with valid direct scores.Source: §C.4 · p. 28, Table 22 · p. 28

D PLURIHARMS EFFICIENCY DETAILS

The efficiency measurements use selected PLURIHARMS test requests and compare SubJudge with Qwen3.5-9B under different thinking budgets. The appendix specifies request selection, generation behavior, and separate protocols for latency versus throughput and memory.

D PLURIHARMS EFFICIENCY DETAILS

  • For each of 50 distinct test questions, one available sequential measurement was selected uniformly with seed 20260920 before target ratings were read for offline evaluation.Predicted scores agreed across available repeats for every question; the statistics describe the 50 selected requests, not across-run variability from three complete repetitions.Source: D PLURIHARMS EFFICIENCY DETAILS · p. 28
  • The 4,096-token thinking-on batch-8 measurement took 1190.255 seconds and returned 50 valid scores, with all 50 requests requiring forced reasoning closure and none exhausting model context.SubJudge and thinking-off used completed configuration measurements on the same 50 questions and a selected batch-8 repetition.Source: D PLURIHARMS EFFICIENCY DETAILS · p. 28
  • At a 256-token thinking budget, mean latency was 8.809 seconds, P95 was 8.896 seconds, and the observed maximum was 8.951 seconds; batch-8 throughput was 0.616 valid requests/s.Peak allocated memory was 24.74 GiB, and batch-8 throughput was 14.67× that of the 4,096-token pass.Source: D PLURIHARMS EFFICIENCY DETAILS · pp. 28–29
  • Timing used 50 sequential requests for latency and a separate batch of 64 for throughput and memory; it included preprocessing through score extraction but excluded model loading and warmup.Table 23 breaks sequential timing into first-position time and later decoding, and defines completion counts for the backbone and SubJudge.Source: D PLURIHARMS EFFICIENCY DETAILS · p. 29, Table 23 · p. 29

Related papers

Loading 2609.33282v1…