PaperlayerView exact revision on arXiv

Source-linked AI summary

Pairwise Ranking Outperforms Single-Action RL for Offline Explanation Selection: A Practical Lesson

Tanay Chowdhury, Saeideh Shahrokh Esfahani

arXiv:2608.18531v1cs.AIcs.LG

TL;DR

LLM-generated explanations are costly to serve, motivating offline generation followed by lightweight request-time selection. Across offline-pool and KG-path evaluations, LambdaRank outperformed single-action RL selectors, reaching BERTScore-F1 = 0.500 on Google Local.

  • Problem

    Selecting high-quality explanations under strict serving-cost and latency constraints remains insufficiently evaluated for offline candidate pools.

  • Method

    The paper pre-generates explanation candidates and compares lightweight offline-pool and knowledge-graph-path selectors across Google Local and MovieLens-1M.

  • Results

    Pairwise LambdaRank outperformed PPO, GRPO, DPO, and distillation variants, reaching BERTScore-F1 = 0.500 on Google Local.

  • Takeaways & Limitations

    Pairwise learning-to-rank should be evaluated before RL when offline labels are available for every candidate.

  • Takeaways & Limitations

    The reported cost and latency figures are measured offline rather than under live production traffic, and production A/B evaluation remains future work.

Abstract

from arXiv · show

Industrial explainable-recommendation systems built on LLMs incur a substantial serving cost: each request triggers an LLM generation, with latency in the hundreds of milliseconds and cost that scales linearly with traffic. We separate generation from selection: explanations are produced ahead of time as a frozen candidate pool (six prompt styles, two commodity LLMs), and a small CPU-resident selector picks one at request time. The stack needs no GPU and returns in under 100 ms. Our primary benchmark is a 2,958-pair XRec Google Local subset, evaluating six offline-pool selectors (LambdaRank, PPO, GRPO, DPO, teacher-student distillation) and three KG-path selectors (random walks, edge-disjoint enumeration, MMR-reranked paths). A 300-pair MovieLens-1M split with Claude-Sonnet-4.5 references serves as an internal cross-dataset check, since no public benchmark exists for this setting. All variants use the same BERTScore-F1 protocol as XRec and G-Refer, averaged across five seeds. LambdaRank reaches F1 = 0.500 on Google Local, exceeding both G-Refer and XRec, and F1 = 0.329 on the MovieLens-1M check. With seed variance below 0.003 F1, the ordering is reliable: pairwise learning-to-rank outperforms single-action RL (PPO, GRPO, DPO), which use only one labelled candidate per rollout, leaving K-1 labels unused. The KG-path family targets a different objective: all three variants reach USR = 1.000 on Google Local and 0.997-1.000 on MovieLens-1M, since per-request path grounding yields a unique output per query, avoiding template-collapse failures affecting cached-LLM outputs. A generator-pool study comparing Claude 3 Haiku and Claude Haiku 4.5 shows small F1 shifts (0.001-0.006) while preserving selector ranking: selector and generator can be evaluated independently, though absolute F1 depends on the generator. End-to-end build cost is near $15 on commodity hardware.

1 Introduction

The paper moves LLM explanation generation offline into a frozen candidate pool, then uses a lightweight selector at serving time. LightGBM LambdaRank best matches reference explanations, while KG-path generation achieves perfect or near-perfect uniqueness on a different metric.

  • System design: Offline generation uses K candidates from six prompt styles and two commodity LLMs, selected at request time by a small CPU-resident model.This relocates inline LLM complexity to the offline path for production traffic under bounded latency.
  • Experimental setting: Fixed candidates and dense BERTScore-F1 labels create a plug-in setting for comparing selectors using 30-dimensional feature vectors.The evaluated methods span offline-pool ranking and KG-path generation families.
  • Main finding: F1 = 0.500 makes LambdaRank the strongest selector on the 2,958-pair Google Local subset, exceeding G-Refer 8B and XRec.The reported gains are 0.041 over G-Refer 8B (0.4592) and 0.069 over XRec (0.4311).
  • KG-path family: USR = 1.000 on Google Local and 0.997–1.000 on MovieLens-1M show that all three KG variants avoid cached-output template collapse through per-request path grounding.The KG-path family therefore targets output uniqueness rather than reference alignment.

2 Problem Setting and Framework Overview

The framework generates a fixed candidate pool offline and selects the highest-scoring explanation online, optimizing BERTScore-F1 against benchmark references within a latency budget. All selector variants share the same candidate features, while online inference remains CPU-efficient.

  • Problem setting: BERTScore-F1 is the optimization target for producing each explanation against the XRec or G-Refer reference within a per-request latency budget.The metric uses the roberta-large encoder with baseline rescaling.
  • Offline stage: K candidates are generated offline for every user–item pair by crossing six prompt styles with two commodity LLMs.The styles range from retrieve-grounded paraphrase and synthesis to adversarial refinement and length-tuned synthesis; the models are Amazon Nova Lite and Claude Haiku.
  • Featurisation: Each candidate is represented by a fixed 30-dim feature vector spanning reference retrieval, query–candidate relevance, linguistic and source-side, and provenance signals.The reference-side group contains 6 dimensions, while query–candidate relevance contains 4 dimensions.
  • Selector inputs: All nine selector variants receive the same input features, with LambdaRank operating directly on 𝜙 and RL variants using padded candidate slots plus user and item embeddings.The RL policy state is s∈R1536+64K.
  • Online stage: At request time, the selector returns the argmax candidate, with total latency under 100 ms at the 99th percentile on an 8-core c5.2xlarge CPU instance.LambdaRank uses one LightGBM forward pass under 1 ms, while MLP-based RL variants use one policy-network pass under 20 ms on CPU.

3 Selector Variants … 3.3 KG-path selection: MMR paths with dual-style generation

The section presents nine selector variants organized by candidate construction: KG-path methods generate candidates from graph paths, while later methods rank candidates from an offline LLM pool. The KG-path variants progress from temperature-biased sampling to edge-disjoint enumeration and MMR reranking with dual-style generation, under a common evaluation protocol.

  • 3 Selector Variants: Nine selector variants are organized by candidate construction, with §§3.1–§3.3 extracting multi-hop KG paths and §§3.4–§3.10 ranking offline LLM-pool candidates.The framework separates selector variants according to how candidates are constructed.
  • 3 Selector Variants: All nine variants use the identical §4.2 evaluation protocol, making their results directly comparable.
  • 3.1 KG-path selection: temperature-biased random walks: KG-path candidates are constructed at training and inference time by sampling up to five paths between each user–item pair using biased random walks.Next-node probabilities are proportional to exp(cos(h_v,h_target)/τ).
  • 3.1 KG-path selection: temperature-biased random walks: The random-walk variant uses τ=0.5 by default for near-greedy walks and τ=1.5 to widen the path distribution.
  • 3.2 KG-path selection: edge-disjoint path enumeration: The edge-disjoint variant extracts up to five structurally diverse paths using Menger-style max-flow decomposition, with fallbacks to node-disjoint and shortest-path alternatives.Recovered paths occupy the same ten-slot action space, padded with empty placeholders when fewer paths are available.
  • 3.2 KG-path selection: edge-disjoint path enumeration: This edge-disjoint method supplies structural diversity deterministically through a graph algorithm rather than stochastic sampling.
  • 3.3 KG-path selection: MMR paths with dual-style generation: The MMR variant samples 20 paths, retains the five most structurally distinct with λ=0.7, renders each using factual and personal-voice prompts, and lets PPO select one.It produces ten candidates per user–item pair and forms a compact request-time candidate set.

3.4 Pool-only heuristic (no learning) · 3.5 LambdaRank (pairwise learning-to-rank) · 3.6 Single-step PPO with adaptive entropy

The section progresses from a structural-score heuristic lower bound to a LightGBM LambdaRank selector, then formulates selection as a one-step PPO problem. PPO uses adaptive entropy and a mixed structural-semantic reward, but wastes the labels of unsampled candidates during training.

  • 3.4 Pool-only heuristic (no learning): The heuristic is a no-learning lower bound for diagnosing featurisation problems and measuring what the candidate pool provides without supervision.A learned selector that fails to exceed it indicates a featurisation issue rather than a learning issue.
  • 3.5 LambdaRank (pairwise learning-to-rank): LambdaRank trains a LightGBM pairwise ranker on quintile-binned per-candidate BERTScore-F1 labels across approximately 130k Google Local candidates and 5,000 query groups.The ranker uses 500 trees, 31 leaves, and a 0.05 learning rate.
  • 3.5 LambdaRank (pairwise learning-to-rank): At inference, LambdaRank selects the candidate with the highest ranker score within each test group.Its inputs are the 30-dimensional feature vector defined in §2.2, covering retrieval and linguistic signals.
  • 3.6 Single-step PPO with adaptive entropy: PPO models selection as a one-step MDP using a 2-layer MLP policy, a separate value network, and roughly 1.08M parameters trained for 500 episodes.The state concatenates user and item embeddings with zero-padded candidate slots, producing 4,096 dimensions for K=40 and 2,688 for MovieLens K=18.
  • 3.6 Single-step PPO with adaptive entropy: PPO combines structural and semantic rewards as r = αR_struct + (1−α)R_sem with α=0.1, preserving grounding without overwhelming the BERTScore-F1 signal.R_sem is rescaled BERTScore-F1, while R_struct is the heuristic’s structural proxy.
  • 3.6 Single-step PPO with adaptive entropy: PPO uses only the sampled action’s reward for each rollout, leaving the other K−1 candidate labels unused and discarding most available dense supervision.The subsequent variants are designed to address this structural limitation.

3.7 Group-relative policy optimisation (GRPO) · 3.8 Direct preference optimisation (DPO)

GRPO densifies one-step policy gradients with within-group reward normalisation, while DPO learns from weighted preference pairs anchored to a frozen reference. DPO also provides the lowest reported RL-variant seed variability, with 0.0006 F1 standard deviation.

  • 3.7 Group-relative policy optimisation (GRPO): GRPO retains PPO’s policy architecture but removes the value network because it is redundant in a one-step bandit.Its advantage uses a within-group z-score instead of a value baseline.
  • 3.7 Group-relative policy optimisation (GRPO): GRPO computes normalisation statistics from all K group rewards despite sampling one rollout action, densifying the gradient.The statistics μ_a and σ_a use the complete reward group.
  • 3.7 Group-relative policy optimisation (GRPO): +0.01 F1 is the lift from replacing PPO’s value-baseline advantage with GRPO’s z-score, matching the full GRPO change.This ablation attributes the gain to normalisation rather than removing the value network.
  • 3.8 Direct preference optimisation (DPO): DPO samples up to 80 preference pairs per training state, retaining pairs whose rescaled F1 gap exceeds δ=2.0.Pairs are ordered as (a+, a−) with F1(a+) > F1(a−) + δ.
  • 3.8 Direct preference optimisation (DPO): DPO weights each pair by its F1 gap and implicitly applies KL regularisation through the student-to-reference log-ratio.Training uses β_dpo=0.1 for 500 episodes, with the reference policy frozen from a random-init MLP snapshot.
  • 3.8 Direct preference optimisation (DPO): 0.0006 F1 is DPO’s across-seed standard deviation, the lowest among the tested RL variants.Pairwise supervision supplies many gradient updates per state, while the reference anchor stabilises training.

3.9 Teacher–student distillation (Stage A only) · 3.10 Distillation plus RL fine-tuning (Stage A+B)

Teacher–student distillation compresses LambdaRank’s ranking signal into a fast neural policy but loses some F1 through softmax smoothing. Adding GRPO fine-tuning worsens performance, as entropy-driven exploration regresses a near-converged student.

  • 3.9 Teacher–student distillation (Stage A only): Stage A trains an MLP student to match a LightGBM LambdaRank teacher’s softmax candidate distribution using KL divergence.The teacher uses the same 30-dimensional features, temperature T=1.0, 200 epochs, and Adam learning rate 3×10−4.
  • 3.9 Teacher–student distillation (Stage A only): 1M parameters and a 10× faster forward pass compress the teacher’s ranking into a neural student.The student uses the MLP architecture applied to PPO, GRPO, and DPO.
  • 3.9 Teacher–student distillation (Stage A only): F1 = 0.4817 ± 0.0003 on Google Local trails LambdaRank’s 0.5003 after distillation.Softmax compression spreads near-uniform probability across high-ranked candidates, preventing the student from recovering the teacher’s sharp argmax.
  • 3.10 Distillation plus RL fine-tuning (Stage A+B): 500 episodes of GRPO fine-tuning with learning rate 1×10−4 and β∈[0.05, 0.005] turn the near-optimal Stage A student into a regression.The procedure uses a tighter entropy schedule intended to recover the remaining F1 gap.
  • 3.10 Distillation plus RL fine-tuning (Stage A+B): Stage A+B scores 0.4767 versus Stage A’s 0.4817, a > 10σ degradation across five seeds.The entropy bonus pushes a near-converged policy back toward exploration.
  • 3.10 Distillation plus RL fine-tuning (Stage A+B): The framework separates candidate-source variation in KG-path methods from selector variation in the offline-pool family.The KG-path family holds selector architecture fixed, while the offline-pool family holds the LLM candidate pool fixed; Table 1 evaluates all nine variants under three metrics.

4 Datasets and Evaluation

The study evaluates methods on Google Local and MovieLens-1M using shared reference explanations and three metrics, with exact-subset rescoring for XRec and G-Refer comparisons. MovieLens KG-path results also show that genre-hub topology limits reachable path diversity and reduces selector effects.

  • Google Local: 2,958 Google Local test pairs remain after dropping 42 uncovered pairs from the canonical 3,000-pair XRec split.The study also samples 5,000 training pairs from trn.pkl and verifies byte-identity with G-Refer’s published predictions.
  • MovieLens-1M: 300 MovieLens-1M pairs form the test split, with 600 for training and 1,100 unused after a seed-42 positional split.The candidate pool contains K = 18 candidates per pair from Claude Haiku 4.5 and Nova Lite, while Claude-Sonnet-4.5 supplies reference explanations.
  • MovieLens-1M: ±0.005 F1 separates all three MovieLens KG configurations, indicating limited selector effect on the genre-hub graph.The graph has 9,941 nodes, 581 k edges, and density 0.012; nearly all user-to-movie paths pass through one of 18 genre nodes.
  • Evaluation metrics: Three metrics evaluate all methods against the same references: BERTScore-F1, BARTScore, and USR.BERTScore-F1 is primary and matches XRec and G-Refer code; BARTScore measures log p(reference | prediction), while USR detects template collapse.
  • Evaluation protocol: 42 pairs distinguish apples-to-apples rescoring from published-paper numbers, which use the full 3,000-pair split and may use a different BERTScore library version.XRec and G-Refer predictions are rescored on the exact 2,958-pair review-covered subset under the study’s evaluation code.

5 Results

Results show that pairwise LambdaRank consistently outperforms single-action RL selectors, while KG-path selectors achieve perfect or near-perfect uniqueness through query-specific grounding. Selector rankings transfer across datasets and generator pools, but absolute F1 depends on the generator.

  • Main benchmark: 0.500 F1: LambdaRank leads Google Local, exceeding G-Refer by +0.041 and XRec by +0.069; all six offline-pool variants beat XRec.The KG-path variants underperform the offline-pool family on F1 but achieve the top USR score of 1.000.
  • Reliability: ≤0.0030 F1 standard deviation: every RL variant remains stable across five seeds, and GRPO exceeds DPO by 0.0046 on Google Local.The GRPO–DPO gap is more than ten times either method’s standard deviation.
  • Offline-pool selectors: 0.02–0.04 F1: LambdaRank beats every trained RL variant on Google Local, using labelled scores from all 40 candidates rather than one sampled action.The pairwise ΔNDCG objective provides a denser training signal than PPO, GRPO, and DPO rollouts.
  • Cross-dataset consistency: 0.04–0.05 F1: LambdaRank’s advantage also appears on MovieLens, with most ordering preserved despite different pool sizes, generators, and reference texts.The only reported local swap is DPO marginally overtaking Distillation-A-only, with MovieLens using K=18 versus 40 candidates on Google Local.
  • KG-path selectors: 0.325–0.327 F1: KG-path variants on Google Local fall to 0.262–0.270 on MovieLens, indicating that their F1 ceiling tracks graph topology.The architectures and training code are identical across datasets; MovieLens has only 18 genre intermediary nodes.
  • Generator robustness: 0.001–0.006 F1: upgrading from Haiku 3 to Haiku 4.5 lowers F1 across methods while preserving selector ordering; USR rises by +0.018–+0.083.Absolute F1 is therefore generator-dependent, and the upgrade costs roughly 4× more per token.

6 Discussion

The discussion emphasizes that pairwise learning-to-rank benefits from dense offline supervision, while RL can regress or exploit BERTScore-F1 through reward hacking. Deployment observations include fallback styles for cold-start items and high review coverage in the test data.

  • Findings: Pairwise learning-to-rank outperforms every trained reinforcement-learning variant because LambdaRank uses labelled BERTScore-F1 scores for every candidate, unlike one labelled action per PPO, GRPO, or DPO rollout.The dense-supervision mechanism explains the observed advantage without requiring RL machinery.
  • Negative results: 0.002–0.005 F1 is lost when distillation is followed by RL fine-tuning, as GRPO’s entropy bonus pushes a near-optimal student toward exploration.Stage A+B consistently regresses relative to distillation alone.
  • Negative results: 0.001–0.006 F1 regression follows upgrading the candidate-pool generator from Claude 3 Haiku to Claude Haiku 4.5, attributed to greater drift from XRec’s trope-heavy references.The newer model is more fluent, but its outputs align less closely with the reference text.
  • Failure modes: BERTScore-F1 reward hacking caused a policy to repeat “user enjoys” five times per output, whereas decoupling generation from selection avoids this failure mode by construction.The repeated tokens appeared frequently in the references, enabling exploitation within hundreds of steps.
  • Deployment observations: 98.6% of pairs (2,958 of 3,000) have review coverage; cold-start items without retrieval coverage fall back to pool styles B, E, and F.The remaining pairs are dropped at evaluation.

7 Related Work

Prior explainable-recommendation methods use transformer generation, knowledge-graph reasoning, or request-time LLM systems. This work instead selects among pre-generated explanations, distinguishing its task from recommendation-oriented graph traversal and adapting preference-based methods to single-step selection.

  • Prior explanation systems: Earlier methods generated explanations from transformer encoders or knowledge-graph reasoning, while XRec and G-Refer combine retrieval with request-time LLM generation.The request-time LLM introduces the latency and cost overhead targeted by the proposed separation of generation and selection.
  • Knowledge-graph recommendation: KGAT, CKAN, and PGPR optimize recommendation accuracy or item ranking, unlike this work’s selection over a pre-generated explanation pool.PGPR uses reinforcement learning for multi-hop graph traversal, whereas this task does not traverse the graph for ranking.
  • Preference-based selection: DPO is adapted from preference-pair classification to a single-step bandit, but LambdaRank directly uses per-candidate F1 labels and performs better.The passage also reports that DPO has high stability but remains bounded above by LambdaRank.
  • KG-path selection: KG-path selection provides structural traceability to graph edges, whereas the offline-pool family is stronger on the reference-aligned F1 metric.Three evaluated variants construct candidates through KG paths, making provenance valuable when it is a business requirement.

8 Conclusion

The framework moves LLM generation offline into a frozen candidate pool, enabling a small CPU-resident selector to return in under 100 ms at a per-query cost ratio on the order of 103 in its favour. Across nine selectors on the 2,958-pair Google Local benchmark, LambdaRank is the strongest BERTScore-F1 performer, significantly outperforming the RL and distillation baselines under a five-seed protocol.

  • Offline serving: Under 100 ms per request, the frozen candidate pool and CPU-resident selector avoid online LLM generation.LLM-based baselines generate explanations on every request, whereas this framework relocates that cost to the offline path.
  • Offline serving: 103 is the order of the proposed approach’s per-query cost advantage, based on public Bedrock and EC2 pricing.The passage reports the cost ratio as being on the order of 103 in favour of the proposed approach.
  • Selector comparison: LambdaRank is the strongest BERTScore-F1 performer across nine selectors on the 2,958-pair Google Local benchmark.Its gap over PPO, GRPO, DPO, and the two distillation stages is statistically significant under a five-seed protocol.
Loading 2608.18531v1…