PaperlayerView exact revision on arXiv

Source-linked AI summary

Safety as a Constraint: Fine-Tuning a LLM Recommender to Explain Itself

Jiashu He, Emma Yanyang Kong, JJ Tan, David Fagnan

arXiv:2609.13657v1cs.AI

TL;DR

Traditional recommender systems forecast what users may find interesting next without explaining why an item is a good match. The paper fine-tunes an in-house recommender to explain recommendations using similar shows in viewing history, with judges for three criteria and constrained GRPO that maximizes faithfulness subject to harmlessness constraints. On the held-out test set, the fine-tuned in-house generator raised all-criteria PASS rate from 0.649 to 0.956 versus the untuned base model; the frontier generator attained 0.652.

  • Problem

    Traditional recommender systems forecast what users may find interesting next without explaining why an item is a good match.

    Source: §1 · p. 1

  • Method

    The paper fine-tunes an in-house recommender to explain recommendations using similar shows in viewing history, with judges for three criteria and constrained GRPO that maximizes faithfulness subject to harmlessness constraints.

    Source: §1 · p. 1, §2 · p. 2, §4 · p. 4

  • Results

    On the held-out test set, the fine-tuned in-house generator raised all-criteria PASS rate from 0.649 to 0.956 versus the untuned base model; the frontier generator attained 0.652.

    Source: §5 · p. 4

  • Takeaways & Limitations

    The conclusion reports that the fine-tuned generator exceeds both thresholds more often than the frontier generator without degrading the model’s original abilities.

    Source: §7 · p. 6

    The study focuses on reference-style explanations, and post-training on high-quality samples does not enhance the model’s internal knowledge of the shows.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Traditional recommender systems are typically trained to predict what item users will interact with next, but not why. However, offering personalized evidence for why a user might like the predicted item is an important way to enhance the service and to raise the likelihood that the user will be genuinely interested in the recommendation. This service can be delivered by integrating a frontier-model call into the member-facing pipeline, but it will add extra cost and latency. In this paper, we train a recommender LLM to generate personalized explanations for its reccomendation, based on the user's watching history at a large video streaming service. We impose two requirements on the generated explanation: it must be faithful to the elements of the shows it links, and it must be strictly non-harmful to the user. To this end, we first train two LLM-judge reward models covering three specific criteria, and propose constrained GRPO to incorporate these different criteria. On a held-out real-world testing set, our fine-tuned model improves the all-three-criteria PASS rate rises from 0.649 to 0.956 under our own judges and from 0.677 to 0.931 under an independent judge, where as the frontier generator performs similar to the untuned recommender baseline. We conduct further experiments to show that the model's language and recommendation abilities remain unchanged. Based on these results, we conclude that an LLM-based recommender can be fine-tuned on other complex tasks without compromising its original recommendation performance, thus provide insights for further agentic user interface powered by a single model.

1 Introduction

The paper addresses recommender systems’ limited ability to explain why an item fits a user, training an in-house recommender LLM to explain recommendations through similar shows in the user’s viewing history. It combines domain-specific judges with constrained optimization to seek faithful explanations while meeting harmlessness requirements.

1 Introduction

  • The proposed recommender explains each recommendation by connecting it to similar shows in the user’s viewing history.The paper motivates explanations as a way to show why a recommended item may fit the user.Source: §1 · p. 1
  • Explanations must be faithful to a real, relevant shared element and harmless by avoiding offense or sensitive associations with user groups.The paper distinguishes this harmlessness standard from broader frontier-model safety; its example flags exclusionary audience cues as unacceptable.Source: §1 · p. 1
  • The approach uses two fine-tuned LLM judges and constrained GRPO, optimizing faithfulness while treating two harmlessness criteria as constraints.The introduction says the constraint weights adapt according to how well the current policy meets the criteria.Source: §1 · p. 2, Figure 1 · p. 2
  • On the held-out real-world test set, all-three-criteria PASS outcomes rose from 0.649 to 0.956 under the authors’ judges and from 0.677 to 0.931 under an independent frontier-model LLM judge.Source: §1 · p. 2
  • The tuned judges outperform a frontier model at judging explanation quality across all three criteria, with rationales on caught negative examples aligning more often with annotators.The paper describes the reward models as reasoners for evaluating explanation quality.Source: §1 · p. 2

2 Reward Modeling

The reward-modeling section builds separate faithfulness and harmlessness judges for three criteria, then evaluates their detection and reasoning against annotators and frontier-model judges. Its safety evaluations target a production standard that differs from conventional model alignment.

2 Reward Modeling

  • One harmlessness reward model assesses two safety criteria, while a separate faithfulness judge evaluates shared elements linking the shows based on metadata.The paper uses separate judges because faithfulness requires recognizing shared show elements, unlike the two safety-related criteria.Source: §2 · p. 2
  • The judges are fine-tuned on human-annotated explanations with criterion labels and annotator reasoning, then their predicted labels are used for online RL.The dataset is split 80/10/10 with approximately 70% PASS cases per split; supervised fine-tuning precedes online GRPO.Source: §2 · p. 3
  • 0.939 CATCH-A-FAIL rate for the tuned harmlessness judge, compared with 0.778 for the frontier-model judge using the same prompt.The paper attributes the gap to the frontier judge being more conservative about assigning FAIL; the untuned open-source model is described as overly aggressive.Source: §2 · p. 3
  • FAIL@8 testing-time scaling marks an example FAIL if any of eight runs fails, improving performance uniformly over the frontier judge for faithfulness evaluation.Under greedy decoding, the faithfulness judge detects 82% of FAIL cases; the authors aim to increase this rate above 90%.Source: §2 · p. 3
  • Tuned judges achieve 1.000 reason-to-label consistency; on correctly flagged harmlessness violations, their rationales score 0.957 versus 0.948 for the frontier judge.The judges achieve a similar consistency rate to the frontier judge on faithfulness.Source: §2 · p. 3

3 Warming Up the Generator

The generator is warmed up with supervised fine-tuning on explanations verified to meet all three criteria before constrained reinforcement learning. The warm-up data is filtered to retain high-quality examples, and the adapter is kept small to preserve output diversity for RL.

3 Warming Up the Generator

  • The generator first imitates explanations verified to meet all three criteria, then proceeds to constrained reinforcement learning.The paper describes this as a two-phase optimization process.Source: §3 · p. 3
  • 952 high-quality explanations that passed all three checks form the SFT dataset, split into 857 training and 95 validation examples.Candidate explanations were generated by a frontier model and rated by human annotators; cross-entropy loss is computed only on the assistant turn.Source: §3 · p. 3
  • The SFT adapter uses LoRA r=16 and α=32, deliberately kept small because an overfit warm-up would reduce output diversity needed for RL..Source: §3 · p. 3

4 Constrained Reinforcement Learning

Constrained reinforcement learning treats faithfulness as the objective and harmlessness as constraints. A primal–dual procedure adjusts the emphasis on each constraint according to whether the current policy meets its threshold.

4 Constrained Reinforcement Learning

  • Faithfulness is maximized subject to strict constraints requiring both harmlessness measures to remain above specified thresholds.The thresholds are set just below the warm-up model’s achieved levels.Source: §4 · pp. 3–4
  • The primal policy update uses GRPO group-normalized advantages and a KL-clipped update relative to the merged warm-up policy.Because the group mean is subtracted, safety terms adjust advantages only when rollouts disagree on a criterion.Source: §4 · pp. 3–4
  • Lagrange multipliers increase when a constraint is violated and decrease, potentially to zero, when it is satisfied, reallocating emphasis to active constraints and faithfulness.This provides per-constraint control throughout learning rather than fixing the reward trade-off in advance.Source: §4 · p. 4
  • The variants compare scalarized rewards, single bias constraints, separate harmlessness constraints, and a joint per-sample constraint on either harmlessness violation.The joint variant constrains the per-sample conjunction; the other constrained variants use expected group-level violations.Source: §4 · p. 4

5 Results

Fine-tuning raises held-out explanation PASS rates, while warm-up addresses a faithfulness failure pattern and recommendation and general-language performance are retained.

5 Results

  • All-criteria PASS rises from 0.649 to 0.956 for the fine-tuned generator, while the frontier generator reaches 0.652, essentially matching the untuned model.The comparison is reported for the held-out test set.Source: §5 · p. 4, Table 2 · p. 5
  • Warm-up closes the faithfulness gap by refocusing explanations on shared show elements instead of listing potentially unsupported metadata or similarity claims.The in-house model’s tendency to mention both titles and explicitly claim similarity gives the faithfulness judge more chances to flag errors.Source: §5 · pp. 4–5
  • The joint safety reward achieves the highest PASS-ALL-3 rate under both judge types by requiring every sample to be non-offensive and non-biased.Batch-level rates can conceal outputs that satisfy one safety property but fail the other.Source: §5 · p. 5
  • Under the independent judge, the fine-tuned model attains PASS-ALL-3 on up to 25.4% more samples than the untuned base and 26.4% more than the frontier generator.The authors use this independent evaluation to check whether gains reflect overfitting to their training reward signal.Source: §5 · p. 5
  • Retrieval normalized MRR increases by +0.0006 to +0.0009 versus the untuned policy, while assessed slices remain unchanged and language benchmarks are unchanged or marginally improved.Table 3 reports ranges over warm-up and four reward designs.Source: §5 · p. 6, Table 3 · p. 6

6 Related Work

Prior work spans reward modeling, constrained optimization, LLM judging, and recommendation explanations, but the authors identify a gap in applying primal-dual RL to deployment and training a generative recommender to explain its own recommendations.

Reward models, overoptimization, and constrained RL

  • Prior constrained-reward approaches include Lagrangian primal-dual optimization, weight interpolation, multi-objective DPO, and multihead reward models.The cited work also motivates constrained objectives with the risk that proxy rewards can rise while true task quality saturates.Source: §6 · p. 6
  • The authors describe their work as the first public application of primal-dual RL to fine-tune an in-house LLM for deployment under several realistic user-facing criteria.Source: §6 · p. 6

LLM-as-judge, reasoning, and recommendation explanation

  • Although recommendation explanations and generative recommenders have been studied, the authors know of no generative recommender fine-tuned to explain its own recommendations.The work also separates reward-model and generator base models because judges may favor their own outputs.Source: §6 · p. 6

7 Conclusion

The paper fine-tunes an in-house recommender LLM as its own explanation generator and reports improved criterion satisfaction without degrading recommendation or general-language performance. It identifies stronger domain-specific knowledge as future work.

7 Conclusion

  • The in-house recommender LLM serves as its own explanation generator, using two reward models across three criteria and constrained GRPO to maximize faithfulness while constraining harmlessness.The fine-tuned generator exceeds both thresholds more often than the frontier-model generator without degrading recommendation capability or general-language performance.Source: §7 · p. 6
  • Future work should strengthen the in-house model’s domain-specific knowledge to improve downstream-task performance rather than rely on task-by-task fine-tuning.Source: §7 · p. 7

Limitations

The work explores fine-tuning the recommender to generate explanations, but its scope is limited to reference-style explanations. Post-training improves explanation quality without improving the model’s internal knowledge of the shows.

Limitations

  • The experiments cover only reference-style explanations.The paper says a production system would ideally support multiple styles personalized to members’ preferences.Source: Limitations · p. 7
  • The paper identifies multiple explanation styles personalized to members’ preferences as a desired production capability beyond its scope.Source: Limitations · p. 7
  • Post-training on high-quality samples improves explanation quality but does not enhance the model’s internal knowledge of the shows.Source: Limitations · p. 7

Ethical Considerations

The paper describes how it safeguards annotators and excludes member data, then specifies the generation and judging prompts used in the work. The prompts constrain explanations to metadata-grounded comparisons and use a high-recall process to assess harmlessness.

Ethical Considerations

  • Annotators identifying harmless explanations were exposed to offensive candidate material, while member data was excluded from evaluations and RL prompts.Those prompts and evaluations use title pairs replayed from the production selection pipeline and catalog metadata.Source: Ethical Considerations · p. 7

A Prompt Specifications

A.1 The deployed generation prompt

  • The deployed generation prompt requires one conditional “Watch if you liked” phrase grounded in supplied metadata and under 110 characters.It uses title names in quotation marks and is shared identically across generators; the template preserves a production typo.Source: §A.1 · p. 10

A.2 The faithfulness judge prompt

  • The faithfulness judge checks whether concrete, central shared elements are present in both titles, rejecting invented, vague, or misplaced claims.Its reason-first scaffold verifies emphasized elements against each title’s tags and synopses before deciding.Source: §A.2 · p. 10

A.3 The harmlessness judge prompt

  • The harmlessness judge uses a high-recall scaffold that flags potentially insensitive framing and passes only when no concern is even arguably problematic.It checks concerns against metadata, including maturity ratings, and identifies the offending span on FAIL.Source: §A.3 · p. 10

B.1 Training Settings.

The training setup specifies substantial judge-decoding costs, data-split safeguards, and a reward designed to discourage degenerate predictions. The audit reports that the served faithfulness judge matches the frontier judge on recall and alignment measures while achieving higher FAIL-precision.

B.1 Training Settings.

  • Each optimization step uses 1,280 judge decodes across two endpoints, and training runs for 188 steps on 3,000 deployment-valid prompt pairs.Each of 16 prompts receives G=8 rollouts; every rollout requires ten judge decodes.Source: §B.1 · p. 10

B.2 Hyperparameters

  • Training uses LoRA adapters on 8B backbones with AdamW and bf16 on 8×A100_80GB; judge and generator GRPO share a trainer but differ in reward and dual block.Source: §B.2 · p. 11

B.3 Annotation splits

  • Judge data splits are stratified 80/10/10 by stable row hash, with no cross-split explanation leakage.Factual FAIL rows are stratified across five failure-mode tags.Source: §B.3 · p. 11

B.4 The two-reference confound

  • The study drops two-reference rows because reference count perfectly predicted PASS among 623 such rows, leaving the full loop scoped to single-reference explanations.The source had zero two-reference FAILs in validation or test, making the shortcut invisible to evaluation.Source: §B.4 · p. 11

B.5 The anti-hacking reward saddle

  • The four-tier reward assigns 0, 0.5, 2, or 4 based on parse failure, wrong label, correct PASS, or correct FAIL; both always-PASS and always-FAIL score 1.55.Neither judge run showed drift: mean batch reward rose above the saddle toward the ≈2.6 ceiling.Source: §B.5 · p. 11

C.1 Reward-design selection and the joint dual

  • Checkpoint selection maximizes all-criteria PASS subject to offensive performance staying at or above the warm-up floor, with faithfulness breaking ties.The selected checkpoint is scored once on test.Source: §C.1 · pp. 11–12

C.2 Reasoning-quality audit

  • Under the served any@8 configuration, the faithfulness judge matches the frontier judge on recall and both alignment measures, with higher FAIL-precision.Table 6 reports the held-out-test reasoning-quality audit.Source: Table 6 · p. 12

Related papers

Loading 2609.13657v1…