PaperlayerView exact revision on arXiv

Source-linked AI summary

CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Production

Yixin Nie, Lin Guan, Zhongyao Ma, Anchit Gupta, Yipin Zhou, Xiao Li, Zhengping Zhou, Raymond Zeng, Gelin Zhou, Shigan Chu, Ajay Thampi, Wancen Mu, Nathan Shuster, Ketong Wang, Lin Chen, Jason Brewer, Derek Hao Hu, Alexander McCauley, Jason Weston, Sem Park, Na Zhang, Kevin Tang

arXiv:2603.01973v1cs.CLcs.AIcs.SI

TL;DR

Social chat models must be improved for subjective engagement objectives that lack the standardized evaluation available to utility-focused assistants. CharacterFlywheel combines iterative data curation, preference modeling, model training, and production evaluation, yielding sustained engagement gains while improving steerability and safety monitoring.

  • Problem

    Socially oriented conversational AI has received less rigorous progress tracking because engagingness is subjective and lacks the objective benchmarks and reward signals available to utility-focused LLMs.

  • Method

    CharacterFlywheel iteratively uses production and internal feedback, reward modeling, data annotation, model training, and monitored evaluation to improve social chat models.

  • Results

    7 out of 8 newly deployed models improved both engagement breadth and depth over baseline, while the process also produced sustained engagement gains and a 78% reduction in instruction violations.

  • Takeaways & Limitations

    Systematic, monitored optimization can produce measurable progress for subjective engagement objectives in production-scale social LLMs.

  • Takeaways & Limitations

    User signal models were unsuitable for direct online RL because inherent biases made them susceptible to reward hacking.

Abstract

from arXiv · show

This report presents CharacterFlywheel, an iterative flywheel process for improving large language models (LLMs) in production social chat applications across Instagram, WhatsApp, and Messenger. Starting from LLaMA 3.1, we refined models across 15 generations using data from both internal and external real-user traffic. Through continuous deployments from July 2024 to April 2025, we conducted controlled 7-day A/B tests showing consistent engagement improvements: 7 of 8 newly deployed models demonstrated positive lift over the baseline, with the strongest performers achieving up to 8.8% improvement in engagement breadth and 19.4% in engagement depth. We also observed substantial gains in steerability, with instruction following increasing from 59.2% to 84.8% and instruction violations decreasing from 26.6% to 5.8%. We detail the CharacterFlywheel process which integrates data curation, reward modeling to estimate and interpolate the landscape of engagement metrics, supervised fine-tuning (SFT), reinforcement learning (RL), and both offline and online evaluation to ensure reliable progress at each optimization step. We also discuss our methods for overfitting prevention and navigating production dynamics at scale. These contributions advance the scientific rigor and understanding of LLMs in social applications serving millions of users.

1 Introduction

Socially oriented conversational AI serves millions of users but has received less systematic research than utility-focused assistants because engagingness is subjective and difficult to evaluate. CharacterFlywheel addresses this gap with iterative production-scale improvement and online validation.

  • Motivation: Social conversational AI emphasizes engaging, human-like interaction, yet its development remains comparatively opaque and lacks rigorous progress tracking.The objectives are more ambiguous and subjective than those of utility-driven systems with standardized benchmarks and verifiable reward signals.
  • Approach: CharacterFlywheel is an iterative improvement process for social chat models deployed across Instagram, Messenger, WhatsApp, and the Web.The ecosystem supports user-created and shared interactive AI characters for entertainment, social connection, and well-being.
  • Approach: The framework evaluates character steerability, safety, engagement breadth, and engagement depth while integrating data curation, reward modeling, SFT, and RL.Reward models estimate and interpolate metric landscapes using human annotations and user behavioral signals.
  • Results: 7 out of 8 newly deployed models improved both engagement breadth and depth over baseline in controlled 7-day A/B tests.The strongest performers achieved 8.8% improvement in breadth and 19.4% improvement in depth.
  • Scope: The report aims to advance rigorous study of social-domain LLMs at real-world product scale serving millions of online users.The work uses a production ecosystem as an environment for validating social chat model improvements.

2 Methodology

CharacterFlywheel treats social-chat improvement as iterative navigation of an unknown, non-differentiable engagement landscape. Its cycle combines curated traffic data, annotation, reward modeling, model training, and evaluation for successive production updates.

  • 2.1 Development Cycle: Engagement breadth and depth are the target metrics for social chat models, but their non-differentiability prevents direct optimization.The development objective is framed as navigating an unknown landscape shaped by engagement.
  • 2.1 Development Cycle: Each iteration samples data near the current model, estimates local engagingness with reward models, and updates the chat model using an estimated direction and step size.Reward-model training is called pre-herding, while chat-model updating is called herding.
  • 2.1 Development Cycle: A full development cycle consolidates post-deployment data, trains reward models, builds rejection-sampling datasets, applies SFT, DPO, and RL, and evaluates checkpoints before deployment.Candidate checkpoints are selected through the evaluation framework, with safety and privacy considerations included in development.
  • 2.2 Data Curation and Annotation: The data pipeline combines internal annotation traffic with millions of production interactions, extracting samples and signals for continuously refreshed training sets.Internal users, data vendors, UX researchers, and content specialists provide targeted feedback alongside production traffic.
  • 2.2.1 Curation Pipeline: Curation filters data for privacy and safety, removes embedding-space redundancy through diversity sampling, and applies multidimensional constraints and stratified sampling.The retained subset is intended to represent target traffic distributions while steering models toward optimization goals.
  • 2.2.2 Annotation: Annotators use static or interactive conversations to rate quality, rank engagingness, rewrite poor responses, and identify false refusals and templated responses.Steerability annotation separately challenges models to follow highlighted character traits or instructions and records violations or rewrites.

2.3 Reward Models

CharacterFlywheel uses surrogate reward models to make non-differentiable engagement objectives optimizable while retaining preference-based quality control. Pointwise and pairwise preference models, supplemented by user-signal models, support training and evaluation.

  • Surrogate rewards: Surrogate models convert non-differentiable engagement metrics into differentiable rewards for optimizing generated responses.The composite signal combines a preference model with auxiliary models trained to predict user behaviors from production interaction data.
  • Reward composition: Preference models provide the primary reward, while user-signal models add auxiliary rewards that balance objectives and mitigate overfitting.Preference data is more controllable, whereas online user-signal data remains noisy despite filtering.
  • Preference models: Pointwise models score responses independently, whereas pairwise models jointly compare two responses to classify which is superior.Both paradigms provide complementary preference signals.
  • Evaluation: Pointwise scores guide reinforcement learning, while pointwise and pairwise win-rates jointly support evaluation and model selection.Using both evaluation paradigms helps mitigate reward hacking.
  • User signals: User-signal models predict behavioral events such as explicit feedback, emoji reactions, and response regeneration from conversation context and model responses.Only p(continue) and p(thumb up) were ultimately used for rejection-sampling data selection because they performed consistently and reliably.

2.4 Fine-Tuning and Alignment

CharacterFlywheel combines rejection sampling, supervised fine-tuning, DPO, and online reinforcement learning in a repeatedly refreshed training loop. The process uses recent traffic outputs and artifact monitoring to optimize engagement while preserving broader response quality.

  • Rejection sampling: Rejection sampling reconstructs a training dataset from user-traffic prompts and candidate-model responses after each model update.The pipeline selects a suitable candidate, generates multiple responses, scores them with a reward model, and retains responses meeting a threshold.
  • SFT and DPO: The initial training stage applies SFT to Llama 3.1 70B using refreshed internal chats, user traffic, safety, capability, and tool-calling data.A small DPO stage then patches urgent safety and style issues while retaining Llama 3.1 data for community-benchmark performance.
  • Reinforcement learning: Online RL follows SFT and DPO, using online DPO and a GRPO variant with importance-sampling corrections for distributed training.The process later switched from online DPO to GRPO because GRPO yielded better engagement in A/B tests.
  • Optimization setup: Single-turn optimization uses fixed partial conversation histories and updates only final responses, reducing simulation complexity but potentially weakening on-policy behavior.Near-policy prompts and tight iteration loops keep this semi-online setup effective for engagement optimization.
  • Overfitting prevention: Artifact mitigation tracks response length, formatting, emoji use, and other stylistic features across preference, rejection-sampling, and checkpoint data.The goal is to prevent superficial artifacts from becoming correlated with reward or selection signals and being shallowly optimized.

2.5 Evaluation and Feedback Loop

The evaluation framework combines offline quality checks with randomized production A/B tests to select improved models and monitor engagement. It measures benchmarks, human preferences, reward-model win-rates, custom response characteristics, and breadth and depth of engagement.

  • Offline evaluation: Offline evaluation checks improvement over the previous model while safeguarding response quality across five evaluation areas.The framework includes community benchmarks, human comparison, reward-model win-rate, custom production metrics, and additional quality checks.
  • Offline evaluation: Community benchmarks test factual and utility-seeking performance, although the stated goal is robust performance rather than state-of-the-art ranking results.The benchmark suite includes standard LLM evaluations listed in Table 3.
  • Offline evaluation: Side-by-side human comparisons present responses from both models while randomly selecting which response continues the conversation to reduce history bias.This evaluates whether the new model outperforms the previous version under a fair comparison procedure.
  • Offline evaluation: Reward-model win-rate measures how often evaluators prefer the new response over the old response on identical prompts, using pointwise and pairwise models.Prompt sets are periodically updated through the evaluation pipeline.
  • Online evaluation: Production A/B tests randomly assign eligible users to updated-model or baseline arms, typically allocating 10% of traffic to each.The tests evaluate engagement improvements for every model update or promising training-recipe change.
  • Engagement metrics: Engagement breadth measures whether units engage, while engagement depth measures aggregate engagement conditional on positive engagement.Percentage lift compares test and control means, with Fieller confidence intervals used for ratio estimators.

2.6 Safety and Privacy

CharacterFlywheel applies layered safety and privacy controls throughout model development and deployment. Fail-closed enforcement rejects unsafe outputs or escalates ambiguous cases, while privacy checks remove entity-identifiable information from downstream processing.

  • Safety framework: The safety framework inherits Llama 3.1 standards and prioritizes minimizing safety violations and false refusals.Automatic and manual evaluations are applied under critical safety rules and risk standards.
  • Layered evaluation: Safety classifiers operate during character generation, character updates, model updates, online interaction, and traffic sampling.Manual review is triggered for uncertain or reported characters.
  • Fail-closed design: Fail-closed design automatically rejects creations, prompts, responses, or model updates that fail required safety rules.Ambiguous cases are escalated to manual review before a final decision.
  • Privacy: Privacy checks combine model-based and rule-based screening for high-risk identifiers during upstream data curation.The stated purpose is to prevent entity-identifiable information from entering annotation and training workflows.

2.7 Image Generation

CharacterFlywheel adds image generation to social chat through explicit user requests and an implicit mechanism that lets the LLM decide when visual content would enrich a conversation. The system treats image generation as an agentic tool-calling task and uses multi-review annotation for subjective cases.

  • Image generation supports CharacterFlywheel’s social-value and engagement goals across explicit and implicit conversational scenarios.The feature is intended to enhance conversational engagement beyond standard utility-focused image generation.
  • Explicit generation occurs when the user directly prompts the model to create an image.
  • Implicit generation lets the LLM autonomously trigger image creation when visual content may enrich the conversation.
  • The LLM learns when to trigger generation and which parameters to provide, including a prompt for a downstream text-to-image model.
  • Multi-review annotation retains high-consensus data for the inherently subjective implicit-generation task.Annotators are required to agree on key dimensions of appropriateness before data is retained.

3 Results

Across iterative model development, CharacterFlywheel improved engagement-related, steerability, refusal, formatting, tone, and quality metrics while sometimes experiencing temporary regressions. The results also show that reward-model overfitting can produce misleading offline gains, making monitoring and iterative correction central to reliable progress.

  • Pre-launch development: 37.4% to 46.2%: human evaluations against GPT-4o improved from V3 to V7 during pre-launch development.Human and reward-model win rates against previous versions consistently exceeded the 50% neutral threshold.
  • Engagement: +8.8% breadth and +11.2% depth: V14 delivered notable engagement lifts, while V12 reached only +0.05% breadth and -2.9% depth.V11 also achieved +4.47% breadth and +18.2% depth, illustrating variation across deployed versions.
  • Overfitting: 70.7% user-traffic RM win rate versus 43.7% internal-traffic RM win rate marked V12’s overfitting warning.The user-traffic result exceeded the typical 50-65% range for successful versions, while the internal result dropped below it.
  • Engagement: Cumulative engagement increased overall despite V12’s setback, although the gains were not entirely attributable to model updates.The reported conclusion links progress to integrating human feedback, reward modeling, online A/B testing, and safeguards against overfitting.
  • General capabilities: 79.5% MMLU and 84.8% IFEval: V7 remained competitive while being optimized for social engagement rather than pure utility.These scores represented modest degradation from the Llama 3 70B baseline values of 83.6% and 87.5%, respectively.
  • General capabilities: 84.8% IFEval and 93.1% ARC Challenge: benchmark performance improved or remained stable across pre-launch versions.IFEval rose from approximately 75% in V2 to 84.8% in V7, supporting retained versatility across production query types.
  • Character steerability: 26.6% to 5.8%: instruction violations fell by 78% relative from V2 to V8.The metric was evaluated on realistic, mildly adversarial interactive sessions involving internal users and domain experts.
  • Character steerability: Engagement-focused annotation also labeled instruction violations and edited responses, showing that engagement and instruction-following need not conflict.

4 Conclusion

CharacterFlywheel shows that systematic, production-scale optimization can improve subjective social-chat objectives while maintaining safeguards and steerability. Across 15 iterations, the framework accumulated engagement gains and exposed remaining challenges in reward hacking, multi-turn optimization, and preference-model generalization.

  • 15 production-scale iterations serving millions of users demonstrated sustained engagement improvements while steerability improved by 78% fewer instruction violations.The process combined systematic optimization with monitoring and safeguards against overfitting.
  • Reward-model overfitting caused engagement to degrade when RM win rates reached 70.7%, motivating safeguards that kept subsequent win rates below 65%.The V12 failure established an empirical boundary for later iterations.
  • Consistent annotation guidelines supported accumulated engagement gains in the iterative preference-modeling approach.
  • The feedback loop resolved temporary degradations such as false refusals, preachy tone, and emoji overuse in subsequent iterations.
  • The report identifies reward-hacking detection, multi-turn optimization, and preference-model generalization as remaining challenges.
  • Reward-model monitoring, preference-data quality, and multi-metric evaluation provide practical guidance for engagement-focused AI systems where single-metric optimization is insufficient.The report frames these techniques as a template for developing and monitoring complex conversational AI systems.

A Evaluation Performance Across Versions

Table 10 reports evaluation results across 15 CharacterFlywheel model versions, including human comparisons, reward-model win rates, and language-match rates. The reported metrics vary across versions and evaluation types, with user and internal reward-model scores listed separately.

  • 15 CharacterFlywheel model versions are covered by the reported evaluation results.
  • Human Win Rate vs GPT-4o (%) increases from 38.5 to 46.2 across the listed early model-version results.
  • Human Win Rate vs Prev. Model (%) reaches 52.5 among the listed early model-version results.
  • Pointwise RM Win Rate (User) (%) peaks at 70.7 in the listed model-version results.
  • Language Match Rates reach 0.856 for Hindi, 0.866 for Romanized Hindi, and 0.980 for Spanish in the listed results.

B Confidence Intervals for Ratio Metrics

The ratio-metric analysis constructs confidence intervals for the test-to-control mean ratio using randomized-group assumptions and Fieller’s theorem. Percentage lift is then reported from the interval, with significance determined by whether the null lift is included.

  • The test and control mean estimators are assumed independent because of randomized assignment and approximately normally distributed.
  • σ2_C denotes the estimated variance of the control-group mean, defined as the squared standard error.
  • User-level aggregate variances account implicitly for within-user temporal autocorrelation over the readout window.Examples include average daily participation and total sends.
  • The target is a 100(1 −α)% confidence interval for the true ratio ρ = µ_T/µ_C, derived using Fieller’s theorem.
  • Solving the Fieller quadratic inequality produces closed-form ratio bounds using the standard-normal critical value z.For 95% confidence, z is approximately 1.96.
  • Percentage lift is reported from the confidence interval, and significance at level α requires the interval to exclude the null lift of 0%.
Loading 2603.01973v1…