Source-linked AI summary
Not All Matches Are Equally Valuable: An Online Experiment of Retention-Focused Recommendation in a Job-Matching Platform
Tatsuya Ute, Chiaki Ichimura, Yuta Saito
TL;DR
The paper asks whether match-focused recommendation in a two-sided job marketplace should account for users’ retention risk, given that low match counts are associated with higher churn. It formulates a retention-aware ranking strategy, boosts churn-risk users, and evaluates it in an online experiment. User churn was directionally lower but statistically inconclusive, while company-side churn showed no deterioration.
Problem
Match-maximizing recommendation may misalign with retention because users with few matches face higher churn risk, while extra matches for already successful users have limited marginal retention value.
Method
The paper applies a retention-focused post-processing intervention that boosts baseline ranking scores for users judged at risk of churn.
Results
User churn was directionally lower under the intervention but statistically inconclusive at conventional levels, with no evidence of deterioration in company-side churn.
Takeaways & Limitations
The case study suggests optimizing marketplace recommenders for downstream retention rather than only immediate match volume.
Takeaways & Limitations
The experiment did not establish whether increased exposure, scouts, or matches mediated the estimated churn effect.
Abstract
from arXiv · showhide
Recommender systems in two-sided matching platforms are commonly optimized for immediate engagement signals such as click-through rate, reply rate, or the total number of successful matches. However, in real-world marketplaces, maximizing matches alone may be misaligned with business goals such as user churn rate and platform revenue, especially when users with fewer matches are at substantially higher risk of churn. In this paper, we study a real job matching platform and show that users with very few recent matches are indeed much more likely to leave the platform, while additional matches for already successful users provide limited marginal value for retention. Motivated by this empirical finding, we formulate a retention-aware recommendation problem and implement a simple post-processing method that adjusts the baseline match-focused ranking to prevent user churn. Specifically, the implemented method gives a score boost to churn-risk users with the goal of increasing their likelihood of obtaining matches and improving retention. We evaluate this practical approach in an online experiment on a real job-matching platform. The treatment group showed directionally lower user churn than the control group, although the estimated effect was not statistically significant at conventional levels, while company-side churn showed no evidence of deterioration. To our knowledge, this is among the first online experimental studies to investigate retention-focused recommendation in a real reciprocal job-matching platform.
1. Introduction
The paper argues that match-maximizing recommendation can conflict with retention because users with few matches face greater churn risk, while extra matches for already successful users add limited retention value. It therefore proposes boosting exposure for churn-risk users and tests this intervention online.
- Maximizing aggregate matches alone may misalign recommendation with platform business objectives such as user retention.
- Users with very few recent matches are substantially more likely to churn, while additional matches for already successful users have limited marginal retention value.
- The proposed post-processing method boosts scores for users judged at risk of churn while leaving other users’ baseline rankings unchanged.
- The online treatment showed directionally lower user churn, although the estimate was statistically inconclusive, with no evidence of deterioration in company-side churn.
- The study presents an early online investigation of retention-focused recommendation in a real reciprocal job-matching platform.
2. Related Work
The related work connects this study to reciprocal recommendation, long-term retention-aware recommendation, and exposure allocation. The paper distinguishes its empirically targeted retention intervention from approaches centered on reciprocal modeling or formal fairness constraints.
- Reciprocal recommendation: Reciprocal recommendation models interactions in which successful outcomes depend on the preferences and behaviors of both sides of a matching market.
- Reciprocal recommendation: The platform setting involves recommendations to companies, company scouting decisions, user responses, and operationally defined matches.
- Long-term and retention-aware recommendation: Retention-aware recommendation extends short-term engagement optimization by considering delayed outcomes such as continued participation and long-term user value.
- Exposure allocation and fairness: Exposure-allocation methods counteract concentration from relevance-only ranking by considering exposure, utility, or competitive disadvantage across marketplace sides.
- Exposure allocation and fairness: This paper differs from fairness-oriented approaches by allocating additional exposure according to empirically defined retention risk rather than optimizing a formal exposure-parity constraint.
3. Problem Formulation
The paper formulates matching as ranking users by expected match probability, then reframes the objective around reducing users below a churn-risk match threshold. This creates a trade-off between prioritizing at-risk users and maximizing immediate company-side match probability.
- The platform estimates match probabilities for user–company pairs in a two-sided job-matching setting.
- Match probability is defined as the product of the company’s scouting probability and the user’s conditional reply probability.
- The standard policy ranks users by descending match probability and optimizes the expected total number of matches.
- Users with match counts below a threshold τ are considered at risk because low recent match volume is associated with substantially higher churn and limited marginal retention gains above the successful range.
- The retention-aware objective seeks to reduce the number of users whose match volume remains below τ rather than treating every additional expected match equally.
- Optimizing this objective requires prioritizing churn-risk users even when they do not have the highest immediate match probability for companies.
4. Implemented Retention-Focused Strategy
The paper implements a lightweight post-processing intervention that boosts rankings for churn-risk users while preserving match-focused recommendations for others. The method targets users with reasonable matching prospects and limits excessive exposure concentration.
- The intervention adds a retention-focused post-processing step to the existing match-probability ranking rather than replacing the recommender.This design avoids retraining the underlying model or directly optimizing a complex retention objective.
- Users with fewer than three recent scouts are treated as the platform-specific high-risk churn region.The cutoff comes from predictive historical analysis rather than a formal breakpoint test and is not presented as a universal threshold.
- For churn-risk users, the method increases exposure by applying a score boost to eligible user–company pairs.The boost is applied only when the baseline score meets score_cutoff, because boosting unlikely matches has little value.
- After score modification, candidates are ranked in descending order of the clipped score s′(u, c).The base boost factor and diversity decay factor can be tuned offline for a predicted retention objective.
- The boost decays with a user’s overall exposure across companies, reducing concentration among users already frequently visible in rankings.The decay uses the number of appearances in top-ranked positions across company rankings.
5. Online Experiment
The authors evaluated the retention-focused ranking in a randomized online experiment on Wantedly Visit, comparing it with a match-maximizing baseline. User-churn estimates favored treatment directionally but were statistically inconclusive, while company-side churn showed no deterioration.
- 5.1. Experiment Design: The online experiment compared a retention-focused treatment policy with a conventional match-maximization baseline on Wantedly Visit.The treatment boosted churn-risk users and applied a smaller boost to users already appearing frequently in recruiter rankings.
- 5.1. Experiment Design: Approximately 3,000 recruiters per group were randomized between November 10 and December 8, 2025.The analysis included users active during the 28-day pre-period who received at least one test-period impression.
- 5.1. Experiment Design: The primary outcome was a binary user-churn flag measuring whether pre-period-active users became inactive during the test period.Company-side churn and a model-based company-churn score were examined as guardrail metrics.
- 5.2. Results: 0.957 was the estimated user-churn odds ratio in the primary overlapping specification, indicating lower estimated treatment-group churn odds than control.The mutually exclusive robustness sample produced an odds ratio of 0.948.
- 5.2. Results: Neither user-churn estimate was statistically significant at conventional levels, so the evidence was directionally positive but statistically inconclusive.The point estimates were directionally consistent across specifications.
- 5.2. Results: Exploratory subgroup estimates were more favorable among users with fewer prior scouts or impressions, but they were not statistically significant.These patterns should not be interpreted as evidence of treatment-effect heterogeneity.
- 5.2. Results: 5.4% was the adjusted point estimate for total matches in treatment relative to control, but the difference was not statistically significant.The intervention was not designed to increase total matches, so this estimate was treated as exploratory.
- 5.2. Results: No evidence indicated deterioration in company-side churn: the treatment–control coefficient difference was +0.006 for actual churn and −0.007 for the churn score.The corresponding p-values were 0.77 and 0.34, respectively.
6. Limitations
The study’s evidence is constrained by its single-platform, short-duration setting and by design and inference limitations. It also does not establish mediation, superiority to alternative allocation methods, or fairness acceptability.
- The experiment covered one job-matching platform for four weeks, limiting generalization to other marketplaces, interfaces, and longer-term retention outcomes.
- Recruiter-level randomization with cross-arm user exposure limits a clean user-level causal interpretation.
- Primary churn effects were not statistically distinguishable from the null at p<0.05, and subgroup analyses were insufficiently powered to establish treatment-effect heterogeneity.
- The study did not verify whether changes in exposure, scouts, or matches mediated the estimated churn effect.
- Churn-risk thresholds and score boosts came from offline analyses rather than a fully specified learned retention objective, while the observational Figure 2 does not establish causality.
- Because the intervention was compared only with the production baseline, it does not establish superiority over congestion-aware or fairness-aware alternatives.
- Reallocating exposure may trade off retention against candidate-side exposure and company-side utility, and the study did not directly evaluate perceived fairness, transparency, or disclosure reactions.
7. Conclusion and Future Directions
The paper argues that match-focused ranking can misalign with retention goals and presents a practical score-boosting intervention for churn-risk users. In the online experiment, user churn was directionally lower but statistically inconclusive, motivating broader downstream-objective optimization and future company-retention work.
- The platform’s match-maximizing objective was misaligned with retention because users with too few matches were more likely to churn.
- The paper formulated a retention-aware ranking problem and adjusted baseline ranking for churn-risk users through a small, controlled score boost.
- User churn was directionally lower in the online experiment, although the estimated effect was statistically inconclusive at conventional levels.
- The case study suggests marketplace recommenders should optimize for downstream objectives rather than only immediate proxies such as match counts.
- Future work will extend the framework to company retention and broaden churn-risk identification with additional signals.
Declaration on Generative AI
The authors disclose using several generative AI systems for language and drafting assistance, then reviewing and editing the resulting content.
- Claude Opus 4.8, Claude Opus 5, GPT-5.5, and GPT-5.6 Sol were used for grammar checking, paraphrasing, rewording, and drafting.
- The authors state that they reviewed and edited the material and retain full responsibility for the publication’s content.