Source-linked AI summary
FORTE: A Production Framework for Feature Evaluation and Deprecation in Industrial Recommender Systems
Vishnu Arun, Adam Obeng, Tianyou Wang, Zilin Ren, Mukuntha Narayanan Sundararaman, Han Wang, Shivin Thukral, Zhixiang Wang, Saurabh Vishwas Joshi, Darren Reger
TL;DR
Feature removal must preserve metrics while gaining efficiency, and evaluating candidate removals competes for limited A/B capacity. FORTE combines temporal feature screening, grouped perturbation, and lineage tracking in a production workflow. Its grouped-perturbation estimates closely agree in total, while marginal importance has limits for correlated or interaction-dependent features.
Problem
Feature removal must preserve metrics while adding features is justified by gains, and validating removals requires retraining plus scarce online A/B tests.
Source: §1.1 · p. 1Method
FORTE combines temporal feature screening, grouped perturbation, and lineage tracking in a production workflow.
Source: §4.1 · p. 3, §6.2 · p. 7, §8 · p. 9Results
The order-averaged grouped-perturbation total was 3.079%, versus 3.084% for the fixed path, with a standard error of 0.008%.
Source: §6.2 · p. 7Takeaways & Limitations
FORTE combines scalable evaluation, validation, and lineage tracking to support production feature deprecation and infrastructure savings.
Source: §8 · p. 9Marginal permutation can misstate importance for correlated features, and first-order importance can miss features that matter only through interactions.
Source: §8.1 · p. 9
Abstract
from arXiv · showhide
Production recommender systems have feature-lifecycle asymmetry: adding new features must produce gains on some metric, while feature removal must not regress any metric. In practice, feature deprecation is hard to prioritize as a feature can fail to be removed if it interacts with other features, its importance changes daily, it cannot be evaluated before production models change or data drift occurs, it is in contention for A/B test space with other launches, it causes regressions in an online A/B test, or it is consumed by multiple models that need to coordinate its deprecation. We present FORTE, a model-agnostic production framework for discovering, validating, and removing features. Its efficient perturbation implementation allows daily feature evaluation. To account for interactions, it evaluates candidate feature sets constructed from all the low-rank features through grouped ablation and its offline results are sufficiently predictive of online A/B tests to allow deprecations to proceed to online testing with confidence. Lineage tracing identifies all downstream consumer models to realize the full savings of removing a feature from the training set. We validate FORTE over 2 large-scale recommender systems and find that it identifies deprecable feature sets which do not cause online regressions, while the impact on efficiency depends on model characteristics. On a search-relevance model, FORTE resulted in 3 nested candidate sets of 32, 48, and 59 features. Online guardrails rejected the largest treatment. The 48-feature set was selected for deprecation, removing 43.3% of the model's features. On a large ranking model, FORTE identified 8 candidates for removal. Retraining and A/B testing confirmed no regression. Its launch reduced the trainable parameters by 29.9%, improved serving throughput by 19.8-25.5%, and reduced annual training costs by hundreds of thousands of dollars.
1 Introduction
Feature removal must preserve model and user metrics while delivering efficiency gains, making deprecation harder to justify and validate than adding features. FORTE addresses this with a workflow for identifying, validating, and removing features across their downstream lineage.
1.1 Feature-lifecycle asymmetry
- Feature removal must preserve user metrics while adding features is justified by gains, and each removal requires retraining and an online A/B test.A/B capacity is limited and typically prioritizes high-impact treatments, while a model with d features has 2^d−1 nonempty candidate removal sets.Source: §1.1 · p. 1
1.2 Deprecation in practice
- Deprecation removes a feature from an already deployed model; its efficiency benefits extend to training and evaluation data only after no model references it.The industry cycle proceeds through proposal, retraining, offline evaluation, online A/B testing, metric validation, audit, and launch, with costs increasing as resources constrain.Source: §1.2 · p. 2
- Industry deprecation can stall because of feature interactions, data or model drift, A/B bandwidth contention, online regressions, and multi-model coordination.These failure modes can prevent the cycle from completing or cause offline results to diverge from online outcomes.Source: §1.2 · p. 2
1.3 FORTE overview; 1.4 Contributions
- FORTE combines daily perturbation-based importance, candidate-group evaluation, online validation, and lineage tracing in an end-to-end workflow for feature removal.Its efficient implementation supports daily individual-feature experiments, and the paper presents case studies from Search Relevance and Pin Ranker.Source: §1.3 · p. 2, §1.4 · p. 2
2 Related Work
Prior work offers several ways to estimate model reliance, but production recommender settings bring dependence, scale, and model-access constraints. FORTE uses global perturbation-based importance within those operational constraints.
Permutation importance
- Permutation importance measures model reliance by testing how much model quality changes when a feature’s signal is destroyed, and existing implementations evaluate features sequentially.Breiman introduced it for Random Forests; Fisher, Rudin, and Dominici generalized it to Model Class Reliance.Source: §2 · p. 2
Feature dependence
- Marginal permutation can distort importance for correlated predictors and create out-of-distribution inputs; conditional PFI addresses dependence but requires estimating conditional structure.FORTE uses fixed nulls for ablation and batch-shuffled values as empirical marginal replacements for permutation, then validates removals online.Source: §2 · p. 2
Scalability and operational fit
- Several attribution alternatives conflict with daily, large-scale production use: retraining methods require repeated fits, Shapley-style methods can be costly, and gradient methods need model internals.TreeSHAP [15] is limited to tree ensembles, while FORTE combines pooled inference, type-aware perturbation, temporal tracking, and online validation.Source: §2 · p. 2
3 Problem Formulation
FORTE compares a frozen model on intact examples with the same model after a feature group is perturbed, reporting a signed normalized metric change. Its objective is to compute temporal importance, form and validate removal candidates, and trace feature lineage.
Offline perturbation
- An offline perturbation compares a frozen model on intact examples with that same model after feature group G is permuted or ablated.The baseline and perturbed conditions yield metrics M0 and MG for the signed normalized change.Source: §3 · pp. 2–3
Feature-importance score
- A negative feature-importance score means perturbation degraded model quality and indicates reliance; scores near zero or above mark deprecation candidates.The sign is adjusted for whether higher or lower metric values are better, aligning its orientation with online A/B results.Source: §3 · p. 3
Deprecation objective
- FORTE aims to compute daily importance for every feature, combine temporal scores into ranked candidates, validate candidates by retraining and A/B testing, and track complete feature lineage.The deprecation funnel narrows candidate sets through validation and deprecation, with each stage having a bottleneck.Source: §3 · p. 3, Figure 1 · p. 3
4 System Design and Methodology
FORTE orders feature-deprecation work around its resource bottleneck: scalable screening comes before retraining and scarce online traffic. It combines temporal candidate selection, joint perturbation, online validation, lineage-aware blocking, and pooled execution.
4 System Design and Methodology
- The deprecation funnel screens the full feature set in amortized-inference stages before candidates reach stages limited by training capacity or online traffic.Online A/B traffic is the bottleneck, so the stages are ordered to reserve that resource for a narrowed candidate set.Source: §4 · p. 3
4.1 Temporal feature importance
- Temporal candidates must have low mean importance, stable low ranking, and a non-positive slope across configured windows and required output heads.The conservative, team-configured gate also considers feature costs and generates nested candidate sets; it is a nomination signal, not a rigorous guarantee of identifying all noncorrelated candidates.Source: §4.1 · p. 3
4.2 Grouped perturbation
- Grouped perturbation tests nested candidate sets jointly, capturing interactions and dependence that individual permutation can miss when correlated candidates substitute for one another.The team first filters for metric neutrality, then retains up to N sets ranked by potential savings; N and set sizes reflect cost priorities, retraining capacity, and A/B traffic.Source: §4.2 · pp. 3–4
4.3 Retraining, online validation, launch review
- Candidate sets move to online testing only after retraining shows neutral metrics; A/B arms must satisfy online metric neutrality and user-behavior guardrails before audit and launch review.Each arm is benchmarked against production control over a test lasting multiple weeks.Source: §4.3 · p. 4
4.4 Feature lineage and deprecation
- A live feature–model mapping identifies when a removed feature has no remaining consumers, after which staged manual blocking prevents it from being reintroduced or read, written, or logged.The blocking process progresses through dataloader, dataset, and ingestion blocks before removal from logging.Source: §4.4 · p. 4
4.5 Dataset Virtualization and Pooled Execution
- FORTE stores an ablated or permuted in-memory copy of the batched dataset so all experiments resolve against one materialization in a distributed inference pass.Each experiment selects feature columns from original or perturbed dictionaries; shared model loading, data I/O, actor startup, and GPU allocation amortize across experiments.Source: §4.5 · p. 4, Figure 3 · p. 4
5 Practical Considerations for Production Deployment
FORTE supports single-permutation feature ranking at production-scale sample sizes while distinguishing AUC theory from empirical PR-AUC evidence. Its production evaluation also relies on feature-type-aware perturbations, current-model tracking, and randomized data arrival.
5.1 Single-Permutation Estimation at Scale
- For fixed data and frozen row-wise inference, AUC permutation variance is O(1/n); averaging R independent draws reduces variance by 1/R, while larger samples reduce single-draw variance.The 1/R identities apply to independent permutation draws; the O(1/n) reference rate depends on row-wise sensitivity and balanced positive and negative counts.Source: §5.1 · p. 5
- For PR-AUC, the proposition establishes no 1/n rate; empirically, increasing the sample from 15k to 150k reduced median per-feature standard deviation by 2.58–3.16×.The comparison is descriptive: samples may differ in composition and class balance, and the proposition does not establish a PR-AUC sample-size rate.Source: §5.1 · p. 5
- At 150k examples, one permutation matched nine-draw rankings closely (0.907–0.959 median correlation; 92–100% top-20 overlap), and 150k×1 beat 15k×10 across all metric–head pairs.The matched-budget comparison found stronger agreement with the 1.5M×1 reference at 150k×1 for all four metric–head combinations.Source: §5.1 · pp. 5–6, Figure 4 · p. 6
5.2 Engineering for Production-Scale Evaluation
- Custom feature-type perturbation operates on the batched representation; a Feature Registry defines type-appropriate nulls consistently across training and evaluation.The registry covers supported features across the ML stack, while batched processing lets one data materialization serve every experiment.Source: §5.2.1 · p. 6
5.2.2 Production model sensor.
- To track daily importance across the model fleet, FORTE resolves each model from serving traffic and follows the latest production checkpoint.Source: §5.2.2 · p. 6
5.2.3 Randomization.
- A two-stage shuffle randomizes rows within output files and dispatches files in randomized order; international data avoids oversampling any single market.This decouples inference-batch arrival order from source-table layout without a full distributed shuffle.Source: §5.2.3 · p. 6
6 Evaluation
The evaluation finds that pooled perturbation sharply amortizes experiment cost, while grouped validation exposes interactions and unstable per-feature attributions that individual screening misses.
6.1 Scaling behavior
- 54× less elapsed time was required for 300 experiments on the 3M-row dataset; experiment-count runtime grew as N^0.20, N^0.23, and N^0.27 for the small, medium, and large sets.For the 30k-row set, a 1000× increase in experiments cost 4.5× the wall-clock time.Source: §6.1 · pp. 6–7
- At 3M rows, each additional experiment cost 41 s versus 45 minutes for a separate pass, because model loading, data I/O, and actor startup are paid once per run.Forward passes and final writes grow with experiment count, but runtime remains far below linear.Source: §6.1 · p. 7
6.2 Grouped perturbation validation
- 38.6% of nonzero fixed-order marginals were negative; the grouped AUC effect was 3.084% versus 6.554% summed individual deltas, peaking at 3.899% after 64 removals.This shows why the joint gate is necessary even when feature groups pass individually.Source: §6.2 · p. 7
- The order-averaged total was 3.079% versus the fixed path’s 3.084% endpoint, with 0.008% standard error, while per-group decomposition remained poorly determined.Attribution error cancels in the sum, making the composed-set estimate an order of magnitude more precise than a typical member.Source: §6.2 · p. 7
- Full-coalition effects differed by head: +3.079% on Closeup AUC versus −3.769% on Clickthrough PR-AUC, and duplicate-slot feature signals received opposing attributions.The same 189 groups could be removable on some metrics but load-bearing on another.Source: §6.2 · p. 7
- Closeup retained only 47–48% of summed individual deltas, Clickthrough AUC reversed sign, and mean Closeup AUC marginal fell from 0.0327% to 0.0030% across the order positions.The decline was −0.0297% with 95% CI [−0.0358, −0.0236], corroborating the cumulative path’s peak and decline.Source: §6.2 · p. 7
- Order averaging broadly preserved the top of the individual-feature ranking but not its zero threshold; fewer than one-third of candidates stayed positive in every order.Agreement in the mean therefore did not imply stable per-feature effects.Source: §6.2 · p. 7
- Per-feature estimates support screening and ranking, but grouped perturbation measures the composed candidate set directly; only 53 of 176 candidates kept their marginal direction across all 20 orders.A 10-order run reproduced the 20-order ranking with 0.975 rank correlation and 95% top-20 overlap.Source: §6.2 · p. 7
7 Industrial Applications
FORTE was evaluated on Search Relevance and Pin Ranker models, with online testing determining which feature removals to launch. The cases show that model architecture affects the efficiency gains, and lineage cleanup can add savings beyond a single-model launch.
7 Industrial Applications
- Search Relevance deprecated a larger share of its features but achieved a smaller parameter reduction because its cross-encoder backbone has orders of magnitude more parameters.The two applications differ in architecture and resulting efficiency impact.Source: §7 · p. 8
7.1 Search Relevance
- All three Search Relevance candidate sets cleared offline gates, but online guardrails rejected the 59-feature arm; the 48-feature set was selected.The 48-feature arm reduced trainable parameters by 2.3% and increased serving throughput by 2%.Source: Table 1 · p. 8, §7.1 · pp. 8–9
- Search Relevance used US and non-US metric-layer segments, whose neutral results indicated no localized signal for non-English queries in those segments.The segmentation reused the global inference and perturbation pipeline.Source: §7.1 · pp. 8–9
7.2 Pin Ranker
- The 8-feature Pin Ranker removal cleared every gate and showed no statistically significant regression in clickthroughs, closeups, or repins; it was launched.The 8-feature set superseded a neutral 2-feature subset.Source: §7.2 · p. 9
- The launched Pin Ranker removal reduced annual infrastructure costs by hundreds of thousands of dollars, with lineage-wide deprecation adding dataset-storage and training-loop savings beyond a single-model launch.After owner review, the eight retired features’ dataset columns and feature-materialization jobs were removed.Source: §7.2 · p. 9
- FORTE’s feature-importance ranking also helped validate an architectural change by showing increased reliance on non-historical features under the ranker’s residual variant.The reported shifts were consistent with the design objective of reducing over-reliance on engagement-biased signals.Source: §7.2 · p. 9
8 Discussion and Conclusion
FORTE combines scalable daily feature evaluation with validation and lineage tracking to support production feature deprecation and infrastructure savings. Its choices prioritize scalable decisions over full causal attribution, with limits on importance estimates, shuffling, temporal comparisons, and generalization.
8 Discussion and Conclusion
- FORTE combines scalable daily feature evaluation with validation and lineage tracking to support production feature deprecation and infrastructure savings.Source: §8 · p. 9
8.1 Limitations
- Marginal permutation can inflate or deflate importance for correlated features, while first-order tests of individual features or manually configured groups can miss interaction-only importance.Source: §8.1 · p. 9
- Temporal importance trends are scoped to a single checkpoint because each analysis window holds the model fixed and new windows refresh to the current production checkpoint.Source: §8.1 · p. 9
- The two-stage downsample is an approximate shuffle, trading exactness for runtime rather than claiming equivalence to a uniform shuffle.Source: §8.1 · p. 10
- The default single permutation per batch is supported by stability measurements on two heads of one production model, which do not establish the choice for every model or metric.Batch size, dataset size, and the repeat count also affect resulting variance.Source: §8.1 · p. 10
- The two Pinterest examples do not establish architecture-independent efficiency findings; the workflow responds with conservative candidate selection, grouped perturbation, retraining, and online testing.Future work includes automatic interaction discovery and conditional importance estimates for correlated features.Source: §8.1 · p. 10
Ethical Considerations
Aggregate-neutral feature removal can still harm subpopulations, and coarse geographic segments may conceal within-segment effects. The paper also notes that energy reductions are expected but were not measured.
Ethical Considerations
- US versus non-US segmentation cannot resolve effects across more than 45 languages and may conceal harm within either segment.Deployments should monitor at a granularity suited to the population served; this mitigation is not a fairness result.Source: Ethical Considerations · p. 10
- Making removal cheap and routine increases the throughput of decisions whose aggregate-metric failure mode can hide distributional harm.Source: Ethical Considerations · p. 10
- Energy use is expected to fall with feature removal, but power, accelerator-hours, and emissions were not measured, so no quantitative energy claim is made.Source: Ethical Considerations · p. 10