Source-linked AI summary
The Missing Piece in Complex Analytics: Low Latency, Scalable Model Management and Serving with Velox
Daniel Crankshaw, Peter Bailis, Joseph E. Gonzalez, Haoyuan Li, Zhao Zhang, Michael J. Franklin, Ali Ghodsi, Michael I. Jordan
TL;DR
Large-scale analytics systems have focused on training complex models, while deployment, serving, and management at scale remain a gap. Velox adds model serving and maintenance to BDAS, and its initial prototype performed basic serving and update tasks while online updates closely recovered offline retraining accuracy.
Problem
Large-scale analytics systems focus on training complex models but leave deployment, serving, and model management at scale largely unaddressed.
Source: §1 · p. 1Method
Velox adds low-latency model serving and online and offline maintenance to BDAS, including monitoring model performance to trigger offline retraining.
Source: §1 · pp. 1–2, §4.3 · p. 4Results
The initial prototype performed well on basic serving and update tasks, and online incremental updates closely recovered offline batch-retraining prediction accuracy.
Source: §4.2 · p. 4, §8 · p. 6Takeaways & Limitations
Velox integrates prediction serving with model maintenance, allowing applications to consume predictions from automatically maintained statistical models.
Source: §8 · p. 6
Abstract
from arXiv · showhide
To support complex data-intensive applications such as personalized recommendations, targeted advertising, and intelligent services, the data management community has focused heavily on the design of systems to support training complex models on large datasets. Unfortunately, the design of these systems largely ignores a critical component of the overall analytics process: the deployment and serving of models at scale. In this work, we present Velox, a new component of the Berkeley Data Analytics Stack. Velox is a data management system for facilitating the next steps in real-world, large-scale analytics pipelines: online model management, maintenance, and serving. Velox provides end-user applications and services with a low-latency, intuitive interface to models, transforming the raw statistical models currently trained using existing offline large-scale compute frameworks into full-blown, end-to-end data products capable of recommending products, targeting advertisements, and personalizing web content. To provide up-to-date results for these complex models, Velox also facilitates lightweight online model maintenance and selection (i.e., dynamic weighting). In this paper, we describe the challenges and architectural considerations required to achieve this functionality, including the abilities to span online and offline systems, to adaptively adjust model materialization strategies, and to exploit inherent statistical properties such as model error tolerance, all while operating at "Big Data" scale.
1. INTRODUCTION
Large-scale frameworks have made complex model training practical, but the analytics stack has largely left model deployment, serving, and management unaddressed. Velox connects trained models to low-latency applications and supports their ongoing maintenance.
1. INTRODUCTION
- Batch frameworks such as Spark suit model training but are a poor fit for interactive predictions, while general-purpose data stores lack model semantics.This leaves model interpretation, serving, and lifecycle management to separate application services and control procedures.Source: §1 · p. 1
- Existing analytics systems emphasize training at scale, while serving models and managing their lifecycle remain missing components of real-world pipelines.BDAS lacked a means to serve data to end users, and industrial users built their own solutions.Source: §1 · p. 1
- Velox provides a low-latency model-serving interface and online and offline maintenance strategies, turning Spark-computed statistical models into end-to-end data products.It exposes models through a generic serving API and supports incremental maintenance, including online strategies.Source: §1 · p. 2
2. BACKGROUND AND MOTIVATION
A song recommender illustrates the trade-offs in building data products from trained models: periodic batch training can leave models stale, while serving predictions can require excessive materialization or repeated bespoke work. Velox addresses challenges in responsiveness, scale, model lifecycle management, and adaptive feedback.
2. BACKGROUND AND MOTIVATION
- Periodic batch training on rating snapshots can leave recommendation models stale until the next training run.The example computes user and song factors periodically with a cluster framework such as Spark or GraphLab.Source: §2 · p. 2
- Precomputing every user-song prediction can materialize billions of values, while online factor-based prediction may require repeated ad hoc implementations.The example contrasts loading all predictions into a low-latency store with computing predictions from latent factors in the application tier.Source: §2 · p. 2
2.1 Challenges and Opportunities
- User-facing services need fast predictions and quick learning from feedback, demands not met together by batch frameworks and conventional low-latency data systems.Velox’s approach includes caching computation, scaling prediction and online training, and fast incremental model updates.Source: §2.1 · p. 2
- Interdependent user and item factors make large models difficult to serve and maintain in a low-latency environment.Velox uses cluster frameworks to initialize training and infer global properties, then incrementally updates models and uses distributed in-memory storage and computation for serving.Source: §2.1 · p. 2
- Model lifecycle management must identify underperforming or outdated models, coordinate retraining, and update online models.Velox tracks performance statistics and version histories for diagnostics and rollbacks, and uses statistics to trigger offline retraining and migrate changes to serving.Source: §2.1 · pp. 2–3
- Recommendation decisions can shape future observations, so adaptive feedback techniques must modify served predictions and observe their outcomes.The paper identifies bandit techniques as Velox’s approach, but the supplied description of that approach is incomplete.Source: §2.1 · p. 3
3. SYSTEM ARCHITECTURE
Velox pairs a model manager for model computation and maintenance with a model predictor for low-latency predictions. It integrates with BDAS storage and Spark, and supports personalized linear models for data-product predictions.
3. SYSTEM ARCHITECTURE
- The model manager maintains declared models using feedback and new data, evaluates performance, and retrains as needed, using Spark for large-scale batch computation.The manager performs fine-grained model updates while delegating large-scale retraining to Spark.Source: §3 · p. 3
- The model predictor provides an up-to-date, low-latency prediction interface, with configurable strategies for pre-materializing predictions.The architecture pairs the predictor and manager with Spark-computed data products stored in a lightweight layer such as Tachyon.Source: §3 · p. 3, Figure 2 · p. 3
- The architecture uses a configurable storage backend, with Tachyon as the default, and co-locates manager and predictor processes with storage workers to maximize data locality.The section introduces further component designs and implementation choices in subsequent sections.Source: §3 · p. 3
- Velox’s current model family assigns each user a d-dimensional weight vector and maps input objects into a d-dimensional feature space.Feature parameters allow the model to incorporate SVMs, deep neural networks, and latent-factor models.Source: §3 · p. 3
- For a recommendation, the predictor reads user weights and feature parameters from Tachyon and evaluates the prediction equation; maintenance updates weights and triggers Spark retraining when quality falls below a threshold.Spark recomputes user models and feature parameters from newly observed data and writes the results back to Tachyon.Source: §3 · p. 3
4. MODEL MANAGEMENT
Velox manages model lifecycles by collecting feedback, updating user-specific models online, and retraining feature parameters offline. Its evaluation strategies monitor performance and identify when retraining is needed.
4.1 Feedback and Data Collection
- Velox collects user-item observations through an API, triggering online updates and storing observations in Tachyon for later Spark retraining.The observation includes a user ID, item data, and correct label.Source: §4.1 · pp. 3–4
4.2 Offline + Online Learning
- Online learning updates user weights continuously, while infrequent offline learning recomputes feature parameters using cluster batch computation.Offline retraining may also update user weights; it invalidates prediction and feature caches, which Velox repopulates with previously cached results.Source: §4.2 · p. 4
- Sherman–Morrison rank-one updates reduce online maintenance from cubic to quadratic time in feature dimension d.The naive implementation achieves acceptable latency across a range of feature dimensions on a collaborative-filtering task.Source: §4.2 · p. 4, Figure 3 · p. 4
- Online updates improve prediction accuracy by 1.6%, compared with a 2.3% increase from full offline retraining.The evaluation initializes feature parameters on half the data, then uses online updates on 70% of the remainder; held-out error is only slightly worse than complete retraining.Source: §4.2 · p. 4
4.3 Model Evaluation
- Velox monitors per-user errors, cross-validation generalization, and bandit-collected validation data, triggering offline retraining when any metric exceeds its configured threshold.The bandit validation pool is designed to be uninfluenced by the model.Source: §4.3 · p. 4
5. ONLINE PREDICTION SERVICE
Velox serves point predictions and topK results, using caching and user-based partitioning to reduce serving costs. It also uses contextual bandits to address feedback loops and assigns new users an average weight vector.
5. ONLINE PREDICTION SERVICE
- Velox caches feature evaluations and final user-item predictions, targeting costs from feature computation or distributed feature lookups.Final-prediction caching is useful when topK calls overlap in their itemsets.Source: §5 · p. 5
- As model size grows, prediction-cache benefits grow; topK latency increases linearly with itemset size.The comparison covers cached predictions and models of several sizes, with 100% cache hit rate as the optimal case.Source: §5 · p. 5, Figure 4 · p. 5
- Partitioning user weights by user ID routes requests to the node holding each model, keeping weight lookups and online writes local.The partitioning also distributes serving load and online-update computation.Source: §5 · p. 5
- New users receive a recent estimate of the average existing user-weight vector, corresponding to the average score across users.Source: §5 · p. 5
- Contextual bandits select items using predicted score plus uncertainty, seeking to reduce uncertainty and limit feedback loops in future training data.The topK interface supports selection of the item with the highest potential predicted rating.Source: §5 · p. 5
6. ADDING NEW MODELS
Velox lets developers add models by implementing a VeloxModel interface that specifies feature transformations, training, and quality evaluation. The system monitors model quality and replaces stale models after offline retraining.
6. ADDING NEW MODELS
- Feature transformations can be computed or looked up from a materialized table, with developers explicitly specifying which strategy applies.The paper describes investigation of automatic analysis of dependencies for choosing strategies.Source: §6 · pp. 5–6
- Developers add models by implementing a VeloxModel instance, specifying feature transformations, offline training, and model-quality evaluation.New feature functions can express a wide range of models and machine-learning techniques.Source: §6 · pp. 5–6
- Velox evaluates loss as new data arrives and marks a model stale if loss increases faster than a threshold, then retrains it offline using the cluster framework.On completion, Velox instantiates a new model and user-weight table, increments the version, and transparently routes prediction requests to them.Source: §6 · p. 6
7. RELATED WORK
Velox relates to predictive database systems, view materialization, and distributed machine learning, while focusing on large-scale, personalized predictive services and online model management.
7. RELATED WORK
- Predictive database systems integrate models or analytics into databases, while Velox emphasizes large-scale personalized services, online training, and feedback loops.The cited systems include Longview, Bismarck, probabilistic databases, and PMML and Oryx3.Source: §7 · p. 6
- Unlike earlier database model management focused on deterministic metadata such as schema, Velox manages statistical machine-learning models.Source: §7 · p. 6
- MauveDB studies model-based view materialization on one node but not latent-feature or personalized models; Columbus demonstrates caching and model reuse.Velox identifies materialized-view maintenance as relevant to model serving.Source: §7 · p. 6
- Distributed machine-learning systems support batch analytics, and Velox leverages their algorithms and batch-processing capacity for online predictive tasks.A Spark SGD variant is one example of work that could improve offline training.Source: §7 · p. 6
8. CONCLUSIONS AND FUTURE WORK
Velox provides scalable model serving and maintenance, using prediction semantics to cache and replicate models while monitoring quality and delegating offline retraining. Its initial prototype performed well on basic serving and update tasks, and online incremental updates closely recovered offline batch-retraining accuracy.
8. CONCLUSIONS AND FUTURE WORK
- Velox serves and maintains machine-learning models at scale, caching and replicating them using prediction semantics and monitoring quality as user patterns change.When needed, Velox delegates offline retraining to existing cluster-compute frameworks, enabling front-end applications to consume predictions from automatically maintained models.Source: §8 · p. 6
- Online incremental updates closely recover the prediction accuracy of offline batch retraining, while the prototype performs well on basic serving and model-update tasks.The initial prototype exposes a RESTful client interface and integrates with Spark and Tachyon for offline training and distributed storage.Source: §8 · p. 6
- Future work includes investigating alternative prediction and modeling APIs, additional dynamically updated multi-armed-bandit techniques, and more efficient top-K support for linear models.The paper also anticipates an alpha code release in early 2015.Source: §8 · p. 6