Source-linked AI summary
MLI: An API for Distributed Machine Learning
Evan R. Sparks, Ameet Talwalkar, Virginia Smith, Jey Kottalam, Xinghao Pan, Joseph Gonzalez, Michael J. Franklin, Michael I. Jordan, Tim Kraska
TL;DR
MLI addresses the difficulty of turning concise ML prototypes into robust, scalable distributed implementations. It introduces high-level abstractions for data and computation while preserving control over distributed execution, and reports competitive scalability with low programming complexity. The system supports concise implementations and, in one experiment, completes a 200K-point dataset in less than 10 minutes while MATLAB runs out of memory.
Problem
ML researchers need scalable distributed ML implementations, but MATLAB and R prototypes are often non-robust and non-scalable, while translating them into production systems can introduce errors.
Method
MLI provides high-level abstractions for data loading, feature extraction, and model development while allowing developers to control communication and parallelization patterns.
Results
MLI/Spark vastly outperforms Mahout and matches specialized systems’ scaling properties, while a 200K-point experiment finishes in less than 10 minutes when MATLAB runs out of memory.
Takeaways & Limitations
MLI provides tools for constructing high-performance distributed ML algorithms without onerous programming complexity.
Abstract
from arXiv · showhide
MLI is an Application Programming Interface designed to address the challenges of building Machine Learn- ing algorithms in a distributed setting based on data-centric computing. Its primary goal is to simplify the development of high-performance, scalable, distributed algorithms. Our initial results show that, relative to existing systems, this interface can be used to build distributed implementations of a wide variety of common Machine Learning algorithms with minimal complexity and highly competitive performance and scalability.
I. INTRODUCTION
MLI addresses the gap between quickly developed but non-scalable ML prototypes and scalable systems that are harder for ML researchers to use. It provides high-level abstractions while retaining control over distributed execution, with competitive scalability and concise code.
- MATLAB and R enable rapid prototyping but commonly produce ad-hoc, non-robust, and non-scalable implementations.
- Low-level distributed systems improve speed and scalability but are less accessible, while high-level systems can be difficult to implement efficiently.
- Losses in translating researchers’ prototypes into production implementations can introduce errors that affect algorithm quality.
- MLI combines high-level abstractions with developer control over communication and parallelization, avoiding reliance on a complex optimizer.
- MLI abstractions target data loading, feature extraction, model training, and testing, while implementations remain concise and readable compared with MATLAB or R.
- MLI/Spark vastly outperforms Mahout and matches specialized systems’ scaling properties, with performance within a small constant factor.
II. RELATED WORK
Existing ML platforms trade off productivity, scalability, generality, and implementation simplicity. MLI is positioned against high-productivity tools, specialized systems, MapReduce libraries, and higher-level distributed abstractions.
- MATLAB and R provide high-productivity numerical computing, but their process-centric designs are not well suited to large-scale distributed data-centric workloads.
- Specialized systems such as liblinear, Vowpal Wabbit, Shogun, and MLPack achieve efficiency for targeted tasks but do not directly simplify general ML design and implementation.
- Mahout and SystemML expose limitations involving new-method development, iterative computation, low-level algebra, or Hadoop-based execution.
- DryadLinq and Hyracks provide distributed data-flow execution but expose low-level APIs requiring ML experts to recast algorithms as dataflow operators.
- GraphLab is suited to some ML tasks, whereas OptiML and SEJITS offer higher-level embedded DSLs for graph and matrix operations.
III. MLI
MLI uses two system-independent data abstractions, MLTable and LocalMatrix, within a structure that connects optimizers, algorithms, and models. These abstractions support data preparation, local linear algebra, and reusable ML workflows.
- MLI consists of MLTable and LocalMatrix APIs used by Optimizers and Algorithms to produce Models, and can be implemented in local or distributed settings.
- MLTable: MLTable supports loading unstructured or semi-structured data, transformations, feature extraction, and parallel batch operations on data partitions.
- MLI predefines Optimization, Algorithms, and Models interfaces to encourage code reuse and provide a consistent external system interface.
- MLTable: MLTable provides a table-like interface resembling SQL tables, R data.frames, or MATLAB Dataset Arrays, with typed columns and common relational operations.
- MLTable: Featurized data can be cast into MLNumericTable, which guarantees numeric columns and treats each row as a feature vector.
- MLTable: An end-to-end text-clustering example applies nGrams() and tfIdf() before Kmeans clustering produces an output model.
B. LocalMatrix
LocalMatrix supplies familiar linear-algebra operations over data partitions rather than globally distributed matrices. Its locality constraint supports algorithms whose local results are combined through global reductions.
- LocalMatrix provides linear-algebra primitives on data partitions whose boundaries are typically determined automatically by the system.
- Developers must structure operations to run locally and combine later through global reduce operations, reflecting a shared-nothing design.
- LocalMatrix resembles matrices in MATLAB, R, and other numerical environments while operating on individual partitions.
- LocalMatrix supports row, column, and slice indexing, matrix-matrix and matrix-scalar operations, and matrix inversion.
C. Optimization, Models, and Algorithms
MLI provides interfaces for optimization, model production, and algorithm development, while its Spark implementation supports concise code and scalability across distributed ML workloads.
- Optimization: MLI treats optimization as a first-class API component and supports introducing new optimizers for iterative approximate solutions.This addresses models without closed-form solutions and cases where closed-form computation grows super-linearly with data size.
- Models and Algorithms: The Algorithm interface accepts data and hyperparameters through train() and returns a Model object that makes predictions.The common interface covers outputs such as predicted classes and collaborative-filtering recommendations.
- Evaluation Platform: MLI evaluates binary classification and matrix factorization on Spark, chosen for iterative workloads and resilience to node failures.Spark provides automatic data replication and computation lineage for recovery in distributed environments.
- Design Claims: MLI implementations produce concise, readable code and support algorithms including linear SVMs, linear regression, and regularized variants.The paper evaluates two problem settings while describing extensions to a broader range of algorithms.
- Scalability: MLI matches the scalability of low-level distributed systems, with performance within a small constant factor, across strong and weak scaling results.The comparison concerns execution times and scalability on the paper’s two example problems.
A. Binary Classification: Logistic Regression
The logistic-regression example formulates binary classification through negative-likelihood minimization and gradient-based optimization, then evaluates distributed scaling against established systems.
- Model: Logistic regression models binary labels by minimizing the negative log-likelihood over a parameter vector.The dataset contains n points with d features, and the optimal parameter vector is denoted w∗.
- Optimization: Gradient descent updates the parameter vector using the negative gradient and a learning rate, while SGD approximates the gradient sum with one summand.The logistic sigmoid is σ(x) = 1/(1 + exp(−x)).
- Evaluation: The experiments compare MLI/Spark with Vowpal Wabbit on the same cluster and MATLAB on a similarly configured single-node machine.The scaling runs use 1, 2, 4, 8, 16, and 32 machines.
- Evaluation: Weak scaling uses up to approximately 200GB of featurized ImageNet data, with roughly 200K images in the 32-node experiment.Each image is represented with 160K dense features, and the experiment covers approximately 20% of the full ImageNet dataset.
Implementation:
MLI implementations combine concise algorithm code with distributed execution and scaling experiments across logistic regression and ALS. The results compare MLI with MATLAB, Mahout, GraphLab, and VW under weak and strong scaling.
- Logistic regression: Logistic regression in MLI defines a gradient function and invokes the SGD optimizer, while aggregating parameters across workers after each round.The implementation uses a MapReduce-style master averaging and broadcast, unlike VW’s AllReduce tree.
- Logistic regression: 35%: VW is on average faster than MLI in weak-scaling logistic regression, but never twice as fast.The comparison excludes VW data-preparation time, which was significant but expected to be a one-time production cost.
- Alternating least squares: ALS alternates optimization of U and V, using a closed-form solution with factors updated in parallel and broadcast after each update.The implementation distributes both the matrix and its transpose to access ratings efficiently.
- Alternating least squares: MLI’s ALS implementation uses sparse matrix representations, linear-algebra primitives, and MLTable functionality for distributed factor updates.Supported operations include transpose, multiplication, solving linear systems, and nonzero-index access.
- Alternating least squares: MLI ALS outperforms MATLAB and MATLAB-Mex at moderate data sizes, remains within 4x of GraphLab, and outperforms Mahout in runtime and scaling.MATLAB and MATLAB-Mex run out of memory before completing the 16x or 25x Netflix datasets.
- Strong scaling: On the 9x Netflix dataset, MATLAB runs out of memory before completion, while GraphLab outperforms MLI by less than 4x.These observations come from strong-scaling experiments.
C. Configuration Considerations
Comparable hardware does not eliminate configuration differences: software systems varied substantially in installation, configuration, and code-execution ease.
- C. Configuration Considerations: Software systems differed drastically in ease of installation, configuration, and executing code despite comparable or identical experimental hardware.The comparison concerns practical configuration considerations rather than computational performance.
Vowpal Wabbit:
VW cluster mode requires careful dataset partitioning and side-channel communication, making data preparation difficult and the system failure-prone.
- Vowpal Wabbit: VW requires equally sized compressed training files, with the file count matching the desired number of concurrent map tasks.Its Hadoop Streaming launch process is combined with an AllReduce side channel using TCP sockets between map tasks.
Mahout:
The comparison highlights setup and integration trade-offs across distributed ML systems, while MLI/Spark emphasizes comparatively easy deployment and scalable algorithm development.
- Mahout: Mahout is easy to set up on an existing Hadoop cluster, but larger problems require careful job-memory tuning for performant completion.Its input formats are reasonably close to MLI’s formats.
- GraphLab: GraphLab performed well in speed and scalability tests but was difficult to set up and integrate with HDFS-based clusters.Its setup requires MPI, dependency installation, manual software distribution, and additional handling for matrices that exceed memory.
- MLI and Spark: MLI/Spark can be configured and launched comparatively easily with dependencies, environment settings, and a single Scala program.New algorithms can be added as Scala classes, with driver programs generated from library examples.
- Conclusion: MLI’s primitives support constructing high-performance distributed ML algorithms without onerous programming complexity.The conclusion positions MLI as a foundational layer for the broader MLBASE system.
APPENDIX
The appendix illustrates MLI’s APIs, data-processing workflow, algorithm implementations, and scaling evaluations through figures covering MLTable, LocalMatrix, logistic regression, and ALS.
- MLTable: Figure A1 illustrates core operations of the MLTable API, while noting that the illustration is not exhaustive.MLTable provides a table-like interface for loading and transforming data.
- Text-data workflow: Figure A2 depicts loading, featurizing, and learning clusters on a text-data corpus.The caption presents these as successive stages of the illustrated workflow.
- LocalMatrix: Figure A3 illustrates the LocalMatrix API.LocalMatrix provides data-local linear algebra operations on dataset partitions.
- Logistic regression: Figure A4 compares logistic-regression code in MATLAB with MLI code shown in the middle and bottom panels.The appendix uses code length comparisons to illustrate concise and readable implementations.
- Scaling and ALS: Figures A5 and A6 cover logistic-regression execution time and strong scaling, while Figures A7 and A8 cover the corresponding ALS scaling views.Figure A9 compares MATLAB and MLI matrix-factorization code for ALS.