Source-linked AI summary
MLlib: Machine Learning in Apache Spark
Xiangrui Meng, Joseph Bradley, Burak Yavuz, Evan Sparks, Shivaram Venkataraman, Davies Liu, Jeremy Freeman, DB Tsai, Manish Amde, Sean Owen, Doris Xin, Reynold Xin, Michael J. Franklin, Reza Zadeh, Matei Zaharia, Ameet Talwalkar
TL;DR
Large-scale machine learning requires efficient support for growing data and iterative computation. This paper presents MLlib, a distributed library integrated with Spark and equipped with broad algorithms and pipeline APIs. MLlib’s community expanded to over 140 contributors, while version comparisons reported a 3× average speedup across algorithms.
Problem
Growing and increasingly complex datasets require scalable statistical and machine learning solutions for large-scale processing.
Method
MLlib provides distributed implementations of common learning algorithms, statistical and optimization primitives, and high-level Spark-integrated APIs for machine learning pipelines.
Results
3× average speedup was reported across MLlib algorithms when comparing versions 1.0 and 1.1, alongside growth to over 140 contributors from more than 50 organizations.
Takeaways & Limitations
MLlib combines Spark’s iterative execution and ecosystem with distributed learning functionality to simplify large-scale machine learning pipeline development.
Abstract
from arXiv · showhide
Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open-source distributed machine learning library. MLlib provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLlib supports several languages and provides a high-level API that leverages Spark's rich ecosystem to simplify the development of end-to-end machine learning pipelines. MLlib has experienced a rapid growth due to its vibrant open-source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed.
1. Introduction
Apache Spark addresses large-scale, iterative machine learning on growing and complex datasets. MLlib extends Spark with distributed algorithms, statistical and optimization primitives, and pipeline tools supported by an active open-source ecosystem.
- Spark is a fault-tolerant cluster-computing system whose efficient iterative execution supports large-scale machine learning applications.
- MLlib provides scalable implementations for classification, regression, collaborative filtering, clustering, and dimensionality reduction.It also includes statistics, linear algebra, and optimization primitives.
- 140+ contributors and Spark integration support MLlib’s rapid development and adoption.Its spark.ml API and Spark ecosystem simplify end-to-end machine learning pipeline development.
- Spark’s ecosystem includes MLlib as a high-level library alongside other components.
2. History and Growth
MLlib evolved from a small Berkeley project into a broader Apache Spark library through open-source development. Its contributor community expanded rapidly alongside the library’s functionality.
- MLlib began in 2012 through the MLbase project and was open-sourced in September 2013 as part of Spark.It was included in the Spark 0.8 release and follows Spark’s Apache 2.0 licensing and release cycle.
- 11 contributors initially developed MLlib with a limited set of standard machine learning methods.
- Over 140 contributors from more than 50 organizations participated in MLlib by the Spark 1.4 release, less than two years later.The open-source community’s growth supported additional functionality.
3. Core Features
MLlib combines distributed learning methods, algorithmic optimizations, pipeline APIs, Spark ecosystem integration, and documentation. These components target practical development of scalable machine learning workflows.
- Supported Methods and Utilities: MLlib supports distributed algorithms for linear models, naive Bayes, tree ensembles, ALS, k-means, and principal component analysis.It also provides convex optimization, distributed linear algebra, statistics, feature extraction, and multiple data and model formats.
- Algorithmic Optimizations: ALS uses blocking, while decision trees reduce communication through feature discretization and parallelize ensemble learning.Generalized linear models parallelize gradient computation using C++-based linear algebra libraries.
- Pipeline API: spark.ml simplifies multi-stage pipeline development and tuning through uniform high-level APIs.The APIs cover preprocessing, feature extraction, model fitting, validation, and swapping in specialized algorithms.
- Spark Integration: Spark integration supplies execution, data-cleaning, featurization, SQL, and structured-data processing components for MLlib workflows.
- Documentation, Community, and Dependencies: MLlib documentation covers supported methods, utilities, code examples, and APIs across supported languages.The user guide also lists open-source dependencies, while an online course explains core distributed-algorithm concepts.
4. Performance and Scalability
MLlib’s performance and scalability were evaluated with ALS on scaled Amazon Reviews data and with cross-version benchmarks. The reported results show continued performance improvements, including a 3× average speedup.
- ALS benchmarking used five iterations on scaled Amazon Reviews datasets across a 16-node EC2 cluster, comparing MLlib with Apache Mahout.The benchmark used MLlib versions 1.1 and 1.4 and Mahout 0.9 on Hadoop MapReduce.
- 3× average speedup was observed across MLlib algorithms when comparing versions 1.0 and 1.1.The improvements reflected algorithmic changes and broader communication-protocol improvements in Spark and MLlib 1.1.
5. Conclusion
MLlib remains in active development, with contribution details and contributor acknowledgments provided through Apache Spark resources.
- MLlib is in active development, and contribution information is available through the Spark Contributing guide.The paper directs readers to the Apache Spark Contributing page for further details.
- The authors acknowledge MLlib contributors through the Spark contributor list and repository history.The Spark GitHub repository and git log can help identify MLlib contributors.
- The Spark repository and git log provide resources for identifying contributors to MLlib.