Source-linked AI summary
Occupy the Cloud: Distributed Computing for the 99%
Eric Jonas, Qifan Pu, Shivaram Venkataraman, Ion Stoica, Benjamin Recht
TL;DR
Many users still struggle with the configuration and management demands of cloud distributed computing, even for simple parallel workloads. The paper proposes stateless functions with remote storage and evaluates them through PyWren. PyWren supports several distributed abstractions and reports competitive performance, while launch overhead and storage remain important limitations.
Problem
Existing cloud data-processing systems impose high configuration and management barriers, leaving many users unable to exploit distributed computing and elasticity.
Method
The paper builds PyWren, a Python prototype using stateless AWS Lambda functions, remote storage, dynamic code injection, and a global scheduler.
Results
PyWren implements image-processing pipelines, BSP-style applications, MapReduce, and parameter servers; a word count on 83M items was 17% slower than PySpark, and 1TB sorting took 3.4 minutes.
Takeaways & Limitations
Stateless functions with remote storage provide a flexible building block for simpler, elastic distributed data processing and may fit future disaggregated computing environments.
Takeaways & Limitations
The current implementation incurs 20-30 seconds of function-launch overhead, while storage throughput limits larger shuffle-intensive workloads.
Abstract
from arXiv · showhide
Distributed computing remains inaccessible to a large number of users, in spite of many open source platforms and extensive commercial offerings. While distributed computation frameworks have moved beyond a simple map-reduce model, many users are still left to struggle with complex cluster management and configuration tools, even for running simple embarrassingly parallel jobs. We argue that stateless functions represent a viable platform for these users, eliminating cluster management overhead, fulfilling the promise of elasticity. Furthermore, using our prototype implementation, PyWren, we show that this model is general enough to implement a number of distributed computing models, such as BSP, efficiently. Extrapolating from recent trends in network bandwidth and the advent of disaggregated storage, we suggest that stateless functions are a natural fit for data processing in future computing environments.
1 Introduction
Cloud distributed computing remains difficult for many users despite widespread platforms, while stateless functions with remote storage offer a simpler and elastic alternative. PyWren demonstrates that this primitive can support higher-level data-processing abstractions with modest performance costs.
- Motivation: Many users still face high barriers to cloud computing because existing platforms require complex cluster, instance, pricing, programming-model, and task-granularity decisions.These difficulties persist even for sophisticated users seeking elasticity.
- Motivation: Embarrassingly parallel workloads such as hyperparameter tuning, Monte Carlo simulation, and feature extraction fit map-reduce, yet most surveyed Berkeley machine-learning graduate students had never written a cluster job.The cited reason was the complexity of setting up cloud platforms.
- Approach: Stateless functions let users submit code to remote containers while inputs and outputs remain in shared remote storage, eliminating direct cluster-management overhead.PyWren uses a global scheduler and remote storage to handle state management.
- Approach: Dynamic code injection combined with remote storage preserves serverless elasticity while enabling data-processing abstractions including MapReduce and parameter servers.Functions scale elastically with input size and are invoked as new inputs arrive.
- Results: 30-40 MB/s per core of S3 read and write performance matched the per-core performance of a local SSD on typical EC2 nodes.Across 2800 simultaneous functions, throughput scaled to 60-80 GB/s to S3.
- Results: A word-count job on 83M items was only 17% slower than PySpark on dedicated servers, while PyWren sorted 1TB in 3.4 minutes.The paper also identifies storage throughput as a major bottleneck for larger shuffles.
2 Is the cloud usable?
Existing cloud data-processing systems impose substantial configuration, planning, and workload-management burdens on users. The paper therefore targets a simple function interface that runs arbitrary existing code in parallel without requiring users to configure servers or frameworks.
- System barriers: Distributed data-processing platforms expose complex abstraction layers whose configuration can require coordinating Spark, HDFS, YARN, the JVM, and host operating-system memory limits.The paper argues that concise application code does not remove the need to master these underlying layers.
- System barriers: Cloud users must plan instance types, cluster sizes, pricing, and workload management before running jobs; even simplifying products still require explicit cluster lifecycle and instance choices.AWS offers 70 instance types across 14 geographic datacenters with differing prices.
- System barriers: Spot-instance elasticity adds preemption and checkpointing concerns, requiring users to protect intermediate data or keep some infrastructure on non-spot instances.These requirements add management complexity and make elasticity difficult to obtain in practice.
- What users want: The stated user goal is to run existing optimized single-machine code in parallel with minimal development time, rather than maximize parallel performance.The target users are more likely to write performant single-threaded code than complex distributed-systems code.
- What users want: Compute-bound workloads can parallelize across stateless functions, while data-bound workloads can use a simpler map-reduce model with outputs persisted in object storage.Examples include parameter sweeps, Monte Carlo simulations, and remote feature extraction.
- Proposal: A simple abstraction for arbitrary cloud functions can cover compute-bound and data-bound workloads without requiring users to set up or configure servers and frameworks.The paper presents this abstraction as the basis for a more accessible distributed-computing system.
3 A Modest Proposal
The proposal uses stateless functions, remote storage, and global scheduling to simplify elastic distributed computing while supporting both embarrassingly parallel and more general workloads.
- Architecture: Serverless stateless functions replace server-oriented abstractions, simplifying programming and deployment for end users.Functions access input and output through shared remote storage, while containers retain no invocation state.
- Systems Components: The system combines a low-overhead runtime, fast scheduler, and high-performance remote storage to execute user-submitted functions.A global scheduler places functions in temporary containers, and runtime dependencies can be annotated at submission.
- Systems Components: Stateless retries provide simple fault tolerance when functions are idempotent and completed writes are tracked atomically in remote storage.A failed function is restarted, potentially elsewhere, on the same input.
- PyWren: A Prototype: PyWren exposes a Python map primitive that launches one stateless function per input element and transparently serializes local computation state.The API mirrors existing Python parallel-processing interfaces.
- PyWren: A Prototype: 18 GFLOPS per core scales beyond 40 TFLOPS with 2800 workers, while S3 reaches over 60 GB/s write and 80 GB/s read throughput.Per-Lambda averages are 30 MB/s write and 40 MB/s read; remote-storage performance therefore scales substantially with worker count.
4 Discussion
The discussion identifies resource allocation, scheduling, storage, and launch latency as key challenges for high-performance serverless data processing. It also marks specialized hardware and coordination-heavy applications as current boundaries of the model.
- Resource balance: Serverless designs must address resource allocation because stateless functions transfer inputs and outputs over the network under low existing limits.The authors suggest using input/output bandwidth as a heuristic for allocating function resources.
- Pricing: PyWren’s disaggregated architecture makes cost estimation simpler, while Lambda costs approximately $0.06 per GB-hour and about twice on-demand instance pricing.The finer-grained billing and elasticity are presented as reasons the premium may be worthwhile.
- Scalable Scheduling: Scheduling must bridge infrastructure-only cluster status with user-only knowledge of job structure and dependencies.The authors identify this separation as an open scheduling problem for stateless-function systems.
- Distributed Storage: Current distributed storage systems limit large shuffle-intensive workloads, motivating more efficient storage and append-capable APIs.The separation of storage and compute translates performance challenges into distributed-storage requirements.
- Launch Overheads: Function invocation can take 20–30 seconds, approximately 10% of execution time, without caching.The overhead partly reflects AWS invocation rate limits and setup time for the custom Python runtime.
- Other applications: Applications requiring GPUs or FPGAs, or extensive coordination among long-running processes, do not fit the current PyWren model well.AWS Lambda does not support the specialized hardware discussed, and particle simulations require coordination that remote-storage stateless functions may not suit.
5 Conclusion
Existing server-oriented cloud data-processing systems create high barriers for many users. The paper proposes stateless functions with remote storage as an elastic, simple foundation for more complex data-processing abstractions.
- 5 Conclusion: Stateless functions with remote storage can provide elasticity and simplicity while serving as flexible building blocks for complex data-processing abstractions.This proposal responds to the barriers created by server-oriented systems.