Source-linked AI summary
InfiniCache: Exploiting Ephemeral Serverless Functions to Build a Cost-Effective Memory Cache
Ao Wang, Jingyuan Zhang, Xiaolong Ma, Ali Anwar, Lukas Rupprecht, Dimitrios Skourtis, Vasily Tarasov, Feng Yan, Yue Cheng
TL;DR
Conventional in-memory object caches are costly for large objects, even though production traces indicate that such objects can be reused. InfiniCache orchestrates ephemeral serverless functions as a request-billed cache, combining erasure coding and delta-sync; its evaluation reports 31–96× improved cost effectiveness over ElastiCache for large objects and comparable performance.
Problem
High memory costs make large-object caching prone to evicting reusable small objects or requiring larger caches, despite large objects’ strong locality and current IMOCs’ lack of large-object support.
Source: §1 · p. 1Method
InfiniCache uses ephemeral serverless-function memory as a request-driven object cache, combining erasure coding, function orchestration, and delta-sync backups.
Source: §1 · pp. 1–2Results
31–96× improvement in cost effectiveness over ElastiCache is reported for large objects, with comparable performance.
Source: §1 · p. 2Takeaways & Limitations
Cache memory is billed only when an object is accessed, distinguishing this pay-as-you-go model from capacity-committed cache services.
Source: §2.2 · p. 3Serverless caching is not cost-effective for small-object-intensive, high-traffic workloads; its hourly cost exceeds ElastiCache above 312 K requests per hour (86 requests per second).
Source: §6 · p. 11
Abstract
from arXiv · showhide
Internet-scale web applications are becoming increasingly storage-intensive and rely heavily on in-memory object caching to attain required I/O performance. We argue that the emerging serverless computing paradigm provides a well-suited, cost-effective platform for object caching. We present InfiniCache, a first-of-its-kind in-memory object caching system that is completely built and deployed atop ephemeral serverless functions. InfiniCache exploits and orchestrates serverless functions' memory resources to enable elastic pay-per-use caching. InfiniCache's design combines erasure coding, intelligent billed duration control, and an efficient data backup mechanism to maximize data availability and cost-effectiveness while balancing the risk of losing cached state and performance. We implement InfiniCache on AWS Lambda and show that it: (1) achieves 31 -- 96X tenant-side cost savings compared to AWS ElastiCache for a large-object-only production workload, (2) can effectively provide 95.4% data availability for each one hour window, and (3) enables comparative performance seen in a typical in-memory cache.
1 Introduction
Large-object caching can be valuable but strains conventional cache capacity; InfiniCache uses ephemeral serverless functions to make cache memory request-driven while addressing function loss and performance challenges.
1 Introduction
- High memory costs make large-object caching prone to evicting reusable small objects or requiring larger, costlier caches.Large objects can consume significant memory and network bandwidth.Source: §1 · p. 1
- Production traces show large objects have strong locality and less stringent speed objectives than small objects, yet current IMOCs do not provide large-object caching.Fast access remains critical for large objects, even though their SLO is less stringent than the sub-millisecond target for small objects.Source: §1 · p. 1
- InfiniCache stores objects in function memory and bills that capacity when an object is accessed, reducing costs versus hourly capacity charges.FaaS providers retain invoked functions and their state between invocations, giving cached objects a sufficient lifetime.Source: §1 · pp. 1–2
- Erasure coding and periodic delta-sync address function reclamation, bandwidth use, stragglers, and the risk of cached-data loss.Functions have limited resources and network constraints, and providers may reclaim them at any time.Source: §1 · p. 2
- 31–96× improvement in cost effectiveness over ElastiCache is reported for large objects, with comparable performance.The evaluation used microbenchmarks and production workloads.Source: §1 · p. 2
2 Background and Motivation
The section motivates large-object caching by documenting its workload characteristics and its tension with small-object performance. It then frames serverless functions as a request-priced cache substrate whose short-lived state and resource limits pose design challenges.
2.1 Large Object Caching
- Large-object caching remains poorly explored across heterogeneous web workloads; IBM Docker-registry traces span over nine orders of magnitude, with more than 20% of objects exceeding 10 MB.The trace analysis used data collected in 2017 from two datacenters.Source: §2 · p. 2, §2.1 · p. 2
- Objects larger than 10 MB occupy more than 95% of storage, risking small-object evictions and consuming network bandwidth that may affect small-object latency.The passage identifies limited cache capacity and bandwidth as the two performance-cost tradeoffs.Source: §2.1 · p. 2
- For the busiest deployment among seven datacenters, requests for objects larger than 10 MB averaged below 3,500 GETs per hour; adding cache capacity raises cost and reduces utilization.The passage contrasts size-threshold exclusion with provisioning more memory.Source: §2.1 · pp. 2–3
- About 30% of large objects are accessed at least 10 times, popularity is long-tailed with the most popular absorbing more than 104 accesses, and 37%–46% are reused within one hour.These trace patterns indicate both repeated access and temporal locality among large objects.Source: §2.1 · p. 3
2.2 Building a Memory Cache on Cloud Functions: Opportunities and Challenges
- The proposed cloud-cache model targets the performance-cost tension with request-driven charging, which FaaS supports through fine-grained per-invocation and execution-time billing.The section says this model charges memory capacity only when a request hits the cached object.Source: §2.2 · p. 3
- Warm functions can retain cached state for tens of minutes to over six hours, but reclamation loses it, and Lambda limits memory, runtime, networking, and QoS.A naive design stores objects in function memory and reinserts them into a new function after reclamation.Source: §2.2 · p. 3
3 INFINICACHE Design
INFINICACHE combines a client library, proxies, and Lambda runtimes to cache erasure-coded objects in serverless-function memory. Its components distribute object chunks, manage Lambda pools, and maintain proxy connections despite functions being ephemeral.
3 INFINICACHE Design
- Applications use a client library to access a cluster of Lambda cache nodes partitioned and managed by proxies; clients and proxies may be co-located.Clients can communicate with any proxy, which is required because Lambda nodes cannot accept inbound connections.Source: §3 · pp. 3–4
3.1 Client Library
- The client library exposes GET and PUT, handles erasure coding, balances requests across proxies, and places encoded chunks on Lambda nodes.Encoding runs in the client library because proxy-side EC processing stalled chunk streaming and significantly reduced transfer performance.Source: §3.1 · p. 4
- GET latency decreased as requests spanned more host VMs, while Lambda functions with ≥1.5 GB memory largely eliminate co-location in same-configuration pools.The measured decrease covered both Lambda-side and client-perceived latency.Source: Figure 4 · p. 4, §3.1 · pp. 4–5
3.2 Proxy
- Each proxy manages a Lambda pool and streams data between clients and nodes; it sends the first d of (d + p) chunks to reduce GET tail latency from stragglers.The proxy also tracks chunk locations, coordinates data migration and delta sync, and evicts objects with a CLOCK-based LRU policy when pool memory is insufficient.Source: §3.2 · p. 5, Figure 5 · p. 5
3.3 Lambda Function Runtime
- The runtime uses a timer to keep a function within its current 100 ms billing cycle when possible, extending execution by a cycle when another request can be served.A preflight PING/PONG lets the proxy check and extend a node’s timeout before forwarding a chunk request, avoiding periodic polling.Source: §3.3 · p. 5
3.4 Reliable Lambda Connections
- Proxies lazily validate each Lambda connection on demand, tracking Sleeping, Active, or Maybe states as nodes are invoked, serve requests, or are replaced during backup.A BYE or timeout can return a connection to Sleeping, while continuous requests require re-validation.Source: §3.4 · p. 6
4 Data Availability and Fault Tolerance
The Lambda study finds availability ranging from 93.36% to 99.76% per hour under observed reclamation patterns, while showing that reclamation behavior varies with warm-up strategy and over time. INFINICACHE combines erasure coding, warm-ups, and delta-sync backups to mitigate data loss, with backup cost growing in importance as more data is cached.
4.1 AWS Lambda Properties
- For 1-minute warm-ups, reclamation peaks fell to 22, 21, and 16 at hour 6, but December showed continuous reclamation at 36 per hour.With 9-minute warm-ups, reclamations clustered around hours 6, 12, and 20–22, and almost all functions were reclaimed.Source: §4.1 · p. 7, Figure 8 · p. 7
4.2 Maximizing Data Availability
- INFINICACHE combines erasure coding, 1-minute warm-ups, and periodic delta-sync backups between peer replicas to improve availability while controlling overhead.Erasure coding permits recovery from up to p lost chunks; delta-sync retrieves only changed data, and peer replicas support failover when one is reclaimed.Source: §4.2 · pp. 7–8
4.3 Data Availability and Cost Analysis
- Object loss is modeled as the probability that at least m chunks are lost, then aggregated over the distribution of reclaimed Lambda nodes.For the illustrated deployment, the model approximates the loss probability using the m-loss term.Source: §4.3 · p. 8
- 93.36%–99.76% availability per hour was estimated for 400 Lambda nodes using 12 erasure-coded chunks, 3-loss threshold, and 1-minute warm-ups.The corresponding 1-minute availability estimate is 99.89%–99.9961%, based on reclamation distributions observed over six months.Source: §4.3 · p. 8
- Backup cost dominates total cost, and its share increases as more data are cached.The system’s cost model includes serving requests, warming functions, and backing up data.Source: §4.3 · p. 8
5 Evaluation
The evaluation tests INFINICACHE on AWS Lambda with microbenchmarks and a 50-hour replay of the Dallas IBM Docker registry trace. It assesses cost, fault tolerance, scalability, and latency against ElastiCache and S3.
5.1 Microbenchmark Performance
- INFINICACHE beats one-node ElastiCache across object sizes, and its (10+1) and (10+2) configurations beat ten-node ElastiCache for larger objects.The paper attributes the larger-object result to first-d data streaming.Source: Figure 11 · p. 9, §5.1 · p. 10
- INFINICACHE throughput, measured in GB/s, scales linearly as client count increases; ideally, it can continue scaling linearly while more Lambda nodes are available to serve GET requests.Source: §5.1 · p. 10
- The (10+1) code offers 10-way I/O parallelism with lower decoding overhead, while (10+0) can suffer higher tail latency from Lambda stragglers.Latency for 100 MB objects improves from 110–290 ms with 512 MB Lambdas to 100–160 ms with 2048 MB Lambdas; gains plateau above 1024 MB.Source: §5.1 · p. 9
5.2 Production Workload
- 31× cost-effectiveness improvement for large-object-only INFINICACHE; disabling backup costs $5.41, 96× cheaper than ElastiCache.With backup enabled, the large-object-only workload costs $16.51; the $5.41 result is for the no-backup configuration.Source: Figure 13 · p. 10, §5.2 · p. 10
- 95.4% availability is achieved for the large-object-only workload, with 1,085 RESET events versus 5,720 for the all-object workload.Without backup, INFINICACHE records 3,912 RESETs, equal to 18.6% of its 21,022 read hits.Source: Figure 14 · p. 10, §5.2 · p. 11
- For about 60% of large requests, INFINICACHE improves latency by at least 100× over S3; it matches ElastiCache at 1–100 MB but is slower below 1 MB.For objects larger than 100 MB, INFINICACHE has consistently lower latencies than ElastiCache.Source: Figure 15 · p. 11, §5.2 · p. 11, Figure 16 · p. 11
6 Discussion
The discussion identifies a cost boundary for small-object caching and describes portability, provider-policy, and future-work considerations.
Small Object Caching
- At the paper’s stated cost models and configuration, INFINICACHE’s hourly cost exceeds ElastiCache above 312K requests per hour (86 requests per second).The hourly cost rises monotonically with access rate.Source: §6 · p. 11
Porting INFINICACHE to Other FaaS Providers
- Porting to Google Cloud Functions is presented as feasible with minor API-related code changes, despite constraints including banned inbound TCP and limited CPU and memory.The paper notes similar constraints for tenants on Google Cloud Functions.Source: §6 · p. 12
Service Provider’s Policy Changes
- Provider policy or pricing changes could affect this design; durable storage atop INFINICACHE is left for future work.The discussion notes that provisioned concurrency pins warm Lambdas but offers no availability guarantee and charges hourly.Source: §6 · p. 12
7 Related Work
Related work spans cost-reduction strategies for cloud storage, use of ephemeral resources, in-memory stores, and serverless computing applications.
Cost-Effective Cloud Storage
- SPANStore reduces cloud-storage costs by distributing data across providers to exploit pricing differences, whereas INFINICACHE uses stateless cloud functions for pay-per-use elasticity.The comparison distinguishes multi-provider storage from function-based caching.Source: §7 · p. 12
Exploiting Spot Cloud Resources
- Unlike spot instances, ephemeral cloud functions have higher churn, cannot host inbound-connection serverful applications, and are automatically cached by providers.INFINICACHE is designed around these function-service characteristics.Source: §7 · p. 12
In-Memory Key-Value Stores
- Most in-memory key-value-store research targets small objects; INFINICACHE targets MB-to-GB objects and uses erasure coding for straggler mitigation and fault tolerance.EC-Cache and SP-Cache also target large objects, using chunking and curated placement for load balancing.Source: §7 · p. 12
New Applications of Serverless Computing
- Prior serverless applications use functions to accelerate computation, while INFINICACHE implements storage atop stateless functions by exploiting transparent function caching.Source: §7 · p. 12
8 Conclusion
The conclusion presents INFINICACHE as an AWS Lambda object cache combining erasure coding and delta-sync backup to balance cost and availability.
8 Conclusion
- INFINICACHE provides request-driven pay-per-use elasticity, improving cost effectiveness 31× over ElastiCache while maintaining 95.4% availability in each one-hour window.Source: §8 · p. 12