Source-linked AI summary
Faa$T: A Transparent Auto-Scaling Cache for Serverless Applications
Francisco Romero, Gohar Irfan Chaudhry, Íñigo Goiri, Pragna Gopa, Paul Batum, Neeraja J. Yadwadkar, Rodrigo Fonseca, Christos Kozyrakis, Ricardo Bianchini
TL;DR
FaaS platforms rely on remote storage for state, while prior caching systems do not adequately accommodate heterogeneous applications, data-driven scaling, or transparent operation. Faa$T provides each application with an auto-scaling cache that unloads and pre-warms with the application and scales for compute, working sets, and object sizes. In experiments, it improved performance by up to 92% and 57% on average while reducing costs for most users compared with state-of-the-art FaaS caching systems.
Problem
FaaS platforms rely on remote storage for state, whose latency and bandwidth limitations restrict efficient support for some applications, while prior caches overlook application diversity and data-driven scaling.
Method
Faa$T ties a transparent cache to each application, unloads it with the application, pre-warms likely objects on reload, and scales using compute demand, data access patterns, working sets, and object sizes.
Results
Up to 92% performance improvement and 57% average improvement were achieved for challenging applications, while costs were reduced for most users versus state-of-the-art FaaS caching systems.
Takeaways & Limitations
Faa$T broadens the applications that can run efficiently on FaaS, including ML inference pipelines and Jupyter notebooks, while reducing reliance on separately provisioned serverful resources.
Takeaways & Limitations
Existing systems may face metadata-management scaling limitations, and some cache approaches restrict supported objects or scale only with computational load.
Abstract
from arXiv · showhide
Function-as-a-Service (FaaS) has become an increasingly popular way for users to deploy their applications without the burden of managing the underlying infrastructure. However, existing FaaS platforms rely on remote storage to maintain state, limiting the set of applications that can be run efficiently. Recent caching work for FaaS platforms has tried to address this problem, but has fallen short: it disregards the widely different characteristics of FaaS applications, does not scale the cache based on data access patterns, or requires changes to applications. To address these limitations, we present Faa\$T, a transparent auto-scaling distributed cache for serverless applications. Each application gets its own Faa\$T cache. After a function executes and the application becomes inactive, the cache is unloaded from memory with the application. Upon reloading for the next invocation, Faa\$T pre-warms the cache with objects likely to be accessed. In addition to traditional compute-based scaling, Faa\$T scales based on working set and object sizes to manage cache space and I/O bandwidth. We motivate our design with a comprehensive study of data access patterns in a large-scale commercial FaaS provider. We implement Faa\$T for the provider's production FaaS platform. Our experiments show that Faa\$T can improve performance by up to 92% (57% on average) for challenging applications, and reduce cost for most users compared to state-of-the-art caching systems, i.e. the cost of having to stand up additional serverful resources.
1 Introduction
FaaS relies on remote storage for state, but its latency and management costs limit efficient support for data-intensive and stateful applications. Faa$T addresses these limitations with application-tied, transparent caching that scales with compute demand, data access patterns, and object sizes.
- Motivation: Remote storage is required for state and cross-function results, but it offers higher latency and lower bandwidth than local memory.Provisioning in-memory storage clusters also adds management overhead and cost.
- Limitations of prior work: Prior FaaS caches often share one cache across applications, disregarding their heterogeneous invocation and data-access characteristics.Caching rarely invoked applications continuously is wasteful, while omitting them can hurt performance.
- Limitations of prior work: Existing approaches commonly use fixed-size or compute-only cache scaling, overlooking changing working sets and the bandwidth costs of large objects.Faa$T instead scales cache resources using data access patterns and object sizes.
- Faa$T design: Faa$T gives each application a transparent local cache that unloads with the application and can pre-fetch popular data when the application reloads.This removes the need for a remote in-memory cache and can reduce remote-storage traffic and provider memory consumption.
- Faa$T design: Faa$T scales cachelet count for data access and reuse patterns in addition to conventional compute-based scaling.This matches cache capacity and bandwidth to application behavior rather than invocation load alone.
- Results: 57% average performance improvement and up to 92% improvement were achieved for challenging applications, while costs were reduced for most users versus state-of-the-art FaaS caching systems.The comparison concerns the cost of provisioning additional serverful resources.
3 Faa$T design
Faa$T is a per-application transparent distributed cache that stores objects across cachelets, preserves metadata across unloading, and pre-warms likely-needed data on reload. It combines cooperative caching, consistent hashing, configurable consistency, and application-specific eviction within provider-managed memory.
- Architecture: Faa$T embeds one cachelet in each application instance and forms a cooperative distributed cache with one cached copy of each object.Cachelets communicate directly, while each application instance runs in a VM or container alongside the FaaS runtime and function code.
- Data access: Local hits, local misses, remote hits, and remote misses define Faa$T’s read paths; accessed objects are cached locally even when another cachelet owns them.Popular objects therefore incur at most one remote hit per cachelet before subsequent local hits, apart from optional consistency checks.
- Data access: Faa$T uses consistent hashing to assign object ownership and reduce metadata tracking and remapping when cachelets are added or removed.The owner handles remote-storage transfers, while cachelets can retain local copies for performance and locality.
- Consistency: Faa$T supports configurable consistency and fault-tolerance trade-offs, including asynchronous writes and reading any cached version for applications that prioritize performance.The default setting writes synchronously to storage and reads the version from storage.
- Pre-warming: When an application unloads, Faa$T records object metadata and later merges it to select objects for pre-warming before the next invocation.Metadata includes object size, access types, access frequency information, and average inter-arrival time; pre-warming occurs while the application is not executing.
- Memory management: Its cachelets are bounded by provider-assigned application memory, while a Memory Daemon evicts objects near capacity using application-tailored policies.The default LRU policy prioritizes non-owned objects, and the evaluated policies improve performance by about 20% on average with multiple cachelets.
4 Scaling Faa$T
Faa$T extends conventional request-based scaling with cache-size and bandwidth-based decisions. It estimates transfer latency for large objects, adds cachelets to partition downloads, and later scales in when access frequency falls.
- Compute scaling: Faa$T retains compute scaling based on incoming requests, in-flight requests, and response-time signals, with each application instance providing compute and cache resources.The Scale Controller collects runtime votes and effects the resulting scale-out or scale-in decision.
- Cache-size scaling: Cache-size scaling responds to cache access and eviction behavior rather than only computational load.When access traffic is substantial but no object has been evicted more than once, Faa$T can vote to maintain capacity; low access frequency triggers scale-in.
- Bandwidth scaling: Bandwidth scaling partitions large-object downloads across multiple cachelets to increase cumulative remote-storage bandwidth and exploit faster inter-instance communication.Faa$T evaluates candidate instance counts using object size, loading latency, and measured bandwidths.
- Bandwidth scaling: Faa$T stops bandwidth-scaling search when transfer latency changes by less than 10% or increases, then requests scale-out and assigns each new instance a distinct S/N byte range.Scale-in is handled through periodic controller voting as object accesses become infrequent.
- Trade-offs: Bandwidth-based scale-out is worthwhile for objects on the order of hundreds of MB, while additional instances can waste computation resources when cache-driven demand is low.Faa$T mitigates this waste by scaling in at low access frequency; providers may also harvest unused resources for low-priority tasks.
5 Implementation
Faa$T is implemented inside a production FaaS runtime, using application data bindings and shared memory to manage cached inputs and outputs with minimal worker changes. Its design supports several languages and can be adapted to platforms with different data-access mechanisms.
- Platform integration: The implementation uses application data bindings to transparently load and manage triggers, inputs, and outputs, while allowing optional Faa$T policies.The system was implemented for a large-scale production FaaS platform and most of the implementation was open-sourced.
- Caching data: Faa$T replaces runtime-worker data messages with shared-memory objects while retaining RPC for control messages.The shared memory area also stores cached data, and workers receive shared-memory addresses rather than copied data.
- Caching data: When inputs are prepared, Faa$T checks the cache before invocation and caches function outputs for reuse and downstream triggering.Caching produced outputs can reduce latency for applications that otherwise use external writes to trigger subsequent functions.
- Language support: Faa$T supports C#, Python, and Node.js, using shared-memory APIs available on Linux and Windows and sharing cache space across containers when needed.Extending support to other languages would require minimal changes according to the implementation description.
- Portability: Faa$T exchanges cachelet data over HTTP and is extensible to other FaaS platforms, but platforms that access data through libraries would require those libraries to be extended.This extension would allow libraries such as Boto3 to consult the cache before remote storage.
6 Evaluation
The evaluation compares Faa$T with native, vanilla FaaS, and caching baselines across application traces and four cache-access scenarios. It includes ML inference and JupyterLess workloads under a production-oriented VM and cost model.
- Methodology: The evaluation uses application traces and separately measures local hits, local misses, remote hits, and remote misses.These scenarios represent the four access paths functions may encounter in the cache.
- Baselines: Faa$T is compared with six baselines, including Native, Vanilla FaaS, InfiniCache, and other caching systems for performance and cost.Native provides local accesses without invocation overheads, while Vanilla accesses objects from remote object storage.
- Workloads: ML evaluation covers SqueezeNet and AlexNet single-model inference plus the multi-stage AMBER Alert pipeline.The pipeline chains bounding-box, people-recognition, and car-recognition models with model sizes of 35MB, 97MB, and 5MB.
- Workloads: JupyterLess evaluation uses five notebooks, including summing a 350MB DataFrame column, capacity planning, FaaS characterization, and counting.Notebook state after each cell executes is stored as JSON data objects.
- Experimental setup: Experiments run one application instance per VM with 8vCPUs, 28GiB of DRAM, up to 500MB/s network bandwidth, and 90MB/s remote-storage bandwidth for large objects.The cost model charges function time and resources separately from VM lifetime and accounts for external caching or storage resources.
- Results: For the JupyterLess 350MB DataFrame workload, Faa$T improves application performance by accessing data in local or remote cache instances.Figure 10 contrasts a pre-loaded native setup with a remote-storage setup before summing.
6.2 Faa$T with applications run in production
Faa$T improves performance over blob storage by exploiting reuse, pre-warming likely objects, and adapting to unload periods and cache size. Benefits are strongest when applications reuse data and access smaller objects frequently.
- Over 40% average improvement is achieved with a 128MB Faa$T cache and pre-warm enabled.About 35% of applications improve by at least 50%.
- Pre-warming becomes more important as the unload period decreases because frequently accessed objects must be available at the next invocation.
- Performance improvement correlates with reuse, with smaller objects reused more often and producing greater gains.
- Faa$T supports current applications with tens to hundreds of KB objects and future workloads accessing objects from hundreds to thousands of MB.
6.3 Comparing Faa$T to existing systems
Across production-style applications, Faa$T improves latency and can substantially reduce cost relative to remote storage, caching systems, and serverful alternatives. Its benefits include transparent caching without requiring users to manage separate resources.
- Faa$T LH improves latency over Vanilla by 50% for AMBER Alert, 87% for AlexNet, and 60% for SqueezeNet.Faa$T LH and RH also outperform Redis for AMBER Alert, while RH matches a manually managed Redis VM.
- Faa$T can run JupyterLess notebooks interactively, with near-native performance when the notebook is nontrivial.
- Faa$T LH and RH improve Native RS performance by 92% and 86%, respectively.
- Faa$T automatically loads objects with spatial and temporal locality for the AMBER Alert pipeline.
- Faa$T remains slower than a Native VM but can be orders of magnitude cheaper because resources are not kept running while applications are idle.
6.4 Is Faa$T pre-warm effective?
Faa$T pre-warms objects based on invocation history before an application runs, substantially reducing AMBER Alert latency relative to both an unloaded cache and a cold start.
- Faa$T pre-warms three models and three labels using application history when the instance is loaded before invocation.
- The evaluation compares a loaded and pre-warmed instance, a loaded instance without pre-warming, and an unloaded cold-start case.
- Pre-warming improves AMBER Alert latency by 58% over no pre-warming and 74% over cold-start execution.The pre-warmed configuration loads three models and three labels totaling 135MB.
6.5 Can Faa$T manage memory effectively?
Faa$T manages memory pressure by evicting cached objects and scaling cache capacity or data loading when object sizes and working sets require it. This preserves application functionality while improving large-object access in suitable cases.
- When heap growth triggers eviction and Faa$T scales to two instances, array allocation succeeds and summation uses all four data access types.Without eviction, allocation fails; without scaling, allocation succeeds with mixed local hits and misses.
- Faa$T opportunistically uses application memory without compromising functionality for applications with large heap requirements.
- Faa$T determines whether to distribute data loading across multiple instances to increase bandwidth for large objects.
- The consistency evaluation reports latency and inconsistencies across different write/read settings, with higher performance as consistency and fault tolerance decrease.
- For a 40MB object, four cachelets download it 47% faster than one when the instances are already loaded.For small 400KB objects, using more than one instance can degrade performance, especially when instances must be loaded.
6.7 Trading off consistency and performance
Faa$T evaluates consistency settings by measuring latency and counter inconsistencies in a shared-state workload. Relaxing consistency reduces latency, with local asynchronous writes matching Cloudburst’s performance while weaker settings trade consistency and fault tolerance for speed.
- Faa$T evaluates five write/read settings using a JupyterLess counter workload shared by five cachelets.The application instances increment a counter in round-robin fashion, targeting a final value of 1K.
- Inconsistencies are measured as the absolute difference between the final counter value and the expected 1K.
- Latency drops as consistency requirements are relaxed across the evaluated settings.
- 800 inconsistencies result from writing and reading locally, the fastest setting, which provides no consistency and the lowest fault tolerance.
- Asynchronous local writes with local reads match Cloudburst’s performance.
6.8 Sensitivity to instance size
Faa$T’s latency decreases as instance size increases for both evaluated applications. The benefit comes from higher network bandwidth, but performance plateaus once accesses saturate bandwidth at 8vCPUs.
- The sensitivity study evaluates the AMBER Alert pipeline and summing a 350MB DataFrame column in JupyterLess.
- Instance memory and network bandwidth scale linearly with vCPUs; a 2vCPU instance provides 8GiB and 1Gbps, while 4vCPUs provide 16GiB and 2Gbps.
- Faa$T latency decreases for both applications as instance size increases because larger instances provide higher network bandwidth.
- Data accesses saturate bandwidth at 8vCPUs, so the 16vCPU instance performs the same as the 8vCPU instance.
7 Related Work
Faa$T differs from prior FaaS caching and storage approaches by remaining transparent, application-tied, and adaptive to data demands. It also complements serverless frameworks and data-placement optimizations rather than replacing them.
- Unlike existing solutions, Faa$T requires no external resources beyond the invoked function, remains transparent to applications, and scales with data size and access patterns.
- Compared with OFC, Faa$T pre-warms objects, supports objects larger than 10MB, scales bandwidth for remote-storage transfers, and keeps one shared-memory copy.
- Faa$T avoids OFC’s machine-learning requirement, reducing decision overhead and management complexity.
- Faa$T can support serverless frameworks for linear algebra, video, analytics, machine learning, and burst-parallel workloads with little to no application change.
- Faa$T’s state-management and instance-scaling focus is orthogonal to Lambdata’s data read/write-intent optimizations.
8 Conclusion
Faa$T is a transparent caching layer that ties cache lifetime to applications, scales with compute and data access demands, and supports configurable consistency. In production experiments, it was faster and cheaper than existing systems for two challenging applications.
- Faa$T ties caching to applications, scales based on compute demands and data access patterns, and provides per-application configurable consistency.
- 57% average faster performance and 99.99% lower cost were measured against existing systems for two challenging applications.