PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast-Convergent Meta-RL via Gradient-Clustered BS Sampling for Edge Caching

Farnaz Niknia, Ping Wang

arXiv:2609.16370v1cs.NIcs.LGcs.SYeess.SY

TL;DR

Independent BSs share a caching problem but differ in traffic, and uniform sampling can produce high-variance meta-gradient estimates. The paper uses gradient-clustered sampling in a PPO/MAML caching framework and proves lower estimator variance under BS heterogeneity.

  • Problem

    Independent BSs have structurally similar caching tasks but different traffic, while uniform sampling can miss traffic regimes and yield high-variance meta-gradients during scalable meta-training.

    Source: §I · p. 1

  • Method

    Each BS runs a local PPO agent, and a shared MAML-style policy is trained using BS batches sampled from gradient-similarity clusters in proportion to cluster size.

    Source: §I · p. 2

  • Results

    Under BS traffic heterogeneity, gradient-clustered sampling yields a strictly lower-variance meta-gradient estimator than uniform random sampling.

    Source: §I · p. 2

  • Takeaways & Limitations

    The variance reduction replaces total gradient variance with the smaller within-cluster component while preserving the known MAML convergence guarantee.

    Source: §I · p. 2, 4) Variance Decomposition of the… · p. 9

Abstract

from arXiv · show

Wireless edge caching networks typically consist of many independent Base Stations (BSs), each facing its own request rate and content popularity profile. Training a Reinforcement Learning (RL) caching agent from scratch at every BS forces each agent to relearn, through slow trial and error, a decision problem that is structurally identical across the network. Meta-reinforcement learning removes this redundancy by learning a shared initialization that adapts to any BS in a few local updates; however, meta-training itself becomes the bottleneck at scale: the meta-gradient must be estimated from a small subset of BSs at each meta-iteration, and sampling this subset uniformly at random yields a high-variance estimate, an issue existing meta-RL caching frameworks leave unaddressed. This paper proposes a meta-reinforcement learning framework for caching across independent, non-overlapping BSs that directly targets this bottleneck. Each BS runs a local Proximal Policy Optimization (PPO) agent, formulated as a Semi-Markov Decision Process (SMDP) over content popularity, size, lifetime, and importance, while a shared meta-policy is learned via a Model-Agnostic Meta-Learning (MAML)-style loop. To scale meta-training and accelerate convergence, we introduce gradient-based clustering, which groups BSs by local gradient similarity and draws from every cluster, in proportion to its size, at each meta-iteration. We prove, via an Analysis of Variance (ANOVA)-style decomposition of gradient variance, that this strategy yields a strictly lower-variance meta-gradient estimator than uniform random sampling under BS heterogeneity.

I. INTRODUCTION

Independent BSs face different traffic but share the same underlying caching problem, making knowledge transfer potentially useful. The paper addresses the scaling bottleneck in meta-training with gradient-clustered sampling and analyzes its variance and convergence effects.

I. INTRODUCTION

  • BSs have distinct request rates and popularity profiles but structurally identical caching decisions; prior multi-node work either coordinates BSs or leaves them isolated.Meta-learning instead optimizes a shared initialization for rapid adaptation to local traffic using a few gradient steps.Source: §I · p. 1
  • Gradient-clustered sampling draws BSs from every gradient-similarity cluster in proportion to its size, representing each traffic regime in every meta-iteration.The framework combines local PPO agents with a shared MAML-style initialization; the sampling procedure periodically clusters BSs using their local gradients.Source: §I · p. 2
  • Under BS traffic heterogeneity, gradient-clustered sampling yields a strictly lower-variance meta-gradient estimator than uniform random sampling, while retaining MAML’s known convergence guarantee.The variance analysis separates within-cluster and between-cluster components.Source: §I · p. 2
  • Simulations report faster, more stable meta-training than uniform random sampling and adapted caching performance matching the strongest baselines without transfer-learning negative transfer.The simulations also empirically confirm the predicted variance decomposition and reduction.Source: §I · p. 2

II. RELATED WORK

  • Single-node caching studies develop DRL methods, while federated caching shares model parameters across nodes; federated personalization does not explicitly optimize a shared initialization for fast adaptation to unseen tasks.The paper frames federated learning and meta-learning as complementary: one addresses where computation and data reside, the other how quickly policies adapt to unseen tasks.Source: A. Single-Node Edge Caching · p. 2, B. Federated Edge Caching · p. 3

C. Meta-Reinforcement Learning for Edge Caching

Prior meta-RL caching work varies tasks and forms training batches in different ways. This paper treats BSs as independent caching environments with heterogeneous traffic, content attributes, and age-dependent utility.

C. Meta-Reinforcement Learning for Edge Caching

  • Prior multi-node meta-RL methods differ in task construction and BS selection; existing tasks often vary popularity while holding request-process temporal statistics fixed.The paper identifies its distinction as treating both popularity and request-arrival dynamics as task variation and using gradient-based rather than random or hand-partitioned BS sampling.Source: C. Meta-Reinforcement Learning for… · p. 4, D. Motivation · p. 4

A. Network Architecture

  • The network has independent, non-overlapping BSs with local caches and no content sharing or inter-BS coordination; each cache stores items from a shared central-server catalog.Each BS’s caching problem is self-contained, so a representative BS formulation applies to all BSs.Source: A. Network Architecture · p. 4

Content Attributes:

  • Caching decisions account jointly for each content type’s popularity, lifetime, size, and importance; popularity varies by BS and time, while the other attributes are shared across BSs.A popular item may still be less beneficial than a moderately popular one if it is large, short-lived, and unimportant.Source: Content Attributes · p. 5

Content Utility:

  • Content freshness starts at zero when generated and increases with age to one at expiration; content utility increases with importance and decreases with freshness.The benefit of retaining an item depends jointly on its freshness and importance to the local population.Source: Content Utility · p. 5

B. Environment Uncertainties

Each independent BS faces changing popularity and request rates, complicating caching decisions over time. The formulation models each BS’s caching task as an SMDP with a state describing cache and content conditions and a binary storage action.

1) Content Popularity Dynamics:

  • Non-stationary popularity can make cached items lose demand while newly popular content goes uncached, degrading the cache-hit rate.Demand shifts with social trends, time of day, and unpredictable events.Source: 1) Content Popularity Dynamics · p. 5

2) Request-Rate Variability:

  • Requests at each BS follow an independent Poisson process with unknown rate λb, which fluctuates and can make policies over-replace items or fail to keep pace with bursts.The BS has no prior knowledge of its request rate or the governing Poisson-process parameters.Source: 2) Request-Rate Variability · p. 5

IV. PROBLEM FORMULATION

  • Because Poisson arrivals create random intervals between decisions, caching is formulated as an SMDP rather than a fixed-step MDP.The SMDP describes a sequential choice upon each request to store the item given cache occupancy, content attributes, and future effects.Source: §IV · p. 5
  • Each BS’s SMDP is self-contained and structurally identical, so the formulation for one representative BS applies to every BS without modification.The independence follows from disjoint coverage areas and the absence of shared caches or inter-BS collaboration.Source: §IV · p. 5

1) State Space:

  • The state includes normalized available capacity, cache occupancy and utility, popularity, importance, lifetime, and content size.Available capacity ranges from zero for a full cache to one for an empty cache.Source: 1) State Space · pp. 5–6

2) Action Space:

  • At each request, the agent chooses whether to cache the item; if storage is full, the lowest-utility items are evicted until the item fits.The binary action value 1 stores the item, while 0 leaves the cache unchanged.Source: 2) Action Space · p. 6

3) System Dynamics:

The caching agent learns from unknown system dynamics and optimizes long-run average reward. Each BS uses PPO, while a shared meta-learning layer and gradient-clustered sampling support learning across BSs.

3) System Dynamics:

  • Unknown transition dynamics and random sojourn times arise from Poisson request arrivals, so reinforcement learning estimates system behavior from experience.The SMDP transition kernel and the time between decisions jointly govern state transitions.Source: 3) System Dynamics · p. 6

4) Reward Function:

  • The reward favors cached content that is popular and useful while penalizing unused cache capacity.The first term is high when cached items are fresh, important, and frequently requested; the second discourages wasting available storage.Source: 4) Reward Function · p. 6

5) Optimization Objective:

  • The infinite-horizon objective maximizes long-run average reward, and solving each independent BS separately is equivalent to the global multi-BS optimization.The reward jointly reflects average cached-content utility and unused capacity.Source: 5) Optimization Objective · p. 6

V. META-REINFORCEMENT LEARNING WITH GRADIENT-CLUSTERED BS SAMPLING FOR MULTI-BS EDGE CACHING

  • Each BS runs an independent PPO agent tailored to local traffic, with a shared meta-policy and gradient-clustered BS batches used for meta-updates.The stated framework uses a MAML-style outer loop and presents a theoretical proof for the sampling strategy.Source: §V · p. 6

1) Why PPO:

  • PPO handles discrete cache-or-do-not-cache actions directly and uses clipped policy updates; the paper selects it over DDPG and DDQL for this setting.The text characterizes PPO’s clipped updates as supporting fast and stable training under fluctuating request rates and popularity.Source: A. Per-BS Caching Policy via Proximal… · p. 6, 1) Why PPO · p. 7

2) PPO Mechanics at a Single BS:

Each BS uses local PPO updates, with clipping and SMDP-aware discounting shaping its policy learning. A MAML-style loop fine-tunes a shared initialization on each BS and updates it using post-adaptation losses.

2) PPO Mechanics at a Single BS:

  • PPO clipping limits how far a BS policy can drift from its previous policy in one update while allowing progress.The clipping parameter is shared across BSs.Source: 2) PPO Mechanics at a Single BS · p. 7
  • Because the decision process is an SMDP, the advantage estimate discounts by the discount factor raised to the sojourn time, giving longer inter-request intervals more discount.The passage contrasts this with a fixed discount per transition.Source: 2) PPO Mechanics at a Single BS · p. 7
  • The value network minimizes squared prediction error against a bootstrapped target formed from immediate reward and discounted value of the next state.Equations (9)–(11) together define the local update each BS executes at every decision epoch.Source: 2) PPO Mechanics at a Single BS · p. 7

B. Meta-Reinforcement Learning Across Base Stations

  • A pretrained shared policy provides a warm start across traffic conditions, intended to reach performance comparable to an environment-specific optimal policy after a few local updates.The initialization is learned with a MAML framework rather than random neural-network weights.Source: B. Meta-Reinforcement Learning Across… · p. 7

1) Meta-Learning Objective:

  • MAML fine-tunes shared θ with a few BS-local gradient steps, then updates θ to reduce average post-adaptation loss across BSs.The adapted policy is tested on its BS without further weight updates before the outer-loop update.Source: 1) Meta-Learning Objective · p. 7

2) Meta-Gradient Estimation:

The paper estimates the shared-policy meta-gradient from a subset of BSs because evaluating every BS at each meta-iteration is costly. It clusters BSs by local-gradient similarity and samples across clusters, establishing lower estimator variance than uniform sampling.

2) Meta-Gradient Estimation:

  • Because evaluating local adaptations for all N BSs at every meta-iteration is costly, the method approximates the meta-gradient using a subset of K ≪ N BSs.The subset is used in the outer-loop update of the shared parameters.Source: 2) Meta-Gradient Estimation · pp. 7–8

1) Motivation:

  • Similar traffic regimes tend to produce similarly directed local gradients, while uniform sampling can omit regimes and yield a high-variance, unrepresentative estimate.The passage contrasts dense urban and lightly loaded suburban BSs as examples of differing regimes.Source: 1) Motivation · p. 8

2) Clustering Procedure:

  • At reclustering intervals, BSs are grouped by cosine similarity of unit-normalized gradients, then each cluster receives draws proportional to its size.Within each interval, BSs are sampled uniformly within every cluster; reclustering less often keeps overhead low while tracking gradual traffic changes.Source: 2) Clustering Procedure · p. 8

D. Convergence Analysis: Gradient-Clustered Sampling Versus Random Sampling

  • Both sampling schemes are unbiased, but gradient-clustered sampling has strictly smaller variance, and the convergence guarantee of carries over with its variance term reduced.The variance gap is given in closed form by the between-cluster variance of local gradients.Source: D. Convergence Analysis:… · p. 8

1) Setup and Sampling Schemes:

  • The schemes use the same budget of m BS evaluations per meta-iteration, with clustered sampling forming a cluster-size-weighted average of local gradients.The clustered budget is divided among clusters in proportion to their sizes.Source: 1) Setup and Sampling Schemes · pp. 8–9

2) Assumptions:

The convergence analysis adopts bounded-gap, smoothness, Hessian, gradient-dispersion, and per-BS estimation-noise assumptions for the finite BS population.

2) Assumptions:

  • The meta-loss is bounded below, local losses are twice continuously differentiable and smooth, and their Hessians are Lipschitz; the inner-loop step size is restricted to α ∈ (0, 1/(6L)].These conditions support the smoothness surrogate and stochastic meta-step-size rule used from.Source: 2) Assumptions · p. 9
  • The analysis assumes bounded dispersion of BS meta-gradients around the population meta-gradient for every θ.This is stated as the finite-population analogue of [8, Assumption 4.5].Source: 2) Assumptions · p. 9
  • Each sampled BS’s stochastic meta-gradient is assumed conditionally unbiased with bounded estimation noise; an ϵ-FOSP satisfies E[∥∇F(θϵ)∥] ≤ ϵ.The noise condition is identified as analogous to [8, Assumption 4.6].Source: 2) Assumptions · p. 9

3) Unbiasedness:

Both sampling estimators are unbiased for the population meta-gradient. The gradient population’s variance decomposes into within-cluster and between-cluster components.

3) Unbiasedness:

  • Both random and gradient-clustered sampling produce unbiased estimates of the population meta-gradient.For clustered sampling, the clusters together contain every BS exactly once.Source: 3) Unbiasedness · p. 9

4) Variance Decomposition of the Gradient Population:

  • The exact ANOVA decomposition splits population gradient variance into size-weighted within-cluster variance and size-weighted variance of cluster means around the population mean.The decomposition holds for any partition of the BS population into clusters.Source: 4) Variance Decomposition of the… · p. 9
  • For m independent uniform BS draws, the random-sampling estimator’s variance follows from the single-draw population variance, with averaging scaling variance by 1/m.The proof uses independence of draws and the 1/m² scaling of the variance of their sum.Source: 4) Variance Decomposition of the… · p. 10

6) Theorem 2 (Variance of Gradient-Clustered Sampling):

Theorem 2 derives the clustered-sampling estimator's variance from independent draws within each cluster and the allocation of draws across clusters.

6) Theorem 2 (Variance of Gradient-Clustered Sampling):

7) Theorem 3 (Exact and Universal Variance Reduction):

Gradient clustering reduces estimator variance by the between-cluster component, and the result connects to steadier meta-updates and a lower convergence floor under stated assumptions.

7) Theorem 3 (Exact and Universal Variance Reduction):

  • The exact variance reduction is σ²_B/m, and it is strict when at least one cluster mean differs from the population mean.The reduction grows with separation between cluster means; gradient-similarity grouping makes clusters internally homogeneous and mutually distinct.Source: 7) Theorem 3 (Exact and Universal… · p. 10

8) Corollary 1 (Batch-to-Batch Stability):

  • With clustered sampling, successive meta-updates differ by the within-cluster term 2σ²_W/m, while random sampling also incurs the between-cluster term 2σ²_B/m.Clustered sampling remains unbiased and samples every BS over time.Source: 8) Corollary 1 (Batch-to-Batch… · pp. 10–11

9) Proposition 1 (ϵ-FOSP Complexity Under Clustered BS Sampling):

  • Under the stated assumptions, clustered sampling preserves MAML's O(∆̄L/ϵ²) iteration complexity while replacing σ²/m with σ²_W/m in the variance term.The resulting accuracy floor is strictly lower at fixed budget m when σ²_B > 0.Source: 9) Proposition 1 (ϵ-FOSP Complexity… · p. 11

10) Discussion:

The discussion links variance reduction to a lower irreducible convergence floor without changing the iteration rate, with the benefit depending on BS heterogeneity and clustering quality.

10) Discussion:

  • Clustered sampling preserves the O(∆̄L/ϵ²) iteration rate but strictly lowers the irreducible expected-gradient floor when σ²_B > 0.The floor difference is governed by the between-cluster variance; finite rollout noise remains part of the floor.Source: 10) Discussion · p. 11
  • The advantage vanishes for identical BS gradients or uninformative clusters, while heterogeneous traffic regimes can create distinct cluster means and larger gains.The re-clustering interval trades overhead against assignment staleness, which can shrink the realized between-cluster variance.Source: 10) Discussion · p. 11
  • At fixed sampling budget, additional meta-iterations cannot remove the floor imposed by finite BS sampling and rollout length; clustered sampling lowers it when σ²_B > 0.Source: 10) Discussion · p. 11

VI. EXPERIMENTAL SETUP AND RESULTS

  • The experimental setup implemented the multi-BS environment and learning agents in Python 3 using TensorFlow.Source: §VI · p. 12

A. Configuration and Parameters

The section describes the heterogeneous BS simulation, comparison methods, and evaluation criteria, then reports faster, more stable convergence with gradient-clustered sampling than with uniform random sampling.

A. Configuration and Parameters

  • The simulation uses 62 BSs (60 for meta-training, two held out), varying Zipf skewness and Poisson request rates, with 50 content types and cache capacity 10,000.Content lifetime, size, and importance are also varied.Source: A. Configuration and Parameters · p. 12

B. Baselines

  • The baselines are gradient-clustered MAML (MGCS), uniformly sampled MAML (MAML-RS), learning from scratch, and single-source transfer learning.MGCS versus MAML-RS isolates the effect of changing BS selection while retaining the MAML outer loop.Source: B. Baselines · p. 12

C. Evaluation Criteria

  • Evaluation tracks meta-loss convergence, empirical meta-gradient variance, held-out-BS adaptation loss and reward, and average-reward level and stability.The held-out traffic configurations are not used during meta-training.Source: C. Evaluation Criteria · pp. 12–13

1) Meta-Training Convergence:

  • Meta-loss: gradient-clustered sampling finishes at 1447 ± 122 versus 1961 ± 300 under uniform random sampling, roughly 26% lower.It drops below 1800 within 20 meta-iterations; its cluster-spanning samples also produce a more stable descent direction.Source: Figure 2 · p. 13, 1) Meta-Training Convergence · p. 13

2) Adaptation to an Unseen Base Station:

On held-out BSs, gradient-clustered sampling matches the adaptation quality of uniformly sampled MAML while requiring fewer training steps. Meta-learning's advantage over training from scratch is most evident on the difficult task; the conclusion identifies fixed clustering settings as a limitation.

2) Adaptation to an Unseen Base Station:

VII. CONCLUSION

  • The cluster count and re-clustering interval are treated as hyperparameters; adaptive selection based on observed gradient dispersion remains for future study.The variance-reduction principle has only been applied to edge caching in this work.Source: §VII · p. 15

Related papers

Loading 2609.16370v1…