PaperlayerView exact revision on arXiv

Source-linked AI summary

Taylor Representations for Model-Free RL in Networked MDPs

Salah Chikhi, Abdelhaq Chaoui, Asuman Ozdaglar, Saurabh Amin

arXiv:2610.05456v1stat.MLcs.LGmath.OC

TL;DR

Networked MDP control must contend with unknown transitions and state–action spaces that grow rapidly with agent count. The paper derives scalable local Taylor critics from expected-reward smoothness and graph locality, then builds model-free linear and neural actor–critic methods. Experiments report competitive performance and scaling, while the linear critic guarantee depends on well-conditioned LSTD systems.

  • Problem

    Unknown transition dynamics and rapidly growing joint state–action spaces make scalable control difficult, while continuous local critics still need a representation that can be learned.

    Source: §1 · p. 1

  • Method

    The paper uses smooth expected future local rewards and graph-local derivative decay to build local Taylor critics, then learns their coefficients with linear LSTD or neural TD in a model-free actor–critic.

    Source: §1 · p. 2, §2 · p. 3, §3.1 · p. 4

  • Results

    LTC/NTC match or outperform baselines across the reported LQR configurations and Kuramoto experiments, and scalability tests show predicted O(N) runtime with near-reference performance up to N = 1000.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The approach provides competitive, scalable continuous networked-MDP control without requiring the transition-model access used by prior constructive spectral approaches.

    Source: §1 · p. 1, §7 · p. 9

    The LTC guarantee assumes well-conditioned LSTD systems, and its coefficient function class is unknown in advance, requiring empirical feature selection.

    Source: §7 · p. 9

Abstract

from arXiv · show

In Networked Markov Decision Processes, transition dynamics are often unknown and the state--action space grows rapidly with the number of agents. In this setting, Taylor representations naturally approximate $Q$-functions, but a naive order-$n$ expansion over $N$ agents requires $\Theta(N^n)$ coefficients. We justify these expansions under smooth expected future local rewards with controlled derivatives. Under this condition, finite-speed information propagation and discounting imply that local-critic Taylor coefficients decay exponentially with the graph distance to the farthest agent involved. Discarding distant-agent coefficients and marginalizing then yield scalable local Taylor representations with a bound controlled by graph locality. Building on these representations, we propose a scalable model-free actor--critic algorithm, establishing finite-sample critic and near-stationarity guarantees for a linear LSTD critic. We then introduce a more expressive neural TD parameterization. Unlike prior constructive spectral methods, our approach covers settings without access to a known local dynamics map, such as hidden switched linear--quadratic regulation. Across three control benchmarks, our method matches or outperforms spectral baselines while scaling efficiently to large graphs.

1 INTRODUCTION

Networked MDPs combine unknown transition dynamics with rapidly growing state–action spaces, while continuous local critics still require representation and learning. The paper uses reward smoothness and graph locality to construct scalable Taylor critics and a model-free actor–critic.

1 INTRODUCTION

  • Unknown transitions and rapidly growing joint state–action spaces make large-scale networked control challenging, and localization alone leaves continuous local critics to represent and learn.The paper targets the fully model-free setting, where only rollout data are available.Source: §1 · pp. 1–2
  • Prior constructive spectral approaches typically assume known local dynamics with additive Gaussian noise, excluding settings such as switched LQR with an unobserved global mode.That setting has state- and action-dependent noise shared across agents and temporally correlated through the Markov mode.Source: §1 · pp. 1–2
  • Under smooth expected future local rewards, finite information propagation and discounting make Taylor derivatives involving distant agents decay exponentially with graph distance.This coefficient-level locality supports discarding distant-agent terms and marginalizing to obtain local critics.Source: §1 · p. 2
  • The resulting Taylor Centroid approach combines a scalable linear critic with finite-sample and near-stationarity guarantees, then extends the coefficient parameterization to neural networks.Unlike prior constructive spectral methods, it needs no known or structurally specified transition kernel.Source: §1 · p. 2

2 PROBLEM SETTING

The paper studies cooperative control over networked agents with local rewards and discounted objectives. Its expected-reward regularity condition supports Taylor expansion and, with finite graph propagation, exponential decay of critic derivatives.

2 PROBLEM SETTING

  • The setting places agents on an undirected graph and uses factorized transitions and policies, with an optional global context that affects local transitions without transmitting physical information between distant agents.Rewards are averages of bounded local rewards, and the objective uses a discount factor.Source: §2 · p. 2
  • The initial distribution is fixed independently of policy parameters, and the cooperative objective is optimized with a local and global actor–critic structure.The policy-independent initial distribution is part of the stated setup.Source: §2 · p. 3
  • Assumption 1 requires smooth expected future local rewards, allowing Q-functions to be differentiable enough for Taylor expansions and, with finite propagation, yielding exponential derivative decay.The uniform derivative-bound formulation can be weakened to discounted summability; the paper retains the uniform version for simplicity.Source: §2 · p. 3
  • This regularity concerns the future reward sequence defining Q, unlike prior polynomial approximations that rely on regularity of the Bellman operator.The paper reports that its condition holds for hidden switched LQR, while the converse implication does not generally hold.Source: §2 · p. 3

3 TAYLOR CENTROID CRITICS

The section turns graph-local derivative decay into Taylor critics whose size and approximation error are controlled by local neighborhoods. It then develops linear LSTD and neural TD coefficient estimators, with different estimation guarantees and trade-offs.

3.1 TAYLOR EXPANSIONS AND SCALABILITY

  • Expected reward regularity makes Taylor derivatives involving agents at distance d decay as O(ρ^d), because finite propagation delays their influence and discounting attenuates it.This coefficient-level decay sharpens value-localization results that do not characterize Taylor coefficients.Source: §3.1 · p. 4
  • Discarding distant derivatives and marginalizing distant variables yields a local Taylor representation centered at the nearest selected centroid, whose cluster index can encode global-state information.The resulting coefficients depend on local states and actions rather than distant variables.Source: §3.1 · p. 4
  • Under Assumption 1, the local Taylor representation combines a graph-localization error with a Taylor remainder and has an N-independent monomial count for fixed neighborhood radius and order.The localization error decreases with radius, while a larger radius also expands the Taylor remainder and adds coordinates.Source: §3.1 · pp. 4–5

3.2 LINEAR AND NEURAL TAYLOR CENTROID CRITIC ESTIMATION

  • LTC shares its coefficient map across clusters, while localization reduces coefficient estimation to a local neighborhood and richer order or radius can make estimation harder.The shared map avoids fitting separate coefficients for potentially rarely visited clusters.Source: §3.2 · p. 5
  • LTC fits marginalized Taylor coefficients with linear features and LSTD, with a finite-sample error bound separating approximation, estimation, and statistical error.The bound is conditional on well-conditioned LSTD systems; increasing centroid count, order, or radius can create estimation trade-offs.Source: §3.2 · pp. 5–6
  • NTC replaces affine coefficient maps with neural networks trained by semi-gradient TD, preserving the structural representation guarantees but not LTC’s finite-sample estimation guarantees.The neural extension is motivated when linear coefficient misspecification dominates.Source: §3.2 · p. 6

4 LEARNING AND POLICY OPTIMIZATION

The section constructs localized policy-gradient estimates using critic-derived one-step TD advantages and establishes a near-stationarity guarantee. The guarantee separates localization, critic, rollout-truncation, and actor-sampling errors.

4 LEARNING AND POLICY OPTIMIZATION

  • The local advantage estimate is the one-step TD residual formed from the reward and the critic's next- and current-state values.The actor uses additional rollouts to form its empirical policy-gradient estimate.Source: §4 · p. 6
  • The policy-gradient estimator retains advantages only from agents in the combined critic and policy neighborhoods, removing dependence on N.It uses localized one-step TD advantages rather than localized Q-values.Source: §4 · p. 6
  • Theorem 3 gives a near-stationarity guarantee under a smooth objective and bounded estimation conditions, with an explicit error floor.Its bound separates errors tied to localization radius, critic accuracy, rollout length, and actor sample size.Source: §4 · p. 7
  • The truncation term accounts for each discarded agent with its own graph-distance decay factor, and can be O(N^-1/2) when the sum of those factors is O(1).This can improve on assigning every discarded agent the same worst-case decay factor.Source: §4 · p. 7

5 ALGORITHM AND COMPLEXITY

Taylor Centroid Actor–Critic learns LTC or NTC critics from rollout data and updates the policy using additional rollouts. LTC iterations scale linearly with the number of agents and are largely parallelizable.

5 ALGORITHM AND COMPLEXITY

  • Taylor Centroid Actor–Critic uses rollout data to fit either a linear LTC critic or a neural NTC critic.Each iteration refits centroids, collects critic rollouts, fits critics across agents, then collects actor rollouts and updates the policy.Source: §5 · p. 7
  • The algorithm specifies Taylor order, critic and policy locality radii, centroid count, and epoch count as inputs.It uses stepsizes η_m = η(m+1)^-1/2.Source: §5 · p. 7
  • LTC iteration time scales linearly with N and is largely parallelizable across agents and samples.The centroids are refit each iteration to track policy-induced shifts in the state distribution.Source: §5 · p. 7

6 EXPERIMENTS

Across the three control benchmarks, Taylor Centroid methods match or outperform the reported baselines, including spectral methods. Experiments also report linear runtime scaling and competitive performance through 1,000 agents.

6 EXPERIMENTS

  • The evaluation compares LTC and NTC with random Fourier features, Spectral-SAC, CDCPG, MLP, and GNN baselines, with RF and CDCPG not directly applicable to hidden switched LQR.The benchmarks are networked LQR, switched LQR, and nonlinear Kuramoto synchronization.Source: §6 · p. 7
  • On LQR, critic error improves sharply from Taylor order 1 to 2 and saturates at order 3; Kuramoto critic error improves nearly systematically with order.On switched LQR, NTC significantly improves over LTC while predicting mode-conditioned value without observing the mode.Source: Figure 1 · p. 8, §6 · p. 8
  • Taylor Centroid matches or outperforms baselines across LQR, switched LQR, and Kuramoto, and matches or outperforms prior spectral baselines across all three environments.On Kuramoto, NTC matches or outperforms LTC and both methods converge near the greedy controller's value.Source: §6 · p. 8, Figure 2 · p. 9
  • LTC shows O(N) runtime scaling and near-reference performance through N = 1000, and outperforms RF in runtime in this experiment.Figure 3 reports runtime per epoch on an H200 GPU and relative optimality gap as functions of N.Source: §6 · p. 8, Figure 3 · p. 9

7 CONCLUSION AND LIMITATIONS

The conclusion presents graph locality as the basis for scalable local critics without transition-model access and identifies limitations in the guarantees, function-class control, and implementation. The benchmarks span known-model LQR, hidden-regime switched LQR, and nonlinear Kuramoto dynamics.

7 CONCLUSION AND LIMITATIONS

  • The LTC finite-sample guarantee assumes well-conditioned LSTD systems, and characterizing that condition requires complementary analysis.The coefficient function class is unknown a priori, preventing sharp control of linear-feature misspecification and requiring empirical feature selection.Source: §7 · p. 9
  • The policy-optimization guarantee reaches a stationary point, not necessarily a global maximum; K-medians can also be challenging in high-dimensional state spaces.Extending finite-sample critic guarantees to NTC remains a future direction.Source: §7 · p. 9

B DEFINITION OF THE ENVIRONMENTS

  • The three environments test complementary settings: exact quadratic value in networked LQR, hidden operating regimes in switched LQR, and nonlinear graph-coupled dynamics in Kuramoto.Switched LQR tests whether the representation can exploit regime structure without transition-model access.Source: B DEFINITION OF THE ENVIRONMENTS · p. 13

B.1 ENVIRONMENT 1: NETWORKED LINEAR–QUADRATIC REGULATION

  • Networked LQR provides an exact quadratic value under the policy class and a known local dynamics map with additive state-independent Gaussian noise.This makes it a controlled Taylor test and a setting suited to the constructive spectral baselines, while Taylor Centroid uses sampled trajectories.Source: §B.1 · p. 14

B.2 ENVIRONMENT 2: SWITCHED LQR WITH AN UNOBSERVED OPERATING MODE

The switched-LQR benchmark tests Taylor Centroid with persistent operating regimes that the controller cannot observe. The section also describes a nonlinear Kuramoto benchmark and points to separate training and architecture details.

B.2 ENVIRONMENT 2: SWITCHED LQR WITH AN UNOBSERVED OPERATING MODE

  • A persistent mode switches among graph-local dynamics on the sparse ring, changing physical behavior without changing the interaction graph.Episodes last T = 20; the mode is redrawn with probability h.Source: §B.2 · p. 14
  • The mode changes about twice per 20-step trajectory on average, and neither actor nor critic observes it.The mode follows an autonomous Markov chain.Source: §B.2 · pp. 14–15
  • Constructive RF and CDCPG baselines are inapplicable because they require a known local dynamics map, which hidden-mode marginalization does not provide.The spectral approaches use explicit random features under structured additive noise.Source: §B.2 · p. 15

B.3 ENVIRONMENT 3: KURAMOTO OSCILLATOR NETWORK

  • The Kuramoto benchmark tests local Taylor representations with nonlinear sinusoidal interactions and a non-quadratic, non-polynomial critic.Its oscillator-synchronization task also stress-tests the regime mechanism, and the number of clusters matters.Source: §B.3 · p. 16

C TRAINING AND EVALUATION PROTOCOLS

C.1 EVALUATION PROTOCOLS

The evaluation compares Taylor Centroid with networked-MARL and generic critics, then probes derivative locality, centroid choices, Taylor order, and representative policy behavior.

C.1 EVALUATION PROTOCOLS

  • LTC and NTC are compared with five baselines on LQR, switched LQR, and Kuramoto, using critic-fit, policy-performance, and scalability evaluations.Policy references include lower bounds on the corresponding optima in switched LQR and structured static-feedback settings.Source: §C.1 · pp. 16–17

C.3 METHOD-SPECIFIC DETAILS

  • LTC uses local Taylor monomials around K-medians centroids with an LSTD fit; NTC keeps the basis but learns coefficients with a neural network trained by semi-gradient TD.The critics use monomials of orders 1, 2, or 3 supported on the κ-hop neighborhood.Source: §C.3 · p. 18

C.4 HYPERPARAMETER SELECTION

  • The reported tables give selected hyperparameters for LQR, switched LQR, and Kuramoto; the Kuramoto table varies choices by Taylor order and identifies LTC’s LSTD ridge.The captions identify selected (η, K) pairs for the two LQR settings.Source: Table 3 · p. 19, Table 4 · p. 20, Table 5 · p. 20

D.1 EMPIRICAL EXPONENTIAL DERIVATIVE DECAY

  • Across all three environments, nonzero-order sensitivities decrease approximately geometrically with graph reach, showing locality in individual Taylor coefficients as well as in Q-function localization.Figure 4 compares measured derivative magnitudes with the theoretical rate.Source: §D.1 · p. 20, Figure 4 · p. 21

D.2 CENTROIDS AND LOCAL APPROXIMATION QUALITY

  • For LTC, adding centroids first improves held-out fit but eventually worsens it as estimation error dominates; this turnover comes earlier at higher Taylor orders.NTC shows less degradation at large K, consistent with its neural coefficient map accommodating variation across centroids.Source: §D.2 · p. 22
  • LQR critic error increases with distance from the assigned centroid, while higher Taylor order typically attenuates this dependence.The two Figure 5 panels illustrate the competing effects of local-region size and coefficient representation.Source: §D.2 · p. 22, Figure 5 · p. 22

D.3 TAYLOR ORDER ABLATION

  • On LQR and switched LQR, order 2 yields a much smaller policy gap than order 1 when κπ ≥2, while order 3 provides no improvement.Under the stated affine-mean Gaussian policies, the critic is quadratic, conditionally on the hidden mode for switched LQR.Source: §D.3 · p. 22, Figure 6 · p. 23

D.4 EPISODE TRAJECTORIES ON THE THREE ENVIRONMENTS

  • Representative trajectories show rapid stabilization on LQR, recovery after unobserved mode changes on switched LQR, and maintained frequency synchronization on Kuramoto.These trajectories are a qualitative overview from one seed.Source: §D.4 · p. 22, Figure 7 · p. 23

D.5 THE LATENT MODES APPEAR AS DATA-DRIVEN OPERATING REGIMES.

The appendix examines data-driven regime structure and the expected-reward regularity condition behind local Taylor representations. Rollout tests and sufficient-condition results show how that condition is assessed and when it holds.

D.5 THE LATENT MODES APPEAR AS DATA-DRIVEN OPERATING REGIMES.

  • 0.62 after training versus 0.35 without control, mode separation rises as learned centroids align with recurrent state regions associated with hidden regimes.The centroids are fitted from physical states alone, and the observed state-distribution shift motivates refitting them during training.Source: §D.5 · p. 24

E.1 INTERPRETATION

  • Assumption 1 requires that a small perturbation of the current physical state not create arbitrarily sharp variations in agent i’s expected local reward after finite steps, although nearby configurations may evolve differently.Source: §E.1 · p. 25

E.2 A ROLLOUT-BASED STRESS TEST

  • Across LQR, switched LQR, and Kuramoto, expected-reward sensitivities stayed controlled over tested horizons, and derivatives involving more distant agents were typically smaller.The finite rollout study is a stress test, not formal verification of a condition uniform over states, policies, agents, and horizons.Source: Figure 9 · p. 25, §E.2 · p. 26

E.3 A WEAKER DISCOUNTED-SUMMABILITY CONDITION

  • Discounted-summable regularity allows finite-horizon sensitivities to grow when discounting dominates, while retaining an N-independent representation size for bounded graph degree and fixed locality and Taylor order.Its decay rate is determined by the discounted sum of the regularity envelope.Source: §E.3 · pp. 26–27

E.4.1 SMOOTH STOCHASTIC DYNAMICS.

  • Smooth stochastic dynamics satisfy Assumption 1 with an N-independent constant under the stated smoothness, noise-density, and bounded-degree conditions.For isotropic noise, the sufficient certificate may scale as O(σ^-q) at derivative order q as σ approaches zero; this need not describe the true critic derivatives.Source: §E.4.1 · p. 28

E.4.2 SMOOTH LINEAR TRANSITION MODELS.

  • Smooth linear transition models also satisfy Assumption 1 with an N-independent constant when their local rewards and current-state feature maps have bounded derivatives and graph degree is bounded.This includes the finite-dimensional spectral-model setting described in the passage.Source: §E.4.2 · p. 29

E.4.3 COMPARISON BETWEEN SMOOTHNESS NOTIONS

This section contrasts expected-reward regularity with broader Bellman-operator smoothness notions, then turns to proof ingredients for the main results. The comparison includes a switched-LQR separation and a boundary for applying standard Mild smoothness to hybrid contexts.

E.4.3 COMPARISON BETWEEN SMOOTHNESS NOTIONS

  • If local rewards are Cq with uniformly bounded derivatives and the augmented MDP is ν-Mildly Smooth for ν > q, then Assumption 1 holds at order q.The one-order shift follows from the paper's Cq convention.Source: §E.4.3 · p. 30
  • Expected-reward regularity controls the particular expected reward sequence, unlike Mild smoothness, which controls Bellman updates on every bounded input; switched LQR can satisfy the former at every finite order without satisfying the latter.The example uses a continuous switching context whose sign selects alternating linear regimes.Source: §E.4.3 · pp. 30–31
  • For a hybrid Markov state with finite-valued global context, standard Mild smoothness is not directly defined without specifying how its smoothness space extends to the discrete coordinate.Assumption 1 instead differentiates only with respect to the continuous physical state and requires bounds uniformly over context.Source: §E.4.3 · p. 31

F PROOFS OF THE MAIN-TEXT RESULTS

  • The proof section establishes finite-propagation and decay, approximation bounds, the finite-sample LTC–LSTD guarantee, and actor-update results; its conditioning convention treats current critic rollouts as independent given the pre-rollout history.Global-state clusters can inject information into local critics, and discarding distant derivatives supports the local Taylor representation.Source: F PROOFS OF THE MAIN-TEXT RESULTS · p. 31

F.1 FINITE PROPAGATION AND EXPONENTIAL DECAY

This section establishes finite-range information propagation and spatial decay to justify localized Taylor approximations, and analyzes their use in LTC–LSTD and localized policy gradients. It also addresses localized actor errors, end-to-end policy optimization, computational scaling, and proofs of additional approximation and smoothness results.

F.1 FINITE PROPAGATION AND EXPONENTIAL DECAY

  • Finite propagation restricts agent i’s conditional state–action law at time t to initial coordinates within radius R_t = t(1 + κπ), supporting spatial decay of local dependence.The spatial-decay argument uses the fact that distant initial coordinates cannot affect the local reward before information reaches the agent’s neighborhood.Source: §F.1 · pp. 32–33

F.2 LOCALIZED TAYLOR CENTROID APPROXIMATION

  • The localized Taylor-centroid critic’s error is controlled by localization and Taylor-remainder terms, with a distribution-weighted bound using the expected state-dependent remainder.The K-medians objective reduces an upper bound on average Taylor remainder under the policy-induced state distribution.Source: §F.2 · pp. 35–36

F.3 FINITE-SAMPLE ANALYSIS OF LTC–LSTD

  • The LTC–LSTD analysis combines a per-agent error bound with a union bound to obtain the stated simultaneous guarantee with probability at least 1 − δ.The per-agent failure probability is at most δ/N.Source: §F.3 · p. 40

F.4 LOCALIZED POLICY GRADIENT WITH ONE-STEP TD ADVANTAGES

  • The policy-gradient identity permits action-independent baselines, while localized advantages make far reward streams contribute zero to the local gradient.For a far reward stream, the localized advantage is independent of the current action.Source: §F.4 · pp. 40–41
  • The localized actor analysis treats spatial truncation bias, one-step TD advantage error, and bias in the localized finite-horizon estimator.It combines spatial-localization, critic-approximation, and finite-horizon bounds for the actor estimator.Source: §F.4 · pp. 41–43

F.5 END-TO-END POLICY OPTIMIZATION

  • Conditioned on critic certificates, fresh actor rollouts yield simultaneous gradient control across iterations with probability at least 1 − δ, supporting the end-to-end optimization guarantee.The proof uses smoothness and ascent updates to bound the minimum policy-gradient norm across iterations.Source: §F.5 · pp. 44–45

F.6 COMPUTATIONAL COMPLEXITY

  • For fixed local parameters, one LTC iteration scales linearly with the number of agents, and critic and actor computations are largely parallelizable.Rollout generation is also linear in the number of agents for fixed local interaction and policy radii.Source: §F.6 · p. 46

G PROOFS OF THE ADDITIONAL APPENDIX RESULTS

  • The appendix outlines proofs for discounted-summability and generalized Taylor Centroid approximation, stochastic and spectral transition-model conditions, and comparison with Mild smoothness.These are identified as auxiliary results from Appendix E.Source: G PROOFS OF THE ADDITIONAL APPENDIX… · p. 47

G.1 DISCOUNTED-SUMMABILITY RESULTS

The appendix establishes discounted control of distant Taylor coefficients and gives conditions under which local expected rewards have bounded derivatives. It also relates these conditions to mild smoothness, including a setting where that property fails.

G.1 DISCOUNTED-SUMMABILITY RESULTS

  • Uniform convergence of the derivative series justifies term-by-term differentiation of the marginalized critic.Differentiation under its probability integral is justified as in Lemma F.4.Source: §G.1 · pp. 47–48
  • Before time τ_d, agent i’s expected reward cannot depend on every state coordinate in β; the resulting envelope Λ_d decreases to zero as graph distance grows.The proof also uses that τ_d is nondecreasing in d.Source: §G.1 · p. 47
  • The generalized coefficient envelope changes coefficient bounds but leaves the Taylor monomial count unchanged.The proof combines the distance-shell contributions with the preceding bound to establish Equation 24.Source: §G.1 · p. 48

G.2 SMOOTH STOCHASTIC AND SPECTRAL TRANSITION MODELS

  • For bounded-degree factorized transitions, local derivative bounds yield expected-reward derivatives bounded independently of network size, and uniform modewise bounds extend the result to finite-mode systems.The derivative bound depends on the order, degree bound, and local bounds.Source: §G.2 · p. 49

G.3 RELATION TO MILD SMOOTHNESS

  • In the switched two-mode setting, the smoothness bound holds uniformly over admissible policies without requiring policy smoothness.After the first transition, the future policy is absorbed into a bounded continuation value.Source: §G.3 · p. 51
  • The augmented MDP can fail mild smoothness: its Bellman image is discontinuous in the continuous context coordinate z.The proof concludes it is not mildly smooth at any positive order.Source: §G.3 · p. 51

H NEURAL NETWORK IMPLEMENTATION DETAILS

The implementation appendix describes neural Taylor critics and two local-value baselines. NTC retains the local Taylor structure while learning nonlinear coefficient maps; the baselines use neighborhood MLPs or graph message passing.

H.1 NTC IMPLEMENTATION

  • NTC replaces LTC’s affine coefficient map with a neural map while keeping an order-n polynomial in local state around the current centroid.The coefficients may depend nonlinearly on local actions, operating regime, and observable context.Source: §H.1 · p. 51
  • Each agent has an independent coefficient network that receives coefficient context, not raw current state; TD training uses SARSA targets and a periodically synchronized target network.The state dependence enters through the explicit Taylor feature vector, so increasing Taylor order changes the network’s output dimension rather than its input dimension.Source: §H.1 · pp. 51–52
  • NTC estimates state value by marginalizing its action-value critic under the current policy, using 16 independent action samples in the implementation.With independent samples, this Monte Carlo marginalization is unbiased; LTC needs only the expected action because its critic is affine in actions.Source: §H.1 · p. 52

H.2 LOCAL MLP CRITIC

  • The local MLP baseline directly estimates local state value from the same κ-hop neighborhood as the Taylor critics, with scalar-ring input dimensions 3, 5, and 7 for κ = 1, 2, and 3.Its policy-optimization networks use two hidden layers and tanh activations.Source: §H.2 · p. 52

H.3 GRAPH NEURAL-NETWORK CRITIC

  • The GNN baseline uses mean aggregation and shared weights across agents, with message passing configured so local value depends only on the κ-hop neighborhood.For Kuramoto, the random-feature map is replaced by a two-layer width-64 GNN with receptive field min{κ, 2}.Source: §H.3 · p. 53
Loading 2610.05456v1…