PaperlayerView exact revision on arXiv

Source-linked AI summary

BE-CTDE: Blockchain-Enabled Multi-Agent Reinforcement Learning with Anomaly Detection and Token Economics

Chih-Wei Huang, Hsin-Chieh Lee, Yun-Chen Yu, Ibrahim Althamary, Rongxing Lu, Phone Lin

arXiv:2610.34121v1cs.MAcs.CRcs.LGcs.NI

TL;DR

CTDE training can leave data untraceable, limit accountability, expose a centralized single point of failure, and permit policy disobedience, while purely on-device training loses global context. BE-CTDE verifies executed trajectories against shared-policy recommendations, applies token incentives, and uses retained balances to weight agents’ contributions to the shared policy update. Under majority adversaries, at N = 64 and 15% obstacles, BE-CTDE’s CO is 24.00% versus Lite’s 37.10% (13.10 pp; 35.3% lower), while SR is 95% versus 89%.

  • Problem

    CTDE training can leave data untraceable, limit accountability, expose a centralized single point of failure, and permit policy disobedience, while purely on-device training loses global context.

    Source: §I · p. 1

  • Method

    BE-CTDE verifies executed trajectories against shared-policy recommendations, applies token incentives, and uses retained balances to weight agents’ contributions to the shared policy update.

    Source: §I · p. 2, A. System Architecture · p. 4

  • Results

    Under majority adversaries, at N = 64 and 15% obstacles, BE-CTDE’s CO is 24.00% versus Lite’s 37.10% (13.10 pp; 35.3% lower), while SR is 95% versus 89%.

    Source: B. Performance Under Adversarial… · p. 9

  • Takeaways & Limitations

    MAPF experiments characterize how behavioral verification and token feedback can support accountable cooperative learning.

    Source: §VII · p. 11

    Committed records do not establish sensor truth or prove that the committed model produced an action.

    Source: B. On-Chain and Off-Chain Function… · p. 8

Abstract

from arXiv · show

Multi-agent reinforcement learning supports cooperative decision-making in Internet of Things systems, but strategic deviations can disrupt coordination and undermine confidence in the training process. This paper proposes Blockchain-Enabled Centralized Training and Decentralized Execution (BE-CTDE), which uses blockchain as an active training-management layer. A behavioral verifier compares executed trajectories with shared-policy recommendations; its flags drive a token mechanism combining slashing, burning, rewards, and compensation. Retained balances determine each agent's contribution to the global policy update, connecting auditable behavior to learning influence. We derive a one-round incentive condition that accounts for action-dependent compensation and the payoff from deviation, and a finite expected exclusion-time bound under explicit conditional detection and accounting assumptions, showing perfect detection is unnecessary. An exact finite-population capacity bound characterizes the limits of the population-relative detector. In multi-agent path finding experiments, BE-CTDE outperforms a state-of-the-art learning-based path planner with substantially higher success rates under attack, even when adversaries form the majority. It also outperforms partially defended baselines in all configurations, with collision rates up to 35% lower than under detection without economic consequences and up to 43% lower than under conventional stake slashing. Together, these results characterize how behavioral verification and token feedback can support accountable cooperative learning.

I. INTRODUCTION

The introduction motivates accountable cooperative learning and presents BE-CTDE as a blockchain-managed training loop that links verified behavior to policy influence. It also situates the approach among blockchain–AI systems and reports improved MAPF outcomes under attack.

I. INTRODUCTION

  • Traditional CTDE faces untraceable data, limited accountability, a centralized single point of failure, and policy disobedience, while purely on-device training loses cooperative global context.The introduction calls for distributed management of learning without sacrificing centralized-training coordination benefits.Source: §I · p. 1

B. AI and Blockchain Integration

  • Prior blockchain-integrated MARL work includes tracking, path planning, and resource management, but LearnChain records updates without changing a deviant agent’s influence on the shared model.The identified gap is carrying behavioral evidence back into the policy update, rather than auditability alone.Source: §I · p. 2, B. AI and Blockchain Integration · p. 2

I. INTRODUCTION

  • A trajectory verifier compares executed behavior with shared-policy recommendations, while slashing, burning, rewards, and compensation affect balances that weight learning contributions.The framework avoids requiring gradient-space similarity between heterogeneous agents.Source: §I · p. 2
  • Finite-time exclusion does not require perfect detection, but depends on history-uniform detection above a threshold and accounting assumptions.The stated threshold is at most 1/3 in the paper’s setting even with full compensation; an exact capacity result makes it hold for every retained population.Source: §I · p. 2
  • The passage reports a result of up to 43% relative to conventional stake slashing, but does not specify the result’s metric or direction.Source: §I · p. 2

A. Blockchain Foundations for IoT

  • BE-CTDE uses a permissioned Ethereum management layer and smart contracts to coordinate training, while neural-network execution remains off-chain.The blockchain foundation discussion describes deterministic smart contracts for automated trust-layer functions.Source: A. Blockchain Foundations for IoT · p. 2

C. Byzantine-Resilient Multi-Agent Learning and Incentive Design

BE-CTDE uses trajectory-level behavior evidence rather than gradient similarity to address heterogeneous MARL agents. Its training architecture connects verification and token accounting to gradient weights, while the cited experiments do not benchmark several related algorithms or transfer their guarantees.

C. Byzantine-Resilient Multi-Agent Learning and Incentive Design

  • Gradient-aggregation defenses such as Krum, coordinate-wise median, and trimmed mean assume an honest majority, while heterogeneous MARL agents need not have aligned honest gradients.BE-CTDE instead compares each executed trajectory with the shared-policy recommendation, a behavioral reference across distinct local subproblems.Source: C. Byzantine-Resilient Multi-Agent… · p. 3

A. System Architecture

  • The architecture assigns blockchain coordination, behavior verification, and token incentives to the management layer while neural-network computation remains off-chain.Agents interact through observations, rewards, and actions; the system description also cautions that recorded physical transitions do not prove sensor truth.Source: Figure 1 · p. 3, A. System Architecture · p. 4

C. Byzantine-Resilient Multi-Agent Learning and Incentive Design

  • The cited experiments neither benchmark the referenced algorithms nor transfer their guarantees to MARL.The related-work distinction is a feedback path in which movement evidence changes token accounts, accounts change gradient weights, and insufficient balance ends training contribution.Source: C. Byzantine-Resilient Multi-Agent… · p. 3

A. System Architecture

  • A trusted coordinator records transitions, computes policy recommendations, and submits sampled records; contract-updated retained balances then determine off-chain gradient weights.The described interface is auditable but does not prove independently reported sensor data are truthful.Source: A. System Architecture · p. 4

B. Cooperative Decision-Making under Partial Observability

The paper frames cooperative IoT learning as centralized policy training with decentralized execution under partial observability. BE-CTDE links behavioral verification and token accounting to training influence, while its threat model and design discussion delimit what that process can control.

B. Cooperative Decision-Making under Partial Observability

  • The Dec-POMDP model represents global configurations and joint actions, while each agent acts from local observations and the agents jointly maximize discounted return.Policies execute decentrally, but the shared parameters are updated centrally from globally aggregated information.Source: B. Cooperative Decision-Making under… · p. 4

C. Threat Model

  • The threat model allows up to M < N agents to replace policy actions to disrupt the environment, while assuming authentic physical observations and an honest training coordinator.Exclusion removes training contribution, not physical presence; network consensus faults and several other attacks are outside the evaluation.Source: C. Threat Model · p. 4

D. Design Problem

  • BE-CTDE targets both trajectory disruption and distortion of centralized updates, with token accounting directly changing agents’ contribution to the latter channel.Its task-level evaluation does not establish the conditional detection assumptions used in the analysis, and learning-layer control cannot remove obstacles or guarantee access to occupied goals.Source: D. Design Problem · pp. 4–5

IV. ON-CHAIN DETECTION AND TOKEN-BASED INCENTIVE MECHANISMS

BE-CTDE separates high-frequency off-chain learning from on-chain behavioral verification and token settlement. Its population-relative verifier compares sampled behavior with shared-policy recommendations, but its own flags are capacity-limited because deviations also affect the population reference.

A. Training Workflow Overview

  • Episode-end detection and settlement supply weights for the next update, while the off-chain learner uses executed actions and rewards at every step to form policy-loss inputs.States are sampled every h steps for verification, whereas neural inference, communication, advantage estimation, and PPO remain off-chain.Source: A. Training Workflow Overview · p. 5, §IV · p. 5

B. Framework-Agnostic Behavioral Consistency Verification

  • The verifier compares each agent’s executed outcome with the shared-policy recommendation at sampled states and flags cumulative residuals above the retained population’s mean-plus-k-standard-deviation threshold.Residuals indicate inconsistency, not necessarily malicious intent; honest sampling or environment-altered outcomes can also produce residuals.Source: B. Framework-Agnostic Behavioral… · p. 5
  • The detector can flag at most Fmax(n), the largest integer strictly below n/(1 + k^2), retained agents at one settlement.Because flagged agents inflate the population mean and spread, this self-masking limits simultaneous detection; threshold ties and zero variance produce no flags.Source: B. Framework-Agnostic Behavioral… · pp. 5–6
  • At n = 8, at most two agents can be flagged, bounding the realized flagged-adversary fraction at 2/3 with three adversaries or 2/5 with five.At a 5/8 adversary ratio, the ceiling remains below 1/2 for every evaluated N, so one-round separation is not guaranteed; Table III labels these as realized-fraction bounds, not conditional detection guarantees.Source: B. Framework-Agnostic Behavioral… · p. 6, Table III · p. 6

C. Token-Based Incentive Mechanism

The token mechanism couples flags to balance changes and uses retained balances to govern continued training participation. Its incentive and exclusion guarantees are conditional on specified behavior, detection, and accounting assumptions.

C. Token-Based Incentive Mechanism

  • At each evidence-bearing settlement, an unflagged retained agent gains its reward and conditional compensation, while a flagged agent forfeits stake.The slashed fraction not allocated as compensation is burned, and rewards are minted.Source: C. Token-Based Incentive Mechanism · p. 6
  • Honest execution strictly dominates a specified deviation exactly when the difference in expected token gains plus the stake-probability difference exceeds the deviation utility.The comparison allows both the flag probability and conditional unflagged gain to vary between honest and deviating actions.Source: C. Token-Based Incentive Mechanism · p. 6
  • With equal unflagged gains, strict deterrence requires deviation utility below (pD − pH)(s + b); when deviation utility is zero, this reduces to pD > pH.The comparison is one-round, and changing a trajectory can also alter peer flags and compensation.Source: C. Token-Based Incentive Mechanism · pp. 6–7
  • Under the exclusion assumptions, an adversary with conditional flag probability uniformly above P* = bmax/(s + bmax) has finite expected exclusion time.For N = 8, the sufficient threshold is at most 1/3 even with full compensation; the bound counts evidence-bearing settlements and requires self-funded re-staking without top-ups.Source: C. Token-Based Incentive Mechanism · p. 7
  • Once excluded, an adversary has zero direct gradient contribution, provided the theorem’s conditions hold for every adversary in the finite initial population.Economic exclusion removes training influence but does not physically remove an agent, and a positive conditional detection floor cannot be certified by pooled detection rates alone.Source: C. Token-Based Incentive Mechanism · pp. 6–7

D. Token-Weighted Policy Aggregation

Token balances weight retained agents’ contributions to the centralized policy update, while gradient clipping bounds their direct preoptimizer influence.

D. Token-Weighted Policy Aggregation

  • After settlement, balances are normalized over the new retained set, and the resulting weights remain fixed during each learning update.The optimizer consumes the weighted aggregate gradient.Source: D. Token-Weighted Policy Aggregation · p. 7
  • With each per-agent loss gradient clipped to norm limit Lg, any contributor set’s direct preoptimizer contribution is bounded by Lg times its total weight.The bound applies to the aggregate vector before the optimizer transforms it.Source: D. Token-Weighted Policy Aggregation · pp. 7–8
  • Exclusion sets an agent’s coefficient to zero, but prior influence may persist in model parameters or optimizer state.The evaluation does not establish a numerical per-agent gradient bound, and updates are skipped if no contributors remain.Source: D. Token-Weighted Policy Aggregation · p. 8

A. Smart Contract Execution Pipeline

The pipeline records interaction data and learner-interface outputs at blockchain coordination points while leaving neural inference and optimization off-chain. At episode end, behavioral flags drive token accounting and determine weights for the next policy update.

A. Smart Contract Execution Pipeline

  • BatchDataStorage records sampled interaction data, while OutputHead, TDError, TrainingGAE, and PolicyLoss interfaces anchor learner outputs and update information.These records persist at the corresponding contract interfaces; neural inference and optimization remain off-chain.Source: A. Smart Contract Execution Pipeline · p. 8
  • At episode end, MaliciousDetection compares recommended and executed behavior to produce residuals and population-relative flags.IncentiveMechanism then applies forfeitures, rewards, compensation, and burning, and updates balances and eligibility.Source: A. Smart Contract Execution Pipeline · p. 8
  • Updated balances and eligibility determine weights for the subsequent policy update, keeping each episode’s detection results aligned with its economic state.This sequence prevents an update from combining flags from one round with balances from another.Source: A. Smart Contract Execution Pipeline · p. 8

B. On-Chain and Off-Chain Function Allocation

Neural computation runs in the learner, while the coordination layer records evidence, detector outputs, economic state, and policy commitments. Hashes and signatures support coordination checks but do not establish the truth or accuracy of submitted content.

B. On-Chain and Off-Chain Function Allocation

  • Observation encoding, Transformer communication, recurrent outputs, PPO losses, gradients, and the token-weighted update run in the learner rather than the EVM.The coordination layer records behavioral evidence, detector outputs, economic state, and policy commitments associated with update inputs.Source: B. On-Chain and Off-Chain Function… · p. 8
  • Policy hashes identify parameter snapshots and interaction commitments identify episode records, enabling checks that participants refer to the same model and data.These commitments do not prove that the model produced an action or that submitted losses, gradients, or physical telemetry are correct.Source: B. On-Chain and Off-Chain Function… · p. 8
  • A signed submission authenticates its sender without establishing the accuracy of its contents.This distinguishes tamper-evident coordination from behavioral verification.Source: B. On-Chain and Off-Chain Function… · p. 8
  • Tuple submission can overlap off-chain computation, but updates dependent on unsettled balance transitions must wait for the required state.Asynchronous submission alone does not establish an end-to-end latency bound.Source: B. On-Chain and Off-Chain Function… · p. 8

C. Solidity Implementation and Geth Architecture

The implementation separates blockchain coordination from the learner and specifies arithmetic and workload constraints. The experiments instantiate BE-CTDE on SCRIMP for MAPF, with defined training and testing conditions, metrics, and common-backbone baselines.

C. Solidity Implementation and Geth Architecture

  • The intended stack uses Solidity 0.8.0 contracts, Geth 1.13.14 nodes, and a Python 3.9.21 learner connected through Web3.py and HTTP-RPC.The intended deployment is a private Ethereum network; the module description does not establish consensus performance.Source: C. Solidity Implementation and Geth… · p. 8
  • Contract arithmetic uses six-decimal scaled quantities and preserves strict detector comparisons, including ties, through a division-free test at k = 1.5.Balance normalization uses the same eligible-agent set, and token division residue must be accounted for so compensation and burning reconcile with slashed deposits.Source: C. Solidity Implementation and Geth… · p. 8

D. Workload and IoT Placement

  • With K ≤ 8 sampling boundaries and N agents, collection plus separate detection and incentive calls requires at most 8N + 2 requests under the stated submission assumption.This excludes deployment, deposit handling, policy-stage commitments, and retries; gas, block capacity, and delay also depend on storage, payloads, and network configuration.Source: D. Workload and IoT Placement · p. 8
  • For an automated guided vehicle fleet, coordinator nodes can host the learner and blockchain interfaces while vehicles execute the transferred policy locally.The chain coordinates training and economic consequences, not each navigation decision; MAPF results establish neither an edge-control timing budget nor a network-throughput claim.Source: D. Workload and IoT Placement · p. 8

A. Experimental Setup

  • The MAPF task uses SCRIMP, four cardinal moves plus staying, local observations, and recurrent Transformer communication; overt adversarial behaviors include blocking, goal occupation, and repetitive movement.The setup trains eight agents on 40×40 grids for 1,838 episodes, sampling residuals every 32 steps and testing four population sizes at three obstacle densities.Source: A. Experimental Setup · p. 9
  • Evaluation reports honest-agent collision rate and all-honest-agent success rate, and compares defense components on a common backbone.The baselines add blockchain coordination, detection, and token-weighted aggregation in stages; CO and SR are interpreted jointly because low collision frequency alone may reflect stalled navigation.Source: A. Experimental Setup · p. 9

B. Performance Under Adversarial Training and Testing

Under adversarial training and testing, full BE-CTDE outperforms Lite across all evaluated conditions. In honest testing, compensation improves collision outcomes over burn-only, while training curves track learning progress rather than exclusion.

B. Performance Under Adversarial Training and Testing

  • At N = 8, full BE-CTDE achieves 100% SR across densities with minority adversaries, compared with 96–97% for Lite and zero for SCRIMP.At N = 32 and 30% obstacles, CO is 23.09% versus Lite’s 27.88%, and SR rises from 92% to 97%.Source: B. Performance Under Adversarial… · p. 9
  • At N = 64 and 15% obstacles with majority adversaries, CO falls from Lite’s 37.10% to 24.00%, while SR rises from 89% to 95%.Full BE-CTDE’s 13.10 pp CO reduction is 35.3%; it achieves SR of at least 93% at N ≤32.Source: B. Performance Under Adversarial… · p. 9
  • Across all 24 ratio, population, and density cells, full BE-CTDE has lower CO and higher SR than Lite.Table V covers 37.5% and 62.5% malicious-agent panels and obstacle densities of 0%, 15%, and 30%.Source: B. Performance Under Adversarial… · p. 9, Table V · p. 10
  • The 100-episode cells share trained policies, and the reported Wilson intervals do not quantify paired method differences; independent training-seed variation is unestablished.Honest-testing and joint-adversarial-testing results concern different transfer conditions and should not be treated as one matched ablation.Source: B. Performance Under Adversarial… · p. 10, C. Effect of Compensation Factor · p. 10

C. Effect of Compensation Factor

  • Compensated variants show faster loss decline, while goal-arrival curves indicate task progress rather than when or whether adversaries were excluded.The compensation study uses honest testing, distinct from the joint adversarial-testing comparison.Source: C. Effect of Compensation Factor · p. 10, Figure 3 · p. 10
  • 43.1% is the largest relative CO reduction for full compensation versus burn-only, from 20.38% to 11.59% at N = 32 and 15% obstacles.Full compensation improves on burn-only in all 24 cells and has the lowest CO among three token variants in 23/24.Source: C. Effect of Compensation Factor · p. 10

D. Relation to the Analysis

The analysis relates finite-population detection capacity and compensation to MAPF outcomes, while the conclusion reports stronger success under attack and fewer collisions than partially defended variants.

D. Relation to the Analysis

  • At λ = 1, the 1/3 threshold is below both capacity ceilings (2/3 and 2/5), but majority-regime exclusion has almost no margin and retained adversaries’ direct contribution is weight-capped.This analysis is consistent with the lower majority-ratio success rate in Table V(b).Source: D. Relation to the Analysis · pp. 10–11
  • 5.4 pp at the 37.5% training ratio and 3.8 pp at 62.5%: λ = 1 reduces collision rate relative to λ = 0 across the twelve cells.The analysis attributes the compensation study’s effect to faster balance reweighting toward unflagged agents, rather than deterrence, because evaluated adversaries use fixed disruptive behaviors.Source: D. Relation to the Analysis · p. 11
  • The reported Smin = 0.1s lies outside Assumption 1; if balances below s end participation, s is the effective retention threshold and the exclusion-time bound applies with Smin replaced by s.For p = 0.5 and λ = 1, the bound is at most 4Sj(0) settlements; otherwise it remains conditional for these trials.Source: D. Relation to the Analysis · p. 11

VII. CONCLUSION

  • BE-CTDE succeeds far more often under attack than SCRIMP, including with an adversarial majority, and collides less than variants that only flag deviations or burn forfeited stake.The conclusion identifies broader threat models, task domains, and multi-operator IoT deployment as future work.Source: §VII · p. 11
  • BE-CTDE links trajectory-residual verification, token accounting, and balance-weighted aggregation in one CTDE workflow.The conclusion frames blockchain as an active training-management layer for cooperative learning.Source: §VII · p. 11
Loading 2610.34121v1…