PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Gradient Inversion for Private Trajectory Reconstruction in Embodied Reinforcement Learning

Sudip Bhujel, Shanghao Shi, Ruiquan Huang, Ning Zhang, Yang Xiao

arXiv:2609.30258v1cs.LG

TL;DR

Ordered policy gradients can expose private embodied observation-action trajectories, while single-frame inversion methods do not use their temporal structure. TRACE reconstructs trajectories autoregressively from gradients; on held-out scenes it reports 18.8 dB PSNR and near-perfect action recovery, with a per-frame runtime of 3–4.5 ms.

  • Problem

    Single-frame gradient inversion methods do not exploit the temporal structure of embodied trajectories, even though exposed gradients can reveal private observations and actions.

    Source: §1 · p. 1

  • Method

    TRACE amortizes gradient inversion across ordered policy-learning gradients using an autoregressive model, with theoretical analysis of temporal gain and discrete-action recovery.

    Source: §1 · p. 2

  • Results

    18.8 dB PSNR with near-perfect action recovery at 3–4.5 ms per reconstructed frame, improving over the learning-based baseline by ∼2 dB PSNR.

    Source: §7 · p. 10

  • Takeaways & Limitations

    Lightweight pruning and mild quantization leave substantial leakage, motivating sequence-aware privacy mechanisms.

    Source: §7 · p. 10

    The study focuses on per-step gradients at broadcast parameters; multi-step client updates may reduce the temporal structure available to the attacker.

    Source: §2 · p. 3

Abstract

from arXiv · show

Distributed learning in embodied reinforcement-learning agents offers a degree of privacy by retaining raw sensor data on-device and transmitting only policy gradients to the server. Yet temporal structure can amplify this leakage beyond single-frame attacks. We introduce Temporal Reconstruction Attack on Consecutive Encodings (TRACE), an amortized temporal gradient-inversion attack that autoregressively reconstructs the sequence of private observation-action trajectories from per-step policy-learning gradients. The attack exploits two structural signals ignored by prior single-frame methods: (i) cross-time correlation between successive embodied gradients, which we formalize via a conditional mutual-information bound, and (ii) closed-form action recovery from policy-head gradient structure, which we prove exact when standard entropy regularization is sufficiently small. On held-out embodied scenes, TRACE reaches $18.8$ dB PSNR with near-perfect action recovery at $3$-$4.5$ ms per reconstructed frame, dominating the learning-based baseline across all reconstruction metrics and exceeding optimization attacks while running orders of magnitude faster. Further evaluation demonstrates TRACE's broader applicability across recurrent, residual, and compact transformer victim architectures, multi-modal inputs, and larger discrete action spaces. Defense experiments suggest that protecting temporal gradient streams may require sequence-aware privacy mechanisms.

1 Introduction

Gradient inversion threatens embodied-RL privacy because ordered gradients can reveal time-ordered camera views and actions, while existing single-frame attacks ignore trajectory structure. TRACE uses temporal context to reconstruct observation-action sequences, with reported transfer to held-out scenes and broader victim settings.

1 Introduction

  • Recovered embodied trajectories can expose camera views and actions, enabling activity profiling, premise mapping, and persistent monitoring.Distributed RL agents may transmit policy gradients while keeping raw environmental observations on-device.Source: §1 · p. 1
  • Unlike single-image or static-batch attacks, TRACE amortizes inversion across a trajectory with a gradient encoder, causal transformer, and joint observation-action decoder.It exploits correlated signatures across consecutive policy gradients, which provide side information for reducing per-step ambiguity.Source: §1 · pp. 1–2
  • The paper develops closed-form discrete-action recovery and an information-theoretic characterization of temporal reconstruction gain.The stated theoretical contributions address action recovery and cross-time gradient structure.Source: §1 · p. 2
  • TRACE shows strong zero-shot reconstruction on held-out scenes and rapid improvement after adaptation with a small subset of target-domain trajectories.The evaluation also covers recurrent, residual, and transformer victims, multi-modal inputs, and larger discrete action spaces.Source: §1 · p. 2

2 Problem Formulation

The threat model gives a curious server the ordered per-step gradients from an embodied RL client and asks it to reconstruct the corresponding observations and actions. The formulation assumes policy learning with A2C or PPO and focuses on per-step gradients evaluated at broadcast parameters.

2 Problem Formulation

  • A passive, honest-but-curious server observes a contiguous sequence of client gradients and seeks to reconstruct the corresponding observations and actions.The client interacts with a private environment after receiving the server's broadcast policy model.Source: §2 · p. 3
  • The adversary is assumed to know the model architecture, checkpoint, action space, and learning rule, but not the private data; auxiliary same-task trajectories are available for offline training.Inference then relies solely on the observed ordered gradients.Source: §2 · p. 3
  • The observation space is RGB and the action space is discrete, while advantage estimates and return targets are derived from the client's hidden local rollout.The supplied passage introduces a unified per-step actor-critic objective for these rollout pairs.Source: §2 · p. 3
  • The study treats gradients as evaluated at broadcast parameters θ0; multi-step client updates may reduce exploitable temporal structure and are outside its focus.A2C and PPO share critic and entropy terms and differ only in their policy-loss term; their policy gradients coincide at θ0.Source: §2 · p. 3

3 TRACE Design

TRACE combines per-step gradient information with causal temporal context to decode observations and actions autoregressively. Its training procedures address the mismatch between ground-truth context during training and predicted context during inference.

3 TRACE Design

  • A gradient encoder and image encoder produce per-step latents, a causal transformer propagates trajectory context, and a decoder reconstructs observations and predicts actions.The attacker compresses high-dimensional gradients into a shared low-dimensional representation; the decoder uses contextualized latents for its image and action heads.Source: §3 · p. 4, §3.1 · p. 4

3.1 Attacker Model

  • The attacker model interleaves gradient tokens with image-context tokens, using lagged image context during teacher-forced warmup and causal attention for sequential decoding.At inference, each decoded observation is encoded and appended as context for later timesteps.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 Autoregressive Training

  • Scheduled sampling gradually replaces ground-truth frames with predictions, while a short-horizon rollout loss trains the model on its own feedback context.Inference follows the sequential feedback pattern across all T steps; the multi-term objective combines pixel, action, perceptual, and rollout losses.Source: §3.2 · p. 5

4 Theoretical Analysis

The analysis asks what per-step policy gradients reveal and why temporal conditioning can improve trajectory reconstruction. It establishes conditions for action recovery and characterizes the added information in past gradients.

4.1 Gradient Leakage and Action Recoverability

  • At the broadcast checkpoint θ0, A2C and PPO produce the same communicated gradient, whose policy, value, and entropy terms carry action and state-dependent information.The policy term carries action information, while value and entropy terms encode additional state-dependent structure.Source: §4.1 · p. 5
  • A fully deterministic policy makes policy-head action recovery ill-posed, whereas moderately confident policies keep the identifiability gap bounded away from zero.The one-hot limit makes every column sum vanish; the K = 2 case also requires sign information from cross-head information such as the value-head gradient.Source: §4.1 · p. 6
  • The policy-head gradient permits exact discrete-action recovery under the stated assumptions, including when entropy regularization is sufficiently small relative to the identifiability gap.For K ≥ 3, the sign-agnostic rule applies; for K = 2, the equal magnitudes require a sign-aware rule.Source: §4.1 · pp. 5–6

4.2 Temporal Conditioning Benefit

  • The exact temporal-gain identity is limited to a Markov victim policy with causally adapted, un-normalized one-step estimators; GAE with λ > 0 and rollout-level normalization fall outside these conditions.Scaling under practical PPO and A2C estimators is discussed informally and validated empirically.Source: §4.2 · p. 6
  • Temporal-context gain is nonnegative, and conditioning on the past gradient stream cannot increase the Bayes-optimal state-reconstruction error.Past gradients help to the extent that they contain information about the state not already captured by the current gradient.Source: §4.2 · p. 6

5 Experimental Evaluation

The evaluation tests TRACE against inversion baselines, on held-out scenes with adaptation, under gradient defenses, and across ablations. Across these tests, the results assess reconstruction fidelity, action recovery, and sensitivity to temporal context and training objectives.

5.1 Comparison with Baselines

  • TRACE achieves the best reconstruction metrics under both PPO and A2C, improving over LtI by about 2 dB and 1 dB PSNR, respectively.Adding temporal conditioning reduces MSE from 0.023 to 0.014 for PPO and from 0.018 to 0.014 for A2C; both learned methods run in milliseconds per frame, versus tens of seconds for optimization-based methods.Source: Table 1 · p. 7, §5.1 · p. 7

5.2 Few-Shot Adaptation

  • At zero-shot transfer to held-out scenes, TRACE reaches 18.8 dB PSNR and SSIM 0.627 with perfect action recovery; adapting on 10% of target trajectories adds 2.1 dB PSNR.With a 50% target-trajectory budget, MSE falls to 0.005 and SSIM exceeds 0.74.Source: Table 2 · p. 8, §5.2 · p. 8

5.3 Defense Evaluation

  • With the attack trained on clean gradients and applied to defended gradients, DP-SGD offers the strongest protection, reducing action accuracy to near-random levels and SSIM to approximately 0.34 for PPO and 0.32 for A2C.Pruning to 10% of gradient entries leaves PPO nearly intact (PSNR = 18.8, Act. Acc. = 99.9%); downstream navigation utility under defenses was not assessed.Source: Table 3 · p. 8, §5.3 · p. 8

5.4 Ablations

  • Extending the trajectory from T=1 to T=8 gains about 6 dB PSNR, while T=64 reaches 20.29 dB PSNR and SSIM 0.666.The study adopts T=8 as a balance between temporal context and optimization complexity.Source: Table 4 · p. 9, §5.4 · p. 9
  • Restricting gradients to terminal policy/value heads lowers visual fidelity to PSNR 14.12 and LPIPS 0.511, while action recovery remains at least 99.6% across components.The authors attribute high-frequency structural cues to earlier CNN and FC layers and behavioral signals to gradients throughout the network.Source: §5.4 · p. 9
  • Removing the action loss reduces action accuracy to 19.0%; dropping LPIPS raises PSNR to 18.94 but LPIPS to 0.401, while removing L1 lowers PSNR to 17.90 and LPIPS to 0.409.The reported trade-offs motivate a multi-objective formulation balancing pixel-level distortion and perceptual similarity.Source: §5.4 · p. 9

6 Related Work

Prior work covers gradient inversion in supervised and distributed learning, learning-based inversion, and privacy risks in RL and embodied AI. TRACE differs by targeting temporally correlated gradients from embodied policy-learning trajectories.

Learning-based inversion

  • Classical attacks optimize each gradient, while learning-based inversion amortizes the cost with feed-forward models; TRACE uses a causal transformer to decode full trajectories without per-gradient search.Prior learning-based methods include Wu et al. [40] and efficiency-focused attacks [16] [31].Source: §6 · p. 9

Gradient inversion attacks in distributed learning

  • Unlike prior single-sample gradient-inversion attacks, TRACE exploits cross-step dependencies in embodied policy-learning gradients rather than treating them as i.i.d. noise.The related work notes prior extensions to Vision Transformers [14] [23], larger architectures [31], and systematic attack analyses.Source: §6 · p. 9

Privacy risks in RL and embodied AI

  • Work on RL and embodied-AI privacy remains limited; TRACE builds on studies of federated robot RL [20] and trajectory-level auditing via cumulative rewards [9] to invert consecutive frames.Related work also studies federated RL under transition-dynamics constraints [15].Source: §6 · p. 9

7 Conclusion

TRACE reconstructs private observation–action trajectories from ordered policy gradients, with high-fidelity, near-real-time results on held-out scenes. The conclusion emphasizes both the leakage risk and the need for sequence-aware protection.

7 Conclusion

  • 18.8 dB PSNR, near-perfect action recovery, and 3–4.5 ms per frame summarize TRACE’s held-out-scene results, about 2 dB above the learning-based baseline.The conclusion also reports that pruning and mild quantization leave substantial leakage.Source: §7 · p. 10

A Broader Impact

  • For the advantage-weighted actor–critic algorithms studied, policy gradients can expose trajectories nearly as directly as raw sensor streams, while stronger perturbations may affect utility or communication efficiency.The conclusion identifies the honest-but-curious setting and notes that per-step pruning and mild quantization leave substantial cross-step leakage.Source: A Broader Impact · p. 14
  • The paper calls for sequence-aware protection across rollouts, but notes that proposed mechanisms may impose prohibitive computational overhead in real-time embodied deployments.The authors also withhold pretrained attacker checkpoints and captured trajectory datasets to reduce misuse risk.Source: A Broader Impact · p. 14

D.1 Proof of Proposition 4.1 (A2C–PPO Equivalence and Decomposition)

  • At the broadcast checkpoint, A2C and PPO have identical policy gradients, so the local gradient decomposition and discrete-action recovery results apply to both.PPO clipping is locally inactive around that checkpoint.Source: §C.2 · p. 15

C.3 Autoregressive Modeling

  • TRACE’s causal transformer conditions each gradient representation on prior gradient and lagged image-context tokens before reconstructing observations and actions.Generated frames are appended sequentially as context during inference.Source: §C.3 · pp. 15–16

Proof of part (i): exact recovery when ce = 0.

D.3 Conditional Sufficiency of st for gt (Lemma D.3)

  • The temporal-gradient argument assumes causally adapted, unnormalized one-step estimators; GAE with λ > 0 and rollout-level advantage normalization violate that factorization.Under the stated assumptions, TD(0) estimators with stop-gradient on the bootstrap value satisfy the condition.Source: Then the per-step gradient is the… · p. 19

D.4 Proof of Theorem 4.5 (Temporal Reconstruction Gain)

The temporal-gain proof relates earlier gradients to improved state reconstruction under stated estimator assumptions. The appendix also qualifies those assumptions and reports different empirical scaling for A2C and PPO.

D.4 Proof of Theorem 4.5 (Temporal Reconstruction Gain)

  • MSE reconstruction using the current gradient and prior gradients is no worse than using the current gradient alone, with the temporal information relation derived under the stated conditional-independence assumptions.The proof expresses the gain through conditional mutual information between the state and earlier gradients given the current gradient.Source: §D.4 · p. 20
  • The conditional-sufficiency result is an idealized reference for these experiments because GAE uses future noise and rollout normalization couples timesteps.The proof’s exact identity and bound hold under assumptions (M) and (B); the practical-setting discussion is explicitly described as intuition, not a proved correction.Source: §D.4 · p. 20
  • A2C reconstruction plateaus at 18.3–18.8 dB PSNR for T = 8, 16, 32, and 64, while PPO rises from 18.74 dB at T = 8 to 20.29 dB at T = 64.The authors interpret this as algorithm- and checkpoint-dependent temporal signal strength, not a theorem-level consequence of the TD(0) idealization.Source: §D.4 · p. 20

D.5 Derivation of Training Objective

  • A three-step autoregressive rollout loss trains TRACE with predicted frames fed back as context, targeting robustness to imperfect context rather than optimizing the model’s own context inputs.The loss combines pixel, action-prediction, and perceptual terms.Source: §D.5 · pp. 20–21

E Evaluation Metrics

The appendix defines image and action-recovery metrics and gives further experimental details. Its additional evaluations report few-shot adaptation results and test TRACE across varied victim configurations, with limits on cross-architecture interpretation.

E Evaluation Metrics

  • PSNR is a logarithmic transformation of MSE in decibels, while SSIM compares luminance, contrast, and structure, and LPIPS measures perceptual distance using deep features.Action accuracy reports the percentage of correctly inferred actions.Source: E Evaluation Metrics · p. 21

F Additional Experiment Details

  • The default navigation evaluation uses AI2-THOR point-goal episodes, with training on 108 scenes and 12 held out for testing.The dataset stores temporally ordered per-step gradients, RGB observations, and actions; color jitter augments recorded observations.Source: F Additional Experiment Details · pp. 21–22

F.1 Runtime of Experiments

  • Table 7 reports end-to-end training runtimes, including training, periodic validation, and final evaluation; timings reflect the measurement hardware rather than a strict hardware requirement.The caption specifies six H200 GPUs by default and two for few-shot adaptation.Source: §F.1 · p. 22, Table 7 · p. 23

F.2 Few-Shot Adaptation

  • With A2C, zero-shot transfer yields PSNR = 18.9, while adaptation on 50% of target trajectories raises PSNR to 25.0 and lowers LPIPS to 0.132.The passage characterizes zero-shot visual fidelity as reasonable and action recovery as near-perfect.Source: §F.2 · p. 23, Table 8 · p. 23

F.3 Applicability across victim configurations

  • The applicability tests use separately trained attackers within AI2-THOR point-goal navigation, not transfer of one attacker between architectures.For the multi-modal victim, only RGB observations and actions are reconstructed, not the goal vector.Source: §F.3 · p. 24
  • 99.5% action accuracy with ten actions accompanies 18.62 dB PSNR for recurrent victims, 19.78 dB for multi-modal victims, and 22.36 dB for the wider residual victim.Tiny ViT reaches 99.0% action accuracy but 13.99 dB PSNR; the authors caution that differing trajectories, dataset sizes, and attacker capacities prevent attributing fidelity differences solely to architecture.Source: §F.3 · p. 24

F.4 Reconstruction under temporal gradient aggregation

With gradient averaging, training from scratch improves action-proportion recovery over uniform guessing in both tested aggregation settings. Image reconstruction and action-histogram recovery differ across settings and attacker training approaches.

F.4 Reconstruction under temporal gradient aggregation

  • Training from scratch reduces mean TV to 0.296, approximately 31% below the uniform baseline mean of 0.428, with lower sequence-averaged TV in 94 of 100 held-out sequences.Source: §F.4 · p. 26
  • 16.03 dB PSNR and histogram TV of 0.268 are reported for Agg8 trained from scratch, about 12% below its uniform baseline of 0.304.The image metric measures last-frame reconstruction, not recovery of every observation or its order within an aggregation window.Source: Table 11 · p. 25, §F.4 · p. 26
  • For Agg4, training from scratch recovers action proportions better than uniform guessing but yields lower PSNR and SSIM and higher MSE and LPIPS than the pretrained attacker.The runs use different training budgets and learning rates, and the experiments do not isolate dataset size, initialization, and schedule effects.Source: §F.4 · p. 26

F.5 Effect of auxiliary-data scale

With more auxiliary training data, TRACE’s visual reconstructions improve while action recovery remains perfect across the tested fractions. A separate qualitative example shows reconstructions under representative defenses using a 20%-adapted PPO attacker.

F.5 Effect of auxiliary-data scale

  • The scaling study trains attackers from scratch on nested subsets of unaugmented auxiliary episodes, holding the victim checkpoint and attacker architecture fixed.Evaluation uses 800 images from the same first 100 held-out sequences, without target-scene adaptation.Source: §F.5 · p. 26
  • PSNR rises from 16.25 to 18.71 dB and LPIPS falls from 0.502 to 0.354 as auxiliary data increases from 10% to 100%, while action accuracy remains 100%.These are single-run results; under the fixed-epoch protocol, larger subsets also receive more training updates.Source: §F.5 · p. 26, Table 12 · p. 27

F.6 Qualitative Defense Example

  • Figure 6 presents T=8 reconstructions under different defenses using the 20%-adapted PPO attacker.The corresponding zero-shot results are reported in Table 3.Source: Figure 6 · p. 27, §F.6 · p. 27

F.7 Extended Ablation Study

The A2C ablations show that longer context improves reconstruction up to a plateau, while gradient components and loss terms contribute differently to image and action recovery. The discussion limits the defense evaluation to practical mechanisms compatible with ordered per-step gradients.

F.7 Extended Ablation Study

  • For A2C, increasing context from T=1 to T=8 raises PSNR from 15.54 to 18.60 dB, then results plateau at 18.26–18.77 dB for T=16, 32, and 64.Unlike A2C, PPO continues improving to 20.29 dB at T=64; the authors suggest algorithm-dependent temporal signal strength as a possible explanation.Source: §F.7 · p. 27, Table 13 · p. 28
  • CNN-only and FC-only gradients recover PSNR within roughly 0.5 dB of the full gradient, while policy/value heads alone lose 2.6 dB but retain 100.0% action accuracy.Removing action loss drops action accuracy to 16.4% without affecting image PSNR; removing LPIPS lowers A2C PSNR by 0.59 dB.Source: §F.7 · p. 28, Table 13 · p. 28

G Discussion

  • Defense evaluation covers pruning, quantization, additive noise, and DP-SGD, but not CENSOR, SVDefense, recomputed rollout-level updates, multi-epoch updates, or multi-client secure aggregation.The latter aggregation settings may diminish or eliminate the ordered temporal signal TRACE exploits.Source: G Discussion · p. 28

H Limitations

The evaluation and theoretical analysis have defined scope boundaries: experiments use point-goal navigation in AI2-THOR, and the action-recovery result concerns categorical policies.

H Limitations

  • Experiments are confined to point-goal navigation in AI2-THOR and do not establish transfer across tasks or simulators.The additional victim architectures and input settings still retain the ordered per-step gradient interface.Source: H Limitations · p. 28
  • The multimodal experiment reconstructs RGB and actions, not goal inputs; depth, proprioception, and off-policy algorithms remain unevaluated.Recurrent captures condition on detached hidden states rather than backpropagating through complete histories.Source: H Limitations · p. 28
  • Theorem 4.2 applies to categorical policies over finite actions, and the study does not establish scalability to substantially larger policies.For Gaussian continuous-control policies, identifiability depends on covariance parameterization and advantage-sign information; the categorical rule does not directly apply.Source: H Limitations · p. 28

Related papers

Loading 2609.30258v1…