PaperlayerView exact revision on arXiv

Source-linked AI summary

Modeling and Understanding Neural Weights of Humanoid Policies with Diffusion

Huixuan Cheng, Bike Zhang, Shankar Sastry, Koushil Sreenath

arXiv:2610.20606v1cs.ROcs.AIcs.LG

TL;DR

Humanoid policies are commonly maintained as separate task-specific networks, and direct generation and interpretation of their weights remain relatively unexplored. The paper models Unitree G1 policy weights with latent diffusion and analyzes their structure. Generated networks outperform distance-matched noisy training weights across motions, while first-layer weights reflect a kinematic prior across RL and IL networks.

  • Problem

    Humanoid skills are typically maintained as separate expert policies, while direct weight-space generation and systematic understanding of encoded structure remain underexplored.

    Source: §I · pp. 1–2

  • Method

    The approach uses a cluster-wise linear autoencoder to compress RL-trained Unitree G1 actor weights, then trains a Diffusion Transformer on the latents.

    Source: §I · p. 2, §II · p. 3

  • Results

    At matched distance from the training set, diffusion-generated networks outperform noise-perturbed baselines in success rate and tracking error across all motions; first-layer weights encode a kinematic prior across RL and IL networks.

    Source: B. Results · p. 4, §VI · p. 7, C. RL and IL distillation for all… · p. 11

  • Takeaways & Limitations

    The findings provide initial insight into humanoid weight structure: first-layer weights reflect the robot’s kinematic prior, and RL- and IL-trained networks show similar structure.

    Source: §VI · p. 7, C. RL and IL distillation for all… · p. 11

    The analysis mainly covers first-layer weights and a limited set of motions on one embodiment; generality remains to be validated, and the identified structures have not yet improved performance or policy generation.

    Source: §V · p. 7

Abstract

from arXiv · show

Recent advances in robot learning have driven rapid progress in humanoid control, enabling increasingly agile behaviors in the real world. However, humanoid control networks are typically obtained through conventional training procedures, while comparatively little attention has been given to directly modeling the distribution of their learned weights and generating functional networks in weight space. In this work, we investigate whether high-performing humanoid control networks can be generatively modeled directly through their weights. We adopt a latent diffusion framework that couples a linear autoencoder with a Diffusion Transformer. The autoencoder compresses actor network parameters into a compact latent space, and the Diffusion Transformer models the distribution of these latents. Trained on networks collected from reinforcement learning for the Unitree G1 robot, our approach generates deployable networks that perform various motions in the real world, including walking, dancing, and spin kick. Beyond generation, we further study the learned weight space to better understand the structure captured by these humanoid control networks. We identify a structural principle in the first-layer weights that consistently reflects the robot's kinematic prior. Notably, this kinematic prior emerges under both Reinforcement Learning (RL) and Imitation Learning (IL) training, suggesting that it is not specific to a particular optimization process but may instead reflect common knowledge associated with humanoid embodiment. Together, these results demonstrate the feasibility of generative modeling in humanoid network weight space, while providing initial insights into the information encoded by their learned weights. Additional results are available on our project website: https://humanoid-weight-diffusion.github.io/

I. INTRODUCTION

Humanoid control skills are usually maintained as separate expert networks, while generating and interpreting these networks directly in weight space remain underexplored. This work models RL-trained Unitree G1 policy weights and analyzes their structure across training paradigms.

I. INTRODUCTION

  • Separate task-specific expert policies motivate directly modeling network weights as a compact, flexible generator of humanoid skills.Potential conditioning inputs include task descriptors, reference trajectories, or other embeddings.Source: §I · p. 1
  • Prior weight-generation work reaches embodied control and real-world quadruped locomotion, while humanoid control weight generation remains largely unexplored.The introduction identifies high-dimensional weight space as part of this gap and notes that network weights can have much lower effective than nominal dimensionality.Source: §I · pp. 1–2
  • Systematic analysis of what humanoid control networks encode in their weights remains largely unexplored, unlike feature interpretation in vision.The introduction notes that robotics studies have mostly examined which weights matter, such as through pruning.Source: §I · p. 2
  • The study collects RL-trained Unitree G1 networks for walking, dancing, and spin kick, then uses a linear autoencoder and Diffusion Transformer to model their weight distribution.It evaluates generated networks in simulation and sim-to-real transfer, while analyzing structures across training paradigms and configurations.Source: §I · p. 2

II. METHODOLOGY

The method builds a dataset of RL-trained Unitree G1 actor weights, compresses each network with a cluster-wise linear autoencoder, and trains a Diffusion Transformer to model the resulting latent distributions.

II. METHODOLOGY

  • For walking, dancing, and spin kick, the dataset retains 6,000 post-convergence actor networks per motion from 750 random-seed runs.Each seed contributes eight networks; actors share a three-layer MLP architecture, with about 30k–40k parameters depending on observation dimensions.Source: §II · p. 2
  • Encoder E uses E1 to compress [w1, b1] and E2 to compress [w2, b2, w3, b3], each into a 1,024-dimensional latent vector; concatenating them forms a 2,048-dimensional representation z for diffusion training.Separate autoencoders are trained for different motions using mean squared reconstruction error to reconstruct network parameters.Source: §II · p. 3
  • With autoencoders frozen, a motion-specific Diffusion Transformer models latent distributions using x-prediction and decodes generated latents into deployable actor weights.The transformer backbone is chosen to capture cross-layer and global weight-space correlations.Source: §II · p. 3

III. WEIGHT GENERATION EXPERIMENTS

The experiments assess generated-network performance, hardware transfer, and whether generated weights generalize beyond memorized training networks. A t-SNE visualization compares training and generated spin-kick weights.

III. WEIGHT GENERATION EXPERIMENTS

  • The evaluation compares generated actors’ success rates and tracking errors with baselines, and assesses sim-to-real transfer on the Unitree G1.These are two of the stated experimental aims.Source: §III · p. 3
  • The t-SNE visualization presents training and generated network weights for the spin-kick motion.Source: Figure 3 · p. 4
  • A separate evaluation asks whether generated weights generalize beyond training weights and their local noisy variants.The section frames this as distinguishing generalization from memorization.Source: §III · p. 4

A. Experimental Setup

The experiments compare generated humanoid policies with RL-collected networks and distance-matched noise perturbations across walking, dancing, and spin kick. They assess simulation performance, hardware deployment, generalization, and weight-space structure.

A. Experimental Setup

  • The setup evaluates walking, dancing, and spin kick in simulation against RL-collected networks and Diffusion Transformers with embedding dimensions 512, 768, and 1024.Success rate and tracking error are the simulation metrics.Source: A. Experimental Setup · p. 4

B. Results

  • Generated networks have slightly lower average performance than RL-collected networks, but their top sample among 100 is comparable and fewer than 10% collapse.The top 90% of generated samples have mean and variance comparable to the RL-collected networks.Source: B. Results · p. 4
  • Increasing the Diffusion Transformer embedding dimension improves performance, with the largest gains for spin kick, followed by dancing and walking.The authors suggest that more dynamic motions are associated with more complex weight distributions requiring higher-capacity models.Source: B. Results · p. 4
  • Networks generated without additional fine-tuning successfully perform walking, dancing, and spin kick on the Unitree G1.Generated spin-kick samples at the boundary of the training distribution also perform the motion in the real world.Source: B. Results · p. 4
  • For spin kick, generated samples largely lie within the training distribution in t-SNE space, while boundary samples can still succeed on hardware.The analysis uses 100 training and 100 generated samples in a shared t-SNE space.Source: B. Results · p. 4
  • At matched weight-space distance, diffusion-generated networks achieve higher success rates and lower tracking errors than noise-perturbed baselines across all motions.The comparison tests whether generation captures structure beyond nearby noisy versions of training weights.Source: B. Results · p. 4
  • Joint-wise weight distributions indicate higher importance for hip pitch, knee, and waist joints, suggesting a kinematic prior.The figure also visualizes the first-layer weight heat map for walking.Source: Figure 4 · p. 5

IV. WEIGHT SPACE ANALYSIS

The weight-space analysis examines first-layer actor weights for structural patterns across inputs, motions, and training settings. It finds consistent joint-importance structure that the authors interpret as a kinematic prior.

IV. WEIGHT SPACE ANALYSIS

  • The analysis focuses on first-layer RL actor weights, whose inputs have direct semantic meaning, and normalizes observations to reduce effects of differing input magnitudes.It examines input-region and motion patterns, RL–IL differences, and effects of domain randomization.Source: §IV · p. 5

A. Kinematic Prior in Weights

  • Joint position and gravity inputs consistently have larger weight magnitudes than last action and command inputs.The heat map uses darker colors to indicate larger magnitudes.Source: A. Kinematic Prior in Weights · pp. 5–6
  • Across motions, normalized joint-wise importance distributions are highly consistent, with waist and both hip-pitch joints most important, followed by other leg joints and then arms.Joint importance is computed from mean absolute weights across 128 hidden nodes and averaged over 50 random seeds.Source: A. Kinematic Prior in Weights · p. 6
  • The first-layer heat maps and per-joint curves show the kinematic prior in both RL- and IL-trained networks.For each motion, the dashed and solid curves largely overlap after distillation.Source: Figure 5 · p. 6, Figure 6 · p. 7

B. RL and IL Weight Comparison

The paper compares RL policies with IL policies distilled from them to test whether first-layer kinematic structure depends on the training paradigm. The patterns are similar, although the resulting networks are not interchangeable by weight interpolation.

B. RL and IL Weight Comparison

  • Each IL network is trained by offline supervised MSE regression on demonstrations collected by rolling out a pretrained RL network.The study distills 50 RL networks per motion and evaluates the resulting networks in simulation.Source: B. RL and IL Weight Comparison · p. 7, Table IV · p. 7
  • RL- and IL-trained networks show similar first-layer kinematic priors, and their normalized per-joint importance curves nearly overlap across motions.The comparison averages over 50 networks for each training paradigm and motion.Source: B. RL and IL Weight Comparison · p. 7
  • Despite similar structure, RL weights have larger magnitudes, and interpolating RL and IL weights does not produce a network that completes the motion.The paper therefore distinguishes shared kinematic structure from direct functional equivalence in weight space.Source: B. RL and IL Weight Comparison · p. 7

C. Effect of Domain Randomization on Weight Space

The domain-randomization comparison finds substantial structured weight changes despite similar overall structures, while the conclusion reports a first-layer kinematic prior across motions. The authors also identify limits in layer, motion, and embodiment coverage.

C. Effect of Domain Randomization on Weight Space

  • Domain randomization (DR) preserves similar overall weight structures while producing substantial, structured differences that resemble RL-trained weights and cannot be represented as a simple linear displacement.The difference weights alone are not a motion-capable network, and linear interpolation between DR and non-DR networks does not work across all motions.Source: C. Effect of Domain Randomization on… · p. 7, Figure 7 · p. 8

V. LIMITATIONS AND FUTURE WORK

  • The analysis is mainly limited to first-layer weights; deeper layers, domain-randomization effects, and generality across motions and embodiments remain insufficiently understood or validated.Future work may extend analysis to deeper layers and use more diverse datasets across motions and embodiments.Source: §V · p. 7

VI. CONCLUSION

  • First-layer weights consistently encode the robot’s kinematic prior across motions, and RL- and IL-trained networks exhibit similar weight-space structure.The conclusion presents these findings as insights into humanoid network-weight structure and interpretability.Source: §VI · p. 7

APPENDIX

The appendix describes the RL actor-weight dataset and autoencoder training choices, then compares cluster-wise and single-cluster autoencoders. With identical latent capacity, the cluster-wise design has lower reconstruction error.

VII. DATA COLLECTION

  • The dataset contains actor weights from RL training; actors use three-layer MLPs with hidden dimensions (128, 128), and eight post-convergence checkpoints are retained per seed and motion.Training runs for up to 4,500 iterations with 4,096 parallel environments, recording checkpoints every 100 iterations after convergence.Source: §VII · p. 9

A. Training Details

  • Autoencoder experiments use unnormalized actor weights, which had lower reconstruction error than normalized weights, and inject Gaussian noise into inputs and latents only during training.The noise augmentation is disabled at inference.Source: A. Training Details · p. 9

1) Multi Cluster vs. Single Cluster:

  • With the same 2048-dimensional latent capacity, the multi-cluster autoencoder consistently achieves lower average MSE reconstruction error than the single-cluster variant.The authors suggest separating parameter groups with distinct magnitude distributions reduces optimization difficulty; Table VII reports the cluster-structure ablation across three motions.Source: 1) Multi Cluster vs. Single Cluster · p. 9, Table VII · p. 9

IX. DIFFUSION TRANSFORMER

A frozen autoencoder maps actor weights to 2048-dimensional latents that a Diffusion Transformer models. Generated weights largely lie within the training distribution, while first-layer heatmaps show a consistent joint-related structural principle across motions.

A. Latent Normalization

  • The frozen autoencoder encodes each actor as a 2048-dimensional latent independent of the original weight dimension, allowing a unified diffusion architecture across motions.Per-dimension z-score normalization and fixed-length patches precede DiT processing; generated latents are denormalized and decoded to recover actor weights.Source: A. Latent Normalization · p. 10, Table VIII · p. 10

A. DiT Memorization vs. Generalization

  • Generated walking and dancing weights largely lie within the training distribution, indicating that the method learns the data distribution.The comparison visualizes 100 training weights and 100 DiT-sampled weights for each motion.Source: A. DiT Memorization vs. Generalization · p. 10

A. Heatmaps for all motions

  • Across walking, dancing, and spin kick, first-layer heatmaps show the same joint-related structural principle described for walking.Additional dancing and spin-kick heatmaps include zoomed views of the joint-position and joint-velocity regions.Source: A. Heatmaps for all motions · p. 10

B. Subspace Similarity Among Motions

First-layer subspaces are highly similar across seeds within a motion and remain substantially similar across motions. RL-to-IL distillation achieves functional equivalence, and heatmaps show a consistent kinematic prior across the motions examined.

B. Subspace Similarity Among Motions

  • Within-motion seed pairs have CKA of 0.92–0.94, while cross-motion pairs have 0.69–0.82, indicating shared kinematic knowledge with motion-specific modulation.The analysis uses linear CKA to compare first-layer subspaces; Fig. 9 summarizes similarity across motions.Source: B. Subspace Similarity Among Motions · p. 10, Figure 9 · p. 11

C. RL and IL distillation for all motions

  • Distilled IL networks achieve performance comparable to their RL teachers in success rate and tracking error, showing functional equivalence.Each student is trained by mean-squared regression on its teacher’s rollout buffer, without environment interaction or reward signals.Source: C. RL and IL distillation for all… · pp. 10–11
  • Across motions, RL- and IL-trained first-layer heatmaps show a kinematic prior consistent with the walking results.The visualizations include dancing and spin kick, with zoomed views of joint-position and joint-velocity regions.Source: C. RL and IL distillation for all… · p. 11

D. Details of domain randomization in RL algorithm

Domain randomization varies fixed physical and sensing parameters across simulated environments while keeping each environment’s settings fixed during training. The weight-space analysis documents actor inputs and outputs and examines first-layer structure across motions and training methods.

D. Details of domain randomization in RL algorithm

  • Domain randomization varies foot-ground friction, torso center-of-mass offset, and per-joint measured-position bias, sampled once per environment and held fixed through training.Without domain randomization, these components are disabled while other environment aspects remain unchanged.Source: D. Details of domain randomization in… · p. 11

XII. INPUT MEANING

  • Across motions, actor outputs correspond to the G1’s 29 joints; walking observations are 99-dimensional, while dancing and spin-kick observations are 154-dimensional.The joint tables specify the actuated-joint order, grouped by body region and ordered left to right and proximal to distal.Source: §XII · p. 11, Table XII · p. 15, Table XIII · p. 15
  • First-layer weights consistently exhibit a structural principle across joints in dancing and spin-kick heat maps.The captions identify the heat maps for both motions but do not specify the principle’s content.Source: Figure 10 · p. 12
  • The first-layer weight comparisons are between RL and IL for dancing and spin kick.The captions identify the motions and training methods being compared but do not state the comparison’s outcome.Source: Figure 11a · p. 13, Figure 11b · p. 14

Related papers

Loading 2610.20606v1…