Meta-Learning via Learned Loss

• The learned meta-loss is applied as a learning signal to optimize regressors, classifiers, or control policies.Source: Figure 1 · p. 1, §I · p. 1
• An inner loop trains an optimizee with the learned loss, while an outer loop updates the meta-loss using the task loss induced by the updated optimizee.Source: §I · p. 1
• ML3 learns adaptive, high-dimensional loss functions through back-propagation to create loss landscapes for efficient gradient-based optimization.Source: §I · p. 1
• Extra information, including exploratory signals or expert demonstrations, can shape the meta-loss during meta-training without being required during meta-test optimization.Source: §I · pp. 1–2
• The framework is evaluated on regression, image classification, behavior cloning, and model-based and model-free reinforcement learning problems.Source: §I · p. 2

II. RELATED WORK

• Meta-learning methods are organized around adapting model parameters, transforming updates with learned optimizers, learning loss or reward representations, or transferring unsupervised rules.Source: §II · p. 2
• ML3 belongs to approaches that learn loss landscapes, with loss parameters transferable across regressors, classifiers, and agent policies.Source: §II · p. 2
• Unlike methods that directly meta-learn model parameters, ML3 separates the learned loss from optimizee parameters so it can transfer across optimizee models.Source: §II · p. 2
• Compared with evolutionary policy gradients, teacher networks, and meta-gradient RL, ML3 uses differentiable loss learning without manually designing the loss parameterization or selecting particular parameters to optimize.Source: §II · p. 2

III. META-LEARNING VIA LEARNED LOSS

• ML3 learns a parametric meta-loss Mφ whose output trains an optimizee fθ by gradient descent, and it learns φ through a task-loss signal.Source: §III · p. 2

A. ML3 for Supervised Learning

• ML3 defines a meta-loss Mφ(y, fθ(x)) from ground-truth targets and predictions, producing the loss used to train a regressor by gradient descent.Source: A. ML3 for Supervised Learning · pp. 2–3
• The outer update evaluates the updated model with a task loss such as mean-squared error, then differentiates that task loss with respect to the meta-loss parameters φ.Source: A. ML3 for Supervised Learning · p. 3
• Meta-loss parameters can be optimized after each inner gradient step or after multiple steps, with the latter requiring back-propagation through the full update chain.Source: A. ML3 for Supervised Learning · p. 3
• In practice, updating φ after each inner gradient step works better, and θ is reset after M inner steps.Source: A. ML3 for Supervised Learning · p. 3

B. ML3 Reinforcement Learning

• The reinforcement-learning objective maximizes expected discounted trajectory reward, with task losses defined from rewards or surrogate policy objectives.Source: B. ML3 Reinforcement Learning · pp. 3–4
• In reinforcement learning, ML3 replaces the optimizee with a stochastic policy and learns loss functions for model-based and model-free policy optimization.Source: B. ML3 Reinforcement Learning · p. 3
• For model-based RL, the meta-loss updates the policy under a learned dynamics model during meta-training, while the meta-network can optimize policies directly without that model at meta-test time.Source: B. ML3 Reinforcement Learning · p. 3
• For model-free RL, ML3 uses mini-batches of trajectory experience and inputs sampled states, actions, policy probabilities, and task information to update high-dimensional policies with SGD.Source: B. ML3 Reinforcement Learning · p. 4

C. Shaping ML3 loss by adding extra loss information during meta-train

• Extra task information can shape the learned loss so optimization is faster with the meta-loss than with a standard loss.The paper considers extra penalties, exploratory rewards, expert demonstrations, and other task information available only during meta-training.Source: C. Shaping ML3 loss by adding extra… · p. 4

IV. EXPERIMENTS

• The experiments compare learned meta-losses trained with standard task losses against methods that add information during meta-training to shape the loss landscape.The evaluation covers generalization and convergence speed, then studies additional information for loss shaping.Source: §IV · p. 4

A. Learning to mimic and improve over known task losses

• The experiments are designed to test whether learned losses mimic and improve standard task losses across supervised and reinforcement-learning settings.The meta-network uses two hidden layers of 40 neurons each in this experiment set.Source: A. Learning to mimic and improve over… · p. 4

1) Meta-Loss for Supervised Learning:

• For supervised regression and classification, the learned meta-loss eventually outperforms direct task-loss optimization on meta-training tasks and improves performance on unseen out-of-distribution tasks.Regression uses ten meta-test tasks and classification uses four; results are averaged across five random seeds.Source: 1) Meta-Loss for Supervised Learning · p. 4, Figure 2 · p. 5

2) Learning Reward functions for Model-based Reinforcement Learning:

• In model-based reinforcement learning, ML3 generalizes from right-quadrant PointmassGoal trajectories to left-quadrant test tasks and speeds learning versus the task loss on PointmassGoal and ReacherGoal.The comparison uses final distance to the target and includes ten different goal positions, including positions requiring generalization.Source: 2) Learning Reward functions for… · p. 5, Figure 3 · p. 5

3) Learning Reward functions for Model-free Reinforcement Learning:

• ML3 requires 5 times fewer samples on average to reach 80% of task performance than PPO on model-free reinforcement-learning tasks.The result is reported for ReacherGoal and AntGoal environments and uses averages over the evaluated model-free tasks.Source: 3) Learning Reward functions for… · p. 5
• The same learned meta-loss can be reused across four policy architectures with only mild performance variation relative to task optimization variance.The comparison covers ReacherGoal and AntGoal, with averages and standard deviations over ten tasks per environment.Source: 3) Learning Reward functions for… · p. 6, Figure 4 · p. 6

1) Illustration: Shaping loss:

• The sine-frequency example uses a single parameter so the effect of reshaping the optimization landscape can be visualized directly.The target frequency is marked in the loss-landscape visualization.Source: 1) Illustration: Shaping loss · p. 6
• Using the ground-truth frequency during meta-training produces a convex learned loss landscape for sine-frequency regression.The meta-network receives targets and predictions, while the squared distance to the ground-truth frequency supplies the shaping loss.Source: 1) Illustration: Shaping loss · p. 6
• Without loss shaping, optimization is prone to getting stuck in a local optimum when using the standard MSE loss.The comparison includes standard MSE, ML3 trained with MSE, and ML3 trained with the shaping loss.Source: 1) Illustration: Shaping loss · p. 6

Physics model

• Rigid-body dynamics (RBD) provides an analytical solution for computing inverse dynamics.u is a vector of joint torques, while ˙q and ¨qdes denote joint angular velocities and desired accelerations.Source: 2) Shaping loss via physics prior for… · p. 6

Physics-prior shaping

• The method uses the simulator-provided ground-truth inertia matrix to shape the meta-loss during meta-training.The task loss is the squared difference between the predicted and ground-truth inertia matrices.Source: 2) Shaping loss via physics prior for… · p. 6

Meta-test evaluation

• During meta-test, the trained physics-shaped meta-loss optimizes inverse-dynamics networks on new trajectories.The approach learns a mapping from joint positions, velocities, and desired accelerations to joint torques.Source: 2) Shaping loss via physics prior for… · p. 6

Results

• The shaped ML3 loss outperforms the MSE loss in all inverse-dynamics evaluation cases.The comparison covers ReacherGoal and the Sawyer seven-degree-of-freedom robot arm.Source: 2) Shaping loss via physics prior for… · pp. 6–7

3) Shaping Loss via intermediate goal states for RL:

• With intermediate goal information, ML3 reaches the MountainCar goal after few policy updates, unlike iLQR and ML3 without extra information.Intermediate goals are available during meta-training but not meta-testing, where the learned loss improves exploration.Source: 3) Shaping Loss via intermediate goal… · p. 7

4) Shaping loss via expert information during meta-train time:

• Expert demonstrations provide another meta-training signal for shaping the loss landscape toward behavioral cloning in robotic-policy tasks.The experiments provide expert state-action trajectories during training, using human demonstrations or iLQR-optimized trajectories.Source: 4) Shaping loss via expert information… · p. 7

V. CONCLUSIONS

• The learned loss becomes well-conditioned for gradient descent and yields significant speed improvements in regression, classification, and benchmark reinforcement-learning tasks.The conclusion presents this as a cross-task outcome of meta-learning the loss function from data.Source: §V · pp. 7–8
• Additional guiding information during training can produce exploratory strategies that significantly improve performance during meta-testing.The extra information is used to shape the meta-loss during training rather than supplied to the environment at meta-test time.Source: §V · p. 8
• The authors identify combining multiple learned meta-losses and adding curiosity rewards as future directions for broader task generalization and exploration.These are proposed future-work directions rather than demonstrated results.Source: §V · p. 8

MBRL implementation

• ML3 for model-based reinforcement learning alternates policy optimization through a learned dynamics model with real-system rollouts that update the dynamics model.The procedure initializes the meta-loss and dynamics model, unrolls and optimizes policies, updates the meta-loss using predicted outcomes, then incorporates real trajectories.Source: APPENDIX · p. 9

MFRL and RL implementation

• ML3 for model-free reinforcement learning repeatedly rolls out policies, optimizes them with the meta-loss, and updates them across sampled training tasks.The appendix describes inner-step policy updates during meta-training and a simpler repeated rollout-optimization loop at meta-test.Source: APPENDIX · p. 9

Policy representation

• For policy optimization, directly inputting Gaussian policy means and standard deviations works better than inputting action probabilities because it directly updates distribution parameters through back-propagation.This design choice is reported for the meta-loss inputs in reinforcement-learning experiments.Source: APPENDIX · p. 9

Environment specifications

• ReacherGoal uses a four-dimensional joint state and two-dimensional torque action, with random initial configurations and goals and performance averaged over ten tasks.Its reward combines goal distance, inverse distance, and action magnitude.Source: APPENDIX · p. 9
• AntGoal evaluates a four-legged agent running to random circular goals, using a 33-dimensional state and a trajectory-distance metric averaged over ten tasks.The meta-network receives goal information as part of the state in both ReacherGoal and AntGoal.Source: APPENDIX · p. 9

Supervised evaluation

• The supervised experiments use sine regression and binary classification with fixed inner and outer learning rates of 0.001, averaging results across five random seeds.Regression uses squared error and classification uses cross-entropy, with neural-network models initialized separately for each task type.Source: APPENDIX · p. 9