PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable Imitation Learning via Generative Adversarial STL Inference and Control

Wenliang Liu, Danyang Li, Erfan Aasi, Daniela Rus, Roberto Tron, Calin Belta

arXiv:2402.10310v2cs.LGcs.SYeess.SY

TL;DR

Offline imitation learning can mimic expert behavior without explaining the task, while reward-based alternatives can be difficult to interpret. This paper combines STL task inference and control synthesis in GAN-inspired training; case studies demonstrate inferred rules and policy adaptation to unseen scenarios.

  • Problem

    Offline imitation learning can lack an interpretable description of the expert's task, while reward-function correctness and adaptation can be difficult to assess or modify.

    Source: §1 · pp. 1–2

  • Method

    The method jointly uses STL inference to describe expert tasks and a policy to satisfy the inferred formula, with policy-generated negatives driving GAN-inspired iterative training.

    Source: §1 · p. 2

  • Results

    Three case studies demonstrate that the method infers STL rules from demonstrations, learns policies to satisfy them, and adapts policies to unseen scenarios by adjusting rules.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    The learned or manually adjusted STL task description supports policy fine-tuning in similar new environments without generating new expert demonstrations.

    Source: §7 · p. 10

    The STL fragment limits performance on tasks that are difficult to express in it, including MuJoCo Swimmer and Walker, where reward levels are poor.

    Source: §6.2 · p. 9, §7 · p. 10

Abstract

from arXiv · show

Imitation learning methods have demonstrated considerable success in teaching autonomous systems complex tasks through expert demonstrations. However, a limitation of these methods is their lack of interpretability, particularly in understanding the specific task the learning agent aims to accomplish. In this paper, we propose a novel imitation learning method that combines Signal Temporal Logic (STL) inference and control synthesis, enabling the explicit representation of the task as an STL formula. This approach not only provides a clear understanding of the task but also supports the integration of human knowledge and allows for adaptation to out-of-distribution scenarios by manually adjusting the STL formulas and fine-tuning the policy. We employ a Generative Adversarial Network (GAN)-inspired approach to train both the inference and policy networks, effectively narrowing the gap between expert and learned policies. The efficiency of our algorithm is demonstrated through simulations, showcasing its practical applicability and adaptability.

1. INTRODUCTION

The paper targets offline imitation learning's difficulty in revealing the task behind expert behavior, then combines STL inference and control synthesis to represent that task explicitly. GAN-inspired training and three case studies support the approach's policy-learning and adaptation aims.

1. INTRODUCTION

  • Behavioral cloning imitates experts without uncovering the task, while inverse reinforcement learning can recover rewards but is costly and difficult to interpret.The resulting reward-function correctness and adaptability are also difficult to assess, manipulate, or extend with human knowledge.Source: §1 · pp. 1–2
  • The approach combines STL inference with control synthesis to learn an explicit task formula and a policy that satisfies it in dynamic environments.Formulas can be manually adjusted for expert knowledge or new conditions, and the policy fine-tuned without new expert data.Source: §1 · p. 2
  • The policy acts as a GAN generator of negative trajectories, while the STL inference network discriminates them from expert demonstrations.Iterative training is intended to move the learned policy toward the expert policy.Source: §1 · p. 2
  • Three case studies illustrate the algorithm, including integrating user-generated rules and adapting a policy to unseen scenarios.The paper also presents bridging the gap between expert and learned policies as a contribution.Source: §1 · p. 2

2. Related Work

Prior work has addressed STL inference and STL-based control, but their integration has received limited attention. The paper distinguishes its approach through template-free neural inference and control trained with offline data.

2. Related Work

  • Earlier STL inference work mined parameters for predefined templates or learned formula structures and parameters through search, trees, enumeration, and neural networks.Other cited work used landmarks to summarize policies.Source: §2 · p. 2
  • STL control synthesis has used optimization and learning-based methods, while cited learning-based approaches considered static environments rather than dynamic ones.Methods include mixed-integer programming, gradient-based optimization, Q-learning, and model-based control.Source: §2 · p. 3
  • Compared with Xu et al. (2018), the approach learns STL formulas without a fixed template or expert intervention, using neural networks for both inference and control.Xu et al. used decision trees and a mixed-integer programming solver; the paper also identifies its GAN integration as novel relative to prior imitation learning.Source: §2 · p. 3

3. Problem Statement and Approach

The problem is to infer an STL description from labeled trajectories and learn a policy that maximizes its robustness. The approach models agent-environment trajectories, represents formula robustness and policy with neural networks, and trains inference and control together adversarially.

3. Problem Statement and Approach

  • The model uses discrete-time agent and environment states, with differentiable agent dynamics, box-constrained controls, known initial-state distribution, and unknown environment trajectories.The agent fully observes the system state, and its dynamics may be known or separately learned.Source: §3 · p. 3
  • The dataset contains equal-length trajectories labeled expert or undesired, may contain only expert samples, and supports an STL task description over state-derived features.The considered STL fragment includes Boolean connectives and eventually and always operators over time intervals.Source: §3 · pp. 3–4
  • STL robustness is a signed measure of satisfaction, and the policy objective is to maximize the inferred formula's robustness.The paper parameterizes robustness inference and the policy with neural networks, from which an STL formula can be decoded.Source: §3 · p. 4
  • Inference minimizes classification error, while control maximizes expected robustness under the inferred formula.The two subproblems are parameterized by separate inference and policy networks.Source: §3 · p. 4
  • Adversarial training addresses insufficient negative examples by adding policy-generated trajectories as negatives and retraining the inference network iteratively.The policy seeks expert-like trajectories that are difficult to classify, while the inference network distinguishes expert from non-expert behavior.Source: §3 · p. 4

4. Neural Network Architectures

The method uses TLINet to infer interpretable STL formulas and an RNN policy to control systems using state history.

4.1. STL Inference Based on Neural Networks

  • Figure 1 distinguishes inference-network training with red arrows from policy-network training with blue arrows.Source: Figure 1 · p. 5
  • TLINet learns formula structure through predicate, temporal, and Boolean modules, without being constrained to a fixed network structure.The modules learn predicate type and spatial parameters, temporal operator type and parameters, and Boolean operator type and formula structure, respectively.Source: §4.1 · pp. 4–5
  • TLINet maps input signals to estimated STL robustness, positive exactly when a signal belongs to the positive class and satisfies the inferred formula.Its smooth robustness approximation supports gradient-based methods and is described as sound.Source: §4.1 · p. 5
  • The inferred network parameters can be decoded and translated into an STL formula.Figure 2 illustrates a five-layer inference network.Source: §4.1 · p. 5, Figure 2 · p. 5

4.2. Recurrent Neural Network Control Policy

  • The policy is an RNN because satisfying an STL specification can require past states as well as the current state.A hyperbolic tangent on the RNN output enforces the control constraint U.Source: §4.2 · p. 5

5. Training of Inference and Policy Networks

Training alternates between inference and policy optimization, then iteratively adds policy rollouts as negative examples and selects the policy with the highest DTW-based performance score.

5.1. Training of Inference Network

  • The inference loss balances formula complexity, margin maximization, and misclassification rate, rewarding formulas satisfied by positive data and violated by negative data.The margin measures signal separation by the inferred formula.Source: §5.1 · p. 6

5.2. Training of Policy Network

  • Policy training estimates the objective by sampling initial states and environment trajectories, rolling out the closed-loop system, and averaging the results.Differentiable dynamics and neural networks allow gradients to be backpropagated; the method resamples and optimizes with Adam.Source: §5.2 · p. 6

5.3. Generative Adversarial Training

  • Each adversarial iteration trains the inference network, trains the policy, and adds negatively labeled policy rollouts to the dataset.If the original dataset has only positive examples, negative trajectories are first generated using a random policy; negative samples are then balanced against expert demonstrations.Source: §5.3 · p. 6
  • The final policy is the candidate with the highest DTW-based performance score, and its corresponding STL formula serves as the interpretation.The score compares policy and expert trajectories under the same initial conditions and environment trajectories.Source: §5.3 · p. 7

6. Experiments

Three case studies evaluate random navigation, MuJoCo control, and autonomous driving. They examine task description with STL formulas, performance against baselines, and adaptation to changed or unseen scenarios.

6. Experiments

  • The experiments compare the method with BC in random navigation and GAIL in MuJoCo, and also test integration of human-generated formulas in autonomous driving.The three case studies include OOD generalization in navigation and adaptation to previously unseen driving scenarios.Source: §6 · p. 7

6.1. Random Navigation

  • In navigation, the learned STL formula closely reproduces the expert’s intended specification, and the learned policy achieves relatively high robustness and success but does not match BC’s performance.The reported statistics use 1,000 random initial states for each network initialization.Source: §6.1 · pp. 7–8
  • After fine-tuning with the same inferred STL formula and no new expert data, the navigation policy retains relatively high robustness and success while BC performance drops significantly.The OOD environment changes the x-coordinates of the three regions; y-coordinates remain random.Source: Figure 3 · p. 8, §6.1 · p. 8

6.2. MuJoCo Tasks

  • Both methods reach the 50-step maximum reward on Inverted Pendulum, while this approach slightly outperforms GAIL on Reacher.Table 2 reports rewards for the different policies; the text gives these comparisons without specific values.Source: §6.2 · p. 8, Table 2 · p. 9

6.3. A Self-Driving Scenario

  • In the original driving scenario, the ego vehicle stops when it detects the other vehicle braking for a pedestrian and continues without slowing when there is no pedestrian.The sampled trajectories show ego and other-vehicle positions for situations with and without pedestrians.Source: Figure 5 · p. 9, §6.3 · p. 10

6.2. MuJoCo Tasks

  • For MuJoCo tasks such as Swimmer and Walker, which are hard to express with the STL fragment in (2), the approach does not reach good reward levels.The authors identify increased inference-network expressivity as future work for addressing this limitation.Source: §6.2 · p. 9

6.3. A Self-Driving Scenario

  • In autonomous driving, a manually supplied speed-limit formula is conjoined with the inferred formula so the policy can incorporate that rule.The formula is used as a supplementary rule in the inference network.Source: §6.3 · p. 9
  • After the speed limit is changed from 10 to 4 for an unseen road-construction scenario, the retrained policy follows the updated limit while maintaining previous rules.The change is made by modifying the formula and retraining the policy network.Source: §6.3 · p. 10

7. Discussion

The discussion weighs interpretability against performance and data requirements. It also describes how explicit STL task descriptions support safety-rule integration and adaptation to similar environments.

7. Discussion

  • The method is less suitable for tasks that are difficult to describe with an STL formula, such as MuJoCo’s Swimmer and Walker.The inference network is not a universal classifier, and these tasks do not reach good reward levels.Source: §7 · p. 10
  • Small datasets can produce conservative, under-approximated STL formulas, and the inference network requires more expert demonstrations than traditional imitation learning.The authors also note that increasingly expert-like negative data may degrade the inferred formula and policy effectiveness.Source: §7 · p. 10
  • Explicit task descriptions allow safety constraints to be added as supplementary STL formulas and policies to be fine-tuned in similar environments without new expert demonstrations.The discussion describes these as practical benefits of the method’s interpretability and adaptability.Source: §7 · p. 10

8. Conclusion and Future Work

The method combines STL inference and control synthesis adversarially to learn task rules from expert demonstrations and a policy that satisfies them. Three case studies illustrate rule adjustment for unseen scenarios; future work targets efficiency and expressiveness.

  • The method combines STL inference and control synthesis in a generative-adversarial manner.Source: §8 · p. 10
  • Three case studies show that the method uncovers expert rules as STL formulas and learns a policy to satisfy them.Source: §8 · p. 10
  • Rules can be manually added or adjusted, and the policy adapted for unseen scenarios.Source: §8 · p. 10

Future Work

  • Future work includes improving computational efficiency and increasing inference-network expressiveness.Source: §8 · p. 10

Appendix A. Decoding and Translating TLINet

The appendix illustrates how learned TLINet parameters specify an STL formula. Module activations determine the formula's structure, which can be translated from the network.

  • The example stacks a predicate layer, a Boolean layer, two temporal layers, and a final Boolean layer.Source: Appendix A · p. 15
  • Spatial parameters a and b, operator selector κ, temporal parameters t1 and t2, and Boolean parameter w guide the modules.In the final layer, w = [0, 1] deactivates the first module and activates the second.Source: Appendix A · p. 15
  • The learned Boolean parameters' module activations determine the STL formula's structure.The example translates to ψ = □[0,10]♢[3,7]((x > 0.9) ∧ (x < −0.7)).Source: Appendix A · p. 15
  • A five-layer TLINet example is translated from learned parameters into an STL formula.Source: Figure 6 · p. 15, Appendix A · p. 15

Appendix B. Experiments Setup and Detailed Results

The appendix describes network setups and detailed experimental results, including random-navigation formula learning and policy training. In that task, the learned formula closely reproduces the expert specification, while policy trajectories progressively converge toward expert behavior.

Appendix B. Experiments Setup and Detailed Results

  • The inference network has three layers in MuJoCo tasks and gains a second Boolean layer for the self-driving case; the policy is a three-layer LSTM with 64 nodes per layer.Source: Appendix B · p. 15
  • Learned STL formulas vary slightly across initializations, and the appendix presents one formula per scenario alongside negative policy trajectories from training.Source: Appendix B · p. 15

B.1. Random Navigation

  • The random-navigation formula learned from an initialization closely reproduces the expert's intended specification.Source: §B.1 · p. 16
  • At iteration 0, random negative trajectories do not push the decision boundary near expert demonstrations; as informative negatives are added, policy trajectories gradually converge toward expert behavior.Source: §B.1 · p. 16

B.2. MuJoCo Tasks

In the MuJoCo tasks, learned STL formulas encode task constraints, including pendulum-tilt limits and Reacher target-reaching requirements. In the self-driving scenario, the formula expresses rules for ego-vehicle speed in relation to another vehicle, while policy trajectories increasingly align with expert demonstrations.

B.2. MuJoCo Tasks

  • The learned Inverted Pendulum formula includes a requirement that the pendulum not tilt positively during [21,45].Its subformula is □[21,45](θ < 0.07); the remaining formula adds system constraints.Source: §B.2 · p. 16
  • For the Reacher, the first four formula subformulas require reaching the target, while the last two describe the expert's strategy under non-unique inverse kinematics.Source: §B.2 · p. 16

B.3. A Self-Driving Scenario

  • The self-driving formula specifies ego-speed rules conditioned on whether the other vehicle continues or decelerates, including a full-stop requirement in the latter case.It also bounds ego velocity over a time window.Source: §B.3 · p. 17
  • During training, learned-policy trajectories increasingly align with expert demonstrations.Source: §B.3 · p. 17, Figure 8 · p. 18
  • Figure 8 plots ego and other-vehicle positions, with blue for ego, orange for the other vehicle, and green for expert ego demonstrations; rows distinguish pedestrian presence.Source: Figure 8 · p. 18
Loading 2402.10310v2…