Source-linked AI summary
Distributional Reinforcement Learning for Energy-Based Sequential Models
Tetiana Parshakova, Jean-Marc Andreoli, Marc Dymetman
TL;DR
Sequential EBMs can express global sequence properties, but their unnormalized potentials are difficult to use for sampling, and existing distillation is restricted to settings where sampling is possible. This paper proposes a distributional Policy Gradient approach for deriving autoregressive policies from sequential EBMs; controlled GAM experiments find performance comparable to distillation.
Problem
Sequential EBMs can represent global sequence properties, but their unnormalized potentials cannot be directly used for sampling, and distillation is limited to cases where sampling is possible during training.
Source: §1 · p. 1, §3.2 · p. 4Method
The paper adapts Policy Gradient to a distributional perspective to derive autoregressive policies from energy-based potentials, using a potential and proposal distribution as inputs.
Source: §1 · p. 2, §3.2 · p. 6, §5 · p. 8Results
Takeaways & Limitations
The DPG algorithm requires only a potential and proposal distribution, while GAMs offer an autoregressive model that can initialize the proposal and separate representation learning from extraction.
Source: §5 · p. 9The distributional transfer of policy gradient from standard RL only scratches the surface; adapting local credit-assignment methods such as actor-critic remains future work.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Global Autoregressive Models (GAMs) are a recent proposal [Parshakova et al., CoNLL 2019] for exploiting global properties of sequences for data-efficient learning of seq2seq models. In the first phase of training, an Energy-Based model (EBM) over sequences is derived. This EBM has high representational power, but is unnormalized and cannot be directly exploited for sampling. To address this issue [Parshakova et al., CoNLL 2019] proposes a distillation technique, which can only be applied under limited conditions. By relating this problem to Policy Gradient techniques in RL, but in a \emph{distributional} rather than \emph{optimization} perspective, we propose a general approach applicable to any sequential EBM. Its effectiveness is illustrated on GAM-based experiments.
1 Introduction
Sequential EBMs can represent global sequence constraints but are harder to sample from and evaluate than locally normalized autoregressive models. The paper applies a distributional Policy Gradient approach to deriving autoregressive models from EBMs and tests it in controlled GAM experiments.
1 Introduction
- The paper separates learning an energy-based representation from exploiting it to produce samples or evaluations, with the latter task being more challenging.Experiments focus on synthetic data with directly controlled conditions.Source: §1 · p. 1
- Distributional Policy Gradient (DPG) adapts policy-gradient techniques to deriving an autoregressive model from an EBM from a distributional, rather than optimization, perspective.The approach is presented as applicable to any sequential EBM.Source: §1 · p. 2
- In synthetic GAM experiments where distillation is feasible, DPG works as well as distillation, with both yielding small-data policies that reduce perplexity over standard autoregressive models.The resulting policies are also described as very similar.Source: §1 · p. 2
- Distillation [15] projects an energy-based representation into a final autoregressive model, but is limited to cases where the representation can be sampled during training.The proposed comparison with DPG is conducted under conditions where distillation is feasible.Source: §1 · p. 2
2 Model
Autoregressive models normalize each next-symbol conditional, while sequential EBMs define sequence potentials that may be difficult to normalize. GAMs combine an autoregressive factor with a global log-linear factor, using predefined sequence features to modulate the local model.
2.1 Background
- Autoregressive models define a sequence distribution through normalized next-symbol conditionals, whereas sequential EBMs define unnormalized potentials whose normalized distributions require a partition function.The model conditions on the preceding sequence prefix and context.Source: §2.1 · p. 2
2.2 GAMs
- A GAM potential multiplies an autoregressive sequence model by a log-linear global factor built from predefined features and context-dependent weights.The autoregressive factor supplies local sequence modeling, while the feature factor modulates it.Source: §2.2 · p. 3
- In the experiments, the autoregressive factor is trained on the sequence dataset and held fixed while the log-linear weights are trained on that data.Source: §2.2 · p. 3
3 Training
Training first builds a sequential energy-based representation from data, then derives an autoregressive policy from it. The second stage treats this as distribution matching rather than reward maximization, and the paper proposes DPG as an alternative to sampling-dependent distillation.
3 Training
- The GAM training procedure has two stages: it starts by training an autoregressive model r = rη1 on D by maximum likelihood, then fits the log-linear weight vector λ to the data using SGD.Source: §3 · p. 3, §3.1 · p. 4
3.1 Training-1: from data to energy-based representation
- Training-1 maximizes data likelihood by matching data and model feature moments, but estimating the model moments can require sampling or Monte Carlo methods.Rejection sampling requires a reasonable upper bound on P(x)/q(x); SNIS estimates the moments without requiring samples from pλ.Source: §3.1 · p. 4
3.2 Training-2: from energy-based representation to distributional policy
- Training-2 starts from an unnormalized potential whose partition function and normalized distribution are unavailable for direct sampling.The potential can be computed for any given sequence.Source: §3.2 · p. 4
- Rather than maximize expected potential, which would concentrate probability on a few sequences, the distributional objective minimizes cross-entropy between the EBM distribution and the policy.The paper frames the goal as approximating p with πθ in cross-entropy.Source: §3.2 · p. 4
- Distributional Policy Gradient (DPG) uses policy-gradient techniques without requiring samples from the EBM distribution, unlike distillation, which requires many such samples.Distillation is equivalent to supervised likelihood training on samples from p and is limited by restrictive rejection-sampling conditions.Source: §3.2 · p. 5
- DPGoff samples from a fixed proposal q, typically chosen to approximate p, and updates q only when the learned policy has better validation perplexity.The learned policy is initialized from q; in GAMs, r provides the initial proposal.Source: §3.2 · pp. 5–6
- DPGon had difficulty converging, likely partly because its sampling policy changes constantly, so the experiments used DPGoff instead.The authors report that DPGoff worked much better than DPGon.Source: §3.2 · pp. 5–6
4 Experiments
Synthetic binary-sequence experiments test whether DPG can match distillation while improving on an initial autoregressive model in data-scarce settings. Across the reported experiments, the methods are comparable, with metric-specific trade-offs.
Experimental setup
- Experiments use synthetic length-30 binary strings filtered for a fixed motif, with training size varied and a five-binary-feature GAM including one motif feature and four distractors.The setup also varies motifs and random seeds; all Table 1 experiments use SNIS for Training-1.Source: §4 · pp. 6, 8
Overall Training: Distillation vs. DPG
- The experiments compare distillation and DPG in Training-2 after SNIS-based Training-1, evaluating test cross-entropy and generated motif frequency.The test set is fixed at 5·10^3 sequences while training-set size varies from 500 to 2·10^4.Source: §4 · p. 7
- For both distillation and DPG, the final policy’s test cross-entropy approaches the true-process entropy and is much lower than the initial autoregressive model’s in small-data settings.As the dataset grows, the initial model becomes strong enough that the two-stage training cannot improve it.Source: §4 · p. 7
- In small-data conditions, the final policy produces the target motif more frequently than the initial model.The motif-frequency improvement is reported alongside the cross-entropy gains.Source: §4 · p. 7
Effectiveness of DPG in approximating p
- When the normalized EBM can be estimated, DPG’s policy has cross-entropy extremely close to that of the EBM, including for an alternative EBM with an exactly computable distribution.The experiments use importance sampling to estimate the partition function for the learned EBM.Source: §4 · p. 8
Results
- Distillation and DPG have comparable efficiency, but their slight advantage reverses by metric: distillation has better perplexity, while DPG has better motif frequency.The table summarizes ratios across motifs, feature sets, and seeds; the supplied passages do not include its cell values.Source: §4 · pp. 7–8, Table 1 · p. 8
5 Conclusion
The paper proposes RL-inspired methods for obtaining distributional policies that approximate normalized distributions associated with sequential EBMs. Controlled synthetic experiments provide initial evidence that the methods work, while the authors identify local credit assignment as a promising direction for further research.
Conclusion
- The paper proposes RL-inspired techniques for finding distributional policies that approximate the normalized distributions associated with energy-based sequence models.The authors report first experimental steps under controlled synthetic conditions.Source: §5 · pp. 8–9
- The authors note that restricting the work to sequential EBMs may be less limiting because many objects can be represented as sequences of actions.Source: §5 · p. 8
- DPGoff is generic in requiring only a potential P(x) and proposal q, while GAMs offer an autoregressive policy r that can initialize the proposal.The authors distinguish learning the energy representation from extracting an autoregressive model from it.Source: §5 · p. 9
- The authors relate Training-1 to Inverse RL because it derives a reward from training data within a maximum-likelihood approach, while Training-2 connects to RL policy learning.They argue that combining both aspects in one direct algorithm would blur the distinction.Source: §5 · p. 9
- The paper frames optimization as an extreme case of distributional sampling at low temperature and suggests this view may inform stable algorithms for standard RL.A related approach is attributed to [13].Source: §5 · p. 9
- The proposed policy-gradient transfer is an initial step; adapting local credit-assignment methods such as actor-critic to EBM sampling remains a future direction.Source: §5 · p. 9
A.1 Related Work
The related work distinguishes this paper’s distributional sampling perspective from prior work focused on sequence inference as optimization. It situates GAMs among global sequence models and RL approaches to sequence generation.
GAMs
- GAMs [15] introduced two-stage training but proposed only a limited-use distillation method for Training-2; the present work makes the RL connection a central focus.Source: §A.1 · p. 13
Global and energy-based sequence models
- Prior global sequence models [1] and energy-based CRF generalizations [3] focus on inference as optimization, while the present work focuses on sampling.Source: §A.1 · p. 13
- Energy-based generative work on non-sequential objects [9] uses GAN-type discriminators and does not exploit RL connections.Source: §A.1 · p. 13
RL for seq2seq
- Other RL approaches to seq2seq combine likelihood and reward-based optimization, including actor-critic methods and integrated training approaches and [13].Source: §A.1 · p. 13
Distributional RL
- Work on distributional RL [4] models the distribution of returns during policy evaluation but still seeks an optimizing policy, unlike this paper’s distributional sampling objective.Source: §A.1 · p. 13
A.2 Rejection Sampling vs. SNIS in Training-1
Training-1 estimates model feature moments for the unnormalized EBM, using rejection sampling or SNIS with the autoregressive model as proposal. In the reported comparison, the two approaches produce similar results.
Training-1
- Estimating the EBM’s model feature moments is difficult because the unnormalized model generally cannot be sampled from directly.The approaches described use the autoregressive model r as a proposal distribution.Source: §A.2 · p. 14
Rejection Sampling
- Rejection sampling uses r as proposal and requires an upper bound on the target-to-proposal ratio; accepted samples are unbiased for estimating model moments.The text states that binary features make it possible to upper-bound the ratio.Source: §A.2 · p. 14
SNIS
- SNIS also uses r as proposal, requires no upper bound, and estimates expectations from importance-weighted samples; its finite-sample estimate is biased but consistent.Source: §A.2 · p. 14
Comparison
- For Training-1, the reported comparison finds that SNIS and rejection sampling produce very similar results; Training-2 uses distillation in that comparison.Source: §A.2 · p. 14, Figure 3 · p. 15
- The appendix notes that SNIS and DPGoff both use importance sampling as an underlying technique.Source: §A.2 · p. 14
A.3 Effectiveness of DPG in approximating p : details
To assess DPG in Training-2 independently of Training-1 quality, the authors test it on two alternative energy-based targets. DPG’s resulting policies closely match the reported target-distribution references in both cases.
A.3 Effectiveness of DPG in approximating p : details
- The experiment compares DPG on Pλ from Training-1 with DPG on an unnormalized white-noise process filtered by a motif predicate.The setup is intended to isolate DPG’s effectiveness in finding a distributional policy for an EBM from the quality of Training-1.Source: §A.3 · p. 15
- The policy obtained by applying DPG to Pλ is very close to the cross-entropy points for pλ.The partition function was accurately estimated by importance sampling under these experimental conditions, allowing the cross entropy to be computed.Source: §A.3 · p. 15
- DPG applied to the white-noise×motif model produces a policy very close to the true distribution’s line.The true line is nearly hidden by the DPG line in Figure 4.Source: §A.3 · p. 15, Figure 4 · p. 16
A.4 Beyond Binary Features in Training-1: Length
The authors extend Training-1 beyond binary features by adding two continuous, interdependent length features. With these features enabled, DPG and distillation perform similarly to their feature-off setups.
A.4 Beyond Binary Features in Training-1: Length
- The feature extension was intended to test continuous, strongly interdependent features and features requiring weaker prior knowledge than specific motifs.The authors identify binary features as a limitation of the earlier technique [15].Source: §A.4 · p. 16
- The length feature has two components, normalized length M and squared normalized length v, whose moments correspond to sufficient statistics for a normal distribution.GAMs are described as roughly matching moments of the dataset.Source: §A.4 · p. 16
- Training-1 learned the length-feature parameters using SNIS without modification or rejection sampling with a modified upper-bound computation for the interdependent components.Source: §A.4 · p. 16
- Adding the length feature leaves DPG and distillation performance rather similar to their feature-off setups.The authors speculate that longer strings may be needed to reveal its impact, because the original autoregressive model may characterize length more weakly there.Source: §A.4 · p. 16, Figure 5 · p. 17