PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking GFlowNet Training: Negative Reinforce Dynamics

Pedro Dall'Antonia, Daniel Csillag, Diego Mesquita

arXiv:2610.38994v1cs.LGcs.AI

TL;DR

Because balance errors can only be corrected on sampled trajectories, GFlowNet training needs exploration, yet the full gradient can lower expected loss by avoiding high-error regions. The paper develops Negative Reinforce Dynamics by tying the parameters of an adversarial GFlowNet–sampler game into a single-policy update. Experiments report improved distribution learning and high-reward discovery relative to Trajectory Balance, with competitive results against auxiliary exploration methods.

  • Problem

    GFlowNet balance can only be corrected on sampled trajectories, while the score term in the full gradient can lower expected loss by avoiding high-error regions without enforcing balance.

    Source: §1 · p. 1

  • Method

    NRD ties the parameters of an adversarial GFlowNet–sampler game, using the score mechanism to reinforce under-allocated trajectories within a single-policy update.

    Source: §1 · p. 2, §8 · p. 10

  • Results

    NRD improves distribution learning and high-reward discovery relative to Trajectory Balance and remains competitive with auxiliary exploration methods.

    Source: §8 · p. 10

  • Takeaways & Limitations

    NRD incorporates adversarial exploration feedback into one model without maintaining a separately trained sampler.

    Source: §8 · p. 10

    Naive NRD can admit spurious interior fixed points with violated Trajectory Balance, so simply reversing sampler direction need not recover the reward-proportional distribution.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Generative Flow Networks (GFlowNets) are models that learn to sample discrete objects proportional to a target reward. They are trained by minimizing balance errors over sampled trajectories, which inherently ties correctness to exploration: if a region is not visited, balance cannot be enforced there. Adaptive exploration is therefore crucial, yet standard training relies on a semi-gradient update that treats the sampling distribution as fixed at each optimization step. On the other hand, the full gradient reveals a second mechanism that reduces the observed errors in an anti-exploratory manner. In this paper, we establish a connection between the full gradient and min-min dynamics between the GFlowNet and a decoupled sampler, showing that the sampler reduces observed violations by avoiding high-error regions. This insight motivates a bridge between two-player games and single-policy dynamics, which underpins the design of our proposed method, Negative Reinforce Dynamics (NRD). By tying the parameters of an adversarial game together, we recover a single-policy dynamic that extends exploration while maintaining correctness. In standard benchmarks, our method outperforms existing approaches in both convergence speed and exploration efficiency, without training a separate teacher or auxiliary exploration policy.

1 INTRODUCTION

GFlowNet balance can only be corrected on observed trajectories, so adaptive exploration matters; however, differentiating through an adaptive sampler can reduce expected loss by avoiding high-error regions. The paper introduces NRD, which uses adversarial sampler feedback in a single-policy update, and reports improved convergence and exploration against standard Trajectory Balance.

1 INTRODUCTION

  • Because balance can only be corrected on observed trajectories, poorly visited regions may remain uncorrected, motivating exploratory sampling and auxiliary samplers or teacher policies.The cited approaches include ε-greedy sampling and methods that expose regions with large balance errors.Source: §1 · p. 1
  • The full gradient has a score term that can lower expected loss by avoiding high-loss regions without enforcing balance, while NRD uses adversarial feedback and ties the players’ weights.The paper frames this as a weight-tied adversarial game between the GFlowNet and sampling policy.Source: §1 · p. 2
  • In tractable population models, NRD converges globally in the free-trajectory setting and reaches positive Jeffreys-divergence thresholds sooner in one-dimensional exponential families under fast normalization.In the two-trajectory setting, its additive advantage is unbounded.Source: §1 · p. 2
  • NRD improves convergence and finite-budget exploration over standard Trajectory Balance while remaining competitive with methods that train auxiliary exploration policies.The method does not require a separately trained sampler.Source: §1 · p. 2

2 BACKGROUND

A GFlowNet aims to sample terminal objects in proportion to reward by enforcing trajectory-level balance. Training uses a sampled expected loss because complete trajectories may be too numerous to evaluate exhaustively.

2 BACKGROUND

  • A GFlowNet targets terminal objects with probability proportional to their nonnegative reward, using a forward policy to construct objects through trajectories on a directed acyclic state graph.The induced distributions are defined over both trajectories and terminal objects.Source: §2 · p. 2
  • Trajectory Balance relates the forward policy, reward, backward policy, and scalar Z on every trajectory; if it holds, terminal probabilities equal normalized rewards.This balance condition is a training criterion for obtaining the target distribution.Source: §2 · p. 2
  • The log-ratio residual is zero exactly when Trajectory Balance holds, and its square gives a nonnegative loss estimated from sampled trajectories because complete trajectories may be combinatorially numerous.With a full-support sampler, zero expected loss is equivalent to zero loss on every trajectory.Source: §2 · p. 3
  • Although standard training uses an adaptive sampler that depends on the forward policy, each update treats the current sampling distribution as fixed when computing the semi-gradient.Source: §2 · p. 3

3 RETHINKING GFLOWNET TRAINING

The section separates balance correction from sampling-policy changes to explain how the full gradient can lower loss without fixing violations. It then connects sampler games to single-policy updates, motivating an adversarial alternative to joint minimization.

3 RETHINKING GFLOWNET TRAINING

  • The section links two-player GFlowNet–sampler games to single-policy updates by identifying their parameters, a correspondence used to motivate the method that follows.Source: §3 · p. 3

3.1 EXPECTED LOSS AS A SURROGATE

  • The semi-gradient corrects balance errors on observed trajectories, whereas the score term can reduce expected loss by shifting sampling toward lower-error regions without directly correcting violations.The paper presents the score mechanism as a potential exploration signal if used in a direction that supports global balance correction.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 TRAINING DYNAMICS OF A GFLOWNET AND ITS SAMPLER

  • When separately parameterized, the GFlowNet corrects balance errors while a min-max sampler seeks trajectories with large unresolved errors; reversing the sampler update changes error avoidance into error seeking.The adversarial formulation requires a separately parameterized sampler, adding computational overhead.Source: §3.2 · p. 4

3.3 FROM TWO PLAYERS TO SINGLE-AGENT DYNAMICS

  • Tying the GFlowNet and sampler parameters in the min-max dynamics yields Naive NRD, but that update can have interior fixed points with violated Trajectory Balance.It therefore need not converge to the reward-proportional distribution, motivating an additional ingredient for NRD.Source: §3.3 · p. 5

4 NEGATIVE REINFORCE DYNAMICS

Tying sampler and GFlowNet parameters makes indiscriminate error-seeking risky: it can reinforce already over-allocated trajectories. NRD retains the balance correction while applying sampler feedback selectively to negative-residual trajectories.

4 NEGATIVE REINFORCE DYNAMICS

  • With a tied sampler and forward policy, increasing the probability of a negative-residual trajectory can expose under-allocation, but doing so for a positive residual reinforces over-allocation.The latter creates a feedback loop in which the violation attracts more samples and is reinforced.Source: §4 · p. 5
  • Because squared Trajectory Balance loss removes residual direction, Naive NRD applies adversarial feedback to both residual signs and can reinforce over-allocated trajectories.The method instead needs to retain the useful error-seeking behavior without that feedback loop.Source: §4 · p. 5, Figure 4 · p. 5
  • NRD’s sampler uses violation magnitude while retaining residual direction, applying sampling pressure to large negative residuals and excluding positive-residual trajectories from its update.Source: §4 · p. 6
  • NRD preserves the standard Trajectory Balance correction and applies the score mechanism only to under-allocated, negative-residual trajectories.In the analyzed settings, this sign-selective change recovers the balanced solution, unlike Naive NRD, which can admit spurious interior fixed points.Source: §4 · p. 6

5 THEORETICAL ANALYSIS

The analysis finds that NRD preserves the target equilibrium and the same first-order behavior as semi-gradient TB near balance, while improving progress under the stated trajectory-level and one-dimensional settings.

5 THEORETICAL ANALYSIS

  • Near any balanced solution under a differentiable parameterization, NRD’s quadratic residual correction leaves the shared linearized dynamics unchanged.The analysis uses exact-expectation continuous-time dynamics; it studies trajectory-level and one-dimensional models to characterize behavior away from balance.Source: §5 · p. 6
  • For every exploration-mixture weight ε, NRD and semi-gradient TB share a unique interior equilibrium at the target distribution and normalizer.The result assumes a finite set of complete trajectories, positive rewards, fixed backward policy, free forward logits, and a jointly learned normalizer.Source: §5 · p. 6
  • At a common interior state, NRD’s trajectory log-ratio discrepancy decreases at least as fast as TB’s, avoiding additional interior equilibria possible under the full expected-loss gradient or Naive NRD.The comparison is pointwise and does not itself guarantee contraction under arbitrary off-policy sampling.Source: §5 · p. 6
  • Figure 5 illustrates faster NRD progress than semi-gradient TB across five exponential-family examples, alongside possible failure of Naive NRD or the full expected-loss gradient.The examples include Poisson, Gaussian location and scale, exponential-rate, and binomial models; the lower panels show Jeffreys divergence over normalized time.Source: Figure 5 · p. 7
  • On-policy, NRD converges from every interior initialization to the target distribution and normalizer, with strictly decreasing discrepancy away from balance.The comparison is also strict for the trajectory-level mixture when ε < 1 and the normalizer is at its conditional stationary value.Source: §5 · pp. 6–7
  • In one-dimensional exponential families with on-policy sampling and fast normalization, NRD reaches every positive Jeffreys-divergence threshold sooner than semi-gradient TB from the same initialization.In the two-trajectory specialization, the additive hitting-time advantage can grow without bound as initial divergence increases.Source: §5 · p. 7

6 EXPERIMENTS

The experiments compare distribution fit and high-reward discovery across distribution-learning and structured-generation tasks. NRD achieves the lowest mean final estimated TV across four distribution-learning tasks and discovers more qualifying peptides than the tested alternatives, while structured-task results show some auxiliary methods can lead on particular tasks.

6.1 EXPERIMENTAL SETUP

  • The evaluation covers multimodal distribution-learning tasks, structured high-reward discovery tasks, and peptide design, using estimated terminal TV, top-200 mean log-reward, and qualifying-sequence counts, respectively.The optimizer-step budgets count policy updates, including auxiliary-policy updates.Source: §6.1 · pp. 7–8, §6.3 · p. 9

6.2 DISTRIBUTION LEARNING AND EXPLORATION

  • NRD has the lowest mean final estimated TV across all four distribution-learning tasks and the lowest estimated TV in each comparison with auxiliary exploration methods.On Rings, NRD ends at estimated TV 0.110 versus TB’s 0.639; the auxiliary-method comparison uses the full optimizer-step budgets and four-seed means with one-standard-deviation bands.Source: Figure 7 · p. 8, §6.2 · p. 8
  • NRD’s distribution-learning gains remain competitive without a separately trained sampler, while some auxiliary methods find higher-reward objects on structured tasks.The comparisons include alternative training objectives and methods with explicitly trained exploration models.Source: §6.2 · pp. 8–9, §6.1 · p. 8

6.3 EXPLORATION WITH SPARSE REWARDS

  • At the end of the peptide-discovery budget, NRD finds a geometric mean of 11,637 distinct qualifying peptides, versus 2,674 for ACE and 111 for TB.NRD’s aggregate advantage is approximately 4.35× over ACE and 105× over TB, but ACE exceeds NRD in one of four seeds.Source: §6.3 · p. 9
  • On Knapsack and all sequence tasks, NRD attains higher final mean log-reward than TB; Bags results are comparable within observed seed variability.NRD remains competitive with auxiliary exploration methods, though some sequence tasks favor ACE or Adaptive Teacher.Source: §6.3 · p. 9

7 RELATED WORK

Prior exploration approaches either alter the single-policy objective or train an additional mechanism to redirect sampling. NRD combines aspects of both while using one policy and deriving its exploration mechanism from that policy’s score term.

7 RELATED WORK

  • Exploration methods split between changing a single policy’s training objective and learning an additional mechanism that redirects sampling toward regions not yet learned.The first category includes Linex(1), f-Trajectory Balance, Contrastive Balance, and Stable-GFlowNet; the second includes ACE, SA-GFN, Adaptive Teacher, and Boosted GFlowNets.Source: §7 · pp. 9–10
  • Methods that learn an additional exploration mechanism incur the cost of training and maintaining extra models and may add hyperparameters such as allocation thresholds and reward exponents.Examples include adversarial, novelty-seeking, discrepancy-driven teacher, and sequential GFlowNet approaches.Source: §7 · p. 10
  • NRD uses one policy without extra hyperparameters beyond TB while directing training toward unresolved balance violations through the policy’s score term.Although analyzed and evaluated with Trajectory Balance, the construction could in principle be applied on top of other balance losses.Source: §7 · p. 10

8 CONCLUSION

NRD ties an adversarial sampler to the GFlowNet, selectively reinforcing under-allocated trajectories to improve discovery with a single model and no additional parameters. The accompanying appendices specify evaluation budgets and the distribution-learning environments and models.

8 CONCLUSION

  • NRD uses adversarial feedback to reinforce under-allocated trajectories, improving distribution learning and high-reward discovery while remaining competitive with auxiliary exploration methods.It trains a single model with no additional parameters or separately trained sampler.Source: §8 · p. 10

A.1 EVALUATION AND OPTIMIZATION BUDGETS

  • Evaluations retain prescribed final budgets rather than selecting best checkpoints; reported bands describe variability across four seeds, not confidence intervals.For grids and structured discovery, the reported means and sample standard deviations use four seeds; peptide counts instead use geometric means and population standard deviations in log-count space.Source: §A.1 · p. 12

A.2 DISTRIBUTION-LEARNING ENVIRONMENTS AND MODELS

  • DiffusionGrid uses 30-step lazy random walks with substantial path multiplicity, while HyperGrid builds objects by coordinate increments on a 15-by-15 state space.The appendix also specifies their targets, policy architectures, exploration rates, and optimization settings.Source: §A.2 · pp. 12–13

A.3 STRUCTURED ENVIRONMENTS, OPTIMIZATION, AND FULL RESULTS

The appendix specifies structured-generation and peptide discovery tasks, evaluation protocols, and comparisons among training objectives. Structured-task endpoint rankings vary by task, while peptide discovery counts use a screened computational oracle.

A.3 STRUCTURED ENVIRONMENTS, OPTIMIZATION, AND FULL RESULTS

  • TB has the highest mean on both Bags tasks, NRD on Knapsack, Adaptive Teacher on Binary32, and ACE on the four nonbinary sequence tasks.Small mean differences on Bags and Seq24/V6 should be read with seed variability; rankings are descriptive, not significance tests.Source: Table 2 · p. 14, §A.3 · p. 14

A.4 PEPTIDE ORACLE, MODEL, AND DISCOVERY PROTOCOL

  • At step 3,000, the reported 11,637 peptide discoveries is the geometric mean of cumulative unique qualifying sequences, not a single-batch count or a sum over seeds.Discovery requires a maximum classifier positive-class probability of at least 0.95; the count has no additional biological-validity filter.Source: §A.4 · pp. 14–15

B.1 GFLOWNET LOSSES WITHOUT AN AUXILIARY EXPLORATION POLICY

  • Linex(1) changes the TB penalty with an asymmetric residual weighting, whereas NRD retains the TB semi-gradient and adds a detached negative-residual score correction.Contrastive Balance instead minimizes squared differences between paired TB residuals, canceling the shared log-normalizer; these comparisons do not exhaust possible losses or hyperparameters.Source: §B.1 · p. 15

B.2 METHODS WITH AN AUXILIARY EXPLORATION POLICY

The auxiliary-policy methods adapt exploration using additional learned components, while NRD is contrasted with maintaining a separate sampler. Recorded wall-time comparisons show task-dependent progress and do not establish a universal advantage.

B.2 METHODS WITH AN AUXILIARY EXPLORATION POLICY

  • ACE maintains a cooperative exploration GFlowNet, while SA-GFN adds a novelty-driven sibling policy and predictor and Adaptive Teacher separately optimizes a teacher and student.These approaches use learned components beyond the principal policy, unlike NRD; the comparison is not an equivalence between finite-step implementations.Source: §B.2 · p. 15

B.3 RECORDED WALL-TIME COMPARISONS

  • On Rings and Moons, NRD's lower estimated TV remains visible over substantial shared recorded-time ranges; 8 Gaussians is less separated from SA-GFN and shows appreciable seed variability.HyperGrid shows earlier NRD progress after an initial plateau, but unequal endpoints and recorded curves do not establish a universal wall-time advantage.Source: Figure 9 · p. 16, §B.3 · p. 16

C SENSITIVITY TO THE EXPLORATION DISTRIBUTION

The exploration-distribution ablation finds task- and noise-dependent effects, with broad transient benefits on structured tasks but no universal ordering. The accompanying proofs establish balance and gradient identities and analyze NRD's local and off-balance dynamics.

C SENSITIVITY TO THE EXPLORATION DISTRIBUTION

  • The ablation compares score corrections based on ∇log PF and ∇log Qε across four seeds and ε values from 0.1 to 0.8, summarizing full-curve error rather than only endpoints.For structured tasks, its FCS measure is a consistency proxy, not exact terminal-space TV or the historical top-200 reward metric.Source: C SENSITIVITY TO THE EXPLORATION… · pp. 16–17
  • The Qε correction reduces error-curve area in 30 of 32 structured task–noise combinations, reaching 45.50% on Binary32 at ε = 0.3, but only 7 of 16 grid combinations favor it.Moons favors PF at every tested noise level, and Knapsack favors PF by less than two percent at its two lowest noise levels; the results do not support a universal ordering.Source: Table 3 · p. 17, C SENSITIVITY TO THE EXPLORATION… · p. 17

D PROOFS OF THE THEORETICAL RESULTS

  • When trajectory balance holds on every complete trajectory, marginalization yields the target terminal distribution; the score-function decomposition separates loss correction from sampling-distribution effects.The standard semi-gradient corresponds to steering the forward policy toward the target via a reverse-KL identity, with a factor of two in the stated identity.Source: §D.1 · pp. 17–18, §D.2 · p. 18, §D.3 · p. 18

D.4 PROOF OF PROPOSITION 1

  • At balance, the NRD correction has zero derivative, so NRD and TB have identical Jacobians and first-order behavior under the analyzed differentiable parameterizations.The correction's scalar factor x^2 1{x < 0} and its first derivative vanish at zero.Source: §D.4 · p. 18

D.5 PROOF OF THEOREM 1

  • In the finite-trajectory model, NRD and TB share the same unique fixed point, the balanced target solution.The proof uses free forward logits, a jointly learned normalizer, and a full-support exploratory mixture.Source: §D.5 · p. 19

D.6 SPURIOUS FIXED POINTS OF THE FULL GRADIENT AND NAIVE NRD

  • The full expected-loss gradient and Naive NRD can have stationary configurations with nonzero residuals, unlike the sign-selective scheme's analyzed balanced fixed point.The appendix constructs valid two-terminal-object GFlowNet instances for those spurious configurations.Source: §D.6 · p. 19

D.7 PROOF OF THEOREM 2

  • NRD's correction cannot increase the spread between maximum and minimum trajectory log-ratios, yielding a pointwise discrepancy comparison with TB.The proof's Dini-derivative argument also handles cases where either extremum is attained by multiple trajectories.Source: §D.7 · p. 20

D.8 PROOF OF COROLLARY 1

The proofs establish convergence and faster discrepancy reduction for NRD in the stated settings. In the two-trajectory specialization, the hitting-time advantage over Trajectory Balance grows without bound as the initial discrepancy increases.

D.8 PROOF OF COROLLARY 1

  • When D > 0, the discrepancy decreases under both NRD and Trajectory Balance, and the distributions and normalizer converge to their balanced targets.The proof also establishes a strict discrepancy-decrease comparison under conditional normalization for ε < 1.Source: §D.8 · pp. 20–21
  • For ε = 1, Qε = µ is independent of the forward policy, so the NRD score term vanishes.Source: §D.8 · p. 21

D.9 PROOF OF THEOREM 3

  • In a one-dimensional exponential family under fast normalization, both dynamics converge to η⋆, while NRD moves strictly faster toward the target at every shared non-target parameter.Source: §D.9 · pp. 21–22
  • For every positive Jeffreys-divergence threshold below its initial value, NRD reaches that threshold sooner, and its Jeffreys divergence is lower than Trajectory Balance's at every t > 0.Source: §D.9 · p. 22

D.10 TWO-TRAJECTORY SPECIALIZATION

  • In the two-trajectory specialization, NRD adds a negative term to the discrepancy dynamics beyond the shared Trajectory Balance term.The two-logit dynamics share a factor-of-two time rescaling relative to a single Bernoulli natural parameter.Source: §D.10 · p. 22
  • The additive hitting-time advantage of NRD over Trajectory Balance is unbounded as the initial discrepancy tends to infinity.Source: §D.10 · p. 22
Loading 2610.38994v1…