Source-linked AI summary
Multi-Objective GFlowNets
Moksh Jain, Sharath Chandra Raparthy, Alex Hernandez-Garcia, Jarrid Rector-Brooks, Yoshua Bengio, Santiago Miret, Emmanuel Bengio
TL;DR
Multi-objective optimization seeks candidates balancing conflicting objectives, which are often imperfect proxies for properties of interest. MOGFNs use conditional GFlowNets to generate diverse Pareto-optimal candidates through scalarized or acquisition-defined sub-problems. Across molecule-to-protein tasks, the paper reports Pareto performance and diversity benefits, including improved sample efficiency and diversity in fluorescent-protein design.
Problem
Multi-objective optimization must balance conflicting objectives that are often imperfect proxies, making diverse candidate sets valuable for expensive downstream evaluations.
Source: §1 · p. 1Method
MOGFNs use reward-conditional GFlowNets for scalarization-defined sub-problems and GFlowNets within an active-learning loop for acquisition-defined sub-problems.
Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4Results
Takeaways & Limitations
The experiments support using MOGFNs to generate diverse Pareto-optimal candidates across a range of tasks, with MOGFN-AL also improving sample efficiency and diversity in fluorescent-protein design.
Source: §1 · p. 2, §7 · p. 9MOGFNs’ benefits are limited in problems where rewards have a single mode.
Source: §7 · p. 9
Abstract
from arXiv · showhide
We study the problem of generating diverse candidates in the context of Multi-Objective Optimization. In many applications of machine learning such as drug discovery and material design, the goal is to generate candidates which simultaneously optimize a set of potentially conflicting objectives. Moreover, these objectives are often imperfect evaluations of some underlying property of interest, making it important to generate diverse candidates to have multiple options for expensive downstream evaluations. We propose Multi-Objective GFlowNets (MOGFNs), a novel method for generating diverse Pareto optimal solutions, based on GFlowNets. We introduce two variants of MOGFNs: MOGFN-PC, which models a family of independent sub-problems defined by a scalarization function, with reward-conditional GFlowNets, and MOGFN-AL, which solves a sequence of sub-problems defined by an acquisition function in an active learning loop. Our experiments on wide variety of synthetic and benchmark tasks demonstrate advantages of the proposed methods in terms of the Pareto performance and importantly, improved candidate diversity, which is the main contribution of this work.
1. Introduction
MOGFNs target diverse Pareto-optimal candidate generation, combining GFlowNets with multi-objective optimization. The paper presents preference-conditional and active-learning variants and reports benefits in candidate diversity and optimization performance.
1. Introduction
- Conflicting objectives can prevent any single candidate from maximizing them all, so multi-objective optimization seeks Pareto-optimal trade-offs.Drug discovery examples include efficacy, synthesizability, and safety objectives.Source: §1 · p. 1
- Because objectives are often imperfect proxies, diverse candidates can offer alternatives for expensive downstream evaluations.The paper notes that proxy scores may not reflect properties such as human efficacy.Source: §1 · p. 1
- Existing multi-objective methods primarily seek Pareto-optimal solutions, while GFlowNets sample candidates with probability proportional to reward, encouraging diversity.The paper describes prior GFlowNet applications to molecule generation and biological sequence design.Source: §1 · pp. 1–2
- MOGFN-PC uses reward-conditional GFlowNets for scalarized sub-problems, whereas MOGFN-AL uses acquisition functions to define sequential sub-problems in multi-objective Bayesian optimization.These variants leverage GFlowNets to generate diverse Pareto-optimal candidates.Source: §1 · p. 2
- MOGFN-PC generates diverse Pareto-optimal candidates, while MOGFN-AL improves sample efficiency and candidate diversity in fluorescent-protein design.The paper also reports empirical validation of reward-conditional GFlowNets through MOGFN-PC.Source: §1 · p. 2
2. Background
The background defines Pareto trade-offs and candidate-space diversity, then reviews scalarization, Bayesian optimization, and GFlowNets as foundations for the proposed methods.
2.1. Multi-Objective Optimization
- Pareto optimality describes solutions for which improving one objective requires sacrificing another; multiple candidates can map to the same Pareto-front point, making candidate diversity important when objectives are proxies.The Pareto set contains non-dominated candidates, and the Pareto front is its image in objective space.Source: §2.1 · p. 2
2.1.1. APPROACHES FOR TACKLING MOO
- Scalarization turns objectives into weighted single-objective problems, while multi-objective Bayesian optimization uses a surrogate and acquisition function across rounds to select candidate batches.Weighted sums capture all Pareto-optimal candidates for convex fronts; weighted Tchebycheff can do so for nonconvex fronts.Source: §2.1.1 · pp. 2–3
2.2. GFlowNets
- GFlowNets construct objects through stochastic trajectories and learn a policy whose marginal probability of generating an object is proportional to its reward.The paper adopts trajectory balance to train the forward and backward policies and partition function.Source: §2.2 · p. 3
3. Multi-Objective GFlowNets
MOGFNs generate diverse Pareto-optimal candidates through two approaches: preference-conditioned modeling of scalarized subproblems and active-learning rounds guided by acquisition functions.
3.1. Preference-Conditional GFlowNets
- MOGFN-PC uses one reward-conditional GFlowNet to model scalarized subproblems, sharing structure across preferences rather than training a separate network for each.The conditional policy can generalize to preference values not seen during training.Source: §3.1 · pp. 3–4
- MOGFN-PC can use any scalarization function; the proposed Weighted-log-sum combines objective rewards in log space to favor simultaneous optimization.The authors hypothesize it may help when a weighted-sum scalar reward is dominated by a single objective.Source: §3.1 · p. 4
- Unlike reward-maximizing MOReinforce, MOGFN-PC samples from the scalarized reward distribution, producing diverse candidates from the Pareto set for a given preference.Training learns conditional forward and backward policies and a log-partition function so the sampling probability is proportional to the scalarized reward.Source: §3.1 · p. 4
3.2. Multi-Objective Active Learning with GFlowNets
- MOGFN-AL generates candidates in each round of a multi-objective Bayesian optimization loop, turning the task into sequential single-objective subproblems defined by an acquisition function.It proposes mutations of existing non-dominated candidates; the mutation reward is the acquisition-function value of the resulting sequence.Source: §3.2 · p. 4
- Mutation-based proposals scale better to longer sequences than token-by-token generation while retaining exploration for diverse candidates.Each mutation specifies a location and replacement token, and its reward is the acquisition value of the resulting sequence.Source: §3.2 · p. 4
4. Related Work
Related multi-objective optimization methods include Bayesian optimization, evolutionary algorithms, and reinforcement learning. The paper contrasts their scope and candidate-generation characteristics with MOGFNs.
4. Related Work
- Evolutionary algorithms solve each MOO instance from scratch, while RL-based methods often focus on a single reward mode; MOGFNs amortize training and sample candidates proportional to reward.Evolutionary algorithms can also be augmented with MOGFNs to generate mutations.Source: §4 · p. 5
Multi-Objective Bayesian Optimization (MOBO)
- MOBO methods learn surrogate objectives to define acquisition functions or use scalarization, but diverse candidate generation is not typically considered and prior diversity work is limited to low-dimensional continuous problems.The cited approaches include preference-based MOBO and LaMBO for multi-objective sequence design.Source: Multi-Objective Bayesian Optimization… · p. 5
- LaMOO and MORBO partition continuous candidate spaces into local regions, limiting their stated scope to continuous problems.Source: Multi-Objective Bayesian Optimization… · p. 5
5. Empirical Results
Across synthetic and benchmark tasks, the experiments provide positive evidence on preference-conditional modeling, Pareto-optimal sampling, diversity, and scaling. MOGFN-AL improves relative hypervolume and candidate diversity in the active-learning task.
5. Empirical Results
- Experiments provide positive evidence for all four questions: preference-conditional modeling, Pareto-optimal sampling, candidate diversity, and scaling to high-dimensional problems.The evaluation samples 128 candidates per preference, selects the top 10, and reports averages over preferences and three seeds.Source: §5 · p. 5
5.1.1. HYPER-GRID
- In the 32 × 32 HyperGrid with three objectives, MOGFN-PC’s distribution differs from the normalized reward distribution by about 10^-4 on average across 64 preferences and captures all modes.The generated Pareto front has an average GD+ of 0.42.Source: §5.1.1 · p. 6, Figure 1 · p. 6
5.1.2. N-GRAMS TASK
- On N-gram tasks, MOGFN-PC models trade-offs for conflicting objectives and generates candidates that jointly maximize correlated objectives; it outperforms baselines on MOO metrics while maintaining diversity.For this task, higher-scoring sequences can have lower diversity; MOReinforce and Envelope-MOQ may struggle with longer trajectories and sparse rewards.Source: §5.1.2 · p. 6
5.2.1. QM9
- On QM9, MOGFN-PC outperforms all baselines in Pareto performance and diverse candidate generation.The task uses HOMO-LUMO gap, synthetic accessibility, molecular weight, and logP rewards.Source: §5.2.1 · pp. 6–7, Table 1 · p. 7
5.2.2. FRAGMENT-BASED MOLECULE GENERATION
- On fragment-based molecule generation, MOGFN-PC consistently outperforms baselines on hypervolume, R2, and candidate diversity.MARS diversity and reward scores are omitted because it lacks preference conditioning, making that comparison unfair.Source: §5.2.2 · p. 7, Table 2 · p. 7
5.2.3. DNA SEQUENCE GENERATION
- On DNA sequence generation, MOReinforce has the best MOO performance but finds a low-diversity quasi-trivial pattern; MOGFN-PC has better diversity and Top-K rewards with slightly lower Pareto performance.Source: §5.2.3 · p. 7, Table 3 · p. 7
5.3. Active Learning
- 0.25±0.01 relative hypervolume for MOGFN-AL versus 0.18±0.03 for LaMBO; its generated Pareto frontier dominates the initial dataset and is more diverse than the baselines’.MOGFN-AL matches LaMBO’s performance within about half the number of black-box evaluations.Source: §5.3 · pp. 7–8, Figure 2 · p. 8
6. Analysis
MOGFN-PC’s Pareto performance depends on preference sampling and scalarization, while the reward exponent β requires a task-specific trade-off between performance, diversity, and training efficiency.
Impact of p(ω)
- Dirichlet(α = 1) gives the best performance, while skewed preference sampling remains close, suggesting interpolation to preferences not sampled during training.α = 1 is uniform over the preference simplex; α < 1 favors corners and α > 1 favors its center.Source: §6 · p. 8, Figure 3 · p. 8
Choice of scalarization R(x|ω)
- Weighted Sum (WS) scalarization performs best; the authors suspect less smooth reward landscapes partly explain poorer results for WT and WL.Top-K Reward is not compared across scalarizations because their rewards are not directly comparable.Source: §6 · p. 8, Figure 3 · p. 8
Impact of β
- β is task-dependent: higher values can improve Pareto performance but may reduce diversity and make optimization harder, so it trades off performance, diversity, and training efficiency.Larger β concentrates reward density near modes of R(x|ω).Source: §6 · pp. 8–9, Figure 3 · p. 8
7. Conclusion
The paper concludes that MOGFNs generate diverse Pareto-optimal candidates across tasks, while their benefits are limited for single-mode rewards. Its two variants train conditional GFlowNets either over preferences or within an active-learning loop.
7. Conclusion
- MOGFNs generate diverse Pareto-optimal candidates, with empirical efficacy demonstrated across tasks from molecule generation to protein design.MOGFN-PC models independent single-objective subproblems with conditional GFlowNets; MOGFN-AL optimizes a sequence of such subproblems.Source: §7 · p. 9
- MOGFN benefits are limited in problems where the rewards have a single mode.The conclusion identifies this as a limitation and refers to Section 5.2.3.Source: §7 · p. 9
A. Algorithms
- MOGFN-PC samples preferences, generates trajectories with a conditional GFlowNet, computes reward R(x|ω)^β and its loss, then updates model parameters.The training inputs include the preference distribution, reward exponent, and uniform-action mixing coefficient.Source: A. Algorithms · p. 13
- MOGFN-AL repeatedly fits a surrogate, trains a GFlowNet to propose mutations from non-dominated candidates, evaluates a batch, and adds it to the dataset.The loop returns an approximate Pareto set after the active-learning rounds.Source: A. Algorithms · p. 13
B. Additional Analysis
MOGFN-PC approaches the reward performance of separately trained single-objective GFlowNets in two tasks; increasing model size helps on 3 Bigrams before performance plateaus.
B. Additional Analysis
- The reward ratio stays close to 1, indicating MOGFN-PC models the reward family at least as fast as a single-objective GFlowNet, and it occasionally surpasses it.The caption reports the occasional surpassing; the main-text analysis describes the ratio as close to 1.Source: B. Additional Analysis · p. 13, Figure 4 · p. 14
- The Figure 4 comparison plots the MOGFN-PC-to-single-objective reward ratio for candidates across preferences in 3 Bigrams and fragment-based molecule generation.The comparison uses mean top-10 rewards averaged over ten preferences.Source: B. Additional Analysis · p. 13, Figure 4 · p. 14
- On 3 Bigrams, larger MOGFN-PC models improve performance, but gains plateau after a point.The authors suggest better training objectives may be needed to use model capacity fully.Source: B. Additional Analysis · p. 14, Table 4 · p. 14
C. Metrics
The paper evaluates Pareto approximations with distance-, volume-, and reference-vector-based metrics, plus reward and diversity measures. Hyper-Grid results show increasing loss and GD+ as reward dimensionality grows.
C. Metrics
- GD+ measures Euclidean distance from an approximation to the true Pareto front while accounting for dominance, and is reported only for Hyper-Grid experiments.It requires knowledge of the true Pareto front.Source: C. Metrics · p. 14
- Hypervolume measures objective-space volume relative to a reference point, while R2 monotonically compares Pareto approximations using uniform reference vectors and a utopian point.The experiments use the origin as the utopian point and transform objectives to minimization.Source: C. Metrics · pp. 14–15
- Top-K Reward averages the highest scalarized rewards among sampled candidates across test preferences; Top-K Diversity averages candidates selected by diversity scores across preferences.Diversity uses a distance threshold, with edit distance for sequences and one minus Tanimoto similarity for molecules.Source: C. Metrics · p. 15
D.1. Hyper-Grid
- The Hyper-Grid task is an n-dimensional gridworld where an agent moves right, down, or stops to receive a non-zero reward; the listed reward functions are normalized between 0 and 1.The reward functions are Branin, Currin, Sphere, Shubert, and Beale.Source: §D.1 · p. 15, Figure 5 · p. 15
- As reward dimension increases, both L1 loss and GD+ increase; the paper attributes this to greater problem difficulty with more rewards.The appendix also presents qualitative visualizations across more preferences.Source: §D.1 · pp. 15–16, Figure 6 · p. 16, Figure 7 · p. 16
D.2. N-grams Task
The N-grams task tests MOGFN-PC on sequence-generation problems with varied objective spaces. Rewards count selected n-grams, while sequence length and n-gram choice shape objective conflicts.
D.2. N-grams Task
- Sequences are generated from 20 characters plus an end token, with maximum length 36 and rewards based on counts of selected n-grams.The experiments also vary the number of objectives and normalize rewards to [0, 1].Source: §D.2 · p. 16
- Unigram counts create conflicting objectives, whereas shared characters in bigrams can let multiple objectives increase together at fixed length.The passage illustrates the reward calculation with the bigrams AB and BA on the sequence ABABC.Source: §D.2 · p. 16
- For sequence generation, each state has only one parent, making the backward policy trivial; the forward policy and log Z are parameterized.The conditional Transformer embeds the partial string and preferences; training uses 10,000 iterations with minibatches of 128.Source: §D.2 · pp. 16–17
Additional Results
Across the additional N-grams results, MOGFN-PC outperforms baselines in Pareto performance while maintaining high diversity. The appendix also details QM9 and fragment-based molecule experiments, including preference-conditioned generation behavior.
Additional Results
- MOGFN-PC outperforms the N-grams baselines in Pareto performance while achieving high diversity.The reported additional results cover the 3-bigram and 3-unigram tasks as well as other objective counts and sequence lengths.Source: Additional Results · pp. 17–18, Table 7 · p. 18
- Its advantage grows with sequence length and number of objectives, and it remains significant for shorter sequences.The tables cover two- and four-objective settings and shorter sequences of length 24.Source: Additional Results · p. 17, Table 8 · p. 18, Table 9 · p. 18
D.3. QM9
- QM9 rewards comprise predicted HUMO-LUMO gap, synthetic accessibility, a molecular-weight target maximized at 105, and a logP target maximized at 2.5.The HUMO-LUMO prediction uses a pretrained MXMNet trained on QM9; synthetic accessibility is transformed as (10 − SA)/9.Source: §D.3 · p. 19
- QM9 generation uses a graph-transformer GNN conditioned on sampled preferences and sampling temperature, with graph-edit actions and a stop action.Preferences are sampled from a Dirichlet distribution and supplied through a virtual node and node embeddings.Source: §D.3 · p. 19
D.4. Fragments
- Fragment-task rewards include an sEH docking proxy, synthetic accessibility, a clipped weight reward favoring molecules under 300, and QED.The sEH proxy uses weights from Bengio et al. (2021a).Source: §D.4 · p. 19
- Fragment generation adds a node by selecting a building-block fragment and an attachment point, using a graph-transformer GNN and a lagged model.The lagged parameters are updated as θ′ = τθ′ + (1 − τ)θ.Source: §D.4 · p. 19
- Figure 8 shows near-Pareto-optimal fragment samples and preference conditioning, while retaining a diverse distribution rather than collapsing to one point.Its off-diagonal panels plot objective pairs with the Pareto front in red; diagonal panels show objective histograms and preference-versus-reward scatter plots.Source: §D.4 · pp. 19–20, Figure 8 · p. 20
D.5. DNA Sequence Design
The DNA sequence experiments compare MOGFN-PC with multi-objective reinforcement learning, while the active-learning section replaces LaMBO candidate generation with GFlowNets. The DNA results show a diversity advantage alongside a Pareto-performance gap, and the active-learning setup uses NEHVI to guide mutation generation.
D.5. DNA Sequence Design
- The DNA task uses A, C, T, and G plus an end token, with free energy and base-pair objectives computed at 310 K and rewards normalized to [0, 1].The inverse-length objective is 30/L.Source: §D.5 · p. 20
- MOGFN-PC does not match MOReinforce’s Pareto performance on DNA aptamers, with short-length energy and pair-count optimization identified as a weakness.Energy and pair-count optimization is reported as very good for the longest sequences; the authors conjecture that hyperparameters or algorithm changes could improve performance.Source: §D.5 · pp. 20–21
- 0.62 versus 19.6 or higher: MOReinforce’s best DNA sequences have lower diversity than MOGFN’s optimal sequences despite better Pareto performance.MOReinforce mostly samples repeating GCGC patterns, while GFlowNets generate other patterns and can insert A and T into GCGC chains to obtain lower energy.Source: §D.5 · pp. 20–21
- DNA hyperparameter sweeps show a trade-off between Top-K diversity and Pareto performance; α < 1 and higher β improve Pareto metrics but slightly reduce diversity.The β observation is consistent with the Bigrams analysis, while the preferred α differs, indicating task dependence.Source: §D.5 · p. 21, Table 14 · p. 21
D.6. Active Learning
- For active learning on red fluorescent protein variants, the objectives are stability and solvent-accessible surface area, with 512 initial candidates.The surrogate combines a convolutional encoder with a multi-task Gaussian process, and the acquisition function is NEHVI.Source: §D.6 · pp. 21–22
- MOGFN-AL replaces LaMBO candidate generation with GFlowNets that generate sequence mutations and receive acquisition-value rewards.The procedure trains for 750 iterations per round, with batch size 16 over 64 rounds, and linearly reduces β by a factor δβ each round.Source: §D.6 · p. 22