PaperlayerView exact revision on arXiv

Source-linked AI summary

Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization

Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D. Lee, Wen Sun, Akshay Krishnamurthy, Dylan J. Foster

arXiv:2407.13399v3cs.AIcs.CLcs.LG

TL;DR

Offline alignment can overoptimize an imperfect reward model, degrading model quality as policies move away from human-labeled data, while KL regularization does not ensure pessimism. The paper introduces χPO, a DPO-based method using χ² regularization, and reports practical offline alignment with provable robustness and single-policy concentrability guarantees.

  • Problem

    Offline alignment can degrade true model quality as policies optimize an imperfect reward model and drift beyond human-labeled data, while KL regularization is insufficient to induce pessimism.

    Source: §1 · pp. 1–2

  • Method

    χPO is a practical offline alignment algorithm that uses χ² regularization to implement pessimism and provides guarantees based on single-policy concentrability.

    Source: §1 · p. 2, §8 · p. 21

  • Results

    The paper presents χPO as the first practical, general-purpose offline alignment algorithm with provable robustness to overoptimization and single-policy concentrability guarantees.

    Source: §8 · p. 21

  • Takeaways & Limitations

    The results add evidence for χ²-divergence’s statistical benefits in reinforcement learning and illustrate interplay between reinforcement learning theory and language modeling.

    Source: §8 · p. 21

    In the illustrative sample-starved case with C_a1 = 2n, competing with the unconstrained optimal action a1 is information-theoretically hopeless.

    Source: §4.3 · p. 12

Abstract

from arXiv · show

Language model alignment methods such as reinforcement learning from human feedback (RLHF) have led to impressive advances in language model capabilities, but are limited by a widely observed phenomenon known as overoptimization, where the quality of the language model degrades over the course of the alignment process. As the model optimizes performance with respect to an offline reward model, it overfits to inaccuracies and drifts away from preferred responses covered by the data. To discourage such distribution shift, KL-regularization is widely employed in existing offline alignment methods, but overoptimization continues to harm performance. Lending theoretical insight into the source of these empirical observations, we first show that the KL-regularization is too weak to prevent overfitting, then raise the following question: is it possible to design an efficient algorithm that is provably robust to overoptimization? We address this question with a new algorithm for offline alignment, $\chi^2$-Preference Optimization ($\chi$PO). $\chi$PO is a one-line change to Direct Preference Optimization (DPO; Rafailov et al., 2023), which only involves modifying the logarithmic link function in the DPO objective. Despite this minimal change, $\chi$PO implicitly implements the principle of pessimism in the face of uncertainty via regularization with the $\chi^2$-divergence -- which quantifies uncertainty more effectively than KL-regularization -- and provably alleviates overoptimization, achieving sample-complexity guarantees based on single-policy concentrability -- the gold standard in offline reinforcement learning. $\chi$PO's simplicity and strong guarantees make it the first practical and general-purpose offline alignment algorithm that is provably robust to overoptimization.

1 Introduction

Offline alignment can overoptimize an imperfect reward model, degrading true model quality as the policy moves beyond the human-labeled data. The paper introduces χPO, a practical DPO variant designed to address this with pessimism and theoretical guarantees.

1 Introduction

  • True model quality can degrade even as reward-model performance improves, as the policy drifts beyond regions covered by human-labeled data.Online feedback can mitigate this drift but is costly and impractical in many settings.Source: §1 · pp. 1–2
  • The paper asks whether offline overoptimization reflects insufficient information in preference data or algorithms that fail to use available data effectively.These alternatives distinguish an information-theoretic limitation from a potential algorithmic improvement.Source: §1 · p. 2

1.1 Contributions

  • χPO replaces DPO’s logarithmic link with one that implements pessimism through χ² regularization and provides single-policy concentrability guarantees.The paper presents it as a practical, general-purpose offline alignment algorithm robust to overoptimization.Source: §1.1 · p. 2
  • For general preference models, single-policy concentrability guarantees are impossible, but iterative self-play χPO obtains guarantees under a new local coverage condition.That condition is stronger than single-policy concentrability but weaker than global and unilateral concentrability.Source: §1.1 · pp. 2–3
  • The analysis combines mixed χ²- and KL-regularization to reformulate the objective as policy-based while preserving pessimism after reparameterization.It also establishes that χ²-regularized RLHF can achieve single-policy concentrability guarantees.Source: §1.1 · p. 3

2 Background

The background formulates alignment as learning a policy from offline preference pairs and reviews reward-based and direct optimization approaches. It explains how inadequate data coverage can produce overoptimization and why existing KL-regularized and pessimistic approaches have limitations.

2.1 Alignment from Human Feedback

  • Offline alignment learns a policy from preference-labeled response pairs sampled from a reference policy, with preferences modeled by Bradley–Terry.The objective is to achieve high true expected reward relative to a comparator policy.Source: §2.1 · pp. 3–4
  • Classical RLHF estimates a reward model and optimizes it with KL regularization, while DPO removes explicit reward estimation through a policy optimization objective.KL regularization is intended to keep the learned policy close to the reference policy.Source: §2.1 · p. 4

2.2 Overoptimization and Insufficiency of KL-Regularization

  • Overoptimization occurs when policy drift from the reference model is accompanied by declining model quality during optimization.The paper focuses on its statistical causes, relating sample inefficiency to inadequate offline data coverage.Source: §2.2 · pp. 4–5
  • Pessimistic methods penalize uncertain policy evaluations and can achieve single-policy concentrability, whereas non-pessimistic methods typically scale with all-policy concentrability.The latter guarantee depends on coverage across all candidate policies, leaving such methods prone to overoptimization when coverage is insufficient.Source: §2.2 · p. 5
  • Prior pessimistic guarantees can rely on linear policy classes that do not directly apply to large language models, while DPO+SFT needs restrictive convexity conditions.Without convexity, the paper reports that DPO+SFT fails to achieve a single-policy concentrability guarantee.Source: §2.2 · p. 5

3 χ2-Preference Optimization

The section develops χ2-regularized reward optimization as a way to make offline alignment pessimistic about uncertain reward estimates, then derives χPO as a DPO-like preference objective. Its main guarantee depends on the coverage of a comparator policy, under stated assumptions.

3.1 Framework: χ2-Regularized Reward Optimization

  • χ2-divergence penalizes deviations more aggressively than KL and quantifies how well reward-model accuracy transfers from the reference policy to a downstream policy.This property motivates using χ2-regularized reward optimization to implement pessimism while tuning the learned policy’s distance from the reference policy.Source: §3.1 · p. 6

3.2 The χPO Algorithm

  • χPO removes separate reward estimation by replacing DPO’s usual logarithmic terms with a new link function in its preference objective.The χ2 regularizer enforces pessimism, while a KL term preserves a barrier property that facilitates reparameterization.Source: §3.2 · pp. 6–7, §3.1 · p. 6

3.3 Theoretical Guarantees

  • The guarantee assumes a realizable policy class and bounded implicit rewards, with its bounds scaling polynomially in the reward-range parameter Vmax.The paper notes that Vmax can be measured and controlled, for example by clipping.Source: §3.3 · p. 8
  • χPO’s sample-complexity guarantee scales only with the comparator policy’s single-policy concentrability, and it is robust to overoptimization for comparators sufficiently covered by the reference policy.The theorem assumes policy realizability and bounded implicit rewards.Source: §3.3 · pp. 8–9
  • Unlike χPO’s comparator-specific coverage dependence, DPO’s sample complexity scales with all-policy concentrability, and its β choices can be overoptimizing or overly conservative.The paper says β must be tuned as a function of comparator coverage to obtain optimal dependence, and data-driven selection is unclear because relevant functionals are unknown.Source: §3.3 · p. 9
  • The statistical guarantees remain unchanged when the KL-term weight parameter γ is any value in (0, 1].The paper states that γ must be positive for the reparameterization to be admissible.Source: §3.3 · pp. 9–10

4 Understanding χPO: The Bias-Overoptimization Tradeoff

The section explains how χPO’s mixed χ2-regularization balances bias against overoptimization, then examines that tradeoff through policy properties, an illustrative example, and the role of Vmax. Its examples show a better tradeoff than DPO under reward-model error.

4.1 Properties of Optimal Policy under Mixed χ2-Regularization

  • The mixed χ2 inverse link grows roughly linearly for z ≥ 1 and exponentially for z ≤ 1, giving the optimal policy a heavier-tailed action distribution than KL-regularization.The χ2 term also yields an upper density-ratio bound that scales inversely with β.Source: §4.1 · p. 10

4.2 The Bias-Overoptimization Tradeoff

  • χPO fits the reward model with the same maximum-likelihood objective as DPO but uses a different inverse link to produce its policy.For both methods, larger β reduces overfitting to reward-model errors, while smaller β reduces bias when the reward model is correct.Source: §4.2 · p. 11

4.3 An Illustrative Example

  • In the same example, χPO preserves more probability on the optimal and reference actions, while DPO assigns more mass to the incorrect greedy action.The comparison concerns policies induced by the same possibly incorrect reward model.Source: §4.3 · p. 12
  • In the constructed bad event, DPO has no β choice that avoids either overoptimization or excessive bias, while χPO achieves a better tradeoff and converges exponentially faster.For χPO, the incorrect greedy action receives at most β^-1/n probability mass.Source: §4.3 · pp. 12–13, Figure 3 · p. 13

4.4 Nontriviality and Role of Vmax Parameter

  • The analysis uses a policy class whose bounded-bias property holds relative to each comparator, with its bound scaling as β times that comparator’s concentrability.The section relates the Vmax assumption to all-policy L∞-concentrability for the chosen χPO class.Source: §4.4 · p. 14

5 Analysis of χPO: Proof Sketch for Theorem 3.1

The proof sketch connects χPO to a mixed χ2-regularized objective through an implicit reward model. That connection lets the analysis control bias and overoptimization and obtain a guarantee based on single-policy concentrability.

Reward-model accuracy

  • The proof first bounds reward-model accuracy under the reference policy, then uses χ2-divergence to control how that accuracy transfers to another policy.The transfer bound motivates χ2-regularization because a reward model accurate under the reference policy may be inaccurate under other action distributions.Source: §5 · p. 15

Proof strategy

  • χPO’s policy optimizes the mixed χ2-regularized RLHF objective under its own implicit reward model, formally connecting the algorithm to pessimism via χ2-regularization.The proof uses this connection to establish the main guarantee.Source: §5 · p. 15

Regret decomposition

  • The regret decomposition separates bias from overoptimization, whose errors are controlled in opposing ways by the regularization parameter β.Large β increases bias; small β increases overoptimization error.Source: §5 · pp. 15–16
  • χ2-regularization cancels the on-policy change-of-measure error term, leaving an overoptimization penalty proportional to β^-1 ε_stat^2.This cancellation supports the improved guarantee’s dependence on single-policy concentrability rather than all-policy concentrability.Source: §5 · pp. 15–16

6 Experiments in Offline Language Model Alignment

In preliminary TL;DR summarization experiments, χPO has higher average winrate than DPO across the tested β and training-epoch choices. Its performance remains more robust as training continues and regularization is reduced.

Experimental setup

  • The evaluation uses the TL;DR dataset, with Pythia-1b as the reference policy and DPO as the comparison baseline.Non-algorithm-specific parameters are shared between the methods for a fair comparison.Source: §6 · p. 16

Results

  • During two epochs of training, χPO’s winrate continues to improve while DPO’s degrades; at the same β, χPO’s learned policy is significantly closer to the reference policy in KL divergence.The winrate curve records measurements every 250 steps; shaded regions show ±1 standard error over three seeds.Source: Figure 4 · p. 17, §6 · p. 17
  • Across tested β and epoch choices, χPO achieves higher average winrate than DPO; DPO degrades completely at β = 0.005, while χPO remains robust over two orders of magnitude of β.The winrate is measured against a baseline using GPT-4o judgments.Source: §6 · pp. 16–17

7 χPO for General Preference Models

For general preferences, single-policy concentrability cannot guarantee offline learning, but iterative χPO achieves guarantees under a local coverage condition weaker than global concentrability. The section also specifies the update procedure and the assumptions behind its guarantees.

7 χPO for General Preference Models

  • The framework extends beyond Bradley–Terry to preferences without transitivity, seeking an ε-approximate minimax winner from logged pairwise comparisons.Source: §7 · pp. 17–18

7.1 Impossibility of Single-Policy Concentrability under General Preferences

  • Even with a realizable preference-model class and minimax-winner L∞-concentrability bounded by 2.6, every n-sample algorithm has constant suboptimality on at least one of two instances.The instances differ only in their ground-truth preference model.Source: §7.1 · p. 18

7.2 Iterative χPO for General Preferences

  • Iterative χPO fits a skew-symmetric preference function by least squares, then iterates policy updates whose regression target is the iteration’s relative reward.The update uses an iteration-dependent reward based on the current policy.Source: §7.2 · p. 19

7.3 Theoretical Analysis of Iterative χPO

  • The theoretical guarantee assumes a realizable preference-function class, policy realizability for each update, and bounded implicit rewards.Source: §7.3 · p. 20
  • Under local coverage—approximate best responses are covered—Iterative χPO trades statistical and approximation error; under full coverage, it learns an ε-approximate minimizer with eO(C⋆/ε²) samples.The local condition does not require coverage of all policies.Source: §7.3 · p. 21

8 Discussion

The discussion presents χPO as a practical general-purpose offline alignment algorithm with provable robustness to overoptimization and single-policy concentrability guarantees. It situates the approach among prior offline-alignment and χ²-regularization work and identifies open directions.

8 Discussion

  • The authors characterize χPO as the first practical, general-purpose offline alignment algorithm with provable overoptimization robustness and single-policy concentrability guarantees.They expect the analysis and design ideas to find broader use.Source: §8 · p. 21
  • Open directions include tight minimax and instance-optimal sample complexity, guarantees for general preference models, and extensions beyond offline alignment to general MDPs.The paper notes lower bounds relevant to preference-model guarantees.Source: §8 · p. 21

A Additional Related Work

  • Some prior general-policy offline-alignment methods require uncertainty bonuses that cannot be faithfully implemented for large language models or rely on non-standard realizability assumptions.Other theoretical approaches are tailored to linear policies or require solving min-max problems over a version space.Source: A Additional Related Work · p. 28
  • Hybrid alignment methods can obtain single-policy concentrability guarantees using online feedback, but assume stronger feedback access than this work and may be unrealistic in some settings.Fully online methods dispense with coverage conditions but require active exploration.Source: A Additional Related Work · p. 29
  • χPO is a special case of a general DPO reparameterization framework, but that framework supplies neither sample-complexity guarantees nor guidance on preferred divergence choices.The paper says χPO does not appear to be a special case of Tang et al.'s general-loss framework.Source: A Additional Related Work · p. 29
  • Prior χ²-regularization work includes single-policy guarantees for contextual bandits, while the paper identifies a difference that makes its own approach suitable for large action spaces.The comparison concerns the regularizer Cπ, which the passage says is always larger in the cited work.Source: A Additional Related Work · p. 29

A.1 Detailed Comparison to DPO + SFT

The comparison argues that DPO+SFT's single-policy guarantee depends on conditions that need not hold for non-convex reward classes. A constructed instance illustrates the issue, while χPO adapts to single-policy concentrability under the stated conditions.

A.1 Detailed Comparison to DPO + SFT

  • For non-convex reward classes, DPO+SFT can fail to achieve a single-policy concentrability guarantee even when the corresponding minimax objective succeeds.The equivalence used in prior analyses is not guaranteed without convexity (Liu et al., 2024; Cen et al., 2024; Fisch et al., 2024).Source: §A.1 · p. 30
  • χPO meaningfully adapts to single-policy concentrability in the constructed instance even when the technical conditions required by DPO+SFT do not hold.The authors contrast this with an optimistic objective studied for online alignment (Xie et al., 2024).Source: §A.1 · p. 31
  • For β ≥ (2 log(n))^-1, the construction does not establish inconsistency, but DPO+SFT can incur O(β) bias when competing with the optimal policy.The passage says the required large β then yields an exponentially slower convergence rate than χPO.Source: §A.1 · p. 31
  • The lower-bound construction uses two contexts, four actions, two reward functions, and a reference policy that assigns probability 1/(2n) to each of two actions in each context.The setup samples n preference tuples from the reference policy and labels them using the Bradley–Terry model.Source: §A.1 · p. 31
  • When the preference sample contains neither a1 nor a2, the two reward functions induce the same DPO loss, allowing DPO+SFT to select the suboptimal policy under adversarial tie-breaking.The event is defined over the sampled preference tuples; under it, the objective reduces to maximizing expected log policy under πref.Source: §A.1 · p. 31

B Sample Complexity Guarantees for χ2-RLHF

The χ²-RLHF extension optimizes a smoothed χ²-regularized RLHF objective using preference data and unlabeled contexts. Its guarantees scale with comparator-specific single-policy concentrability, and smoothing avoids certain dependence on policy-ratio bounds.

B Sample Complexity Guarantees for χ2-RLHF

  • χ²-RLHF uses preference data to estimate a reward model and unlabeled contexts to optimize an empirical regularized RLHF objective, with PPO-compatible policy optimization.The unlabeled-context dataset can tighten guarantees when it is large, and the optimization step allows tolerance for computational error.Source: B Sample Complexity Guarantees for… · pp. 32–33

Theoretical guarantees.

  • The analysis assumes the reward class contains the true reward and the policy class contains an optimal policy for the regularized RLHF objective.The reward class is also bounded as specified in Assumption B.1.Source: Theoretical guarantees · p. 33
  • χ²-RLHF's sample complexity scales with the single-policy concentrability coefficient Cπ⋆ and its guarantee holds simultaneously for all comparator policies.The tuned-β bound separates reward-estimation error from context-sampling error; a larger unlabeled-context dataset can make the latter much smaller.Source: Theoretical guarantees · p. 34
  • Unlike χPO's stated guarantee, Theorem B.1 has no dependence on Vmax or quantities such as maxπ Cπ.The paper attributes this in part to the explicit reward class, which can naturally be normalized or clipped to the reward range.Source: Theoretical guarantees · p. 34
  • Smoothing with η balances clipping benefits against bias; without smoothing, a guarantee depending on maxπ Cπ is unavoidable for χ²-RLHF.The theorem uses η = β/Rmax for its compactly stated main guarantee.Source: Theoretical guarantees · pp. 33–34

C Experiment details

The appendix gives implementation and evaluation details, extends χPO to a token-level MDP formulation, and notes that its main theorem also covers a mixed KL/χ² variant. The supplied preliminaries passage states no substantive result.

C Experiment details

  • The implementation builds on the TRL DPO trainer; the authors report training instability from policy ratios and better stability and performance with a more general link function and a small grid search.The link mixes KL and χ² regularization, and the stated evaluation uses GPT-4o winrates against SFT-dataset responses on 512 validation prompts.Source: C Experiment details · pp. 35–36

D Applying χPO to the Token-Level MDP

  • In the token-level MDP, prompts are initial states, generated tokens are actions, and response trajectories are composite actions under the contextual-bandit interpretation.The paper states that Theorem 3.1 applies as-is to the token-level χPO objective.Source: D Applying χPO to the Token-Level MDP · p. 36

F Proofs for Section 3

  • The appendix analyzes a χPO variant mixing KL and χ² regularization, with χPO recovered by setting γ = 1.The paper says Theorem 3.1 follows directly from the mixed-regularization theorem at that parameter choice.Source: F Proofs for Section 3 · p. 37

F.1 General Version of Theorem 3.1

This section analyzes a mixed χ²/KL-regularized version of χPO and gives its sample-complexity guarantee under stated assumptions. The original χPO is the special case γ = 1.

F.1 General Version of Theorem 3.1

  • Theorem F.1 gives a sample-complexity guarantee for mixed χ²/KL-regularized χPO, with the original algorithm recovered at γ = 1.The general statement covers β > 0 and γ ∈ (0, 1].Source: §F.1 · pp. 37–38
  • The guarantee assumes policy realizability and bounded implicit rewards, and holds simultaneously for comparator policies.The theorem also gives a tighter statistical rate when β is chosen as a function of the comparator policy.Source: §F.1 · p. 38
  • The mixed-regularization analysis says γ changes the bound by only a small multiplicative factor, but its practical effects may be understated.γ can affect Vmax and the policy realizability condition, which the theorem may not fully capture.Source: §F.1 · p. 38
  • The proof connects the learned policy to an RLHF objective by showing its implicit reward estimate transfers beyond the reference-policy distribution with a χ²-dependent factor.The learned policy maximizes the RLHF objective using that implicit reward model.Source: §F.1 · pp. 38–39

F.1.1 Proofs for Supporting Lemmas

The supporting proofs establish reward-estimation and policy-optimization links used to analyze χPO, then invoke the general theorem for its original γ = 1 form and a corollary.

F.1.1 Proofs for Supporting Lemmas

  • The reward-based maximum-likelihood objective is treated as estimation of Bradley-Terry preference models, with error bounded under the reference-policy distribution.Clipping reward differences keeps statistical error independent of Vmax, and realizability follows when the true reward is represented up to an action-independent shift.Source: §F.1.1 · pp. 41–42

F.2 Proof of Theorem 3.1

  • The proof of Theorem 3.1 follows by instantiating Theorem F.1 with γ = 1.The policy optimization in Algorithm 1 is the corresponding special case of Eq. (40).Source: Proof of Theorem 3.1 · p. 45

F.3 Proof of Corollary 3.1

  • The Corollary 3.1 proof applies Theorem 3.1 with policy class ΠR and uses absolute constants in its bound.The supplied proof passage introduces a further parameter choice but does not state it here.Source: §F.3 · p. 46

G Proofs for Section 4

These proofs establish properties of the link function and normalization term used in Section 4’s policy expressions.

G Proofs for Section 4

  • The proof establishes that ϕ and its inverse are strictly increasing.It begins by examining the derivative of ϕ.Source: G Proofs for Section 4 · p. 46
  • The inverse link is expressed using the increasing principal Lambert W function and bounded separately for z ≥ 1 and z ≤ 1.The proof combines the two regimes to establish the stated inverse-function result.Source: G Proofs for Section 4 · p. 46
  • For rewards in [0, Rmax], the proof bounds the normalization constant Zβ,r(x) within [−β, Rmax].It then uses bounds on ϕ⁻¹ to bound the corresponding policy expression.Source: G Proofs for Section 4 · p. 47

H.1 Proof of Theorem 7.1

The proof constructs two preference-model instances and bounds the sum of their duality gaps for every policy, then treats a case separately.

H.1 Proof of Theorem 7.1

  • The construction uses one context, four actions, and two preference models defined by skew-symmetric loss functions.The deterministic policies selecting b and c are minimax winners for the two losses, respectively.Source: §H.1 · pp. 47–48
  • For every policy π, DG(π; P1) + DG(π; P2) ≥ 1/2.The proof uses this bound in applying the Le Cam two-point method.Source: §H.1 · p. 48, Case (2): π(a) + π(c) < 1 · p. 48

Case (2): π(a) + π(c) < 1

  • In Case (2), the proof obtains DG(π; P1) > 1/2, which is enough for the aggregate duality-gap bound.It then invokes the Le Cam two-point method using the Bretagnolle-Huber inequality variant.Source: Case (2): π(a) + π(c) < 1 · p. 48

H.2 Proof of Theorem 7.2

The proofs bound the policy gap using regularized policy updates and concentration results, supported by least-squares and three-point lemmas. They also establish a general regret bound for χ2-RLHF and smoothed and non-smoothed corollaries.

H.2 Proof of Theorem 7.2

  • The H.2 analysis uses a regularized RL objective and compares policies within ΠC, the set bounded in χ2-divergence from the reference policy.The objective’s minimizer is denoted πt+1, and the analysis invokes a regularized policy mirror-descent guarantee.Source: §H.2 · p. 49

H.3 Proofs for Supporting Lemmas

  • The supporting lemmas use least-squares regression concentration and a three-point lemma to bound terms in the proof.The concentration result applies uniformly over policies and iterations; the three-point lemma is stated for any three distributions p1, p2, p3.Source: §H.2 · p. 50, §H.3 · p. 52

H.2 Proof of Theorem 7.2

  • With probability at least 1 −δ, the proof combines bounds on the decomposed terms to conclude DG(bπ) ≤ 2ℓ⋆(eπ, bπ).The decomposition reduces the gap to four terms; the proof uses concentration and bounds for those terms.Source: §H.2 · pp. 49, 52

Proofs for Appendix B

  • The Appendix B proofs establish a general regret bound for Algorithm 3 and derive smoothed and non-smoothed χ2-regularization corollaries.The theorem and corollaries concern the policy produced by χ2-RLHF; the non-smoothed corollary sets η = 0.Source: Proofs for Appendix B · p. 54

Related papers

Loading 2407.13399v3…