Source-linked AI summary
Entropic Control for Homeostatic Optimization: Pre-empting Catastrophic Forgetting via Dynamic Feedback
Shangxi Wu, Xiaorong Yan, Xiaojun Meng, Yuxiang Zhang, Jiansheng Wei
TL;DR
Supervised fine-tuning can cause catastrophic forgetting, while existing strategies struggle to balance retaining general capabilities with adapting to downstream tasks. ECHO uses output-entropy feedback to regulate replay weight, and evaluations report that it alleviates this dilemma across model scales and tasks.
Problem
SFT can erode pre-training capabilities as it improves downstream task performance, creating a plasticity-stability dilemma that existing mitigation strategies do not readily resolve.
Source: §1 · p. 1Method
ECHO uses output entropy as a reference-free stability signal and adjusts replay weight through a closed-loop controller when entropy deviates from the pre-trained equilibrium.
Source: §1 · p. 2, §3.2 · p. 4, §3.2.1 · p. 4Results
Takeaways & Limitations
ECHO regulates replay without a frozen reference model, strengthening replay only when entropy deviates from the pre-trained equilibrium.
Source: §1 · p. 2, §6 · p. 9Experiments cover representative high-interference tasks, and the authors identify longer training horizons, larger scales, broader alignment settings, and richer control strategies as areas for further study.
Source: Limitations · pp. 9–10
Abstract
from arXiv · showhide
Supervised Fine-Tuning (SFT) of LLMs suffers from catastrophic forgetting, and prevailing mitigation strategies struggle to reconcile the plasticity-stability dilemma. To address this, we propose Entropic Control for Homeostatic Optimization (ECHO), a closed-loop control framework. By decomposing the KL divergence, we identify output entropy as a dominant observable component of KL variation. This enables us to leverage entropy as a computationally efficient online proxy to monitor forgetting risks in real-time, bypassing expensive reference-model inference. Consequently, ECHO employs an elastic controller that dynamically modulates the gradient weight of replay data via a proportional control law, using deviations from the pre-trained entropic equilibrium as an error signal. Empirical evaluations across 0.6B to 9B parameter architectures demonstrate that ECHO lies on, and in several comparisons extends, the observed plasticity-stability frontier and maintains high operational robustness against varying data mixing ratios.
1 Introduction
SFT can improve downstream performance while eroding capabilities acquired during pre-training, and existing mitigation strategies face computational costs or fail to adapt replay to changing training dynamics. ECHO uses output-entropy feedback to dynamically regulate replay, and evaluations place it on or beyond the observed plasticity-stability frontier.
1 Introduction
- SFT’s plasticity-stability dilemma arises because downstream updates can improve new-task performance while degrading general capabilities acquired during pre-training.Reference-model KL regularization adds computational and memory overhead, while static replay ratios can suppress early drift yet hinder later task adaptation.Source: §1 · p. 1
- Output entropy accounts for the larger share of KL variation, making it a lightweight, reference-free proxy for monitoring fine-tuning health.The cross-entropy component remains comparatively stable during fine-tuning.Source: §1 · pp. 1–2
- ECHO turns replay into a feedback control mechanism that adjusts replay weight according to deviations from pre-trained prediction entropy.It relaxes constraints when the model is stable and amplifies replay when abnormal entropy deviations arise.Source: §1 · p. 2
- ECHO lies on, and in several comparisons extends, the observed plasticity-stability frontier while outperforming static mixing baselines.The evaluations also report virtually negligible computational and memory overhead during optimization.Source: §1 · p. 2
2 Related Works
Prior work addresses forgetting through output-distribution regularization or data replay, but the described approaches use fixed coefficients, schedules, or mixing ratios rather than responding to instantaneous stability. These limitations motivate the need for closed-loop control of replay intensity.
2.1 Regularization & Alignment
- Reference-free alignment and weight-deviation methods reduce costs but rely on static coefficients or preset schedules, so they cannot adjust constraint intensity to real-time drift.KL regularization against a pre-trained reference can effectively double computational and memory overhead.Source: §2.1 · p. 2
- Across both categories, the described approaches do not regulate constraint or replay intensity through real-time feedback on model stability.Regularization methods use static coefficients or preset schedules; replay approaches lack an intrinsic closed-loop mechanism.Source: §2.1 · p. 2, §2.2 · p. 2
2.2 Data Replay & Mixing
- Data-replay approaches use static mixing ratios that constrain the plasticity-stability trade-off, while surprise-based replay selection remains primarily open-loop.The passage identifies the absence of intrinsic feedback that modulates replay intensity based on instantaneous stability.Source: §2.2 · p. 2
3 Methodology
ECHO reframes fine-tuning as closed-loop control, using output entropy as a reference-free indicator of distribution drift and entropy deviations to guide replay weighting.
3.1 Motivation and Empirical Validation
- KL decomposition separates reference-current cross-entropy from negative current-policy entropy; entropy captures uncertainty shifts associated with diversity collapse or high-entropy uniformity.The paper treats entropy as a sensitive indicator of representational degradation, not as a sufficient guarantee of distributional closeness.Source: §3.1 · pp. 3–4
- Output-entropy deviation was approximately 1.54× cross-entropy deviation, and early entropy deviation negatively correlated with final performance retention.The 1.54× comparison and dominant entropy fluctuations are reported for the SFT settings studied; the correlation study measured MMLU accuracy at step 150.Source: §3.1 · pp. 3–4
3.2 The ECHO Framework
- ECHO estimates prediction entropy on anchor batches and compares it with a pre-training setpoint calibrated once from the original model.The anchor-batch entropy uses logits already produced during the anchor-loss forward pass; entropy matching is a feedback signal rather than a guarantee of distributional closeness.Source: §3.2 · p. 4, §3.1 · p. 4
3.2.1 The Elastic Feedback Loop
- An EMA smooths entropy state, absolute deviation from Href drives replay weight, and saturation bounds that weight using fixed controller hyperparameters.The EMA filters high-frequency batch variance; the controller has no learnable parameters, and its gain was set by a one-time sensitivity sweep and kept fixed across experiments.Source: §3.2.1 · pp. 4–5
3.2.2 Optimization Objective
- The objective adds anchor loss to SFT loss with the feedback-driven replay weight, which relaxes toward baseline when entropy is stable and rises with drift.The stronger anchor-loss weight under larger entropy error is intended to restore stability while retaining plasticity when error is near zero.Source: §3.2.2 · p. 5
4 Experiments
ECHO is evaluated across code, math, and reasoning tasks, with results covering plasticity, stability, mixing-ratio robustness, scaling, and training dynamics. Across these evaluations, it balances downstream learning with capability retention and remains competitive with static mixing across ratios and larger models.
4 Experiments
- The experiments assess disturbance rejection and stability, downstream-task plasticity, and operational robustness across data-mixing ratios.They cover code generation, mathematical reasoning, and logical inference, using Alpaca-GPT4 as anchor data.Source: §4 · p. 5, §4.1 · p. 6
4.1 Experimental Setup
- ECHO uses AdamW, a global batch size of 32, and 40,000 samples for mixed-training setups; evaluations include MMLU and task-specific benchmarks.The main models are Qwen3-0.6B and Llama-3.2-3B, with additional evaluation on Qwen3-1.7B, Qwen3-8B, and Gemma2-9B.Source: §4.1 · p. 6
4.2 Main Results
- ECHO matches or surpasses Vanilla SFT on downstream tasks in several comparisons while balancing capability retention, placing on and sometimes extending the observed plasticity-stability frontier.On Llama-3.2-3B GSM8K, ECHO scores 59.34% versus Vanilla SFT’s 58.83%; on Qwen3-0.6B HumanEval pass@1, it scores 38.41% versus 31.10%. KL-SFT retains general capability more strongly but does not match ECHO’s specialized gains.Source: §4.2 · pp. 6–7, Table 2 · p. 7
4.3 Robustness to Data Mixing Ratios
- Across mixing ratios γ from 0.25 to 4.0, ECHO matches or outperforms Static Mixing across Code, Math, and Reasoning, and is never worse in 21 configurations.On GSM8K at γ = 2, ECHO scores 56.5% versus Static Mixing’s 55.3%; it retains an advantage at γ = 4.Source: §4.3 · p. 7, Figure 3 · p. 8
4.4 Generalization and Scaling Analysis
- On Qwen3-8B and Gemma2-9B, ECHO outperforms Static Mixing, with the authors reporting robustness across architectures and tokenizer designs.The evaluation uses a balanced 1:1:1:1 mixture of coding, mathematics, reasoning, and general anchor data; Table 3 reports single-run results.Source: §4.4 · p. 8, Table 3 · p. 8
4.5 Qualitative Analysis
- During training, ECHO maintains lower entropy on anchor data than Vanilla SFT, while its replay weight λt starts high, settles lower, and can rise again when later batches threaten stability.The authors interpret the lower entropy and persistent negative entropy difference as suppression of forgetting.Source: §4.5 · pp. 8–9, Figure 4 · p. 8
5 Hyperparameter Sensitivity Analysis
The analysis examines how basal anchoring weight λ and proportional gain Kp shape the stability–plasticity trade-off. It selects λ = 0.5 and Kp = 5.0 as the default balance.
5 Hyperparameter Sensitivity Analysis
- MMLU increases from 42.4 at λ = 0.3 to 44.2 at λ = 1.0, while HumanEval decreases from 46.0 to 42.4 over λ ∈[0.3, 1.5].MMLU is 43.3 at λ = 1.5, so it no longer improves beyond λ = 1.0.Source: §5 · p. 9
- Lower λ favors rapid adaptation but risks forgetting, whereas higher λ enforces rigidity; Kp acts as a threshold-based dynamic safety valve.At lower Kp, feedback is insufficient to counter distribution drift; Kp = 10.0 imposes stricter entropic constraints.Source: §5 · p. 9
- MMLU rises from 42.9 at Kp = 5 to 44.6 at Kp = 10.0, while stronger entropic constraints can hinder code-generation learning.The authors identify λ = 0.5 and Kp = 5.0 as the default configuration, balancing forgetting mitigation with downstream learning.Source: §5 · p. 9
6 Conclusion
ECHO regulates replay weights using stability signals and output entropy as a reference-free KL surrogate. Evaluations across model scales and task types report reduced forgetting while preserving general capabilities and improving downstream adaptation.
6 Conclusion
- Across Qwen3-0.6B to Gemma2-9B and math, code, and reasoning tasks, ECHO alleviates the plasticity–stability dilemma of static mixing.The reported outcome is preservation of general capabilities alongside improved downstream adaptation, without frozen reference models.Source: §6 · p. 9
- ECHO is a closed-loop framework that regulates replay weights through stability signals.It strengthens replay only when entropy deviates from the pre-trained equilibrium.Source: §6 · p. 9
- The framework uses output entropy as an efficient, reference-free KL surrogate by linking prediction entropy to distribution drift.Source: §6 · p. 9
Limitations
The paper identifies limits in the evaluated scope and controller design. Further work could test ECHO across broader training horizons, model scales, data distributions, and continual alignment settings.
Limitations
- The applicability of output entropy as a stability signal remains to be examined under longer training horizons, larger models, and more complex data distributions.Source: Limitations · pp. 9–10
- Experiments cover representative high-interference code, mathematical reasoning, and logical inference scenarios, leaving broader continual alignment settings for future study.Suggested settings include multilingual instruction tuning, safety preference optimization, and multimodal model updating.Source: Limitations · p. 10
- The implementation uses a simple proportional control law, while adaptive gains or richer control strategies remain unexplored options.Source: Limitations · p. 10
Impact Statement
ECHO is presented as a resource-efficient approach to continual fine-tuning that aims to preserve general capabilities during task adaptation. The impact statement connects this to easier model updates and long-term maintenance.
Impact Statement
- Using entropy feedback instead of a frozen reference model reduces the memory and computational overhead of preserving general capabilities during task adaptation.Source: Impact Statement · p. 10
- The lower overhead can help researchers and practitioners with limited hardware resources update models more conveniently.Source: Impact Statement · p. 10
- By mitigating catastrophic forgetting during adaptation, ECHO aims to reduce unintended degradation in general capabilities and support long-term model maintenance.Source: Impact Statement · p. 10