HY-WU (Part I): An Extensible Functional Neural Memory Framework and An Instantiation in Text-Guided Image Editing

Mengxuan Wu; Xuanlei Zhao; Ziqiao Wang; Ruicheng Feng; Zhangyang Wang; Kai Wang

Source-linked AI summary

HY-WU (Part I): An Extensible Functional Neural Memory Framework and An Instantiation in Text-Guided Image Editing

Mengxuan Wu, Xuanlei Zhao, Ziqiao Wang, Ruicheng Feng, Zhangyang Wang, Kai Wang

arXiv:2603.07236v2Mar 7, 2026cs.CV

TL;DR

Foundation models need adaptation mechanisms that handle evolving, heterogeneous objectives without repeatedly overwriting one shared parameter point. HY-WU implements functional memory by generating instance-conditioned operator updates on the fly. In text-guided image editing, it achieves the best overall WU-Eval score and supports conditional routing as the key mechanism, while long-horizon online continual learning and retrieval baselines remain unevaluated.

Problem
Static adaptation stores new behaviors in one shared parameter point, creating structural brittleness when deployment objectives are heterogeneous and evolving.
Method
HY-WU implements functional memory as a neural module that maps instance conditions to generated operator updates for conditional inference.
Results
HY-WU achieves the best overall WU-Eval performance with a score of 4.27, while alignment ablations identify conditional routing as decisive rather than parameter capacity alone.
Takeaways & Limitations
The results support adapting by learning a conditional family of parameter points instead of optimizing a single shared solution.
Takeaways & Limitations
Part I does not benchmark retrieval-based memory baselines for editing or evaluate long-horizon online continual-learning protocols.

Abstract

from arXiv · show

Foundation models are transitioning from offline predictors to deployed systems expected to operate over long time horizons. In real deployments, objectives are not fixed: domains drift, user preferences evolve, and new tasks appear after the model has shipped. This elevates continual learning and instant personalization from optional features to core architectural requirements. Yet most adaptation pipelines still follow a static weight paradigm: after training (or after any adaptation step), inference executes a single parameter vector regardless of user intent, domain, or instance-specific constraints. This treats the trained or adapted model as a single point in parameter space. In heterogeneous and continually evolving regimes, distinct objectives can induce separated feasible regions over parameters, forcing any single shared update into compromise, interference, or overspecialization. As a result, continual learning and personalization are often implemented as repeated overwriting of shared weights, risking degradation of previously learned behaviors. We propose HY-WU (Weight Unleashing), a memory-first adaptation framework that shifts adaptation pressure away from overwriting a single shared parameter point. HY-WU implements functional (operator-level) memory as a neural module: a generator that synthesizes weight updates on-the-fly from the instance condition, yielding instance-specific operators without test-time optimization.

1 Introduction

Foundation-model adaptation is constrained by evolving objectives and a static interface that stores behavior in one shared parameter point. HY-WU instead uses functional memory to generate instance-conditioned operator updates, with text-guided image editing as its stress-test instantiation.

Motivation: Continual learning and personalization require memory interfaces that store and apply new behaviors as objectives, domains, and user preferences evolve.The paper frames these requirements as architectural constraints for deployed foundation models.
Static adaptation: Static adaptation applies one shared update uniformly, making heterogeneous objectives brittle when their feasible parameter regions are separated.The paper identifies infeasible sharing and over-specialization as structural failure modes of this single-point constraint.
Functional memory: Functional memory represents adaptation as a neural module that maps instance conditions to operator updates, allowing multiple specialized operators to coexist.The generator synthesizes updates on demand rather than storing behavior only in a shared parameter point or retrieved context.
Stress test: Text-guided image editing provides a transformation-centric stress test because objectives can be directional, mutually exclusive, and strongly instance-dependent.Part I focuses on conditional LoRA generation for heterogeneous editing objectives rather than full online continual-learning protocols.
HY-WU: HY-WU synthesizes LoRA updates on-the-fly from hybrid conditions through a scalable conditional parameter-generation framework.Its training objective optimizes the generator directly from downstream loss, avoiding checkpoint reconstruction and large weight-target datasets.
Mechanism and scope: The paper analyzes condition–parameter alignment and reports that HY-WU gains arise from routing and structured generated updates rather than capacity alone.The broader scope is a memory-first adaptation interface, not a new image-editing backbone.

2 Instantiation (Part I): Text-Guided Image Editing as a Procedural Functional-Memory Stress Test

Part I instantiates HY-WU as functional memory for text-guided image editing, a stress test where objectives can conflict and optimal transformations depend on each input instance. The TI2I system uses conditional operator specialization to address these properties while mapping the framework to measurable diagnostics and explicit scope boundaries.

Mapping the Memory-First Framework to the Editing Instantiation: HY-WU implements the memory-first framework through functional operator memory and instance-conditioned operator synthesis rather than a uniformly applied adaptation.The Part I mapping connects abstract components to concrete TI2I modules and validation diagnostics.
Task Definition and Terminology: The editing task maps an input image I and instruction p to an edited image that follows the instruction, preserves irrelevant content, and maintains structural coherence.The implementation uses a diffusion-based conditional generator with LoRA as the structured operator interface.
Task Definition and Terminology: Personalized editing specializes the transformation rule for each instance, unlike T2I personalization, which expands persistent representational memory for concepts, identities, or styles.Identity consistency, style coherence, and subject-specific constraints require changing selected attributes while preserving core characteristics.
Why Editing is a Procedural-Memory Stress Test: TI2I editing provides a stress test because objectives can be directional, mutually exclusive, and strongly dependent on instance content.Examples include conflicting transformations such as restore versus age and blur versus deblur.
Mapping the Memory-First Framework to the Editing Instantiation: Part I does not benchmark retrieval-augmented editing or agent-style external memory, and LoRA serves as an interface rather than an inherent framework requirement.Later series parts are intended to expand the application domain and memory substrate.
Mapping the Memory-First Framework to the Editing Instantiation: The evaluation separates parameter count, modular isolation, and conditional routing by comparing static, isolated, and instance-conditioned adaptation baselines.Additional controls preserve parameter count while removing instance–update correspondence to test the role of conditional routing.

3 HY-WU Implementation for Text-Guided Image Editing

HY-WU synthesizes instance-specific LoRA operators from image-and-text conditions while keeping the foundation model frozen. Its implementation combines on-the-fly downstream-loss training, rank-anchored parameter tokenization, structured transformer attention, and end-to-end optimization without checkpoint reconstruction.

Pipeline: HY-WU extracts hybrid image-and-text conditions, transforms them into parameter tokens, detokenizes those tokens into LoRA adapters, and injects them into a frozen foundation model.The generator is trained end-to-end through the generated-adapter forward pass.
Parameter Tokenization: Rank-anchored tokenization reorganizes heterogeneous LoRA matrices into consistent r × d tokens while preserving layer, module, rank, and channel-segment structure.The inverse process detokenizes the generated tensor back into LoRA matrices for backbone injection.
Neural Network Transformer: The final LoRA B projection is zero-initialized to support stable training of the parameter generator.The architecture also uses cross-attention to inject extracted image and text conditions.
Neural Network Transformer: The Neural Network Transformer maps conditions to parameter tokens using factorized intra-layer and inter-layer attention instead of quadratic attention over the flattened sequence.Intra-layer attention captures dependencies within backbone blocks, while inter-layer attention captures relationships across depth.
Training: On-the-fly optimization trains the parameter generator directly with downstream editing loss, avoiding pre-collected adapter checkpoints and reconstruction losses.This removes the need to train, store, and load large-scale LoRA checkpoint datasets.

4 Model Performance

HY-WU demonstrates strong performance in human and automatic evaluations of text-guided image editing, outperforming leading open-source models and remaining competitive with strong closed-source systems. It also generalizes across architectures and benefits from increased generator and adapter capacity.

Public benchmarks: HY-WU ranks #1 on GEdit-Bench among open-source models and #2 on ImgEdit-Bench among public models.On GEdit-Bench, it leads semantic consistency and overall score in English and all three reported metrics in Chinese; on ImgEdit-Bench, it achieves 5 top-1 and 1 top-2 results across 9 sub-tasks.
Automatic evaluation: HY-WU achieves the best overall WU-Eval performance with an overall score of 4.27, including gains of +0.27 in consistency and +0.23 in structure over the strongest open baseline.It also obtains the highest scores on consistency, structure, and quality while remaining competitive with leading closed-source systems.
Architecture generalization: Integrating HY-WU improves overall WU-Eval scores by +0.10 and +0.04 across two backbone architectures, with the largest improvements in consistency and structure.Alignment remains comparable, and GSB win rates of 56.5% and 53.0% against the corresponding base models indicate perceptual improvements.
Scaling study: Increasing NNT size and generated LoRA capacity generally improves performance, reaching a 53.8% win rate for both 5B and 7B NNT configurations.The highest editing win rate occurs at LoRA rank=64, although the scaling results come from a small-scale experimental setting.
Qualitative results: Qualitative results show that HY-WU better preserves identity, pose, background, clothing geometry, and facial structure while transferring reference attributes.These gains appear across costume transfer, virtual try-on, and identity-preserving face replacement scenarios.

5 Analyses and Discussions

The analyses show that static adaptation compromises under conflicting editing objectives, while conditional parameter generation routes instances to specialized operators. Performance and geometry analyses indicate that correct instance–parameter alignment, rather than parameter count alone, drives the gains.

Conflict-controlled editing: Static adaptation compromises when mutually exclusive editing objectives share one operator, whereas Parameter Generation preserves clearer directional behavior across opposing tasks.Single LoRA over-specializes, Shared LoRA compromises, and PG maintains sharper restoration/deblurring and more appropriate aging/blurring outputs.
Gradient conflict: Gradient conflict is structured and pervasive: intra-task gradients have mean cosine similarity ≈0.56, while a representative inter-task pair has mean ≈−0.30.Aggregating opposed gradients into one static update provides mechanistic evidence for compromise solutions.
Ablation results: 56.5% win rate and 4.02 overall WU-Eval score make full PG the strongest ablation, with gains concentrated in Consistency, Structure, and Quality.Alignment remains comparable to static baselines, while the agreement between human preference and automatic metrics supports perceptual relevance.
Capacity versus conditionality: 51.5% vs 51.7% win rates and 3.99 vs 3.98 WU-Eval scores show that SFT and Shared LoRA perform nearly identically despite SFT’s larger capacity.PG improves structural and perceptual metrics beyond both static baselines, separating capacity from conditionality.
Routing alignment: 48.0% and 48.3% win rates show that Average PG and Shuffle PG collapse toward base-level performance when instance–parameter correspondence is removed.Because parameter count and compute remain unchanged, the ablation isolates conditional alignment as necessary for PG’s benefit.
Parameter-space geometry: Generated LoRA parameters form semantically organized, multi-modal regions in which coarse editing semantics and finer object or scene factors structure nearby sub-regions.The alignment appears in both global clusters and local neighborhoods, and proximity correlates with semantic proximity across vision and language.

6 Related Work

Related work frames foundation-model memory as a choice among parameter, activation, and retrieval substrates, while continual learning and personalization confront interference from heterogeneous objectives. HY-WU positions conditional operator routing as functional memory and trains generation directly from downstream loss rather than checkpoint reconstruction.

Memory interfaces: Foundation-model memory spans persistent parameter memory, volatile activation memory, and external retrieval memory, each trading off reuse, update cost, freshness, and context limits.Hybrid systems can combine retrieval evidence with parameter-efficient specialization.
Operator-level memory: Retrieval and activation memory add information while preserving a fixed operator, so they do not directly express differing transformation rules.The paper treats retrieval-based memory as complementary and does not benchmark it for image editing in Part I.
Continual learning and personalization: Continual-learning and personalization methods reduce interference with task- or user-specific modules, but can increase stored-module growth and still leave incompatible objectives to reconcile.HY-WU instead treats adaptation as conditional operator routing rather than repeated overwriting.
Capacity versus conditionality: Increasing trainable capacity up to full SFT still yields one shared inference operator, motivating a distinction between capacity and conditionality.Static updates may compromise under heterogeneity even when they are more expressive.
Positioning HY-WU: HY-WU optimizes its generator directly through downstream task loss, replacing checkpoint reconstruction with on-the-fly synthesis of instance-conditioned updates.This addresses the checkpoint-conditioned regime common in prior weight-generation approaches.

7 Open Roadmap: The HY-WU Series and a Memory-First Agenda

Part I establishes functional memory through conditional LoRA generation for transformation-centric image editing, while defining open evaluations for retrieval complementarity, online continual learning, scaling, broader operator interfaces, and deployment governance.

Scope of Part I: Part I demonstrates instance-conditioned operator routing for heterogeneous transformation objectives but does not benchmark retrieval-based editing baselines or long-horizon online continual learning.These remain explicit next steps for the HY-WU series.
Retrieval complementarity: The roadmap proposes comparing retrieval-only, retrieval plus static parameter memory, and retrieval plus HY-WU to test when content memory and operator memory are complementary.Task design separates facts or exemplars from transformation rules and procedural control.
Online continual learning: Online continual learning for HY-WU must preserve generator routing and operator synthesis while preventing the update family from collapsing toward a low-diversity compromise.Proposed update axes include scoped generators and hybrid consolidation.
Memory-first scaling: The proposed scaling agenda varies generator capacity, update bandwidth, token budget, and conditional diversity while holding the backbone fixed.The report introduces these architectural tools but does not yet present scaling curves.
Beyond LoRA: Future functional-memory interfaces include alternative PEFT parameterizations, modality-specific operator deltas, and hierarchical synthesis across coarse and fine behaviors.These extensions motivate questions about expressivity per bandwidth and controllability per compute.
Broader systems agenda: Longer-horizon applications include video identity consistency and agentic procedural skills, where functional memory can condition operator shifts while retrieval supplies factual or episodic context.Deployment also requires explicit memory scope, rollback, deletion, and privacy protections.

A Project Contributors

The project credits list project leads, core contributors, contributors, benchmarking assistance, and external contributors or advisors. The supplied materials also include contributor-affiliation context and example image-editing prompts.

Project leadership: Qinglin Lu and Kai Wang are identified as Project and Tech Leads.
Core contributors: Mengxuan Wu, Xuanlei Zhao, Ziqiao Wang, Ruicheng Feng, Atlas Wang1, and Kai Wang are listed as Core Contributors.
Contributors: Jiale Tao, Junshu Tang, Haoyu Yang, Zhentao Yu, Xiang Wen, Chunyu Wang, Shuai Shao, and Han Hu are listed as Contributors.
Data and benchmarking: Jihong Zhang, Qiuyong Xiao, Liyu Wang, Jiaxin Lin, Feng Lu, Lei Wang, Shiyu An, Shuo Jiang, and Zhichao Hu assisted with data and benchmarking.
External contributors and advisors: External contributors and advisors include researchers affiliated with NUS, UCSD, SYSU, NTU, Princeton, SZTU, UCF, the University of St. Gallen, THU, Oxford, and other institutions.Atlas Wang1’s footnote states that the contribution was independent and should not be interpreted as representing Tencent.