PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual Visual Learning under Evolving Semantic Concept Shift

Ismail Lamaakal, Chaymae Yahyati, Yassine Maleh, Khalid El Makkaoui, Ibrahim Ouahbi

arXiv:2608.23903v1cs.CVcs.CL

TL;DR

The paper studies evolving semantic concept shift, where visual evidence may stay similar while taxonomies, policies, or concept definitions change. SemReWrite combines semantic-change representation, sparse-supervision localization, structured memory, and selective low-rank rewriting; experiments report a stronger balance between revised-semantic learning and preservation of unaffected knowledge than competing adaptation strategies.

  • Problem

    Visual adaptation methods generally assume stable task semantics, but long-lived systems may require revised interpretations of the same visual evidence while selectively retaining valid knowledge.

  • Method

    SemReWrite combines old–revised semantic representations, sparse revised supervision for affected-region localization, structured semantic memory, input-dependent low-rank rewriting, preservation, and obsolete-decision suppression.

  • Results

    Across EvoShift-Bench settings, SemReWrite achieves a stronger balance between learning revised semantics and retaining unaffected knowledge than prompt replacement, fine-tuning, parameter-efficient adaptation, and continual-learning strategies.

  • Takeaways & Limitations

    Selective revision is evaluated through separate rewriting, preservation, obsolete-retention, and joint selective-revision measures across diverse semantic transitions and benchmarks.

  • Takeaways & Limitations

    The method cannot reliably recover revised distinctions that depend on information absent from the visual input without additional metadata or modalities.

Abstract

from arXiv · show

Visual foundation models are commonly adapted under the assumption that the appearance of incoming data may change while the semantic meaning of the prediction task remains fixed. In long-lived visual systems, however, taxonomies, policies, and concept definitions can themselves evolve, causing the same visual evidence to require a different interpretation. We study this setting as evolving semantic concept shift and introduce SemReWrite, a framework for selectively updating obsolete visual--semantic mappings while preserving knowledge that remains valid. SemReWrite represents changes between old and revised semantic specifications, combines semantic discrepancy with sparse revised supervision to localize affected visual regions, and uses an input-dependent low-rank rewriting mechanism together with structured semantic memory, preservation, and obsolete-decision suppression. We further introduce EvoShift-Bench, spanning ImageNet, iNaturalist, CUB-200-2011, and DomainNet, with semantic transitions including class split, merge, boundary revision, insertion, partial redefinition, recurrence, and mixed semantic--appearance shift. To explicitly evaluate selective semantic revision, we introduce Rewrite Accuracy (RA) and Preservation Accuracy (PA) for affected and unaffected regions, respectively, Obsolete Retention (OR) for measuring residual outdated semantic associations, and the Selective Revision Score (SRS), which jointly summarizes rewriting and preservation performance. Experiments show that SemReWrite achieves a stronger balance between learning revised semantics and retaining unaffected knowledge than prompt replacement, conventional fine-tuning, parameter-efficient adaptation, and continual-learning strategies.

I. INTRODUCTION

Evolving semantic concept shift occurs when visual evidence remains similar but the semantic rules assigning meaning to it change. SEMREWRITE addresses this by selectively rewriting obsolete mappings while preserving knowledge that remains valid.

  • Motivation: Evolving semantic concept shift changes task semantics over time, unlike conventional adaptation, which mainly addresses changed visual distributions under fixed meanings.The same visual evidence may require a different interpretation after a taxonomy, policy, or concept-definition revision.
  • Motivation: Conventional fine-tuning may damage valid knowledge, continual learning may preserve invalid knowledge, and prompt-only adaptation may not identify the visual boundary that must change.The desired behavior is selective modification rather than preserving or rewriting everything.
  • Motivation: Unlabeled visual streams cannot reliably reveal semantic revisions when the observable input distribution remains unchanged.This motivates combining explicit semantic side information with sparse supervision under the revised specification.
  • Contributions: SEMREWRITE represents semantic changes, localizes affected visual regions, rewrites them with input-dependent low-rank adaptation, preserves valid knowledge, and suppresses obsolete mappings.The framework also uses structured semantic memory to represent evolving concept relationships.
  • Contributions: EVOSHIFT-BENCH covers splits, merges, boundary revisions, insertions, partial redefinitions, recurring changes, and mixed semantic–appearance shifts with separate rewriting, preservation, and obsolete-retention measures.These evaluations target selective revision across multiple forms of semantic evolution.

B. Identifiability of Semantic Concept Shift

Semantic concept shift may be statistically unidentifiable from unlabeled images when visual marginals stay fixed but labeling mechanisms change. SEMREWRITE therefore combines revised semantic specifications with sparse revised examples to determine what changed and where it changed.

  • Identifiability: Semantic revisions can remain invisible when the same images are governed by a revised labeling rule.Unlike appearance changes, semantic changes need not alter the marginal visual distribution.
  • Identifiability: If two environments share the same visual marginal but differ in conditional labeling, unlabeled observations have identical likelihood under both environments.The observable sample distribution is therefore the same despite different semantic interpretations.
  • Identifiability: Under equal priors, no decision rule using only unlabeled visual observations can identify the active labeling mechanism with error below 1/2.The proposition follows because the total-variation distance between observable distributions is zero.
  • Methodological consequence: SEMREWRITE uses the revised specification to communicate what concepts mean and sparse revised examples to indicate where distinctions occur in visual representation space.This separates semantic-change description from visual localization.
  • Semantic representation: The method encodes old and revised specifications in a frozen vision-language space, representing persistent, inserted, and deleted concepts through semantic transition vectors.For persistent concepts, cosine distance quantifies semantic movement but does not alone determine the affected visual region.

D. Affected Concept Localization

SEMREWRITE localizes which inputs require semantic revision by combining specification-level discrepancy with sparse visual supervision, then uses the resulting gate to control selective rewriting. A structured semantic graph and compact low-rank updates preserve semantic structure while limiting changes.

  • Affected Concept Localization: The affected-concept localizer estimates how strongly each input should be modified and uses that score to control selective rewriting.It combines semantic and visual evidence rather than relying on specification discrepancy alone.
  • Affected Concept Localization: The semantic discrepancy compares old and revised predictive distributions with normalized Jensen–Shannon divergence, producing a bounded signal in [0, 1].The measure remains finite when categories are added or removed.
  • Affected Concept Localization: Sparse revised supervision trains a sample-level localizer because semantic discrepancy alone can overestimate regions whose correct labels remain unchanged.The localizer uses visual representations to refine the estimated affected region.
  • Structured Semantic Memory: A typed semantic graph represents classes, attributes, parts, and hierarchical relations so splits, merges, insertions, and redefinitions update concept structure coherently.Adapted class prototypes remain anchored to revised textual definitions while graph relations are preserved.
  • Selective Rewriting: Frozen visual and text encoders plus a rank-r residual UV^⊤ provide compact parameter-efficient rewriting, modulated by a(x) so stable inputs remain close to their original representations.The trainable parameters are the low-rank rewrite, semantic residuals, and localization head.

1) Learning the Revised Decision Rule

SEMREWRITE separates learning the revised rule from removing obsolete decisions and preserving valid ones. Its objective combines revised supervision, selective preservation, obsolete-decision suppression, semantic consistency, gate supervision, and residual regularization.

  • Learning the Revised Decision Rule: Revised cross-entropy supervision moves affected decision regions toward the new semantic rule using few labeled examples.The examples connect the revised semantic definitions to actual visual observations.
  • Learning the Revised Decision Rule: Preservation distillation is restricted to unchanged concepts and weighted by unaffectedness, avoiding explicit constraints toward obsolete decisions.Inputs with a(x) near 1 can change more freely, while inputs with a(x) near 0 remain close to the previous model on valid concepts.
  • Learning the Revised Decision Rule: Obsolete-decision suppression is applied when an affected sample retains a conflicting old prediction that still exists in the revised label space.Removed classes are suppressed structurally because they no longer exist in the revised classifier.
  • Learning the Revised Decision Rule: Residual regularization constrains low-rank visual updates and keeps semantic prototypes near their text-derived initialization.This limits unnecessary modification under sparse revised supervision.
  • Learning the Revised Decision Rule: The complete objective jointly learns revised predictions, erases conflicting old relations, protects valid relations, constrains semantic geometry, trains localization, and limits adaptation complexity.These complementary terms implement the principle of preserving what remains true and rewriting what has become false.

G. Measuring Selective Semantic Revision

The paper evaluates selective semantic revision by separating performance on affected and unaffected regions, then tests the framework across benchmark families, shift types, backbones, and supervision budgets.

  • Rewrite Accuracy (RA) measures correct predictions under revised semantics for affected examples, while Preservation Accuracy (PA) measures retained validity on unaffected examples.
  • Selective Revision Score (SRS) summarizes rewriting and preservation using a harmonic mean, penalizing strongly imbalanced RA and PA.
  • Benchmark construction: EvoShift-Bench spans ImageNet, iNaturalist, CUB-200-2011, and DomainNet to represent hierarchical, biological, fine-grained, cross-domain, and mixed semantic transitions.
  • Semantic shift protocols: The benchmark includes split, merge, boundary revision, insertion, partial redefinition, recurrence, and mixed semantic–appearance shifts.
  • Experimental setup: Experiments use CLIP ViT-B/16 primarily, additionally test SigLIP and DINOv2, and evaluate revised-supervision budgets from 1 to 16 examples per changed concept.

C. Comparison Methods

The comparison evaluates SemReWrite against prompt, fine-tuning, parameter-efficient, test-time, and continual-learning methods across semantic revisions and longer adaptation sequences. Results emphasize selective rewriting: SemReWrite improves changed concepts while preserving stable concepts and suppressing obsolete decisions.

  • Comparison methods: Baselines include unchanged and text-only models, fine-tuning and probing methods, parameter-efficient adaptation, prompt methods, and test-time adaptation approaches.
  • Main results: Across five semantic shift types, SemReWrite achieves the strongest performance, including boundary revision and partial redefinition where the label space need not expand.
  • Main results: Full fine-tuning improves revised-task accuracy but weakens SRS by changing semantically valid knowledge, whereas LoRA improves the revision–preservation trade-off.
  • Selective Forgetting: SemReWrite combines the highest RA with PA close to the unchanged model and substantially lower OR than methods that retain obsolete decisions.
  • Annotation Efficiency: SemReWrite achieves strong one-shot SRS and improves with additional revised examples, indicating that semantic specifications help when visual supervision is limited.
  • Annotation Efficiency: SemReWrite raises RA from 79.2% to 92.6% while PA remains above 92%, whereas Full FT’s PA falls from 84.8% to 74.2% as supervision increases.
  • Continual Semantic Evolution: After five sequential revisions, PA remains above 92% and OR below 10%, while recurrence evaluation measures the recovery gap after returning from S2 to S1.

6) Discussion

The discussion distinguishes learning revised semantics from preserving valid knowledge and suppressing obsolete decisions. Ablations and sensitivity analyses show that selective localization, preservation, erasure, semantic-change representation, and low-rank rewriting jointly support this trade-off, while several failure conditions bound the method’s applicability.

  • Discussion: SEMREWRITE jointly optimizes revised-rule learning, unaffected-knowledge preservation, and obsolete-mapping removal, reflected by high RA and PA with low OR.The experiments contrast full fine-tuning, continual learning, and prompt-only adaptation with the intended selective-revision behavior.
  • Ablation studies: Removing affected-region localization keeps RA relatively high but substantially lowers PA, showing that localization determines where revisions are applied.Without spatial selectivity, rewriting approaches ordinary global adaptation.
  • Ablation studies: Removing preservation maximizes RA but sharply reduces PA, whereas removing erasure increases OR and lowers RA, confirming complementary roles.Preservation protects valid knowledge; erasure suppresses outdated semantic decisions.
  • Ablation studies: Replacing the semantic concept graph or explicit semantic delta with independent or revised-text-only representations reduces adaptation quality, supporting structured semantic-change modeling.The explicit semantic delta produces the larger degradation when removed.
  • Ablation studies: Full-backbone rewriting slightly increases RA but considerably lowers PA, so low-rank rewriting offers a better revision–preservation trade-off with fewer modified parameters.The discussion reports additional parameter, memory, and adaptation-time details in the supplementary material.
  • Failure modes and limitations: SEMREWRITE is most useful when semantic evolution affects only part of the learned mapping and revised concepts are visually grounded and sufficiently specified.Near-global revisions favor conventional retraining, while ambiguous, visually unobservable, subtle, or contradictory revisions create failure conditions.

APPENDIX A EXTENDED RESULTS AND ADDITIONAL ANALYSIS

Extended experiments show that SemReWrite’s advantage persists across datasets and representation families, while localization, semantic supervision, and selective rewriting govern robustness, capacity, and long-horizon behavior.

  • Per-Dataset Results: SemReWrite maintains the strongest SRS across ImageNet, iNaturalist, CUB-200-2011, and DomainNet.The advantage persists on taxonomy-driven and attribute-driven benchmarks, including mixed semantic–appearance shift in DomainNet.
  • Foundation-Model Backbones: The ranking between adaptation strategies remains stable across CLIP, SigLIP, and DINOv2 despite moderately lower performance with DINOv2.The result indicates that the benefit is not solely tied to CLIP’s jointly pretrained text space.
  • Affected-Region Localization: More accurate affected-region localization is associated with stronger selective revision and remains informative even with one revised example.Localization improves as revised supervision increases, while AUROC remains well above chance in the one-shot setting.
  • Semantic-Shift Severity: Performance degrades more slowly for SemReWrite as semantic-shift severity increases, although its advantage narrows when nearly the entire input space is revised.Selective localization has its largest relative advantage at low and moderate severity, where substantial knowledge remains valid.
  • Robustness: SemReWrite remains robust under moderate prompt variation and label noise, but ambiguous specifications and high noise reduce revision quality.Ambiguous specifications substantially reduce RA and SRS; high noise contradicts the semantic description and deteriorates localization.
  • Capacity and Long-Horizon Behavior: The complete configuration jointly preserves high RA and PA while keeping OR low, and its performance declines more slowly across repeated semantic revisions.Beyond adapter rank r = 8, gains become marginal while PA decreases; selective rewriting also reduces interference across repeated updates.
  • Summary: Across extended analyses, selective rewriting preserves reusable knowledge while suppressing semantic decisions that have become obsolete.The authors frame effective semantic evolution as preserving valid associations and deliberately rewriting invalid ones.

APPENDIX B COMPLETE THEORETICAL ANALYSIS

The theoretical analysis establishes that unlabeled visual observations cannot identify a semantic revision when visual marginals are unchanged, and formalizes revised risk as a weighted combination of rewriting and preservation performance.

  • Unlabeled Non-Identifiability: When two environments share the same visual marginal but differ in conditional labeling, no unlabeled-sample decision rule can identify the active semantic mechanism.Under equal priors, the minimum achievable error is 1/2.
  • Unlabeled Non-Identifiability: The non-identifiability result requires equal priors only for the numerical error value, not for the absence of information in the observations.With unequal priors, the optimal strategy selects the more probable hypothesis.
  • Risk Decomposition: Revised risk decomposes into affected-region error weighted by πA and unaffected-region error weighted by πU.Rewrite Accuracy governs changed semantics, while Preservation Accuracy governs inputs whose previous interpretation remains valid.
  • Risk Decomposition: Aggregate accuracy can remain high while affected-region performance is poor when the affected-region mass πA is small.This is why aggregate accuracy alone is insufficient for evaluating semantic revision.
  • Risk Decomposition: When πU is small, selective preservation becomes less important and global adaptation can become competitive with selective rewriting.This is the near-global-shift failure mode.

C. Selective Rewriting and Localization Error

The localization analysis separates rewriting from preservation by routing affected inputs to a rewriting predictor and unaffected inputs to a preservation predictor, then quantifies the cost of routing errors.

  • Selective Routing: SEMREWRITE uses an affected-region localizer to choose between rewriting and preserving the previous representation.The rewriting residual is activated for affected inputs and suppressed for stable inputs.
  • Localization Error: Localization error consists of false negatives that fail to rewrite changed examples and false positives that unnecessarily rewrite stable examples.These two errors formalize the competing risks of under-revision and interference.
  • Localization Error: The hard-routing localization error is πA P(bg = 0 | X ∈ At) + πU P(bg = 1 | X ∈ Ut).The first term captures missed rewriting on affected inputs, and the second captures unnecessary rewriting on unaffected inputs.
  • Soft Routing: A soft router assigns probability a(x) to rewriting and 1 − a(x) to preservation, yielding a corresponding decomposition of soft localization error.The two terms measure insufficient rewriting on affected examples and unnecessary rewriting on stable examples.
  • Soft Routing: The soft selective-routing theorem bounds routed risk using rewriting risk, preservation risk, and localization error.This establishes that incorrect activation can limit performance even when rewriting and preservation mechanisms are individually effective.

3) Bound for a Deterministic Gated Residual

The deterministic gated-rewrite analysis shows that continuous residual scaling inherits a localization-dependent risk bound under a Lipschitz assumption, while the Selective Revision Score has basic boundedness and monotonicity properties.

  • Bound for a Deterministic Gated Residual: The deterministic gated-rewrite theorem assumes conditional risk is Lipschitz in the continuous rewriting gate.This assumption supports a bound for continuously scaling the residual transformation and is unnecessary for hard routing.
  • Bound for a Deterministic Gated Residual: The resulting risk bound separates rewrite error, preservation error, and localization error.The probability-weighted form is tighter and matches the probability weighting in global risk.
  • Bound for a Deterministic Gated Residual: When the learned gate approaches the oracle gate, the gated model approaches the performance of the oracle selective router.The oracle activates rewriting on affected regions and preservation on unaffected regions.
  • Properties of SRS: SRS is bounded above by the arithmetic mean of Rewrite Accuracy and Preservation Accuracy and is non-decreasing in either argument.These properties formalize its sensitivity to both revision and preservation performance.
  • Properties of SRS: SRS equals zero exactly when either Rewrite Accuracy or Preservation Accuracy is zero, and equals one exactly when both equal one.The score is symmetric and equals either argument when the two arguments are equal.

1) Penalty for Imbalanced Revision and Preservation

SRS penalizes imbalance between rewriting accuracy and preservation accuracy, even when their arithmetic mean is fixed. It therefore favors balanced semantic adaptation rather than allowing one dimension to compensate for failure in the other.

  • For a fixed arithmetic mean, SRS decreases as the discrepancy between RA and PA increases.
  • SRS explicitly prefers balanced semantic adaptation.
  • The SRS bound implies that achieving SRS ≥ s requires min{RA, PA} ≥ s/(2−s).
  • An SRS of at least 0.90 therefore cannot coexist with extremely poor rewriting or preservation performance.

APPENDIX C COMPLETE IMPLEMENTATION DETAILS

The implementation fixes pretrained visual-language or vision-only backbones, preprocessing, semantic prompting, graph memory, localization, and low-rank rewriting settings across semantic-transition experiments. Adaptation uses frozen encoders, normalized representations, structured semantic memory, and continuous affected-region weighting.

  • Backbones and preprocessing: SEMREWRITE uses frozen CLIP ViT-B/16 and SigLIP Base/16 encoders, with standard 224 × 224 preprocessing.The vision-only evaluation uses frozen DINOv2 ViT-B/14, projected into the frozen CLIP text-embedding space.
  • Backbones and preprocessing: Visual and textual embeddings are normalized before similarity or compatibility computation.DINOv2 representations are projected into CLIP semantic space before normalization.
  • Semantic prompting: Prompts use one deterministic construction rule per benchmark family, with old and revised specifications sharing the same linguistic template.CUB boundary-revision prompts directly encode the changed semantic rule.
  • Structured semantic memory: Semantic memory is a typed graph containing semantic nodes, relations, embeddings, and class residuals, but no raw image exemplars.Graph contents vary by benchmark, including taxonomic relations, visual attributes, and semantic-parent nodes.
  • Localization and rewriting: The affected-region localizer is a two-layer MLP with 256 hidden units, GELU, dropout probability 0.1, and one sigmoid output.The default rewriting mechanism uses the resulting probability continuously rather than thresholding it.
  • Localization and rewriting: The visual adapter uses trainable low-rank matrices initialized so the adapted model initially reproduces the frozen visual representation.Rank 8 is used in all main experiments, while other ranks are evaluated only in supplementary sensitivity analysis.

I. Loss Coefficients

The main experiments use fixed optimization, data-stream, seed, and evaluation protocols, with coefficients that separately weight preservation, obsolete-decision suppression, graph consistency, localization supervision, and residual regularization.

  • Optimization: AdamW optimizes all trainable SemReWrite parameters with gradient clipping and cosine-decayed learning rates over a maximum of 20 epochs.Main experiments train all methods for the complete 20 epochs without evaluation-label early stopping.
  • Loss coefficients: The default objective coefficients are λp = 1.0, λe = 0.5, λs = 0.1, λg = 0.5, and λr = 10−4.These coefficients control preservation, obsolete-decision suppression, semantic-graph consistency, affected-region supervision, and residual regularization, respectively.
  • Loss coefficients: The erasure objective weights affected examples by continuous a(xi), avoiding an additional training threshold for localization.
  • Data protocol: Each optimization step combines revised labeled examples for new-label, gate, and erasure losses with unlabeled examples for preservation.The default step uses 64 revised and 64 unlabeled samples when possible, and unlabeled sampling never uses affectedness labels.
  • Fair comparison: Supervised and continual-learning baselines receive the same revised few-shot budget as SemReWrite, and no baseline receives affected-region annotations.Hyperparameters are selected on held-out training transitions and fixed across datasets and shift types.
  • Reproducibility: Main experiments repeat adaptation across three random seeds while fixing benchmark definitions generated with seed 2026.Only adaptation and few-shot selection vary across experimental seeds.
Loading 2608.23903v1…