PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal Activation Verbalizer: A Unified Framework for Cross-Model Activation Explanation

Haiyan Zhao, Zirui He, Guanchu Wang, Ali Payani, Yingcong Li, Mengnan Du

arXiv:2605.25903v2cs.CLcs.LG

TL;DR

Existing activation verbalization methods largely explain only their own models’ activations, limiting cross-model readout of hidden representations. UAV uses a shared decoder and donor-specific adapters, including adapter-only transfer, and remains competitive with self-explanation baselines across multiple tasks while exposing distinct roles for decoder tuning and activation alignment.

  • Problem

    Existing activation verbalization methods are mostly limited to self-explanation, while activation-grounded natural-language readouts are needed across heterogeneous donor models.

  • Method

    UAV maps heterogeneous donor activations into soft tokens for a shared decoder and supports adapter-only transfer by reusing frozen decoder-side LoRA.

  • Results

    UAV remains competitive with strong self-explanation baselines across classification, fact retrieval, and gist summarization while enabling cross-model verbalization.

  • Takeaways & Limitations

    Decoder-side tuning mainly improves task behavior, whereas the adapter provides activation-grounded factual and semantic information for explanations.

  • Takeaways & Limitations

    UAV remains limited on fine-grained fact retrieval and abstract gist recovery, focuses on short-context inputs, requires donor- and layer-specific adapters, and relies on LLM judges.

Abstract

from arXiv · show

Activation verbalization explains hidden representations in natural language, but existing methods are mostly limited to self-explanation, where each model explains only its own activations. We introduce Universal Activation Verbalizer (UAV), a framework that uses a shared decoder to explain activations from heterogeneous donor models. UAV learns a lightweight adapter that converts donor activations into soft tokens in decoder's embedding space, and further supports adapter-only transfer by reusing a frozen decoder-side LoRA while training only a new adapter for another donor. Across classification, fact retrieval, and gist summarization, UAV remains competitive with strong self-explanation baselines while enabling cross-model verbalization across model families and scales. Ablations show that decoder-side tuning mainly improves task behavior, whereas the adapter provides the activation-grounded factual and semantic information needed for faithful explanations. Code and data are available at https://github.com/hy-zhao23/ActExp.

1 Introduction

Activation verbalization offers a human-readable readout of information in hidden states, but existing training-based methods largely explain only their own models’ activations. UAV addresses this limitation with a shared decoder for heterogeneous donors and remains competitive across several activation-grounded tasks.

  • Activation verbalization describes information contained in hidden activations in natural language rather than predicting predefined labels or discovering individual features.The verbalizer receives the question and activation-derived soft tokens without the source text.
  • UAV uses a shared decoder to verbalize activations from heterogeneous donor models, enabling cross-model explanation across model families and scales.The framework combines a donor-specific adapter with decoder-side adaptation.
  • UAV introduces adapter-only transfer by freezing a decoder-side LoRA learned from one donor and training only a new adapter for another donor.This reuses decoder-side instruction-following behavior while adapting to a new activation space.
  • Across classification, fact retrieval, and gist summarization, UAV remains competitive with strong self-explanation baselines while enabling cross-model verbalization.Larger decoders generally improve performance, and the evaluation spans Llama, Gemma, Yi, and Qwen-family donors.
  • The ablations separate roles: decoder-side tuning improves task behavior, whereas the adapter recovers activation-grounded factual and semantic information.This distinction motivates treating the adapter as the component carrying donor-specific information.

2 Related Work

Prior activation-interpretability methods include bottom-up analyses of decodable features and top-down methods that directly verbalize activations. UAV builds on lightweight representation-alignment methods to address the cross-model gap in top-down verbalization.

  • Bottom-up methods probe or decompose frozen activations to identify linguistic, syntactic, semantic, or interpretable feature information.Probing trains lightweight classifiers, while sparse autoencoders study feature structure.
  • Shared-decoder verbalization should be interpreted as a representation readout rather than a causal or mechanistic explanation.
  • Top-down methods directly verbalize target activations, using prompt patching or activation injection with training-free or fine-tuned language models.These approaches include Patchscopes, SelfIE, LatentQA, and AO.
  • Existing top-down approaches are described as unable to perform cross-model activation verbalization.Their limitation motivates a shared external decoder with donor-specific alignment.
  • UAV adapts MLP projectors, Q-Formers, and adapter transfer ideas to align heterogeneous donor activations with a shared decoder.The adapter bridges donor representations and decoder-readable embeddings, while adapter-only transfer reuses decoder-side LoRA.

3 UAV Framework

UAV maps donor activations into decoder-readable soft tokens and trains them in two stages: activation-to-text alignment followed by explanation-oriented instruction tuning. It supports both full donor-specific adaptation and adapter-only transfer with a shared decoder.

  • 3.1 Problem Statement: UAV defines activations as layer- and position-specific hidden states that summarize the preceding input tokens and support questions about input knowledge or gist.The donor model and verbalizer may be different models.
  • 3.2 Architecture Design: The framework first aligns donor activations with the decoder embedding space, then jointly trains the adapter and decoder-side parameters for instruction-following explanations.The second stage targets both fine-grained attributes and topic-level gist questions.
  • 3.2 Architecture Design: The first stage freezes the decoder and trains an adapter to reconstruct the input prefix from a donor activation, using either an MLP or Q-Former-style attention adapter.Adapters are trained separately for activations from each donor layer.
  • 3.2 Architecture Design: UAV converts each donor activation into n continuous soft tokens that are prepended to the prompt and injected at the decoder embedding layer.These activation-conditioned tokens guide the decoder to reconstruct input prefixes and answer activation-grounded questions.
  • 3.3 Two Training Strategies: Full adaptation trains a donor-specific adapter and decoder-side LoRA for each donor, whereas adapter-only transfer freezes the decoder and LoRA and trains only a new donor adapter.The transfer strategy tests whether unseen donor activations can be aligned to an already adapted decoder space.

4 Experiments

UAV is evaluated across open-ended classification, fact retrieval, and gist summarization, where it remains competitive with self-explanation while supporting cross-model verbalization. Results show that decoder tuning improves task behavior, whereas activation adapters provide input-specific factual and semantic information.

  • Comparison with Activation Verbalization Baselines: UAV’s full two-stage adaptation achieves the strongest overall ROUGE-L and BERTScore among self-explaining methods, outperforming AO and LatentQA, with especially clear gains on fact retrieval.Training-based self-explanation baselines outperform training-free patching methods, whose fact-retrieval BERTScore is near zero or negative.
  • Adapter-Only Transfer: Adapter-only transfer remains close to full adaptation, and for Qwen3-4B it slightly outperforms full self-explanation on overall metrics.For Llama-3.1-8B, adapter-only transfer incurs only a moderate drop compared with full adaptation while reusing a frozen decoder-side LoRA.
  • Shared Decoder Across Donor Models: Cross-donor verbalization remains consistent across Llama, Gemma, and Yi families with Qwen3-4B as a shared decoder, while Llama-3.1-8B and Yi-1.5-34B outperform Gemma donors.These comparisons indicate that performance depends on donor-specific alignment factors as well as donor scale.
  • Ablations: Decoder-side LoRA alone improves comprehension-style behavior but remains much weaker on fact retrieval and gist summarization, showing that the adapter supplies input-specific information.The base decoder and randomly initialized adapter perform poorly; Stage-1 alignment helps modestly, while joint training preserves original-task performance better than sequential fine-tuning.

5 Conclusions and Future Work

UAV provides a shared-decoder framework for explaining heterogeneous donor activations across model families and scales. Its ablations distinguish decoder-side task behavior from adapter-provided activation-grounded information.

  • UAV enables cross-model activation verbalization with a shared decoder across model families and scales while remaining competitive with self-explanation methods.
  • Decoder-side tuning improves task behavior, whereas the adapter provides activation-grounded factual and semantic information.
  • Future work targets stronger activation alignment, multi-layer features, and richer factual or summarization supervision for fact retrieval and gist understanding.

Limitations

UAV remains constrained when recovering fine-grained attributes or abstract semantic content from a single activation, and its current setting is limited in several practical ways.

  • UAV remains limited in fact retrieval and gist summarization, especially for fine-grained attributes or abstract semantic content from a single activation.
  • The current setting focuses on short-context inputs, leaving long-context and document-level activation explanation for future work.
  • Adapter-only transfer still requires a donor- and layer-specific adapter, which may be costly for large-scale layer-wise analysis.
  • Source-grounded evaluation relies on LLM judges rather than human annotators.

A Additional Method Details

The method trains an adapter to convert donor activations into decoder-readable soft tokens, then optimizes activation reconstruction and explanation-oriented answer generation with teacher forcing. It studies MLP and Q-Former-style adapter designs.

  • A.1 Teaching Forcing: The adapter maps a donor activation into decoder-readable soft tokens used as the verbalizer’s conditioning representation.
  • A.1 Teaching Forcing: Stage 1 reconstructs the input prefix from the activation, freezing decoder-side parameters while optimizing only the adapter.
  • A.1 Teaching Forcing: Stage 2 prepends soft tokens to a textual prompt and trains autoregressive answer prediction under teacher forcing.
  • A.2 Adapter Designs: The MLP adapter applies layer normalization and projection, with an optional bottleneck before producing output soft tokens.
  • A.2 Adapter Designs: The Q-Former-style adapter projects one activation into latent context slots, then uses learnable queries and cross-attention to produce activation-conditioned soft tokens.
  • A.2 Adapter Designs: The Q-Former design keeps output-token count mainly in the learnable query matrix while leaving context projection and attention parameters unchanged.

B.1 Data Sourcing

The training corpus combines heterogeneous factual, semantic, classification, and user-intent sources under a controlled short-context budget. Offline QA generation supplies factual and comprehension-oriented supervision for activation alignment and explanation training.

  • Data Curation: The corpus covers factual knowledge, semantic comprehension, classification attributes, and natural user intents, with most sequences bounded to 64 tokens.
  • Raw Text Sources: Raw text comes from 17 sources spanning Wikipedia subtypes, scientific and classification datasets, and natural user requests.
  • Length Normalization: Inputs are normalized with the Qwen3-4B tokenizer, truncated to sentence or paragraph prefixes when needed, and filtered to satisfy the short-context constraints.
  • Data Curation: All 17 sources support Stage 1 reconstruction, while 15 sources with synthesized QA pairs also support Stage 2 generative QA finetuning.
  • QA Generation: Wikipedia examples use factual QA with subtype-specific attribute hints and anti-leak constraints, while other sources use comprehension QA and gist supervision.
  • QA Generation: Gist supervision uses one-sentence paraphrases paired with varied semantically equivalent prompts to reduce dependence on a fixed question template.

C.1 Source-Grounded LLM-as-a-Judge Evaluation

Source-grounded judging shows that UAV recovers coarse topic information more reliably than fine-grained factual details, while paired comparisons find self- and cross-decoding largely comparable overall.

  • Source-grounded evaluation: 64–70% of fact-retrieval predictions reach the topic threshold, but only 24–30% reach the detail threshold; gist reaches just 8–16% and 4–6%, respectively.Topic and detail scores use a threshold of 3 on a 0–5 scale.
  • Qualitative failure analysis: Qualitative analysis shows that fluent predictions can receive partial topic credit while changing the source’s actor, action, or event.The example remains related to security forces and Lebanon but is factually incorrect at the detail level.
  • Statistical comparison: The confidence intervals for UAV versus AO and LatentQA exclude zero, but improvements over LatentQA are small and cross-model differences vary by donor and metric.For Gemma-3-4B, both intervals include zero; for Llama-3.1-8B, ROUGE-L favors self-decoding slightly while BERTScore is approximately tied.
  • Efficiency and stability: UAV reduces per-epoch training cost from 7.1 to 2.7 GPU-hours in Stage 1 and from 26.5 to 11.6 in Stage 2, while worsening best validation loss by 0.059.Caching donor activations avoids storing the donor model or backpropagating through it during verbalizer training.
  • Adapter architecture: QF-LoRA achieves the best Tok-F1 and chrF scores with fewer adapter parameters than the largest MLP variants, motivating the default architecture.The results suggest that decoder-side adaptation is important for exploiting attention-based adapter representations.

D.4 Dataset-Level and Error Analysis

Dataset-level results favor concrete entity-centered retrieval, whereas abstract, affective, and general-language tasks are weaker or mixed. Error analysis shows that fluent generation can still be irrelevant or factually imprecise.

  • Dataset-level performance: BERTScore reaches 0.475 on wikipedia_person and 0.450 on wikipedia_place, but only 0.282 on wikipedia_generic.Sentiment and emotion datasets range from 0.274 to 0.319, and topic and general-language results are mixed.
  • Dataset-level performance: Performance depends on the specificity and semantic structure of the target information, with concrete entity retrieval outperforming more abstract or affective tasks.The proposed explanation involving pretraining consistency is explicitly interpretive rather than directly established.
  • Error analysis: Weak activation alignment can produce fluent but input-irrelevant content, while incomplete fine-grained information can cause confusion between semantically similar entities.These failure modes show that generation fluency does not necessarily imply faithful recovery from the donor activation.
  • Complementary metrics: UAV achieves the strongest overall Token-F1 and chrF for Qwen3-4B activations under both full adaptation and adapter-only transfer.Adapter-only transfer from Llama slightly improves over the corresponding full self-explanation setting on both metrics.
  • Complementary metrics: For Llama-3.1-8B activations, the full self-verbalizer performs best among UAV variants, while an external Qwen3-4B decoder remains competitive and adapter-only transfer substantially beats training-free baselines.SelfIE and PatchScope perform much worse than trained verbalizers, especially on fact retrieval.

D.6 Additional Results on Cross Model Decoding

Cross-donor results remain stable across semantic, lexical, and character-level metrics, with donor scale and family-specific representation structure both influencing decoding quality.

  • Cross-donor results: Yi-1.5-34B achieves the best Token-F1 and chrF among donor models, while Gemma-3-12B improves over Gemma-3-4B.The shared decoder is Qwen3-4B-Instruct-2507.
  • Cross-donor results: Llama-3.1-8B remains stronger than both Gemma models despite being smaller than Gemma-3-12B, showing that scale alone does not determine decoding quality.The comparison indicates an additional role for donor-family representation structure.
  • Cross-donor results: UAV adapts the same Qwen3-4B decoder to multiple donor families while preserving cross-donor behavior across semantic, lexical, and character-level evaluation.Remaining variation reflects both donor scale and family-specific representation structure.

D.7 Task-level comparison between self-decoding and external-decoder verbalization

Self-decoding is generally strongest, especially for fact retrieval, but cross-model verbalization remains competitive for coarse and semantic tasks while enabling a shared decoder across donors.

  • Task-level comparison: Self-decoding usually outperforms cross-model verbalization, with the most consistent gap appearing on fact retrieval.Classification and gist summarization show more mixed comparisons, and some cross-model settings match or exceed self-decoding on individual metrics.
  • Decoder size: Increasing the decoder from 0.6B to 4B improves most metrics, while Qwen3-14B performs best in most tasks; Qwen3-8B does not consistently outperform Qwen3-4B.Classification is relatively stable across decoder sizes, whereas fact retrieval is more sensitive to decoder capacity.
  • Adapter and decoder contributions: Full UAV obtains the strongest chrF scores on information-intensive tasks, whereas decoder-side LoRA alone mainly improves classification-style performance.LoRA-only variants remain much weaker on fact retrieval and gist summarization, supporting the adapter’s role in recovering activation-grounded information.
  • Layer-wise results: Layer 27 achieves the best Token-F1 and chrF++ among evaluated Qwen3-4B layers, with middle-to-late layers generally outperforming early layers.Training and validation losses decrease across all examined layers.
  • Training data scale: Increasing Stage-2 supervision from 200K to 951K QA pairs remains stable and generally improves evaluation, with 951K producing the strongest overall performance.The gains support using larger and more diverse query-oriented supervision for activation interpretation.
  • Experimental qualification: Architecture comparisons are qualified because several MLP and cross-attention variants differ in LoRA rank, learning rate, or microbatch size beyond architecture.Matched subgroups and layer sweeps provide more direct comparisons than comparisons across all adapter families.

E.4 Baseline training settings

The baseline configurations specify training recipes for Activation Oracles, LatentQA, and zero-training Patchscope-Direct/SelfIE, alongside model revisions and evaluation examples. The evaluation spans classification, fact retrieval, and open-ended comprehension or gist prompts, with qualitative examples organized by task category and ROUGE-L level.

  • Activation Oracles Stage-2 uses LoRA rank 64, alpha 128, dropout 0.1, all-linear targets, bfloat16 computation, checkpointing, clipping at 1.0, and seed 42.The reported Qwen and Llama runs use learning rate 10−4, maximum 10 epochs, evaluation every 250 optimizer updates, and early stopping with patience six.
  • LatentQA uses LoRA rank 64, alpha 128, dropout 0.1, all-linear targets, AdamW, learning rate 5 × 10−5, weight decay 0.01, bfloat16 computation, and up to five epochs.Its scheduler advances once per epoch rather than per optimizer update, making the effective learning rate nearly constant; final settings require verification from archived metadata.
  • Patchscope-Direct and SelfIE are zero-training baselines using greedy decoding, with one injected placeholder for Patchscope-Direct and five repeated placeholders for SelfIE.Neither baseline uses an optimizer, learning-rate schedule, LoRA module, or learned checkpoint.
  • The main decoder and donor use fixed Hugging Face revisions for Qwen3-4B and Llama-3.1-8B-Instruct, while additional model revisions are provided with released configuration files.
  • Evaluation covers classification, fact retrieval, and open-ended sources, with R-L reported as sample-level ROUGE-L and qualitative examples grouped by category and score level.Representative test examples pair each input with an evaluation question and reference answer; non-Wikipedia sources use comprehension questions, while Wikipedia sources use fact-retrieval questions.
Loading 2605.25903v2…