PaperlayerView exact revision on arXiv

Source-linked AI summary

NinaXander: Feasibility and Limits of Composing Frozen Language Models Across Architecture Families via a Shared Latent Space

Takanori Kotama, Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino, Takahiro Katagiri

arXiv:2609.38261v1cs.CLcs.AI

TL;DR

Prior work had not examined recombining frozen language-model blocks across architecture families without retraining the composed model. NinaXander connects such blocks through a shared-latent adapter; one configuration sharply reduces Transformer KV cache, but the composed models do not match Pythia and perform worse on WikiText.

  • Problem

    Prior model connections either stayed within architecture families or fine-tuned the composition, leaving frozen cross-family recombination unexamined.

    Source: §I · p. 1

  • Method

    NinaXander trains a shared-latent adapter to convert intermediate representations between frozen parent models, enabling block compositions at different connection layers.

    Source: C. Shared-Latent Adapter · p. 4, B. Residuals and Notation · p. 4

  • Results

    84.4%: Pythia→RWKV (L=4) reduced Transformer KV cache by this amount with accuracy not significantly different from RWKV alone, but did not exceed Pythia and performance fell sharply on WikiText.

    Source: §VII · p. 14

  • Takeaways & Limitations

    The results show that frozen models from different families can be recombined post hoc, but the study’s significance is not a higher-performing new model.

    Source: §VII · p. 14

    The adapter was trained with one random seed, and evaluation data were also used for learning-rate adjustment and checkpoint selection, which may make read-out accuracies somewhat optimistic.

    Source: §VI · p. 14

Abstract

from arXiv · show

In this paper we propose NinaXander, a series of composed language models obtained by connecting layers of frozen language models from different architecture families with a single trained shared-latent adapter. A composed model runs the first layers of one model, converts the resulting intermediate representation once with the adapter, and then runs the remaining layers of the other model. Once the adapter is trained, several composed models that connect at different layers are obtained without retraining. Using the recurrent RWKV-4-Raven-7B and the Transformer-based Tulu-Pythia-6.9b, abbreviated as RWKV and Pythia, this study examines whether frozen models from different families can be recombined post hoc. The composed models answered multiple-choice questions, and those whose generations we examined produced syntactically well-formed text. The configuration that combines the first 5 layers of Pythia with the remaining 27 layers of RWKV reduced the Transformer key-value (KV) cache by 84.4% with accuracy not significantly different from that of RWKV alone. In multiple-choice accuracy, however, no composed model matched the parent model Pythia, and language-modeling performance decreased sharply on WikiText, a corpus of Wikipedia articles outside the training domain. The correspondence between intermediate representations was also obtained in one favorable case, with a shared tokenizer, the same depth, and the same hidden width, and does not show that the models share a general semantic space.

I. INTRODUCTION

The paper asks whether frozen models from different architecture families can be recombined, addressing a gap left by approaches that retrained weights. It frames the challenge as converting intermediate representations between model-specific coordinate systems while managing different inference costs.

I. INTRODUCTION

  • The study tests frozen cross-family composition, where prior model connections either stayed within Transformers or fine-tuned the composed model.Earlier architecture changes to reduce the KV cache also required retraining weights.Source: §I · p. 1

A. Residual Stream and Inference Cost

  • A residual stream carries hidden vectors between blocks, but a representation from one model may not match the coordinate system expected by another model’s next block.Transformer cache size grows as O(T) with context length T, whereas recurrent linear-attention layers retain a fixed-size O(1) state.Source: A. Residual Stream and Inference Cost · p. 2

B. Model Stitching

  • Prior stitching studies examined image-recognition models, while this study connects recurrent and Transformer language models through a common latent space and one intermediate conversion.The paper evaluates representational agreement separately from recovery and task performance.Source: B. Model Stitching · p. 2

C. Composing Frozen Large Language Models

Prior language-model composition methods connect frozen models within narrower architecture settings, fine-tune the composition, or do not build a composed generator. NinaXander instead recombines frozen blocks in activation space, complementing weight-space merging.

C. Composing Frozen Large Language Models

  • Prior LLM composition includes Transformer stitching and cross-attention, while Manticore combines Transformer and Mamba blocks but fine-tunes the whole hybrid model.Other proposals use routing or exchange hidden states between frozen experts and a hub.Source: C. Composing Frozen Large Language… · p. 2

D. Representation Similarity

  • Representation-similarity measures such as linear CKA and fitted per-layer linear maps are auxiliary indicators; the study focuses on functional stitching through reconstruction.Prior work reports that representational similarity need not reflect task performance.Source: D. Representation Similarity · p. 3

E. Model Merging and Knowledge Fusion

  • Weight-merging methods assume identical or compatible weights, whereas NinaXander recombines frozen blocks by converting activations and is therefore complementary to them.FuseLLM instead aligns tokens across vocabularies and additionally trains a target model.Source: E. Model Merging and Knowledge Fusion · p. 3

F. Position of This Study

The paper’s target is a single-point connection between frozen recurrent and Transformer models, evaluated as a generator with both parent models and language-model heads frozen. Its chosen models share conditions that make layerwise pairing favorable, so other mismatched configurations remain outside scope.

F. Position of This Study

  • The study addresses cross-family stitching that prior work had not examined, connecting a recurrent linear-attention model and a Transformer at one point with both parent models and language-model heads frozen.It also reports the resulting reduction in Transformer KV cache.Source: F. Position of This Study · p. 3

A. Target Models

  • RWKV-4-Raven-7B and Tulu-Pythia-6.9b share a tokenizer, 32-block depth, and 4096 hidden width, enabling direct pairing of token-position residuals and corresponding layers.Pairs differing in tokenizer, depth, or hidden width are outside the study’s scope.Source: A. Target Models · p. 3

B. Residuals and Notation

The method standardizes model residuals and uses a shared-latent adapter to reconstruct each model’s representations and align their latent vectors. Cross-family read-outs use one model’s encoder with the other model’s decoder.

B. Residuals and Notation

  • Residuals are hidden vectors passed between blocks, and the method standardizes each model’s residuals using layer-specific training-data statistics before adapter processing.The inverse operation restores the original scale.Source: B. Residuals and Notation · p. 4

C. Shared-Latent Adapter

  • A shared-latent adapter uses an encoder and decoder for each model, mapping standardized residuals into a common space and recovering residuals from that space.The same adapter serves all 32 layers without conditioning on layer index.Source: C. Shared-Latent Adapter · p. 4
  • The training loss combines each model’s self-reconstruction error with latent alignment, weighted by λ; the study fixes λ=1 and noise standard deviation σ=0.4 without tuning them per layer, direction, or task.The authors report no ablation varying these settings.Source: C. Shared-Latent Adapter · pp. 4–5
  • Cross-family read-outs combine one model’s encoder with the other’s decoder, and paired residuals are used even though no cross-family target is directly predicted.The loss aligns the two latent vectors to support these read-outs.Source: C. Shared-Latent Adapter · p. 5
  • Noise before decoding is intended to prevent a token-independent common mode from dominating the latent, because low per-token variation makes residual recovery harder.The adapter has 268.5M parameters, 1.9% of the combined 14.25B parent-model parameters.Source: C. Shared-Latent Adapter · p. 5

D. Configurations and Chimera Models

The section defines how frozen parent models are joined at a switch layer and describes the read-outs and evaluations used to assess configurations. The evaluation covers recovery and alignment, generated text, multiple-choice accuracy, and language modeling.

D. Configurations and Chimera Models

  • The four compared configuration types are RWKV self-reconstruction, RWKV→Pythia, Pythia self-reconstruction, and Pythia→RWKV.The self-reconstruction controls pass through the same adapter and switch layer as the corresponding cross-family configuration.Source: D. Configurations and Chimera Models · p. 5
  • A chimera runs blocks 0 through switch layer L of a front model, converts its residual once, then runs blocks L+1 through 31 and the back model’s language-model head.Each chimera runs 32 blocks total; Pythia→RWKV (L=4), for example, runs five Pythia blocks followed by 27 RWKV blocks.Source: D. Configurations and Chimera Models · p. 5

E. Evaluation Metrics

  • Read-out accuracy is the coefficient of determination R2, measuring recovery of a true residual; cross-family read-outs are compared with self-reconstruction controls.R2=1 indicates perfect recovery, 0 matches a per-dimension mean predictor, and a negative value is worse than that predictor.Source: E. Evaluation Metrics · p. 5
  • Layer-level alignment uses centered latent correlation and varying fraction, with per-layer means reported across 32 layers.The evaluation data comprise separate 112-token sequence sets for main values and per-layer analysis; fitted linear-map R2 uses the same samples for fitting and evaluation and can therefore be inflated.Source: E. Evaluation Metrics · p. 6
  • The language evaluation uses deterministic argmax generations, ARC and closed-book SciQ multiple-choice accuracy, and next-token cross-entropy on Alpaca and WikiText.SciQ results are not directly comparable with standard-protocol results that provide support passages; WikiText is used to assess performance outside the adapter’s training domain.Source: E. Evaluation Metrics · pp. 6–7

F. Training Setup

The adapter was trained on paired residuals from Alpaca sequences, with layers sampled uniformly during optimization and the checkpoint selected by evaluation A→B performance.

F. Training Setup

  • Training used residuals from 2,075 Alpaca training-split sequences of 112 tokens, yielding about 232k token positions and 7.44M residual pairs, iterated about 228 times.The sequences came from the beginning of the training split.Source: F. Training Setup · p. 7
  • Optimization ran for 416,000 iterations with AdamW, effective batch size 4096, learning rate from 10^-3 to 10^-5, and gradient norm clipped at 1.0.Each iteration uniformly sampled one of the 32 layers and drew an effective batch from that layer’s residual pairs; the learning rate halved after four consecutive evaluations without improvement.Source: F. Training Setup · p. 7
  • The iteration-415,000 checkpoint was selected because it had the best evaluation A→B score, not because of B→A or multiple-choice results.The evaluation reference was updated only when a score exceeded it by more than a relative 10^-3.Source: F. Training Setup · p. 7

IV. RESULTS

The adapter produced aligned, cross-decodable representations on held-out text, but the evidence depends on token correspondence and does not establish that same-index layers are inherently special.

A. Shared Representation Between Corresponding Layers

  • Centered latent correlation reached 0.901 on text unused for training, while cross-family read-out accuracies were 0.559 for A→B and 0.590 for B→A.Self-reconstruction read-outs were 0.695 for A→A and 0.711 for B→B.Source: A. Shared Representation Between… · p. 7
  • Shuffling token correspondence reduced centered correlation from 0.901 to 0.000 and cross-family read-out accuracy from 0.561 to −0.592 for A→B and 0.592 to −0.609 for B→A.Self-reconstruction accuracy does not change under this shuffle control.Source: A. Shared Representation Between… · p. 7
  • Same-index layer pairs were not intrinsically special: mean linear CKA was 0.05 on the diagonal versus 0.03 off it, and the diagonal was most similar in only 2 of 32 rows.Fitted R2 averaged 0.87 on the diagonal and 0.84 off it, but was measured on the fitting rows; Fig. 3 shows both all-pair comparisons.Source: A. Shared Representation Between… · p. 7, Figure 3 · p. 8
  • The raw latent correlation of 0.965 exceeds centered correlation 0.901 because the token-independent common mode accounts for 65% of the latent’s squared norm.The varying fraction was 0.350, so the paper uses centered correlation as its alignment measure; read-outs assess whether the varying component carries information.Source: A. Shared Representation Between… · pp. 7–8

B. Self-Reconstruction Compared with Cross-Family Read-Outs

Cross-family read-outs underperform self-reconstruction, while adapter nonlinearity contributes to read-out and task accuracy in the tested configurations. The experiments do not establish that nonlinearity is generally necessary for connecting model families.

B. Self-Reconstruction Compared with Cross-Family Read-Outs

  • At all 32 layers, cross-family read-out accuracy was no higher than self-reconstruction, though substantial error was already present at the self-reconstruction stage.The ordering is experimental, not guaranteed by theory; improving cross-family conversion alone would not remove the self-reconstruction error.Source: B. Self-Reconstruction Compared with… · p. 8
  • Pythia-to-RWKV read-outs slightly exceeded RWKV-to-Pythia read-outs, indicating RWKV residuals were easier to recover from latents encoded by Pythia.The correlation coefficient and the four coefficients of determination in Fig. 4 use different scales and should not be compared directly.Source: B. Self-Reconstruction Compared with… · p. 8
  • At latent width 4096, Pythia self-reconstruction reached only 0.711, and the source of the remaining reconstruction error is unresolved.The study did not determine whether the limit comes from noise, encoding, latent normalization, decoding, or training noise.Source: B. Self-Reconstruction Compared with… · p. 8

C. Nonlinearity of the Adapter

  • The adapter's imitation affine map explained 86.6% of output variance for RWKV→Pythia and 88.0% for Pythia→RWKV, leaving the trained map nonlinear in both directions.The affine map approximated the trained adapter on evaluation rows.Source: C. Nonlinearity of the Adapter · p. 9
  • The direct affine map scored 0.487 for RWKV→Pythia and 0.535 for Pythia→RWKV, while the adapter scored 0.558 and 0.576, respectively.The adapter's gains were +0.071 and +0.041 on five representative layers; these differ from the 32-layer means because the layers and rows differ.Source: C. Nonlinearity of the Adapter · p. 9
  • The affine accuracy control differed from the adapter by −3.4 to +2.0 points for Pythia→RWKV across eight conditions, unlike the 8–23-point losses for RWKV→Pythia.At L=4 on ARC the adapter was 2.0 points higher (p=0.005), while at L=8 on SciQ the affine control was 3.4 points higher (p=0.003); differing objectives, data, and regularization prevent a causal interpretation.Source: C. Nonlinearity of the Adapter · p. 9, Table I · p. 9
  • Removing the nonlinear part lowered mean read-out accuracy to −0.41 for RWKV→Pythia and −0.30 for Pythia→RWKV, with accuracy varying non-monotonically across α.Even at α=0, non-affine layer normalization remains; the encoder and decoder projections were jointly trained with the nonlinear part.Source: C. Nonlinearity of the Adapter · p. 9, Table II · p. 9
  • Disabling the nonlinear part of the jointly trained adapter lowered task accuracy by 39.6 points for RWKV→Pythia and 11.2–36.6 points for Pythia→RWKV (α: 1→0).This result concerns removing the nonlinear part from a jointly trained map; it does not show that nonlinearity is generally necessary for connecting families.Source: Table II · p. 9, C. Nonlinearity of the Adapter · p. 10

D. Accuracy Compared with the Parent Models

Both cross-family chimera directions answered ARC and SciQ questions, but none matched Pythia's accuracy. Only RWKV→Pythia at L=4 significantly exceeded RWKV, and only on SciQ using length-normalized scores.

D. Accuracy Compared with the Parent Models

  • All 16 chimera conditions were significantly less accurate than Pythia, whose accuracy was 65.3% on ARC and 81.0% on SciQ.Every configuration nevertheless exceeded random-choice accuracy, about 25%, by a wide margin.Source: D. Accuracy Compared with the Parent… · p. 10
  • Only RWKV→Pythia (L=4) significantly exceeded RWKV, and only on SciQ using length-normalized scores; Pythia→RWKV never significantly exceeded RWKV.RWKV alone scored 61.8% on ARC and 67.0% on SciQ.Source: D. Accuracy Compared with the Parent… · p. 10
  • RWKV→Pythia accuracy declined monotonically with deeper switches, whereas Pythia→RWKV was non-monotonic and substantially exceeded it at L=16 and L=24.Cross-family read-out accuracy alone did not predict the accuracy ranking at matched switch depths.Source: D. Accuracy Compared with the Parent… · p. 10
  • Self-reconstruction controls also lost accuracy relative to their parent models, with significant differences at every Pythia switch depth and at deeper RWKV switch depths.For RWKV self-reconstruction, the difference was significant on both tasks at L=16 and 24, and only on ARC at L=8.Source: D. Accuracy Compared with the Parent… · p. 10
  • Differences between a chimera and a parent cannot be attributed to conversion error alone because they also include differences between the front parent models.With RWKV as back model, Pythia→RWKV exceeded the RWKV self-reconstruction control at deep switches, including +8.2/+12.0 points on SciQ at L=16/24.Source: D. Accuracy Compared with the Parent… · p. 10

E. KV Cache Reduction

Pythia→RWKV (L=4) offers a cache–accuracy trade-off point unavailable from RWKV-first configurations, while comparisons with early exit and runtime measurements show important limits to interpreting that advantage.

E. KV Cache Reduction

  • Pythia’s KV cache grows with context length, whereas RWKV uses a fixed-size recurrent state; the reported reduction applies to cache, not resident weights.The adapter occupies 0.50 GiB in FP16, and chimera resident weights including the adapter are 13.49–14.55 GiB.Source: E. KV Cache Reduction · pp. 10–11
  • Pythia→RWKV (L=4) reduces Transformer layers from 27 to 5 versus RWKV→Pythia (L=4), with two-task mean accuracy changing from 65.25% to 65.70%.Fig. 5 presents the trade-off between KV cache reduction and SciQ accuracy.Source: E. KV Cache Reduction · p. 11, Figure 5 · p. 11
  • Adding Pythia-first configurations yields a high-reduction trade-off point unavailable from RWKV-first configurations, rather than configurations that uniformly outperform the parents.RWKV alone needs no KV cache, and its accuracy does not differ significantly from Pythia→RWKV (L=4).Source: E. KV Cache Reduction · p. 11
  • For configurations retaining 17 or fewer Transformer layers, chimera models significantly exceed matched-layer Pythia early exits on both tasks, but the comparison does not equalize computation.Pythia→RWKV (L=4) scores 62.2% versus 26.8% on ARC and 69.2% versus 30.5% on SciQ against an early exit after five Pythia blocks.Source: E. KV Cache Reduction · p. 11, Table VI · p. 11
  • At context length 512, RWKV→Pythia (L=4) takes 1,128 ms and Pythia→RWKV (L=4) takes 5,652 ms in the measured implementation.The authors caution that these measurements depend on the implementation and do not compare the architectures’ computational cost.Source: E. KV Cache Reduction · pp. 11–12

F. Effect of Domain Shift

On text outside the adapter’s training domain, chimera-model perplexity rises sharply, although Pythia→RWKV is less affected than RWKV→Pythia. The examined generations are syntactically well formed but can be factually inaccurate or lose the requested format.

F. Effect of Domain Shift

  • At context 512, Pythia→RWKV (L=4) perplexity is 3.85 on Alpaca versus 45.00 on WikiText; at 2048, WikiText perplexity remains 34.78 versus RWKV’s 8.27.At context 512, RWKV→Pythia (L=4) has perplexities of 6.40 on Alpaca and 93.87 on WikiText; Alpaca overlaps parent instruction data.Source: F. Effect of Domain Shift · p. 12, Figure 6 · p. 12
  • For Pythia→RWKV, a ridge-fitted affine control has lower WikiText perplexity than the adapter at every tested switch layer and context length, suggesting conversion may contribute to out-of-domain degradation.The control is fitted on WikiText tokens not used for evaluation, making the comparison unfavorable to the adapter.Source: F. Effect of Domain Shift · p. 12

G. Generation Examples of the Chimera Models

  • The L=4 generation examples are syntactically well formed, but both directions answer the water-composition prompt incorrectly and may lose content or instruction format.The choice of L=4 followed evaluation results; RWKV alone also gives the wrong water answer and drifts after the primary-colors prompt.Source: G. Generation Examples of the Chimera… · p. 12, Table VII · p. 13

V. DISCUSSION

The discussion confines the learned alignment to tokenwise correspondence under favorable conditions, while the conclusion presents NinaXander as a way to recombine frozen models, not to produce a higher-performing model.

V. DISCUSSION

  • The learned representation had ρctr =0.901, but the evidence supports tokenwise correspondence only under favorable conditions, not a general shared semantic space.The comparison used models with the same tokenizer, hidden width, and depth, with inputs in a domain close to Alpaca; meaning, vocabulary, position, and syntax were not disentangled.Source: §V · p. 13
  • A low-capacity affine map already achieves substantial cross-family prediction, which the authors cite against adapter capacity alone explaining the observed alignment.Smith et al. report stitching success across tasks and modalities, including pseudo-representations; the authors note that stitching success does not itself mean models represent the same information.Source: §V · p. 13
  • Zhang and Xin [35] found that transferred activations could score R2 = 0.883 yet reduce multi-step reasoning performance, motivating separate evaluation of alignment, decodability, and receiver use.NinaXander’s objective does not include the receiving model’s output, and the contribution of its per-dimension unstandardization was not isolated.Source: §V · p. 13

VI. LIMITATIONS OF THIS STUDY

  • Verbatim matching found 19 ARC and 4 SciQ evaluation questions in Pythia’s released mixture, but the accuracy changes after their removal were at most 0.22 points.The detected overlap counts are lower bounds, and ARC and SciQ results are best read as retention of learned ability rather than acquisition of new ability.Source: §VI · pp. 13–14
  • The adapter used one random seed, evaluation data informed learning-rate and checkpoint choices, and multiple comparisons were not corrected.Each composition used one switch layer; the likelihood-based multiple-choice evaluation does not directly measure correct answer generation, and four examples per direction are only qualitative supplements.Source: §VI · p. 14
  • Long-context cross-entropy was measured only to 2048 tokens, beyond the converted RWKV configuration’s specified 1024-position range, and does not directly measure long-range reading comprehension.The original checkpoint’s training context length could not be confirmed; several architecture and training-data ablations were left for future work.Source: §VI · p. 14

VII. CONCLUSION

  • Pythia→RWKV (L=4) reduced the Transformer KV cache by 84.4% with accuracy not significantly different from RWKV, but no chimera exceeded Pythia.Configurations reducing Transformer layers to 17 or fewer exceeded a Pythia early exit with the same KV cache size by a wide margin on both tasks; performance also decreased sharply on WikiText.Source: §VII · p. 14

APPENDIX

The appendix describes the released inference artifacts, adapter-training setup, evaluation data, and reproduction materials.

APPENDIX

  • The adapter was trained on 4 NVIDIA V100 GPUs, and evaluation sequences came from Alpaca beginning at character position 2,000,000.Layerwise standardization statistics were computed from training residuals and also used for unstandardization.Source: APPENDIX · p. 16
  • The repository includes experiment code, raw evaluation data, aggregate tables, reproduction procedures, and a script checking reported numbers against those tables.For the RWKV back model in Pythia→RWKV, options were evaluated with batch size 1.Source: APPENDIX · p. 16
  • Three inference repositories provide the shared adapter and standalone chimeras for both directions, with L=4 as the default switch layer.The default was selected after seeing evaluation results, with no independent data split; the artifacts bundle frozen parent models, tokenizer, adapter, and custom inference code.Source: APPENDIX · p. 16
  • The inference artifacts bundle the two frozen parent models in FP16 and use the AI2 AI Model License for non-commercial research.The repositories include model cards in English, Japanese, and Simplified Chinese, attribution notices, Safetensors weights, and SHA-256 hashes.Source: APPENDIX · p. 16
Loading 2609.38261v1…