PaperlayerView exact revision on arXiv

Source-linked AI summary

Think Twice: The Metis-1.6 Technical Report

Giulianno Vollmer

arXiv:2610.37903v1cs.LGcs.AIcs.CL

TL;DR

Language models have often pursued capability through scale, which raises training and serving costs; Metis-1.6 tests a different recipe with two looped hybrid models. Logos scores 52.93 across 40 post-trained benchmarks and lies on the data and compute Pareto frontiers, though its architecture comparisons are confounded by the optimizer used.

  • Problem

    Scaling language models can increase capability but also makes training and serving more expensive, motivating a test of whether better data, algorithms, and architectures can offer a cheaper route.

    Source: §1 · p. 3

  • Method

    Metis-1.6 builds two models from scratch around a hybrid Mamba-2 and attention stack that runs twice per token with shared weights.

    Source: §3 · p. 6

  • Results

    52.93 mean across 40 post-trained benchmarks places Logos seventh of 17, 0.75 points behind Qwen3-4B, while it sits on both data and compute Pareto frontiers.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The report says diagnosed failures have already changed how Metis-1.7 and MoRE are built.

    Source: §12 · p. 23

    Architecture conclusions are confounded because attention trained with AdamW while other parameters used plain Muon; 10B-token ablations used a smaller proxy and one run per arm.

    Source: §11 · p. 23

Abstract

from arXiv · show

Metis is a bet that better data, better training methods and new architectures can buy what a bigger parameter count cannot, at a fraction of the cost. Metis-1.6 is its latest step: two language models built from scratch by a single researcher, Praxis (2.94B parameters, 0.46B active per pass) and Logos (10.23B parameters, 1.18B active per pass). Both run a hybrid Mamba-2 and attention mixture-of-experts stack twice for every token, so each token gets a second look. Each was trained on 1.02T tokens in total: 804.76B of pretraining, 18B of context extension to 131,072 tokens, and 200B of supervised fine-tuning. Metis-1.6 was never meant to challenge frontier models, and across 40 post-trained benchmarks it mostly does not: Logos averages 52.93, seventh of 17 models in its comparison set and within a point of Qwen3-4B. But it gets there on roughly a twelfth of the median training data of the comparators that disclose it, and with fewer active parameters per token than any model that beats it, which puts it on the Pareto frontier for both. In mathematics it ranks second of 30 models, with 67.50% on AIME 2025 and 63.64% on HMMT February 2026. It is weak at code, instruction following and long context. The rest of the report is the part I found most valuable: what went wrong. The optimizer I configured never ran, and attention trained with AdamW for all of pretraining. A 1.77B-parameter n-gram memory learned to switch itself off. Five fine-tuning choices cost capability. And a length reward under group-mean normalization punished correct solutions for being long once a group's pass rate passed 2/3, a sign inversion I prove and fix with a one-line clamp. I close with the 10B-token ablations behind MoRE (Mixture of Recursive Experts), the looped architecture of Metis-1.7.

1 Introduction

The introduction frames Metis-1.6 as a small-scale test of whether better data, training methods, and architecture can offer a cheaper path to capability. It reports benchmark results alongside a candid account of training and post-training failures.

Introduction

  • The report tests the claim that better data, algorithms, and architecture can buy capability more cheaply than parameter scaling, and presents 10B-token ablations for MoRE.The author says Metis-1.6 was not intended to beat frontier models.Source: §1 · pp. 3–4
  • The report pairs a unified evaluation of open models with a complete training lineage, checkpoint hashes, and exact parameter accounting.The same harness evaluated 27 models at the base stage and 28 after post-training.Source: §1 · p. 4
  • The audit found the declared AdaMuon optimizer never ran, leaving every attention matrix trained with AdamW.A parameter-routing mismatch caused the discrepancy.Source: §1 · p. 4
  • Both models learned to close the n-gram memory gates; removing Logos’s 1.77B-parameter table changed held-out loss by less than 10−3 nats.Source: §1 · p. 4
  • The report identifies five supervised fine-tuning failure modes, a verifier imbalance, and a length-reward sign inversion under group-mean normalization, with a one-line fix.Source: §1 · p. 4

Results in brief

  • 52.93 mean across 40 post-trained benchmarks places Logos seventh of 17 models, 0.75 points behind Qwen3-4B; it sits on both data and compute Pareto frontiers.Logos used 1.02T tokens, versus a 12T median among comparators disclosing counts.Source: §1 · p. 4, Figure 1 · p. 5

2 Related work

The related work situates Metis-1.6 among research on efficient hybrid and recurrent-depth models, mixture-of-experts and memory designs, optimizers, and verifiable-reward training.

Hybrid state-space and looped models

  • Hybrid state-space models combine mostly Mamba-2 layers with some attention, while looped models reuse blocks or stacks across recurrent depth.Metis-1.6 uses three attention layers among twenty in Logos and two among twelve in Praxis, and loops its entire stack twice.Source: §2 · p. 5

Mixture-of-experts, residual streams, and memory

  • The design draws on fine-grained mixture-of-experts methods and multi-stream residual mixing, alongside conditional hashed n-gram memory.The report says Metis models learned not to use the memory.Source: §2 · p. 5

Optimizers and reinforcement learning

  • The optimizer and reinforcement-learning background covers orthogonalized momentum methods, group-normalized rewards, length control, and on-policy distillation.The report connects its length-reward sign inversion to analyses of group objectives.Source: §2 · p. 6

3 Architecture: one stack, two passes

Metis-1.6 reuses a hybrid stack twice per token, combining Mamba-2, sparse attention, latent mixture-of-experts blocks, controlled residual streams, and cross-pass memory.

Loop and depth memory

  • Logos’s 20-layer stack runs twice per token with shared weights, yielding 40 sequence-mixing and expert stages while storing only half as many.Praxis uses 12 layers and 24 stages; a depth memory carries notes from the first pass into the second.Source: §3 · pp. 6–7, Figure 2 · p. 7

Hybrid trunk

  • Logos places grouped-query attention at layers 5, 10, and 15 and uses Mamba-2 elsewhere; Praxis places attention at layers 4 and 8.Neither model uses positional encoding, and pass-indexed rank-8 updates let attention differ between passes.Source: §3 · p. 6

Latent mixture-of-experts

  • Each layer’s experts operate in a shared 1,024-dimensional latent space; Logos stores 10.23B parameters but activates 2.36B per token.One shared expert and four routed experts run, with routed-expert weights renormalized over the selected experts.Source: §3 · p. 6

Residual streams

  • A pass-conditioned controller reads, writes, and mixes four residual streams at each sub-layer boundary.The 4 × 4 mixing matrix is projected toward doubly stochastic matrices using Sinkhorn–Knopp iterations.Source: §3 · p. 6

Vocabulary and context

  • Both models use a 65,536-token byte-level BPE vocabulary and support 131,072-token contexts; released models fix two passes and four experts.Checkpoints also contain adaptive-compute heads, but these do not affect the released models.Source: §3 · p. 7

4 Pretraining and context extension

Metis-1.6 pretraining used a three-phase 804.76B-token corpus, but the configured optimizer did not run as intended. An 18B-token context extension substantially improved long-context retrieval while short-context performance barely moved.

4.2 The optimizer I thought I had

  • The 804,755,808,835-token pretraining corpus used three phases, and the declared AdaMuon optimizer never ran; every attention matrix instead trained with AdamW.The phases allocated 68.9% to broad filtered text, 26.1% to mathematics, science, software engineering and analytical text, and 5.0% to curated technical material.Source: §4.2 · pp. 8–9, §4.1 · p. 8
  • The optimizer mismatch did not stop training or appear in the loss curve, so the report treats its architectural conclusions as conditional on that optimizer.The attention parameter-routing mismatch caused attention projections, including pass-specific low-rank updates, to use AdamW throughout pretraining.Source: §4.2 · p. 9

4.3 Context extension

  • Logos needle retrieval at 131,072 tokens rose from 19.53% to 66.93% after 18B context-extension tokens, while its seven-task short-context mean moved from 62.65 to 62.24.RULER reached 60.14% at 128K, 1.32 points above the 12B-token gate; full-length document tail loss fell from 5.689 to 1.495 nats.Source: §4.3 · p. 9

5 A memory the models chose to forget

The models’ hashed n-gram memory was a large component that learned to contribute almost nothing. The released checkpoints omit it, while Metis-1.7 explores a different, off-accelerator memory design.

5 A memory the models chose to forget

  • Hashed preceding 2-grams and 3-grams supplied 16 retrieved vectors, projected and added to residual streams through learned gates at two layers per pass.Early Mamba-2 convolutions already compute local n-gram compositions, while hash collisions can mix frequent and rare n-grams.Source: §5 · p. 11
  • After training, n-gram gates were nearly closed and removing the memory changed held-out loss only slightly; released checkpoints omit it.The memory tables held 1,774,549,800 parameters in Logos and 602,180,520 in Praxis; released Logos was very slightly better without the memory.Source: §5 · p. 11
  • Metis-1.7 replaces this version with a 50B-parameter per-row ternary memory stored off-accelerator and trained with quantization-aware straight-through estimation.The stated goal is to hold long-tail facts without competing with the network for accelerator memory.Source: §5 · p. 12

6 Supervised fine-tuning, and five things I would do differently

Supervised fine-tuning first built reasoning ability, then broadened Logos into an assistant. The report identifies five training-data and objective choices to change in Metis-1.7.

6 Supervised fine-tuning, and five things I would do differently

  • Stage 2 retained reasoning and improved assistant performance: Logos scored 71.07% on IFEval and 58.60% on BFCL v4, alongside 60.00% on AIME in think mode.Stage 2 used 80.00B tokens across domains including instruction following, tool calling and conversation.Source: §6 · p. 12

6.1 Five failure modes and their fixes in Metis-1.7

  • Deleting reasoning blocks from 15–25% of traces left answers referring to unseen work; direct-mode IFEval was 21.84% versus 32.76% for think mode.The proposed fix replaces deletion-made examples with 12.0B tokens of natively generated direct-mode answers verified against ground truth and unit tests.Source: §6.1 · p. 12
  • Teacher traces could repeat verification after deriving answers, so the proposed filter removes traces with over two post-solution checks or excessive thinking on easier prompts.The filter preserves genuine 16K–64K-token derivations in software engineering and olympiad mathematics.Source: §6.1 · p. 12
  • Early-2025 teachers supplied 6.93M rows, or 74.2% of UltraData-SFT-2605; the proposed revision removes them and keeps only late-2026 teachers.The removed rows amount to about 18B tokens.Source: §6.1 · pp. 12–13
  • Sparse identity and open-conversation data left Logos sounding like its teachers, motivating 2.5B tokens of house-voice writing and 600M tokens of identity data.The original training included 4,800 identity rows and 15M tokens of open conversation.Source: §6.1 · p. 13
  • Stage 2 teacher-text cross-entropy applied unbounded gradients to low-probability stylistic tokens, collapsing generation entropy before reinforcement learning.The proposed replacement is a reference-anchored sampling objective mixed with 10% standard cross-entropy; reference log-probabilities are precomputed offline.Source: §6.1 · p. 13

7 Reinforcement learning on Logos

The Logos reinforcement-learning and distillation study exposed how an imbalanced brevity curriculum and group-mean-centered length rewards could reduce reasoning capability. It also documents operational fixes and why the final distillation run did not capture specialist peaks.

7.1 Teaching brevity with the wrong curriculum

  • Three quarters of the reward came from checklist-solvable tasks, and no science problems were included, so the curriculum taught brevity without guarding science reasoning.The proposed Metis-1.7 bridge balances domains and uses science verifiers while reducing response length.Source: §7.1 · p. 14

7.2 Punished for being right

  • When a prompt group's correct rate exceeds 2/3, long correct samples receive negative advantages under the median split and λ = 0.5.The general threshold depends on the correct fraction p, the short-correct fraction q, and the length penalty λ.Source: §7.2 · p. 14
  • At step 78, once high-pass groups exceeded p > 2/3, the 1,240-token length budget pushed down longer correct derivations as GPQA Diamond fell to 34.34% and HMMT February 2026 to 24.24%.Earlier reasoning-specialist gains included an 8.08-point rise on think-mode GPQA Diamond at step 40 and HMMT February 2026 reaching 42.42% at step 50.Source: §7.2 · pp. 14–15
  • Clamping correct-sample advantages at zero preserves the positive signal for short correct answers while removing the negative signal for long correct ones.The code, knowledge, and agentic specialists used the clamp; the reasoning specialist was not rerun with it.Source: §7.2 · p. 15

7.3 Agents waiting on containers

  • Starting sandboxes only at first tool calls, closing episode environments, and retiring cancelled records fixed the three rollout bottlenecks; the agentic specialist then led post-GSPO checkpoints on several reported suites.It scored 52.80% on think-mode BFCL v4, 47.20% on direct-mode BFCL v4, 84.10% on think-mode IFEval, 38.89% on direct-mode GPQA Diamond, and 67.14% on think-mode MMLU.Source: §7.3 · p. 15

7.4 A distillation step too small to matter

  • The final distillation improved several scores but did not capture the reasoning or agentic specialists' peaks.The single optimizer step ran at 10−7, a tenth of the target rate, and moved weights about 100 times less than a real run; the best reasoning teacher had also been pruned and longer trajectories were truncated at 8,192 tokens.Source: §7.4 · pp. 15–16

7 Reinforcement learning on Logos

  • Table 5 reports pass@1 and median generated tokens across eight suites, with the released model in the first column and bold marking each row's best entry.The study began from the released Logos checkpoint and tested reinforcement learning and specialist distillation; none of those stages became the released model.Source: §7 · p. 13, Table 5 · p. 16

8 Praxis post-training

Praxis followed a two-stage post-training path: cold-start fine-tuning established contest-math ability, and overall fine-tuning improved assistant capabilities. The released model's reported results show both math performance and remaining weaknesses across other tasks.

8 Praxis post-training

  • 27.08% on pooled AIME 2025 and 2026 in think mode followed Praxis cold-start fine-tuning, up from 0.00% on contest mathematics.The stage also reached 17.05% on HMMT February 2026 and 31.57% on GPQA Diamond, while IFEval and BFCL v4 remained at 22.22% and 14.40%.Source: §8 · p. 16
  • IFEval rose by 42.34 points in think mode to 64.56% and by 50.39 points in direct mode to 66.48% after overall fine-tuning.BFCL v4 rose by 38.00 points in think mode to 52.40% and by 23.20 points in direct mode to 50.40%.Source: §8 · p. 16
  • 27.08% on AIME 2025 and 31.25% on AIME 2026 are among the released Praxis results, alongside 24.17% on HMMT November 2025, 60.31% on AMC 2023, and 41.25% on Brumo 2025.These contest-math results are reported as avg@8.Source: §8 · p. 16
  • On RULER, released Praxis scores 75.54% at 4K, 56.23% at 32K, and 42.38% at 128K.It also scores 76.80% on GSM8K, 78.80% on MATH-500, and 62.80% on HumanEval; HumanEval+ is 50.00%.Source: §8 · pp. 16–17

9 How good is Metis-1.6, really?

The report evaluates Metis-1.6 and comparison models with shared prompts, scorers, and verifiers, then compares post-trained performance across benchmarks and domains. Logos is strongest in mathematics, while its overall results remain below several larger-data models and weak in several other domains.

9.1 Protocol

  • Base-model comparisons use one harness across 50 benchmarks, reporting a 44-benchmark short-context mean and an all-50 mean only for models that can run every length.Post-trained comparisons also use identical prompts, scorers, and verifiers; developer-published scores are not directly comparable with this harness.Source: §9.1 · pp. 17–18

9.2 Where it shines and where it struggles

  • Logos averages 52.93, ranking seventh of 17 models in its comparison set.Qwen3-4B scores 53.68, while the released-model comparison covers 40 benchmarks.Source: §9.2 · p. 18, Table 8 · p. 19
  • Across eleven math benchmarks, Logos scores 65.83, ranking second among 30 models and 2.77 points behind Gemma 4 12B; no comparator that publishes training data beats it in mathematics.It is highest in its set on HMMT February 2025 (62.92), HMMT February 2026 (63.64), and IMO-AnswerBench (57.00), but third on AIME 2025 (67.50).Source: §9.2 · pp. 18, 20
  • On active parameters per token, Logos holds the compute frontier in mathematics and science and knowledge, while models with fewer active parameters beat it in six other domains.The figure compares all 30 models; its panels share the domain order and vertical axes with the training-data figure.Source: Figure 4 · p. 20
  • In code and tools Logos ranks 17th of 30, instruction following 18th, safety 19th, and long context 24th.Think-mode RULER falls from 68.33 at 4K to 34.17 at 128K, and LongBench v2 is 13.20.Source: §9.2 · p. 20
  • Every model averaging above Logos that discloses training data used at least 28T tokens, compared with 1.02T for Logos; those models activate 3.57B to 10.94B parameters per token versus Logos's 2.36B.The report cautions that the comparison is uncontrolled because models differ in data, post-training, and size.Source: §9.2 · p. 21

9.3 Base models

  • The base Logos and Praxis checkpoints average 31.03 and 24.66 across 44 short-context benchmarks, trailing most open models of similar size.Post-training lifts both models substantially, mostly through mathematics and reasoning; Logos scores 70.50 on RULER at 64K and 60.14 at 128K.Source: §9.3 · p. 21

10 Toward Metis-1.7 and MoRE

Metis-1.7's MoRE design and optimizer were selected through two controlled 10B-token campaigns. The winning looped design outperformed both the Metis-1.6 loop and a larger non-recursive variant on paired evaluation windows.

10 Toward Metis-1.7 and MoRE

  • Metis-1.7 uses 12.48B neural parameters, with 0.794B active per pass and 1.59B per token over two passes.The model also adds a 422M-parameter vision encoder and a 204.8M-parameter speculative drafter.Source: §10 · p. 21
  • The winning loop design combines per-layer dynamic expert counts, per-pass routers, a gated cross-pass residual highway, a cosine-decayed first-pass auxiliary loss, and untied mixers and embeddings.The campaign compared eleven variants while holding the Metis-1.6 loop's tied Mamba-2 and attention weights as the starting design.Source: §10 · pp. 21–22
  • 2.5529 nats for the winning loop design beat the Metis-1.6 loop by −0.0464± 0.0008 nats and a non-recursive model with twice the expert parameters by −0.0245 ± 0.0008 nats.The loop campaign trained each arm for 10B tokens and evaluated it on the same 256 windows.Source: §10 · pp. 21–22, Table 9 · p. 22
  • 2.37104 nats and perplexity 10.708 beat AdamW by −0.18190 ± 0.00299 nats across all 256 windows in the optimizer campaign.The campaign held the winning loop design fixed and trained each arm for 10B tokens on a smaller proxy.Source: §10 · pp. 21–22, Table 10 · p. 22
  • The report says Metis-1.6 trained every attention matrix with AdamW, although its configured optimizer differed from what actually ran.The stated follow-up included strict optimizer-group assertions and separate orthogonalization of fused matrix subparts.Source: §10 · p. 22

11 Limitations

The report identifies evaluation-contamination, harness, optimizer, and reinforcement-learning limitations that constrain how its results should be interpreted.

11 Limitations

  • Several recent contests were absent from the decontamination manifest, including benchmarks where Logos is strongest relative to its comparators.The report names a dedicated overlap check as the first addition.Source: §11 · pp. 22–23
  • Comparators may score better under their developers’ recommended settings than under the shared Lernex harness.Source: §11 · p. 23
  • Architecture results are confounded because attention used AdamW while other components used plain Muon; proxy ablations used one run per arm.Their standard errors therefore reflect evaluation noise rather than run-to-run variance.Source: §11 · p. 23
  • The advantage clamp was used in short code, knowledge, and agentic specialist runs, but not in the reasoning specialist where sign inversion was observed.Source: §11 · p. 23

12 Conclusion: talk, use, and what comes next

Logos is presented as a useful small model with strong contest mathematics, though it loses most comparisons outside mathematics. The report also treats diagnosed failures as lessons already shaping Metis-1.7 and MoRE.

12 Conclusion: talk, use, and what comes next

  • Logos finishes ahead of open models trained on up to 38T tokens, within a point of one trained on 36T, and on the Pareto frontier for data and compute.The conclusion says it stands alongside the best small models from large labs in contest mathematics, while losing most head-to-head comparisons outside mathematics.Source: §12 · p. 23
  • The report lists an optimizer mismatch, an unused memory, unwanted fine-tuning habits, a length reward that punished correct reasoning, and an ineffective distillation step.Source: §12 · p. 23
  • The diagnosed failures have already changed how Metis-1.7 and MoRE are built, continuing an iterative process of building, measuring, and learning.The report describes each release as adding something the preceding model could not do.Source: §12 · p. 23
  • The author makes no promises about Metis-1.7 and intends to continue publishing both results and mistakes openly.Source: §12 · p. 23

A Base-model evaluation

  • Tables 11 and 12 compare base checkpoints after context extension with open base models using the same harness, prompts, and scorers.Table 11 covers the Logos comparison set; Table 12 covers Praxis.Source: Table 11 · p. 28, Table 12 · p. 30

B Post-trained evaluation

The appendices document post-trained comparisons, evaluation protocols, and reinforcement-learning telemetry. The supplied table descriptions establish scope and setup but do not provide benchmark score values.

B Post-trained evaluation

C Base-model evaluation protocols

  • The evaluation appendix specifies benchmark-specific prompts, scoring methods, and handling of items that exceed context windows.The protocol descriptions include text-only scoring for Humanity’s Last Exam and pass@1 sampling for code benchmarks.Source: C Base-model evaluation protocols · pp. 37–38

D Reinforcement-learning telemetry

  • Telemetry tables track response-cap changes; the agentic specialist table also records online tail retirement and the advantage clamp, with no truncation.Its cap tightens from 20,480 to 4,096 tokens, and the deferral queue stays empty.Source: Table 17 · p. 39, Table 16 · p. 39
Loading 2610.37903v1…