Source-linked AI summary
Capability-Stratified Degradation in Ternary Language Models
Anirudh Malik, M Sparsh Mehra, Poojith Devan
TL;DR
The paper asks which capabilities survive when a pretrained language model is converted to ternary weights and whether the result remains useful for adaptation. It creates and diagnoses Cloe through ternary conversion, representation probes, guarded evaluation, and downstream fine-tuning. The evidence shows capability-stratified degradation: broad pretrained knowledge is damaged, while selected task performance and task-specific learnability remain useful.
Problem
The paper examines which capabilities survive ternary conversion of a pretrained model and whether the compressed model remains useful for adaptation.
Method
The study converts a pretrained model to ternary weights and evaluates behavior, representations, learnability, execution validity, and deployment measurements.
Results
Cloe shows capability-stratified degradation: specialist knowledge declines, while ten tasks retain a mean 77.1% of teacher performance and downstream adaptation reaches 95.6% teacher retention on SST-2 and 79.4% on XSum.
Takeaways & Limitations
Ternary conversion is not a general-purpose teacher replacement but can provide a compact substrate for selected task-specific models.
Takeaways & Limitations
The reported capability profile reflects a single 72.4M-token recovery budget, and the complete checkpoint is approximately 6.47 bits/parameter because embeddings remain BF16.
Abstract
from arXiv · showhide
Extreme low-bit inference offers a route toward smaller models and constrained deployment. Ternary language models restrict weights to $\{-1,0,+1\}$, approaching the limit of $\log_2 3 \approx 1.585$ bits/weight. The practical question for a pretrained model is not simply whether weights can be quantised but which capabilities survive and whether it remains useful for adaptation. We explore this by converting Qwen3.5-0.8B (752M parameters) to ternary weights using 72.4M tokens of quantisation-aware training (QAT). The resulting model, Cloe, is evaluated across 29 benchmarks, representation diagnostics, and downstream fine-tuning. The evidence shows non-uniform degradation. A linear probe recovers 43.76% of MMLU answers from the full-precision teacher's representations but only 26.19% from Cloe (near chance), indicating specialist factual information is lost. However, Cloe retains measurable performance on ten tasks, averaging 77.1% of teacher performance. Crucially, fine-tuning raises Cloe to 89.8% on SST-2 (95.6% of the matched teacher) and reaches 79.4% teacher retention on XSum. We attribute degradation to a combination of quantisation-induced information loss and incomplete recovery due to the limited QAT budget. We also highlight an evaluation pitfall: standard answer-letter scoring failed (Cloe emitted "A" on 98.6% of MMLU questions), necessitating continuation scoring. Ultimately, ternary conversion is unsuitable as a drop-in general replacement yet remains valuable as a compact substrate for task-specific models.
CLOE V1.1 - OneBit AI
Cloe examines whether a pretrained 752M-parameter model remains useful after ternary conversion, emphasizing capability-specific retention rather than a single aggregate score.
- Cloe is evaluated through capability, representation, learnability, and deployment analyses rather than a single benchmark score.
- The central finding is capability-stratified degradation, with specialist knowledge and language-modelling quality declining while commonsense judgement and learnability remain measurable.
- Ternary weights constrain parameters to {-1, 0, +1}, approaching log2 3 ≈ 1.585 bits per weight.
- The study asks whether pretrained factual, linguistic, and task-relevant structure survives conversion and whether the resulting model remains adaptable.
3. Learnability: Can the compressed model still acquire a new task through supervised adaptation?
This section positions Cloe as a controlled ternary conversion and examines whether its compressed representation supports useful deployment and task-specific adaptation.
- Deployment: Deployment evaluation distinguishes packed storage, runtime memory, and throughput instead of treating them as interchangeable benefits.
- Research positioning: The study is framed as capability characterization, asking whether failed benchmarks reflect missing information, impaired output channels, representational collapse, or repairable failures.
- Base model: Cloe is derived from Qwen3.5-0.8B, a 752M-parameter model with 24 transformer layers and a 248,320-entry tied vocabulary.
- Ternary parameterisation: Every targeted linear layer contains exactly three weight values, with measured entropy of 1.5821 bits/weight near the ternary limit.
- Ternary parameterisation: The conversion uses abs-mean scaling, a straight-through estimator, and RMSNorm before low-precision matrix multiplication.
- Execution verification: Verification across 186 targeted layers indicates that reported behavior comes from quantised computation rather than an accidentally retained full-precision path.
4 Evaluation Framework
The evaluation framework combines behavioral validity checks with representation and learning analyses so that low scores are not mistaken for capability loss and high scores are not accepted without discrimination.
- The framework evaluates models from behavior to representation to learning, separating task failure from lost information or reduced adaptability.
- Answer-letter failure: 22.97% MMLU accuracy was misleading because Cloe emitted “A” for 98.6% of 14,042 questions.
- Answer-letter failure: Continuation scoring replaces answer-letter elicitation with length-normalised log-probability over candidate answer text.
- Guarded evaluation: A capability must exceed chance or majority and balanced-accuracy thresholds by two standard errors.
- Guarded evaluation: Balanced accuracy exposed CoLA, MRPC, and BoolQ as degenerate constant-label outputs despite superficially strong raw accuracies.
- Guarded evaluation: Invalid or degenerate measurements are reported for validity but excluded from headline capability results.
5 Capability Retention
Ternarisation produces a capability-stratified profile rather than uniform degradation: Cloe retains substantial performance on several discriminative commonsense and plausibility tasks, but the aggregate applies only to a selected interpretable subset.
- Teacher–Cloe comparison: 77.1% mean retention is measured across ten benchmarks satisfying discrimination guards, not across all evaluated tasks.The guard requires majority-class and chance-level balanced-accuracy criteria; remaining tasks are reported separately.
- Teacher–Cloe comparison: 86–89% teacher retention remains on RACE, PIQA, and CommonsenseQA, while QASC and SciQ remain above 80%.These results indicate that several plausibility, commonsense, and relatively accessible-knowledge capabilities remain usable after conversion.
- Evaluation scope: 77.1% should be interpreted as a selected interpretable-subset mean, not as retention of 77.1% of all general intelligence.Tasks where both models collapse or the measurement instrument is degenerate do not provide a defensible retention denominator.
- Teacher–Cloe comparison: The representative retained-capability profile is non-uniform, with several commonsense and plausibility-oriented tasks remaining substantially above chance.The figure emphasizes the spread of outcomes rather than a single scalar loss.
6 Locating the Lost Information
The study tests whether failed benchmark performance reflects inaccessible hidden-state information or information removed by ternarisation. A linear probe finds that Cloe’s final-layer representations retain little recoverable specialist MMLU signal beyond chance.
- Diagnostic question: The representation diagnostic asks whether poor benchmark performance reflects inaccessible information or information removed by conversion.This separates output-head accessibility from information loss in the hidden states.
- Linear-probe diagnostic: A multinomial logistic-regression probe uses final-layer states from 8,000 MMLU auxiliary examples and evaluates all 14,042 test questions.Synthetic-representation validation showed 100% recovery for explicitly encoded answers and approximately chance without signal.
- Linear-probe diagnostic: 26.19% probe accuracy for Cloe is only about 1.2 percentage points above four-way chance, whereas the teacher reaches 43.76%.The comparison supports substantial loss of specialist MMLU information in Cloe’s representations rather than only a poorly calibrated output head.
7 Representation Capacity Survives
Cloe retains substantial representational diversity despite losing specific specialist information. Matched fine-tuning shows that this surviving structure can support useful downstream learning.
- Representation diagnostics: 602.1 effective rank for Cloe versus 557.5 for the teacher shows that its hidden-state geometry has not collapsed.Specific knowledge loss therefore coexists with substantial representational diversity.
- Downstream adaptation: Cloe reaches 89.8% on SST-2 after fine-tuning, retaining 95.6% of the matched teacher.It begins at 61.9%, and the measured learning gap is approximately 4.13 percentage points ±2.39.
- Downstream adaptation: Cloe improves from 11.1 to 19.0 ROUGE-L on XSum and reaches 79.4% of the matched teacher.Its score also exceeds the LEAD-1 baseline by approximately 2.4×, supporting genuine summarisation learning.
- Interpretation: The combined results separate preserved representational capacity from lost pretrained knowledge and support task-specific adaptation after compression.The evidence motivates viewing Cloe as a substrate for specialised deployment rather than a general-purpose teacher replacement.
9 Capability Profile: What Changes and What Survives?
Cloe exhibits a capability-stratified profile rather than a single scalar loss: severe losses coexist with useful commonsense performance, hidden-state diversity, and downstream adaptability.
- What changes: Cloe loses most severely in specialist factual retrieval, arithmetic, and language modelling.The capability map contrasts these losses with surviving commonsense, representational, and adaptation properties.
- What survives: Cloe retains useful performance on a subset of commonsense and plausibility tasks while showing substantial hidden-state diversity.This profile is why conversion should not be interpreted as a uniform loss of intelligence.
- Evaluation implication: The relevant evaluation question for a narrow supervised application is whether Cloe retains enough capacity to learn the required task at the target deployment cost.This shifts assessment from broad teacher knowledge retention toward task-specific learnability.
10 Deployment and Efficiency Characterisation
Deployment measurements show a trade-off: packing sharply reduces persistent storage and improves throughput in the tested path, but runtime memory and initialization remain separate concerns.
- Scope: The deployment comparison does not establish a universal speedup over all full-precision implementations.The latent representation has different runtime semantics, so the result is a representation-level measurement.
- Storage and throughput: 634.7 MB packed storage versus 1.50 GB latent storage is the clearest measured deployment benefit.The comparison concerns packed and latent representations rather than a universal full-precision baseline.
- Storage and throughput: The packed path provides 22.3% higher prefill throughput and 24.2% higher decode throughput in this comparison.These are measurements from the particular tested execution path.
- Engineering trade-offs: Peak memory is approximately 1.67 GiB for the packed path, and storage compression does not translate one-to-one into runtime-memory savings.Loading and inference behavior must therefore be evaluated separately from artifact size.
11 Failure Modes as Operating-Boundary Conditions
The study frames failure modes as operating boundaries: Cloe is not a general-purpose teacher replacement, but may suit narrow objectives after task-specific adaptation.
- Operating boundary: Cloe should not be presented as a general-purpose replacement because broad pretrained knowledge is substantially damaged.The supported interpretation preserves its representational diversity and task-specific learnability as useful properties.
- Specialization: Extreme ternarisation supports a deployment paradigm focused on specialization rather than preserving every capability of a general-purpose model.The paper distinguishes compact specialised models from universal replacements.
- Specialization: Applications with narrow or well-defined objectives may value a compact substrate whose task-specific capability is learned after compression.These are deployment hypotheses rather than claims that every scenario has been experimentally validated.
- Scope boundary: The results do not establish a universal advantage across hardware, model scales, or workloads.The supported implication is limited to compact storage, tested packed-path inference benefits, and selected downstream adaptation results.
13 Discussion
The study separates stored knowledge, representational capacity, and learnability, showing that ternarisation can substantially reduce factual knowledge while preserving useful representations and downstream adaptation. It also argues that evaluation should combine behavioural, representation-level, and learning-level diagnostics, while noting unresolved limits on interpreting the results.
- Stored knowledge is substantially reduced, particularly for specialist factual information.
- Representational capacity is not obviously destroyed, as indicated by effective rank.
- Learnability remains substantial on the tested downstream tasks, allowing near-teacher performance after supervised adaptation despite weak broad-knowledge results.
- Answer-letter bias and teacher collapse can make benchmark accuracy or retention ratios misleading.
- A useful evaluation stack combines behavioural benchmarks with representation-level and learning-level diagnostics.
- The study does not establish general superiority over PTQ, general-purpose replacement, universal device-cost reduction, or an irreducible ternarisation effect because no recovery-budget ablation was performed.
14 Conclusion
Cloe exhibits a structured capability profile rather than uniform degradation: specialist knowledge declines sharply, while selected task performance, representational capacity, and adaptation remain useful. The engineering results further show that storage footprint, runtime memory, and throughput must be assessed separately.
- Specialist factual knowledge falls from 43.76% for the BF16 teacher to 26.19% for Cloe, close to the 25% chance level.
- Cloe retains 77.1% mean teacher performance across ten interpretable tasks and reaches 95.6% of the matched teacher on SST-2 and 79.4% on XSum.
- These results characterize a single 72.4M-token recovery regime rather than an irreducible capability limit of ternary representation.
- The packed artifact occupies approximately 634.7 MB on disk, while runtime memory does not decrease in direct proportion to disk footprint.
- Extreme quantisation can destroy broad pretrained knowledge without destroying the ability to represent and learn useful application-specific behaviour.