CoS-VQ: Coordinated Shared-Scale Vector Quantization for Large Language Models

1 Introduction

• LLM weight storage and repeated weight access strain memory and bandwidth, motivating compression that preserves model capabilities while reducing deployment costs.Source: §1 · p. 1
• High-dimensional trellis and lattice VQ can improve quantization quality but may restrict independent lookup and local modification of small codewords.The resulting question is whether low-dimensional codewords can retain those capabilities while matching structured methods through broader joint decisions.Source: §1 · p. 1
• Shared-scale coupling lets joint changes across positions improve reconstruction after single-index updates stall, even when the scale is re-optimized each time.The cited figure reports further normalized true layer-output error reduction after switching to joint updates, with multi-index changes accounting for most accepted updates.Source: §1 · p. 2, Figure 1 · p. 2
• CoS-VQ combines multi-probe candidate pools, a residual-aware tangent separable upper bound, and lower-envelope interval search to jointly update hard indices and shared scales.The solver avoids exhaustive codeword-combination enumeration and verifies accepted changes against true layer-output error.Source: §1 · p. 2
• CoS-VQ is reported to achieve superior low-bit quantization quality, with joint-optimization and codeword-dimension ablations supporting the design.The introduction's Figure 2 covers language-modeling perplexity and five downstream tasks, but the supplied caption gives no cell values.Source: §1 · p. 3, Figure 2 · p. 3

Vector quantization methods

• Existing VQ designs range from explicit, additive, and residual codebooks to structured lattice or trellis encodings, with different search, storage, and decoding trade-offs.The passage contrasts costly nearest-neighbor search and cache demands for unstructured high-dimensional codebooks with constraints or specialized kernels for structured encodings.Source: §2 · p. 3

Quantization optimization methods

• Prior optimization methods use sequential compensation, beam search, second-order refinement, alternating updates, or closed-form solutions for discrete assignments and scales.Source: §2 · p. 3
• CoS-VQ instead jointly optimizes hard indices and their shared positive scale using scale elimination, a residual-aware tangent separable upper bound, and lower-envelope interval search.This avoids heuristic enumeration of the Cartesian product while retaining low-dimensional codebooks with independent lookup and local rewriting.Source: §2 · p. 3

Shared-scale vector quantization

• Weights are partitioned into d-dimensional positions, with positions in a scale group reconstructed using a shared scale and accessible codebooks.The default codeword dimension is d = 6, and the dimension is ablated at matched actual bitwidth.Source: §3 · p. 3
• The joint selection range m controls how many indices may change per round, while the codebook, index bitwidth, scale-group partition, and scale count remain fixed.Positions outside the mutable set use singleton candidate pools because scale changes also affect their reconstructions.Source: §3 · pp. 3–4

Layer-output error

• Each round optimizes a complete scale group against calibration layer-output error, using finite candidate pools and a feasible positive scale interval.The objective is the squared norm of the residual after applying the group's modification through a linear operator; floating-point scales are restricted to legally representable values.Source: §3 · p. 4

Joint coupling under a shared scale

• Re-optimizing a shared scale couples candidate choices across positions, while the true output error also contains quadratic cross-position interactions.The procedure activates at most four existing scale groups per row, and m bounds mutable indices within each group.Source: §3 · p. 4

4 Method

• Proposed updates are checked against true calibration error and accepted only when they meet the prescribed decrease threshold.After joint search, optional scale fine-tuning freezes the hard indices and uses the same calibration data.Source: §4 · p. 4, §4.2 · p. 6

4.1 Residual-Aware Majorization and Candidate Construction

• The residual-aware majorizer preserves the exact first-order residual term while replacing cross-position quadratic curvature with a block-diagonal upper bound.The bound matches the objective and its continuous-weight gradients at the current point, but can be conservative.Source: §4.1 · pp. 4–5
• Multi-scale probes combine local codeword shortlists so candidates are not limited to those preferred at the current shared scale.Pools use the fixed original codebook; probes expand search coverage rather than the deployed representation or number of stored scales.Source: §4.1 · p. 5

4.2 Lower-Envelope Joint Index–Scale Optimization

• Lower-envelope breakpoints partition positive scales into representative intervals, coordinating candidate rankings without enumerating all codeword combinations.The number of intervals grows at most linearly in candidate-pool sizes rather than as their product.Source: §4.2 · pp. 5–6

5.1 Experimental Setup

• Evaluations cover four Qwen3 models, two Llama-3.1 models, and Qwen3.8-27B with thinking enabled, using perplexity and zero-shot task benchmarks.The main quantization setting uses d = 6, K = 4096, and a 12-bit index; calibration uses 320 RedPajama sequences.Source: §5.1 · pp. 6–7

5.2 Main Results

• At a 2-bit target, CoS-VQ achieves the lowest average perplexity on five of six models and the highest average downstream accuracy on three.On Qwen3-4B, PPL.AVG improves from 16.13 to 15.83 and ACC.AVG from 60.77 to 61.42 versus LLVQ.Source: Table 1 · p. 7, §5.2 · p. 7

5.3 Ablation Studies

• Joint optimization reduces WikiText-2 perplexity to 13.68/9.76 and raises average zero-shot accuracy to 62.58/69.43 on Qwen3-4B/8B.Scale-only refinement reaches perplexities of 21.01/13.85 from 22.69/14.41; the ablation attributes most gains to coordinated hard-index updates.Source: Table 2 · p. 7, §5.3 · pp. 7–8
• On Qwen3.8-27B with thinking enabled, CoS-VQ improves WikiText-2/C4 perplexity, LiveCodeBench v6, and GPQA-D over both quantized baselines, while matching them on AIME 2025.Against QTIP, LiveCodeBench v6 rises from 73.65 to 74.92 and GPQA-D from 81.11 to 82.83; all three methods score 93.33 on AIME 2025.Source: Table 3 · p. 7, §5.3 · p. 8

5.4 Efficiency

• Decode throughput rises from 21.9 to 36.7 tok/s on Llama-3.1-8B-Instruct, a 1.68× speedup at 2K context, while prefill remains comparable to BF16.Offline quantization takes 1.3 hours for Qwen3-4B and 3.5 hours for Qwen3-32B.Source: Table 4 · p. 8, §5.4 · p. 8
• Under an approximately 2-bpw budget, d = 6 outperforms d = 4 and d = 5 on all four downstream tasks, while d = 8 raises bitrate to 2.59 bpw.Scale fine-tuning consistently reduces WikiText-2 perplexity across four Qwen3 models, particularly from 9.78 to 8.51 on Qwen3-14B, with minor regressions on some HellaSwag and WinoGrande results.Source: Table 5 · p. 8, §5.4 · p. 8, Table 6 · p. 8

• CoS-VQ preserves independently accessible, locally rewritable low-dimensional codewords through coordinated shared-scale selection.Source: §6 · p. 8
• For Qwen3.8-27B, COS-VQ outperforms QTIP and LLVQ on language modeling, coding, and selected reasoning benchmarks, supporting joint selection as a promising alternative to increasing codeword dimensionality.Source: §6 · pp. 8–9
• Joint selection is presented as a promising alternative to increasing codeword dimensionality.Source: §6 · p. 9

• The theoretical optimality guarantee applies to the surrogate objective, not the original layer-output error.Candidate quality depends on multi-scale probe coverage and the chosen codebook; experiments cover selected LLM families and benchmarks, leaving broader architectures and extremely low-bit settings for future evaluation.Source: Limitations · p. 9
• Inference efficiency depends on hardware and codebook lookup implementations, and performance across deployment environments remains unexplored.Source: Limitations · p. 9

Ethical Considerations

• Energy savings depend on hardware and serving conditions and were not independently measured.The paper also notes quantization may affect factual reliability, fairness, and safety-related behavior.Source: Ethical Considerations · p. 9
• The evaluations were not dedicated assessments of safety, bias, or privacy.Quantization can alter model outputs, potentially affecting factual reliability, fairness, and safety-related behavior.Source: Ethical Considerations · p. 9
• The authors recommend retaining access controls and safety safeguards, alongside task-specific evaluations, before practical deployment.They note that reduced deployment costs may also facilitate harmful applications of LLMs.Source: Ethical Considerations · p. 9

A.1.1 Scheduling and Full-Group Scope

• The joint-search unit is a complete existing scale group, not only the positions whose indices may change.All group positions contribute to the objective and true-error verification because changing the shared scale affects fixed-index reconstructions too.Source: §A.1.1 · p. 12
• The surrogate-optimality claim is conditional on the candidate pools selected by the group and mutable-position schedules.Those schedules are fixed independently of the interval optimizer.Source: §A.1.1 · p. 12
• Each batch activates at most four existing scale groups per matrix row, while a complete sweep processes all eligible groups.When more than four groups are eligible, later batches handle the remainder; the limit does not add stored scales.Source: §A.1.1 · p. 12

A.1.2 Multi-Scale Candidate-Pool Construction

• Multi-scale probe shortlists form a deterministic truncated candidate pool that retains the current index but is not guaranteed to contain the full-codebook or surrogate optimum.Candidates are ranked by probe-occurrence count, then best probe rank and fixed codeword index; positions outside the mutable group have singleton pools.Source: §A.1.2 · p. 12

A.1.3 Full Joint-Search Pseudocode

• The interval solver minimizes the surrogate over candidate pools and legal scales, including representable-scale neighbors around each interval’s continuous minimizer and explicit boundary handling.The algorithm packs a proposal before evaluating true error and accepts it only if the true error falls by at least η; surrogate minimization alone does not ensure that strict threshold is met.Source: §A.1.3 · p. 12, §A.3.3 · p. 16

A.1.4 Complexity and Implementation Notes

• Envelope-stage complexity excludes substantial total-algorithm costs, including operator and majorizer construction, codebook scoring, and packed-candidate verification.Memory for cross-position blocks can dominate, and the base method verifies only its single best surrogate proposal; reranking all interval representatives is a distinct variant.Source: §A.1.4 · p. 13

A.2 A Checkable Coordinated-Update Instance

• The two-position example shows that refitting the shared scale after either single-index change worsens error, while jointly changing both indices to (3, 3) at s = 1/3 gives zero error.The current assignment has error 1/61; changing only position 1 gives 9/29 and changing only position 2 gives 2/5. The example demonstrates possible coordinate-wise stagnation, not guaranteed recovery by a particular majorizer or truncated pool.Source: §A.2 · p. 14, Table 7 · p. 15

A.3.1 Exact Group Objective and Block Majorization

• The exact group objective uses the full current layer residual, while a block-diagonal majorizer supplies a tangent upper bound that makes the surrogate separable across positions.The first-order term therefore includes interactions with fixed out-of-group reconstruction errors, although only in-group variables are optimized.Source: §A.3.1 · p. 14

A.3.2 Lower Envelopes and Candidate-Pool Optimality

• Lower envelopes of affine candidate-comparison lines partition legal scales into at most B = Σ_i(K_i − 1) switching points and B + 1 intervals, avoiding exhaustive index combinations.For 20 mutable positions with eight candidates each, the bound is 141 representative intervals rather than 8^20 combinations; this bounds envelope intervals, not pool generation or majorizer cost.Source: §A.3.2 · p. 15
• Surrogate optimality within the selected pools does not certify either the full-codebook optimum or the true-objective optimum, and tied boundary assignments can have different true errors.The majorizer’s slack bounds surrogate-induced suboptimality relative to the true minimizer in the same pools; a loose majorizer can miss true-improving joint moves.Source: §A.3.2 · p. 16, §A.3.3 · p. 16

A.3.4 Conditions for Preserving Envelope Combinations

• Implementation and evaluation must account for stored representation details, legal-scale precision, and verification on reconstructed stored weights, while reporting search and other runtime components separately.Envelope combinations are preserved under a sufficient pairwise ranking-sign condition, but interval-minimizing scales may still change; the condition does not restore omitted off-diagonal interactions.Source: §A.3.4 · p. 17, §A.4.2 · p. 17, §A.4.1 · p. 17

A.5 A Deterministic Lower-Envelope Example

• Four representative intervals meet the Eq. 9 bound, but each requires quadratic evaluation within its interval; minimizing linear envelopes alone is incorrect.The example illustrates envelope geometry, not a prescribed optimal scale or a guarantee about true layer-output error.Source: §A.5 · p. 18

A.6 Supplementary Results

• PPL.AVG is the arithmetic mean of WikiText-2 and C4 perplexities, while ACC.AVG is the macro-average of six zero-shot task accuracies.The six tasks are ARC-Challenge, ARC-Easy, HellaSwag, LAMBADA OpenAI, PIQA, and WinoGrande.Source: §A.6 · p. 18
• Per-task tables cover Qwen3-4B, Qwen3-8B, Qwen3-14B, Qwen3-32B, Llama-3.1-8B-Instruct, and Llama-3.1-70B-Instruct.They report WikiText-2 and C4 perplexities and zero-shot accuracies; bold marks the best quantized-method result.Source: Table 8 · p. 19, Table 9 · p. 19, Table 10 · p. 19