PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamics as Code: On Model Compression via Dynamic System

Fan Gao, Wei Su, Juntong Fan, Renfeng Peng, Hongyu Liu, Jinqiao Duan, Feng-Lei Fan

arXiv:2610.11115v1cs.LG

TL;DR

Model compression must reduce the storage demands of increasingly large pretrained networks, but DS-based compression has lacked a predictable relationship between compression ratio and error. The paper proves a conditional trajectory-covering bound and builds a broader DS compression framework with acceleration and error-control techniques. Experiments report high compression ratios for Hilbert mappings alongside limited increases in perplexity on two models.

  • Problem

    The relationship between compression ratio and reconstruction error remains unexplored, and the broader design space of DS-induced mappings has not been examined.

    Source: §I · p. 2

  • Method

    The paper proves an irrational-winding covering guarantee and combines multiple dynamic-system families with acceleration and outlier-based error control.

    Source: §I · p. 2, §VII · p. 13

  • Results

    Hilbert achieves the highest evaluated compression ratios, reaching 2.67× and 3.00× with PPL increases of 0.35 and 0.51, respectively.

    Source: §VI · p. 12

  • Takeaways & Limitations

    The authors present DS-based compression as a principled approach with a controllable error profile and engineering techniques intended to make it practical at scale.

    Source: §VII · p. 13

Abstract

from arXiv · show

The escalating size of pretrained neural networks has rendered model compression a prerequisite for deployment under stringent memory and compute constraints. With the irrational winding as an example, earlier work introduced a dynamic system (DS) paradigm that reconceptualizes compression as compact weight representation: high-dimensional parameters are encoded by the index of a trajectory produced by a dynamic system, from which the vector is recovered during decompression. This mechanism is fundamentally distinct from pruning, quantization, knowledge distillation, and low-rank decomposition. Along this direction, we prove that under a Diophantine condition, a finite trajectory of \(M = O(\epsilon^{-(d+\nu)})\) states in the irrational winding constitutes an \(\epsilon\)-net over the \(d\)-dimensional weight space, thereby linking state resolution, decompression error, and compression ratio in a predictable manner. Furthermore, we propose a generalized DS-based model compression framework by unifying four DS families---space-filling curves (Hilbert, Peano, Morton/Z-order, Snake), chaotic systems (Lorenz), congruential and pseudo-random generators (LCG, PCG), and low-discrepancy sequences (Halton). Also, we introduce the KD-tree and coordinate-template acceleration to scale to large models as well as outlier identification to control the error. Experiments on ResNet-18 and Qwen2.5-1.5B/Qwen1.5-7B validate that DS-based compression achieves competitive compression ratios without post-hoc retraining, with controllable decompression error and flexible state-space design, establishing it as a principled and practical compression approach.

I. INTRODUCTION

Large pretrained models strain deployment resources, while dynamic-system compression offers compact weight representation but leaves compression–error predictability and the choice of mappings open. This work develops theoretical and practical responses to those challenges.

I. INTRODUCTION

  • The unresolved challenges are predicting the compression-ratio–error relationship and examining mappings with different coverage, locality, and indexability properties.Earlier irrational-winding compression left the minimum trajectory size needed for a given error intractable.Source: §I · pp. 1–2
  • The paper links irrational-winding trajectory length to reconstruction error under a Diophantine condition and expands DS compression beyond a single mapping family.It also describes an optimization procedure that partitions at breakpoints and finds closed-form minimizers on continuous segments.Source: §I · p. 2
  • The framework covers space-filling curves, chaotic systems, pseudo-random generators, and low-discrepancy sequences.Examples include Hilbert, Peano, Morton/Z-order, Snake, Lorenz, LCG, PCG, and Halton.Source: §I · p. 2
  • The authors report competitive performance on ResNet and the large language models Qwen2.5-1.5B and Qwen1.5-7B.The introduction characterizes the experiments as systematic and comprehensive.Source: §I · p. 2

A. Classical Model Compression Methods

  • Classical methods have distinct trade-offs: distillation typically needs additional training and task data, while low-rank compression depends on weight rank across layers, architectures, and tasks.Practical pruning speedups can also depend on specialized kernels, storage formats, and hardware support.Source: A. Classical Model Compression Methods · pp. 2–3
  • Classical compression methods exploit sparsity, reduced numerical precision, teacher–student transfer, or low-rank structure, whereas DS compression represents weights compactly.The paper describes its approach as orthogonal to pruning, quantization, knowledge distillation, and low-rank decomposition.Source: §I · p. 1, A. Classical Model Compression Methods · p. 3

B. Model Compression via dynamic systems

Earlier DS compression work used irrational winding, while the authors examine a broader set of dynamic-system mappings. They describe how these systems generate trajectories and point sets with differing geometric properties.

B. Model Compression via dynamic systems

  • Earlier work applied irrational winding to language models, segmentation models, and image compression; this paper examines a broader family of mappings under a unified framework.The prior LLaMA2-7B result was approximately 2.6× compression.Source: B. Model Compression via dynamic systems · p. 3
  • Figure 1 contrasts structured lattice traversals, Lorenz’s low-dimensional attractor, deterministic pseudo-random LCG/PCG sets, and Halton’s low-discrepancy coverage.The caption notes that all eight mappings are visualized after normalization to the same local domain.Source: Figure 1 · p. 3

C. Dynamic Systems and Deterministic Mappings

  • Dynamic systems map an initial phase-space state to future states under an evolution rule, generating trajectories or point sets with differing locality, coverage, regularity, and indexing properties.The paper groups examples into space-filling curves, pseudo-random generators, chaotic systems, and low-discrepancy sequences.Source: C. Dynamic Systems and Deterministic… · p. 3

III. THEORY

The theory addresses the unexplored relationship between irrational-winding trajectory length and approximation error. Under a linear-form Diophantine condition, finite trajectory points form an ε-net over the weight space, giving a coverage guarantee.

III. THEORY

  • The unresolved theoretical question was how trajectory length θ relates to approximation error ε for irrational-winding compression.Earlier work established feasibility, while this work studies that relationship under mild conditions.Source: §III · p. 4
  • Under a linear-form Diophantine condition, finite trajectory points form an ε-net, so every β in T^d lies within ε of some point.The theorem applies for sufficiently small positive ε.Source: §III · pp. 4–5
  • The linear-form Diophantine condition is presented as broadly applicable, with common irrational numbers offered as examples.The text cites a two-dimensional example satisfying the condition with exponent ν = 2.Source: §III · p. 5

A. Dynamic System Families

The framework treats dynamic systems as plug-and-play components and considers four families of mappings. They differ in how they generate and index states, from lattice traversals to deterministic sequences and chaotic trajectories.

A. Dynamic System Families

  • The framework covers space-filling curves, Lorenz chaotic trajectories, congruential and pseudo-random generators, and low-discrepancy sequences.The instantiated methods include Hilbert, Peano, Morton/Z-order, Snake, LCG, PCG, and Halton.Source: A. Dynamic System Families · pp. 5–6
  • Space-filling curves map scalar indices bijectively to lattice points, with Hilbert, Peano, Morton/Z-order, and Snake as the listed traversals.Hilbert, Morton/Z-order, and Snake use m = 2^k; Peano uses m = 3^k.Source: A. Dynamic System Families · p. 5
  • The Lorenz family retains consecutive post-transient trajectory states and projects the three-dimensional states into a two-dimensional template space using PCA.The trajectory is not lattice-based; its spatial distribution follows the Lorenz attractor geometry.Source: A. Dynamic System Families · p. 5
  • LCG and PCG form multidimensional states by sliding windows over scalar recurrences; PCG applies a deterministic output permutation to the LCG state.The permutation changes the state-set structure while preserving deterministic generation and reconstruction.Source: A. Dynamic System Families · pp. 5–6
  • Halton instantiates the low-discrepancy family, mapping an index to coordinates through radical-inverse functions in chosen prime bases.For the two-dimensional implementation, the bases are 2 and 3.Source: A. Dynamic System Families · p. 6

B. The Framework of Model Compression via Dynamic Systems

The framework compresses grouped weight vectors by representing them with indices into states generated by a selected dynamic system. It combines tensor-wise preprocessing with family-specific state assignment and reconstruction.

B. The Framework of Model Compression via Dynamic Systems

  • The framework treats each dynamic system as a plug-and-play component for model compression.Weights are represented by low-dimensional scalar parameters that index high-dimensional states.Source: B. The Framework of Model Compression… · pp. 6–7
  • Each parameter tensor is flattened, divided into D-dimensional groups, and padded when needed; padding is discarded after decompression.The number of scalar entries is made divisible by D before partitioning.Source: B. The Framework of Model Compression… · p. 6
  • Grouped weights are shifted and rescaled into a layer-specific local hypercube before being represented by trajectory states.The hypercube center and size can differ across layers.Source: B. The Framework of Model Compression… · p. 7
  • Space-filling curves quantize weights to a lattice and invert the traversal to obtain an index, while other families use nearest-state assignment from finite candidate sets.The framework specifies candidate-set assignment for pseudo-random and QMC mappings, and a trajectory-state set for Lorenz.Source: B. The Framework of Model Compression… · p. 7
  • The compressed representation stores the state index rather than the vector, and decompression uses that index to recover the corresponding state.For SFCs, lattice quantization and deterministic traversal jointly provide the state ordering and index.Source: B. The Framework of Model Compression… · p. 7

C. Engineering Techniques

The engineering techniques target assignment speed for large models and representation error that could impair the compressed model. They combine family-specific search acceleration with error checks and outlier handling.

C. Engineering Techniques

  • For Lorenz states, a reusable KD-tree supports nearest-neighbor queries with average O(log(S)) complexity per query.The tree is built once over the candidate states and reused across layers.Source: C. Engineering Techniques · p. 8
  • For space-filling curves, a reusable coordinate template reduces assignment and decompression lookup overhead.The template is generated in a canonical local coordinate system and translated or scaled to align with each layer's domain.Source: C. Engineering Techniques · p. 8
  • For LCG, PCG, and QMC, exact nearest-state assignment scans candidates in memory-bounded batches and blocks at O(SD) cost per grouped vector.The minimum distance and corresponding index are updated across blocks.Source: C. Engineering Techniques · p. 8
  • The method increases S quadratically until the decompression error is acceptable and directly stores a tensor when its mean MAE exceeds the threshold.Radial classes scale farther outlier vectors more strongly, with the number of classes selected using an MAE-based threshold.Source: C. Engineering Techniques · p. 8

V. ANALYSIS EXPERIMENTS

The analysis varies grouping dimension and system scale across eight dynamic-system methods and 254 ResNet-18 configurations, then compares their compression–accuracy trade-offs. Results show that state-grid geometry, resolution, and coverage help explain method behavior and identify representative configurations.

V. ANALYSIS EXPERIMENTS

  • Hilbert, Morton/Z-order, and Snake produce identical compression and accuracy under matched configurations because they share grid points and differ only in index assignment.The resulting behavior depends on grid geometry and resolution rather than traversal order.Source: §V · p. 9
  • As scale increases, compression ratio and accuracy drop can move in opposite directions; at D = 4, the drop reaches 0.25% with compression above 5×, while D = 5 reaches 5.88× below a 1% drop.The scale trade-off depends on the balance between index cost, fallback storage, and the errors of newly compressed tensors.Source: §V · p. 9
  • At per-axis resolution 26, Hilbert, Snake, and Z-order reach 4.84–5.13× compression with at most 0.40% Top-1 drop; Peano uses a different resolution schedule, while QMC surfaces are less monotonic.Peano’s ternary resolutions do not match the binary schedules used for the other listed curves, so observed differences cannot be attributed to traversal alone.Source: §V · p. 9
  • At D = 3, 5.50× compression is achieved below a 1% Top-1 drop; at D = 4, 6, accuracy degradation fluctuates more as scale increases.At D = 3, increasing scale from 28 to 222 reduces the drop from 5.22% to 0.16%, while compression falls from 5.65× to 4.15%.Source: §V · p. 9
  • LCG reaches 5.47× compression below a 1% Top-1 drop at D = 3, while PCG stays below a 1% drop only at D = 2.LCG’s smallest drop is 0.41% at D = 4; PCG’s higher-dimensional settings show substantially larger degradation.Source: §V · p. 10
  • Lorenz is evaluated only at D = 2; increasing scale from 218 to 224 reduces compression from 3.36× to 2.56× while the Top-1 drop stays within 0.03–0.12%.Accuracy preservation is already close to saturation at scale 218, while further trajectory extension increases index cost.Source: §V · p. 10
  • Across methods, compression quality depends on how effectively a finite state space covers grouped-weight distributions, not simply on trajectory complexity or index traversal.Systems with the same number of states can behave differently, while different indexing rules can yield identical results.Source: §V · p. 10
  • Selected Pareto configurations achieve 4.80–5.77× compression with 0.14–0.59% accuracy drop, excluding Lorenz; Lorenz achieves 3.36× with 0.12%.Table II summarizes the representative ResNet-18 configurations selected from method-wise Pareto frontiers.Source: §V · pp. 10–11, Table II · p. 11

VI. COMPARATIVE EXPERIMENTS

Across two language-model scales, dynamic-system configurations offer distinct compression–quality trade-offs, and Hilbert compares competitively with existing weight-only methods. Downstream results are broadly competitive, though task-level changes vary.

Comparison with Existing Compression Methods

  • Across both models, six DS configurations occupy distinct compression–quality operating points; Hilbert has the highest compression ratios, while Lorenz stays near baseline PPL at 1.68×.Hilbert reaches 2.67× and 3.00× with PPL increases of 0.35 and 0.51; Lorenz's increases are 0.03 and 0.02.Source: §VI · pp. 11–12, Figure 5 · p. 11
  • On Qwen2.5-1.5B, Hilbert reaches 2.67× compression with a PPL increase of 0.35, versus GPTQ at 2.68× and 0.98.ZeroQuant-INT4, SQuant-INT4, and PTQ-INT4 reach 2.74× with PPL increases of 4.24, 4.27, and 5.88, respectively.Source: §VI · p. 12, Figure 6 · p. 12, Table IV · p. 13
  • On Qwen1.5-7B, Hilbert reaches 3.00× compression with a PPL increase of 0.51, versus IR at 2.91× and 1.17.The INT4 baselines reach 3.21× but have PPL increases of 2.68, 2.67, and 3.27.Source: §VI · p. 12, Figure 6 · p. 12, Table V · p. 13
  • Hilbert's average downstream scores are 66.52 and 66.12, changing by −0.79 and +0.63 points on Qwen2.5-1.5B and Qwen1.5-7B, respectively.On Qwen1.5-7B, its average score exceeds those of the three conventional INT4 PTQ baselines despite their somewhat higher compression ratios.Source: §VI · p. 12, Table IV · p. 13, Table V · p. 13
  • Hilbert's scores stay within approximately one percentage point on several broad-coverage tasks, while GSM8K, CB, MultiRC, and WSC fluctuate more across methods and scales.PPL changes do not necessarily align with individual task-score changes, so isolated improvements do not indicate consistent capability gains.Source: §VI · pp. 12–13

VII. CONCLUSION

The conclusion presents dynamic-system compression as a theoretically grounded, unified framework, with acceleration and error-control techniques for scale. Experiments confirm competitiveness, while automatic configuration selection and fuller overhead characterization remain open questions.

VII. CONCLUSION

  • The paper establishes an irrational-winding theoretical foundation and extends compression to a unified dynamic-system framework.KD-tree and coordinate-template acceleration support fast state assignment; outlier identification and threshold-based fallback address error control.Source: §VII · p. 13
  • Experiments confirm that dynamic-system-based compression is competitive.Source: §VII · p. 14
  • Automatic system and configuration selection and fuller characterization of compression and deployment overhead remain open questions.Source: §VII · p. 14

Supplementary Materials of “Dynamics as Code:

The supplied supplementary-materials section identifies the paper and its authors and lists its index terms.

Supplementary Materials

I. DETAILED SPACE-FILLING CURVE CONSTRUCTIONS

The supplement specifies four scalar-index-to-lattice traversal constructions and analyzes method-wise and global compression–accuracy Pareto frontiers. The methods differ in locality, frontier behavior, and attainable compression–accuracy trade-offs.

I. DETAILED SPACE-FILLING CURVE CONSTRUCTIONS

  • The four space-filling mappings are deterministic bijections from scalar indices to finite D-dimensional lattice points.Their explicit traversal rules supplement the shared formulation and compression role used in the main manuscript.Source: §I · p. 16

A. Hilbert Traversal

  • Hilbert traversal recursively reorients lower-order patterns within subcubes using coordinate permutations and axis reflections to preserve continuity and locality.Source: A. Hilbert Traversal · p. 16

B. Peano Traversal

  • Peano traversal recursively maps scalar indices through first-level subblocks, with an orientation operation for the corresponding lattice state.Source: B. Peano Traversal · p. 16

C. Morton/Z-order Traversal

D. Snake Traversal

  • Snake traversal uses a regular alternating scan pattern to provide a low-complexity grid-ordering rule.Source: D. Snake Traversal · p. 16

II. ANALYSIS EXPERIMENTS

  • No method dominates the global Pareto frontier; Hilbert, Peano, LCG, and QMC support its high-fidelity, moderate-compression region, while Lorenz and PCG contribute no frontier point.Hilbert, Morton/Z-order, and Snake frontiers coincide under matched configurations.Source: §II · pp. 16–17, Figure S2 · p. 17
  • Method-wise, Hilbert and Peano have stable but differently located frontiers; QMC and LCG span broader, less regular ranges, while PCG is competitive only at D = 2.Lorenz has a narrow frontier with stable accuracy preservation but limited compression capacity.Source: §II · p. 17, Figure S1 · p. 17
  • Representative non-Lorenz configurations achieve 4.80–5.77× compression with 0.14–0.59% Top-1 accuracy drops; Lorenz reaches 3.36× with a 0.12% drop.Configurations were selected near the Pareto-frontier knee, prioritizing roughly 5× compression and a drop below 0.6%; Lorenz instead uses its highest-compression near-lossless configuration.Source: §II · p. 17, Figure S1 · p. 17

Related papers

Loading 2610.11115v1…