Understanding Deep Learning via Entropy Space Theory

1 Introduction

• The authors argue that complex, high-dimensional model spaces hinder understanding AI's black box and motivate a new coordinate system linking human understanding and AI [10, 13].They characterize deep learning as minimizing information losses while quantitatively compressing space.Source: §1 · p. 2
• Existing frameworks include geometric deep learning based on symmetry and invariance [19], RKHS [6, 20, 21], reproducing kernel Banach spaces [23, 24], and free boundary neural operators [25].The introduction notes limitations including the lack of a widely accepted axiom for geometric deep learning and RKHS's limited inner-product structure for deep learning.Source: §1 · p. 2
• Entropy space comprises quotient sets of all partitions into disjoint, non-empty subsets and is proved to be a normed space within a formal axiomatic framework.Its toolkit includes a norm and addition, subtraction, multiplication, and Cartesian product.Source: §1 · p. 2
• The unified coordinate system places the input state space at the origin, degeneracy in the positive direction, and ranks compression by maximal information entropy.The paper also gives a universal definition of deep-learning depth.Source: §1 · p. 2

2.1 Definition of entropy space

• Unlike constrained microstates W, Ω covers all possible system states; Ω0 treats each state as distinct origin, equivalence classes simplify it, and Ω is independent of network parameters.The paper says the topology can cover model possibilities with arbitrary precision, while Ω is independent of network parameters.Source: §2.1 · pp. 3–4
• Entropy space is the topological space (Ω, Ω(Ω)), where Ω contains all possible states and Ω(Ω) collects quotient sets induced by every partition into disjoint, non-empty subsets.For finite Ω, the number of partitions is bounded by the Bell number B(|Ω|).Source: §2.1 · p. 3

2.2 Addition and subtraction

• Addition corresponds to union and subtraction to set difference, and entropy space is closed under both operations in the field of ΩR.Addition has identity Ω∅, inverse elements, and commutative and associative laws.Source: §2.2 · pp. 4–5

2.3 Multiplication

• Scalar multiplication maps aΩx to Ω(a · Ω(x)) and is closed in ΩR; it has identity 1 and satisfies distributive and associative laws.The stated distributive laws cover both entropy-space addition and scalar addition.Source: §2.3 · pp. 5–6

2.4 Cartesian product

• The Cartesian product of n entropy spaces forms an n-dimensional entropy space, with each Ωj serving as its j-th coordinate.The product is defined as Ω1×...×Ωj×...×Ωn.Source: §2.4 · p. 6

3.1 Metric space

• Theorem 5 defines distance as ρ(Ωx, Ωy) = |Ωx△Ωy| and establishes that it makes entropy space a metric space.The symmetric difference is the union of elements in either set but not both; its cardinality is the distance.Source: §3.1 · pp. 6–7

3.2 Vector space

• Theorem 6 states that entropy space is a vector space, with its field operations satisfying the vector-space axioms.The proof cites closure under addition and scalar multiplication, identity and inverse elements, and the distributive, commutative, and associative laws.Source: §3.2 · p. 7

3.3 Normed space

• Theorem 7 concludes that entropy space is a normed space under the distance ρ(Ωx, Ωy) = |Ωx△Ωy|.The conclusion combines the vector-space and metric-space results.Source: §3.3 · p. 7

4.1 Shannon-Boltzmann isoentropy

• Under MaxEnt, a space’s maximum information entropy is log of its number of states, and it is zero for the empty set.MaxEnt assigns equal probability to every state.Source: §4.1 · p. 7
• Shannon-Boltzmann isoentropy groups quotient sets with equal maximum entropy, where Shannon and Boltzmann entropy coincide under equal state probabilities.The authors describe it as a bridge between the physical and digital worlds.Source: §4.1 · p. 9
• For a float32 perceptron output, Hm(Ω0(y)) = log 2^32 despite potentially much greater input entropy, so the perceptron compresses the information space.The output’s maximum entropy is bounded by its allocated memory, regardless of its value’s uncertainty.Source: §4.1 · p. 9

4.2 Reinterpret of deep learning

• Depth tracks ΔHm = Hm(Ω0) − Hm(Ωx), and deep learning is a system whose depth changes markedly during computation, rather than one defined by layer count.The authors present this as a broader definition that can include human minds and distinguish non-deep-learning systems.Source: §4.2 · p. 9
• Mappings are called degeneracy, regeneration, or co-isoentropy when maximum entropy decreases, increases, or remains equal; these directions characterize depth change.Input resolution is treated as discrete and finite, even when its state space is astronomically large.Source: §4.2 · p. 10
• The coordinate system uses an input-dependent basis, a norm based on ρ(Ω0, Ωx), origin Ω0, and degeneracy as its positive direction.It can extend to arbitrary dimensions and represent multimodal states as products of modality-specific spaces; the endpoint of degeneracy has Hm = 0.Source: §4.2 · p. 10, Figure 1 · p. 11
• Network architecture quantitatively compresses node spaces, parameters select the compressed regions and are optimized by gradient descent and backpropagation, and activations compress most states toward zero to make room for features.The entropy-space topology is described as covering all possibilities of each node.Source: §4.2 · p. 10

• The main contribution is entropy space, a novel normed space proposed as a unified mathematical framework for deep learning models.Source: §5 · p. 11
• Entropy space offers a brief explanation of why deep learning can automatically learn features.Source: §5 · p. 11
• The entropy-space coordinate system has the potential to support cross-architecture learning and artificial general intelligence as a mathematical foundation.Source: §5 · p. 11