Prediction, Retrodiction, and The Amount of Information Stored in the Present

INTRODUCTION

• The paper extends computational mechanics’ minimal causal representations from forward time to reverse time, linking excess entropy to both representations [1, 2].It presents the work as a detailed account of the theory, proofs, and calculation methods.Source: INTRODUCTION · p. 1

OPTIMAL CAUSAL MODELS

• Causal states group histories with identical future predictions, and their transitions form an ǫ-machine, the process’s unique minimal unifilar model.The states are optimally predictive; unifilarity means each state and observed symbol determine at most one next state.Source: OPTIMAL CAUSAL MODELS · p. 2

INFORMATION PROCESSING INVARIANTS

• The ǫ-machine connects the process’s entropy rate, statistical complexity, and excess entropy, clarifying the relationship between information production, stored history, and prediction.The paper addresses the prior lack of a direct general method for calculating excess entropy from an ǫ-machine.Source: INFORMATION PROCESSING INVARIANTS · p. 2, EXCESS ENTROPY · p. 3

EXCESS ENTROPY

• The paper provides explicit expressions for excess entropy in terms of the ǫ-machine, extending earlier direct calculations beyond the special case of spin systems.Earlier work had established that statistical complexity upper-bounds excess entropy.Source: EXCESS ENTROPY · p. 3

RETRODICTION

• Retrodiction scans measurements in reverse time and groups futures that yield the same distribution over pasts, defining retrodictive causal states and a reverse-scan ǫ-machine.The paper distinguishes forward and reverse entropy rates and statistical complexities.Source: RETRODICTION · p. 3
• For stationary processes, optimally predicting the future and retrodicting the past are equally effective, although the effort required can differ.The hidden process lattice depicts how forward and reverse scans lead to their respective causal states.Source: RETRODICTION · p. 3, Table I · p. 3

EXCESS ENTROPY FROM CAUSAL STATES

• The paper analyzes a four-variable mutual information over past, future, and both causal states to derive relationships among entropy rate, statistical complexity, and excess entropy.The resulting ǫ-machine information diagram has five independent variables, rather than the fifteen in a generic four-variable diagram.Source: EXCESS ENTROPY FROM CAUSAL STATES · p. 4
• Excess entropy is the mutual information between predictive and retrodictive causal states, so the past–future information can be represented as shared information between the two state sets.This connects observed-sequence information to the process’s causal structure.Source: EXCESS ENTROPY FROM CAUSAL STATES · p. 4
• The relationship between the forward and reverse ǫ-machines is expressed through a bidirectional machine, which unifies their causal structure.The theorem and companion proposition connect excess entropy explicitly to causal structure.Source: EXCESS ENTROPY FROM CAUSAL STATES · p. 4

THE BIDIRECTIONAL MACHINE

• Bidirectional causal states partition the joint process into a subset of forward and reverse causal-state pairs.The machine describes scanning forward or backward and switching to the causal state for the opposite direction.Source: THE BIDIRECTIONAL MACHINE · pp. 4–5
• Switching maps send distributions over reverse states to forward states and vice versa, and both maps are onto.The maps are defined through conditional distributions between forward and reverse causal states.Source: THE BIDIRECTIONAL MACHINE · p. 5
• The bidirectional machine's statistical complexity is defined as the information needed to optimally predict and retrodict a process.The text identifies the bidirectional machine as the representation that uses causal information from both past and future.Source: THE BIDIRECTIONAL MACHINE · pp. 4–5
• Excess entropy is exactly the difference between the directional and bidirectional structural complexities.The bidirectional representation is also more efficient than ignoring either direction or their relationship.Source: THE BIDIRECTIONAL MACHINE · p. 5

Upper Bounds

• The new upper bounds for excess entropy are tighter than Eq. (7), with simplified proofs; one bound is E ≤ H[S−] = C−.The bound follows from basic information inequalities.Source: Upper Bounds · p. 5

Causal Irreversibility

• Causal irreversibility is the difference between the conditional entropies H[S+|S−] and H[S−|S+].Predicting and retrodicting may require different amounts of information storage.Source: Causal Irreversibility · pp. 5–6
• A zero measure does not imply identical forward and reverse machines; the periodic process 123123… is a counterexample despite differing reversed-word probabilities.Vanishing measure indicates reversibility for some processes that are not microscopically reversible.Source: Causal Irreversibility · p. 6
• A nonzero causal irreversibility measure implies that a process is not microscopically reversible.Microscopic reversibility means the process has the same word distribution when scanned in either direction.Source: Causal Irreversibility · pp. 5–6
• The authors conjecture that zero causal irreversibility captures reversibility up to isomorphism, based on case studies, and note that excess entropy does not control this asymmetry.They report no cases with zero measure but non-isomorphic left- and right-scan processes.Source: Causal Irreversibility · p. 6

Process Crypticity

• Model building is framed as decrypting hidden organization from measurements, with this treatment restricted to self-decoding without sideband communication, prior knowledge, or disciplinary assumptions.The text says model building with such additional knowledge requires solving the self-decoding case first.Source: Process Crypticity · p. 6
• The text identifies χ(M+, M−) as the distance between a process's forward and reverse ǫ-machines.It states that this distance is nonnegative, symmetric, and satisfies a triangle inequality.Source: Process Crypticity · p. 6
• Crypticity is the amount of internal state information not locally present in the observed sequence.The process therefore hides χ bits of information.Source: Process Crypticity · p. 6
• When crypticity is low, much of the stored information is present in observed behavior, so E ≈ C±μ.At the other extreme, the text describes measurements appearing nearly independent despite a highly structured process.Source: Process Crypticity · p. 6
• The bidirectional machine's statistical complexity bounds the process's crypticity.The text introduces crypticity as a measure of how accessible internal state information is from observations.Source: Process Crypticity · p. 6

ALTERNATIVE PRESENTATIONS

• Alternative presentations need not be ǫ-machines, and their states are denoted Rt rather than causal-state variables St.The ǫ-machine is described as the process's unique, minimal unifilar presentation.Source: ALTERNATIVE PRESENTATIONS · p. 7

Time-Reversed Presentation

• A time-reversed presentation is constructed by interchanging current and next states using probability rules, and it has the same stationary state distribution as the original machine.The construction generates symbols in reverse order.Source: Time-Reversed Presentation · p. 7

Mixed-State Presentation

• A mixed-state presentation uses distributions over machine states induced by observed words as its states.Many words can induce the same mixed state, and the resulting presentation applies to any machine generating the process in a forward scan.Source: Mixed-State Presentation · p. 7

CALCULATING EXCESS ENTROPY

• Starting from M+, the procedure constructs f M+ and then M−, obtaining Pr(S+|S−) for a direct calculation of excess entropy.The procedure can also start from M− to obtain M+.Source: CALCULATING EXCESS ENTROPY · p. 8
• U is not guaranteed to produce a minimal presentation; minimality of U(T(M)) remains conjectural, although an appropriate sum still yields the desired conditional distribution.Source: CALCULATING EXCESS ENTROPY · p. 8
• When the mixed-state presentation is an ε-machine, each mixed-state distribution corresponds to exactly one reverse causal state.This lets the conditional variable be expressed as a reverse causal state when calculating excess entropy.Source: CALCULATING EXCESS ENTROPY · p. 8

CALCULATIONAL EXAMPLE

• Figure 1 shows the RnC forward machine, its time-reversed presentation, and the resulting mixed-state presentation; the time-reversed presentation is not unifilar.The stationary distributions of a machine and its time-reversed presentation are identical.Source: Figure 1 · p. 9, CALCULATIONAL EXAMPLE · p. 9
• In the RnC example, mixed states are extended until no new states appear, and transient states are removed before calculating excess entropy.The recurrent reverse causal states are D, E, and F.Source: CALCULATIONAL EXAMPLE · pp. 9–10

• The examples section applies the laid-out calculation procedure to several processes, including two familiar from symbolic dynamics.Source: EXAMPLES · p. 10

Even Process

• The Even Process has two recurrent causal states but cannot be represented as any finite Markov chain over measurement sequences.Source: Even Process · p. 10
• For every self-loop probability p, χ = 0 and E = Cμ for the Even Process.The paper gives Cμ = H(1/(2 − p)) and hμ = H(p)/(2 − p).Source: Even Process · p. 10
• Figure 2 presents the forward and reverse ε-machines alongside the bidirectional machine; the forward and reverse machines are the same.Source: Figure 2 · p. 10, Even Process · p. 10
• At p = 1/2, C±μ and E are both approximately 0.9183 bits.The bidirectional-state distribution is Pr(S±) = (2/3, 1/3).Source: Even Process · p. 11
• Although E is difficult to estimate empirically because convergence is extremely slow, the time-symmetric measures expose properties that can be explored across process space.Source: Even Process · p. 11
• The one-to-one mapping between forward and reverse states implies Ξ = 0 and χ = 0, allowing prediction and retrodiction to switch without loss of information.Source: Even Process · p. 11
• As p varies, the Even Process’s information-processing properties persist across the family, and Figure 3 depicts E as the area bounded by the curves.Source: Even Process · p. 11, Figure 3 · p. 11

Golden Mean Process

• The Golden Mean Process has identical forward and reverse ǫ-machines, yet its three-state bidirectional machine has nonidentity state maps and χ>0.It is therefore causally reversible (Ξ=0) but cryptic.Source: Golden Mean Process · p. 11, Figure 4 · p. 12
• Its crypticity is χ=H(p)/(2−p), despite transition matrices and causal-state probabilities matching those of the Even Process.A quick comparison of the forward state-transition diagrams does not reveal an overt difference from the Even Process’s ǫ-machines.Source: Golden Mean Process · p. 12
• For the Golden Mean Process, E=0.2516 bits while C±µ=log2 3≈1.5850 bits, giving χ≈1.3334 bits.The excess entropy is substantially less than the statistical complexities.Source: Golden Mean Process · p. 12
• Direction switching mixes bidirectional causal states, costing χ+ bits of memory and generating χ− bits of uncertainty; complete round trips geometrically reduce state knowledge.The paper characterizes the reduction by the number of complete switches needed to halve state knowledge.Source: Golden Mean Process · pp. 12–13
• At p=0, the Golden Mean Process becomes period-2 with E=Cµ=1 bit; at p=1, it becomes period-1 with no memory.As p approaches 1, E vanishes rapidly and internal state memory becomes dominated by crypticity.Source: Figure 5 · p. 12, Golden Mean Process · p. 13

Random Insertion Process

• For the Random Insertion Process at p=q=1/2, the forward and reverse ǫ-machines differ; C+µ≈1.8464 bits, hµ=3/5 bits per measurement, and Ξ≈0.3245 bits.The differing machines indicate causal irreversibility.Source: Random Insertion Process · p. 13, Figure 6 · p. 13
• The reported excess entropy E = I[S+; S−] is 1.2464 bits, below statistical complexity µ = H[S±] ≈ 2.1219 bits, yielding crypticity χ ≈ 0.8755 bits.The passage characterizes this as a moderately cryptic process.Source: Random Insertion Process · p. 14
• Across its parameter space, the Random Insertion Process borders noncryptic reversible, semicryptic irreversible, cryptic reversible, and cryptic irreversible process classes.The plotted paths also include typical cases within the cryptic, irreversible region.Source: Random Insertion Process · p. 14, Figure 7 · p. 14

CONCLUSIONS

• Empirical excess-entropy estimates can converge so slowly, even in benign low-complexity cases, that estimation becomes computationally intractable.The authors say this difficulty was revealed by exact analysis and would not have been clear without closed-form expressions.Source: CONCLUSIONS · pp. 14–15
• The method yields closed-form excess entropy from an ε-machine, and the authors argue statistical complexity—not excess entropy—is the information stored in the present.The calculation connects time reversal of ε-machines with the mixed-state-presentation algorithm.Source: CONCLUSIONS · p. 15
• Crypticity measures stored state information not shared in the measurement sequence, while modeling with causal representations exposes it alongside causal irreversibility.The authors describe crypticity as potentially necessary modeling overhead because the ε-machine is minimal.Source: CONCLUSIONS · p. 15
• Causal irreversibility Ξ is a scalar measure related simply to excess entropy and crypticity, broader than Boolean microscopic reversibility.The authors argue microscopic reversibility can miss structural properties.Source: CONCLUSIONS · p. 15
• The framework pairs predictive and retrodictive causal models through switching maps and introduces a bidirectional machine.The authors present this expansion of the representation space as central to the time-agnostic perspective.Source: CONCLUSIONS · p. 15
• The bidirectional machine’s operational meaning warrants further attention, and its nonunifilarity may not yet be fully appreciated.The authors suggest considering a unifilar representation and possible extensions beyond one-dimensional processes.Source: CONCLUSIONS · p. 15

APPENDIX: THE MIXED-STATE PRESENTATION IS SUFFICIENT TO CALCULATE THE SWITCHING MAPS

• Although it remains conjectural that U(f M+) yields an ε-machine, the operation suffices to calculate Pr(S+|S−).This is the condition needed for the appendix’s switching-map calculations.Source: APPENDIX: THE MIXED-STATE PRESENTATION… · p. 15
• For machines A and B with U(A) = B, the mixed-state presentation makes B’s states mixed states of A.The appendix develops the generic conditional distribution Pr(RA|RB) between their states.Source: APPENDIX: THE MIXED-STATE PRESENTATION… · p. 15
• Machine B is deterministic under the construction, and its mixed states refine the causal states of the corresponding ε-machine.The appendix states H[R′|R, X] = 0 and that RB refines SB.Source: APPENDIX: THE MIXED-STATE PRESENTATION… · pp. 15–16
• Pasts inducing the same state in B belong to the same causal state of B’s ε-machine.The proof uses that equal mixed states give equal future distributions, making the pasts causally equivalent.Source: APPENDIX: THE MIXED-STATE PRESENTATION… · p. 16
• The conditional distribution Pr(RA|SB) is obtained from Pr(RA|RB) using refinement, Bayes’ rule, and stationary state probabilities.The procedure applies U to A, identifies which RB states lie in SB, then divides by the stationary probability of the corresponding causal state.Source: APPENDIX: THE MIXED-STATE PRESENTATION… · p. 16
• This method applies to excess-entropy calculation when the resultant machine T(M+) is not an ε-machine.The appendix identifies T(M+) as the input machine, with states in one-to-one correspondence with S+, and points to the resulting calculation.Source: APPENDIX: THE MIXED-STATE PRESENTATION… · p. 16