PaperlayerView exact revision on arXiv

Source-linked AI summary

From Source Reconstruction to Predictive State Preservation: An Information-Theoretic Framework for AI-Native Communication

Yi Wang, Linglong Dai

arXiv:2609.01131v1cs.IT

TL;DR

AI-native communication must determine what source information matters for prediction rather than assuming source reconstruction is the fidelity objective. The paper uses a target-conditioned predictive state to define fidelity and analyze reception, compression, and sequential updating, showing that predictive-state preservation organizes these problems while exposing a boundary between fixed-target sufficiency and recursive closure.

  • Problem

    Conventional source coding assumes the fidelity object is already specified, leaving unclear what should be preserved when communication serves prediction.

  • Method

    The paper defines the source-induced predictive state as the conditional distribution of the specified future given the source observation and shared context, then uses it across fidelity, reception, compression, and sequential prediction.

  • Results

    The predictive state is sufficient and minimal for exact preservation; log-loss distortion equals predictive information lost through communication, matched compression needs no raw-source distinctions beyond the state, and sequential prediction may require the recursively closed full-future state.

  • Takeaways & Limitations

    Communication design should begin by identifying which source distinctions matter to the terminal prediction objective, before choosing distortion, compression, transmission, or receiver design.

Abstract

from arXiv · show

AI-native communication increasingly aims to support prediction rather than reproduce every detail of the source. This shift raises a basic question left implicit by conventional source coding: what should be preserved when the terminal goal is prediction? We take the source-induced predictive state as the fidelity object. It is the distribution of the specified future conditioned on the source observation and shared context. We show that this state is sufficient and minimal for exact predictive preservation. The terminal prediction loss then defines communication distortion as lost predictive performance rather than source reconstruction error. Under logarithmic loss, this distortion equals the conditional mutual information lost through communication. Using the receiver-side predictive state as a Bayes reference, we separate AI-receiver error into predictive value lost in communication, receiver-available value unusable by the model family, and family capability not realized by the deployed model. The same predictive state also suffices for matched compression. For finite-alphabet memoryless sources, access to the raw source gives no rate-distortion advantage over coding the state directly. Applying the same target-conditioned construction to sequential prediction reveals a dynamic boundary. A state induced by a fixed horizon is minimal for that horizon but may not support recursive updating as the target window shifts. Taking the entire future as the target yields a minimal full-future state that updates recursively and admits a Markov representation. Together, these results shift AI communication from source reconstruction to predictive-state preservation.

I. INTRODUCTION

The paper argues that predictive communication must first identify what source distinctions matter for the specified future, rather than defaulting to source reconstruction. It uses a target-conditioned predictive state to organize fidelity, reception, compression, and sequential prediction.

  • Motivation: Predictive communication must identify its fidelity object before defining distortion or coding.Source realizations are equivalent when they induce the same predictive distribution for the specified future.
  • Predictive state: The source-induced predictive state preserves the full predictive distribution of the specified future given the source observation and shared context.It is a target-conditioned representation, not necessarily a transmitted probability vector or learned embedding.
  • Contributions: The framework connects predictive-state preservation to loss-based fidelity, constrained AI reception, matched compression, and sequential refinement.These components use a common target-conditioned state to relate communication and prediction objectives.
  • Contributions: Under matched predictive-state fidelity, raw-source distinctions within a predictive class provide no rate-distortion advantage over coding the state directly.The compression result concerns finite-alphabet memoryless sources and matched distortion.
  • Sequential refinement: A fixed-horizon predictive state may be sufficient for its target yet fail to support recursive updating when the prediction target shifts.The full-future state is introduced as the recursively closed sequential refinement.

C. Predictive Equivalence and the Quotient Source

Predictive equivalence groups source realizations that induce the same target distribution, yielding a quotient representation that retains exactly the distinctions relevant to prediction. The predictive state is sufficient and minimal, while the reconstruction boundary depends on the target.

  • Predictive equivalence: Predictive equivalence identifies source realizations that induce the same predictive state for a fixed context.The relation is equality of probability measures on the specified target space.
  • Quotient source: The quotient map gives a distribution-valued representation of the source distinctions relevant to the specified prediction target.The induced map is a bijection onto the attainable predictive-state space.
  • Sufficiency and minimality: The predictive state is sufficient because, once it and the shared context are known, the raw source adds no information about the target.This is the predictive sufficiency property used in the theorem.
  • Sufficiency and minimality: Every admissible predictively sufficient representation must retain enough information to recover the predictive state.The recovery is expressed through a measurable map from the representation and context.
  • Reconstruction boundary: When the prediction target is the source itself, the predictive quotient reduces to the original source space; genuine future targets may permit merging predictively equivalent realizations.Thus exact source reconstruction is recovered only when prediction requires source identity.
  • Receiver state: At the receiver, communication replaces the source-side predictive state with the conditional barycenter of states compatible with received information.This mapping provides the static predictive-fidelity interface between source and receiver.

III. PREDICTIVE FIDELITY AND INFORMATION ACCOUNTING

The paper defines predictive distortion through excess terminal prediction risk between source- and receiver-side predictive states. Under proper losses, this distortion is nonnegative and vanishes exactly when communication preserves the predictive state.

  • Loss-induced fidelity: Proper losses define predictive distortion as the excess terminal prediction risk from reporting one predictive distribution instead of the true one.The induced divergence is nonnegative, directed, and need not be a metric.
  • Loss-induced fidelity: The receiver-side predictive state is Bayes-optimal among reports based on receiver information, so communication distortion is the Bayes-risk increase from source-side to receiver-side information.This compares terminal risks rather than source reconstruction errors.
  • Communication identity: E[DL(SX∥SY)] = HL(F | Y, C) − HL(F | X, C) ≥ 0 under the proper-loss communication identity.The identity holds under the theorem’s Markov, admissibility, and well-definedness assumptions.
  • Communication identity: Under strict properness, zero expected predictive distortion occurs if and only if SX = SY almost surely.The same condition characterizes exact predictive preservation under the stated Markov relation.
  • Loss-induced fidelity: Different proper losses can induce different fidelities on the same predictive quotient, while relative entropy arises specifically under logarithmic loss.The prediction target selects the relevant source distinctions, and the terminal loss determines the cost of confusing predictive states.

B. Log-Loss Information Accounting

Under logarithmic loss, predictive distortion has an exact information-theoretic form: it is the conditional mutual information about the target unavailable from receiver-side information. The framework then distinguishes communication loss from limitations of the receiver’s model family and deployed predictor.

  • Log-loss identity: Under logarithmic loss, predictive-state fidelity is measured by relative entropy, and averaged communication distortion equals unavailable conditional mutual information about the prediction target.This connects the loss-based predictive quantity to a Shannon-theoretic representation.
  • Log-loss identity: I(F; X | C) = I(F; Y | C) + Dp(X →Y | C), so communication partitions source-side target information into receiver-available information and predictive distortion.The decomposition assumes finite mutual-information terms.
  • Log-loss identity: Dp(X →Y | C) = H(F | Y, C) − H(F | X, C), expressing distortion as the increase in conditional target entropy after communication.The entropy form applies when the relevant conditional entropies are well-defined and finite.
  • Predictive sufficiency: The predictive state preserves all selected-target information from the raw source: I(F; SX | C) = I(F; X | C).Thus quotienting the source to its predictive state loses no information about the specified target.
  • Predictive sufficiency: Exact predictive preservation is equivalent to receiver predictive sufficiency, SX = SY almost surely, or F ⊥⊥X | (Y, C).These equivalent conditions state that the receiver retains all source information relevant to the target.
  • Receiver model limitations: The receiver-side predictive state is the Bayes-optimal report given (Y, C), providing the reference for separating communication loss from model-family and deployment limitations.The admissible report class must contain this state.

B. Communication, Family, and Deployment Gaps

The paper decomposes end-to-end AI-receiver loss into communication, predictor-family, and deployment gaps, each representing a distinct bottleneck with separate zero-loss requirements.

  • Communication, Family, and Deployment Gaps: End-to-end AI-receiver loss splits into predictive value lost in communication, receiver-available value unusable by the predictor family, and unrealized deployed-model capability.The decomposition uses source- and receiver-side Bayes references and separates the three stages additively.
  • Communication, Family, and Deployment Gaps: All three gaps are nonnegative, and their sum equals the difference between deployed-model risk and source-informed Bayes risk.The stated risk identities insert receiver Bayes risk and family-optimal risk between the endpoint risks.
  • Losslessness: Under strict properness and attainment, losslessness requires predictive sufficiency at reception, family realizability of the receiver Bayes state, and deployment at the family optimum.These are the three conditions identified for zero communication, family, and deployment gaps.
  • Losslessness: Zero communication gap guarantees losslessness only for the unconstrained Bayes receiver; end-to-end losslessness also requires zero family and deployment gaps.Losslessness is defined relative to the source-informed Bayes predictor, not zero intrinsic prediction risk.

Whenever the displayed quantities are finite, Theorem 3

Under logarithmic loss, the communication gap becomes conditional mutual information, while receiver-side gaps quantify failures to exploit information already available at the receiver.

  • Log-Loss Specialization: Under logarithmic loss, the communication gap equals the conditional mutual information I(F; X | Y, C).This identifies predictive information lost through communication with a Shannon-theoretic quantity.
  • Log-Loss Specialization: Only the communication gap has a universal Shannon-information representation; family and deployment terms remain receiver-side excess-risk terms.Communication removes predictive information before reception, whereas the other terms concern exploiting information already available there.
  • Restricted Receiver Interfaces: With a restricted receiver interface U, the family gap equals I(F; Y | U, C) under log loss, measuring predictive value discarded by the interface.The interface retains received information only through U while preserving separate access to C.
  • Layerwise Diagnosis: Three receivers can share the same one-bit deployed log-loss excess risk while differing in whether communication, family, or deployment is the sole nonzero gap.Their remedies differ: improve the link or representation, enrich the interface or family, or improve training, optimization, or calibration.

V. MATCHED COMPRESSION ON THE PREDICTIVE QUOTIENT

Matched compression evaluates raw-source and predictive-state encoders under the same predictive-state distortion, showing that within-class raw-source distinctions cannot improve fidelity.

  • Matched Compression: The indirect encoder observes the raw source, whereas the direct encoder observes only its predictive state, with both evaluated under the same matched criterion.The comparison isolates whether raw-source distinctions beyond the predictive state provide compression value.
  • Matched Compression: Predictive-equivalence classes group source realizations that are indistinguishable to the fidelity criterion.The source-space distortion factors through the predictive state, so it is constant within each class.
  • Fiber-Averaging Reduction: Any reconstruction dependence on distinctions within a predictive-equivalence class consumes rate without improving matched distortion.Fiber averaging constructs a state test channel with the same joint law and distortion, while no greater conditional mutual information is required.
  • Fiber-Averaging Reduction: The fiber-averaging reduction replaces every raw-source test channel with a predictive-state channel preserving the joint law of state, context, and reconstruction.This establishes the single-letter reduction underlying the operational compression result.

C. Rate-Distortion Preservation

For finite-alphabet memoryless sources, matched rate-distortion performance is preserved when coding the predictive state directly rather than accessing the raw source.

  • Rate-Distortion Preservation: The single-letter reduction combines fiber averaging with conditional rate-distortion theory to compare raw-source and predictive-state coding.The two coding problems differ only in the encoder observation while sharing the predictive-state distortion.
  • Rate-Distortion Preservation: Theorem 4 establishes equality of the indirect raw-source and direct predictive-state rate-distortion functions for every distortion level D ≥ 0.Both directions follow by preserving distortion and matching or reducing the conditional mutual-information rate.
  • Matched-Fidelity Boundary: Access to raw-source distinctions within a predictive-equivalence class does not improve the rate-distortion tradeoff under matched predictive-state fidelity.This conclusion depends on the distortion factoring through the predictive state.

D. Log-Loss Closure and Information-Bottleneck Form

Under finite-alphabet assumptions and logarithmic loss, matched compression reduces to the predictive quotient: the Bayes distortion is conditional predictive information, and the rate-distortion formulation becomes an information-bottleneck problem on that quotient. The section remains a static characterization, while sequential recursive state representation is treated separately.

  • D. Log-Loss Closure and Information-Bottleneck Form: The finite-source extension uses a full probability-simplex reproduction alphabet and an extended-valued distortion under finite expected distortion.The source, observation, and context are finite, and the operational functions are defined for finite expected distortion.
  • D. Log-Loss Closure and Information-Bottleneck Form: Under logarithmic loss, the Bayes-optimal reconstruction is the posterior predictive law given the auxiliary and shared context.This yields distortion I(F; X | M, C).
  • D. Log-Loss Closure and Information-Bottleneck Form: Fiber averaging lets raw-source channels be replaced within predictive-state fibers without increasing rate or changing Bayes log-loss distortion.Predictive sufficiency gives F ⊥ M | (S, C), preserving the posterior predictive law.
  • D. Log-Loss Closure and Information-Bottleneck Form: The indirect and direct operational rate-distortion functions coincide after reducing the source to the predictive quotient.The reduction uses classical conditional log-loss source coding together with the quotient construction.
  • D. Log-Loss Closure and Information-Bottleneck Form: The relevance constraint can be expressed entirely through the predictive quotient rather than the raw source.The state channel and predictive sufficiency imply the needed conditional independences.
  • D. Log-Loss Closure and Information-Bottleneck Form: The information identity reduces to I(F; S | M, C), showing that the common rate-distortion form is entirely on the predictive state.The displayed chain begins with I(F; X | M, C) and eliminates the residual term through predictive sufficiency.
  • D. Log-Loss Closure and Information-Bottleneck Form: The result is static and finite-alphabet memoryless; sequential predictive states are addressed at the state-representation level rather than through a dynamic coding theorem.The sequential section instead examines fixed-horizon and recursively closed state representations.
  • A. Finite-Horizon Predictive State: For a fixed horizon k, the k-step predictive state is sufficient and minimal for preserving the current k-step predictive law.Sequential use additionally requires a measurable update map, which fixed-target minimality does not guarantee.

B. Finite-Horizon Sufficiency Does Not Imply Recursive Closure

A finite-horizon predictive state preserves the current target law minimally, but shifting the target can expose information that the state discarded. Consequently, fixed-horizon sufficiency alone does not generally provide recursive updating, motivating a full-future refinement.

  • B. Finite-Horizon Sufficiency Does Not Imply Recursive Closure: Information outside the current finite horizon may be irrelevant now but become relevant after one target-window shift.This moving-target obstruction prevents a generally valid finite-horizon update.
  • B. Finite-Horizon Sufficiency Does Not Imply Recursive Closure: For every k ≥ 1, a binary process can make the shifted state non-measurable with respect to the current finite-horizon state.The construction establishes a counterexample to recursive closure for every finite horizon.
  • B. Finite-Horizon Sufficiency Does Not Imply Recursive Closure: The counterexample begins with W1 = A, followed by B1 through Bk, and then returns to A at Wk+2.The process may be extended arbitrarily thereafter.
  • B. Finite-Horizon Sufficiency Does Not Imply Recursive Closure: At t = 1, the current target is independent of the revealed history, so its predictive law is uniform over {0, 1}^k.After observing B1, the shifted target includes A and therefore changes its predictive dependence.
  • B. Finite-Horizon Sufficiency Does Not Imply Recursive Closure: The failure occurs because Wt+k+1 lies outside the current target but enters the next target.Thus current-horizon irrelevance does not imply future update irrelevance.
  • B. Finite-Horizon Sufficiency Does Not Imply Recursive Closure: Particular processes may admit closed finite-horizon updates, but predictive sufficiency alone does not guarantee such closure.The negative result is therefore general rather than a prohibition for every process.
  • B. Finite-Horizon Sufficiency Does Not Imply Recursive Closure: Taking the entire future as the target retains distinctions that can become relevant at any later finite horizon.The full-future state determines every finite-horizon predictive state, whereas the finite-horizon state retains only the current marginal.

D. Full-Future Sufficiency and Recursive Closure

The full-future predictive state refines finite-horizon states into a representation that preserves the entire future, updates recursively, and has a Markov representation. It is minimal among full-future sufficient representations, distinguishing recursive sequential sufficiency from fixed-target sufficiency.

  • D. Full-Future Sufficiency and Recursive Closure: A recursive predictive representation must provide both full-future sufficiency and recursive closure under new observations.These are defined as separate requirements for a state process derived from history.
  • D. Full-Future Sufficiency and Recursive Closure: A measurable disintegration kernel updates the remaining future-tail law after the next observation.The kernel gives the conditional law of tail coordinates 2, 3, … given the first coordinate.
  • D. Full-Future Sufficiency and Recursive Closure: Under standard Borel assumptions, the full-future predictive state satisfies full-future sufficiency and minimality.Theorem 5 states these properties for the conditional law of the entire future tail.
  • D. Full-Future Sufficiency and Recursive Closure: The full-future state admits a Markov transition kernel, so its next state depends on the past only through the current state.The next observation has the current state’s first-coordinate marginal, and the update is given by the disintegration kernel.
  • D. Full-Future Sufficiency and Recursive Closure: The full-future state is a measurable factor of every other full-future sufficient representation, including every recursive predictive representation.This establishes its minimality within the class of recursively useful full-future representations.
  • D. Full-Future Sufficiency and Recursive Closure: Finite-horizon and full-future states solve different problems: current k-step preservation versus entire-future preservation with recursive closure.Fixed-target predictive sufficiency and recursive sequential sufficiency are therefore distinct requirements.
  • VII. CONCLUSION: The predictive-state framework makes fidelity-object selection precede distortion design and coding for prediction-oriented communication.The paper identifies the source-induced predictive state as the object that preserves the specified predictive law.
Loading 2609.01131v1…