PaperlayerView exact revision on arXiv

Source-linked AI summary

Finite-Sample Limits of Entropy-Based Structure Identification in Discretized Nonlinear Systems

Pratishtha Shukla, James Nutaro

arXiv:2609.03074v1eess.SYcs.ITcs.SYmath.IT

TL;DR

Discretization and stochasticity can limit entropy-based identification of the inputs that drive an output, even as sample size grows. The paper analyzes FIR mask selection using a resolution-stochasticity ratio and compares entropy-based selection with MSE-based prediction. It characterizes regimes for recovery, excess prediction risk, and sample requirements, and applies the analysis to grid reliability data.

  • Problem

    The paper addresses when entropy-based mask selection can recover input structure under measurement stochasticity and discretization, a setting where the reliability of mask-ranking criteria is not precisely understood.

    Source: §I · p. 1

  • Method

    The paper develops a theoretical analysis of entropy- and MSE-based mask selection using a resolution-stochasticity quantity to characterize when entropy selection is reliable.

    Source: §I · p. 1

  • Results

    Entropy masking identifies an informative mask when ρ < 1 but loses discriminative power otherwise, and entropy over-selection induces excess prediction risk.

    Source: §VI · p. 8

  • Takeaways & Limitations

    The regime-based guidance distinguishes entropy masking for explainability from the robust MSE criterion for prediction, with a permutation test offered to screen spurious entropy reductions.

    Source: C. Regime-Based Decision Principle · p. 6

    The excess-risk bound treats K(ME) as fixed and may understate expected risk when mask size varies; the threshold ρc assumes Gaussian stochasticity.

    Source: §VI · p. 8

Abstract

from arXiv · show

Discretization fundamentally limits structure identification in stochastic systems. When system stochasticity exceeds the discretization resolution, entropy-based methods lose their ability to distinguish which input drives the output. We study this in Fuzzy Inductive Reasoning (FIR), a nonparametric framework for learning dynamical systems from discretized measurements, where the choice of input variables determines both predictive accuracy and the interpretability of the learned input--output relationships. Entropy-based selection targets explainability, i.e., identifying which variables causally drive the output, while mean-squared-error-based selection targets prediction. We introduce a resolution-stochasticity ratio that governs when entropy-based selection is reliable. Three results follow. First, entropy-based selection is consistent below this threshold but loses discriminative power above it, regardless of sample size. Second, using the entropy-selected variables for prediction instead of the MSE-selected ones incurs a closed-form excess prediction risk that grows with input complexity and shrinks with sample size. Third, reliable identification of the causally relevant inputs requires data that scales with the number of input combinations and inversely with the strength of the entropy signal. The theory is validated on a two-state Markov model and demonstrated on a distribution grid reliability dataset analyzing the impact of infrastructure investment, where the goal is to explain which investments drive reliability improvements rather than merely predict outcomes.

I. INTRODUCTION

FIR mask selection must balance identifying variables that explain the output with selecting inputs that predict it well. The paper analyzes when entropy-based selection recovers input structure and how discretized FIR models represent dynamics through input masks and rule tables.

II. FUZZY INDUCTIVE MODEL

  • FIR learns input–output dynamics by discretizing variables into qualitative levels and constructing probabilistic transition rules from observed patterns.The framework has been applied to systems with partially unknown dynamics, including autonomous space systems and complex energy demand processes.Source: §I · p. 1, §II · p. 2

I. INTRODUCTION

  • Entropy-based selection targets explainability, while MSE-based selection targets prediction; the paper analyzes how measurement uncertainty and discretization affect reliable structure recovery.Too few inputs may omit relevant structure, while too many produce sparse rule tables and degraded predictions.Source: §I · p. 1
  • The analysis characterizes when entropy selection is reliable, quantifies excess prediction risk and recovery data needs, and demonstrates infrastructure-investment analysis on grid reliability data.The stated scope extends beyond FIR to discretized nonparametric models with sparse rule tables.Source: §I · pp. 1–2

A. System and Mask

  • A mask selects variable–lag pairs as predictors, and its choice affects model quality; using both previous values in the example yields more informative, near-certain predictions.The discovery of a best mask is the goal of the FIR procedure.Source: A. System and Mask · p. 2

B. Input Cells and the Rule Table

A mask partitions discretized inputs into cells, and FIR estimates a conditional output distribution for each cell; together, these estimates form the rule table.

B. Input Cells and the Rule Table

  • Each unique combination of mask-selected input values defines a cell, and a fixed mask has K(M) possible cells regardless of the number of observations.For two inputs with three bins each, the example has nine cells.Source: B. Input Cells and the Rule Table · p. 2
  • FIR estimates each cell’s conditional output distribution from observed counts, and the collection of these distributions forms the model’s rule table.The cell distribution is estimated from the number of times each output occurs for that cell.Source: B. Input Cells and the Rule Table · p. 2
  • The analysis assumes each cell appears at least once in the dataset.Source: B. Input Cells and the Rule Table · p. 2

C. Mask Selection Objectives

Entropy and MSE evaluate different properties of a mask: conditional output uncertainty and prediction error, respectively. Increasing mask size expands the cell space and raises data-sparsity concerns.

C. Mask Selection Objectives

  • Entropy depends on the full conditional output distribution, whereas MSE depends only on the conditional mean; their difference underlies the behavior analyzed in the paper.Entropy uses in-sample coarsened conditional entropy, while model quality is measured by out-of-sample prediction error.Source: C. Mask Selection Objectives · p. 3

D. Sparsity Parameter

  • As mask size grows, the number of cells increases exponentially, and limited data can leave many cells sparsely populated or empty.Estimating a distribution requires more data per cell than estimating a mean.Source: D. Sparsity Parameter · p. 3
  • Sparsity regimes are characterized using the largest candidate mask’s cell count: small λ indicates sparse rule tables, while large λ indicates well-populated cells.Source: D. Sparsity Parameter · p. 3

E. Resolution-Stochasticity Ratio

The resolution-stochasticity ratio compares intrinsic output variability with discretization resolution. Because discretization bounds coarsened conditional entropy, high stochasticity can erase entropy differences between masks.

E. Resolution-Stochasticity Ratio

  • The ratio ρ compares intrinsic variability with output-bin width, measuring how many discrete output values that variability spans.The theoretical analysis assumes Gaussian intrinsic stochasticity, while the rule table estimates conditional output distributions directly from cell counts without a distributional assumption.Source: E. Resolution-Stochasticity Ratio · p. 3
  • When ρ < 1, cells produce distinct output distributions; when ρ > 1, their output distributions are approximately uniform.The figure’s color indicates the dominant output bin in each input cell.Source: E. Resolution-Stochasticity Ratio · p. 3, Figure 1 · p. 3

III. THEORETICAL RESULTS

  • A minimal sufficient mask contains the variable–lag pairs carrying all predictive information about the output, with no redundant pair.Source: §III · p. 3
  • Coarsened conditional entropy is bounded above by log2 q, with equality only when the output bins are equiprobable.The intrinsic coarsened entropy is the conditional entropy under the minimal sufficient mask; unlike continuous entropy, it is bounded by the finite output alphabet.Source: §III · p. 3

A. Main Results

The results characterize when discretized entropy can distinguish the minimal input mask and when MSE can still predict without uniquely recovering it. Entropy selection is reliable in the resolved regime but loses discrimination as stochasticity approaches or exceeds the discretization ceiling.

1) Resolution Regime and Consistency:

  • For ρ<1, entropy masking is consistent as sample size grows, and omitting a relevant variable raises conditional entropy by pooling distinguishable cell-wise distributions.Distinct conditional means make relevant variables identifiable rather than increasing entropy under the minimal sufficient mask.Source: 1) Resolution Regime and Consistency · p. 4
  • For ρ>1, conditional entropy approaches log2 q for every mask, so entropy selection loses discriminative power even as sample size grows.The threshold ρc=1 is necessary under Gaussian stochasticity, but discrimination degrades continuously rather than failing abruptly at the boundary.Source: 1) Resolution Regime and Consistency · p. 4
  • Near ρ=1, entropy differences across masks shrink and discrimination degrades as their entropies approach the discretization ceiling.The ceiling regime has Hq(M) approaching log2 q for all masks.Source: 1) Resolution Regime and Consistency · p. 4
  • MSE masking is prediction-consistent for every ρ, but masks containing the minimal sufficient mask can share the same variability-floor error without uniquely recovering it.The result concerns prediction consistency, not structural identification.Source: 1) Resolution Regime and Consistency · p. 4

2) Closed-Form Excess Risk:

The excess prediction risk from entropy-driven over-selection is nonnegative and increases with mask complexity and stochasticity while decreasing with sample size. The bound assumes balanced cells; when occupancy is sparse, empty cells add bias and the stated bound does not apply.

2) Closed-Form Excess Risk:

  • The excess risk δ is nonnegative, increases with over-selection and stochasticity, and decreases as sample size grows.The theorem assumes balanced cells and evaluates prediction error on observations independent of those used to estimate cell means.Source: 2) Closed-Form Excess Risk · pp. 4–5
  • Each additional input variable multiplies the number of cells by qi, increasing per-cell estimation variance and the prediction cost of over-selection.The excess risk is the difference between the risks for the entropy-selected mask and the minimal sufficient mask.Source: 2) Closed-Form Excess Risk · p. 5
  • When λ<1, empty cells introduce bias absent from the variance decomposition, so the bound does not apply.Under Poisson occupancy, the expected per-cell variance inflation is λ/(1−e^−λ).Source: 2) Closed-Form Excess Risk · p. 5

3) Sample Complexity Scaling:

Reliable entropy-mask recovery requires enough data to overcome finite-sample bias toward larger masks and estimation variability. The sufficient sample requirement scales with the number of candidate input combinations and inversely with the squared effective entropy signal.

3) Sample Complexity Scaling:

  • The sufficient sample requirement scales with K(Mmax) and inversely with the square of the effective entropy signal Hq(ε).The bound also captures dependence on α, though its constant Cα is not tight.Source: 3) Sample Complexity Scaling · p. 5
  • Finite-sample entropy estimates favor larger masks because their plug-in bias grows with cell count, while variance falls as samples per cell increase.Additional variables do not reduce true conditional entropy for masks containing the minimal sufficient mask.Source: 3) Sample Complexity Scaling · p. 5
  • When λ<1, the discretization ceiling can prevent the entropy signal from overcoming estimation bias unless Ns scales with K(Mmax).The ceiling is Hq(ε)≤log2 q.Source: 3) Sample Complexity Scaling · p. 5
  • Because σε is unknown in practice, q∈{2,3} with regime diagnosis is offered as a guideline; data-adaptive q selection is left for future work.Non-Gaussian stochasticity shifts the regime boundary, while retaining the resolved/saturation structure.Source: 3) Sample Complexity Scaling · p. 6

B. Deterministic Regime and Mask Faithfulness

In the deterministic regime, entropy uniquely identifies the minimal structural mask, whereas perfect-prediction masks under MSE need not be unique. With stochasticity and a noisy proxy input, entropy can prefer the structural variable while MSE becomes indifferent as proxy noise vanishes.

B. Deterministic Regime and Mask Faithfulness

  • With no intrinsic stochasticity, the minimal sufficient mask uniquely minimizes entropy, while any mask achieving perfect prediction has zero MSE and need not be structurally minimal.Redundant or correlated inputs can reproduce the same partition; a longer lag may perfectly predict output in a periodic system.Source: B. Deterministic Regime and Mask… · p. 6
  • When ρ<1 and X2 is an imperfect proxy for X1, entropy prefers the true structural mask because conditioning on X2 leaves additional uncertainty.The proposition assumes X1=g(X2)+η with η independent of X2 and positive variance.Source: B. Deterministic Regime and Mask… · p. 6
  • As proxy noise σ²η approaches zero, MSE becomes indifferent between the proxy and structural masks, a distinction that matters when identifying causal drivers rather than predicting outcomes.The proxy-mask prediction error approaches that of the structural mask.Source: B. Deterministic Regime and Mask… · p. 6

C. Regime-Based Decision Principle

Mask selection should follow the goal and the data regime: entropy supports structure identification when reliable, while MSE targets prediction; permutation testing screens entropy contributions against chance.

C. Regime-Based Decision Principle

  • Entropy masking is recommended for explainability in the resolved regime, while MSE is the robust criterion when prediction alone is required.The excess prediction risk from using the entropy-selected mask instead of the MSE-selected mask is bounded by δ.Source: C. Regime-Based Decision Principle · p. 6
  • A permutation test shuffles input values across observations and compares observed entropy reduction with the permuted null distribution.A variable is retained when its entropy contribution is statistically significant (p < α).Source: C. Regime-Based Decision Principle · p. 6
  • The test provides practical evidence against over-selection when exact recovery of M* cannot be guaranteed.The passage points to the grid reliability application as an example.Source: C. Regime-Based Decision Principle · p. 6

IV. EXPERIMENTAL VALIDATION

The two-state Markov-chain experiments validate the theory’s prediction-risk bound and distinguish informative-mask recovery from exact recovery as variability changes.

Experimental setup

  • The validation uses a two-state symmetric Markov chain, all 31 subsets of five candidate lags, three bins, 3,000 samples, and 100 Monte Carlo runs per ε.The entropy estimator uses the Miller–Madow correction.Source: §IV · p. 6

Experimental results

  • 0.0004 to 0.012: excess risk δ grows monotonically and remains below the Theorem 2 bound, with the gap widening at high entropy.The comparison is shown in Fig. 2(b).Source: §IV · pp. 6–7, Figure 2 · p. 7
  • Within 1–2% at every variability level, MSE for MR tracks the variability floor ε(1−ε), confirming Theorem 1(3).The reported paired t-test gives p < 0.001.Source: §IV · p. 6, Figure 2 · p. 7
  • 100%: ME includes the true lag at every variability level, but exact entropy-mask recovery occurs only at the lowest level, reaching 12% at ε = 0.01.MR exact recovery declines from 93% to 11% as variability increases; entropy masking selects all five lags except at the lowest variability level.Source: Figure 2 · p. 7, §IV · p. 7

V. APPLICATION: DISTRIBUTION GRID RELIABILITY

The grid application analyzes utility reliability to identify investment-related inputs and evaluate prediction costs. It finds an association between AMI and CAIDI in a recovery state, while entropy-selected inputs incur greater prediction error than the MSE-selected mask.

V. APPLICATION: DISTRIBUTION GRID RELIABILITY

  • The analysis uses 2013–2024 data from 1,300 U.S. distribution utilities, with lagged CAIDI, revenue per customer, and AMI penetration as inputs.The application is described as a multi-variable coupled scenario with mixed discretization resolutions.Source: §V · p. 7

A. Regime Diagnosis

  • With ρ < 1 and λ ≈ 1.57, the data is in the resolved regime near the sparse boundary; the decision rule calls for entropy identification, permutation confirmation, and MSE prediction.Approximately 21% and 35% of candidate cells are empty under the stated Poisson occupancy estimates.Source: A. Regime Diagnosis · p. 7

B. Input Structure Mask Analysis

  • The entropy-selected mask includes AMI lags 2–3, income lag 1, and reliability lag 1; in the recovery state, high versus low AMI shifts poor-reliability probability from 0.410 to 0.389 and good-reliability probability from 0.171 to 0.197.The AMI permutation test reports p = 0.003; its shuffles preserve yearly marginal distributions while breaking the AMI–CAIDI association.Source: B. Input Structure Mask Analysis · p. 7, Figure 3 · p. 8

C. Prediction

  • 79%: substituting the entropy-selected ME for the MSE-selected MR incurs excess prediction error in the grid application.The empirical excess is approximately 23,315, below the Theorem 2 bound of 31,806; Fig. 4 compares observed and predicted CAIDI for the two masks.Source: C. Prediction · pp. 7–8, Figure 4 · p. 8

VI. CONCLUSION

  • The paper identifies two limitations: the excess-risk bound treats mask size as fixed, and the threshold ρc assumes Gaussian stochasticity.When mask size varies, Jensen’s inequality implies that the bound understates expected excess risk; the regime structure itself does not depend on the specific distribution.Source: §VI · p. 8
Loading 2609.03074v1…