Source-linked AI summary
SAIVE: Selecting AI Valuable Entities
Inna Voloshchuk, Hayden Jananthan, Jeremy Kepner
TL;DR
Large data lakes can remain unanalyzed by AI because their volume is prohibitive, while selecting useful fields is important for aggregate analysis. The paper derives a closed-form approximation of the Zipf-Mandelbrot exponent and uses its range as a cheap, expert-free column filter. Synthetic results support separating power-law from non-power-law columns, with the proposed range dropping 92.6% of light-tailed columns.
Problem
Data volume can make AI analysis of data lakes prohibitive, motivating cheaper ways to select columns for analysis.
Source: §I · p. 1Method
The paper models value frequencies with a Zipf-Mandelbrot distribution and uses an estimated exponent α, filtered against the empirical range [1.5, 3], to screen columns.
Source: §III · p. 2, A. Approximating α · p. 3Results
92.6% of light-tailed columns were dropped by the α̂_m ∈ [1.5, 3] filter, and synthetic results support separating power-law from non-power-law columns.
Source: §V · p. 4Takeaways & Limitations
The approximate α formula may provide a cheap way to remove non-informative data and reduce fields, labor, and compute needed for downstream analysis.
Source: §VI · p. 5The estimator assumes i.i.d. value frequencies, although a real table's frequencies are dependent when conditioned on its fixed record total.
Source: §V · p. 4
Abstract
from arXiv · showhide
Data lakes store large amounts of telemetry, with logs from network sensors, hosts, and applications containing possibly hundreds of fields for every event. Large enterprises are then left with data lakes that cannot be analyzed efficiently with AI. Aggregate analysis looks at persistent shifts in behavior over time. Many of the fields and columns in data lakes are not useful as they do not contain information that is sufficiently diverse or concentrated to support AI analysis. SAIVE is a simple method for examining a few rows in a large table and applies a histogram of histograms filtering criterion to select the fields that for AI analysis is more likely to yield useful results. This paper provides a principled foundation for the SAIVE heuristics by assuming of a Zipf-Mandelbrot power-law distribution of the underlying data. Constraining the Zipf-Mandelbrot exponent alpha to a reasonable range provides a a practical, cheap, expert-free filter for selecting AI valuable entities in large data sets.
I. INTRODUCTION
Large telemetry data lakes are often left unanalyzed because their scale makes AI analysis difficult, despite interest in finding persistent behavioral shifts. SAIVE builds on field-selection heuristics and seeks a principled basis for filtering columns using a constrained power-law model.
I. INTRODUCTION
- Telemetry volume, velocity, and variety make AI analysis of subtle behavioral changes across long periods and many actors complex.The paper notes that organizations seek cheap statistics to reduce the amount of data requiring analysis.Source: §I · p. 1
- Prior scalable-analysis heuristics for selecting useful fields form the basis of SAIVE, which examines column-level criteria.The supplied introduction names counts of nonempty entries, distinct values, and the most popular value, but does not specify the observed criteria.Source: §I · p. 1
- The paper aims to ground SAIVE heuristics in a power-law model by constraining the Zipf-Mandelbrot slope parameter α to a reasonable range.The motivation cites prior work modeling relevant quantities with a modified Zipf-Mandelbrot distribution,.Source: §I · p. 1
- The paper introduces associative-array preliminaries, derives a closed-form approximation of α, evaluates it on synthetic data, and discusses results and limitations.The roadmap places these topics in Sections II through V, followed by a conclusion in Section VI.Source: §I · p. 1
II. ASSOCIATIVE ARRAYS
The associative-array construction turns each table column into counts of value frequencies, then into a histogram of those counts. This histogram of histograms is the distribution modeled in the subsequent analysis.
II. ASSOCIATIVE ARRAYS
- For each column, the construction makes an N × N_unique associative array A_c from the column's unique values.The passage specifies one associative array per column.Source: §II · p. 1
- Each row of A_c has exactly one nonzero entry, and summing rows yields the frequency count for each distinct column value.The resulting 1 × N_unique vector is a histogram of values in that column.Source: §II · p. 1
- A further array D_c groups values by their count, producing n_c, a histogram of histograms.Each n_c entry counts how many distinct values occurred exactly d times; normalizing n_c gives probabilities p_c(d).Source: §II · p. 2
III. ZIPF-MANDELBROT DISTRIBUTION
The paper models column value frequencies as independent Zipf-Mandelbrot samples and treats the exponent α as a potential filter for columns. It motivates an acceptable α range empirically and relates α to the distribution's slope.
III. ZIPF-MANDELBROT DISTRIBUTION
- Column value frequencies d are modeled as i.i.d. samples from a Zipf-Mandelbrot distribution on {1, 2, ..., N_max}.The model is applied to actual frequency counts, not category rankings; the paper frames useful columns as having both repeated information and diversity.Source: §III · p. 2
- The model is written x_1, ..., x_k i.i.d. ∼ Zipf-Mandelbrot(α, δ), with α as the distribution slope and δ fitting the head at small sampled values.Small α near 1 allows very large x_i values, while large α makes large x_i values rare; heavy-tailed means and variances may be undefined.Source: §III · p. 2
- The paper proposes estimated α as a potential filter, retaining columns within the empirically motivated range 1.5 to 3.The range is motivated by prior empirical work finding many quantities fit a modified Zipf-Mandelbrot distribution with α between 1.5 and 3.Source: §III · p. 2
A. Approximating α
SAIVE approximates the Zipf–Mandelbrot exponent from column frequency data, fitting the upper tail and using a fixed cutoff to support automated filtering. Estimated α serves as a proxy for whether a column is informative within the chosen range.
A. Approximating α
- Replacing the likelihood normalization sum with an integral and taking the untruncated limit yields an approximation to the discrete estimator for α.The estimator begins with maximum likelihood under the Zipf–Mandelbrot model.Source: A. Approximating α · p. 2
- Because the smallest observations may not follow a power law, the estimator fits the upper tail using the largest m observations.The Mandelbrot offset matters most at the lower end, where sampled observations may also be unreliable.Source: A. Approximating α · p. 2
- Choosing a cutoff m separately for every column requires repeated Hill plots and statistician judgment, so the procedure uses one fixed m.A universal m is a practical compromise for automatic selection without expert intervention.Source: A. Approximating α · pp. 2–3
- The column model treats categorical values as meaningless labels and estimates α from their frequencies.Prior work reports typical α values between 1.5 and 3 for real-world datasets.Source: Figure 1 · p. 3
- Estimated α acts as a proxy for informativeness: columns outside the empirically chosen range [1.5, 3] can be removed before downstream analysis.The range is motivated by results in.Source: A. Approximating α · p. 3
IV. SYNTHETIC DATA EXPERIMENT
The experiment evaluates the α filter on synthetic columns generated from controlled distributions, using a fixed sample size and cutoff. It varies distribution families and parameters, then summarizes estimated exponents and filter acceptance across repeated trials.
IV. SYNTHETIC DATA EXPERIMENT
- The synthetic experiment tests whether the estimator-based filter removes columns generated by processes without power-law exponents.Each trial draws counts until their total reaches N, then evaluates the estimated α.Source: §IV · p. 3
- Trials use N = 8192 and δ = 1; the offset was chosen to separate edge cases, with δ above approximately 0.27 needed to place the all-unique case above the α threshold of 3.N matches the row count of cyber-log tables in [8].Source: §IV · p. 3, Figure 2 · p. 4
- The generating families include heavy-tailed Zipf, Zipf–Mandelbrot, and log-normal distributions; light-tailed normal, exponential, gamma, and uniform distributions; and two deterministic edge cases.The edge cases are columns with all unique values or a single repeated value.Source: §IV · p. 3
- Light-tailed and log-normal parameters are sampled log-uniformly over a broad range, while power-law α is sampled from 1.25 to 3.5 and δ from −1 to 3.5.The narrower power-law ranges reflect expected bounds for network quantities; values of α well outside them would commonly be filtered out.Source: §IV · p. 3
- The evaluation fixes m = 100, although the appropriate cutoff can vary by sample or may not exist for some distributions.The paper identifies testing different m values as a possible extension.Source: §IV · p. 3, Figure 2 · p. 4
- Across 10,000 trials per distribution, the experiment estimates α from the largest m observations while retaining the Mandelbrot offset.The estimator is an approximation from fitted through order statistics; Figure 2 summarizes the estimates, with values capped at 10 for display.Source: §IV · pp. 3–4, Figure 2 · p. 4
V. RESULTS
On controlled synthetic distributions, the approximate exponent estimator behaves differently for power-law and non-power-law columns, supporting a filtering rule. Results also identify an i.i.d. modeling approximation and the need to test the filter on real cyber logs.
V. RESULTS
- The approximate estimator behaves differently on power-law and non-power-law columns, suggesting an α-based rule can separate them.The authors suggest this may remove many columns that do not represent network quantities of interest.Source: §V · p. 4
- For Zipf and Zipf-Mandelbrot distributions, the estimated exponent is meaningful; heavy-tailed log-normal values show a large spread because they are not power-law distributed.Source: §V · p. 4
- 92.6% of light-tailed columns fall outside the α̂_m ∈ [1.5, 3] filter; their estimates also spread widely across sampled parameters.Three percent of estimates above 10 were truncated for Figure 2.Source: §V · p. 4
- The estimator assumes i.i.d. entries in d, although real table frequencies are dependent because they sum to the fixed row count N.The authors describe this as an approximation and note Poissonization as a way to remove the dependency.Source: §V · p. 4
- The controlled synthetic experiment has not yet tested the criterion on real cyber logs or whether retained fields are useful for aggregate analysis.Source: §V · p. 4
VI. CONCLUSION
The conclusion presents a closed-form approximation of the Zipf-Mandelbrot exponent as a way to filter columns for aggregate analysis. The proposed range is empirically motivated, and the authors characterize the approach as a cheap way to reduce downstream analysis effort.
VI. CONCLUSION
- When column names and value meanings are unknown, tables can be represented as associative arrays over categories and their frequencies.Source: §VI · p. 5
- The authors model value-frequency distributions with a Zipf-Mandelbrot model and derive a closed-form approximation of exponent α for identifying network quantities of interest.Source: §VI · p. 5
- Columns with α̂_m outside [1.5, 3] are discarded by the proposed filter.Source: §VI · p. 5
- The approach uses cheap-to-compute statistics to remove non-informative data, reducing fields for later analysis and the associated labor and compute.The authors qualify the results as being on controlled distributions.Source: §VI · p. 5