ALICE: In-context, Zero-shot, Mutual Information Estimation

1 Introduction

• Finite-sample MI estimation is difficult because it depends on the full joint density, is unbounded, and is dominated by rare high-density events; guarantees are fragile in high dimensions.Existing sample-based estimators also require fitting anew for each distribution, with computational and tuning costs and risk of failure.Source: §1 · p. 1
• A KL identity expresses mutual information as an integral of squared velocity differences, reducing estimation to evaluating the joint and block-conditional velocity fields.ALICE predicts fields from context samples of an unseen distribution and uses three masked queries to obtain the fields needed for MI estimation.Source: §1 · p. 2
• ALICE is a single model trained once on synthetic distributions that predicts rectified-flow velocity fields from samples, without per-distribution training.Its inference supports any joint width and context length.Source: §1 · p. 2
• ALICE is competitive from contexts of a few hundred samples to tens of thousands, while existing neural estimators need hundreds of thousands of samples per distribution for high accuracy.The existing estimators degrade sharply below that sample scale, whereas biology and neuroscience datasets often have at most a few thousand pairs.Source: §1 · p. 2
• With 1,000 samples, ALICE is the most accurate estimator by a factor of at least two, and its zero-shot accuracy matches estimators trained per distribution.In the zero-shot setting, it is also competitive with trained neural estimators at their full budget.Source: §1 · p. 2

2.1 Mutual information estimation

• MI is estimated by combining joint and block-conditional velocity fields under a shared perturbation, then Monte Carlo averaging their aligned block-wise differences.The three evaluations use one joint and two block-conditional fields.Source: Figure 1 · p. 3, §2.1 · pp. 3–4

2.2 The in-context velocity field

• A clean sample context determines which distribution-specific velocity field the model represents, while the query specifies where that field is evaluated.The model implements this context-conditioned mapping with an attention-based transformer.Source: §2.2 · p. 4

2.3 Architecture

• Scalar tokens and induced latents let one parameter set handle variable widths and context sizes, with induced attention cost linear in context length.A context-derived relation graph carries cross-coordinate dependence; queries are processed independently conditional on shared context states, which can be cached.Source: §2.3 · pp. 4–5

2.4 Pretraining

• Pretraining samples synthetic joint distributions and random noising masks, regressing velocity targets without requiring MI labels.The corpus varies dependence structure, conditional behavior, and support geometry; masks cover joint, block-conditional, and partial-observation fields.Source: §2.4 · p. 5

Protocol

• The benchmark covers 40 tasks with joint widths from 2 to 100 and a closed-form ground-truth MI; all estimators receive budgets of 1k, 5k, or 10k samples.Neural estimators are trained and tuned separately on each distribution, while Alice and InfoAtlas are evaluated zero-shot on the same contexts.Source: §3 · p. 6

Results

• 0.092, 0.063, and 0.060 nats: Alice Base has the lowest 40-task MAE at 1k, 5k, and 10k samples, versus 0.195 (CCA), 0.070 (MINDE), and 0.065 (MINDE), respectively.At 1k samples, Alice Base outperforms every competitor by a factor of at least two.Source: §3 · p. 6
• At 1k samples, Alice has the lowest MAE in all four task groups; at 10k, it leads in the three Gaussian-based groups.The Small model has 19M parameters versus 85M for Base; on the seven width-50 and width-100 tasks, Base improves from 0.153 nats at 1k to 0.086 at 10k while Small rises from 0.197 to 0.241.Source: §3 · p. 6

Small-data regime

• At 1k samples, MINDE has 0.353 nats MAE, InfoNCE 0.887, and D-V 1.229; NWJ diverges at 1k and remains at 1.261 nats at 5k.The reported comparison says trained estimators need the increase from 1k to 5k or 10k samples to become practically usable.Source: §3 · p. 6

Inference compute

• At 8192 samples, Alice Base reaches 0.060 nats MAE for 1.6 · 10^13 FLOPs per task, versus InfoAtlas at 0.274 nats for 1.9 · 10^13 FLOPs.The figure compares inference FLOPs per task and seed against MAE on the 40-task benchmark.Source: Figure 3 · p. 7, §3 · p. 7

4.1 Analysis of multivariate single-cell signaling responses

• For NF-KB, Alice estimates dose-specific divergences from 536–1,307 cell samples per dose using 1,024-cell contexts, then derives MI, capacity, and pairwise discrimination.The analysis uses 15,632 cells measured at 11 TNF-α concentrations across 40 frames.Source: §4.1 · p. 7
• The NF-KB trajectory reaches about 1.3 bits of capacity versus 1.2 bits for the best single frame, and improves discrimination of high doses from 0.56 to 0.67 PCD.The prefix capacity saturates by frame 12; the trajectory encodes dose in the timing and amplitude of the first response peak.Source: §4.1 · p. 8

4.2 Promoter Identification

• For promoter identification, Alice scores each L = 4 or 6 base window separately using 1,024 context pairs and 1,024 held-out pairs.A fixed injective embedding preserves MI, and Alice handles the one-dimensional label and L-dimensional window natively.Source: §4.2 · p. 8
• Alice’s promoter-label MI profile peaks at 30–32 bases upstream of the TSS, within the documented tata-box band, and also shows a smaller TSS peak.Info-SEDD locates the tata-box with a single peak but has a substantially higher bias floor.Source: §4.2 · pp. 8–9

4.3 Brain Region Activity Patterns

• Alice estimates O-information separately for each of 72 sessions from 128-flash contexts, where each flash supplies six area-level variables.Sessions contain about 150–200 independent flashes per flash type, too few for per-session estimator training.Source: §4.3 · p. 9
• In novel-image sessions, visual-area redundancy peaks at 100–150 ms; novelty raises redundancy in 25 of 27 mice for change flashes and 28 of 36 for repeats.The change-flash difference is absent in the first window and not seen at the peak in familiar-image sessions.Source: §4.3 · p. 9

5 Conclusion and Limitations

• Alice is a single Transformer trained once to estimate an unseen distribution’s joint and block-conditional rectified-flow velocity fields from samples.A known identity then uses these fields to estimate MI.Source: §5 · p. 10
• At matched budgets of 1k, 5k, and 10k samples, Alice has the lowest error on Beyond Normal; at 1k, its error is lower by at least a factor of two.Every competitor is trained and tuned on each distribution.Source: §5 · p. 10
• In three scientific applications, Alice reproduces dedicated-estimator findings in the small-data regime typical of biology and neuroscience, using a few forward passes per estimate.The data types and distributions were absent from its synthetic training corpus.Source: §5 · p. 10
• The implementation is research code that has not been thoroughly optimized; larger models, higher-dimensional training data, and longer training contexts might improve benchmark results.These are proposed directions, not reported improvements.Source: §5 · p. 10

A Related Work

• Diffusion-based MI estimators express information through score, denoiser, velocity, or bridge differences, but train a network per distribution; Alice amortizes that step.InfoAtlas is the closest amortized estimator: Alice uses one attention-conditioned network, supports unseen joint widths, and uses an exact velocity-difference identity rather than a variational bound.Source: A Related Work · p. 16

B Mutual Information as a Velocity-Difference Integral

• The appendix derives the velocity-form KL identity and its joint-versus-conditional form, which underlies the estimator.The stated results connect to the I-MMSE relation and prior velocity-form relative-entropy work.Source: B Mutual Information as a… · p. 16

B.1 Velocity and score

• Rectified flow interpolates clean samples to Gaussian noise, and its velocity field is defined as a conditional expectation along that path.The appendix states smoothness, finite-second-moment, and Gaussian-tail assumptions for the derivation.Source: §B.1 · pp. 16–17

B.2 KL divergence in velocity form

• The velocity-form KL identity uses a time-weighted squared velocity difference; its integrand is bounded at every time, motivating velocity prediction.The alternative score-difference form has a weight that diverges as time approaches zero.Source: §B.2 · p. 18

B.3 Mutual information

• The joint-versus-conditional MI identity requires the fields to use shared noise; independent perturbations break the orthogonality argument.The proof relates the MI integral to projections of the conditional and joint velocity fields.Source: §B.3 · p. 18

B.4 Conditional variant: a discrete input as clean evidence 

• For discrete inputs, MI can be estimated by comparing the response velocity conditioned on each input value with the response marginal velocity.Only the response is noised; a joint context supplies the conditional field, while a shuffled context supplies the marginal field.Source: §B.4 · p. 19

C Estimation algorithm

• The reference estimation algorithm splits context from evaluation data, caches one context encoding, and makes three shared-noise velocity queries per point and time.It averages weighted block-wise velocity differences.Source: C Estimation algorithm · p. 19

D Alice Details

• The Alice details section specifies token layout, time conditioning, relation-graph attention, the induced context bottleneck, boundary parameterization, and model family.Figure 7 depicts a forward pass through these components.Source: D Alice Details · p. 19

Size-independent representation

• The model uses shared projections and coordinate-shared parameters, so the same weights support any joint width; validity masking makes padded context equivalent to truncated context.The context is encoded once and cached, and queries are evaluated independently through cross-attention.Source: §D.1 · pp. 19–20

Context-derived relation graph

• A graph computed from clean context controls cross-coordinate information exchange, using learned nonlinear feature covariances to describe pairwise dependence.Its learned, signed gated edges permit additive or subtractive messages, while shared maps keep parameter count independent of joint width.Source: §D.1 · pp. 20–21

Induced context bottleneck

• Two cross-attention layers compress each coordinate’s context tokens into K induced latents, and deeper blocks operate on those latents at cost independent of context length.Queries combine a latent global summary with context-derived local detail.Source: §D.1 · pp. 21–22, Figure 7 · p. 21

Boundary parameterization

• Because the MI estimator weights squared velocity differences by (1 − t)/t, Alice enforces the boundary condition v0(z) = z by construction.The passage notes that violations would create positive bias where the weight is largest.Source: §D.1 · p. 22

Model family

• Alice’s parameter count is independent of joint width and context length, and a trained checkpoint can be used at any joint width without modification.Model sizes vary backbone capacity while sharing the induced-latent count, relation-feature width, and time-feature resolution.Source: §D.1 · p. 22

Implementation

• The section presents the reference implementation of Alice.Source: E Training and Implementation Details · p. 22

Induced context bottleneck

• Figure 9 describes the induced context bottleneck: a shared learned bank queries each coordinate’s context tokens twice to produce K context-specific latent vectors.Source: Figure 9 · p. 23

Model variants

• Table 1 lists configuration values and parameter counts for the two Alice variants.Source: Table 1 · p. 23

E.1 The training corpus

• The corpus contains 70,000 episodes at each of 14 joint widths, drawn from copula mixtures, latent warps, manifolds, and nonparametric regressions.Additional corpus portions include same-sign factor covariances and a plane-rotation warp.Source: §E.1 · pp. 23–24

E.2 Batch construction

• Batches use a shared sampled context length, disjoint context and query samples, and marginal rank normalization fitted on the context and applied to queries.Batches are dimension-homogeneous; variable context length is implemented by truncation or a validity mask.Source: §E.2 · p. 24

E.3 Objective, noising indicators, optimizer

• Training uses masked velocity MSE on noised coordinates, with targets available from the sampled noise and time, so no ground-truth density or MI labels are needed.The indicator mixture includes all-noised, block-conditional, and random partial-observation patterns.Source: §E.3 · p. 24
• Context lengths are sampled from 128 up to the training window, with a 1024-sample first phase followed by a 2048-sample phase.The two phases run for 500,000 and 60,000 steps, respectively.Source: §E.3 · pp. 24–25

• Alice-Small training took 2 days and 22 hours on 2x H200 GPUs, compared with 5 days and 10 hours for Alice-Base.The reported rates were about 8,674 and 4,300 optimizer steps per hour, respectively.Source: §E.4 · p. 25
• InfoAtlas training is reported, to the authors’ understanding, to require 2 weeks on 16x H800 GPUs.Source: §E.4 · p. 25
• All experiments used a single H200 GPU for inference.Source: §E.4 · p. 25

• Alice Base has the lowest aggregate MAE at each budget; at 1k samples, all neural estimators exceed 0.24 nats and CCA, the best classic estimator, has 0.195.Source: F Ground-truth benchmark: details and… · p. 25, Table 2 · p. 26

Joint width

• Alice Base’s wide-task MAE improves from 0.153 to 0.086 nats with more context, while Alice Small degrades from 0.197 to 0.241.InfoAtlas reaches 1.0 nats on wide tasks, where it uses a sliced fallback; Table 3 includes 33 tasks of width at most 10 and seven tasks of width 50 or 100.Source: F Ground-truth benchmark: details and… · p. 25, Table 3 · p. 26

Error analysis

• At 10,000 samples, 35 of 40 tasks are within 0.1 nats of ground truth for Alice Base, with mean signed error −0.005 nats.Its largest errors include underestimation on spiral embeddings and overestimation on dense multinormal tasks.Source: F Ground-truth benchmark: details and… · p. 25, Table 4 · p. 27
• Alice Small shares the spiral underestimation and dense-multinormal overestimation patterns, and also overestimates three sparse width-50 tasks by 0.24 nats each.Source: F Ground-truth benchmark: details and… · p. 26

• The study has no ground-truth MI estimates, so its results are compared with the biological conclusions of Jetka et al. (2019).Source: G Single-cell signaling responses:… · p. 28

Estimand and metrics

• The analysis estimates each dose’s divergence from the output mixture, then uses Blahut–Arimoto to compute capacity from the per-dose divergences.Conditional fields are cached, while the mixture field is re-estimated as the input distribution is updated.Source: G Single-cell signaling responses:… · p. 28

Results in full

• Single-frame capacity peaks at 1.12–1.16 bits at minutes 15–21, falls to 0.04 bits at minute 66, and rises to 0.40 bits at minute 93.The peak is about 1 bit in Jetka et al. (2019).Source: G Single-cell signaling responses:… · p. 28
• Prefix capacity reaches 1.34 bits with the first nine frames, exceeding the best single-frame capacity, while trajectory capacity is about 1.05 bits versus 1.3 bits in Jetka et al. (2019).Source: G Single-cell signaling responses:… · p. 28

Dataset

• The dataset contains 1,497 promoter and 1,497 non-promoter sequences, each 251 bases, making the promoter label uniform and bounding each window's MI by ln 2 nats.The promoter sequences span positions −200 to +50 around the annotated TSS.Source: H Promoter identification: technical… · p. 29

Relation to previous work

• Earlier approaches localized elements through drops in classification accuracy after random-base substitution; Alice instead profiles MI, with irrelevant windows near zero and motif-overlapping windows high.Foresti et al. (2026) used a discrete-diffusion MI estimator to reproduce the earlier protocol.Source: H Promoter identification: technical… · pp. 29–30

Encoding

• Each base is mapped through a fixed injective embedding; dither removes ties in Gaussian-copula normalization, and the reported results use the minimal width K=1.Injectivity preserves MI exactly for every K.Source: H Promoter identification: technical… · p. 30

Results

• Both window lengths peak inside the tata-box band: L=4 reaches 0.31 nats at −30; L=6 reaches 0.34 nats at −32.The analysis covers 248 windows for L=4 and 246 for L=6.Source: H Promoter identification: technical… · p. 30

Discussion

• The profile resolves the tata-box at −30 and a TSS-straddling initiator; top tata-box 6-mers each occur in about 4% of promoters versus about 0.4% for the most frequent non-promoter 6-mer.The initiator is characterized by a pyrimidine/purine pair around the TSS.Source: H Promoter identification: technical… · p. 30

Estimator

• The estimator defines Ω-info as TC−DTC and expresses both terms as time integrals of squared score differences; shared noised coordinates let velocity differences represent score differences.For N = 2, the conditional-field expression is the MI estimator; common random draws are used across terms, requiring 2N + 1 velocity evaluations per Monte Carlo row.Source: I Brain Region Activity Patterns:… · pp. 30–31

Data and preprocessing

• Neuropixels recordings covered six visual areas across paired familiar- and novel-image sessions, with five 50-ms windows constructed from post-flash spike counts.Each flash began after 250 ms of image presentation and 500 ms of gray screen; the per-area dimension was 50/s, with s = 50 ms in this analysis.Source: I Brain Region Activity Patterns:… · p. 31
• Analyzed change flashes were image changes; non-change flashes were non-omitted flashes at trial positions 4–10, and the released reward and lick filters had no effect.Positions 1–3 were excluded because responses to repeated images decrease before leveling off from the fourth repeat.Source: I Brain Region Activity Patterns:… · p. 31

Protocol details

• Correlated flashes make nominal row counts misleading: effective draws were 185 of 251 change rows and 146–205 of 1,447 non-change rows.Estimates used 128 context rows, up to 512 evaluation rows, 64 time draws per evaluation row, and five context draws; contiguous runs of 32 rows were assigned to context or evaluation.Source: I Brain Region Activity Patterns:… · pp. 31–32
• Randomly assigning context rows shifted estimates by 0.01 nats, whereas cross-session contexts raised peak medians and largely erased familiar–novel differences.At the peak, cross-session contexts changed non-change medians from 0.49 to 0.75 nats and change medians from 0.70 to 0.83 nats.Source: I Brain Region Activity Patterns:… · p. 32

Per-window results

• In novel-image sessions, Ω-info was positive in every window for every mouse; maxima fell at 100–150 ms for 78% of change-flash mice and 47% of non-change-flash mice.Figure 11 complements these results with familiar-image sessions and within- and between-session contrasts.Source: I Brain Region Activity Patterns:… · p. 32, Table 6 · p. 32, Figure 11 · p. 33
• Familiar-image sessions showed flat change-flash profiles and late-rising non-change profiles, while familiar and novel sessions already differed by −0.16 nats before the visual response.Figure 11 shows the session and flash-type contrasts; Table 7 reports paired contrasts and signed-rank tests.Source: I Brain Region Activity Patterns:… · p. 32, Figure 11 · p. 33, Table 7 · p. 33
• At peak windows, mouse identity accounted for 38% of estimate variance, experience level 23%, flash type 8%, and context-draw variability 10%.These variance shares use all 63 sessions with both flash types and five context draws each.Source: I Brain Region Activity Patterns:… · p. 32, Table 8 · p. 34