PaperlayerView exact revision on arXiv

Source-linked AI summary

Finite Probes Suffice: Identifiability and Universality for Weight-Space Learning

Soutrik Sarangi, Yonatan Sverdlov, Adir Dayan, Haggai Maron, Nadav Dym

arXiv:2609.33901v2cs.LGcs.AI

TL;DR

Learning neural functionals has developed through parameter-based and probe-based methods, but probe methods have lacked theoretical guarantees and finite probes may seem to reveal only partial function information. This paper proves identification and universality results for probing, including more informative hidden-layer representations, and introduces HIDDENPROBE. HIDDENPROBE consistently improves over ProbeGen and strong weight-space baselines across diverse tasks, with state-of-the-art performance in several.

  • Problem

    Probe-based methods have shown empirical success, but their theoretical guarantees are limited, and finite evaluations provide only partial information about a network’s realized function.

    Source: §1 · pp. 1–2

  • Method

    The paper proves finite-probe identification and universality results, studies hidden-layer traces, and proposes HIDDENPROBE to learn network properties from them.

    Source: §1 · p. 2, §1.2 · p. 3

  • Results

    HIDDENPROBE consistently improves over ProbeGen and strong weight-space baselines across diverse tasks, achieving state-of-the-art performance in several.

    Source: §8 · p. 10

  • Takeaways & Limitations

    Hidden-layer probing is an effective approach across different architectures and tasks, and can match or surpass ProbeGen with substantially fewer probes, often half as many.

    Source: §8 · p. 10

    The hidden-probe identifiability result requires non-polynomial, injective activations and holds only outside an exceptional Lebesgue-null parameter set.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Learning properties of neural networks has recently attracted growing interest, with existing approaches operating either directly on network parameters or through probe-based representations of network behavior. While probing methods have shown strong empirical performance, their theoretical foundations remain limited. In this work, we study when finite probe-based representations are sufficient for learning neural functionals. We establish general identification and universality results for probing, and show that using intermediate hidden representations can provide significantly more informative representations than relying only on final outputs. Motivated by these results, we introduce HIDDENPROBE, a simple architecture for learning from hidden probe responses. Across a range of neural functional benchmarks, including both MLPs and Transformers, HIDDENPROBE consistently improves over existing probing methods and achieves state-of-the-art performance. Our code is publicly available on GitHub.

1 INTRODUCTION

Neural functionals map networks to properties, but weight-space methods must handle parameter symmetries, while probe-based methods lack comparable theoretical guarantees. The paper studies finite-probe identification and universality, then motivates hidden-layer probing as a more informative representation and introduces HIDDENPROBE.

1 INTRODUCTION

  • Weight-space learning processes network parameters directly, but distinct parameterizations can represent the same function, often requiring more complex symmetry-aware architectures.Probe-based methods instead represent networks through evaluations on selected inputs, but their strong empirical performance has lacked a theoretical justification.Source: §1 · p. 1

1.1 PREVIOUS WORK

  • Prior work had not systematically studied property prediction from complete activation traces across all hidden layers, motivating analysis of identification and universality from final outputs and hidden traces.Existing approaches include weight-space methods, final-output probing, and selected hidden-layer probes.Source: §1.1 · pp. 2–3

1.2 PRELIMINARIES

  • Final-layer probing learns network properties from outputs at selected inputs; hidden-layer probing retains intermediate computations on those same probes.A neural functional maps a network to a vector-valued property, and ProbeGen is a representative final-layer method with learned probes.Source: §1 · p. 1, §1.2 · p. 3

1.3 OVERVIEW

  • For analytic networks, the paper gives a 2q + 1 probe identification and universality result, while hidden-layer probing lowers sufficient probe counts for MLPs and self-attention and motivates HIDDENPROBE.The paper reports state-of-the-art results for HIDDENPROBE on several standard neural-functional datasets.Source: §1 · p. 2, §1.3 · pp. 3–4

2 IDENTIFYING ANALYTIC NEURAL NETWORKS FROM PROBE EVALUATIONS

For analytic parametric functions, generic final-layer evaluations at 2q + 1 probes uniquely identify the realized function. This finite-sample result underpins probe-based universality, though the proof discussion notes that the guarantee does not extend to ReLU networks.

2 IDENTIFYING ANALYTIC NEURAL NETWORKS FROM PROBE EVALUATIONS

  • 2q + 1 generic final-layer probes identify analytic parametric functions: matching evaluations at every probe is equivalent to equality of the realized functions on Rd.The theorem applies to analytic families, including MLPs and Transformers when their activations are analytic.Source: §2 · p. 4
  • The finite-probe identification argument is specific to analytic functions; the proof discussion gives ReLU networks as a case where no finite probe set suffices.The paper relates the analytic result to finite-dimensional parameterization and a finite-witness theorem.Source: §2 · p. 4
  • The finite identification result implies that probe-based models with 2q + 1 probes can approximate all neural functionals, without exhaustive input-domain sampling.The paper notes that this sufficient count remains much larger than typical practical probe counts, motivating hidden probes.Source: §2 · p. 4

3 PROBING MLPS

For analytic MLPs, hidden-layer probe activations identify the realized function with a probe count governed by maximal width rather than total parameter count. ReLU networks cannot be exactly identified from any finite probe set, but sampled probes can provide approximate identification in L2(µ), with fewer samples when hidden activations are used.

3.1 HIDDEN PROBES PROVIDE USEFUL INFORMATION:

  • For analytic MLPs, N > dmax generic hidden probes identify parameters, and therefore the realized function, for almost every parameter pair outside an exceptional Lebesgue-null set.The activation must be analytic, injective, and non-polynomial; the exceptional set is a lower-dimensional zero set with empty interior.Source: §3.1 · pp. 4–5
  • For analytic activations, final-layer identification instead requires a number of probes proportional to the MLP parameter dimension qD.The hidden-probe bound depends on maximal width, and examples in the appendix match the dmax bound; final-layer probe tuples can fail for every N ≤ 2qD − 2.Source: §3.1 · pp. 4–5

3.2 IDENTIFYING RELU NETWORKS FROM PROBE EVALUATIONS

  • No finite probe set identifies all ReLU networks: distinct three-hidden-neuron networks can agree on every probe.A nonzero ReLU hat function can be constructed with support avoiding any finite probe set.Source: §3.2 · p. 5, Figure 2 · p. 5
  • With probability at least 1 − δ, matching N1 = C1·Lq ln(q) final-layer samples or N2 = C2·q ln(q) hidden-layer samples yields L2(µ) error at most ϵ for ReLU networks.The guarantee assumes a compactly supported input distribution and compact parameter set; hidden activations can reduce the sample count, especially for deeper networks.Source: §3.2 · p. 5

4 IDENTIFYING MULTI-HEAD ATTENTION FROM PROBES

The section studies identifying multi-layer softmax attention blocks from probes, including their complete hidden activations. Structured probes exploit attention’s softmax structure to identify hidden representations with O(dh) probes under stated conditions.

4 IDENTIFYING MULTI-HEAD ATTENTION FROM PROBES

  • Generic final-layer probes also identify the attention stack’s output function, but require 1 + 2q probes.The section notes that these parameter-based counts are O(d²) per standard MHA layer, while structured probing uses O(dh) probes.Source: §4 · p. 6
  • N = d(2hmax + 1) + 1 structured probes identify every layer’s hidden outputs for almost every target, using probes of token length at most max{3, 1 + d/hmax}.The theorem provides an open set of probe tuples for the stated identification guarantee.Source: §4 · p. 6
  • RECOVERMHA first uses one-token probes to determine W, then an affinely independent-token probe to recover attention weights and softmax log-ratios to recover M0.For the single-layer, single-head case, the algorithm uses d + 1 probes.Source: §4 · p. 6

5 HIDDENPROBE: LEARNING USING HIDDEN NEURAL ACTIVATIONS

HIDDENPROBE learns network properties from learned probes evaluated across hidden layers, rather than relying only on final outputs. Its MLP and Transformer instantiations aggregate responses at each layer and then combine them to predict the target property.

5 HIDDENPROBE: LEARNING USING HIDDEN NEURAL ACTIVATIONS

  • HIDDENPROBE uses learned probes and all hidden layers, processing each layer’s probe responses with a permutation-invariant aggregator before a global MLP predicts the target property.The paper describes separate instantiations for MLPs and Transformers.Source: §5 · pp. 6–7
  • For MLPs, HIDDENPROBE uses a Set Transformer to respect independent hidden-neuron permutations that preserve the represented function.The paper contrasts these layerwise permutations with entangled symmetries in weight-space representations; Transformer probing avoids GL(n) parameterization symmetries by depending on functional behavior.Source: §5 · p. 7

6 UNIVERSALITY OF PROBE-BASED LEARNING

Under compact-domain and continuity assumptions, the section establishes universality for probe-based learning of neural functionals. Final-layer probing has a general analytic-model guarantee, while hidden-probe universality uses fewer probes under architecture-specific assumptions and an almost-everywhere parameter restriction.

6 UNIVERSALITY OF PROBE-BASED LEARNING

  • For analytic models with q parameters, 2q + 1 final-layer probes let an MLP approximate any continuous functional uniformly over the compact parameter domain.The theorem assumes compact input and parameter domains and approximation error below any chosen ϵ.Source: §6 · p. 7
  • HIDDENPROBE achieves universality with N = dmax + 1 probes for specified analytic MLPs or N = d(2H + 1) + 1 for pure softmax MHA stacks.For compact parameter subsets avoiding an appropriate Lebesgue-null set, the theorem gives approximation error below any ϵ.Source: §6 · p. 7

7 EXPERIMENTS

HIDDENPROBE generally outperforms ProbeGen and other weight-space and probe-based baselines across the evaluated tasks. Its results also show that hidden-layer information can yield competitive performance with fewer probes.

Experimental setup

  • HIDDENPROBE is evaluated on INR image classification, prediction of CNN test accuracy, and Transformer test-accuracy prediction.The experiments emphasize comparison with ProbeGen, which uses only final-layer probes.Source: §7 · pp. 7–8

INR classification

  • HIDDENPROBE achieves the best performance on MNIST, Fashion-MNIST, and CIFAR-10 INR classification, second-best on augmented CIFAR-10, and beats ProbeGen on all four datasets.ProbeGen ranks second on three of the four datasets.Source: §7 · p. 8, Table 1 · p. 8

Test accuracy prediction

  • HIDDENPROBE achieves the best performance on all five test-accuracy prediction benchmarks and outperforms ProbeGen in every case.CIFAR-10-WP contains heterogeneous architectures with up to four hidden layers.Source: §7 · p. 8, Table 2 · p. 8
  • On CIFAR-10-WP, where model architectures vary, probe-based models significantly outperform weight-space models.HIDDENPROBE is best across the datasets, while ProbeGen ranks second on four of five.Source: §7 · p. 8

Transformer accuracy prediction

  • HIDDENPROBE achieves state-of-the-art performance on MNIST and AGNews Transformer benchmarks, while output-only ProbeGen is also competitive.The comparison covers the full datasets and reports Kendall’s τ.Source: §7 · pp. 8–9, Table 3 · p. 9

Varying probe count

  • HIDDENPROBE generally improves as probe count increases and is comparable to or better than ProbeGen with half as many or fewer probes.ProbeGen also improves with more probes, but to a weaker extent; higher test accuracy or Kendall’s τ is better, depending on the task.Source: Figure 3 · p. 9, §7 · p. 9

8 CONCLUSION

The paper concludes that finite probes can identify neural networks and support universal learning of their properties, with hidden-layer information reducing the required probe count. HIDDENPROBE uses hidden responses to learn neural functionals across MLPs and Transformers.

8 CONCLUSION

  • Across MLPs and multi-head attention blocks, hidden-layer information reduces the number of probes needed for identification; HIDDENPROBE improves over ProbeGen and weight-space baselines across tasks.The authors suggest adapting such methods to larger architectures, including language models or diffusion kernels, as future work.Source: §8 · pp. 9–10

A.1 HIDDENPROBE: A PROBE-BASED MODEL FOR LEARNING ON WEIGHT SPACES

  • Hidden-neuron permutations preserve an MLP’s realized function, so HIDDENPROBE uses permutation-invariant set encoders for hidden-layer responses.The hidden-layer representations transform equivariantly under the corresponding neuron permutations.Source: §A.1 · p. 14
  • HIDDENPROBE concatenates hidden-layer representations in layer order with final probe outputs, then maps the representation to the target with a predictor.Probes, layer-wise encoders, and the predictor are trained jointly, and learned probes are shared across target networks.Source: §A.1 · p. 14

A.2 HIDDENPROBE FOR TRANSFORMERS

  • For Transformers, HIDDENPROBE learns a shared bank of probes with varying sequence lengths because singleton and multi-token probes provide complementary reconstruction information.The reconstruction arguments associate singleton probes with the value matrix and multi-token probes with attention weights.Source: §A.2 · p. 14
  • The Transformer model records block outputs, normalized token states, and decoder outputs, then encodes channel trajectories and final logits to predict the target.A shared MLP encodes channel trajectories before a Transformer and MLP produce the output.Source: §A.2 · pp. 14–15
  • HIDDENPROBE is invariant by design to head permutations and specified invertible transformations of attention matrices because these preserve recorded block responses and classifier outputs.The authors contrast this with symmetry handling as a challenge in prior work.Source: §A.2 · p. 15

B PROOF OF TECHNICAL RESULTS

For an analytic parameterized function, the theorem establishes that agreement on a finite set of generic probes is equivalent to agreement everywhere. The proof applies the Finite Witness Theorem to pairs of parameters.

Theorem 1

  • For analytic f: R^d × R^q → R^m, agreement at 2q+1 generic probes is equivalent to the two parameterized functions agreeing everywhere on R^d.The claim holds for almost any choice of probes and for all parameter pairs in the stated domain.Source: B PROOF OF TECHNICAL RESULTS · p. 15
  • The result makes finite-probe identification possible: matching outputs on the selected probes certifies equality of the realized functions.Source: B PROOF OF TECHNICAL RESULTS · p. 15

Proof

  • The proof applies the Finite Witness Theorem to an analytic function of parameter pairs and probe inputs.It treats each parameter pair as a point in R^(2q) and uses 2q+1 probe samples.Source: B PROOF OF TECHNICAL RESULTS · p. 15

B.1 IDENTIFICATION OF ANALYTIC MLPS FROM HIDDEN PROBES

The section establishes that intermediate-layer probe responses can identify analytic networks with fewer probes than final outputs alone. For fully connected MLPs under stronger activation assumptions, they can also recover the weights for generic targets.

B.1 IDENTIFICATION OF ANALYTIC MLPS FROM HIDDEN PROBES

  • The full hidden-probe record contains sampled states at every layer, including the final output, and equality of these states implies equality of the realized functions.The section also gives an induction argument that establishes equality layer by layer.Source: §B.1 · pp. 15–16
  • For analytic sequential models, N ≥ q + qmax + 1 generic probes with intermediate observations identify every pair of models that represent different functions.The bound applies simultaneously to all parameter pairs for almost every probe tuple.Source: §B.1 · p. 15
  • For fully connected MLPs with analytic, injective, non-polynomial activation, N ≥ 1 + max_i d_i generic probes uniquely identify generic targets from all hidden-layer activations.The excluded parameter set is closed, nowhere dense, and Lebesgue-null.Source: §B.1 · p. 19
  • For generic targets, the proof of the identification theorem yields an algorithm that reconstructs the network weights from hidden activations.The algorithm initializes the probe inputs as layer 0 and returns the recovered layer blocks as the parameter tuple.Source: §B.1 · p. 20

B.2 LOWER BOUNDS ON PROBE COUNTS

The lower-bound constructions show that the probe requirements cannot generally be reduced below near the established sufficient bounds. The thresholds depend on whether probes observe final outputs or hidden activations.

B.2 LOWER BOUNDS ON PROBE COUNTS

  • Both lower bounds use shallow universal MLPs with widths D = (1, m, 1), q = 3m + 1, and compact parameter spaces with nonempty interior.The constructions establish the stated worst-case necessity across widths m ≥ 1.Source: §B.2 · p. 22
  • For sin, N ≤ 2q − 2 final-layer probes fail on a nonempty open set, placing the 2q + 1 sufficient bound within two probes of the 2q − 1 lower bound.The authors do not claim these bounds are tight for every individual activation.Source: §B.2 · pp. 22–24
  • For tanh, N ≤ dmax hidden probes cannot identify generic targets, while dmax + 1 probes suffice generically, making the threshold sharp in the worst case.The lower bound holds for every probe tuple and almost every interior target in the stated shallow family.Source: §B.2 · pp. 22, 24

B.3 RELU NETWORK IDENTIFICATION FROM PROBES

Finite probes cannot exactly identify every ReLU network, but random probes support high-probability approximate identification under the paper’s stated setting. The section also distinguishes final-output observations from richer hidden-activation observations.

B.3 RELU NETWORK IDENTIFICATION FROM PROBES

  • No finite probe set exactly identifies all ReLU networks: three-hidden-neuron networks can agree on every probe yet differ elsewhere.A localized ReLU function supported in an interval avoiding the probes gives the counterexample.Source: §B.3 · p. 24
  • The section frames ReLU identification as approximate agreement on a set of input measure at least 1−ε, rather than exact recovery everywhere.The probes are sampled independently from the same measure used to evaluate agreement.Source: §B.3 · p. 25
  • The proof bounds disagreement-set complexity by counting polynomial tests, then applies a random-sampling bound to ensure large disagreement sets are detected.The setup uses compact input and parameter spaces; bounded outputs under these conditions support the stated Ls reconstruction objective.Source: §B.3 · pp. 25–26
  • With i.i.d. probes from μ, compact-domain ReLU networks matching a target’s final outputs have disagreement measure at most ε with probability at least 1−δ.The guarantee is over probe samples and applies to every parameter in the stated compact class.Source: §B.3 · p. 27
  • Matching the target’s hidden activations on random probes likewise bounds disagreement on any hidden-layer neuron to measure at most ε with probability at least 1−δ.This is the all-hidden-activation guarantee, distinct from the final-output guarantee.Source: §B.3 · pp. 27, 29

B.4 FUNCTION-LEVEL IDENTIFICATION OF MULTI-HEAD ATTENTION

This section studies finite-probe identification of multi-head attention, showing that fixed probes with hidden access recover canonical representations and distinguish standalone layers, while identifying generic multilayer targets up to canonical equivalence. It also describes independent layerwise reconstruction and notes that the theorem does not guarantee solver convergence or out-of-distribution error from nonzero residuals.

Relation to query-based attention recovery

  • The focus is generic identification of multilayer MHA stacks from shared, nonadaptive probes with hidden access, using complete tokenwise layer outputs rather than only scalar final-token attention-oracle outputs.The section contrasts these observation models with query-based attention-recovery work.Source: §B.4 · p. 30

Lemma 9 (Three-token separation)

  • Two MHA layers agree on every three-token input if and only if they agree at every sequence length.The separation argument is a matrix-output adaptation of Kim & Kim (2026, Proposition 3.1).Source: §B.4 · p. 30

Lemma 10 (Countably many candidates from a fixed probe batch)

  • The fixed batch of d(2H + 1) probes yields at most countably many effective layer candidates for almost all probe tuples and parameters.The batch comprises d singletons, d(H + 1) triples, and d(H −1) sequences of length s + 1.Source: §B.4 · pp. 30–31

Theorem 14 (Structured probing for softmax attention)

  • With N = d(2H + 1) + 1 fixed probes of lengths at most max{3, 1 + d/H}, generic probe tuples admit a closed exceptional parameter set of dimension at most D−1.The probe set is chosen independently of the target; outside the exceptional set, the theorem's identification guarantee holds for every target parameter.Source: §B.4 · p. 33

Proposition 15 (Fixed-probes identification)

  • For almost every probe tuple and target, any stack matching the complete hidden transcript has the target’s canonical representation in every layer.The result uses a shared probe set and applies to generic targets and probe tuples.Source: §B.4 · p. 33

Multilayer identification

  • For multilayer targets outside EP, no canonically different competitor shares the transcript, and canonical agreement implies equality of all layer maps and composed functions at every length.The exceptional set EP is closed and has Hausdorff dimension at most D − 1; the identifiable set U is open, dense, and full measure.Source: §B.4 · p. 34

Multilayer MHA reconstruction from a fixed set of probes

  • Multilayer reconstruction fits each layer independently because probes provide its training inputs as preceding target states, and raw-factor optimization preserves prescribed head counts and rank bounds.The layer fits may run in parallel.Source: §B.4 · p. 34

Correctness and scope

  • The theorem guarantees feasible layer fits and all-length correctness, but not global convergence of numerical least squares or an OOD error bound from nonzero residuals.No head alignment between layers is required.Source: §B.4 · p. 35

The special case of one layer

  • For one layer, N = 4dh + 1 fixed probes distinguish every pair of targets exactly when their functions agree at every sequence length.The probe set is fixed for given d and h, with a common probe length n∗.Source: §B.4 · p. 35

B.5 LEARNING FUNCTIONALS AND OPERATORS FROM PROBES

This section establishes conditions under which finite probe representations support learning continuous functionals and operators, and describes HIDDENPROBE implementations for INR classification, CNN-zoo accuracy regression, and Transformer tasks. It also clarifies that the theory motivates the design but does not automatically guarantee results for the finite-width transformer classifiers used in the experiments.

B.5 LEARNING FUNCTIONALS AND OPERATORS FROM PROBES

  • A continuous representation that separates represented functions yields universal MLP approximation of continuous functionals and operators over compact domains.The separation condition requires equal representations to imply equal functions; the converse need not hold.Source: §B.5 · p. 36
  • For real-analytic families, almost every final-output probe bank of N ≥ 2q + 1 points identifies represented functions and enables MLP approximation of continuous functionals and operators.The probe evaluation map is one-to-one with a continuous inverse on its image.Source: §B.5 · p. 36
  • For MLPs and softmax-attention stacks, hidden-probe representations support continuous functional and operator readouts on compact parameter sets excluding a lower-dimensional exceptional set.For each good probe bank, the excluded set has Hausdorff dimension at most D − 1; the respective HIDDENPROBE families approximate the readouts.Source: §B.5 · p. 37

C.1.1 INR CLASSIFICATION

  • For INR classification, HIDDENPROBE uses 128 learned coordinate probes and permutation-invariant neuron tokens built from probe trajectories, global statistics, and layer embeddings.The targets use sine activations, which are analytic but non-injective, so the stated parameter-identification theorem for injective activations does not directly apply.Source: §C.1.1 · pp. 39–40

C.1.2 HIDDENPROBE FOR CNN-ZOO ACCURACY REGRESSION

  • For CNN-zoo accuracy regression, HIDDENPROBE uses a shared bank of 128 learned image probes and pools hidden channel responses within each layer to predict accuracy.Its hidden branch uses additional global statistics; channel pooling respects channel permutations, and the probes require no additional target-network queries.Source: §C.1.2 · p. 40

C.2 HIDDENPROBE FOR TRANSFORMERS

  • For MNIST and AGNews Transformers, HIDDENPROBE records three hidden layers and logits for 256 learned probes, then encodes channel trajectories and jointly pools response and logit tokens.The bank combines learned sequences and learned images, and the predictor receives recorded responses and fixed semantic identifiers rather than target weights or identities.Source: §C.2 · pp. 40–41
  • Theory motivates shared probe banks with varied sequence lengths and intermediate responses, but does not guarantee reconstruction or target identification in the finite-width experiments.Experiments use complete transformer classifiers with feedforward and normalization layers; universality concerns capacity-growing architecture families, not this configuration or its learned parameters.Source: §C.2 · p. 41

D.1 FUNCTION-LEVEL RECONSTRUCTION ON SYNTHETIC NEURAL NETWORKS

The synthetic reconstruction experiments compare final-output observations with complete hidden-layer observations. Across analytic and ReLU MLPs and an MHA stack, hidden activations support more accurate reconstruction with fewer probes.

D.1 FUNCTION-LEVEL RECONSTRUCTION ON SYNTHETIC NEURAL NETWORKS

  • For an analytic tanh MLP, hidden activations enable essentially exact reconstruction, while final-output reconstruction improves more gradually and retains substantially larger error.The target has widths (1, 4, 4, 1), with training on [−1, 1] and evaluation on [−3, 3].Source: §D.1 · p. 42, Figure 4 · p. 42
  • For a challenging ReLU sawtooth target, hidden activations achieve near-exact reconstruction with far fewer probes, whereas final-output reconstruction fails with the tested probe budget.The target has two hidden layers of width 4 and many breakpoints, described as atypical.Source: §D.1 · p. 42, Figure 5 · p. 42
  • With nine probes, hidden activations enable essentially exact MHA-stack reconstruction, versus about 31 probes for final outputs alone.The MHA-stack comparison uses final outputs or complete hidden-layer observations.Source: §D.1 · p. 43, Figure 6 · p. 43

D.2 FULL INR CLASSIFICATION COMPARISON

  • The extended INR-classification comparison includes HIDDENPROBE and prior weight-space and probing methods across INR benchmarks.The comparison includes earlier baselines and results reported in prior evaluations; unavailable results are marked “–”.Source: §D.2 · p. 43, Table 4 · p. 44

D.3 FULL TEST-ACCURACY PREDICTION COMPARISON

  • The extended test-accuracy prediction comparison covers weight-space, graph-based, and probing approaches, including HIDDENPROBE, with Kendall’s τ as the reported metric.Results not reported for a benchmark are marked “–”.Source: §D.3 · pp. 43–44, Table 5 · p. 44

D.4 TRANSFORMER ACCURACY PREDICTION ACROSS THRESHOLDS

On MNIST-Transformers and AGNews-Transformers, HIDDENPROBE achieves the highest Kendall’s τ across the evaluated accuracy thresholds. The appendix also describes parameter-count and runtime comparisons and the tuning protocol.

D.4 TRANSFORMER ACCURACY PREDICTION ACROSS THRESHOLDS

  • HIDDENPROBE achieves the highest Kendall’s τ across all evaluated accuracy thresholds on both MNIST-Transformers and AGNews-Transformers.Tables 6 and 7 report these threshold results.Source: §D.4 · p. 44, Table 6 · p. 45, Table 7 · p. 45

D.5 NUMBER OF PARAMETERS COMPARISON

  • The parameter-count comparison reports trainable parameters in millions across three INR classification tasks, CIFAR-10-GS regression, and MNIST-Transformers.Published configurations may differ from those used in accuracy comparisons; ProbeGen and HIDDENPROBE counts are author-provided, and dashes indicate unverified counts.Source: §D.5 · p. 45, Table 8 · p. 46

D.6 TIME MEASUREMENT

  • The runtime comparison measures popular deep weight-space models, ProbeGen, and HIDDENPROBE on the MNIST INR benchmark.The runtime table covers the full 55,000-target INR training split with batch size 32.Source: §D.6 · p. 46, Table 9 · p. 46

D.7 TUNING BUDGET

  • ProbeGen and HIDDENPROBE use the same validation-based tuning protocol, with test performance excluded from model selection for test-accuracy prediction.Each configuration is first trained for 20 epochs with seed 0; the selected configuration is retrained for five seeds and 60 epochs, with mean and standard error reported. CIFAR-10 INR classification selects by validation accuracy and trains final runs for 30 epochs.Source: §D.7 · pp. 46–47

Related papers

Loading 2609.33901v2…