Source-linked AI summary
Curvature Cryptanalysis of Smooth Transformer Feed-Forward Networks
Munawar Hasan, Apostol Vassilev
TL;DR
The paper asks whether black-box chosen-input access to smooth transformer FFNs leaks internal parameter geometry beyond behavioral outputs. It uses shared Hessian factorization and vector-output finite differences to recover hidden directions, achieving high alignment with 16 projected Hessians and supporting high-fidelity substitutes. The scope remains tied to the stated block-level oracle and does not guarantee recovery of all parameters or global functional equivalence.
Problem
Behavioral fidelity alone does not establish recovery of hidden FFN geometry, motivating structural extraction under black-box raw-output access.
Method
The attack factorizes projected input Hessians, which share rank-one factors from first-layer weights, using finite differences and vector-output stencil reuse.
Results
With T = 16 projected Hessians from 8,193 queries, mean DirRec reaches 0.9644 for GELU and 0.9476 for SiLU, while recovered directions support substitutes exceeding 93% top-1 agreement.
Takeaways & Limitations
Under the stated oracle model, smooth FFN curvature exposes internal first-layer geometry that can be reused for functional replacement beyond behavioral imitation.
Takeaways & Limitations
The local identifiability result does not guarantee recovery of signed row scales, biases, or second-layer parameters, nor global functional equivalence; the analysis also has a boundary for nonlinear preprocessing such as layer normalization.
Abstract
from arXiv · showhide
We show that smooth two-layer feed-forward networks (FFNs) expose an additional structural model extraction channel under a chosen-input raw-output oracle at the FFN branch; consider transformer FFN branches with GELU or SiLU activations under chosen-input raw-output access, without access to parameters, gradients, or internal activations; exploit a second-order leakage channel in which projected input Hessians form different mixtures of the same hidden symmetric rank-one factors induced by the FFN input weights. We formalize resulting Hessian collection as a partially symmetric decomposition to establish conditions for local identifiability and stability to exploit vector-output stencil reuse to reduce the structural query cost by a factor of 16. On independently trained CIFAR-10 vision transformers, only 16 projected Hessians, corresponding to 8193 black-box queries, recover the hidden FFN directions with average absolute cosine alignment above 0.94, with 95.1 % of GELU and 91.9 % of SiLU directions exceeding 0.90 alignment. Recovery remains high across independently trained models, repeated extraction runs, and all transformer blocks. The recovered structure supports functional extraction too. Keeping the recovered directions fixed and fitting only the remaining FFN parameters yields high-fidelity substitutes with more than 93 % top-1 agreement, while test accuracy remains within 0.90% and 0.62% of the GELU and SiLU targets. Output rounding and Gaussian noise substantially reduce recovery under a fixed attack configuration, but adapting the finite-difference step restores average alignment to 0.9603 and 0.9398. This is an end-to-end path from black-box second-order observations to hidden FFN-structure recovery and functional replacement. Under the stated oracle model, smooth FFN curvature exposes internal parameter geometry that behavioral fidelity alone cannot reveal.
1 Introduction
The paper asks whether black-box access can reveal transformer FFN parameter geometry, not merely behavior, and develops curvature-based structural extraction to answer that question. Projected Hessians expose shared hidden directions, which can support both direction recovery and functional replacement.
- Motivation: Behavioral agreement can approximate outputs without revealing whether two networks share internal representations or weights.This motivates distinguishing functional imitation from structural extraction.
- Research question: The paper studies whether chosen-input raw-output access can reveal a transformer FFN’s internal parameterization and support high-fidelity functional extraction.The setting provides black-box branch outputs but no parameters, gradients, or internal activations.
- Approach: Projected input Hessians contain shared symmetric rank-one factors induced by first-layer weights, enabling recovery of normalized hidden directions from different mixtures.The analysis targets smooth FFNs with GELU or SiLU activations.
- Results: Using 16 projected Hessians, recovery reaches average absolute cosine alignment 0.9644 for GELU and 0.9476 for SiLU, while functional substitutes exceed 93% top-1 agreement.Accuracy drops are 0.90 and 0.62 percentage points for GELU and SiLU, respectively.
- Theory: The Hessian collection is formulated as a partially symmetric decomposition with local identifiability up to sign and permutation, plus stability under perturbed observations.The theory also derives a necessary measurement-count condition.
- Attack design: Vector-output stencil reuse reduces query cost because one finite-difference stencil provides the Hessian of every output coordinate simultaneously.The attack estimates projected Hessians with centered finite differences and recovers the shared dictionary through joint optimization.
2 Related Work
Prior extraction work primarily targets behavioral imitation or exploits piecewise-affine activation boundaries, while derivative-based research shows that smooth-network derivatives can reveal hidden weights. This paper applies that curvature perspective to internal transformer FFN directions and links structural recovery to surrogate completion.
- Behavioral and cryptanalytic extraction: Earlier model-extraction attacks used prediction queries to construct substitutes that reproduce remotely deployed model functionality.Later work framed extraction cryptanalytically and studied query-based recovery for ReLU networks.
- Contrast with ReLU methods: ReLU-based approaches exploit piecewise-affine activation boundaries or local changes, unlike the smooth-curvature mechanism studied here.The paper instead uses second-order information from smooth activations.
- Derivative-based identification: Mathematical identification work shows that first- and second-order derivatives can reveal hidden weight structure through rank-one matrices induced by hidden vectors.Extensions address smooth deep and vector-valued networks through Hessian-derived entangled weights.
- Transformer extraction: Transformer extraction studies have examined grey-box systems, public encoders, private classifiers, and behavioral indistinguishability of surrogates.These settings differ from the internal FFN branch targeted in this paper.
- Positioning: The paper recovers normalized first-layer directions of an internal smooth FFN from second-order oracle measurements, then retains them during surrogate completion.This connects parameter-geometry recovery to high-fidelity functional extraction.
- Defenses: Existing output defenses perturb or redirect returned information, but this work evaluates their effect specifically on curvature extraction.The robustness question concerns whether rounding or additive noise suppresses finite-difference measurements.
3 Extraction Framework
The framework defines a chosen-input raw-output oracle for a smooth two-layer FFN and separates structural recovery of first-layer directions from subsequent functional fitting. Structural recovery is evaluated up to sign and permutation, while system-level fidelity is assessed separately.
- Oracle model: The oracle returns a designated FFN branch’s output vector for adaptively chosen inputs while hiding parameters, gradients, and intermediate activations.Queries operate in FFN representation space rather than necessarily on raw images or token sequences.
- Target FFN: The target branch is gθ(x) = W2ϕ(W1x + b1) + b2 with smooth activations such as GELU or SiLU.For affine residual wiring, branch outputs can be recovered from residual outputs by subtracting the known input.
- Assumptions: The adversary is assumed to know the architecture, activation family, residual wiring, and target block, while parameters and the hidden width or target-block location are not all inferred within scope.The framework explicitly treats block-level representation-space access as the threat model.
- Extraction objectives: Structural extraction recovers normalized rows of W1, whereas functional extraction fits a surrogate branch whose outputs approximate the target.The two objectives are evaluated separately, though the surrogate may preserve complete-classifier behavior.
- Evaluation: Functional fidelity is evaluated through output discrepancy, classifier agreement, accuracy, KL divergence, and centered-logit mean-squared error.These metrics quantify empirical fidelity without selecting a single functional-extraction tolerance pair.
- Recovery ambiguity: Because projected Hessians contain u_j u_j^T, direction recovery is defined only up to sign and permutation.The matched direction-recovery metrics require the true directions for scientific evaluation and are unavailable during extraction.
4 Curvature Leakage and Identifiability
Smooth FFN curvature produces projected Hessians that share hidden rank-one factors, enabling structural recovery of first-layer directions under suitable local nondegeneracy conditions. The analysis also identifies residual, normalization, activation, and perturbation boundaries for this leakage channel.
- Shared Hessian structure: Projected Hessians of smooth FFNs are different mixtures of shared symmetric rank-one factors induced by first-layer weight directions.Scalar output projections and varying query–projection pairs change the coefficients while preserving the rank-one atoms.
- Residual and normalization effects: The residual identity contributes no second-order signal, so observing residual output preserves the nonlinear FFN branch curvature exactly.The identity path is linear in the input and therefore has zero Hessian.
- Residual and normalization effects: LayerNorm before the FFN introduces additional curvature, so the primary analysis uses the post-normalization representation as the FFN query variable.An oracle before LayerNorm must separate normalization curvature from FFN curvature.
- Shared Hessian structure: The curvature measurements form a partially symmetric decomposition whose first two modes share each normalized direction and whose third mode records measurement-specific coefficients.This structure constrains each atom to u_j u_j^T rather than an arbitrary matrix in the observed Hessian span.
- Identifiability and stability: Local identifiability holds when the restricted differential is injective, yielding unique normalized directions and coefficients up to permutation and sign symmetries.The theorem is conditional and does not claim injectivity for every FFN or probe–projection design.
- Identifiability and stability: The necessary dimension bound is not sufficient, while local stability is analyzed through the smallest singular value of the restricted differential under Hessian perturbations.The dimension count is conservative because the relaxed factorization treats measurement coefficients as unconstrained nuisance parameters.
- Activation scope: Smooth-curvature leakage vanishes almost everywhere for ReLU networks, although finite-difference stencils crossing activation boundaries can produce a different nonsmooth signal.Boundary localization is outside the scope of the smooth-Hessian analysis.
5 Curvature-Based Extraction Attack
The attack estimates multiple projected input Hessians from chosen-input vector-output queries and jointly factorizes them to recover shared first-layer direction atoms. Vector-output stencil reuse produces the optimized structural query budget, while optional completion fits remaining branch parameters with the recovered directions fixed.
- Attack pipeline: The attack first estimates projected input Hessians from chosen-input oracle responses, then jointly factorizes them to recover shared first-layer directions.Changing probes or output projections changes mixture coefficients while preserving the shared rank-one direction atoms.
- Hessian estimation: Centered finite-difference stencils estimate diagonal and mixed Hessian entries, cancel constant and first-order terms, and incur O(h^2) truncation error under fourth-order smoothness.The step size trades lower truncation error against greater amplification of oracle perturbations.
- Query complexity: For d = 64, one complete Hessian requires 8,193 vector-output oracle queries, while the optimized attack counts structural queries as P(2d^2 + 1).The shared-stencil construction enables offline production of arbitrary projected Hessians at each probe location.
- Attack pipeline: The scalar projected response is computed locally from the complete vector output, so output projections require no additional oracle queries.A single vector-valued stencil can therefore support Hessians for multiple output projections.
- Identifiability limits: Increasing the number of measurements beyond the available output rank cannot create new independent information when directions are invisible, weakly conditioned, or rank-limited.Additional probes can nevertheless provide robustness when curvature is weak or output dimension is insufficient.
- Functional completion: Optional completion fits the remaining branch parameters with the recovered direction matrix fixed, adding Nfit queries to the structural budget.The structural output is the hidden first-layer direction dictionary; functional replacement is evaluated separately under additional access assumptions.
6 Experiments
Experiments show that curvature measurements recover hidden FFN directions reliably across models, activations, extraction settings, and transformer blocks, while also supporting high-fidelity functional replacement. Recovery requires sufficiently diverse Hessian measurements and remains vulnerable to adaptive finite-difference scaling despite fixed-step perturbation defenses.
- Structural recovery: 95.1% of GELU and 91.9% of SiLU hidden directions exceeded 0.90 absolute cosine alignment across independently trained checkpoints.GELU DirRec ranged from 0.9615 to 0.9663, while SiLU ranged from 0.9452 to 0.9492.
- Measurement budget: T = 4 produced no directions above 0.90 alignment, T = 8 yielded partial recovery, and T = 16 entered a stable high-recovery regime.At T = 8, GELU achieved DirRec = 0.8565 ± 0.0192 and SiLU achieved 0.7972 ± 0.0271.
- Measurement budget: Low Hessian reconstruction loss at T = 4 did not certify correct structural recovery, showing that observation fitting and parameter identification can diverge.Reconstruction loss is useful for restart selection only once measurements are sufficiently informative.
- Optimization: 94.5% of hidden directions exceeded 0.90 alignment with random multi-restart joint optimization, outperforming the tested sequential-pursuit initializations.Primary experiments used random initialization, joint Adam optimization, and restart selection by observed Hessian reconstruction loss.
- Functional replacement: 93.24% ± 0.45% GELU and 94.89% ± 0.40% SiLU top-1 agreement was obtained after fixing recovered directions and fitting the remaining FFN parameters.Accuracy drops were 0.90 ± 0.05 and 0.62 ± 0.12 percentage points for GELU and SiLU, respectively.
- Functional replacement: 12,000 completion queries were sufficient for functional fidelity to largely stabilize, with only marginal improvement from doubling the budget.The ablation reports rapid improvement at lower budgets and effective saturation by Qfit = 12,000.
- Perturbation robustness: 0.9603 alignment under rounding and 0.9398 under Gaussian noise was restored by increasing the finite-difference step from 0.01 to 0.30.At h = 0.30, Hit0.90 reached 0.9297 under rounding and 0.9219 under Gaussian noise.
- Structural recovery: 0.9644 for GELU and 0.9476 for SiLU mean directional recovery was achieved using 16 projected Hessians and 8,193 structural queries.Recovery remained high across independently trained models, extraction seeds, and transformer blocks.
7 Conclusion
The paper presents a black-box curvature attack that factorizes shared Hessian rank-one components to recover transformer FFN first-layer directions. The recovered geometry also enables high-fidelity functional substitutes, while the mechanism is scoped to smooth two-layer FFNs under the stated oracle assumptions.
- 16 projected Hessians constructed from 8,193 queries recover hidden FFN directions with mean DirRec values of 0.9644 for GELU and 0.9476 for SiLU.Recovery remains high across training seeds, extraction runs, and transformer blocks.
- Joint factorization of projected Hessians exposes normalized first-layer direction dictionaries through shared rank-one factors induced by first-layer weights.The method turns second-order black-box responses into a structural extraction channel.
- 95.1% of GELU and 91.9% of SiLU hidden directions exceed 0.90 absolute cosine alignment.
- Fixing recovered directions and fitting remaining FFN parameters produces surrogates with more than 93% top-1 agreement and target-accuracy gaps within 0.90 and 0.62 percentage points for GELU and SiLU.
- The analysis extends under the same architectural and oracle assumptions to encoder-only, decoder-only, and encoder–decoder transformers with smooth two-layer FFNs, but many language models use gated FFNs such as SwiGLU.