The Linear Representation Hypothesis Needs a Group Action

• Different implicit equivalences can make methods that appear to study the same representation probe different objects and hypotheses.This can happen even within a single study, and assumptions may shift between estimation, comparison, and intervention.Source: §1 · p. 1
• An unnormalized difference-in-means vector acts as a magnitude-sensitive displacement, while its normalized use defines a projective direction.The two effective interventions therefore treat the putative refusal direction as objects in different mathematical spaces.Source: §1 · p. 1
• A representation claim specifies its object space, equivalence group, producing procedure, and asserted predicate.The procedure must transform equivariantly, the predicate must remain invariant, and the group must account for function-preserving architectural reparameterizations.Source: §1 · pp. 1–2
• The Linear Representation Hypothesis is a family of claims because displacement, decoding, superposition, and subspace formulations use different objects and transformation laws.Admissible symmetry also depends on the reading point, and a multi-stage analysis must be checked as a composition.Source: §1 · p. 2
• Making equivalence explicit is part of stating a representation claim, and may narrow conclusions while clarifying when analyses test the same hypothesis.The paper situates this framework as a way to audit common quantities and recent interpretability analyses.Source: §1 · p. 2

• A displacement feature is a vector in V that transforms as v → Av under affine changes, preserving constant-offset relations.Word-embedding offsets and methods such as difference-in-means and contrastive activation addition use this formulation.Source: §2 · p. 2
• Linear decodability treats a probe as a covector in V∗, transforming as w → A−⊤w; adding a bias permits affine transformations.Probe weights and steering vectors may both be stored as d-dimensional arrays, but they obey different transformation laws and cannot be canonically identified without additional structure [3].Source: §2 · p. 2
• Superposition represents an activation as a baseline plus weighted feature directions, with activation-coordinate changes acting on the baseline and directions.Its latent-coordinate ambiguity is distinct from equivalence under changes of activation coordinates; sparsity or other constraints generally reduce but do not eliminate it [18].Source: §2 · p. 2
• Even for a fixed object space, procedures can have different symmetry requirements: difference-in-means needs no inner product, whereas PCA does.Both can return a direction in P(V), but PCA requires an inner product to rank directions by explained variance.Source: §2 · pp. 2–3
• Cosines and spectral quantities require similarity geometry, while absolute norms and intervention magnitudes require fixed scale, although the formulations do not intrinsically require an inner product.Thus the metric can enter through the evidence supporting a hypothesis rather than through the hypothesis itself.Source: §2 · p. 3
• The paper treats the equivalence group as part of the claim, unlike approaches that choose a nuisance group to define a representation-similarity metric.The group is constrained by the architecture and analysis pipeline.Source: §2 · p. 3

3.1 Representation Equivalence

• Representation equivalence declares which transformations count as changes of description, and the choice of group determines which quantities can remain invariant.A larger group leaves fewer invariant quantities; for claims about the model, the group must include transformations realized by function-preserving reparameterizations.Source: §3.1 · p. 3

3.2 Common Choices of Transformation Group

• Affine, similarity, and isometry groups preserve progressively stronger structures, from affine relations to angles and then norms, distances, and absolute scale.A larger equivalence group therefore imposes stronger invariance requirements and permits fewer quantities to be attributed to the representation.Source: §3.2 · p. 4

3.3 The Object and the Procedure

• A representation claim specifies an object space M with a G-action, a procedure F producing the object, and an invariant predicate P.F must be equivariant and P invariant; this factorization matters because the intermediate object and its transformation law may be used in later analyses.Source: §3.3 · p. 4
• A displacement belongs to V while a probe belongs to V∗, so identifying them requires additional structure such as a metric.An inner product identifies them equivariantly under isometries; passing to projective directions permits similarity equivariance, while an ℓ2 probe penalty leaves only isometric equivariance.Source: §3.3 · p. 4

3.4 The Specification

• Admissibility requires an equivariant procedure, an invariant predicate, and a group containing architecture-induced equivalences.Claims are directly comparable only when they use the same equivalence and G-space, or an explicit equivariant map connects different spaces; restricting a group can make additional predicates well defined, but the architectural condition must be rechecked.Source: §3.4 · p. 5

4.1 The Architectural Floor

• A claim that changes under a function-preserving parameter transformation is a property of the parameterization, not the model function.At a reading point, admissible equivalences must include the symmetry induced by such transformations.Source: §4.1 · pp. 5–6
• The same architecture can induce different symmetries at different reading points, so equal-dimensional representations cannot be identified by shared coordinates alone.At query or key sites, the induced GL(dhead) symmetry rules out angular and metric predicates based on a fixed inner product; at the residual stream it acts trivially.Source: §4.1 · p. 6

4.2 Composition of Analysis Stages

• A composite analysis is admissible when its stages are equivariant under compatible actions and its final predicate is invariant.When a stage is not equivariant, admissibility must be established for the composite; an intermediate change of object space can alter the action seen by the next stage.Source: §4.2 · p. 6
• For compatible stage actions, the pipeline is limited by its most restrictive stage unless the composite has stronger invariance.A direction estimated under Gaff, compared by cosine under Gsim, and calibrated by a norm under Giso is guaranteed admissible only under Giso.Source: §4.2 · p. 6

5 A Symmetry Audit of Common Representation Quantities

• Table 1 records the largest group under which each common representation quantity is well defined; using a smaller group narrows the claim it supports.Its rows are grouped by preserved group, and the final column names the structure fixing each restriction.Source: §5 · p. 6
• Dictionary learning is restricted to isometries on activations by its Euclidean reconstruction loss, while a coordinatewise sparsity penalty restricts latent mixing to signed permutations.The encoder and decoder can absorb affine changes to activations without changing latent codes, but preserving Euclidean loss for all residuals requires A⊤A = I.Source: §5 · p. 6

6.1 Identification by Storage Format

• Probe coefficients live in V∗, whereas steering vectors are displacements in V, so treating both length-d arrays as the same direction requires an explicit identification.Comparing their directions by cosine is meaningful under similarity geometry, but reusing probe coefficients as an additive displacement is equivariant only under isometries.Source: §6.1 · p. 7

6.2 Normalization and Calibration

• Normalization and calibration assign meaning to vector length, so equal numerical coefficients need not denote equal displacements across intervention conventions.Contrastive Activation Addition normalizes magnitudes across behaviours but not layers; Activation Addition leaves the vector unnormalized, while Wollschläger et al. calibrate an optimized refusal direction to match a difference-in-means norm.Source: §6.2 · pp. 7–8
• For directional ablation, sampling unit directions directly within refusal cones avoids the directional bias induced by normalizing arbitrary convex combinations.The sampling procedure is adapted to the projective object consumed by the intervention.Source: §6.2 · p. 8

6.3 Unnamed Reading Points

• The reading point determines the relevant symmetry, and key-space geometric quantities used to explain KeyDiff are not invariants of the attention computation.KeyDiff uses pairwise key cosine for eviction, while ordinary dot-product attention depends on the query-key bilinear pairing.Source: §6.3 · p. 8

Q WK at the parameter level [27]; with RoPE, it is the family W⊤

• Head-space cross-Gram singular values and singular-vector cosines depend on chosen coordinates, whereas comparing column spaces in the residual stream targets an object preserved by the parameter symmetry.The residual-stream column span remains unchanged under the stated query-projection transformation.Source: Q WK at the parameter level [27]; with… · p. 8
• Gauge-related parameter settings compute the same function but can make cosine-based key-space eviction select different tokens.The geometric quantities offered to explain the rule—including key and query cosines, L2 norms, PCA, and log det(KK⊤)—are not invariants under the attention gauge, including with RoPE.Source: Q WK at the parameter level [27]; with… · p. 8

6.4 Inconsistency Within a Single Analysis

• Using one difference-in-means estimate as both an unnormalized displacement and a projective direction changes its G-space, making the combined conclusion stronger than either use establishes.The displacement magnitude matters in V, while the normalized projection uses only the direction in P(V).Source: §6.4 · p. 8
• Li et al. calibrate probe-derived intervention directions using projected activation standard deviation, making the construction compatible with Gsim without identifying V∗ and V outright.The identification of a probe direction with an intervention direction still depends on the chosen similarity geometry.Source: §6.4 · pp. 8–9
• SVCCA's singular-value truncation is not preserved by general invertible transformations, so its later CCA stage cannot restore the affine invariance lost in preprocessing.The recurring issue is that assumptions introduced at one stage may be hidden when stating the final claim.Source: §6.4 · p. 9

7 The Same Omission Beyond Linear Representation

• Representations of different-width models need not share a representation space, so comparisons use structures induced on shared inputs and are indexed by the input distribution.Two representations may induce the same structure on one distribution and differ elsewhere.Source: §7 · p. 9
• With width at least the number of sampled inputs and full-row-rank activation matrices, arbitrary-invertible-linear-invariant similarity cannot distinguish the representations, making finite-sample convergence nearly trivial.Existing evaluations operate closer to Gsim, with effective equivalence set by the alignment measures used.Source: §7 · p. 9
• Nearest-neighbour agreement, kernel-based alignment, and angular quantities inherit geometric commitments, so the hypothesis cannot be assessed before its comparison object and convergence equivalence are specified.The transformation group remains open, and the truth value turns on its choice.Source: §7 · p. 9

8 The Specification Principle

• A claim’s strength is determined by the largest group under which its predicate survives; reporting it under a smaller group loses content without invalidating it.Table 1 is organized by the largest preserving group, not merely a group under which the quantity is defined.Source: §8 · p. 9
• A study should report its G-space, equivariant object-producing procedure, asserted predicate, reading point, and the symmetry of composed stages.These choices should be recoverable from the claim rather than reconstructed from implementation details.Source: §8 · pp. 9–10
• Making assumptions explicit can narrow conclusions, require maps between spaces, or make quantities conditional on a metric or parameterization; Giso is inadmissible when architecture realizes transformations outside it.A property varying across Im ρ belongs to θ and its producing procedure, not fθ, unless stability across seeds and training runs is shown.Source: §8 · p. 10
• Park et al. specify an inner product for orthogonality, while Golechha et al. argue that high-dimensionality and whitening can induce near-orthogonality.Making geometry explicit turns their disagreement into a question about which inner product supports the orthogonality predicate.Source: §8 · p. 10

9 Conclusion

• The Linear Representation Hypothesis is a family of claims distinguished by their equivalence choices, specified through a G-space, equivariant procedure, and predicate subject to architectural constraints.The same omission applies to representation claims beyond linearity.Source: §9 · p. 10

A.1 Groups and Symmetries

• A group action induces equivalence by placing objects in the same orbit, so the chosen group determines which transformations count as equivalent descriptions.A symmetry preserves a specified structure.Source: §A.1 · p. 13

A.2 Linear Spaces

• Vectors and dual vectors are distinct absent added structure; inner-product quantities depend on the chosen inner product, while projective and Grassmannian spaces represent directions and subspaces.Affine transformations preserve affine relations but generally not lengths or angles.Source: §A.2 · pp. 13–14

B Proof of Proposition 3.2(i)

• For dim V ≥ 2, every GL(V)-equivariant map from V* to V is zero; this conclusion fails in dimension one, where nonzero maps exist.The proof uses the stabilizer of a nonzero dual vector and its fixed subspace.Source: B Proof of Proposition 3.2(i) · p. 14

C.1 The rotary commutant

• For distinct rotary frequencies and sufficiently separating relative positions, admissible key-site transformations are independently scaled rotations; coincident frequencies enlarge the commutant to GLm(C) blocks.The admissible transformations are determined by the commutant of the rotary operators.Source: §C.1 · pp. 14–15

C.2 Consequences at a key site

• Independent rotary-plane scales invalidate norms, cosines, and many spectral quantities; only uniform scaling would rescue angular quantities, which the architecture does not impose.The RoPE restriction leaves the Section C.3 criterion invariant but does not restore the excluded quantities.Source: §C.2 · p. 15
• The RoPE restriction preserves the Section C.3 criterion because the key differences and Ω transform oppositely under the restricted gauge.Source: §C.2 · p. 15

C.3 What an invariant version looks like

• The QK gauge preserves realized attention logits, so predicates expressible through those logits remain admissible at these sites.Source: §C.3 · p. 15
• The query-weighted distance measures mean squared attention-logit change between keys and yields identical eviction sets across gauge-related settings, unlike the cosine rule.When the realized query second moment is proportional to identity, it reduces to Euclidean key distance.Source: §C.3 · p. 15

C.4 Head-space quantities

• Cross-Gram singular values are not invariant under general within-head basis changes unless the spanning matrices are orthonormalized, although residual-stream subspaces remain unchanged.Fokou [28] interpret the singular values as principal angles between head subspaces.Source: §C.4 · pp. 15–16