PaperlayerView exact revision on arXiv

Source-linked AI summary

From the Transformer to a Formal Semantics of Coordinated Transformations

Marco Giunti, Fabrizia Giulia Garavaglia

arXiv:2610.39735v1cs.AIcs.CLcs.LO

TL;DR

The paper asks how to formally describe the semantic organization of coordinated transformation systems without tying the account to Transformer-specific mathematics. It develops a semantics coordinating vectors, parameter structures, and property-level maps, and concludes that attention and residual semantics can be invariant across mathematical instantiations of the same local organization.

  • Problem

    The paper asks what formal semantic organization coordinated transformations exhibit once interpretations are specified, while distinguishing this question from explaining representation or its grounding.

    Source: §2.2 · p. 3

  • Method

    The framework assigns vectors extensional reference and ordered intensional interpretations, gives parameters relative second-level interpretations, and coordinates transformations with property-level maps.

    Source: §1 · pp. 1–2

  • Results

    Attention and residual semantics remain invariant across different mathematical instantiations of the same local organization, and both are modular abstract system types.

    Source: §9.5 · p. 27

  • Takeaways & Limitations

    The framework separates shared algebraic-semantic organization from functional architecture and supports semantic analysis beyond the standard Transformer’s particular mathematics.

    Source: §1 · p. 2, §9.5 · p. 27

    The framework does not explain what makes a state representational, how its content arises, or what grounds its representational relations.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

We develop a formal semantics for a general class of coordinated systems of parameterized transformations, taking the Transformer as a paradigmatic starting point but abstracting from its particular mathematical instantiation. Representational vectors receive both extensional reference and ordered intensional interpretations, while parameter structures acquire relative, second-level interpretations that determine how argument properties are preserved, replaced, or combined. This yields the notion of a transforming concept: a parameterized transformation whose numerical action is coordinated with a corresponding property-level map. We then distinguish the common algebraic-semantic core of these systems from their architecture-specific functional organization, allowing different mathematical instantiations and mechanical realizations of the same abstract system type. The framework is applied to attention heads and residual connections, providing semantic explanations that depend only on the shared algebraic-semantic core and their local functional organization and therefore generalize beyond the standard Transformer. These analyses also reveal the modularity of the framework: complex architectures can be decomposed into simpler semantically coordinated subsystems whose organization may recur across different abstract system types.

1 Introduction

The paper generalizes a concept-based interpretation of Transformer processing into a formal semantics for coordinated transformation systems. It defines transforming concepts and separates shared semantic organization from architecture-specific organization and realization.

1 Introduction

  • The paper asks what formal semantics can describe coordinated systems of transformations beyond the standard Transformer and Transformer architectures.It allows parameterized transformations to operate singly or in parallel and to connect through direct or operational output–input relations and output–parameter relations.Source: §1 · p. 1
  • Representational vectors refer extensionally to objects, while their positions receive property-based intensional interpretations.Parameter structures receive relative, second-level interpretations specifying how argument properties are preserved, replaced, or combined.Source: §1 · p. 1
  • The semantic account does not address whether or how its interpretations are representationally grounded.Source: §1 · p. 2
  • A parameterized transformation is a transforming concept when its numerical action coordinates with a property-level map determined by its relative second-level interpretation.Intensional invariance is independent of whether numerical values are preserved, and dimensional preservation does not entail intensional invariance.Source: §1 · p. 2
  • The framework separates a shared algebraic-semantic core from architecture-specific functional organization, permitting different mathematical instantiations and mechanical realizations of one abstract architecture.Attention heads and residual connections are analyzed using the common core and their local organization, rather than the standard Transformer’s particular mathematics or overall architecture.Source: §1 · p. 2

2 From the Transformer to Coordinated Systems of Transformations

Starting from an interpretation of Transformer processing as concepts transformed by concepts, the paper asks how to formalize the semantic organization of a broader class of coordinated systems. It distinguishes that task from explaining representation or its grounding.

2.1 Transformee and transforming concepts

  • In TR1, vectors representing tokens were called transformee concepts, while parameterized transformations acting on them were called transforming concepts.The account distinguished static parameters fixed through training from dynamic parameters generated from current input during inference.Source: §2.1 · p. 2

2.2 The semantic problem

  • The paper asks how mathematical structures and transformations are semantically organized once interpretations are specified, rather than what makes internal states represent.Source: §2.2 · p. 3
  • The framework formalizes assigned semantic interpretations but does not explain what makes a state representational, how its content arises, or what grounds its representational relations.It takes vectors to refer to objects and positions to be interpreted by properties, while leaving room for different theories of representation and metasemantic grounding.Source: §2.2 · p. 3

2.3 Beyond the Transformer

  • The paper extends the formal-semantic question beyond Transformer-specific mathematics to systems coordinating parameterized transformations and operations.Outputs may become later inputs, pass through operational blocks, be regrouped, or supply later parameters.Source: §2.3 · p. 3

3 Generalized Transformations, the Value Algebra, and Their Coordination

The paper constructs a generalized value algebra and uses it to define parameterized transformations and the ways they can be organized into coordinated systems. The system description distinguishes direct transmission, function-mediated modification, and outputs used as later parameters.

3 Generalized Transformations, the Value Algebra, and Their Coordination

  • The generalized value algebra abstracts from Transformer-specific numerical operations and supports a common treatment of matrix transformations, componentwise transformations, and vector addition.Its blank element represents absence of a contribution and is not identified with zero.Source: §3 · p. 3, §3.1 · p. 4, §3.2 · p. 6

3.1 Generalized values, vectors, and matrices

  • The algebra permits arbitrary binary operations without assuming they are associative, commutative, distributive, or distinct.Additive and multiplicative extensions give the blank element identity and absorbing roles, respectively.Source: §3.1 · p. 4

3.2 Parameterized vector transformations and the value algebra

  • Matrix-parameterized transformations combine input-to-output contributions, while vector-parameterized transformations act componentwise; the diagonal case unifies componentwise multiplication and vector addition with matrix-defined transformations.In the diagonal case, the componentwise transformation depends on the multiplicative-role operation, not the additive-role operation.Source: §3.2 · pp. 5–6

3.3 Parallel transformations

  • Parallel transformations share input and output arities but need not be identical; different parameter vectors can apply different transformations to different rows.This covers positional-encoding, masking, and residual additions in standard Transformer processing.Source: §3.3 · pp. 6–7

3.4 Direct output–input interconnections

  • A direct output–input interconnection transmits output values to a later input without further mathematical modification.Routing may regroup or redirect values, but does not mathematically modify them.Source: §3.4 · p. 7

3.5 Operational blocks and operational output–input interconnections

  • An operational output–input interconnection mediates the transfer of values to a later transformation through a function that modifies them.Operational blocks are non-parameterized vector-to-vector functions, distinct from transformations in the paper’s technical sense.Source: §3.5 · pp. 7–8

3.6 Output–parameter interconnections

  • An output–parameter interconnection uses output values as parameters of a later transformation rather than as its input argument.The resulting coordinated system may combine parameterized transformations and operations through direct, operational, and output–parameter interconnections; its characterization remains semantic-free at this stage.Source: §3.6 · p. 8

4 An Intensional–Extensional Formal Semantics for Representational Vectors

Representational vectors combine object reference with ordered, position-specific property interpretations, while parameter structures receive intensional interpretations without referring to objects. The resulting semantics specifies how vector values relate to properties while leaving the grounding of those assignments open.

4.1 Objects, values, and properties

  • Properties are intensional entities whose extensions can vary across occurrences, even when the same property appears in multiple positions or vectors.Distinct properties may also coincide extensionally in particular occurrences.Source: §4.1 · p. 8, §4.3 · p. 9
  • Parameter-position interpretations use a special preservation element that is not an ordinary property of objects.The null property corresponds exactly to a blank parameter value; the preservation element is reserved for parameter structures.Source: §4.1 · pp. 8–9, §4.4 · p. 10

4.2 Intensional interpretation of vector positions

  • Each representational-vector position pairs a numerical value with an intensional property, and these assignments remain distinct.A null property corresponds exactly to a blank value.Source: §4.1 · p. 8, §4.2 · p. 9

4.3 Extensional reference and features

  • The paper distinguishes a property as an intensional entity from a feature as that property considered for a represented object or occurrence.A vector is called a representational vector, or transformee concept, when it has both an intensional interpretation and an extensional reference.Source: §4.3 · p. 9

4.4 The interpretation of parameter vectors and matrices

  • A parameter vector or matrix has position-wise intensional interpretations but no extensional reference to an object.Its values therefore do not represent features of an object.Source: §4.4 · pp. 9–10, §4.5 · p. 10

4.5 Formal semantics, representation, and grounding

  • The formal semantics specifies representation structurally but does not explain what makes a property assignment appropriate or what grounds a vector’s reference.Parameter values lack the occurrence-relative relation between a property’s extension and an object.Source: §4.5 · p. 10
  • A representational vector combines an extensional reference to an object with an ordered intensional interpretation of its positions.Each property’s extension is defined relative to its occurrence in that vector and position.Source: §4.3 · p. 9, §4.5 · p. 10

5 Transformational Semantics and Transforming Concepts

The section coordinates transformations of values with maps over properties, defining when parameterized transformations count as transforming concepts. Its semantics distinguishes property preservation, replacement, and combination, and separates these semantic relations from parameter genesis and dimensional change.

5.1 The property algebra

  • The property algebra makes projection preserve, replace, or omit an argument property, while property sum combines implicated properties idempotently.Repeated contributions interpreted by the same property still contribute distinct numerical values.Source: §5.1 · p. 11

5.2 Relative interpretations and second-level interpretations

  • A matrix-defined point of view can combine implications from multiple argument properties, whereas vector parameterization is restricted to the corresponding property.Vector parameterization can preserve that property, replace it with an implicated one, or contribute no property.Source: §5.2 · pp. 12, 14
  • Projection can replace argument properties with distinct properties they implicate, while property sum combines those implications without itself creating or removing abstraction.The framework interprets progressively more abstract representations through this replacement-and-combination account.Source: §5.2 · p. 13

5.3 Transforming concepts

  • Operational blocks are not transforming concepts because they lack parameter structures with relative interpretations and parameter-induced property maps.They preserve the input’s reference and ordered interpretation while allowing values and occurrence-relative extensions to change.Source: §5.3 · pp. 14–15

5.4 Intensional invariance and dimensionality

  • A transforming concept may preserve or change its ordered interpretation, and dimensional preservation alone does not guarantee intensional invariance.Dimension-changing transformations are necessarily intensionally changing; residual additions are identified as a dimension-preserving case of intensional change.Source: §5.4 · p. 15

5.5 The genesis of values and interpretations

  • The genesis of parameter values and interpretations is independent of whether a transforming concept is intensionally invariant.The paper’s genetic classification is not exhaustive, and it does not adjudicate among accounts grounding interpretive and representational relations.Source: §5.5 · p. 16
  • The first three genetic configurations have Transformer instances, while the fourth is formally permitted but has no clear instance.Learned bias and scaling vectors, trainable positional vectors, sinusoidal positional vectors, masking, rescaling, and temperature are examples of intensionally invariant configurations.Source: §5.5 · p. 16

5.3 Transforming concepts

  • A parameterized transformation is a transforming concept when its value-level action is coordinated with a property-level map determined by a relative, second-level parameter interpretation.The two maps remain formally distinct, while their results are semantically coordinated.Source: §5.2 · p. 13, §5.3 · p. 14, §5.5 · pp. 16–17

6 Semantic Composition in Coordinated Systems

Semantic composition is specified through compatibility between local semantic structures and a system’s architectural organization. The account distinguishes uniform operations and several interconnection types, requiring no additional global semantic mechanism.

6 Semantic Composition in Coordinated Systems

  • The section combines the architecture of coordinated systems with the semantics developed for vectors and transformations rather than defining a recursive semantics for arbitrary compositions.It identifies semantic conditions for the principal architectural forms of coordination.Source: §6 · p. 17

6.1 Interpretive uniformity in operations

  • A semantically coordinated operation requires its parallel transformations to share common ordered interpretations of corresponding input and output positions.The individual transformations, parameter values, and relative interpretations may differ.Source: §6.1 · p. 17

6.2 Direct interconnections and semantic transmission

  • Direct interconnections transmit already interpreted vectors or reorganize them according to the architecture, but routing alone cannot determine a reference for vectors with different referents.Transformer Concat illustrates concatenating vectors and property tuples for the same token.Source: §6.2 · p. 17

6.3 Operational interconnections and intensional invariance

  • Operational interconnections may modify values while preserving the transmitted vector’s extensional reference and ordered intensional interpretation.The operational block is intensionally invariant.Source: §6.3 · p. 17

6.4 Output–parameter interconnections and second-level interpretation

  • An output–parameter interconnection changes a transmitted structure’s semantic role, assigning it a relative, second-level interpretation as a parameter rather than retaining its output interpretation.Referential provenance may remain relevant, depending on the architecture.Source: §6.4 · p. 18

6.5 Semantic coordination at system level

  • A coordinated system is semantically coordinated when its local semantic constraints are compatible with its architecture of transformations, operations, blocks, and interconnections.System-level coordination follows from how locally interpreted structures are connected, without a further global semantic mechanism.Source: §6.5 · p. 18

7 Abstract Functional Architecture, Mathematical Instantiation, and Realization

The paper separates a shared algebraic-semantic core from architecture-specific functional organization, then distinguishes abstract system types from their mathematical instantiations and realizations. This permits different implementations to preserve an abstract type while allowing distinct routes from abstract systems to the physical level.

7.1 Abstract functional architecture

  • An abstract functional architecture combines the shared algebraic-semantic core with an architecture-specific organization of transformations, operations, interconnections, and operational-block roles.The paper specifies the core in detail but only some constraints on admissible organizations, leaving room for multiple architectures.Source: §7.1 · p. 19

7.2 Mathematical instantiations

  • A mathematical instantiation supplies a value-algebra model and functions for the architecture’s roles, while leaving property domains and semantic assignments open to admissible variation.The value domain is fixed, but its operations may vary; property domains may vary, and property sum is only partially constrained.Source: §7.2 · pp. 19–20

7.3 Functional-architectural invariance

  • Different numerical operations or functions can instantiate the same abstract architecture when its algebraic-semantic structure and functional organization remain preserved.The abstract system type is individuated by functional organization within the common framework, not by particular operations or functions.Source: §7.3 · p. 20

7.4 Abstract system types and the Transformer

  • The Transformer is one abstract system type sharing the common core; other types may share that core while differing in organization and interconnections.The standard Transformer serves as an exemplar, not a restriction on the framework.Source: §7.4 · p. 20

7.5 Mathematical instantiation, mechanical realization, and the physical level

  • A mathematical instantiation and a mechanical realization are distinct relations to an abstract type, and neither determines the other.A type may admit different instantiations and realizations independently.Source: §7.5 · p. 20
  • Computational simulation and physical construction are separate routes from an abstract system to the physical level.Simulation programs the functional organization and numerically evaluates the specified functions; construction realizes a mathematically instantiated mechanical organization in material components.Source: §7.5 · p. 21

8 Attention and Residual Connections

The paper explains attention heads and residual connections through the shared algebraic-semantic core and their local functional organizations, rather than specific numerical instantiations. The analyses characterize attention as token-indexed values of a shared property and residual connections as capable of changing intensional interpretation despite dimensional preservation.

8.1 Attention scores and token-indexed points of view

  • Attention constructs one more abstract property from key interpretations, whose values differ across token-indexed points of view.When key properties differ from corresponding query properties, projection replaces at least one query property before combination; the resulting property is more abstract than each query property.Source: §8.1 · p. 23

8.2 Value aggregation and the semantic output of attention

  • Attention uses those values to regulate value-vector contributions while preserving the value representation’s ordered intensional interpretation and the query-originating token’s reference.Interpretive uniformity and idempotent property sum help preserve the output properties; different mathematical instantiations can change numerical values while preserving semantic organization under the stated constraints.Source: §8.2 · p. 24

8.3 Residual connections and intensional change

  • A residual output takes the module output’s intensional interpretation; it is intensionally invariant exactly when that interpretation matches the input’s.When the interpretations differ, the residual connection remains dimension-preserving but is intensionally changing.Source: §8.3 · pp. 24–25

8.4 Complementarity and modularity

  • Attention and residual connections are modular abstract types, so their semantic analyses can recur in other architectures containing structures of the same functional kinds.The complementary analyses show that semantic organization may persist across mathematical variation, while mathematical similarity need not entail semantic identity.Source: §8.4 · p. 25

9 Philosophical Significance, Scope, and Conclusion

The paper gives formal semantics an analytical role before questions of representational grounding, then extends semantic analysis from represented contents to the transformations acting on them. It uses that framework to distinguish shared semantic organization from mathematical and physical realization, while identifying limits and open questions.

9.1 Formal semantics before representation and grounding

  • Formal semantics characterizes interpretation once specified but neither decides which account of representation is correct nor explains why particular properties or referents are assigned.C1 separates extensional vector reference, ordered intensional interpretation, and occurrence-relative property extension from the grounding of genuine representational relations.Source: §9.1 · p. 25

9.2 From representing contents to transforming them

  • The analysis extends from represented contents to transformations: parameter structures interpret argument properties, and transforming concepts coordinate numerical action with property-level maps.Such structures can specify preservation, replacement, or combination of properties without themselves referring to objects.Source: §9.2 · p. 26

9.3 Sameness, variation, and modular explanation

  • Different mathematical systems can instantiate the same abstract functional type when they preserve its algebraic-semantic core and architecture-specific organization.Attention and residual connections are modular abstract system types: their semantic organization depends on local functional organization and the shared core, not mathematical similarity alone.Source: §9.3 · pp. 26–27
  • Computational implementation and physical construction are distinct routes to the physical level, not necessary successive stages.The paper distinguishes these routes while leaving their further realization questions open.Source: §9.3 · p. 26, §9.4 · p. 27

9.4 Scope and open questions

  • The framework does not derive representational status or content, and its preliminary genetic classification does not explain how systems acquire content.It also leaves generalized neural realizability and the relation to Conjecture T for future work, without theorizing mechanical, neural, or physical realization.Source: §9.4 · p. 27

9.5 Conclusion

  • The proposal begins from TR1’s Transformer interpretation and asks whether its semantic organization can be generalized beyond the standard Transformer’s mathematics and neural organization.The conclusion frames the result as a basis for further questions about realization, implementation, and construction, rather than answers to them.Source: §9.5 · p. 27
  • The framework distinguishes vector semantics, transforming concepts, and abstract functional architecture, then applies these distinctions to attention and residual connections.Its conclusion is conditional: the results concern coordinated transformations interpreted in the specified manner, not every vector transformation or mathematically adequate model.Source: §9.5 · p. 27

Related papers

This paper cites

Loading 2610.39735v1…