4-Tensor Attention Model for Semantic Physical Reality

1.1 Implementation Scheme

• The intended pipeline compresses frames into semantic states, predicts future states with 4-tensor attention, then renders those predictions back into frames.This paper implements and evaluates only the attention block.Source: §1.1 · p. 4, Figure 1 · p. 5

1.2 Scope of This Paper

• The evaluation isolates the update from visual encoding and decoding, using narrative windows as a semantic-layer next-sentence task.The image-to-text encoder, text-to-image renderer, and planner are outside the test.Source: §1.2 · p. 4

1.3 The Semantic State Sequence

• A semantic state is a fixed-length token sequence, and a window collects the most recent states along the temporal axis.The running example uses T = 3 and N = 8; experiments use T = 4 and N = 16.Source: §1.3 · p. 5

1.4 Rank-4 Representation

• The rank-4 representation pairs token position x with time t and separates semantic and temporal-context fibers, while allowing their internal interactions.The fibers are distinct tensor factors, but the general bilinear projection permits cross-fiber interaction; the architecture does not force transition rules to separate from content.Source: §1.4 · pp. 5–6

1.5 Score Tensor and Joint Softmax

• Each query-key score contracts both fibers, and a single softmax normalizes jointly over all key positions and time slices without a temporal causal mask.Padding keys are masked; otherwise every query can attend to every slice in the window.Source: §1.5 · p. 6

1.6 Attention Output

• The attention output is the A-weighted sum of values over the key grid.Source: §1.6 · p. 6

1.7 Prediction Targets and the Learning Objective

• The model reads logits at each position and trains against the window shifted by one slice, but only the final slice predicts a sentence absent from the input.Because there is no temporal causal mask, earlier targets are present in later input slices; the final-slice prediction is scored in one parallel forward pass.Source: §1.7 · p. 7

1.8 Autoregressive Inference

• At inference, the predicted state is fed into a shifted window and the same weights predict the next state, but this paper reports only single-step results.Ground-truth states from S(3) onward never enter the window during repeated rollout.Source: §1.8 · p. 7

2.1 Overview of the Tensor Pipeline

• The state tensor at temporal slice t is the base unit, and the rank-4 input is built sequentially, with each stage adding a signal to a designated fiber.Source: §2.1 · p. 8

Stage A: Initialization by Dual-Dictionary Token Embedding

• Each token begins as the outer product of vectors from separate semantic and temporal-context dictionaries, giving it rank-3 structure across both fibers.Source: Stage A: Initialization by… · p. 8

Stage B: Spatial Position Injection

• The spatial position embedding lies entirely in the semantic fiber and broadcasts identically across the temporal-context fiber.Source: Stage B: Spatial Position Injection · p. 8, §2.1 · p. 8

Stage C: Temporal Stacking

• The T sentence tensors are stacked along a new temporal axis, yielding the rank-4 configuration (N, dx, T, dt).Source: Stage C: Temporal Stacking · p. 9

Stage D: Temporal-Order Injection

• The temporal-order embedding uses both x and t; a column-independent signal would fail to distinguish the same entity's changing causal roles across time.Source: §2.3 · p. 9

2.3 Joint-Phase Temporal Embedding

• The joint-phase embedding uses phase x + t, while adding the spatial embedding distinguishes positions that share that phase.Source: §2.3 · p. 9

2.4 Final Assembly of the Input Tensor

• The assembled input adds S, P, and the joint-phase temporal embedding as one tensor, while retaining the two fibers as distinct factors that the attention weights may mix.Source: §2.4 · p. 10

3 Forward Pass and Pipeline Overview

• The forward pass proceeds from attention through residual and MLP blocks, normalization, vocabulary readout, and cross-entropy, with a temporal-index shift aligning predictions to target sentences.Source: §3 · p. 10, §3.1 · p. 10, §3.8 · p. 14

3.2 Residual After Attention

• The attention output shares the input's (x, n, t, q) space, and dropout is applied only to the residual branch, preserving the identity pathway.Source: §3.2 · p. 11

3.3 Event Position MLP Block

• At each (x, t), the MLP jointly flattens the semantic and temporal-context fibers without assuming separability, using hidden width dff = 4dxdt.Source: §3.3 · p. 11

3.4 Residual After MLP and Readout-Boundary Normalization

• Readout-boundary LayerNorm acts locally over the joint fibers; the paper says it stabilizes logits because unscaled residual magnitudes can saturate the vocabulary softmax and collapse the cross-entropy gradient.Source: §3.4 · pp. 11–12

3.5 Vocabulary Readout and Structured Factorization

• The vocabulary readout factorizes each token's map into a rank-1 pair, tying its semantic factor to the token embedding and learning a temporal-context factor.Source: §3.5 · p. 12

3.5.1 Sequence-Independent Matrix Pre-computation

• A sequence-independent matrix pre-computation avoids materializing an intermediate tensor of size O(NTdt|V|) for the vocabulary logits.Source: §3.5.1 · p. 12

3.6 Structural Constraints Imposed by the Loss Objective

• The loss preserves spatial position x and time t while contracting away latent coordinates n and q; the implemented readout replaces the general bilinear map with a rank-1 factorization.Source: §3.6 · p. 13

3.8 From Embedding-Level Loss to Cross-Entropy Loss

• Vocabulary softmax is computed independently at each (x, t), and cross-entropy trains each logit slice at t against the sentence at t+1.Source: §3.7 · p. 13, §3.8 · pp. 13–14

4 Enabling Construction of the Forward Pipeline

• The implementation fixes operation order, in-memory axes, numerical guards, and parameter definitions; preserving tensor shapes alone may still yield a different function if checks are omitted.Source: §4 · p. 14

4.1 Index convention and storage

• The stored layout uses a shared flattening convention for attention, the MLP, normalization, and vocabulary projection, so fiber order matters even when the product dxdt matches.Source: §4.1 · p. 14

4.2 Parameters, buffers, and initialization

• Learned parameters include semantic and temporal token embeddings, full-fiber attention maps, per-layer normalization and MLP weights, and a final LayerNorm; positional tables are fixed sinusoids.Source: §4.2 · p. 15

4.3 Step 1: construction of the input tensor

• Input construction adds token content, spatial position, and a joint-phase temporal embedding based on x + t; stacking the sentences adds no arithmetic.Source: §4.3 · pp. 15–16
• Padding embeddings begin at zero, but added positional signals make padding slots nonzero; the implementation identifies them from token IDs, not activation norms.Source: §4.3 · pp. 15–16

• Each head applies full-fiber query, key, and value maps to the input, contracting over the semantic and temporal-context fibers.Source: §4.4 · p. 16
• The score contracts query and key fibers and is evaluated over the joint key grid (x′, t′), producing a score tensor of shape (B, N, T, N, T).Source: §4.4 · p. 16
• Padding is masked on key axes with the finite minimum score value; all-padding queries therefore receive a uniform distribution, while padding queries are excluded from loss.Source: §4.4 · p. 17
• A joint softmax normalizes the key axes together, and the resulting attention weights contract with values to return an output matching the residual stream's shape.Source: §4.4 · p. 17

4.5 Multi-head sum and the Pre-LN layer

• Each head uses a normalized copy of the residual as input, while residual additions use the unnormalized stream.The MLP also operates on the flattened fiber at each position, with hidden width 4 dxdt.Source: §4.5 · pp. 17–18, W2 GELU(W1v + b1) + b2 · p. 18

W2 GELU(W1v + b1) + b2

• The MLP output is added to the post-attention stream through dropout, and each of L layers is followed by a final LayerNorm on the residual.The final norm is additional to the branch normalizations because the residual stream is a raw sum.Source: §4.5 · p. 17, W2 GELU(W1v + b1) + b2 · p. 18

4.5 Multi-head sum and the Pre-LN layer

• The attention and MLP branches use distinct LayerNorm modules and independent dropout masks at the same rate; these are the only dropout sites.Residual additions retain the unnormalized stream.Source: §4.5 · p. 18

4.6 Readout and the training loss

• The readout ties the semantic embedding Esem through W′ rather than allocating a (dx, dt, |V|) parameter tensor.The (n, q) fiber pair is eliminated at readout and does not appear in the loss.Source: §4.6 · p. 18, Table 2 · p. 19
• Targets are shifted by one sentence, and mean token cross-entropy ignores padding targets while retaining the separate forward key mask.The reported last-slice score uses the final sentence axis, S(4) for windows with T = 4.Source: §4.6 · pp. 18–19

4.7 End-to-end training procedure

• Each example is split into an input prefix and shifted target, then passed through the layers, final norm, tied logits, and ignore-index cross-entropy.The target is permuted to align its axes with the logits before loss evaluation.Source: §4.7 · p. 19
• Teacher forcing supplies ground-truth sentences for every training loss term; predicted sentences appear only in gradient-free diagnostic rollouts.Evaluation uses the same forward procedure with dropout disabled.Source: §4.7 · p. 19

4.8 Faithfulness checks

• The activation layout is (B, N, T, dx, dt), with the shared flattening index ϕ(n, q) = n dt + q used across the MLP, final norm, and readout.Swapping the fiber order can preserve the product size while changing the intended mapping.Source: §4.8 · p. 20
• Padding masks derive from token identifiers, mask keys rather than queries, and are shared across layers within one forward call.The MLP receives no mask, and padding targets are removed from loss without replacing the forward key mask.Source: §4.8 · p. 20
• The readout materializes M from Esem and W′ before multiplication, avoiding a much larger intermediate activation.Training uses logits and integer targets directly; vocabulary softmax and one-hot targets are diagnostic only.Source: §4.8 · p. 20

5 Reference Implementation

• The implementation’s reference listing follows the design operations in pipeline order while omitting shape checks, optimizer wrapping, and diagnostic warnings.The listed operations are arithmetic source statements with shortened comments.Source: §5 · p. 20

5.1 Input construction

• Token embeddings form an outer-product fiber tensor, then spatial and temporal position embeddings are broadcast over their respective axes.The returned input layout is (B, N, T, dx, dt).Source: §5.1 · p. 21

5.2 Rank-4 attention head

• Rank-4 attention scores are reshaped so one softmax normalizes jointly over the key grid of spatial and temporal positions.Padding is applied on key axes before the joint softmax.Source: §5.2 · pp. 21–22

5.3 Multi-head sum, Pre-LN, and the MLP

• Each head retains a full (dx, dt) fiber, and head outputs are summed without a 1/sqrt(H) factor.Attention and MLP branches use pre-normalization and residual additions.Source: §5.3 · p. 22

5.4 Tied readout, shifted loss, and the model step

• The readout constructs M from Esem and W′, multiplies flattened outputs by M, and computes cross-entropy against shifted targets.The reported token statistics use non-padding tokens from S(T) only.Source: §5.4 · pp. 22–23

6.1 Joint Softmax Jacobian and Score Tensor Gradient

• The joint-softmax Jacobian yields a row-wise score gradient that subtracts the attention-weighted average incoming gradient across the key/value grid.Rows remain algebraically independent because each probability row depends on scores sharing its query index.Source: §6.1 · p. 24

6.2 Gradients with Respect to Q, K, and V

• Query and key gradients sum score gradients over opposite index pairs, reflecting their distinct row and column roles.The value gradient is the transpose of the forward contraction, distributing upstream gradients through attention weights across queries for each key/value position.Source: §6.2 · p. 24

6.3 Chain Rule to the Projection Weights

• Gradients for the rank-4 Q, K, and V weights are outer products of upstream projection gradients and normalized inputs, summed over positions.The gradient then continues from the normalized input through LayerNorm to the residual.Source: §6.3 · p. 25

6.4 Gradient-Descent Update Rule

• The displayed joint parameter update is plain gradient descent for exposition, while training uses AdamW.The update is stated in terms of learning rate η and cross-entropy L.Source: §6.4 · p. 25

7 Comparative Baseline: Standard 1-Dimensional Autoregressive Attention

• The baseline predicts the same future semantic state from the historical context as the 4-tensor model, while serving as a representation comparison rather than an ablation.The target is excluded from the input to prevent causal leakage.Source: §7 · pp. 25–26

7.1 1-D Sequence Packing and Standard QKV Architecture

• The baseline removes padding and concatenates sentences in temporal order, then uses lexical embeddings, 1-D positional encodings, and causal self-attention over the flattened string.It relies on positional encodings to approximate temporal boundaries rather than representing sentence and spatial axes explicitly.Source: §7.1 · p. 26

7.2 Informational Parity via Free-Running Autoregressive Decoding

• To preserve informational parity, the baseline generates each target token from the context and its own earlier predictions, without access to gold target tokens.An error in one prediction becomes part of the context for the next step.Source: §7.2 · p. 27

7.3 Architectural Alignment and Capacity Matching

• At H = 4 and L = 2, the models have nearly matched total parameter counts but allocate parameters differently between attention and token tables.The 4-tensor model has 100.7M attention and 4.7M token-table parameters; the baseline has 33.6M and 75.2M, respectively.Source: §7.3 · p. 27, Table 3 · p. 28

7.4 Baseline design pipeline

• The baseline’s free-running training loop scores target tokens and appends its argmax predictions, while the 4-tensor model’s loss covers all T target sentences.The baseline loss covers only the tokens of S(T), whereas only S(T) is absent from the 4-tensor input.Source: §7.4 · p. 29

7.5 Code for the fair comparison

• The comparison implementation packs the context after a BOS token and keeps the target sentence, followed by EOS, outside the input prefix.The code description identifies the same window, excludes gold S(T), and aligns the content-token denominator with the 4-tensor last-slice score.Source: §7.5 · p. 30

8 Experimental Setup

• The experiments train both models on semantic windows and report mean cross-entropy on non-padding tokens of the last sentence.Both models use Pre-LN, dropout on both residual branches, and a final LayerNorm before tied readout.Source: §8 · p. 30

8.1 Corpus and windows

• The ROCStories pool contains 101,671 stories, with windows of four context sentences and a fifth target sentence, each padded or truncated to 16 tokens.The vocabulary has 36,711 tokens and is built over the full pool, including validation.Source: §8.1 · p. 31

8.2 Optimization and checkpoint rule

• Both models use d_x = 64, d_t = 32, dropout 0.1, and AdamW at learning rate 3 × 10^-4 with weight decay 0.01.Checkpoint selection uses different losses: teacher-forced window loss for the 4-tensor model and free-running S(4) loss including EOS for the baseline; reported last-sentence cross-entropy excludes EOS.Source: §8.2 · p. 31

9 Experiment Data

• Table 5 reports LS(4) over 32,173 non-padding tokens of the last sentence, and a positive ∆ means lower 4-tensor cross-entropy.Source: §9 · p. 31, Table 5 · p. 32
• 5.845, 5.694, and 5.667 were the lowest logged baseline LS(4) values, each above the corresponding 4-tensor minima of 5.562, 5.554, and 5.531.Source: §9 · p. 32, Figure 2 · p. 33
• Validation checkpoint minima were 1.305, 1.305, and 1.314 for 4-tensor, versus 5.481, 5.310, and 5.297 for baseline selection loss.Source: §9 · p. 32
• 5.562 vs 5.873, 5.554 vs 5.701, and 5.531 vs 5.667 last-slice cross-entropies favor 4-tensor over baseline at H=2, L=2; H=4, L=2; and H=4, L=3, respectively.Source: §9 · p. 32, Table 5 · p. 32
• At H=4, L=2, parameter counts are 172.5M vs 175.9M; on two RTX 3080 GPUs, runs took 2.4 vs 45.2 hours.Source: §9 · pp. 32–33

10 Discussion

• The rank-4 factorization separates semantic and temporal-context fibers at input and readout boundaries while allowing arbitrary cross-fiber interaction internally.Source: §10 · p. 33
• Nested containment is proposed as a minimal setting to test whether coupled agent–environment dynamics can acquire an operation that fills its own argument slot.Source: §10 · pp. 33–34
• Figure 3 separates training-loss panels: the upper row is the 4-tensor shifted-window loss, while the lower row is the baseline free-running loss on S(4), including [EOS].Source: Figure 3 · p. 34
• A multi-step rollout would score later sentences under free-running inputs, but no value of this metric is reported.Source: §10 · p. 34
• The methods are identified as subject to a pending Korean patent application, filed October 6, 2026.Source: §10 · p. 34
• Figure 4 separates validation shifted-window loss for 4-tensor from baseline selection loss on S(4), with open circles marking selected minima.Source: Figure 4 · p. 35