PaperlayerView exact revision on arXiv

Source-linked AI summary

The Limits of Binding in Dual Encoders

Kin Ian Lo

arXiv:2608.15971v1cs.LGcs.CLcs.CV

TL;DR

Dual encoders often fail to bind attributes to the correct objects, raising questions about whether this reflects fundamental geometric limits or contingent training and coding constraints. The paper proves conditional depth, objective, and geometry obstructions, finding that deployed failures are primarily incentive and code-structure limits while a depth ceiling remains at CLIP scale.

  • Problem

    It remains unclear which limits on compositional binding are fundamental to dual-encoder geometry rather than attributable to architectures, objectives, or data.

  • Method

    The paper establishes three conditional, measurable obstructions involving recursive-code depth, contrastive-objective incentives, and the smoothness–binding geometry frontier.

  • Results

    A depth ceiling remains at natural-language depth for CLIP-sized models, while deployed encoders occupy only ≈25–35% of the smoothness ceiling and its per-item cap tracks SugarCrepe difficulty at r = 0.99.

  • Takeaways & Limitations

    Binding failure in deployed dual encoders is primarily an incentive and code-structure limit, whereas the depth ceiling persists after those contingent factors are fixed.

  • Takeaways & Limitations

    The depth theorem retains one explicit concentration-estimate gap, and the frontier measurement covers two-object color binding rather than relations or counting.

Abstract

from arXiv · show

Dual-encoder models such as CLIP score an image-caption pair by a single inner product of two independently computed unit vectors, and fail at binding, often scoring near chance when asked to distinguish "a red car and a blue dog" from "a blue car and a red dog". We give a mathematical account of when this failure is necessary and when it is contingent. Working within the ideal-encoder framework proposed by Kang et al., we first show the relevant axioms are satisfiable, so every impossibility must enter through an added, checkable hypothesis. We then prove three such obstructions. Depth: for recursive role-binding codes the swap margin obeys an exact law $m(D) = 2b^{-D}$ in the nesting depth D, with a finite-dimension version holding up to one explicitly flagged concentration estimate; the resolvable depth grows only logarithmically in the dimension and is single-digit at CLIP scale, the nesting depth of ordinary language. Objective: architecture-free throttle theorems showing that the contrastive objective's entire reward for binding is bounded by the rate at which training contrasts a caption against its own swap, a rate that vanishes at web scale, and that exactly reversed binding costs only that rate times the mean binding margin; both are verified in simulation. Geometry: a tight smoothness-binding frontier: the closer the two swap-related captions must embed to a shared paraphrase anchor, the smaller the binding margin can be, with an exact constant. Measuring its text-only diagnostic across 18 deployed text encoders, every model sits at roughly 25-35% of its ceiling, and the induced per-item ceiling tracks SugarCrepe's subset difficulty at r = 0.99. Binding failure in deployed dual encoders is thus not a dimension or smoothness limit today, but an incentive and code-structure limit, with a proved depth ceiling that remains once those are fixed.

1 Introduction

Dual encoders use a single image–text inner product and empirically fail compositional binding on swapped attribute–object captions. This paper shows that binding is not unconditionally impossible, but is constrained by conditional depth, objective, and smoothness obstructions.

  • Motivation: Single-inner-product dual encoders often score swapped captions nearly as highly as correct captions, with ARO and Winoground near chance and SugarCrepe swaps weakest.These models support retrieval, zero-shot classification, and image-generation conditioning, but fail on negatives that permute attribute assignments.
  • Possibility framework: The axioms are satisfiable as low as N = 2k, so no unconditional theorem says dual encoders cannot bind; every limitation requires a natural, checkable hypothesis.Random factored sign codes realize the axioms at deployed scales.
  • Depth: m(D) = 2 b^-D exactly for recursive role-binding codes in nesting depth D, with a finite-dimensional multiplicative-error version requiring one explicitly flagged concentration estimate.The replace margin is b^-D, and the resolvable depth grows only logarithmically with dimension.
  • Objective: At web scale, vanishing swap-negative rate π collapses the contrastive objective’s reward for binding and anti-binding penalty; the exactly reversed scorer costs precisely π E[M] more.The bag-of-words competitor is ε-optimal with ε ≤ (π + πimpl) log 2 plus an exponential tail; both constants match simulation within 0.006.
  • Geometry: Across 18 deployed text encoders, every model uses approximately 25–35% of the smoothness–binding ceiling, while SugarCrepe’s per-item ceiling tracks subset difficulty at r = 0.99.The exact frontier is M(s) ≤ 2δ when swap-related captions are δ-close to a pairing-neutral paraphrase anchor.
  • What does not explain failure: Dimension does not explain the observed failure, and a sign-agnostic matcher generalizes across disconnected exposure components at accuracy 1.0.The paper therefore treats dimension and exposure-graph connectivity as insufficient explanations of deployed binding failure.

2 Setup

The setup formalizes scenes, captions, swaps, and dual-encoder scoring, then defines benchmark competence through margin-retrieval and constituent-ranking axioms. It identifies the swap binding margin as the central quantity and specifies the embedding conventions used throughout.

  • Scene representation: A k-scene is a set of k object–attribute atoms with distinct objects, paired with a caption asserting exactly those atoms and an image depicting them.An atom is a pair (o, a) meaning object o has attribute a.
  • Scene representation: A swap exchanges two atoms’ attributes while preserving the scene symbol multiset, creating the binding-critical hard negative used by ARO and SugarCrepe.The original and swapped captions contain the same multiset of object and attribute symbols.
  • Dual-encoder axioms: A dual encoder maps images and captions to unit vectors in S^(N−1) and scores each pair by their inner product, with competence specified by margin retrieval and constituent ranking.The axioms require scene captions to outrank captions describing different scenes and present objects to outrank absent objects.
  • Binding margin: The binding margin controls swap performance: benchmark accuracy on a swap item is 1[M(s) > 0], under unit-vector, coordinatewise-product, and ±1 sign-vector conventions.The recursive depth law gives m(D) = 2b^-D; for b = 2 and D = 3, m = 1/4.

3 The depth ceiling of recursive binding

Recursive role-binding codes have an exact zero-crosstalk depth law: swap margins decay as 2b^-D, while finite-dimensional resolvable depth grows only logarithmically with dimension. Shared self-inverse roles can additionally alias distinct structures, so the result is class-relative and motivates tree-side scoring or cleanup.

  • Finite-dimensional ceiling: 1 level per b^2-fold increase in N describes the finite-dimensional resolvable-depth staircase, with D* = Θ(log N / log b).The finite-N result depends on one explicitly identified concentration estimate; its multiplicative-error signature is confirmed in simulation.
  • Simulation: At N = 65,536, m·b^D stays within 0.014 of 2 for b = 2 and within 0.006 for b = 3 across the tested depths.The sweep covered b = 2 to depth 10, b = 3 to depth 7, and N from 256 to 65,536, using 24 seeds per configuration.
  • Aliasing: Shared self-inverse roles make leaves with equal parity profiles exactly interchangeable, yielding zero unnormalized swap margin at any dimension.Fresh per-level roles restore m·b^D ≈ 2.00, whereas aliased far swaps fall two to five orders of magnitude below the law.
  • Scope and escape routes: The no-go applies to recursive role-binding encoders rather than arbitrary encoders, with tree-side scoring and chunking with cleanup as escape routes.Text encoders that score along syntactic derivations instantiate the tree-side route.

4 What the objective pays for binding

The contrastive objective offers only a vanishing, architecture-independent reward for binding because explicit-swap and implicit collision rates vanish at web scale. Consequently, exactly reversed binding is cheap, while controlled simulations verify the theorem constants within 0.006.

  • 4 What the objective pays for binding: Architecture-free bounds apply to arbitrary measurable scorers, so they cover dual encoders a fortiori.The analysis compares each scorer with a pairing-blind bag-of-words scorer and a reflection that reverses binding exactly.
  • 4 What the objective pays for binding: L(¯S)−L(S) ≤ (π + πimpl) log 2 + e^−(∆−2R), limiting binding’s objective advantage to this narrow sliver.Here π is the explicit-swap probability, πimpl is the implicit collision rate, ∆ is cross-class separation, and R is within-class spread.
  • 4 What the objective pays for binding: π E[M] bounds the loss gap to the exactly reversed scorer, making below-chance binding cheap to select.Small perturbations such as pooled-code cross-term interference or implicit bias can therefore select the reversed scorer.
  • 4 What the objective pays for binding: 0.006 is the maximum controlled-simulation discrepancy for both theorem constants.The companion measurements also verify linear small-π binding dose–response, equivalence between matched-term up-weighting and raising π, and cheap below-chance binding.

5 The smoothness–binding frontier, and where deployed models sit

The smoothness–binding frontier gives a per-item ceiling of cap = 2δ, while pairing-blind encoders collapse swaps to chance. Across deployed models, usage remains roughly 0.25–0.35, indicating binding failure is not smoothness-limited; NegCLIP is the exception, and the ceiling predicts SugarCrepe difficulty.

  • Smoothness–binding frontier: Pairing-blind caption maps make the original and swapped captions identical, forcing M(s) = 0 and chance swap accuracy.This holds for every γ > 0 and every image map under label-independent tie-breaking.
  • Smoothness–binding frontier: The smoothness–binding frontier sets the per-item ceiling at cap = 2δ, with usage defined as d/cap ∈ [0, 1].Usage near 1 indicates smoothness-limited binding; usage far below 1 indicates the ceiling is not binding.
  • Deployed models: NegCLIP alone climbs across the wedge, reaching usage 0.349 through caption-level swap negatives rather than LoRA fine-tuning.LoRA-rank-4 fine-tunes move along the wedge by spending smoothness without gaining separation.
  • Benchmark prediction: Across 14 models × 4,757 SugarCrepe items, subset-level mean d tracks accuracy at r = 0.99, with swap subsets lowest.The ordering matches the theoretical ceiling’s prediction of benchmark difficulty.

6 What does not explain the failure

Dimension does not explain swap failure: the constituent-ranking axioms are satisfiable up to an exact threshold, including positive-margin retrieval, and random factored sign codes realize this possibility at deployed scales. Exposure-graph connectivity is presented as a tempting but unresolved alternative, while the supplied table introduces a frontier diagnostic across 18 deployed text encoders.

  • Dimension: k ≤ ⌊N/2⌋ is the exact satisfiability threshold for constituent-ranking axioms over scene sizes j ≤ k, with positive-margin retrieval also achievable.The result applies to concept systems with at least N + 2 objects.
  • Dimension: Random factored sign codes realize the possibility direction cheaply at deployed scales, so dimension does not explain swap failure there.The possibility direction is independently published twice and established mathematically in the cited passage.
  • Exposure-graph connectivity: Exposure-graph connectivity is a tempting conjecture because disconnected training components create gauge ambiguity in rank-one completion.The supplied passage introduces this explanation but does not provide its resolution.
  • Frontier diagnostic: 18 deployed text encoders are evaluated with a frontier diagnostic reporting swap separation d, smoothness gap δ, ceiling cap = 2 2δ, usage = d/cap, and largest per-quartet usage.Rows are sorted by usage, with protocol and rigor checks deferred to Appendix H.

7 Related work

This work formalizes and extends Kang et al.’s ideal-encoder program while distinguishing binding obstructions from prior retrieval-capacity and semantic-resolution limits. Its frontier is independent of dataset size N, whereas the depth ceiling grows only logarithmically with N.

  • Ideal encoders: Kang et al. initiated the ideal-encoder program, which this work formalizes and extends.The extension is presented in Appendix B.
  • Capacity limits: Retrieval-capacity bounds in the LIMIT line and its possibility counterpart are orthogonal to this work’s binding results.Those bounds vanish as N grows, unlike the frontier of §5.
  • Capacity limits: The §5 frontier does not involve N, while the §3 depth ceiling grows only logarithmically in N.This contrasts the binding analysis with dimension/capacity limits for retrieval.
  • Semantic resolution: Finite semantic resolution limits identification under non-compositional representations, but does not address swap or margin behavior.The passage distinguishes this prior limitation from the paper’s swap/margin content.

8 Discussion … E.5 The swap margin, in full

The paper argues that binding failure is contingent on added structural, statistical, or semantic hypotheses, while smoothness and depth limits remain after those fixes. Its discussion also flags one explicit proof gap, scope limitations, and the quantitative behavior of pooled compositional codes.

  • 8 Discussion: Today’s failures arise from vanishing swap-negative rate, pooled-code leakage, and training-distribution effects, all fixable in principle.Measurements show that only raising π moves models up the predicted wedge.
  • 8 Discussion: The smoothness–binding frontier and depth ceiling D∗ = Θ(log N / log b) survive those fixes, with the ceiling binding recursive role-binding codes at CLIP-scale language depth.The discussion describes measured slack on smoothness and a natural-language-depth ceiling for CLIP-sized models.
  • Appendix; 8 Discussion: The appendices provide definitions, axioms, proofs, protocols, raw numbers, and negative results, while Lemma L.7’s concentration estimate remains the single deliberate proof gap.The finite-N depth result depends on this explicitly flagged estimate; its expectation form is proved and its multiplicative-error signature is simulated.
  • A Notation and glossary; B The formal setup; B.1 Concept systems, scenes, and negatives; B.2 Dual encoders and the axioms: Dual encoders score unit image and caption vectors by one inner product, and binding is measured by M(s) = ⟨i(Is), t(c(s)) − t(c(σs))⟩.The swap instance requires M(s) ≥ γ, while benchmark correctness is equivalent to M(s) > 0.
  • C Possibility: the axioms have solutions; C.1 Convex-geometry vocabulary; C.2 The dimension dichotomy: For k ≤ ⌊N/2⌋, placements satisfy all axioms with γ > 0; therefore every binding impossibility requires an additional hypothesis.If k > ⌊N/2⌋, the constituent-ranking axioms are impossible, establishing the dimension dichotomy.
  • D Obstruction I: pairing-blind encoders; B.1 Concept systems, scenes, and negatives: Pairing-blind caption maps assign identical embeddings to a scene and its swap, forcing M(s) = 0 and chance swap accuracy under label-independent tie-breaking.Separating swaps therefore requires parse-sensitive information rather than word-multiset invariance.
  • E Obstruction II: pooled codes and leakage; E.1 The homomorphism form: what “compositional” means here; E.2 Pooled codes, gain, leakage; E.3 The norm-cancellation lemma; E.4 Random factored codes and their primitives: In compositional coding, binding survives symmetric outer pooling only when the pooled units are bracketed bound constituents rather than words.Typed object and attribute codebooks carry role information, while bracketing supplies the load-bearing structure.
  • E.5 The swap margin, in full; E.4 Random factored codes and their primitives: 2/k is the pooled code’s swap margin, with all single-edit margins Θ(1/k), so meaningful retrieval margins satisfy γ ≲ 1/k.Under kL ≤ 1/5 and L < 2/(19k), the theorem pins the swap margin at 2/k up to the stated regime.

E.6 Constituent ranking fails for the plain code; augmentation · E.7 Dimension rates and the leakage dichotomy

The plain factored code cannot certify constituent ranking: present and absent objects receive no separating signal, with the criterion occurring at rate 0.00–0.01 in simulation. Augmenting with object presence repairs ranking but halves swap and attribute-replace margins, while dimension rates and leakage separate viable codes from additive and uncontrolled learned codes.

  • E.6 Constituent ranking fails for the plain code; augmentation: 0.00–0.01: The plain factored code’s constituent-ranking event occurs at this rate in simulation because its scores cannot separate present from absent objects.The obstruction applies to every scene and every object, present or absent.
  • E.6 Constituent ranking fails for the plain code; augmentation: Adding an object-presence term repairs the plain code’s unbinding obstruction by pooling presence alongside each bound pair.The repair addresses the fact that bound products are dissimilar to their raw factors.
  • E.6 Constituent ranking fails for the plain code; augmentation: 0.1243 vs. 0.1247: The augmented swap margin matches the predicted halving at k = 8, while object-replace and add margins match plain-code values to 3 decimals.Attribute-edit margins halve because the presence component cancels when the object multiset is preserved.
  • E.6 Constituent ranking fails for the plain code; augmentation: No single pooled vector maximizes retrieval and binding margins: mixing u ⊙v with αu trades the two continuously.Within the pooled class, retrieval and binding compete for the same normalization budget.
  • E.7 Dimension rates and the leakage dichotomy: N = O(k2 log MK): The whole hard core admits a uniform margin γ = c/(2(k + 1)), c < 1, for the plain factored code.The weakest member is add/drop, so no γ = c/k bound with large c covers the full family.
  • E.7 Dimension rates and the leakage dichotomy: ℓ≈0: Factored codes satisfy all axioms at N = O(k2 log MK), whereas additive codes have margin exactly 0 and generic learned codes provide no guarantee.The learned-code example has ℓ≈+0.17 with below-chance extrapolation to unseen pairings.
  • E.7 Dimension rates and the leakage dichotomy: Θ(1/k): Pooled atom codes separate swapped scenes at this margin while giving identical embeddings to order variants within each scene class.Pooling over words is invariant under both order variants and swaps, which destroys binding.

F Obstruction III: the training distribution … G.2 Anchors, smoothness, and the guard against tautology

The paper identifies training-distribution conditions that throttle binding in dual encoders, then derives a smoothness–binding frontier requiring pairing-neutral anchors to avoid tautological bounds. Web-scale contrastive training lacks the positive swap rate and coverage conditions needed for reliable binding.

  • F Obstruction III: the training distribution: Contrastive training separates a caption from its swap only insofar as the observed pair distribution distinguishes them.Population InfoNCE’s optimum is s*(I,c) = log p(c|I)/Q(c) + h(I).
  • F Obstruction III: the training distribution: π = O(1/|C|): in-batch negatives over a caption corpus yield asymptotically no swap signal and select a bag-of-words-equivalent solution.In the real-COCO π-sweep, swap accuracy rises 0.62 → 0.70 as π goes from 0 to 1, while retrieval metrics stay flat.
  • F Obstruction III: the training distribution: Held-out binding is unconstrained for atom pairs never co-observed, so generalization depends on encoder class and pair coverage.Factored-structure codes follow an occupancy law with threshold ρ ≈ 2–3; additive codes remain at chance, while generic MLP codes can fall below chance.
  • F Obstruction III: the training distribution: The data-side requirements are a positive swap rate π plus pair coverage or a factored-class code, neither of which standard web-scale contrastive training provides.These conditions are presented as Paper 1’s explanation of the status quo; the present paper studies the frontier after they are met.
  • G.1 The two-jobs tension: A swap preserves words, objects, and attributes while changing wiring, forcing embeddings to balance semantic similarity against binding.Paraphrases should be close for retrieval, similarity search, and deduplication, whereas swapped captions should be far apart for swap benchmarks.
  • G.2 Anchors, smoothness, and the guard against tautology: δ-smoothness requires both pairings to lie within cosine-radius 1 −δ of a unit anchor determined only by the scene’s symbol multiset.Examples include a bag-of-atoms caption embedding or a paraphrase-family mean constructed without pairing information.
  • G.2 Anchors, smoothness, and the guard against tautology: Pairing-neutrality is essential: an anchor allowed to depend on the pairing reduces the bound to the zero-content midpoint identity.The measurement protocol therefore constructs anchors without reference to the pairing, so proximity to both variants yields a genuine ceiling.

G.3 The theorem … I.1 Protocol

The smoothness–binding frontier gives a tight per-item ceiling on binding margins, yet deployed models operate far inside it. Across the model zoo and SugarCrepe, the evidence points to code, data, and training incentives—not smoothness—as the dominant current limitation.

  • G.3 The theorem: Theorem G.3 establishes a tight smoothness–binding frontier: shared-anchor smoothness upper-bounds binding, and the bound is achievable.The theorem’s upper bound applies for every image map and dimension, while its tightness construction reaches the frontier.
  • G.3 The theorem: Corollary G.4 makes the ceiling text-only and per-item: |M(s)| ≤ d(s), while class-level smoothness similarly caps cross-class separation.The anchor-relative theorem creates a jointly achievable (δ, M) frontier, not a claim that approaching the bound is inherently desirable.
  • G.4 The diagnostic: Usage ≈1 indicates smoothness-limited binding, whereas usage ≪1 indicates constraints from code, data, or training incentives instead.The diagnostic is defined as d/cap and distinguishes whether a model spends its available smoothness budget on binding.
  • H.3 Findings: 13 base models spanning three objectives, five corpora, two text architectures, and a 10× scale range all land at usage 0.25–0.32.No base model is within 3× of the ceiling; NegCLIP reaches usage 0.349, or 2.9× closer.
  • H.3 Findings: CLIP B/32 →L/14 raises d 0.237 →0.293 and cap 0.936 →1.131, leaving usage nearly fixed as scale increases.This indicates that scale moves models along the frontier rather than toward it.
  • H.3 Findings: Across all 18 models, larger d coincides with larger δ, with correlations from +0.10 to +0.82 and positive direction in every model.The result is directionally consistent with the predicted d–δ coupling; no exponent was fitted.
  • H.4 Measurement rigor: Only NegCLIP raises d 0.237 →0.320 and frontier usage 0.253 →0.349 against its architecture-matched baseline.It uses caption-level swap negatives; the comparison requires matching the QuickGELU graph and verified adapter loading.
  • I.1 Protocol: On SugarCrepe, pooled accuracy rises monotonically from near-chance to 0.93 across within-model d-deciles, with 73% of errors below median d.The evaluation covers 14 models, 4757 items, and 66,598 decisions; CLIP B/32 reproduces 0.775 overall accuracy and 0.61 on swap-object.

I.2 Three levels of validation … K.1 Setup

The paper validates its ceiling-based binding diagnostic across items, subsets, and models, while distinguishing contingent obstructions from a terminal smoothness frontier. It then formalizes the contrastive objective’s dependence on swap-negative coverage and specifies limitations and open extensions.

  • I.2 Three levels of validation: 0.990 Pearson correlation links subset mean-d with SugarCrepe subset accuracy, because swap negatives have intrinsically smaller ceilings than replace negatives.The structural reading is that identical-vocabulary swaps yield small d, whereas word-changing negatives yield large d.
  • I.2 Three levels of validation: 0.58 to 0.93 accuracy rises monotonically across d deciles, a +35-point span, while 73% of errors occur on below-median-d items.The low-end near-chance interpretation additionally assumes sign-symmetric sub-ceiling margins; the high end is permitted and observed.
  • I.3 Caveats: The validation is limited to synthetic 2-object color–object binding, one anchor, text-side diagnostics, and a single benchmark, without claims about unrelated capabilities.Naturalistic multi-relation captions and anchor robustness remain future work.
  • J.1 The obstruction map: three contingent gates, one terminal wall: Three obstructions are contingent—geometry, code structure, and swap-negative coverage—whereas the smoothness frontier binds every encoder, dimension, and training regime.Measurements place current models far below the terminal frontier, returning near-term attention to the contingent obstructions.
  • J.2 Relation to Paper 1: Relative to Paper 1, this work supplies the axiom system, organizing theorems for code classes, and a terminal bound that engineering cannot move.Paper 1 established quantitative code-ablation, leakage, and swap-rate effects; this paper provides their principled complement.
  • J.3 Related work and differentiation: The paper claims the only tight two-sided trade-off among the compared limits and the only measured frontier position for a deployed model zoo.Its contribution emphasizes the anchor quantifier, converse, and measurements alongside an elementary upper-bound proof.
  • J.4 Open problems: Open problems include margin-robust dimension, smoothness versus neighborliness, naturalistic multi-relation binding, and how recursive nesting affects depth-dependent margins.The depth question was unresolved at program start and is tied to grammar-structured composers’ performance on deeper-than-trained inputs.

K.2 The theorems … L.2 The contraction lemma and the exact law

The theorems show that contrastive training rewards binding only through swap-contrast frequency, while recursive role-binding margins contract geometrically with depth. Simulations verify the objective predictions, and the paper identifies unresolved selection, finite-sample, prior-separation, and multi-negative questions.

  • K.2 The theorems: ε ≤ (π + πimpl) log 2 + e−(∆−2R): the pairing-blind scorer is ε-optimal, so binding reward is limited by swap-contrast frequency and an exponential tail.At web scale, this threshold collapses; binding cannot follow from ε-optimality unless ε is smaller than the threshold.
  • K.2 The theorems: λπ + πimpl: up-weighting matched terms is theoretically equivalent to increasing the explicit swap rate.The equivalence follows because λ multiplies the matched case of the proof.
  • K.2 The theorems: π E[M]: exactly reversed binding incurs this loss distance, making small leakage or implicit bias sufficient to select below-chance binding.The theorem derives the explicit-swap contribution exactly, while implicit collisions and unmatched contrasts add bounded terms.
  • K.3 Verification, and a loop closed with the frontier: 0.290 + c1 = 0.476 vs. 0.477 and 1.261 + c0 = 1.943 vs. 1.949: Monte Carlo recovers both predicted matched constants, with error ≈0.005.At noise 0.25, the corresponding comparisons are 0.5445 vs. 0.5460 and 1.9455 vs. 1.9511.
  • K.4 Honest scope and open items: Not established: training’s selected solution inside the sliver, finite-sample guarantees, separation of instance binding from text priors, and the multi-negative softmax extension.The paper flags an Ω(1/π^2) testing lower bound as a natural finite-sample target.
  • L The depth ceiling of recursive composition: Θ(1/k): flat pooled codes have margins scaling with the number of atoms, whereas recursive composition yields a logarithmic ceiling on nesting depth resolvable by one pooled vector.The section motivates recursion as the productive structure of language and measures the depth law across three orders of magnitude.
  • L.1 Recursive role-binding: Sign-binding is an inner-product isometry: all distortion arises from superposition and normalization, not from binding itself.The recursive construction uses role-bound sums of child codes, with shared or fresh role schemes.
  • L.2 The contraction lemma and the exact law: m_swap(D) = 2 · b^-D and m_replace(D) = 1 · b^-D: in the zero-crosstalk idealization, sibling and far swaps obey the same exact geometric depth law.More generally, a level-ℓ0 edit with total slot deficit G has root margin G · b^-(D−ℓ0), independent of edit location.

L.3 Finite dimension: a multiplicative error law … N The updated map, and open problems

Finite dimension preserves the ideal multiplicative depth law up to a single explicitly outlined concentration step, yielding a logarithmic resolvable-depth ceiling that is single-digit at CLIP scale. Structural aliasing can make some deep swaps invisible, while the connectivity conjecture is refuted: swap binding requires coverage, not cross-component connectivity.

  • L.3 Finite dimension: a multiplicative error law: Finite-N codes preserve the ideal law multiplicatively when N ≥ C b^2D^2 log(bQ/η), with m · b^D constant to relative error o(1) whenever b^D ≪ N/polylog.The high-probability theorem depends on the single unproved concentration step in Lemma L.7.
  • L.4 The law, measured: At N = 65,536, simulated products m · b^D match predicted constants 2 for swaps and 1 for replacement within 0.014 and 0.006, respectively.At smaller N, the maximum swap deviation |m_sib b^D − 2| declines from 0.20 at N = 256 to 0.012 at N = 65,536.
  • L.4 The law, measured: 2 log b + O(1) is the resolvable-depth scaling, increasing one level per b^2-fold increase in dimension.At CLIP’s N = 512, pure crossing gives 4.5 for b = 2 and 2.8 for b = 3; with a 10/N floor, verified ceilings are 2 and 1.
  • L.5 Structural aliasing: some deep edits are invisible at any dimension: With shared sign-vector roles, equal parity profiles produce exactly unchanged unnormalized codes, so the swap margin is identically zero at any dimension.Under per-node normalization, only crosstalk-sized norm-mismatch residuals remain, without a 2b^-D term.
  • L.5 Structural aliasing: some deep edits are invisible at any dimension: For b = 3 at even depths, aliased far-swap margins are two to five orders of magnitude below the depth-law prediction at every measured N.Fresh per-level roles repair this structural aliasing, whereas flat shared roles do not merely attenuate deep structure.
  • L.6 Status and positioning: The no-go result is class-relative: the finite-depth claim applies to bounded-norm homomorphic combinators, while unconstrained encoders can look up any finite set of deep captions.A general-encoder theorem therefore requires a margin-complexity argument or another measurable hypothesis.
  • M What does not limit binding: a conjecture refuted / M.1 The conjecture / M.2 The refutation: The connectivity conjecture fails because sign-agnostic matching succeeds across components whenever both atoms are exposed; coverage, not connectivity, is the information threshold.The conjecture proposed that cross-component recombination was information-theoretically impossible in the factored world.
  • M.3 What survives / N The updated map, and open problems: Sign-naive gauge commitment yields 0.27–0.37 accuracy on cross-component tests, below chance, while the occupancy curve acc(ρ) ≈ 1/2 + 1/2(1−e^−ρK)(1−e^−ρM) is a theorem target.The below-chance result reflects committing to an arbitrary per-component gauge; the occupancy law is distinct from the giant-component percolation threshold.

N.1 The causal story, as the results now support it · N.2 Open problems (Part II update)

The results support a causal story in which binding is representable, geometrically permitted, and population-optimal, while deployed systems fail mainly because weak objective incentives and leaking composition codes leave outcomes to training pressures. Remaining limits are the smoothness frontier and depth ceiling, with open problems involving solution selection, finite-sample detection, and one unresolved concentration estimate.

  • N.1 The causal story, as the results now support it: Binding is representable with factored codes using N = O(k2 log MK), and the geometry permits it.The population objective at universal capacity even prefers binding.
  • N.1 The causal story, as the results now support it: π + πimpl log 2 nats is the proved width of the objective’s reward for binding, leaving capacity, regularization, early stopping, and implicit bias decisive.Theorems K.1–K.4 establish this throttle.
  • N.1 The causal story, as the results now support it: ℓ≈+0.17 leakage in learned composition codes and a π E[M]-cheap exactly-reversed solution explain why below-chance binding is observed.The reversed solution is cheap inside the objective’s incentive sliver.
  • N.1 The causal story, as the results now support it: D∗= Θ(log N/ log b) is a terminal depth ceiling, sitting at natural-language depth for CLIP-sized models, while the smoothness frontier has measured slack today.Both limits survive proposed fixes.
  • N.1 The causal story, as the results now support it: Dimension and exposure-graph connectivity are eliminated as candidate causes of binding failure.These eliminations are reported in §C and §M.
  • N.2 Open problems (Part II update): Selection inside the (π + πimpl) scale remains open: SGD tie-breakers may be non-benign, as suggested by leakage and gauge commitment.The unresolved question is which solution dynamics select when the objective is effectively indifferent.
  • N.2 Open problems (Part II update): Ω(1/π2) is the finite-sample lower bound for detecting the correct binding sign from rate-π swap-contrast events.This bound follows from two-point testing.
  • N.2 Open problems (Part II update): The depth-law concentration step remains incomplete: law (7) is proved except for one concentration estimate.Theorems L.4–L.8 establish the rest of the law.
Loading 2608.15971v1…