PaperlayerView exact revision on arXiv

Source-linked AI summary

When the Canonical Completion Is Wrong: Formalizing and Measuring the Jump in Large Language Models

Dai Shi, Xiaoyu Li, José Miguel Hernández-Lobato

arXiv:2608.26187v1cs.CLcs.AIcs.LGcs.LO

TL;DR

The paper addresses the lack of a formal definition and measure for abductive jumps in LLMs. It formalizes four stages and measures the override stage using certified finite extension problems. The Kan-default rate is zero in all 248 constrained trials, but the test covers selection rather than constraint generation or framework invention.

  • Problem

    The field lacks a formal definition and measure for whether LLMs can make abductive jumps from evidence to new axioms.

  • Method

    The paper defines certified jump instances whose unique-up-to-renaming correct completion differs from the canonical Kan completions and measures model override of those defaults.

  • Results

    0 Kan-default rate occurred across all 248 constrained trials involving nine certified instances and four frontier models.

  • Takeaways & Limitations

    The measured override step is not the bottleneck; if incapacity remains, it lies in generating constraints or inventing the framework.

  • Takeaways & Limitations

    The test measures abduction as selection with stated constraints and a delimited answer space, not abduction as generation.

Abstract

from arXiv · show

Whether large language models (LLMs) can perform the abductive leap from evidence to a new system of axioms, commonly referred to as a jump, has recently attracted considerable debate. A prominent position holds that LLMs are structurally incapable of such jumps, while recent studies challenge both its mechanism and its evidence. However, the debate remains difficult to settle, since the field still lacks a formal definition of the jump and a measure to test either side. In this paper, we develop a formal account of the jump in four steps and measure the second. The steps ask what the default completion of partial data is, when abandoning it is forced, when the abandonment is correct, and how successive jumps compound. Specifically, we define a jump instance as a finite extension problem with a machine-checked certificate that a correct completion exists, is unique up to renaming, and differs from the canonical completion of the data. The canonical completion is given by the left and right Kan extensions and is also what models produce without constraints, so it serves as the default. We prove that jump instances are well-posed and establish a family theorem that certifies instances of unbounded difficulty without enumeration. We further formalize when a jump is correct and how successive jumps compound. Finally, we run the measurement on nine certified instances and four frontier models. The Kan-default rate is zero in all 248 constrained trials, so the models do jump at this step and abandon the excluded default every time. Failures at higher difficulty stem from exhausted reasoning budgets or constraint errors, never from reverting to the default. These results indicate that the second step is not the bottleneck. If the disputed incapacity is real, it lies in generating the constraints or inventing the framework. Code can be found at: https://github.com/EEthanShi/kan-jump-test.

1 Introduction

The paper formalizes the jump from default completion to a constraint-selected alternative and measures whether models can perform this override. Across nine certified instances and four frontier models, models abandoned excluded Kan defaults in all 248 constrained trials, while the paper limits its claim to selection rather than generation.

  • Problem: The paper asks whether LLMs can abandon a canonical completion when stated constraints rule it out.It targets the selecting half of abduction, with constraints and the answer space already specified.
  • Default: Canonical completions are defined by left and right Kan extensions, which serve as measured defaults for unconstrained models.The protocol calibrates each model’s default rather than assuming it coincides with a Kan extension.
  • Formulation: A jump instance certifies that a unique-up-to-renaming correct completion exists and differs from both Kan extensions, forcing non-canonical structure.Matched controls instead select the canonical completion, separating default lock-in from other causes of failure.
  • Measurement: 0 Kan-default rate occurred in all 248 constrained trials across nine certified instances and four frontier models.Higher-difficulty failures came from exhausted reasoning budgets or constraint errors rather than default reversion.
  • Scope: The experiment measures only the second step, leaving constraint generation and invention of a genuinely new mathematical language for future work.The test covers abduction as selection, not abduction as generation.
  • Theory: The paper proves well-posedness, finite answer spaces, and a family theorem certifying instances of unbounded difficulty without enumeration.It also formalizes internal versus predictive correctness and chains in which one certified answer becomes the next stage’s data.

2 Related work

The paper positions its contribution between debates about abductive generation, categorical accounts of generalization, creativity, and existing systematicity benchmarks. Its distinction is to certify when the canonical completion is wrong and test whether learners override it.

  • The jump debate: The jump debate disputes whether abduction requires embodiment and whether machine-generated mathematical discoveries challenge incapacity claims.The paper’s Kan-default rate requires only a computable default and a certified exclusion, not a full theory of the jump.
  • Category theory and learning: Prior categorical and error-minimization work motivates treating Kan extensions as a null hypothesis for completion from partial data.The paper calibrates whether each model’s unconstrained default actually coincides with a Kan extension.
  • Formal abduction and computational creativity: Existing creativity accounts model blending, transformation, or theory discovery, but do not provide the learner-facing test developed here.Those approaches respectively use universal gluing, meta-level search, or structural grammars.
  • Benchmarks for systematicity and abduction: Systematicity and ARC-style benchmarks generally grade canonical or intended rules without certifying that the canonical completion is wrong.The paper instead certifies canonical exclusion, making failures more attributable.

3 Preliminaries

The preliminaries represent partial-data completion as finite functor extension into FinSet, quotienting away invented-element renamings. Left and right Kan extensions provide the canonical, constraint-blind completions against which non-canonical extensions are tested.

  • Finite categories as tables: A finite category is given by finite objects, typed morphisms, identities, and composition, while a FinSet-valued functor gives a concrete labeled structure.Functor assignments must respect typing, identities, and composition.
  • The extension problem: An extension completes a partial functor F0 on new objects and morphisms while agreeing exactly with F0 on the observed subcategory.The extension fiber is Ext(F0) := { F : C →D | F ◦K = F0 }.
  • Gauge: Gauge equivalence identifies extensions that differ only by permutations of invented elements at unobserved objects.Scoring is invariant under these renamings, so no invented-element labeling is rewarded or penalized.
  • Kan extensions: Lan pushes observed values forward along routes, whereas Ran assigns matching tuples across routes out of a hidden object.Both are finite, choice-free completions that restrict back to the original data.
  • Why Kan is the distinguished canon: Kan extensions are distinguished because they are the left and right adjoints to restriction and are unique up to canonical isomorphism.The paper treats them as universal constraint-blind completions while measuring each learner’s actual default.
  • Worked micro-example: In the micro-example, Lan copies the input through the hidden stage, while Ran copies the output; other factorizations posit structure not forced by the data.Jump instances are designed so only such non-canonical factorizations satisfy the constraints.

4 Formulation

Section 4 formalizes measurable jump instances by separating constraint-blind canonical completion from constraint-guided admissibility, with calibration and override metrics for Tier 1.

  • 4.1 Instances: A Tier-1 instance supplies finite category tables, partial data, constraints, and a declared bound on new-object sizes.The semantic object is an explicit finite table, and the bound must be entailed by the constraints.
  • 4.1 Instances: The canon library is fixed before generation, and its distinguished default consists of the left and right Kan extensions.Canonical operators read only the data and never the constraint text or size bound.
  • 4.3 Jump and control instances: A jump instance requires admissible completions to exclude every canonical-library output while remaining identifiable up to renaming of invented elements.Support conditions ensure every new object is connected to the observed data.
  • 4.3 Jump and control instances: Matched control instances instead pin exactly a target canonical completion while preserving the same format and structural conditions.This pairing distinguishes failure to extend functorially from extending only canonically.
  • 4.4 Chance: The chance level is the ratio of admissible strict tables to the bounded answer space, using a format-aware guesser that ignores constraints.The bounded space is finite because the declared size bound restricts new objects.
  • 4.5 Jump capability: calibration, margin, Kan-default rate: Learners receive data, constraints, and the bound, and succeed only by producing an explicit bounded functor in the admissible set.Outputs outside the bounded extension space are counted separately as format failures.
  • 4.5 Jump capability: calibration, margin, Kan-default rate: Calibration replaces constraints with neutral filler to identify the learner’s unconstrained default before measuring default-canonicity, constrained canonicality, and the override gap.The instrument is inapplicable when calibration validity fails; then no jump claim is licensed.
  • 4.5 Jump capability: calibration, margin, Kan-default rate: The triple (DC, KD, ∆) operationalizes Tier-1 override capability by separating canonical solving from inability to extend at all.A high canonical-default rate with controls passed and little override gap is the falsifiable non-jumping pattern.

5 Well-posedness and existence

The paper proves that its jump definitions are well-posed, constructs a certified seed, and extends it to an infinite family whose difficulty can grow without enumeration.

  • 5 Well-posedness and existence: Propositions 1–3 establish canonical existence, gauge-invariant scoring, and finite identifiable answer spaces, making chance levels computable.Canonical representatives exist for every instance, while admissibility and scoring are invariant under renaming invented elements.
  • 5 Well-posedness and existence: Proposition 4 constructs a seed jump instance in which the singleton Kan extension violates the nontrivial-endomorphism constraint.The admissible class is a single gauge component of size 3 with hidden object size 3.
  • 5 Well-posedness and existence: Each admissible seed completion selects a point, makes the outgoing map constant, and adds two invented elements carrying a nontrivial Z/2 symmetry.These hidden elements are absent from both the observed data and the Kan extension.
  • 5 Well-posedness and existence: The seed is easy for frontier models, so benchmark difficulty must scale the category, relations, and coupled constraints under the integrity discipline.The paper also distinguishes controls, where canonical solving succeeds, from jumps, where constraint-blind canonical learners fail.
  • 5.1 The family theorem: certification without enumeration: The family theorem certifies every pointed chain with hidden prime cycles as a jump instance, with one fixed point and one pi-cycle at each hidden object.Incoming maps are constant onto the fixed point and outgoing maps are constant.
  • 5.1 The family theorem: certification without enumeration: The admissible set is one gauge orbit, and its cardinality follows a closed-form product over the hidden prime cycles.The theorem also gives a transfer recursion for the bounded extension-space size.
  • 5.1 The family theorem: certification without enumeration: Primality is necessary for identifiability because composite cycle orders permit shorter compatible cycles and multiple cycle lengths.The transfer-matrix count replaces enumeration with station-by-station recursion.
  • 5.1 The family theorem: certification without enumeration: The theorem supports efficient, non-memorizable benchmarks: instances of arbitrary difficulty can be regenerated with fresh vocabulary after one proof.Enumeration corroborates the closed forms at seven configurations, including |Ext4| = 1,257,409 and |Adm| = 27 at (3, (2, 2, 2)).

6 Correctness and chained jumps

The paper distinguishes internal from predictive correctness and formalizes how certified commitments become data for later stages, where accumulated constraints can eliminate earlier admissible alternatives.

  • Internal versus predictive correctness: Internal correctness means satisfying all stated constraints, whereas predictive correctness requires agreement with held-out ground truth.Under identifiability, the two notions coincide; their divergence requires ambiguity and withheld future evidence.
  • Internal versus predictive correctness: A sound 2-ambiguous extended instance can contain two internally correct components while held-out probes make exactly one predictively correct.The distinguishing evidence lies entirely outside the model’s observed system, formalizing the Nordström situation.
  • Chains, monotonicity, and entrenchment: A successor scheme turns a commitment’s hidden objects into observed data and adds constraints over newly introduced objects.Chains iterate this incorporation, committing at every stage.
  • Chains, monotonicity, and entrenchment: Incorporation is well-defined: gauge-equivalent commitments yield data-isomorphic successor instances with equal truth values, chance, and corresponding admissible-component sizes.This makes each chain stage another valid jump instance, independent of representative choice within a gauge component.
  • Chains, monotonicity, and entrenchment: Accumulated knowledge only narrows admissibility, while retrospectively correct commitments decrease with later stages and eventually stabilize.Retrospective correctness requires that a commitment can be continued through a specified later stage; ultimate correctness requires continuation indefinitely.
  • Chains, monotonicity, and entrenchment: In a two-stage chain, the Y3 component remains extendable but the Y4 spectator component is excluded by the next-stage probe.The construction leaves open whether models revise a commitment eliminated by later evidence or entrench the earlier error.

7 Experiments

The experiments evaluate four frontier models on nine certified instances and measure whether they override the Kan default under stated constraints. The Kan-default rate is zero in all 248 constrained trials, while higher-difficulty failures reflect search, budget, or constraint issues rather than default reversion.

  • Setup: Four frontier models are evaluated on nine certified instances, with chance levels ranging from 0.13 down to 8 × 10−7.Six instances are enumeration-certified and three m = 3 instances are certified by the family theorem.
  • Main results: The Kan-default rate KD is identically zero across the main results.The experiment therefore measures whether models override their own default when constraints exclude it.
  • Instrument validation: 98% of non-truncated calibration answers coincide with the Kan default, validating it as a per-model measure rather than an assumption.The conditional per-model rates are 35/36, 36/36, 35/36, and 23/24.
  • Main results: In all 248 constrained trials, no model produced the excluded Kan default, with chance levels down to 8 × 10−7.Jump accuracy was 41/44 under greedy decoding and 185/204 at temperature 0.7, with neither setting producing a Kan answer.
  • Difficulty effects: At m = 3, Gemini 3.1 Pro solves every sample, while other failures arise from constraint errors, exhausted reasoning budgets, or empty outputs.DeepSeek V4’s accuracy drops to 0.33, with 42 to 50% of samples across the three arms exhausting the reasoning budget.
  • Deployment effects: Undersized output budgets bias results toward the default, and provider content filters can refuse every task rendering, so deployment effects require control.The strictest filter of one vendor refused all renderings, including a pure-mathematics statement.

8 Scope, protocol, and limitations

The instrument measures only the override step, with constraints and codomain supplied rather than generated. Its main boundaries concern scope, certification scalability, and possible mismatch between certifiable and diagnostically hard instances.

  • Scope and protocol: The test measures overriding a measured canonical default with a constraint-pinned non-canonical extension, not abductive generation.It provides no supervision on the hidden extension and therefore does not address generating constraints or frameworks.
  • Scope and protocol: The constraints are given, the codomain D is handed over, and fixing FinSet excludes invention of a genuinely new codomain.Tier 2 is intended to host the jump proper, where D is not fixed.
  • Integrity and benchmark scope: Six of nine instances clear every constraint floor, while three fall below one or two size-constraint floors, as low as 1.4%.The retained instances have no single constraint within a factor of eighty of isolating the admissible component.
  • Limitations: The program may fail if brute-force-certifiable instances are too easy or diagnostically hard instances cannot be certified, making scalable certification the open engineering problem.The paper identifies the possible non-intersection of certifiable and diagnostic regimes as a primary failure mode.
  • Library dependence: Non-canonicity is relative to the preregistered library L = {Lan, Ran}, although passing matched jump and control arms refutes every constraint-blind single-completion operator.The conclusion is tied to the chosen library, while Proposition 5 supplies the stated robustness against constraint-blind operators.

A Deferred proofs

The appendix proves that Kan extensions can be strictified to agree with observed data, characterizes gauge equivalence, and analyzes the finite seed instance. It also proves that the Kan default is excluded by the seed constraints.

  • Deferred proofs: Finite limits and colimits define the left and right Kan extensions, which can be strictified to restrict exactly to F0 on the observed subcategory.The strictification is functorial and independent of further choices up to gauge equivalence.
  • Deferred proofs: Gauge transformations preserve object assignments and sizes, so the canonical extensions form a union of gauge components.This establishes the quotient structure used to compare extensions up to renaming.
  • Deferred proofs: Rooted isomorphism with identity on observed objects is exactly the gauge-orbit relation, with finiteness following from finite hom-sets.The proof identifies natural isomorphisms fixing the observed subcategory with gauge tuples.
  • Seed instance: The seed extension count is |Ext4| = 25, obtained by summing n·I(n−1) for n = 1 through 4.The relations require an involution fixing the image point, while the Kan extensions are singleton and violate K1.
  • Deferred proofs: The matched jump output lies in the admissible set, whereas the canonical output lies in CanonL and is disjoint from it; the control always returns the canonical output.This proposition supplies the formal separation between constrained jump and default behavior.

B The certified seed as a benchmark item

The certified seed is presented as a nonce-vocabulary three-stage signal pipeline whose constraints uniquely select a non-canonical extension. Its control condition instead admits the Kan-default singleton.

  • Benchmark item: The seed translates three category objects into NARV, QUILB, and SORM, with dax, rell, and fen realizing the three morphisms.Rules R1–R3 encode the category relations, while observations and requirements encode F0 and K.
  • Jump condition: The correct answer has QUILB = {p, q, r}, dax mapping n1 to p, constant rell, and fen fixing p while swapping q and r.These properties characterize the single admissible gauge orbit.
  • Jump condition: The Kan-default answer is the singleton QUILB with identity fen, reproduces every observation, and violates exactly D2.It is therefore the diagnostic failure mode under the jump constraints.
  • Control: The matched control replaces D1–D2 with a one-state bound, making the Kan extension itself the unique admissible answer.The control separates constraint-driven override from ordinary canonical completion.
  • Integrity checks: The benchmark preregisters non-pinning, propagation-hardness, and a matched-budget generic CP/SAT reading-comprehension baseline.These checks assess whether individual constraints or local propagation isolate the answer and how much raw search suffices.

D Proof of the family theorem

The family theorem constructs finite chain instances whose constraints force one fixed point and one prime-length cycle at each hidden station. Their Kan defaults remain trivial, while closed-form counts certify increasing difficulty.

  • Family construction: The family uses finite chain categories with singleton inter-object hom-sets and cyclic endomorphisms of prime order.The category is modeled over FinSet, with the observed subcategory containing the endpoints and their composite.
  • Extension structure: Each extension assigns hidden sets, prime-order endomorphisms, and connecting values; the relations force fixed incoming images and orbit-constant outgoing maps.Prime order restricts cycles to lengths 1 or p_i.
  • Forcing and uniqueness: The constraints force |F(b_i)| = 1 + p_i, consisting of exactly one fixed point and one p_i-cycle.The incoming map is constant onto the fixed point, and the outgoing map is forced constant.
  • Forcing and uniqueness: The admissible data form a single gauge orbit after choosing a fixed point and cyclic order at each station.The construction therefore satisfies the intended identifiability condition.
  • Kan default: Lan(b_i) = Ran(b_i) = [1] for every hidden station, so the all-singleton trivial extension violates exactly K1_i.Every hidden station receives a path from the observed endpoint, establishing the stated strong J4 condition.
  • Counting and sharpness: The transfer recursion reproduces all seven enumerated counts, including 1,257,409 at (3, (2, 2, 2)) with N = 4.For composite p_i, a smaller admissible profile exists and J3 fails, showing primality is sharp.

E Proofs for Section 6

The paper proves structural properties of successive jump instances, including invariance, monotonicity, stabilization, and separation of admissible completions.

  • Characterization: Proposition 6 characterizes the witness-isomorphism predicate as selecting exactly the C0-rooted gauge component.Gauge transport extends by identities on held-out objects, and conjugation preserves functoriality.
  • Separation: Proposition 7 shows that relaxing the seed size bound permits exactly two admissible structures, Y3 and Y4, with |Adm| = 15.The two components differ by a spectator fixed point outside the incoming image.
  • Well-definedness: Theorem 2 establishes that incorporation is well-defined under arbitrary automorphism families and preserves admissibility and the canonical class.The proof uses conjugation invariance, extension of gauges by identities, and iso-naturality of Lan and Ran.
  • Monotonicity: Lemma 1 proves monotonicity: admissible sets decrease along the instance ladder as each later-stage functor restricts to an earlier admissible commitment.Descending the ladder yields the decreasing sequence, with the final term equal to the last stage’s admissible set.
  • Stabilization and entrenchment: Theorem 3 proves that retrospectively correct sets stabilize within finite admissible sets and formalizes an entrenchment scheme using a marked-cone limit constraint.At the second stage, the constraint forces the new object’s value to biject onto the fixed set of the commitment’s t.

F Experimental details

The experiments evaluate four models through calibration, controls, constrained jumps, and automated grading, while documenting sensitivity to size bounds and access conditions.

  • Sensitivity to the size bound: The declared size bound enlarges the answer space and lowers chance levels as N increases.For m=2, p=(2, 2), the count rises from 4,387 at N=4 to 117,867 at N=5, while chance falls from 2.1 × 10−3 to 7.6 × 10−5.
  • Sensitivity to the size bound: The released instance uses the minimum admissible bound N = 4, and above-chance conclusions survive either bound convention.The largest minimal-bound chance is 0.44 at (1, (3)), compared with accuracies of 0.83 and above; padding is excluded by the family constraints.
  • Models and access: The evaluation used GPT-5.6 Luna Pro, Claude Sonnet 5, Gemini 3.1 Pro, and DeepSeek V4 Pro through a single aggregator API.One provider’s strictest tier refused every rendering, so the reported data point for that family uses its mid tier.
  • Sampling: Sampling produced 464 graded answers, combining jump, calibration, control, and wording-arm samples.The earlier 165-answer seed pilot was released but excluded from all counts.
  • Grading: The grader classified parsed answers by admissibility and default status, with certified agreement on all 138,691 enumerated extensions of six released instances.The released test suite also passed adversarial renaming and junk-answer tests.
  • Aggregates: The Kan-default rate is 0 in all 248 constrained jump samples.The 22 jump failures comprise 15 truncations, 3 empty answers, 3 rule violations, and 1 valid-but-inadmissible completion.
  • Aggregates: Calibration answers were the Kan default in 129 of 132 nonempty, nontruncated cases, while controls were passed in 69 of 72 cases.The three control failures were truncated or empty DeepSeek answers at m = 3.
Loading 2608.26187v1…