PaperlayerView exact revision on arXiv

Source-linked AI summary

The Illusion of Replacement: Rethinking Specialized Machine Learning Models in the Foundation Model Era

Kiyan Rezaee

arXiv:2608.28980v1cs.CLcs.AIcs.LG

TL;DR

Whether language-based models can replace specialized architectures for structured data remains an open question beyond predictive accuracy. This review synthesizes 159 papers across nine modalities using distinctions between describing, preserving, computing, and learning structure, and finds strong performance in specific regimes but no evidence of general architectural replacement when structure is tested directly.

  • Problem

    The central question is whether language-based models can acquire and exploit structural inductive biases traditionally encoded in specialized architectures, since predictive accuracy alone cannot establish structural preservation or computation.

  • Method

    The review synthesizes 159 papers across nine modalities using representational regimes and a describe/preserve/compute/learn framework.

  • Results

    Language-mediated systems can match specialized architectures in identifiable regimes, but direct tests of representation, computation, or architecture rarely support broader replacement.

  • Takeaways & Limitations

    When language-only representations are insufficient, specialization frequently relocates through tools, structural tokens, or architectural scaffolds rather than disappearing.

  • Takeaways & Limitations

    Evidence coverage is uneven, and most benchmarks were not designed to test representation preservation or replacement directly.

Abstract

from arXiv · show

Can the specialized architectures that machine learning has traditionally built for structured data be replaced by language-based models? This question is examined through a review of 159 papers (2016--2026) across nine modalities, with predictive accuracy considered alongside structural representation and computation. A distinction is made between performing a task and preserving and computing the structure that makes the task tractable, and existing approaches are organized into eight representational regimes, ranging from language-only systems to fully specialized architectures. Language-mediated models are found to be highly competitive in specific settings, including extreme few-shot prediction, discretized symbolic tasks, textually annotated knowledge graphs, and large-scale single-modality pretraining. However, whenever structural representation or computation is directly evaluated rather than accuracy alone, no evidence of general architectural replacement is found. Instead, a recurring pattern is observed across independent research communities: when language alone is insufficient, the missing structure is reintroduced through a graph module, structural tokens, specialized attention, or another non-linguistic component. In this sense, specialization more often relocates than disappears. Moreover, although performance of language-based models is improved by scaling, whether the gap to a structure-aware architecture can eventually be eliminated remains untested.

1 Introduction

Language-based models can perform competitively on structured-data tasks, but accuracy alone does not establish that they preserve or compute the underlying structure. Across modalities, specialization often reappears through explicit structural components rather than disappearing.

  • Relocated specialization: Independent communities repeatedly supplement insufficient language representations with non-linguistic structure, including structure tokens, graph attention, and graph-based chemistry tools.This convergence suggests that specialization is often relocated into another component.
  • Why specialization exists: Specialized architectures encode structural assumptions such as translation equivariance, permutation invariance, and relational inductive bias.These assumptions constrain the hypothesis space explored by the model.
  • Positive results: Language-mediated models show strong results in extreme few-shot prediction, text-based knowledge-graph completion, and text-symbol spatial grids.Text-symbol spatial grids reach 84–91% accuracy versus 60–73% for pixel-based formulations.
  • Controlled evidence: Controlled evaluations challenge claims that language components are responsible for apparent gains.Removing language-model components from time-series models preserved performance with roughly 1000× fewer parameters, while changing only graph textual encoding altered accuracy by 61.8%.
  • Structural fidelity: Causal positional encoding can make decoder-only transformers non-permutation-invariant even when the model describes the relevant structural property.Description and computation can therefore diverge.
  • Review framework: The review distinguishes predictive task performance from where structural information resides and what the model can preserve, compute, and learn.Systems retaining jointly trained or specialized components cannot attribute their performance to language alone.

2 Scope, Methodology, and Problem Formulation

The review uses a cross-modal systematic synthesis to clarify disagreements about whether language models replace specialized architectures. It traces the field’s shift from asking whether structure can be expressed in language toward asking what is transferred, preserved, computed, and learned.

  • 2.1 Survey methodology: The PRISMA-based review searches 2016–2026 literature on language models for structured and non-linguistic data across multiple complementary streams.The streams cover tabular data, graphs, time series, vision and multimodal data, in-context learning, learning theory, universal representations, and reported limitations.
  • 2.2 Positioning relative to existing surveys: Existing surveys are mostly modality-specific, while this review provides a common framework spanning nine modalities and connects empirical findings with learning theory.The review identifies no prior survey spanning more than two or three of these modalities under one consistent framework.
  • 2.3 What the field actually disagrees about: The framework separates whether a system can describe, preserve, compute, and learn structural regularities.This separates questions that are often conflated when evaluating substitution.
  • 2.3 What the field actually disagrees about: The review distinguishes empirical, mechanistic, definitional, and evaluative disagreements because each requires a different form of resolution.These categories concern measured results, explanations, terminology, and scrutiny of reported evidence respectively.
  • 2.3 What the field actually disagrees about: Controlled studies qualify strong universal-substitution claims through ablations, encoding sensitivity, and tests of permutation invariance.The reviewed evidence includes a 61.8% accuracy change from textual encoding and unchanged performance after removing language-model components from three forecasters.
  • 2.3 What the field actually disagrees about: The review defines replacement as language-model computation implementing a structural function, distinguishing it from feature engineering, model search, or pipeline construction.A language model’s role in directing a specialized system is not treated as architectural substitution.
  • 2.3 What the field actually disagrees about: Tabula-8B’s reported 5–15 percentage-point few-shot advantage was reassessed after instruction tuning and contamination analysis.The re-analysis recovered 92.2% of the reported gain through instruction tuning the same base model without tabular exposure.
  • 2.4 Historical evolution: The field’s trajectory moved from language-based expression and competitive performance toward controlled ablations, contamination audits, scaling specialized models, and joint pretraining.The review characterizes this as increasing precision about what language interfaces transfer, preserve, compute, and learn.

3 A Unifying Taxonomy: Representation Regimes and the Four-Way Distinction

The review separates where structural computation occurs from what a system does with structure. Its taxonomy distinguishes language-only, scaffolded, and non-linguistic regimes, while evidence shows that accuracy matching does not establish structural replacement.

  • Representation regimes: Eight regimes locate structural computation across language-only systems, scaffolded language systems, and architectures using non-linguistic representations.The boundary between Levels 6 and 7 marks the shift from language as computational medium to non-linguistic representations as the primary structural carrier.
  • What replacement means: Matching accuracy is insufficient evidence of replacement because it may reflect pretrained knowledge, preserved information without invariant computation, or information unavailable to the specialized baseline.The review therefore treats functional, structural, computational, and other senses of replacement as non-equivalent.
  • Recurring pattern: Across modalities, specialization commonly relocates into structural encoders, graph scaffolds, structural tokens, or other non-linguistic components rather than disappearing.Protein, tabular, and chemistry systems independently reintroduced specialized structure after language-only approaches proved insufficient.
  • Four-way distinction: Descriptive competence does not guarantee preservation, computation, or efficient learning, and the literature documents failures in each direction.For example, textual encoding can alter accuracy without changing description, while causal positional encoding can violate permutation invariance even when relevant information is preserved.
  • Four-way distinction: The regime ladder and the describe/preserve/compute/learn framework are complementary: one asks where computation occurs, while the other asks what the system does there.They are logically independent, so a model can occupy a given regime yet fail at preservation or computation.

4 Major Methodological Families: A Modality-by-Modality Synthesis

The review applies one cross-modal procedure: identify each task’s specialized inductive bias, classify the language-mediated alternative, and test its structural and computational claims against controlled evidence. Across modalities, the resulting evidence supports narrow functional successes rather than broad replacement.

  • Review framework: The synthesis identifies each modality’s specialized architecture and inductive bias before classifying the alternative’s representational regime and testing its evidence.It also isolates remaining specialized computation and evaluates whether systems describe, preserve, compute, and learn relevant structure.
  • Tabular data: Tabular methods have strong non-linguistic baselines: TabPFN performs in-context learning over feature–value pairs and can match tuned gradient-boosting ensembles on small datasets at low inference cost.Later variants extend this paradigm to datasets with up to 10,000 rows.
  • Tabular data: Language-mediated tabular approaches are strongest when labeled data are extremely scarce, but gradient-boosted trees overtake them as more labeled data become available while using substantially less computation.A reported 5–15 percentage-point advantage for Tabula-8B did not survive later scrutiny because most improvement was recoverable through instruction-tuning and contamination was identified.
  • Tabular data: Broader tabular evaluations find foundation models strongest in small-to-medium IID settings, while tree-based and deep-learning methods remain competitive or superior under other conditions.The review concludes that tabular evidence supports only narrow functional replacement, with no direct demonstration of structural replacement.

4.2 Graph-Structured Data

Graph evidence separates fluent description of graph properties from preserving and computing graph structure. Language-based systems can perform well when given textual information or specialized encoders, but direct tests do not establish replacement of graph architectures.

  • Structural preservation: Graph neural networks enforce permutation invariance by construction, whereas decoder-only language models do not automatically inherit this guarantee.Fine-tuning can reduce sensitivity to node relabeling while increasing sensitivity to formatting and structural changes.
  • Hybrid architectures: GraphGPT and LLaGA succeed by routing graph structure through learned graph encoders into the language model’s embedding space.Their results therefore support specialized structural encoders coupled to language models, not natural language as the computational substrate of graph reasoning.
  • Knowledge-graph completion: SimKGC substantially outperforms TransE, ComplEx, and RotatE, but its textual entity and relation descriptions give it information unavailable to the structural baselines.The comparison therefore does not establish recovery of graph topology from triples alone.
  • Conclusion: Direct tests show that models can describe permutation invariance fluently while violating it in their predictions.Accordingly, graph data provides direct evidence against structural replacement, while SimKGC’s functional comparison is weakened by its information advantage.

4.3 Time-Series Forecasting

Time-series forecasting offers strong controlled evidence against automatic replacement by pretrained language models. Language-compatible sequence representations are easy to construct, but the tested language components often do not contribute to temporal computation.

  • Temporal inductive bias: Temporal forecasting requires biases for order, locality, and cross-time dependence that pretrained language models have no inherent reason to respect.Specialized architectures encode these properties directly.
  • Controlled ablations: Across thirteen datasets and two metrics, language-free variants outperformed original LLM-based forecasters in most comparisons using roughly 1000× fewer parameters and up to three orders of magnitude less training time.Randomly reinitializing pretrained weights also matched or exceeded pretrained performance.
  • Controlled ablations: Shuffling the input sequence had little effect on predictions despite temporal order being central to the task.The result is decisive for the three tested architectures, but its generalization to the broader family of LLM-adapted forecasters remains open.
  • Non-linguistic alternatives: Non-linguistic time-series foundation models remain competitive: Chronos uses quantized numerical values without natural-language input, while TiRex uses xLSTM and achieves state-of-the-art performance on GIFT-Eval.Independent work also reports underperformance of LLM-based methods in epidemic forecasting and identifies leakage risks in benchmarks.
  • Joint pretraining: Chronicle jointly pretrains language and time-series representations through shared attention blocks rather than attaching a pretrained language model to the task.This is a different, preliminary hypothesis from the ablated architectures tested by Tan et al.

4.4 Vision and Spatial Reasoning

Across vision and adjacent modalities, language-based systems perform well when structure is discretized or retained in a specialized encoder, but raw structural replacement remains unsupported. The same pattern appears in chemistry, code, point clouds, and protein structure.

  • Vision and Spatial Reasoning: 51.26% accuracy was achieved by the best evaluated vision-language model across fourteen computer-vision tasks, versus 95.7% for humans.A separate study found roughly 58% average accuracy across four pixel-geometry tasks.
  • Vision and Spatial Reasoning: 84–91% accuracy was achieved by text-symbol grids on spatial localization, compared with 60–73% for the identical pixel-based task.Text-symbol representations preserve task-relevant spatial information before language processing.
  • Vision and Spatial Reasoning: Raw visual inputs have not established structural replacement under perturbations such as patch permutation, whereas discretized spatial representations come closer to functional replacement.The distinction is between preserving structure before language processing and asking language to recover it from raw inputs.
  • Adjacent Modalities: MoLFormer outperformed graph neural networks across ten molecular-property benchmarks after training on 1.1 billion molecules represented as SMILES.This is a genuine functional substitution result within chemistry, but it does not generalize uniformly across the modality.
  • Adjacent Modalities: CRUXEval found frontier models achieved only 67% and 63% accuracy when predicting short Python functions’ input/output behavior, despite strong code-generation performance.Fluent generation therefore does not establish reliable execution simulation.
  • Adjacent Modalities: PointLLM achieved strong 3D captioning by combining a specialized point-cloud encoder with a language model, while multimodal models still failed basic binary spatial relations.Protein models similarly restored structure-derived information after sequence-only representations proved insufficient.

4.6 Language as Orchestrator: The Genuine Growth Area

Language models show a practical role as orchestrators of specialized computation rather than replacements for it. They can search over, compose, evaluate, and refine specialized components while those components perform the underlying structural computation.

  • Orchestration: LLMs generate, select, evaluate, and refine specialized computational components instead of performing the underlying structural computation themselves.This orchestration role is formalized as Level 6 in the paper’s representational taxonomy.
  • Orchestration: LLM-FE consistently improved over previous automated feature-engineering methods by generating and evaluating feature-transformation programs within an evolutionary search.The language model contributes search and evaluation over specialized methods.
  • Orchestration: The evidence supports orchestration—searching, composing, evaluating, and refining machine-learning pipelines—not replacement of the specialized models being selected or invoked.The underlying structural computation remains with those specialized models.

4.7 Why These Comparisons Are (Not) Scientifically Valid

Performance comparisons are informative only when data regime, compute budget, and information availability are made explicit. Otherwise, reported advantages may reflect unequal resources rather than representational superiority.

  • Comparison Conditions: Language-mediated methods can have advantages in extreme few-shot settings but lose those advantages as labeled data accumulate.Data regime therefore changes the meaning of reported performance differences.
  • Comparison Conditions: Removing the language-model component in Tan et al.’s ablation reduced parameter count by roughly 1000× without reducing predictive performance.Accuracy-only comparisons can therefore obscure substantial computational differences.
  • Comparison Conditions: SimKGC and MoLFormer benefit from textual descriptions or large domain-specific pretraining corpora unavailable to their specialized baselines.These information advantages confound representational choice with information access.
  • Comparison Conditions: Comparisons that do not control or report data regime, compute budget, and information availability should be interpreted cautiously.A performance difference may reflect unequal data, computation, or information rather than a genuine representational advantage.

4.8 Cross-Modal Synthesis

Across modalities, language-mediated systems show narrow successes but no direct-test evidence of general structural replacement. The recurring response is to retain or restore non-linguistic structure, while conclusions remain weaker where only benchmark accuracy has been evaluated.

  • Cross-Modal Synthesis: Under matched conditions, TabPFN, TiRex, GNNs, and specialized vision architectures can outperform language-mediated alternatives.Language-mediated systems obtain narrower advantages under specific data or information conditions.
  • Cross-Modal Synthesis: At least three modalities independently restored missing structure through graph attention, structure-derived tokens, or graph-based chemistry tools.CARTE, SaProt, S-PLM, and chemistry systems implement different versions of the same corrective strategy.
  • Cross-Modal Synthesis: Language models perform well when task-relevant structure can be discretized before processing, but serialization alone does not automatically preserve continuous, relational, or invariant structure.Discretizability may moderate when language-mediated representations can substitute for specialized architectures.
  • Cross-Modal Synthesis: MoLFormer’s success at very large scale suggests domain-specific pretraining can partially compensate for representational limitations, but matched-budget compensation remains unestablished.Smaller SMILES-only models showed structural failures.
  • Cross-Modal Synthesis: No modality provides unambiguous evidence of structural replacement that survives a direct test.Direct tests are strongest in time series, graphs, and raw-pixel vision; several other modalities lack equivalent tests.
  • Cross-Modal Synthesis: The review’s central claim is strongest where structural properties are tested directly and weaker where evidence relies primarily on benchmark performance.The remaining question is whether the convergence reflects current-model limitations or a fundamental limitation of language-mediated computation.

5 Datasets and Benchmarks

The review distinguishes datasets, tasks, and benchmarks, showing that benchmark protocols usually evaluate predictive performance rather than representation, computation, or replacement. Benchmark bias and incomplete comparisons further limit replacement claims, while structural and mechanistic evaluation remains rare.

  • Benchmark scope: A benchmark combines a dataset, task, split, metric, and baseline, so its score concerns one specific performance comparison.A benchmark score is not a direct measurement of representation, computation, or replacement.
  • What benchmarks measure: Almost every reviewed benchmark evaluates predictive performance, while only component ablations and direct invariance-violation metrics provide representation- or replacement-level evidence.TabArena, TALENT, and GIFT-Eval measure in-domain accuracy; BLINK and MMVP evaluate reformatted or targeted accuracy; WN18RR and FB15k-237 measure link-prediction accuracy.
  • Benchmark bias: TabArena and TALENT favor language-mediated methods by concentrating on small-to-medium IID tabular datasets, where tabular foundation models perform most strongly.A broader, purpose-built alternative weakens that advantage substantially under a different evaluation regime.
  • Benchmark bias: Exact-match and structured-output scoring can penalize fluent, format-variable answers even when the underlying computation is correct.This output-format bias is documented across state-of-the-art language models in the wider evaluation-methodology literature.
  • Comparison limits: NLGraph and GraphQA usually lack GNN baselines, so their results cannot establish whether language-mediated reasoning replaces specialized graph computation.Other documented boundaries include MLE-Bench’s easier Lite subset and BLINK’s multiple-choice bottleneck.
  • Coverage gaps: Information preservation, structural computation, and mechanistic replacement are evaluated almost nowhere, with coverage also uneven across modalities.Chemistry, protein structure, and point clouds each have one benchmark entry, while code has none.

6 Evaluation Methodology and Theoretical Foundations

The review argues that accuracy metrics do not establish structural or mechanistic replacement, and combines direct behavioral tests with theoretical results to clarify when architecture-specific structure matters. Scaling improves performance within representational regimes, but existing studies do not measure whether it closes the gap to matched-bias specialized models.

  • 6.1 What current metrics can and cannot distinguish: Accuracy and task-specific scores measure output performance, not whether language-mediated representations preserve or compute the required structure.Mechanistic replacement asks whether a specialized component can be removed without loss under matched conditions.
  • 6.1 What current metrics can and cannot distinguish: Only two reviewed studies use direct invariance-violation metrics, while controlled ablation provides the strongest available mechanistic-replacement evidence.The metric maxπ |f(x) −f(π(x))| tests behavior under a structure-preserving transformation.
  • 6.2 Why the empirical pattern has a theoretical floor: Formal results show that some structural costs do not vanish through scale alone unless the relevant structure enters through architecture, representation, or training data.This is weaker than impossibility in principle but stronger than an empirical claim of insufficient scale.
  • 6.2 Why the empirical pattern has a theoretical floor: Transformers can be minimax-optimal for specific function classes when pretraining matches target tasks, and posterior-variance risk can decay exponentially with demonstrations under stated assumptions.These are formal results for restricted settings rather than general guarantees for structured computation.
  • 6.2 Why the empirical pattern has a theoretical floor: One-layer softmax attention cannot solve three-way matching, function composition, or binary-relation composition regardless of width or precision, whereas Strassen attention can solve them.The result identifies an architectural barrier removable by changing the computed operation rather than enlarging the existing mechanism.
  • 6.2 Why the empirical pattern has a theoretical floor: Permutation-invariant architectures provide formal representation guarantees, while standard transformers must learn permutation invariance from data; explicit invariance can match transformer in-context performance with roughly an order-of-magnitude lower resource use.The constructive bound uses latent dimension 2DN for vector dimension D and multiset size N.
  • 6.3 Scaling is real, sub-linear, and does not by itself close the gap: Reported scaling exponents are positive and sub-linear, but studies scale different quantities and do not use a common matched-bias baseline.Ma et al. report an exponent of approximately 0.4 for tabular in-context learning with respect to model parameter count and pretraining data size, separately.
  • 6.3 Scaling is real, sub-linear, and does not by itself close the gap: Scaling reduces loss within a representational regime, but no identified study directly measures whether the performance gap to a matched-bias specialized architecture vanishes asymptotically.The relevant unmeasured quantity is ∆(N) = L_general(N) − L_specialized(N), not language-mediated loss alone.

7 Comparative Synthesis

Across modalities, language-mediated systems are competitive in bounded regimes but do not replace specialized architectures when structural representation or computation is directly tested. The recurring solution is to restore missing inductive bias through architectural modification, explicit structural channels, or orchestration.

  • Mechanistic convergence: Architectural or representational corrections—not simply greater scale, pretraining, or compute—restore performance when language-mediated systems fail structurally.The reviewed examples add attention scaffolds, vocabulary extensions, or structural tools.
  • Replacement claims: A pure, unmodified language model counterexample was not identified; missing bias was instead restored either externally or by modifying attention and positional mechanisms.Set-LLM provides the internal-modification case, while CARTE, SaProt, and the chemistry approach add external structural channels.
  • Comparative evidence: Specialized inductive biases retain measurable computational or statistical advantages, including 230× faster inference for TabPFN than tuned boosting ensembles.Other examples include language-free state-of-the-art forecasting, permutation-invariant attention, and predictable scaling in a billion-parameter graph architecture.
  • Comparative evidence: Language-mediated systems are predictively competitive in extreme few-shot, symbolic or discretized, and tool-orchestration settings, but the evidence is conditional across data regimes and structural complexity.The review does not establish extension across matched computational budgets.
  • Replacement claims: Broad architectural replacement claims are refuted within the review’s evidence base across representation, computation, information availability, and structural invariance.Accuracy matching establishes, at most, functional replacement under the comparison’s data, compute, and information conditions.
  • Limitations: The review’s conclusions are limited because benchmark coverage is uneven and most benchmarks were not designed to test representation preservation or replacement directly.Therefore, absent positive evidence may reflect either a failed test or a test that was never performed.
  • Limitations: Formal results support that scale alone does not resolve non-permutation-invariance and certain compositional-reasoning failures.These claims follow from positional or causal structure and attention’s computational form rather than from insufficient parameter count.

8 Open Problems and Future Work

The review identifies a missing cross-regime experiment that compares language-mediated and matched specialized models while controlling information and sample budgets. It proposes falsifiable structural-complexity, depth, invariance, demonstration, and task-property tests, plus targeted ablations and scaling-law reanalysis.

  • Central gap: No existing study measures the loss gap between language-mediated and structure-aware architectures as model or data scale increases.The proposed metric is ∆(N) = Lgeneral(N) − Lspecialized(N).
  • Central gap: The central experiment varies the source of structural bias while holding information content and sample budget fixed across synthetic tasks with known ground truth.Candidate channels include architectural constraints, natural-language instructions, and in-context demonstrations.
  • Central gap: A flat or non-monotonic ∆(c) curve under these controls would refute the review’s central empirical hypothesis.The gap is defined against structural complexity c at fixed scale.
  • Research questions: RQ1 tests whether in-context sample efficiency diverges from a matched specialized architecture as permutation, locality, sparsity, or compositional complexity increases.The proposed hypotheses distinguish an architectural ceiling from depth-based recovery.
  • Research questions: RQ2 directly measures whether stating invariance in language narrows output-distribution gaps across a dimensionality sweep.Downstream accuracy alone cannot distinguish inert bias from genuine invariant behavior.
  • Research questions: RQ3 separates demonstration effects from memorization by varying target-task distance from the pretraining mixture along a matched structural-complexity axis.The review contrasts in-support generalization with out-of-support failure.
  • Research questions: RQ4 asks whether symmetry-group order, minimum description length, or discretizability predicts when language-mediated performance gaps are small or catastrophic.Existing text-symbol versus pixel-grid results motivate discretizability as a candidate moderator.
  • Actionable studies: Future work should reanalyze scaling laws as ∆(N)-versus-scale curves relative to matched-bias baselines rather than isolated loss curves.The review states this requires no new training run.

9 Conclusion

Across nine modalities, behavioral competitiveness does not establish replacement: when language-only representations are insufficient, systems often reintroduce non-linguistic structure. Scaling improves performance, but whether the gap to matched-bias architectures closes remains unresolved.

  • Across modalities, language-mediated systems can match specialized architectures predictively, but direct tests of representation, computation, or architecture rarely support broader replacement.
  • Accuracy, statistical efficiency, computational cost, and structural equivalence are distinct claims; benchmark performance alone cannot establish structural replacement.
  • When language-only representations prove insufficient, successful systems frequently restore non-linguistic structure through tools, specialized tokens, or architectural scaffolds, relocating rather than eliminating specialization.
  • Language can function as an interface for directing and composing specialized components, but this integration is distinct from architectural replacement.
  • The unresolved scaling question is whether the gap to a matched-bias architecture, ∆(N), decreases and eventually closes, not merely whether L(N) decreases.
  • Existing theoretical constraints do not directly predict the performance gap as a function of scale or structural complexity, motivating a falsifiable framework for identifying substitution boundaries.
Loading 2608.28980v1…