PaperlayerView exact revision on arXiv

Source-linked AI summary

The Illusion of Replacement: Rethinking Specialized Machine Learning Models in the Foundation Model Era

Kiyan Rezaee

arXiv:2608.28980v1cs.CLcs.AIcs.LG

TL;DR

Specialized architectures may be replaceable by language-based models for structured data, but predictive accuracy alone cannot establish whether structural representation and computation have been replaced. Reviewing 159 papers across nine modalities, this paper finds no general architectural replacement where structure is directly evaluated; instead, specialization often relocates into non-linguistic components.

  • Problem

    The central question is whether language-based models acquire and exploit structural inductive biases traditionally encoded explicitly in specialized architectures.

    Source: §1 · p. 2

  • Method

    The review analyzes 159 papers across nine modalities using predictive accuracy alongside structural representation and computation, organizing approaches into eight representational regimes.

    Source: §1 · p. 2, §3.3 · p. 14

  • Results

    Across modalities, language-mediated systems can match specialized architectures on accuracy, but direct tests of representation, computation, or architecture provide no evidence of general replacement.

    Source: §9 · pp. 40–41

  • Takeaways & Limitations

    When language-only representations are insufficient, successful systems frequently reintroduce specialization through tools, specialized tokens, or architectural scaffolds.

    Source: §9 · p. 41

    The evidence base is uneven, and most benchmarks were not designed to test representation preservation or replacement directly.

    Source: §7.4 · p. 34

Abstract

from arXiv · show

Can the specialized architectures that machine learning has traditionally built for structured data be replaced by language-based models? This question is examined through a review of 159 papers (2016--2026) across nine modalities, with predictive accuracy considered alongside structural representation and computation. A distinction is made between performing a task and preserving and computing the structure that makes the task tractable, and existing approaches are organized into eight representational regimes, ranging from language-only systems to fully specialized architectures. Language-mediated models are found to be highly competitive in specific settings, including extreme few-shot prediction, discretized symbolic tasks, textually annotated knowledge graphs, and large-scale single-modality pretraining. However, whenever structural representation or computation is directly evaluated rather than accuracy alone, no evidence of general architectural replacement is found. Instead, a recurring pattern is observed across independent research communities: when language alone is insufficient, the missing structure is reintroduced through a graph module, structural tokens, specialized attention, or another non-linguistic component. In this sense, specialization more often relocates than disappears. Moreover, although performance of language-based models is improved by scaling, whether the gap to a structure-aware architecture can eventually be eliminated remains untested.

1 Introduction

This review asks whether language-based models can replace specialized architectures for structured data, distinguishing predictive task performance from preserving and computing the structural assumptions that make tasks tractable. Across modalities, it argues that language-mediated systems often relocate specialization into structural tokens, graph modules, specialized computation, or external tools rather than eliminate it.

Cross-modal motivation

  • Across protein, tabular, and chemistry research, language-based approaches were supplemented with non-linguistic structural channels after careful evaluation exposed their limitations.Foldseek-derived structure tokens, graph-attention scaffolds, and graph-based tools restored missing structural capabilities in otherwise language-mediated systems.Source: §1 · p. 1

Why specialization exists

  • Specialized architectures encode unavoidable assumptions about data structure, including translation equivariance, permutation invariance, and relational inductive biases [15].These architectural choices constrain the hypothesis space because no learning algorithm can outperform all others across every possible data-generating process.Source: §1 · pp. 1–2

Research question

  • The review therefore evaluates whether language-based models acquire and exploit structural inductive biases, not merely whether they produce plausible predictions on structured inputs.It asks whether language is a general representation or an interface to underlying computation, and whether prompting can recover inductive biases enforced by specialized architectures.Source: §1 · p. 2

Empirical tension

  • Positive accuracy results coexist with failures in structural preservation, invariant computation, efficient learning, and benchmark-independent generalization [34] [21] [10].Examples include 1000× smaller time-series models matching or outperforming language-model systems, a 61.8% accuracy change from textual encoding alone, and text-symbol grids reaching 84–91% versus 60–73% for pixels.Source: §1 · p. 2

Meaning of replacement

  • “Replacement” can mean reproducing outputs, matching accuracy, preserving robustness, reproducing structural computation, or orchestrating specialized tools, and these claims are not equivalent [1].Language-model orchestration is a genuine capability, but it does not establish architectural substitution when specialized computation remains responsible for the result.Source: §1 · pp. 2–3

Review framework

  • The review addresses this gap with a PRISMA-based corpus of 159 papers across nine modalities, using an eight-level representational taxonomy and separate tests for describing, preserving, computing, and efficiently learning structure.It also treats hybrid systems such as Chronicle, TabPFN, and PatchTST as counterfactuals because their performance cannot be attributed to language alone, while identifying a still-unresolved experiment that varies only the channel supplying structural bias.Source: §1 · pp. 3–4

2 Scope, Methodology, and Problem Formulation

This review examines whether language-based models can replace specialized architectures by synthesizing 159 papers across nine modalities under a framework that distinguishes describing, preserving, computing, and learning structure. It shows that apparent disagreements often reflect different experimental conditions, mechanisms, definitions, and evaluation standards, motivating a more precise historical and structural analysis.

2.1 Survey methodology

  • The PRISMA-based review screened literature from 2016–2026 across nine structured and non-linguistic modalities, retaining 159 independently verified papers.Searches combined modality-specific queries with backward and forward citation tracing and targeted searches for theoretical and recent work.Source: §2.1 · p. 4

2.2 Positioning relative to existing surveys

  • Unlike prior modality-specific or limited cross-modal surveys, this review applies one framework linking architectural inductive bias, representation, computation, and learning across modalities.Its central contribution is the four-way describe/preserve/compute/learn distinction and synthesis of empirical findings with learning theory.Source: §2.2 · pp. 4–5

2.3 What the field actually disagrees about

  • The literature’s disagreements are classified as empirical, mechanistic, definitional, or evaluative because each requires a different form of resolution.The review uses these categories to distinguish disputes about measured results, causal explanations, terminology, and the scrutiny applied to reported evidence.Source: §2.3 · pp. 5–6
  • Empirical and evaluative disputes qualify strong replacement claims: controlled ablations found no performance loss after removing language components, while 92.2% of Tabula-8B’s reported gain was recovered without tabular exposure [28].A graph encoding change altered accuracy by 61.8% [21], and systematic comparable audits remain unavailable across most reviewed benchmarks.Source: §2.3 · pp. 5–6
  • The review defines replacement as implementing the structural function within the language model rather than delegating it to an external tool, while noting that inferential bias can sometimes match architectural bias.Mittal et al. show this for exchangeable sequence modeling, leaving generalization across modalities and structural properties unresolved.Source: §2.3 · p. 6

2.4 Historical evolution

  • Before LLMs, research primarily encoded structure directly into computation, while general-purpose architectures such as Perceiver IO and Gato pursued cross-domain processing without eliminating specialization costs.Examples included equivariant convolutions [15], permutation-invariant set architectures [47], and relational message passing.Source: §2.4 · pp. 6–7
  • From 2022 onward, language-mediated approaches tested whether textual representations and pretrained models could reduce specialization, but 2024–2026 evidence shifted toward ablations, contamination audits, theoretical limits, joint pretraining, and renewed specialized scaling.Chronicle jointly pretrains language and time-series representations, while orchestration increasingly uses language models to direct specialized models or tools rather than replace their computation.Source: §2.4 · pp. 7–8

3 A Unifying Taxonomy: Representation Regimes and the Four-Way Distinction

The section separates where structural computation occurs from what a model does with structural information, showing why language-based systems cannot be assessed through accuracy or language-model presence alone. It defines eight representational regimes and four independent properties—describe, preserve, compute, and learn—for evaluating replacement claims.

3.1 Eight representational regimes

  • The eight regimes locate structural computation across language-only reasoning, executable artifacts or tools, separate encoders, and architectures without language, rather than forming a uniformly graduated scale.Levels 1–4 differ by linguistic scaffolding; Levels 5–6 move computation outside the language model; Level 7 uses separate encoders; Level 8 removes language entirely.Source: §3.1 · p. 9, Figure 2 · p. 10
  • The taxonomy’s boundaries distinguish orchestration from replacement and recur across modalities, while compositional cases such as protein structure tokens combine Level 7 computation with Level 2 representation.Time-LLM is classified at Level 7 because removing its language component does not hurt performance, and the same boundary separates modalities including tabular, graph, time-series, vision, point clouds, and knowledge graphs.Source: §3.1 · p. 11

3.2 What “replace” means

  • Replacement claims must distinguish functional, statistical, computational, practical, robustness, universality, and structural properties, because matching one does not establish the others.Tan et al. ’s ablated forecasters match functional performance with roughly 1000× fewer parameters, but this does not establish statistical replacement; apparent functional or statistical replacement usually fails or lacks direct structural testing.Source: §3.2 · pp. 12–13, Table 1 · p. 12

3.3 The central distinction: describe, preserve, compute, learn

  • Describe, preserve, compute, and learn are independent dimensions: descriptive ability can coexist with encoding loss, non-invariant computation, or inefficient learning, so each property requires separate evaluation.Fatemi et al. [21] report a 61.8-point accuracy swing from textual encoding alone, while the synthesis finds descriptive competence relatively uniform but preservation, computation, and learning established unevenly and less often.Source: §3.3 · pp. 13–15

4 Major Methodological Families: A Modality-by-Modality Synthesis · 4.1 Tabular Data

Across modalities, the review applies a common test of whether language-mediated systems merely perform tasks or also preserve and compute the structures underlying specialized learning. In tabular data, language models show narrow few-shot benefits, but contamination, scaling limits, and absent structural evidence prevent claims of replacement.

4 Major Methodological Families: A Modality-by-Modality Synthesis

  • The review uses one analytical template: identify the specialized inductive bias, classify the language alternative, evaluate positive and qualifying evidence, isolate remaining specialized computation, and test structural learning claims.The framework distinguishes describing, preserving, computing, and learning relevant structure, with the resulting cross-modal classification summarized in Table 2.Source: §4 · pp. 15–16, Table 2 · p. 16

4.1 Tabular Data

  • Tabular learning traditionally uses tree ensembles and synthetic-prior transformers because its sparse, nonlinear feature interactions differ from the smooth sequential structure used for language pretraining.This motivates comparing language-mediated models against general-purpose specialized learners rather than narrow classical baselines.Source: §4.1 · pp. 16–17
  • Specialized tabular learners remain strong baselines: TabPFN can match tuned gradient-boosting ensembles on small datasets at low inference cost without language.TabPFN-v2, TabPFN-2.5, and TabICLv2 extend this paradigm to datasets with up to 10,000 rows.Source: §4.1 · p. 17
  • TabLLM [34] and LIFT [16] exploit pretrained knowledge with very few labels, but gradient-boosted trees overtake them as labeled data increase while using substantially less computation.These approaches serialize rows through prompting or few-shot demonstrations, occupying Levels 2 and 4.Source: §4.1 · p. 17
  • Tabula-8B initially reported a 5–15 percentage-point advantage, but 92.2% of the improvement was recovered through instruction-tuning without tabular exposure, alongside detected train–test contamination.Silvestri et al. also found contamination in four of eight widely used tabular benchmarks, while Bordt et al. [10] reported poor genuinely in-context statistical performance as dimensionality increases.Source: §4.1 · p. 17
  • Across 142 IID, temporal, and grouped-shift datasets, tabular foundation models were strongest in small-to-medium IID settings, while tree-based and deep-learning methods remained competitive or superior elsewhere.A separate unreplicated study reports systematic LLM degradation with dimensionality while classical baselines remain comparatively stable.Source: §4.1 · p. 17
  • Tabular language-mediated systems achieve comparable prediction mainly in extreme few-shot settings, but no study demonstrates replacement of specialized tabular structure or inductive bias.Tabula-8B’s stronger claim did not survive contamination analysis, while evidence supports descriptive competence and restricted functional performance rather than structural replacement.Source: §4.1 · p. 17

4.2 Graph-Structured Data

Graph data directly exposes the gap between describing structure and preserving it computationally: language models remain sensitive to serialization, while successful systems typically retain explicit structural pathways.

Permutation invariance

  • 61.8-point accuracy swings under different textual encodings show that decoder-only models do not automatically preserve graph permutation invariance [21].Relabeling the same graph can substantially degrade graph-isomorphism detection, and fine-tuning may trade reduced relabeling sensitivity for increased formatting sensitivity.Source: §4.2 · pp. 17–18

Structural pathways

  • GraphGPT and LLaGA couple learned graph encoders to language models [13], while GraphInfer-Bench reports LLM approaches weaker than conventional GNNs.LLM agents with extensive tool access also struggle with sophisticated graph analysis.Source: §4.2 · p. 18

Knowledge-graph completion

  • SimKGC outperforms TransE, ComplEx, and RotatE, but textual entity and relation descriptions give it information unavailable to structural baselines.Smaller fine-tuned models can also outperform zero-shot frontier models on the same task, implicating task-specific adaptation.Source: §4.2 · p. 18

Overall finding

  • Direct tests find no general structural replacement: language models violate permutation invariance, and graph–LLM systems retain non-linguistic structural pathways.Changing only textual encoding can shift accuracy by 61.8 points, while Levels 7–8 systems use graph encoders or other structural components.Source: §4.2 · pp. 17–18

4.3 Time-Series Forecasting

Time-series forecasting offers strong controlled evidence against automatic replacement of temporal inductive biases by pretrained language models. Across tested architectures, language components add little computational or structural value, while jointly learned multimodal approaches remain preliminary.

4.3 Time-Series Forecasting

  • Temporal forecasting depends on order, locality, and cross-time dependence, but pretrained language models have no inherent reason to encode these inductive biases.Thus, language-compatible sequence representation is distinct from computing over temporal structure: shuffling should matter to genuine temporal computation, yet had little effect in the tested models.Source: §4.3 · pp. 18–19
  • 1000× fewer parameters and up to three orders of magnitude less training time accompany equal or better performance when language components are removed from three LLM forecasters across thirteen datasets and two metrics.Randomly reinitializing pretrained weights also matches or exceeds pretrained performance, while input-sequence shuffling has little effect; generalization beyond the three tested architectures remains open.Source: §4.3 · pp. 18–19
  • Chronos and TiRex achieve strong time-series results without natural-language LLM components, while numerical methods can outperform LLM approaches in epidemic forecasting and benchmark leakage can inflate foundation-model performance.Chronos quantizes numerical values through T5, whereas TiRex uses xLSTM and achieves state-of-the-art performance on GIFT-Eval.Source: §4.3 · pp. 18–19
  • Chronicle jointly pretrains language and time-series representations through shared attention blocks, testing a different multimodal hypothesis than attaching language models to time series.The result is preliminary because the model is relatively small and lacks independent replication.Source: §4.3 · p. 19

4.4 Vision and Spatial Reasoning · 4.5 Adjacent Modalities: Does the Pattern Generalize?

Vision benchmarks show that language-mediated systems remain unreliable at preserving and exploiting continuous spatial structure, while symbolic spatial encodings and several adjacent modalities support only qualified, structure-dependent forms of substitution. Across these modalities, specialization is often retained or reintroduced when topology, execution semantics, or geometry is essential.

4.4 Vision and Spatial Reasoning

  • Visual-language systems span language-mediated, visual-token, and specialized regimes, so their performance does not show that language itself computes visual structure.Grid2Matrix finds that visual encoders preserve more information than language outputs express, while encoder, positional-encoding, and objective changes leave spatial deficits persistent.Source: §4.4 · p. 19
  • 54.93% multiple-choice and 40.93% open-ended accuracy trail humans’ 87.57% and 64.93% on spatial reasoning, while patch permutation changes predictions little.BLINK similarly reports 51.26% versus 95.7% for humans across fourteen vision tasks, and another study reports roughly 58% average accuracy on pixel geometry [22].Source: §4.4 · p. 19
  • Text-symbol grids reach 84–91% accuracy versus 60–73% for pixels, and text-only reasoning outperforms vision-augmented models on synthetic mazes and maps.These results support language as a substrate for spatial structure already discretized before processing, not general replacement of continuous visual computation.Source: §4.4 · p. 20

4.5 Adjacent Modalities: Does the Pattern Generalize?

  • Across five thinner adjacent-modality literatures, sequence-based representations work when structure is shallow, redundant, or recoverable through large pretraining; critical topology, execution semantics, and geometry retain explicit channels.These conclusions are directional because the adjacent-modality evidence is considerably thinner.Source: §4.5 · pp. 20–21
  • MoLFormer, trained on 1.1 billion SMILES molecules, outperforms supervised and self-supervised graph neural networks across ten molecular-property benchmarks, establishing functional substitution at scale.Smaller SMILES-only models show structural blindness to rings and branching until graph information returns, while surveys find general-purpose LLMs increasingly orchestrate specialized chemistry tools [33]; structural replacement remains untested.Source: §4.5 · p. 20
  • CRUXEval reports 67% and 63% accuracy for predicting short Python functions’ input/output behavior despite strong code generation, leaving the specialized-baseline boundary untested [32].Code therefore cleanly separates fluent description from reliable execution simulation because code is already symbolic.Source: §4.5 · p. 20
  • Point-cloud multimodal models outperform text-only systems on some spatial tasks yet fail basic binary relations, while PointLLM uses a specialized encoder for strong 3D captioning.This Level 7 configuration retains structural representation in the non-linguistic encoder and uses language mainly as the interaction interface.Source: §4.5 · pp. 20–21
  • SaProt and S-PLM restore structure-derived information after sequence-only protein representations proved insufficient, illustrating relocated rather than replaced specialization.The mechanism parallels chemistry and graph learning: once structural information becomes task-critical, an explicit structural channel returns.Source: §4.5 · p. 21

4.6 Language as Orchestrator: The Genuine Growth Area

Language-mediated systems are gaining practical value as orchestrators that search, compose, evaluate, and refine specialized components rather than replace their underlying structural computation. Evidence supports orchestration, while the language model’s independent contribution remains an open question.

4.6 Language as Orchestrator: The Genuine Growth Area

  • Language models orchestrate specialized computation by generating, selecting, evaluating, and refining components, while the underlying structural computation remains specialized.This evidence supports orchestration in the specific sense of Table 1, not general architectural replacement.Source: §4.6 · p. 21
  • 64% of evaluated Kaggle competitions earned medals with MLE-STAR, which retrieves baseline models and refines them through targeted, ablation-guided experimentation.LLM-FE similarly improves automated feature engineering by generating and evaluating feature-transformation programs within evolutionary search [1].Source: §4.6 · p. 21
  • A planner-guided multi-agent system reduced feature-engineering time from three weeks to one day at industrial scale.This demonstrates the practical scalability of language-mediated orchestration without showing replacement of specialized structural computation.Source: §4.6 · p. 21
  • Whether orchestration benefits arise from the language model itself or from the search and tool-use process it enables remains an open question.Component-ablation logic used in time-series forecasting may help isolate the language model’s contribution.Source: §4.6 · p. 21

4.7 Why These Comparisons Are (Not) Scientifically Valid

Meaningful comparisons between methods require explicit control and reporting of data regime, compute budget, and information availability; otherwise, performance differences may reflect unequal resources rather than representational superiority.

Validity conditions

  • Removing Tan et al.’s language-model component reduces parameter count by roughly 1000× without reducing predictive performance, making accuracy-only comparisons potentially misleading.Compute budget is therefore a necessary comparison condition, alongside data regime and information availability.Source: §4.7 · p. 21
  • Comparisons that omit data regime, compute budget, or information availability can confound representational choice with unequal data, computation, or information access.Language-mediated methods may excel in extreme few-shot settings but lose that advantage as labeled data accumulate; SimKGC and MoLFormer benefit from textual descriptions or large domain-specific pretraining corpora unavailable to specialized baselines.Source: §4.7 · pp. 21–22

4.8 Cross-Modal Synthesis

Across modalities, language-mediated systems show conditional strengths but do not provide unambiguous evidence of replacing specialized structural computation. The strongest convergence is that structural pathways are retained or restored when task-relevant relations are continuous, relational, or otherwise lost through serialization.

Cross-modal convergence

  • Matched comparisons show specialized baselines can outperform language-mediated alternatives, while language systems gain narrow advantages under extreme few-shot conditions or additional textual information.Examples include TabPFN, TiRex, GNNs, and specialized vision architectures; Tabula-8B’s apparent advantage disappears under closer scrutiny, and Tan et al. found language was not responsible for the underlying computation.Source: §4.8 · p. 22

Structural restoration

  • When language- or sequence-based representations miss task-relevant structure, systems independently restore explicit non-linguistic channels through graph attention, structure-derived tokens, or graph-based tools.CARTE, SaProt, S-PLM, and chemistry systems implement this recurring corrective strategy despite using different architectures.Source: §4.8 · p. 22

Discretizability

  • Language models perform more reliably when task-relevant structure is discretized symbolically, but remain less reliable for continuous, relational, or invariant structure such as raw geometry, topology, and temporal ordering.The key condition is whether structural relations survive serialization in a form that subsequent computation can exploit.Source: §4.8 · p. 22

Scaling and open questions

  • Large-scale domain-specific pretraining can partially compensate for representational limitations, as illustrated by MoLFormer, but whether this holds under matched compute, data, and information budgets remains unestablished.Smaller SMILES-only models exhibit structural failures, qualifying the interpretation of MoLFormer’s success.Source: §4.8 · pp. 22–23

Evidence for structural replacement

  • No modality provides unambiguous evidence of structural replacement that survives a direct test, although evidence is strongest for time series, graphs, and raw-pixel vision.Direct tests include component ablation, controlled serialization and permutation analysis, and patch-permutation experiments; tabular learning, chemistry, code, and many Table 3 benchmarks lack equivalent tests.Source: §4.8 · p. 23

5 Datasets and Benchmarks

Benchmarks measure performance under particular dataset, task, split, metric, and baseline choices, but almost never directly test representation, computation, or replacement. Their design introduces directional biases and leaves major gaps in structural evaluation, modality coverage, and fair comparison.

Benchmark scope

  • Benchmark scores support only comparisons under specific dataset, task, split, metric, and baseline choices, not direct claims about representation, computation, or replacement.A single accuracy number cannot distinguish functional, statistical, computational, or structural replacement.Source: §5 · p. 23
  • Almost every reviewed benchmark evaluates predictive performance alone, while only Tan et al.’s ablation and two invariance metrics provide representation- or replacement-level evidence.Table 3 shows that TabArena, TALENT, GIFT-Eval, BLINK, MMVP, WN18RR, and FB15k-237 primarily measure task-level accuracy.Source: §5 · p. 23, Table 3 · p. 24

Benchmark bias

  • Regime selection, contamination, and information asymmetry systematically favor language-mediated methods in several settings, whereas output-format scoring can disadvantage them independently of reasoning quality.TabArena and TALENT emphasize IID tabular data; overlap and leakage favor pretrained models; WN18RR and FB15k-237 give text-based methods information unavailable to structural baselines, while exact-match scoring creates the reverse bias.Source: §5 · p. 25

Benchmark limitations

  • Some benchmarks cannot support replacement claims because they omit specialized baselines, use easier subsets, or impose score bottlenecks unrelated to the target capability.NLGraph and GraphQA usually omit GNN baselines; MLE-Bench Lite is easier than the full benchmark; BLINK’s multiple-choice format may constrain visual scores.Source: §5 · p. 25

What benchmarks do not test

  • Information preservation and structural computation are evaluated almost nowhere, and genuine structural or computational replacement is tested only by Tan et al.’s component-level ablation.Reported end-task success in Level-6 orchestration systems cannot currently separate language-model contributions from specialized-tool contributions.Source: §5 · p. 25

Coverage gaps

  • Coverage is uneven: chemistry, protein structure, and point clouds each have one benchmark, code has none, and no benchmark directly tests serialization sensitivity or invariance.The section motivates a controlled experiment holding information content and sample budget fixed while measuring direct invariance violation.Source: §5 · pp. 25–26

6 Evaluation Methodology and Theoretical Foundations

The evaluation framework distinguishes predictive accuracy from representation fidelity, computational behavior, and true mechanistic replacement, which current benchmarks rarely measure directly. Theoretical results show that structural bias can enable efficiency or solve tasks unavailable to restricted mechanisms, while scaling improves language-mediated models but has not been shown to close matched-bias gaps.

6.1 What current metrics can and cannot distinguish

  • Accuracy and task scores measure outputs, not whether models preserve task-relevant structure, compute it, or can replace specialized components without loss.Table 4 is dominated by accuracy-based metrics; only direct invariance tests target structural computation, while mechanistic replacement requires controlled ablation.Source: §6.1 · p. 26, Table 4 · p. 27
  • Only two reviewed studies measure structure-preserving invariance, whereas controlled ablations provide the strongest evidence that a component is unnecessary.Aggregate means can conceal failures on structurally harder cases, so benchmark accuracy alone cannot establish computational replacement.Source: §6.1 · p. 26

6.2 Why the empirical pattern has a theoretical floor

  • Formal results support both favorable matches and architecture-specific limits: transformers can be optimal for selected function classes, but one-layer softmax attention cannot solve several compositional tasks regardless of width.Kim et al. and Wakayama and Suzuki establish positive results under stated assumptions, while Kozachinskiy et al. show Strassen attention removes the compositional barrier by changing the mechanism.Source: §6.2 · p. 28
  • Theoretical explanations remain unsettled: Hill et al. find distribution-shift failures inconsistent with general OLS execution, while prompting alone did not reliably induce order-invariant behavior in Yoon et al..Pretraining diversity can improve generalization above a critical task-diversity level, but Azizian and Hasan identify a robustness–sample-efficiency trade-off, leaving the sample complexity of learning graph or permutation-invariant structure unresolved.Source: §6.2 · pp. 28–29
  • Encoding structure by construction can reduce resource requirements: permutation-invariant architectures achieve a 2DN latent-dimension bound, while explicit invariance matches transformer performance with roughly an order of magnitude fewer parameters.Tabaghi and Wang provide the formal bound, and Chiu et al. report the empirical parameter-efficiency result.Source: §6.2 · pp. 28–29

6.3 Scaling is real, sub-linear, and does not by itself close the gap

  • Approximately 0.4 scaling exponents show sub-linear improvement for tabular in-context learning, but no reviewed study directly measures whether scaling eliminates the gap to a matched-bias architecture.Reported scaling determinants differ across modalities, including model depth, architecture type, and scale range; therefore individual loss improvements do not establish convergence of Δ(N).Source: §6.3 · pp. 29–31, Figure 4 · p. 30

7 Comparative Synthesis

Across modalities, language-mediated models are behaviorally competitive in specific regimes but do not generally replace specialized structure. The recurring remedy is to restore missing inductive bias through modified attention, structural representations, or separate structural pathways, while several conclusions remain evidence-limited.

7.1 Why the cross-modality pattern is mechanistic?

  • Cross-modality convergence is best explained by shared architectural constraints: when sequence representations omit task-relevant structure, successful systems restore it internally or through a separate structural pathway.This pattern is supported by domain-agnostic limits on permutation invariance and compositional operations, while scale, prompting, and additional data alone do not resolve the tested failures.Source: §7.1 · pp. 31–32

7.2 The claim map

  • Specialized inductive biases remain necessary, whereas language provides a genuine interface and orchestration capability and predictive competitiveness only in identifiable regimes.Supporting examples include TabPFN’s 230× inference speedup over tuned boosting ensembles, PatchTST’s state-of-the-art long-horizon forecasting, Set Transformer’s permutation-invariant attention, and GraphBFF’s predictable scaling.Source: §7.2 · p. 32, Table 5 · p. 33
  • General architectural replacement is not supported: accuracy matches establish at most functional replacement under tested conditions, while direct representation and computation checks do not establish structural replacement.Language-mediated systems remain competitive in extreme few-shot prediction, discretizable layouts, textually annotated knowledge graphs, and large-scale single-modality pretraining, but these advantages do not generalize to stricter replacement senses.Source: §7.2 · pp. 32–33

7.3 Answers to the central questions

  • Language is primarily an interface, with high confidence in tested regimes, because it can describe and orchestrate domains without reliably preserving permutation invariance, temporal structure, or continuous spatial geometry.Prompting rarely recovers exact computational invariance: Yoon et al.’s prompting and formatting intervention failed on real listwise tasks, while Kozachinskiy et al. identify an architectural ceiling.Source: §7.3 · p. 34

7.4 Limitations and failure modes

  • The review’s conclusions are limited by uneven modality coverage and benchmarks that often were not designed to test representation preservation or replacement directly.Formal results support scale-independent failures for non-permutation-invariance and compositional reasoning, whereas contamination and the vision-to-language bottleneck require more cautious, task-specific interpretation.Source: §7.4 · pp. 34–35

8 Open Problems and Future Work

The review identifies an untested relationship between structural complexity, scale, and the performance gap between language-mediated and specialized architectures. It proposes controlled synthetic experiments and concrete follow-up studies to measure this gap structurally, computationally, and functionally.

8.1 The precise gap

  • No existing study measures how the gap between matched language-mediated and specialized architectures varies with model scale or structural complexity.The proposed quantities are Δ(N) across scale and Δ(c) at fixed scale, using synthetic tasks, controlled information and sample budgets, and direct invariance-violation metrics.Source: §8.1 · pp. 36–37

8.2 Research questions and hypotheses

  • The open questions test whether depth, explicit instructions, demonstrations, or task properties determine when language-mediated models match or fall short of specialized architectures.They distinguish architectural ceilings from depth limitations, inert from useful linguistic bias, smooth in-support generalization from support-boundary transitions, and discretizability from other complexity moderators.Source: §8.2 · pp. 37–38

8.3 Minimal experimental design

  • The minimal protocol compares five representation arms across synthetic structural task families while measuring direct invariance violations, prediction cost, accuracy, and contamination resistance.The arms include matched specialization, naïve serialization, explicit invariance instructions, demonstrations alone, and lossless canonical serialization; fresh generators, randomized names, and a held-out structural regime control contamination.Source: §8.3 · pp. 38–39

8.4 Concrete future directions

  • Future work should report Δ(N)-versus-scale curves, test depth on Kozachinskiy et al.’s three compositional tasks, and ablate textual descriptions in SimKGC and specialized tools in orchestration agents.These studies target missing comparisons involving matched structural baselines, multi-layer standard attention, information availability in knowledge graphs, and the division of labor between language models and orchestrated tools.Source: §8.4 · p. 40

9 Conclusion

Across nine modalities, behavioral parity with specialized architectures does not establish replacement: direct tests of representation, computation, and architecture rarely support broader replacement claims. Instead, specialization often relocates into non-linguistic components, leaving the scale-dependent performance gap unresolved.

Conclusion

  • Language-mediated systems can match specialized architectures predictively, but direct tests show parity rarely extends beyond narrower replacement regimes.Identified regimes include extreme few-shot prediction, discretizable spatial layouts, textually annotated knowledge graphs, and sufficiently large pretraining.Source: §9 · p. 40
  • Accuracy, statistical efficiency, computational cost, and structural equivalence are distinct claims, so benchmark performance alone cannot establish architectural replacement.Apparent success beyond identified regimes may involve contamination, format familiarity, or information supplied through non-linguistic channels.Source: §9 · pp. 40–41
  • When language-only representations are insufficient, successful systems frequently reintroduce specialization through tools, specialized tokens, or architectural scaffolds.Language can also direct and compose specialized components, but this represents integration rather than replacement; the pattern recurs across communities without being established as universal.Source: §9 · p. 41
  • The unresolved question is whether the gap to a matched-bias architecture, ∆(N), decreases and eventually closes with scale.Kozachinskiy et al. and Goddard et al. constrain respective settings but do not predict the performance gap as a function of scale or structural complexity.Source: §9 · p. 41

Related papers

Loading 2608.28980v1…