PaperlayerView exact revision on arXiv

Source-linked AI summary

Shaping Schema via Language Representation as the Next Frontier for LLM Intelligence Expanding

Zhiqin Yang, Yuhan Liu, Jingwen Fu, Pei Fu, Bo Han, Masashi Sugiyama, Nanning Zheng

arXiv:2605.09271v1cs.AI

TL;DR

Natural language can constrain how LLMs express and use knowledge on complex tasks. This paper formalizes schema shaping through language representation, reviews evidence, and experimentally shows that representation choices change performance and internal activations.

  • Problem

    Natural language has limited expressive capacity, while stored knowledge does not guarantee effective activation, organization, or use in LLMs.

  • Method

    The paper formalizes schema and language representation, synthesizes prior practices, and conducts controlled experiments comparing representations of identical tasks.

  • Results

    Across reviewed practices and controlled experiments, deliberate language representations produce performance gains, while representation changes systematically alter task performance and internal feature activations.

  • Takeaways & Limitations

    The findings position language representation design as a promising direction for expanding LLM intelligence beyond reliance on parameter scaling alone.

  • Takeaways & Limitations

    The formalization assumes language representations accurately describe questions and answers and posits a schema space with a small approximation value ϵ.

Abstract

from arXiv · show

Although natural language is the default medium for Large Language Models (LLMs), its limited expressive capacity creates a profound bottleneck for complex problem-solving. While recent advancements in AI have relied heavily on scaling, merely internalizing knowledge does not guarantee its effective application. Defining language representation as the linguistic and symbolic constructs used to map and model the real world, this paper argues that shaping schemas through advanced language representation is the next frontier for expanding LLM intelligence. We posit that an LLM's knowledge activation and organization -- its schema -- depends heavily on the structural and symbolic sophistication of the language used to represent a given task. This paper contributes both a formalization of this claim and the empirical evidence to support it. With a new formalization, we present multiple lines of evidence to support our position: Firstly, we review recent empirical practices and emerging methodologies that demonstrate the substantial performance gains achievable through deliberate language representation design, even without modifying model parameters or scale. Secondly, we conduct controlled experiments showing that LLM performance and its internal feature activations vary under different language representations of the same underlying task. Together, these findings highlight language representation design as a promising direction for future research.

1 Introduction

The paper argues that natural language’s limited representational channel constrains how LLM knowledge is activated, organized, and used. It proposes shaping schemas through increasingly sophisticated language representations as the next frontier for expanding LLM intelligence, supported by formalization, review, and controlled experiments.

  • LLMs can internalize vast knowledge through scaling, but its presence alone does not guarantee effective activation, organization, or use.
  • Natural language provides a narrow channel for encoding, accessing, and composing real-world complexity during inference.
  • A schema is the representational and organizational framework determining how knowledge is invoked, related, and operationalized for a task.
  • The paper formalizes schema, language representation, and representation design within a unified sophistication framework spanning levels 0 through 3.
  • The paper supports its claim through a review of empirical practices and methodologies, controlled experiments isolating language representation effects, and a roadmap toward AI-constructed formal languages.

2 Background: Language and Intelligence

General intelligence involves integrating cognitive functions for advanced reasoning and complex problem-solving, with schemas organizing knowledge and guiding interpretation. Language encodes cognitive schemas that shape thought, while LLM research indicates schema-like structures and shows that inputs and prompting can alter their activation and performance.

  • 2 Background: Language and Intelligence: General intelligence integrates diverse cognitive functions to enable advanced reasoning and complex problem-solving.Schema provides a framework for understanding how knowledge is organized through connections to prior experiences.
  • 2 Background: Language and Intelligence: Schemas organize knowledge by connecting prior experiences and guiding interpretation of new information.
  • 2 Background: Language and Intelligence: Language bridges cognition and interaction by encoding cognitive schemas that shape how people think and act.The weak Sapir–Whorf hypothesis holds that linguistic systems subtly guide thought by framing interpretive schemas rather than determining thought absolutely.
  • 2 Background: Language and Intelligence: LLMs exhibit schema-like structures that shape performance, alongside low-level semantic correlation structures resembling those observed in humans.
  • 2 Background: Language and Intelligence: Different input content can activate distinct schemas in LLMs, while in-context information modulates embeddings and attention weights across layers.Chain-of-thought prompting can elicit reasoning capabilities even when the provided reasoning is invalid.

3 Language Representation Design

This section formalizes language design as selecting a language space that represents questions and answers while inducing a mapping close to the target task. It distinguishes language design from prompt engineering and frames schema alignment as the objective governing prediction accuracy.

  • Language Design Formalization: Language design maps questions and answers into a common language space so the induced mapping approximates the target function.The target mapping is f: Q → A, while the language model operates on linguistic representations.
  • Language Design Formalization: Prompt engineering transforms only the input question, whereas language design changes both question and answer representations under stricter representation constraints.Prompt transformations may add information or hints before encoding, while the language map is typically required to be an isomorphism.
  • Schema Representation: A schema space S supports mappings from questions to schemas and schemas to answers, with each task associated with a target schema.The framework introduces πs: Q → S and πa: S → A, and denotes the target schema for task f as sf.
  • Schema Alignment: Schema mismatch SM(L) measures the extra bits needed to reroute language-evoked internal patterns toward task-required patterns, with SM(L) = 0 defining a schema-matched language.The framework links prediction discrepancy to schema mismatch and recasts language design as minimizing SM(L) over admissible language spaces.

4 Expanding the Intelligence Frontier with Language Representation Design

Language representation design addresses natural language’s instability and expressive limits by optimizing existing methods and extending reasoning into domains requiring formal models. Controlled logic-circuit experiments show that representation choice changes both LLM performance and the internal reasoning schemas it induces.

  • Section scope: The section studies representation design beyond the natural-language baseline, covering optimization of established methods, expansion into new domains, and controlled experimental validation.Levels 1–2 optimize existing methods, Level 3 addresses new domains, and Section 4.3 provides supporting experiments.
  • Limitations of Natural Language: Natural language produces unstable outputs because it lacks explicit logical constraints, contains semantic ambiguity, and forces models to infer schemas from noisy structural cues.These limitations suggest that performance bottlenecks often reflect an inadequate interface rather than absent latent capability.
  • Frontier Expansion: Language representation design expands the intelligence frontier by constructing formal models for high-dimensional scientific logic, physical dynamics, causal relations, and state evolution.Scientific formalization creates executable and verifiable reasoning spaces, while world modeling represents physical laws and actionable state transitions.
  • Experimental Design: The logic-circuit experiment uses |Q| = 100 semantically equivalent questions encoded into 15 distinct representations, preserving deterministic task semantics across formulations.Circuits contain AND, OR, NOT, XOR, NAND, and NOR gates, with randomly generated topologies and Boolean input assignments.
  • Internal Schema: KAI and KOI act as complementary proxies for schema activation and organization, with Canonical Boolean Expressions supporting efficient attention anchoring and logical-path reuse.Petri Net performs poorly because non-sequential dependencies cause distant-reference jumps and structural oscillation, showing that syntax and topology affect schema stability and performance.
  • Performance Results: Canonical Boolean Expressions achieve the highest accuracy, while Natural Language remains broadly adaptable but incurs higher token usage and reasoning costs.The performance gap is attributed to alignment between representation syntax and the circuit task’s underlying logic.

5 Alternative Views

The section contrasts scaling and tool augmentation as routes to advancing LLM intelligence with the paper’s view that language representation design is the next frontier. It argues that new structured, interpretable, and learnable representations could reorganize intelligence by supporting reasoning, collaboration, and interpretability.

  • Alternative View 1: Scaling advocates view intelligence as an emergent, asymptotic property of increasing model size and training data that could master any language-expressible task.This perspective treats intelligence as a continuum rather than a discrete architectural or representational leap.
  • Alternative View 2: Tool augmentation emphasizes system composition, with an LLM orchestrating search engines, code interpreters, and symbolic planners to extend problem-solving without further scaling.The LLM serves as the cognitive core in a collaborative system of specialized tools.
  • Compared with Our View: The paper argues that scaling and tool augmentation rely on existing human-designed languages that were not optimized for large models’ task-specific inductive and representational biases.These substrates include natural language, programming languages, and formal symbolic notations.
  • Compared with Our View: New language representations could be structured, interpretable, and learnable by humans and machines while encoding reasoning processes, abstractions, and world models more naturally.The proposed representations are intended to bridge the gap between existing languages and model-specific representational needs.
  • Compared with Our View: Under this view, intelligence does not merely scale; it reorganizes through language as infrastructure for higher-order reasoning, collaboration, and interpretability.The claim identifies language design as the source of the next qualitative leap in intelligence.

6 Open Problems and Roadmap

The paper identifies principled design, model adaptation, and theoretical understanding of language representations as open problems, and proposes a four-phase roadmap to overcome the natural-language bottleneck. The roadmap begins with systematic representation design and representation-aware alignment.

  • Open Problems: Principled methods for designing, adapting, and theoretically grounding effective language representations remain largely open despite growing empirical evidence of performance gains.These challenges define the emerging research frontier.
  • Q1: Systematic Design: Effective representation design requires identifying essential structural elements, relating task properties to representation choices, and finding design principles that transfer across domains.Candidate elements include symbolic constraints, intermediate variables, modular decomposition, and control tokens.
  • Representation Adaptation: Models may fail to exploit effective representations because pretraining distributions mismatch task-specific constructs, motivating instruction tuning, representation-aware finetuning, in-context strategies, and architectural biases.These methods aim to help models internalize and use new schemas.
  • Theoretical Understanding: A theoretical account is needed to explain how linguistic structure, abstraction level, and compositional primitives shape LLM activations, attention patterns, feature compositions, and internal schema organization.The passage highlights uncertainty about when representations promote more compositional schemas.
  • Roadmap: The proposed roadmap treats representation design as a core modeling decision and begins with optimization-based systematic design, followed by representation-aware finetuning for new language designs.The second phase aims to avoid catastrophic interference while adapting models to optimal representations.

7 Conclusion

The paper argues that language representation design is a critical next frontier for expanding LLM intelligence and supports this claim through empirical review and controlled experiments. These experiments examine both task performance and internal feature activations under representation design.

  • Language representation design is presented as a critical next frontier for expanding LLM intelligence.
  • The paper surveys empirical practices and emerging methodologies that achieve substantial performance gains through deliberate representation design.
  • Controlled experiments demonstrate that language representations affect both task performance and internal feature activations.

A Proof of Proposition 3.6 · B More Related Information about Our Position

The proof derives prediction-error bounds by relating Fisher–Rao distances in action space to representation-space geometry through the Fisher Information Matrix, spectral bounds, geodesic integration, and KL divergence. Related analyses position language representation design as a way to induce task-specific schemas and expand LLM intelligence beyond scaling alone.

  • A Proof of Proposition 3.6: The proof defines prediction error as the squared Fisher–Rao distance between target and language-induced action distributions.This establishes the output-space distance used throughout the argument.
  • A Proof of Proposition 3.6: The action mapping pulls the Fisher metric back from action space to schema-representation space, where its local geometry is determined by the Fisher Information Matrix.The chain rule connects action-space velocity to representation-space velocity through the Jacobian of the action mapping.
  • A Proof of Proposition 3.6: The quadratic form of the Fisher Information Matrix is bounded between its minimum and maximum eigenvalues for every schema and tangent vector.The bound is expressed as λmin(Iπa(s))∥v∥2 ≤ v⊤Iπa(s)v ≤ λmax(Iπa(s))∥v∥2.
  • A Proof of Proposition 3.6: Information geometry connects schema mismatch to KL divergence because KL divergence is the canonical divergence associated with the Fisher Information Metric.The proof uses the local metric relation and the stated global relationship between squared Fisher–Rao distance and KL divergence.
  • B More Related Information about Our Position: The paper compares alternative intelligence-expansion perspectives to illustrate their limitations and the frontier potential of language representation design for current LLMs.Table 2 provides the comparative overview, while Figure 3 visualizes language design.
  • A Proof of Proposition 3.6: The final proposition follows by substituting 2 SM(L) for D2π into the integrated inequalities and simplifying the coefficients.The displayed final result is obtained after the substitution and coefficient simplification.
  • B More Related Information about Our Position: Different language representations induce distinct schemas, and well-designed representations activate appropriate knowledge regions and organize them into task-specific reasoning structures.The example describes a structured representation activating geo-spatial, logistics, and environmental knowledge and organizing it into a coherent reasoning graph.

C More Detailed Experimental Setup · C.1 Detail about Different Language Format for Logic Circuit Simulation Task

The experiment converts each circuit instance into 15 linguistic representations spanning natural language, technical notations, graph and matrix encodings, execution traces, and formal symbolic formats. These representations are deliberately designed to test trade-offs among structural fidelity, compactness, parsing simplicity, logical clarity, execution transparency, and formal semantics.

  • C.1 Detail about Different Language Format for Logic Circuit Simulation Task: Natural Language extensively describes gate functions, input connections, and processing flow, requiring redundancy and producing the longest prompts.This format is intended to expose natural language’s limitations for structured tasks.
  • C.1 Detail about Different Language Format for Logic Circuit Simulation Task: Netlist Language, Graph Adjacency Notation, and Matrix Representation encode circuits through hardware-style declarations, explicit graph edges, or binary connection matrices.These formats emphasize structured circuit description, graph structure, or dense numerical encoding.
  • C.1 Detail about Different Language Format for Logic Circuit Simulation Task: Lisp Tree Notation recursively represents gate operations, while Dataflow Language organizes gates into pipeline stages to capture hierarchical dependencies or temporal signal flow.The tree format resembles abstract syntax trees, whereas the dataflow format expresses computation as sequential transformations.
  • C.1 Detail about Different Language Format for Logic Circuit Simulation Task: Partial Truth Table, Layered Execution Plan, and Signal Propagation Trace make computation progressively explicit through evaluation traces, computational layers, or time-step signal values.These formats emphasize execution order and provide increasingly explicit views of signal flow.
  • C.1 Detail about Different Language Format for Logic Circuit Simulation Task: Compact Gate Notation, Reverse Polish Notation, and Dependency Chain Language target compactness, parsing simplicity, and explicit causal or logical dependencies.Their mechanisms include abbreviated gate syntax, postfix operators, and dependency arrows with logical operators.
  • C.1 Detail about Different Language Format for Logic Circuit Simulation Task: Constraint Satisfaction Format, Canonical Boolean Expression, and Petri Net Notation represent circuits as Boolean constraints, complete output formulas, or token-based concurrent systems.These formats emphasize declarative satisfaction, elimination of intermediate signals, or concurrency-theoretic semantics.
  • C.1 Detail about Different Language Format for Logic Circuit Simulation Task: The first seven representations cover common circuit descriptions, whereas the latter eight optimize compactness, parsing simplicity, logical clarity, execution transparency, or formal semantics.The design rationale organizes the formats by the aspect of circuit description they prioritize.

C.2 Formal Definition of KAI and KOI

KAI measures how strongly a language representation focuses attention on task-relevant logic, while KOI measures the consistency of reasoning structure across model layers. Both operate on attention matrices already computed during inference with O(LN^2) complexity.

  • Knowledge Activation Index (KAI): KAI quantifies focused computational state by combining normalized attention purity with attention mass assigned to critical logical nodes across layers.A power factor increases sensitivity to high-fidelity activations during aggregation.
  • Knowledge Activation Index (KAI): A high KAI indicates that the representation suppresses semantic noise and concentrates computational budget on task-relevant logic nodes.
  • Knowledge Organization Index (KOI): KOI measures consistency and convergence of internal reasoning by comparing vectorized attention structures between adjacent layers.Its higher-order power transformation exposes structural drift that residual similarity can mask.
  • Knowledge Organization Index (KOI): A high KOI suggests a consistent logical flow in which the representation propagates dependencies across layers with minimal structural correction or cognitive oscillation.
  • Complexity and Robustness of KAI and KOI: O(LN^2) is the computational complexity of both metrics for L layers and sequence length N, using attention matrices already computed during a single forward pass.They require no additional forward or backward computation and are negligible relative to inference cost.

C.3 Experimental Results

The experiments compare GPT-5-chat and Qwen3-32b across language representations for logic-circuit simulation, relating reasoning accuracy, internal dynamics, and computational efficiency. Across architectures, CBE is identified as optimal, while representation choice produces accuracy–efficiency trade-offs.

  • GPT-5-chat and Qwen3-32b were evaluated on logic-circuit simulation using different language representations, with Figure 5 comparing internal dynamics, reasoning accuracy, and computational efficiency.
  • 100% accuracy was achieved by four representations in GPT-5-chat, while Canonical Boolean Expressions (CBE) simultaneously delivered the highest accuracy and superior computational efficiency.
  • GPT achieved higher accuracy than Qwen3-32B with Graph Adjacency Notation but showed markedly lower efficiency, indicating a computational cost for deciphering suboptimal representations.
  • Natural Language maintained high accuracy but consumed nearly 4× as many completion tokens as the optimal language.

C.4 Representations Induce Disjoint Internal Geometries · C.5 Attention Pattern Analysis · C.6 Examples for different language representation

Across equivalent logic-circuit tasks, language representations produce sharply distinct internal geometries that persist through the network, while attention analysis compares representation-specific processing across early, middle, and late layers. Examples span 15 representation formats used during inference.

  • C.4 Representations Induce Disjoint Internal Geometries: 0.93 silhouette score and 96.8% between-format variance ratio show sharply disjoint final-layer clusters for identical logic-circuit problems rendered in 15 representations.Each cluster corresponds to one surface representation, indicating that representational identity dominates final-layer geometry despite identical logical content.
  • C.4 Representations Induce Disjoint Internal Geometries: Silhouette and variance-ratio metrics remain ≥0.82 across layers 0, 16, 32, 48, and 64, with 0.94 at layer 0 and 0.93 at layer 64.Representation-specific subspaces are already separated at the embedding layer and persist through the final layer.
  • C.4 Representations Induce Disjoint Internal Geometries: The observed geometry is dominated by how the question is written rather than what it asks, challenging representation-invariant internalization of logic circuits.The disjoint clusters accompany an order-of-magnitude accuracy gap between the best and worst formats.
  • C.5 Attention Pattern Analysis: Attention weights are extracted from Qwen3-32B at layers 6, 24, and 48 to examine how language representations shape processing schemas during logic-circuit simulation.The layers correspond to early processing, middle processing, and output generation.
  • C.5 Attention Pattern Analysis: For each layer and representation, heads are selected by attention-distribution variance as Most Varied, Medium Varied, or Least Varied to compare processing strategies.These roles respectively target focal attention, structural dependencies, and stable baseline processing.
  • C.5 Attention Pattern Analysis: Figures 7–15 visualize attention heatmaps across representations, layers 6, 24, and 48, using red for high attention and blue for low attention.The visualizations organize rows by language representation and columns by heads with differing attention-distribution variances.
Loading 2605.09271v1…