PaperlayerView exact revision on arXiv

Source-linked AI summary

Structured but Silent: Probing Capability Requirements in LLM Hidden States

Kyojun Choo, Minsoo Song, Yunju Kang, Chanjun Park

arXiv:2610.08018v1cs.CL

TL;DR

Tool-use research often focuses on triggering or selecting tools, leaving open whether query-side capability requirements are represented before generation. TACIT tests this with a three-axis taxonomy and pre-generation linear probes; the probes decode requirements across four model families more reliably than the models verbalize them.

  • Problem

    Whether multidimensional capability requirements implied by queries are encoded in LLM hidden states before generation remains an open question.

    Source: §1 · p. 1

  • Method

    TACIT defines eight classes across Source, Transformation, and World Effect and trains linear probes on pre-generation hidden states from four open-weight LLM families.

    Source: §1 · p. 1

  • Results

    Across four models, probes achieve 92.50–94.78% joint accuracy and exceed four-shot verbalization by 25.33–38.15 percentage points on the 920-query test set.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Capability labels are more reliably accessible through linear probes than expressed through generation under the evaluated prompting protocol.

    Source: §6 · p. 9

    The experiments do not test whether models use probed representations when selecting or executing tools, and tool availability, permissions, planning, and execution feedback are outside the evaluation.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Reliable tool use requires more than triggering a mechanism or matching a query to an API description. Before selecting a specific tool, an agent must first infer the capability requirements implied by the user query. In this paper, we investigate whether these query-side capability requirements are linearly decodable from LLM hidden representations prior to generation, and how this hidden-state accessibility compares with explicit verbal classification. We introduce TACIT, a framework that decomposes external requirements along three fundamental axes: Source, Transformation, and World Effect, defining eight structurally distinct capability classes. Using 1,600 balanced training queries from benchmarks, synthetic examples, and new domain scenarios, we train linear probes on pre-generation hidden states from four open-weight LLM families. Our empirical results demonstrate that fine-grained capability structures are linearly decodable with high accuracy across all models. Crucially, however, we expose a representation-to-verbalization gap: these same models are significantly less reliable when asked to explicitly classify the same queries in natural language. This disconnect indicates that information about required external capabilities is linearly accessible in LLM hidden representations but not reliably expressed, a phenomenon we define as "structured but silent."

1 Introduction

Tool-use methods often focus on triggering or selecting APIs, leaving open whether models represent the capability requirements implied by queries before generation. TACIT formalizes those requirements and finds them decodable from hidden states, while explicit verbal classifications are less reliable.

1 Introduction

  • Query-side capability inference is a prerequisite distinct from deciding which specific tool matches a query.The paper frames that inference as an intermediate abstraction for tool use.Source: §1 · p. 1
  • Whether multidimensional capability structures are encoded in hidden states before generation remains an open question beyond prior binary tool-triggering and hallucination judgments.Existing approaches also optimize tool selection through prompting, tool-description tuning, or output filtering.Source: §1 · p. 1
  • TACIT defines eight capability classes from Source, Transformation, and World Effect axes to represent external requirements implied by a query.The framework compares hidden-state probe predictions with explicit verbal classifications of the same requirements.Source: §1 · pp. 1–2
  • The study trains linear probes on 1,600 balanced queries and pre-generation hidden states from four open-weight LLM families.The models are Llama, Mistral, Qwen, and Gemma.Source: §1 · p. 1
  • 92.50%–94.78% joint accuracy across four models, while probes exceed four-shot verbalization by 25.33–38.15 percentage points on the 920-query test set.On deceptive queries, probes also beat the strongest lexical or sentence-embedding baseline by 19.55–23.64 percentage points in joint accuracy.Source: §1 · p. 2

2 Related Work

Prior work probes information in LLM representations and studies when that information fails to appear in generated text. This paper extends those lines of work to multidimensional capability requirements for tool use.

2.1 Internal Representations of Language Models

  • Prior representation studies uncover latent structures for truthfulness and factual knowledge, while tool-use studies examine hallucination risk, tool necessity, or binary triggering decisions.The paper positions fine-grained capability structures as a complementary representational question that precedes concrete tool selection.Source: §2.1 · p. 2

2.2 The Disconnect Between Knowledge and Generation

  • Prior work finds that internal signals about truthfulness, errors, or factual knowledge may not emerge in generated answers, suggesting generation can distort accessible information.The cited examples include internal veracity tracking alongside false outputs and hidden knowledge that fails to surface during token generation.Source: §2.2 · p. 2
  • Earlier representation-to-generation studies primarily address single-dimensional properties; this work evaluates the gap for structured requirements spanning three binary capability axes.It measures the discrepancy between pre-generation probes and explicit verbal classifications across four LLMs.Source: §2.2 · p. 2
  • Figure 1 depicts final-prompt hidden-state probes for Source, Transformation, and World Effect, followed by tests of deceptive-cue robustness and verbalization differences.The caption describes comparisons with lexical and sentence-embedding baselines and with the same model’s verbal classifications.Source: Figure 1 · p. 3

3 The TACIT Framework

TACIT is a diagnostic framework for testing whether query-side capability requirements are linearly decodable before generation. It combines a three-axis taxonomy, balanced query datasets, pre-generation representations, and axis-wise probes.

3 The TACIT Framework

  • TACIT evaluates whether query-side capability requirements are linearly decodable from hidden states before text generation, using probe predictions and verbalization comparisons.The framework also compares probes against lexical and sentence-embedding baselines.Source: §3 · p. 3

3.1 Three-Axis Capability Taxonomy

  • Source distinguishes available from external information, Transformation distinguishes retrieval-only from processing, and World Effect distinguishes read-only from state-changing queries.The three binary axes form eight classes; World Effect labels follow explicit query verbs, with state-changing verbs taking precedence when verbs conflict.Source: §3.1 · pp. 3–4

3.2 Dataset Construction

  • Training combines 800 benchmark-derived and 800 domain-scenario queries; the 920-query main test set includes 720 scenario and 200 benchmark-held-out queries.The domain-scenario component supplies 160 validation queries, and each scenario family stays within one split.Source: Figure 2 · p. 4, §3.2 · p. 4
  • The benchmark-derived corpus samples queries across capability classes, while the domain-scenario dataset uses shared scenarios to reduce associations between domains and labels.The scenario dataset contains 1,680 queries in 210 families across 25 domains; five domains are reserved for testing.Source: §3.2 · p. 4

3.3 Pre-generation Representation Extraction

  • For each query, the method extracts the final prompt token’s hidden state in one forward pass without generating output tokens.Under causal self-attention, that terminal position can attend to the preceding query.Source: §3.3 · p. 5

3.4 Axis-wise Linear Probing

  • Independent ℓ2-regularized logistic-regression probes are trained for each capability axis at each transformer layer.The probes use a fixed inverse regularization strength of C = 1.0, and activation features are standardized using training-split statistics.Source: §3.4 · p. 5

4 Experimental Setup

The experiments evaluate capability classification across complementary query sets, four open-weight models, text-based baselines, and four-shot verbalization.

4.1 Evaluation Datasets

  • The 920-query main test set combines new scenario families, five held-out domains, and 200 queries from benchmark sources not used in training.The 400 queries from five additional domains are reserved for testing; scenario families do not overlap across training, validation, and test splits.Source: §4.1 · p. 5
  • A separate 220-query deceptive set uses misleading wording for one capability axis while retaining a determinate gold label.It contains 77 Source, 70 Transformation, and 73 World Effect queries.Source: §4.1 · pp. 5–6

4.2 Models

  • The study probes Llama 3.1 8B Instruct, Qwen 3 8B, Gemma 2 9B IT, and Mistral 7B Instruct v0.3.Llama is the primary subject for in-depth architectural analysis; the other models establish cross-model generalization.Source: §4.2 · p. 6

4.3 Baselines

  • Text baselines include word-level TF-IDF, word-and-character n-grams, and frozen E5-large-v2 sentence embeddings, each trained on the same 1,600 queries.The baselines are evaluated on the same main and deceptive test sets as the hidden-state probes.Source: §4.3 · p. 6

4.4 Evaluation Metrics

  • Evaluation reports per-axis accuracy and joint three-axis accuracy, selecting each probe's layer by validation accuracy and breaking ties by joint validation accuracy.Main-test estimates and paired method comparisons use 95% percentile bootstrap intervals with 10,000 resamples.Source: §4.4 · p. 6

4.5 Verbalization Setup

  • Four-shot verbalization uses the same 920 test queries for all four models, with a shared prompt specifying TACIT and four fixed labeled examples.The prompt requests labels for all three capability axes.Source: §4.5 · p. 7

5 Results

Across the tested models, pre-generation probes classify capability requirements reliably, outperform text baselines on misleading-cue queries, and exceed explicit four-shot verbalization.

5.1 RQ1: Are External Capability Requirements Linearly Encoded Before Generation?

  • Llama's axis-wise probes achieve 94.02% joint three-axis accuracy (95% CI: 92.50–95.43%) before token generation.Seven of eight TACIT classes reach at least 90% accuracy; (E,T,M) is lower at 87.83%. Pairwise probe-weight cosine similarities are near zero, which alone does not establish semantic independence.Source: §5.1 · p. 7

5.2 RQ2: Do Probes Capture Abstract Taxonomy or Surface Lexical Shortcuts?

  • On deceptive queries, probes achieve 79.09% target-axis and 60.45% joint accuracy, versus 61.82–62.27% and 35.91–38.64%, respectively, for text baselines.Probes also outperform BoW on all three targeted axes, with the largest difference for Transformation (+22.86 percentage points).Source: Table 5 · p. 7, §5.2 · pp. 7–8

5.3 RQ3: Does Linearly Decodable Capability Information Surface in Explicit Generation?

  • On Llama 3.1 8B, probe joint accuracy is 94.02% versus 68.04% for four-shot verbalization, a 25.98-percentage-point difference.The probe is jointly correct while verbalization is not on 29.6% of queries; the reverse occurs on 3.6%.Source: Table 6 · p. 7, §5.3 · p. 8, Figure 3 · p. 8

5.4 RQ4: Does the "Structured but Silent" Phenomenon Generalize Across Models?

  • Across four model families, probe joint accuracy is 92.50–94.78% on the main set; four-shot verbalization reaches 54.35–69.46%, leaving a 25.33–38.15-point gap.On deceptive queries, probe joint accuracy is 58.18–62.27%, versus 38.64% for BoW, 35.91% for n-gram, and 37.27% for E5.Source: Table 7 · p. 8, §5.4 · pp. 8–9

6 Conclusion

TACIT probes recover query-side capability requirements from hidden states across four open-weight model families, while explicit classification is substantially less accurate. The conclusion frames causal links to tool use and downstream behavioral improvement as future work.

6 Conclusion

  • TACIT analyzes query-side capability requirements relevant to tool use.It is a three-axis framework.Source: §6 · p. 9
  • 92.50–94.78% joint accuracy across four open-weight LLM families shows that capability requirements are linearly decodable from pre-generation hidden states.The probes also outperform lexical and sentence-embedding baselines on deceptive queries, although accuracy decreases under misleading wording.Source: §6 · p. 9
  • 25.33–38.15 percentage points separate probe joint accuracy from explicit four-shot classification across the four models.The paper names this pattern “structured but silent”: probe-accessible capability labels are less reliably expressed through generation under the evaluated prompting protocol.Source: §6 · p. 9
  • Future work should test whether decodable representations causally influence tool selection and improve behavior in agents that execute tools.Source: §6 · p. 9

Limitations

The evaluation probes query-side capability requirements rather than the full process of tool use, and it leaves several data, evaluation, and model-scope boundaries unresolved.

Limitations

  • The study does not test whether models use probed representations when selecting or executing tools; tool availability, API schemas, permissions, planning, and execution feedback are outside scope.Testing causal use would require representation interventions and downstream tool-behavior tests.Source: Limitations · p. 9
  • Dataset construction may introduce artifacts, and training on all eight classes does not establish generalization to an entirely unseen capability class.Some original-corpus classes align with particular benchmarks and query styles; synthetic examples supplement underrepresented Mutate classes, and unseen-domain tests cannot rule out every stylistic shortcut.Source: Limitations · p. 9
  • 58.18–62.27% probe joint accuracy on deceptive queries leaves sensitivity to misleading cues, especially for Source and Transformation.The LM-generated and screened evaluation is a targeted stress test, not an exhaustive robustness measure.Source: Limitations · p. 9
  • The probe–verbalization gap reflects different access conditions: supervised hidden-state probes versus a fixed four-shot prompt and output parser.It does not establish that models could never express the labels under other prompts or training procedures.Source: Limitations · p. 9
  • Empirical scope is limited to four open-weight models in the 7–9B parameter range; generalization beyond this scope remains unresolved.The text identifies substantially larger models, multilingual settings, and explicit reasoning modes as open empirical questions.Source: Limitations · p. 9

Ethical Considerations

The study reports no human participants or PII and uses publicly accessible benchmark datasets under their licenses. It also notes potential bias from proprietary models used in data work and describes diagnostic-use boundaries.

Ethical Considerations

  • No human participants were involved, no PII was collected, processed, or used, and benchmark datasets were publicly accessible and used under their licenses.Source: Ethical Considerations · p. 9
  • Proprietary models supported data generation, provisional labeling, and automated review, which may introduce distributional or stylistic biases.The authors report using explicit taxonomy definitions, query- and family-level review, and targeted revision as quality checks.Source: Ethical Considerations · pp. 9–10
  • The diagnostic method requires intermediate activations, is tailored to open-weight models, and uses probes limited to abstract task-level capability labels.The paper says these probes have no utility beyond functional diagnostic applications.Source: Ethical Considerations · p. 10

A Dataset Source Rationale and Construction Details

The appendix section describes dataset sources, construction, and review procedures.

A Dataset Source Rationale and Construction Details

A.1 Original Corpus: Sources, Sampling, and Review

The original corpus supplies training examples for TACIT classes, while Observe and Mutate class candidates are supplemented and reviewed before balanced retention.

A.1 Original Corpus: Sources, Sampling, and Review

  • The experiments sample 100 queries per TACIT class from a 1,600-query corpus, yielding 800 original-corpus training queries.The source corpus has 200 queries per class.Source: §A.1 · p. 13
  • Observe-class candidate pools draw on corresponding Table 9 sources, with FreshQA supplementation for the limited-coverage (E,R,O) class.The target pools are approximately 400 candidates per class, or approximately 600 for (E,R,O).Source: §A.1 · p. 13
  • Mutate requests come from BFCL v3, are deduplicated and provisionally labeled by GPT-4.1-mini, then supplemented through target-class-conditioned synthesis.Synthesis targets approximately 400 candidates per class before review.Source: §A.1 · p. 13
  • Gemini 2.5 Flash independently predicts capability labels and assesses quality; reviewed curation retains 200 examples per class, prioritizing benchmark-derived examples where available.Source: §A.1 · p. 13

A.2 Domain-Scenario Dataset

The domain-scenario dataset varies TACIT capabilities within shared domains and scenarios, then uses family-based splits and review to produce training and evaluation data.

A.2 Domain-Scenario Dataset

  • Domain cards specify entities, supplied or stable information, external state, transformations, and state-changing actions, enabling axis variation within a domain.For example, calendar queries vary supplied versus current availability, direct retrieval versus time calculations, and information requests versus event creation.Source: §A.2 · p. 13
  • Each of 210 scenario families contains eight queries, one per TACIT class, within a shared domain and broad scenario, reducing domain-label association.Source: §A.2 · p. 13
  • The final domain-scenario dataset contains 210 scenario families and 1,680 queries that pass automated acceptance criteria.Initial item-level review accepted 1,635 queries (97.32%); combined item- and family-level concerns flagged 52 for targeted revision and renewed review.Source: §A.2 · p. 13
  • The 20 seen domains provide 800 training, 160 validation, and 320 in-domain test queries, while five held-out domains provide 400 test queries.Families remain within one split, though broad themes and operations may recur.Source: §A.2 · pp. 13–14
  • The final training set combines 800 original-corpus and 800 domain-scenario queries; validation uses only the separate 160-query domain-scenario set.Benchmark settings and API workflows informed domain specifications rather than supplying benchmark utterances.Source: §A.2 · pp. 13–14

A.3 Deceptive Evaluation Set

The deceptive evaluation set tests whether capability labels remain determinate despite wording that suggests an incorrect label on the target axis.

A.3 Deceptive Evaluation Set

  • The deceptive set contains 220 queries: 77 target Source, 70 Transformation, and 73 World Effect, each with a misleading cue but a determinate gold label.Source: §A.3 · p. 14
  • GPT-5 generates candidates from target-axis, gold-label, and deception specifications, and Claude Sonnet 5 independently labels them before acceptance review.Review checks correctness, naturalness, ambiguity, identifiable cues, unsupported premises, and overlap with examples or templates.Source: §A.3 · p. 14
  • The 220 accepted queries are frozen before evaluation and are not filtered or selected based on probe or baseline performance.The set is not used for probe training.Source: §A.3 · p. 14

B Layer-wise Probe Accuracy

Table 11 reports per-layer validation accuracy for each capability-axis probe across four models and identifies jointly selected best layers.

B Layer-wise Probe Accuracy

  • Table 11 reports per-layer validation accuracy across four models, with bold layers selected for the best joint three-axis validation accuracy.When per-axis peaks tie, selection maximizes simultaneous validation accuracy across all three axes.Source: B Layer-wise Probe Accuracy · p. 14

C Prompts

Appendix C documents the taxonomy and prompts used to construct and label queries, review examples, evaluate deceptive queries, and elicit verbal classifications. It also presents per-layer validation results for four models.

C.1 Capability Axis Definitions

  • The shared capability-axis definitions guide benchmark labeling and review, query synthesis, domain-scenario construction, deceptive-query generation and review, and verbal classification.The benchmark-source held-out set has 200 queries; the separately constructed domain-scenario dataset uses 25 domains, with 20 for training, validation, and in-domain testing and five held out within that component.Source: §C.1 · p. 14, Table 9 · p. 15, Table 10 · p. 15
  • Joint validation accuracy was 0.9813 for Llama 3.1 8B Instruct, 0.9875 for Qwen 3 8B and Gemma 2 9B IT, and 0.9938 for Mistral 7B Instruct v0.3.The reported best layers were S = L14 (43.75%), T = L7 (21.88%), W = L16 (50.00%) for Llama; S = L36 (100.00%), T = L20 (55.56%), W = L26 (72.22%) for Qwen; S = L23 (54.76%), T = L10 (23.81%), W = L23 (54.76%) for Gemma; and S = L20 (62.50%), T = L8 (25.00%), W = L22 (68.75%) for Mistral.Source: §C.1 · pp. 16–17, Table 11

C.2 Gold-Label Review Prompt

  • Gemini 2.5 Flash reviews benchmark-derived labels without seeing proposed labels, and disagreements or quality failures are flagged before dataset finalization.The reviewer rejects ambiguous, mislabeled, vague, malformed, or multi-goal queries and accepts only a coherent request with an unambiguous label and confidence of at least 0.85.Source: §C.2 · p. 20

C.3 Synthetic Supplementation Prompts

  • GPT-4.1-mini supplements classes with insufficient benchmark coverage using prompts for four Mutate classes from BFCL v3 and (E,R,O) from FreshQA.Each system prompt combines the shared capability-axis definitions with class-specific generation constraints.Source: §C.3 · p. 20

C.4 Domain-Scenario Generation and Review Prompts

  • GPT-5-mini generates domain-scenario queries, Gemini 2.5 Flash reviews them at query and scenario-family levels, and GPT-5-mini revises queries flagged by either review.A prompt “block” is a scenario family containing one query for each of the eight TACIT classes.Source: §C.4 · p. 22

C.5 Deceptive Evaluation Prompts

  • The deceptive evaluation uses 220 accepted challenge queries; because controls also change the target-axis gold label, challenge-control performance differences cannot be attributed solely to misleading wording.The paper reports method comparisons on challenge queries in the main text and supplies generation and review prompts for reproducibility.Source: §C.5 · p. 31

Generation Prompt

  • The generation rubric labels Source A when the named target, required parameters, and stable information are explicit, and E when identifying the target requires fetching external or time-varying information.It labels Transformation R for a direct lookup returned as-is and T when non-trivial computation is required, including arithmetic, aggregation, ranking, comparison, conditional logic, or inference.Source: Generation Prompt · pp. 31–32
  • The rubric labels World-Effect O for information-only requests and M for explicit state-changing actions, following the query’s literal verb rather than inferred intent.Challenge-control pairs preserve the topic, entities, wording, and other requirements where possible while flipping only the assigned axis; each query must include the context needed to label it.Source: Generation Prompt · p. 32
  • Queries should be under 100 words and plausible, avoiding invented facts, contrived jargon, ambiguous units, and advice or subjective-choice requests.Fictional local record values are allowed only when explicitly supplied as scenario data.Source: Generation Prompt · p. 32
  • Prompts should not coach labels: actions must be explicit, and external inputs must be genuinely absent rather than inferable from intention or a supplied snapshot.For A snapshots, ask about the fixed record; for E, a necessary external input must be missing from the query.Source: Generation Prompt · p. 32
  • Taxonomy labels follow required operations, not keywords: stored fields indicate R, required derivation indicates T, and quoted action words do not request execution.For R, make clear the field is stored; for T, name the required operation in metadata and ensure the query requires it.Source: Generation Prompt · p. 32
  • Matched pairs should change exactly one axis while preserving other requirements, such as operands and calculation, and materializing a result can flip T to R.Source edits replace supplied snapshots with named external records; effect edits change execution or delivery while keeping the calculation.Source: Generation Prompt · p. 33
  • Each challenge-control output includes labels and specifications for available and external information, processing, and requested state change, plus cue and edit metadata.The prescribed JSON also records a misleading cue, a shortcut hypothesis, and why exactly one axis flips.Source: Generation Prompt · p. 33
  • Source demonstrations contrast using a supplied account snapshot with requiring a live dashboard value, and a battery snapshot with an unstated mutable value.The examples keep the request ordinary and do not require a clock word to signal external information.Source: Generation Prompt · p. 33
  • Transformation demonstrations distinguish retrieving a supplied form field from dividing its values, and require the derived operation rather than relying on a keyword.The examples preserve the same values while changing direct reading into arithmetic.Source: Generation Prompt · p. 34
  • World-effect demonstrations distinguish showing a path quoted in a note from deleting the file at that path, so quoted instructions remain data unless execution is requested.Source: Generation Prompt · p. 34
  • The generation example pairs challenge label ATM with an arithmetic-difference query and contrasts it with ATO, where reporting explicitly creates an app note rather than a conversational response.Source: Generation Prompt · p. 35
  • Each generated pair must be topic-specific, distinct from examples, and consistent with its assigned labels, target axis, and inventory-drawn domain, mechanism, and style.Curated seeds and demonstrations are fixed examples, not evidence of evaluated-model errors.Source: Generation Prompt · p. 35
  • Generation attempts start at zero, and later attempts may include error lists from up to the two most recent attempts.The first attempt has an empty previous_attempts array; subsequent entries use an errors field containing error messages.Source: Generation Prompt · p. 35

C.6 Verbalization Inference Prompt

  • Verbal classification uses four fixed labeled training examples shared across all four models, with the instruction, taxonomy, examples, and query combined in one user message.There is no separate system message because Gemma 2’s chat template does not support a system role.Source: §C.6 · p. 40
  • The classifier prompt requires labels on Source, Transformation, and World-Effect in the exact specified format, without explanations or extra text.The response format restricts labels to A or E, R or T, and O or M, respectively.Source: §C.6 · pp. 40–41
  • The verbalization demonstrations include retrieval of a recipient’s verification status labeled E/R/O and setting an alarm labeled A/R/M.Source: §C.6 · p. 40
Loading 2610.08018v1…