PaperlayerView exact revision on arXiv

Source-linked AI summary

Encoded but Not Actionable: Auditing the Decode-Generate-Steer Gap in Frozen LLMs for Geometric Constraints

Man Liang, Xinzhao Cheng, Faizan Wajid

arXiv:2608.17843v1cs.CLcs.AI

TL;DR

It remains unclear what geometric constraint information frozen LLMs encode and whether that information guides generation or responds to interventions. This paper audits these properties across matched CAD reasoning tasks and finds that decodability often fails to translate into behavioral expression, influence, or control.

  • Problem

    It remains unclear whether frozen LLMs’ encoded geometric constraint information informs generation, activation-level influence, and behavioral control.

  • Method

    The paper evaluates six frozen LLMs with matched CAD prediction tasks and a four-part audit of decodability, generation, activation influence, and steerability.

  • Results

    Across six models, pretraining improves pairwise constraint decoding more than sketch-level DOF classification, while generation and interventions do not reliably express or control decoded information.

  • Takeaways & Limitations

    Decodability, behavioral expression, activation-level influence, and control are empirically distinct in the tested CAD setting.

  • Takeaways & Limitations

    Behavioral and intervention results cover one forced-choice formulation, two backbones, and specific prompts, corruption schemes, intervention directions, and token positions.

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated strong performance on structured reasoning tasks, but what they encode and whether it informs model behavior remain unclear. We investigate this question through geometric reasoning, using parametric CAD constraints as a controlled testbed for separating local pairwise relations from sketch-level constraint status. By probing the hidden states of six frozen decoder-only LLMs, we examine four properties: linear decodability, forced-choice generation, activation-level influence, and behavioral steerability. Pretraining substantially improves the decoding of local geometric relations, and this advantage persists after accounting for positional cues with shuffled-order controls. In contrast, sketch-level DOF status is already highly decodable from randomly initialized representations and improves only modestly with pretraining, indicating that much of its probe performance is available without learned weights. Further analyses show that decodable information is not always actionable. Generation often fails to express this information, and on the two intervention-tested backbones, activation-restoration effects at the patched entity position vanish while decodability persists across depth. Mean-difference steering also does not reliably control outputs. These results show that decodability, generation, activation-level influence, and steerability can diverge in the tested setting. The audit provides a controlled way to distinguish failures to encode geometric structure from failures to express or control encoded information.

1 Introduction

The paper audits whether frozen LLMs encode geometric constraints and whether that information is expressed, used, or controllable in behavior. Using CAD constraint reasoning, it identifies a systematic gap between encoded and actionable information.

  • Motivation: Structured outputs for scientific and industrial use must satisfy interacting physical, geometric, safety, and global-consistency constraints.
  • Problem: Output-level CAD evaluation cannot determine whether successful generation and repair rely on internal constraint representations.
  • Method: The audit tests linear decodability, forced-choice generation, activation-level influence, and steering across six frozen decoder-only LLMs, with a cross-dataset P1 check.Evaluations use SketchGraphs and Fusion 360 Gallery.
  • Findings: Local pairwise relations contain learned information that remains linearly decodable after controls for random initialization, input structure, and entity order.
  • Findings: Global constraint status benefits little from pretraining, while activation-patching sensitivity vanishes even as decodability persists.Restoration sensitivity is concentrated in early layers and disappears at the patched entity position.
  • Contribution: Mean-difference steering does not reliably control predictions, motivating multi-level audits that distinguish encoding failures from failures to express, use, or control information.

2 Related Work

Related work frames geometric reasoning as a local-to-global problem and distinguishes representations, probing, and interventions as complementary analysis tools. It also motivates controls for separating learned information from probe, architectural, and input-driven effects.

  • Geometric reasoning and CAD generation: Parametric CAD separates pairwise geometric constraints defining local relations from remaining degrees of freedom characterizing sketch-level constraint status.This local-to-global structure supports controlled analysis of geometric reasoning in engineering workflows.
  • Geometric reasoning and CAD generation: Prior CAD representations range from explicit primitive-and-constraint graphs to construction histories, while generative models target constrained sketches or CAD construction sequences.SketchGraphs models primitives and explicit pairwise constraints; Fusion 360 Gallery, DeepCAD, and CADParser represent construction histories.
  • Representation analysis and intervention: Linear probing tests which information is linearly decodable from frozen hidden states, including world states and spatial or temporal information.Prior applications extend beyond linguistic attributes to game-playing models and LLMs.
  • Representation analysis and intervention: High linear-probe accuracy does not establish that information was learned through pretraining because it can reflect probe capacity, architectural bias, or input information.Prior controls include shuffled-label tasks, selectivity, random encoders, and simpler input representations; this work adds shuffled-label, random-initialization, and pure-input controls.
  • Representation analysis and intervention: Activation patching and related interventions test whether intermediate representations affect outputs and help identify behaviorally relevant circuits.The cited intervention methods include activation patching, activation addition, and representation engineering.

3 Method

The method audits frozen decoder-only LLMs on geometry-only CAD serializations through matched decoding and behavioral tasks, evaluating linear decodability, forced-choice generation, activation restoration, and steering. It combines controlled labels, serialization, representation extraction, probes, and order and initialization controls to separate accessible information from behavioral influence.

  • Evaluation pipeline: The audit uses geometry-only CAD sketches, three matched prediction tasks, and two activation-level interventions.The four-part pipeline evaluates linear decodability, forced-choice generation, activation restoration sensitivity, and steering-based control.
  • Prediction tasks: P1 classifies eight pairwise relation labels from concatenated entity representations, while P2 decodes sketch-level constraint status from mean-pooled entity representations.P1 covers seven geometric relations plus NOCONSTRAINT; HORIZONTAL and VERTICAL are excluded because they are primarily unary.
  • Behavioral task: P3 matches P1 with zero-shot forced-choice completion over the same eight classes, held-out pairs, label space, and macro-F1 metric.Appendix G additionally reports a four-shot control across five exemplar sets.
  • Serialization and representations: Serialized inputs contain entity types and numeric geometry parameters but omit EdgeOp annotations, with entity character spans mapped to token spans for pooling.Representations are extracted from frozen decoder-only LLMs by mean-pooling token hidden states within entity spans at eight evenly spaced relative-depth locations.
  • Probes and controls: Layer-wise probes use ℓ2-regularized logistic regression on balanced data with class-stratified 75/25 splits, supplemented by initialization, shuffled-label, entity-count, and shuffled-order controls.The main P1 split is at the entity-pair level; the shuffled-order control preserves entity identities, geometry, and labels while permuting serialization order.
  • Activation-level influence and steerability: Activation patching on Qwen2.5-3B and Llama-3.1-8B corrupts entity i embeddings with Gaussian noise and restores clean hidden states at tested layers to measure restoration rate.Restoration rate is defined as the fraction of corruption-informative examples for which patching recovers the clean prediction.

4 Experimental Setup

The experiments compare six frozen decoder-only LLMs with same-architecture random baselines under a fixed probing protocol. Evaluation uses macro-F1 and selectivity, plus a common-layer dissociation index and matched generation comparisons.

  • Models: Six frozen decoder-only LLMs spanning Qwen2.5, Mistral-7B, and Llama-3.1-8B are compared with same-architecture randomly initialized models.All models use identical extraction and probing protocols, balanced subsets, and a fixed sampling seed.
  • Metrics: Probe performance is measured with macro-F1 against uniform-class references of 0.125 for P1 and 0.333 for P2, alongside selectivity.Selectivity is defined as F1_task − F1_shuffled to control for probe memorization.
  • Dissociation index: The dissociation index evaluates whether pretraining improves local P1 decodability more than global P2 decodability at the pretrained model’s peak P1 layer.Pretrained and random-initialized P1 and P2 scores are all read at this common layer; random baselines partially control for architectural and dimensional differences.
  • Generation comparison: P3 uses the same held-out entity pairs, eight-class labels, and macro-F1 as P1, enabling a matched comparison between linear decodability and forced-choice generation.Unless otherwise noted, results use single-split F1 estimates with seed 0.
  • Uncertainty: Reported 95% confidence intervals use 1,000 bootstrap resamples, while DI uncertainty combines P1 and P2 component uncertainty in quadrature.DI uncertainty is not estimated with a direct paired bootstrap.

5 Results

Across six frozen models, local geometric relations are strongly decodable and benefit from pretraining beyond positional shortcuts, whereas sketch-level status is largely decodable without learned weights. However, decodable information is poorly expressed in generation and does not reliably support activation-based control.

  • Linear decodability: P1 reaches peak macro-F1 of 0.714–0.734 versus 0.125 chance, while pretraining improves over random initialization by 0.127–0.185 even after entity-order shuffling.Shuffling lowers P1 macro-F1 by 0.10–0.13, but pretrained models retain a 0.026–0.075 advantage over random initialization.
  • Linear decodability: P2’s pretraining gain is only 0.037–0.048 at the P1-selected layer, indicating that most sketch-level DOF-status decodability is present without learned weights.At P2’s own peak, the gain remains 0.037–0.047; entity count alone reaches macro-F1 0.419.
  • Robustness across models: The P1–P2 dissociation persists across model scales and families: DI remains positive at every scale, and all six models have positive DI with 95% confidence intervals excluding zero.Across scales, DI ranges from 0.106 to 0.167; selected P1 peaks span layers 10–21, while P1 selectivity exceeds P2 selectivity across families.
  • Generation: P3 forced-choice generation trails matched P1 decoding by 0.460–0.700 macro-F1 across all six models, showing that decodable information is not reliably expressed in predictions.Mistral-7B collapses to Coincident on 99.8% of examples, whereas Qwen2.5-7B achieves the highest P3 macro-F1 of 0.259.
  • Activation influence: Activation restoration is early and transient: it peaks at layer 4 and falls to zero by layer 16 on Qwen2.5-3B and by layer 12 on Llama-3.1-8B, before decodability peaks.Peak restoration rates are 0.781 [0.722, 0.846] and 0.876 [0.821, 0.924], respectively, with no later resurgence.
  • Steerability: Mean-difference steering produces no target-class flips at any tested strength for either P1 or P2 on either intervention-tested model.On Qwen2.5-3B, steering changes about four times as many labels as matched random directions at α = 8, but none reach the intended target class.

6 Discussion

The discussion shows that decodability does not guarantee generation, activation-level influence, or steerability, and that interpretability methods support narrower conclusions than behavioral control. It also identifies practical safeguards, dataset and labeling limitations, and open questions about generalization beyond SketchGraphs.

  • Interpretation: Decodable geometric information is not necessarily expressed, retained at the probed position, or controllable through mean-difference steering.Decodability persists across depth, whereas restoration effects at the tested entity position are early and transient.
  • Interpretation: Linear probing establishes information recoverability, while patching and steering support narrower claims about intervention effects rather than specific feature mediation or universal control.Patching affects outputs at a tested position without identifying the mediating decodable feature; steering failure rules out only the tested direction.
  • Practical implications: Reliable CAD outputs should not rely on LLM representations alone because P2 is highly decodable without pretraining and strong P1 decodability does not ensure accurate constraint predictions.Suggested safeguards include explicit validity supervision, structured state tracking, and solver-based verification.
  • Limitations: The conclusions are limited to geometry-only SketchGraphs inputs, with P1 excluding primarily unary Horizontal and Vertical constraints and P2 using heuristic Gr¨ubler-style DOF labels rather than solver verification.The P2 evaluation covers coarse DOF status, not global consistency, solvability, or redundancy, and redundant constraints may receive incorrect labels.
  • Limitations: Experimental uncertainty and behavioral coverage are limited by fixed splits or seeds, one random-init checkpoint per architecture, a single P3 formulation, Qwen2.5-3B-focused four-shot control, and interventions on two backbones.Patching uses corruption-informative pairs (n=169 and 145), while steering tests mean-difference directions up to α = 8.
  • Future work: Generalization beyond SketchGraphs remains open, motivating evaluation on other geometric corpora and CAD-native models with solver-derived validity signals and broader global-reasoning measures.The Fusion 360 Gallery experiment provides initial cross-dataset evidence for P1 but lacks matched three-class labels for P2.

7 Conclusion … B Representation and Split Controls

The audit separates whether frozen LLMs encode geometric information from whether they express, use, or respond to it. Pretraining strongly benefits pairwise constraint decoding, while sketch-level DOF information is already highly decodable from random representations and remains weakly actionable.

  • 7 Conclusion: The four-part audit separately evaluates linear decodability, forced-choice generation, activation-level influence, and behavioral steerability under matched conditions.This framework tests where representational evidence does or does not translate into model behavior.
  • 7 Conclusion: Pretraining contributes substantially more to pairwise constraint decoding than to sketch-level DOF classification across six models.Randomly initialized models already achieve high probe performance on the DOF task.
  • 7 Conclusion: P3 generation remains below supervised P1 decoding, and four-shot control on Qwen2.5-3B narrows but does not close the gap.The passage reports the comparison qualitatively without providing numerical values.
  • 7 Conclusion: Activation restoration affects predictions mainly at early layers, vanishes at the patched entity position, and leaves decodability persistent across depth.This result was observed on the two backbones tested with interventions.
  • 7 Conclusion: Mean-difference steering at the strongest restoration layer produces no targeted class flips.The intervention therefore does not reliably control outputs in the tested setting.
  • 7 Conclusion: Decodability, behavioral expression, activation-level influence, and control are empirically distinct properties.A high probe score shows that information can be linearly recovered, but provides limited evidence that models will express, use, or respond to interventions on it.
  • A.1 Geometry-Only Serialization and Matched Tasks: EdgeOp relations supervise or evaluate the task but never appear in the frozen LLM input.Geometric entities are serialized as the model input in the matched protocol.
  • A.2 P1/P2 Constraint-Type Gallery: The task includes seven pairwise EdgeOp relations plus sampled NOCONSTRAINT, while P2 uses under-, well-, and over-constrained labels.For P2 label checks, entity count alone reaches 0.419 macro-F1 versus 0.333 uniform, 0.679–0.691 random-init, and 0.719–0.732 pretrained probes; constraint satisfaction reaches 100% for well-constrained and 97.5% for over-constrained sketches.

B.1 Random-Init Controls … C.1 P3 Per-Class Accuracy

Randomly initialized models reproduce P2 performance far more closely than P1, while chance-normalized dissociation remains positive across all six pretrained models. P3 generation exhibits distinct class-specific failure modes, including near-collapse to COINCIDENT in Mistral-7B and uneven accuracy in Qwen2.5-7B.

  • B.1 Random-Init Controls: Random-init P1 peaks range from 0.549 to 0.598, below the pretrained range of 0.714–0.734.Table 3 reports independently selected task-specific peaks from one random initialization per architecture; these controls are descriptive and do not determine DI.
  • B.1 Random-Init Controls: Random-init P2 peaks reach 0.681–0.695, above the pure-input baseline of 0.380 and close to the pretrained range of 0.719–0.732.Across all six architectures, random-init representations reproduce P2 performance much more closely than P1 performance.
  • B.2 Chance-Normalized Dissociation Index: Chance-normalized DI remains positive for all six models and closely tracks raw DI.The normalization rescales each gain by task-specific headroom above its uniform-class reference level.
  • B.2 Chance-Normalized Dissociation Index: The P1–P2 dissociation cannot be explained by the tasks’ different uniform-class reference levels.Reference terms cancel within each pretrained–random-init contrast, leaving task-specific headroom as the rescaling factor.
  • C P3 Diagnostics: P3 diagnostics report both predicted-class frequency and accuracy conditional on the true class.Table 5 presents these quantities together, with fractions of all predictions followed by accuracy among examples with the corresponding true label.
  • C.1 P3 Per-Class Accuracy: Mistral-7B collapses almost entirely to COINCIDENT and succeeds mainly when COINCIDENT is correct.The model’s generation failure mode is concentration on a single predicted class.
  • C.1 P3 Per-Class Accuracy: Qwen2.5-7B distributes predictions across more classes, with PARALLEL predicted most often but Tangent predicted most accurately.Its class-specific accuracy is uneven rather than concentrated in the most frequently predicted class.

C.2 P3 Content-Free Prior Control … E.1 P1 Activation-Patching Layer Grid

Content-free class priors align with dominant real-task predictions but cannot quantify their contribution, while sketch-level splitting leaves P1 performance unchanged. Activation restoration declines or vanishes before peak decodability in both tested models.

  • C.2 P3 Content-Free Prior Control: Mistral-7B’s highest content-free prior is COINCIDENT, whereas Qwen2.5-7B favors PARALLEL, followed closely by TANGENT.These preferences align with dominant real-task predictions, suggesting class priors contribute to observed patterns.
  • C.2 P3 Content-Free Prior Control: Content-free probabilities do not determine how much prior bias explains real-task argmax frequencies.The control measures probability mass, whereas the real-task analysis reports argmax frequencies.
  • D P1 Sketch-Level Split Check: Sketch-level splitting yields macro-F1 0.700 ± 0.002 versus 0.699 ± 0.009 for the original pair-level split.The averages come from five split seeds and indicate sketch overlap does not materially affect P1 performance in this setting.
  • E.1 P1 Activation-Patching Layer Grid: The activation-patching grid evaluates four-layer intervals plus each model’s P1 decodability peak, using 169 Qwen2.5-3B and 145 Llama-3.1-8B corruption-informative pairs.Restoration rates use 95% confidence intervals from 1,000 bootstrap resamples; layer 0 is a sanity check and excluded from strongest nontrivial-layer selection.
  • E.1 P1 Activation-Patching Layer Grid: Qwen2.5-3B restoration falls from 0.781 [0.722, 0.846] at layer 4 to 0.604 [0.533, 0.675] at layer 12 and reaches zero by layer 16.These layers precede the P1 decodability peak at layer 21.
  • E.1 P1 Activation-Patching Layer Grid: Llama-3.1-8B restoration decreases from 0.876 [0.821, 0.924] at layer 4 to 0.759 [0.690, 0.821] at layer 8, then remains zero from layer 12 onward.Restoration is zero at the model’s P1 decodability peak at layer 14, so patched-position effects disappear before peak decodability.

E.2 Steering at the Restoration Peak

At layer 4, mean-difference steering failed to produce target-class flips for either model or task across tested intervention strengths. Although Qwen2.5-3B showed more prediction changes than random directions at α ≥2, the changes did not reliably reach the intended class, while Llama-3.1-8B remained at the random baseline.

  • Evaluation setup: Steering was evaluated at layer 4 using mean-difference and matched random directions across α ∈{0.5, 1, 2, 4, 8}.Experiments used 200 examples, 10 random-direction seeds, and two independent runs.
  • Targeted steering: Mean-difference directions produced no target-class flips for either task or model at any tested α.Random-direction target-flip rates also remained at or below 0.1%.
  • Model-specific effects: For Qwen2.5-3B, mean-difference directions caused more label changes than random directions when α ≥2, but none reached the intended class.Table 7’s P1 label-change rate counts any prediction change, not only changes to the intended class.
  • Model-specific effects: For Llama-3.1-8B, mean-difference steering label-change rates remained at or below the random baseline.Together with the absence of intended-class flips, the observed changes did not support reliable targeted steering.

F Cross-Dataset P1 Check · G P3 Few-Shot Prompting Control

A cross-dataset P1 check on Fusion 360 Gallery reproduces an intermediate-to-late probing peak for Qwen2.5-3B. Four-shot prompting improves P3 forced-choice performance over zero-shot, but remains far below P1 linear-probe performance and varies with exemplar selection.

  • F Cross-Dataset P1 Check: F Cross-Dataset P1 Check: Fusion 360 Gallery reconstruction data were evaluated with Qwen2.5-3B on 13,600 balanced entity pairs, covering P1 only.The dataset lacks matched three-class DOF-status labels for P2.
  • F Cross-Dataset P1 Check: F Cross-Dataset P1 Check: 0.643 macro-F1 peaks at layer 26, corresponding to 72.2% relative depth.The peak occurs in the Fusion 360 Gallery P1 check.
  • F Cross-Dataset P1 Check: F Cross-Dataset P1 Check: Macro-F1 changes little between layers 21 and 31, forming a broad intermediate-to-late plateau consistent with SketchGraphs.This describes the layerwise pattern observed in the cross-dataset check.
  • G P3 Few-Shot Prompting Control: G P3 Few-Shot Prompting Control: Four-shot evaluation used Qwen2.5-3B, five independently sampled exemplar sets, and one labeled example from each of four sampled classes.Each prompt was followed by the same eight-way forced-choice task used in zero-shot evaluation.
  • G P3 Few-Shot Prompting Control: G P3 Few-Shot Prompting Control: Evaluation pairs and scoring procedures remained unchanged across the four-shot prompting comparison.This isolates prompt-format sensitivity while retaining the zero-shot evaluation setup.
  • G P3 Few-Shot Prompting Control: 0.138 ± 0.013 mean macro-F1 was achieved with four-shot prompting, versus 0.081 zero-shot, across five exemplar sets.Individual four-shot scores ranged from 0.117 to 0.157, and every tested set improved over zero-shot prompting.
  • G P3 Few-Shot Prompting Control: 0.218 ± 0.011 mean accuracy was obtained with four-shot prompting, while performance varied with exemplar selection.The reported individual macro-F1 scores ranged from 0.117 to 0.157.
  • G P3 Few-Shot Prompting Control: 0.557 was the remaining gap between the best four-shot result and the 0.714 P1 linear-probe score, compared with a 0.576 gap at the four-shot mean.Even the best four-shot result remained below the P1 linear-probe score.
Loading 2608.17843v1…