PaperlayerView exact revision on arXiv

Source-linked AI summary

CircuChain: Disentangling Competence and Compliance in LLM Circuit Analysis

Mayank Ravishankara

arXiv:2602.15037v1cs.SEcs.AI

TL;DR

Engineering circuit reasoning requires both physical correctness and compliance with explicit analytical conventions, but existing evaluations leave this distinction underexplored. CircuChain addresses the gap with paired Control/Trap tasks and multi-stage verification, finding that stronger physical reasoning can coincide with convention violations. The benchmark therefore exposes a reliability gap between competence and compliance in mathematically rigid settings.

  • Problem

    Existing evaluations do not adequately distinguish numerical circuit-solving competence from adherence to user-specified sign, direction, polarity, and methodological conventions.

  • Method

    CircuChain uses paired Control/Trap circuit tasks across five topologies and combines symbolic solving, SPICE simulation, and LLM-based error classification.

  • Results

    The evaluated models exhibit a Compliance–Competence Trade-off: stronger physical reasoning coincides with more convention violations on Trap cases, while smaller models adhere more consistently but make more physics errors.

  • Takeaways & Limitations

    Numerical correctness alone is insufficient for specification-critical circuit reasoning, motivating compliance-aware evaluation and training.

  • Takeaways & Limitations

    CircuChain covers five linear topologies and text-only reasoning, and its failure attribution relies on an LLM-based judge despite secondary audit agreement.

Abstract

from arXiv · show

As large language models (LLMs) advance toward expert-level performance in engineering domains, reliable reasoning under user-specified constraints becomes critical. In circuit analysis, for example, a numerically correct solution is insufficient if it violates established methodological conventions such as mesh directionality or polarity assignments, errors that can propagate in safety-critical systems. Yet it remains unclear whether frontier models truly apply first-principles reasoning or rely on entrenched training priors that conflict with explicit instructions. We introduce CircuChain, a diagnostic benchmark designed to disentangle instruction compliance from physical reasoning competence in electrical circuit analysis. CircuChain consists of counterbalanced Control/Trap problem pairs across five canonical circuit topologies, augmented with systematic variations in sign conventions, current orientations, and polarity definitions. A multi-stage verification pipeline, combining symbolic solvers, SPICE simulation, and an LLM-based error taxonomy, enables fine-grained attribution of failures to convention errors, physics errors, arithmetic mistakes, or hallucinations. Across 100 tasks per model, we observe a consistent Compliance-Competence Divergence. The strongest model evaluated exhibits near-perfect physical reasoning but a high rate of convention violations when Trap conditions deliberately invert natural sign patterns. Conversely, weaker models display lower physical fidelity yet superior adherence to explicit instructions. These results suggest that increased model capability does not guarantee improved constraint alignment and highlight the need for new evaluation frameworks that stress instruction-following under mathematically rigid domains. CircuChain provides one such framework and offers actionable insights for both engineering education and AI alignment research.

I. INTRODUCTION

CircuChain addresses the gap between numerical circuit-solving accuracy and adherence to explicit methodological conventions. Its paired benchmark and verification pipeline expose a Compliance–Competence Trade-off in which stronger physical reasoning can coexist with convention violations.

  • Engineering solutions remain invalid when they produce correct numbers through incorrect sign conventions, current directions, or other methodological violations.
  • Convention Blindness is the tendency to override explicit circuit-analysis instructions in favor of implicitly learned priors.
  • CircuChain uses 100 SPICE-verified tasks across five classical topologies to distinguish physical reasoning competence from instruction compliance.
  • Control and Trap instances invert current directions, polarity assignments, or node labels to test whether models follow prompts when they conflict with training priors.
  • Stronger models show higher convention-violation rates on Trap cases, while smaller models follow instructions more consistently but make more fundamental physics errors.
  • A multi-stage pipeline combines symbolic solving, SPICE simulation, and LLM-based error classification to separate physics errors from instruction-following errors.

II. RELATED WORK

Prior reasoning benchmarks emphasize computational competence and final-answer correctness, while engineering validity also depends on procedural fidelity. CircuChain studies the complementary failure mode in which capable models resist explicit instructions that conflict with learned solution patterns.

  • Existing mathematical and scientific benchmarks commonly evaluate final-answer accuracy without assessing adherence to domain-specific rules.
  • In engineering, solutions violating sign conventions, reference directions, or requested methods remain invalid despite numerical correctness.
  • CircuChain uses a physically grounded, controlled setting to examine this inverse alignment failure separately from sycophantic agreement.
  • Convention Blindness describes models privileging typical internal solution patterns over arbitrary sign conventions or reference directions specified by users.

C. Circuit and EDA Benchmarks

Existing circuit and EDA benchmarks assess broad topology, conceptual, multimodal, and design capabilities, but may conflate perception with reasoning. CircuChain instead isolates text-based symbolic reasoning and procedural compliance across fixed circuit-analysis tasks.

  • CIRCUIT evaluates topology interpretation and conceptual reasoning, while MMCircuitEval extends circuit evaluation to schematic images, SPICE generation, and design synthesis.
  • Multimodal circuit benchmarks can conflate diagram-perception errors with reasoning errors, motivating CircuChain’s text-only netlist descriptions.
  • CircuChain evaluates numerical correctness together with user-defined sign conventions, variable definitions, and requested analytical methods.
  • The benchmark contains 5 fixed topologies and 50 problem instances evaluated under Mesh/KVL and Nodal/KCL modes, producing 100 scored subtasks per model.
  • Its topology set includes a 3-Loop Supermesh, 2-Mesh T-Network, Wheatstone Bridge, Ladder Circuit, and Dependent Source Network.

B. Control vs. Trap Variants

Control and Trap variants preserve circuit topology while changing regimes and source orientations so that physically correct answers conflict with common textbook intuitions. These variants target stereotyped solution behavior through source, loading, and dependency stresses.

  • Trap instances preserve the underlying topology but create counter-intuitive solutions, such as negative mesh currents under a clockwise definition.
  • Source-dominance reversal changes source magnitudes or polarities so net current opposes common left-to-right expectations.
  • Asymmetric loading uses extreme resistance ratios such as R1 ≪R2 to create stiff coupled equations that penalize informal simplifications.
  • Dependency stress sets dependent-source gains to strongly couple controlling and controlled variables, increasing exposure to sign and omitted-term errors.

C. Prompting and Global Conventions

CircuChain uses fixed prompting and convention rules across circuit-analysis evaluations, while generating and verifying instances through paired NGSPICE and Python solutions. The evaluation covers 100 scored subtasks per model.

  • C. Prompting and Global Conventions: A fixed prompt template defines mesh direction, reference node, PSC, required analysis method, and named output variables.
  • D. Ground Truth Generation and Dual Verification: Parameterized NGSPICE netlists instantiate fixed topology templates with instance-specific component values and source orientations.
  • D. Ground Truth Generation and Dual Verification: Ground truth records node voltages and branch currents from DC operating-point analysis.
  • D. Ground Truth Generation and Dual Verification: NGSPICE and an independent Python mesh-and-nodal solver must agree within numerical tolerance for every recorded variable.No cases failed to converge.
  • E. Automated Evaluation Pipeline: The benchmark evaluates 5 models on 100 scored subtasks each.
  • C. Prompting and Global Conventions: Each model is tested on two analysis modes per instance under the fixed evaluation setup.

1) Stage 1: Solution Generation:

The evaluation parses structured model answers against verified circuit values, then uses a reference-guided judge to classify failures into convention and competence categories.

  • 1) Stage 1: Solution Generation:: Models are queried at deterministic settings and must return structured final_answer fields for requested variables.
  • 1) Stage 1: Solution Generation:: Predicted variables are extracted with rule-based parsing and a constrained structured fallback when formatting fails.
  • 1) Stage 1: Solution Generation:: A subtask passes only when all required variables satisfy the hybrid tolerance.The tolerance includes a small ϵabs for near-zero values.
  • 1) Stage 1: Solution Generation:: GPT-5 judges failed subtasks using the problem, model derivation, and verified values without solving the circuit itself.
  • 1) Stage 1: Solution Generation:: The rubric distinguishes sign-convention and method violations from physics setup, calculation, and hallucination errors.
  • 1) Stage 1: Solution Generation:: Compliance Errors comprise sign-convention and method violations, whereas Competence Errors comprise physics setup, calculation, and hallucination.

IV. RESULTS

CircuChain evaluates five models on 100 subtasks each and reports judge-labeled accuracy with uncertainty intervals. GPT-5 and Claude Opus 4.5 form the top accuracy tier, with overlapping confidence intervals.

  • IV. RESULTS: 5 models are evaluated on 100 scored subtasks each, split evenly between KVL and KCL.
  • IV. RESULTS: GPT-5 and Claude Opus 4.5 lead overall accuracy at 66% and 65%, respectively, followed by o4-mini at 57%.
  • IV. RESULTS: At N=100, GPT-5 reports 66% [56.3, 74.5] and Claude Opus 4.5 reports 65% [55.3, 73.6].
  • IV. RESULTS: The GPT-5 and Claude Opus 4.5 binomial 95% confidence intervals strongly overlap, indicating no statistically meaningful separation at this sample size.

B. Robustness Under Trap Regimes (Control vs. Trap)

Control and Trap conditions test robustness to counter-intuitive parameter regimes. Most models lose accuracy under Traps, while Claude Opus 4.5 shows the opposite pattern.

  • B. Robustness Under Trap Regimes (Control vs. Trap): The Control–Trap split is n=52 Control and n=48 Trap per model.
  • B. Robustness Under Trap Regimes (Control vs. Trap): GPT-4o shows the largest degradation under Traps, while o4-mini is relatively stable.
  • B. Robustness Under Trap Regimes (Control vs. Trap): Claude Opus 4.5 exhibits an inverted gap, with Trap accuracy exceeding Control accuracy.The reported relationship is Trap > Control.
  • B. Robustness Under Trap Regimes (Control vs. Trap): Most models degrade under Trap regimes.

C. Topology Sensitivity

Accuracy varies substantially by circuit topology, with ladder circuits easy for frontier models and the 2-mesh opposing-source T-network difficult. Failure attribution also reveals different compliance–competence profiles across models, with moderate agreement in the automated-label audit.

  • Topology-specific accuracy: GPT-5 reaches 100% on ladder circuits, while the 2-mesh opposing-source T-network yields only 30% accuracy; Claude Opus 4.5 scores 95% and 50%, respectively.Each topology contributes N=20 subtasks per model.
  • Failure attribution: GPT-5 failures are dominated by compliance errors at 34%, whereas Claude Opus 4.5 has 7% compliance errors and 28% competence errors.The percentages are reported as fractions of all subtasks under the judge taxonomy.
  • Judge reliability: A secondary audit on N = 50 failure cases achieves 72.0% fine-grained agreement with κ = 0.53 across five error categories.The audit uses Claude 3.5 Sonnet as an independent judge.
  • Judge reliability: Aggregating errors into Compliance versus Competence increases agreement to 80.0% with κ = 0.57.Most disagreements occur between neighboring subcategories such as Setup and Calculation errors.

V. DISCUSSION

CircuChain finds that models can separate in overall problem-solving accuracy from adherence to user-specified circuit conventions. The discussion connects this divergence to method sensitivity, conventional priors, and risks for educational and safety-critical workflows.

  • The Compliance–Competence Trade-off: GPT-5 achieves 66% overall accuracy with 34% compliance failures and no competence-attributed failures, while Claude Opus 4.5 reaches 65% accuracy with 7% compliance and 28% competence errors.The benchmark treats competence and compliance as orthogonal properties.
  • The Compliance–Competence Trade-off: Scaling raw problem-solving capability does not automatically improve adherence to arbitrary user constraints.The paper links this pattern to a possible decoupling between symbolic comprehension and execution, while describing standard-solution priors as a plausible interpretation.
  • Method sensitivity: GPT-5 performs better on nodal KCL subtasks than mesh KVL subtasks, scoring 72% versus 60%, while GPT-4o fails all KCL subtasks at 0%.The paper identifies algebraic sensitivity and representational familiarity as possible contributing factors.
  • Implications: Convention violations can mislead learners and, in sign-sensitive control workflows, a polarity inversion can change negative feedback into positive feedback and alter stability properties.The paper recommends explicit convention checks and verification layers for such deployments.

D. Limitations and Future Work

CircuChain demonstrates a compliance–competence divergence while remaining intentionally limited in topology, modality, and attribution methodology. The paper therefore calls for broader circuit coverage, additional auditing, and evaluation under varied convention contracts.

  • Scope boundaries: CircuChain covers five linear circuit topologies and evaluates text-only reasoning under a fixed convention contract.The task suite is diagnostic rather than exhaustive.
  • Future work: Future work should expand to richer topologies, AC/phasor analysis, diagram-grounded inputs, additional auditors, rule-based checks, and varied convention contracts.The current audit reports fine-grained κ ≈0.53 and dichotomy κ ≈0.57.
  • Main finding: The benchmark’s reported divergence includes GPT-5’s 0% competence errors and 34% compliance errors versus Claude Opus 4.5’s 7% compliance and 28% competence errors.These figures come from judge-labeled aggregation across 100 subtasks per model.
  • Implications: Numerical correctness is insufficient when user-declared mesh direction, polarity, or required method defines validity.The conclusion recommends compliance-aware evaluation and training beyond accuracy-only metrics.
Loading 2602.15037v1…