Uncovering Why LLMs Fail: From Internal Mechanisms to Adversarial Reasoning Trajectories

1 Introduction

• Prior work links failures to overlooked evidence, unsupported associations, incomplete reasoning, violated constraints, and commitment or verification problems.These concerns span ordinary reasoning and safety evaluations, where unsafe completion identifies a safeguard failure but not the internal process reorganized by an adversarial prompt.Source: §1 · pp. 1–2
• Endpoint labels leave unresolved how failures arise inside the generation pipeline, because the same failure category can result from different internal routes.Existing endpoint, calibration, rationale, and mechanistic analyses provide partial views but are rarely connected in a common process-level account.Source: §1 · pp. 1–2
• CauTrace maps generation traces to seven functional-state coordinates for support and constraint processing, answer commitment, and verification.It examines representation propagation, temporal-state coordination, and whole-trace trajectory topology.Source: §1 · p. 2
• Across Qwen and LLaMA-family models, failed traces show weaker support-state update agreement and lower relative verification activity, while the same output label can follow different trajectories.Paired harmful-benign adversarial flips also accompany larger internal-state changes.Source: §1 · p. 2

2.1 Reliability and Reasoning Analytics

• Endpoint benchmarks quantify factuality, hallucination, instruction following, and harmful behavior, but the same failed label can arise through different internal routes.Calibration and consistency methods estimate unreliability, while textual rationales can diverge from operative computation and do not identify which process stage produced a score.Source: §2.1 · pp. 2–3

2.2 Adversarial-Prompt and Safety Analytics

• Attack benchmarks characterize harmful outcomes and threat models, while safety-mechanism studies locate refusal directions, template-region signals, or attention vulnerabilities.These approaches are generally organized around a particular refusal mechanism, attack family, or localized component rather than a task-general generation process.Source: §2.2 · p. 3

2.3 Mechanistic and Representation Analytics

• White-box decoders and instrumentation reveal layer-wise information or intervention effects, while mechanistic studies localize behavior to components or circuits.Those units do not by themselves describe how functional roles coordinate over an entire generation trace; CauTrace jointly analyzes propagation, temporal coordination, and trajectory.Source: §2.3 · p. 3, Figure 1 · p. 4

3 Methodology

• The three analytics form a hierarchy: representation analysis tracks where deviations persist, temporal analysis locates miscoordination, and trajectory chains describe the route to the output.The levels are intended as mutually complementary measurements rather than independent feature inspections.Source: §3 · p. 4

3.1 Functional-State Coordinates

• Seven coordinates represent premise binding, knowledge retrieval, reasoning composition, operational computation, rule tracking, answer commitment, and self-verification.The first five cover support or constraint formation, AC selects a candidate answer, and SV evaluates whether it remains supported or revisable; states may overlap and need not follow a fixed order.Source: §3.1 · pp. 4–5

3.2 Propagation of Representation

• Representation analytics tests whether support-state changes persist through Attention and MLP updates, remain visible in value-projected representations, and align with the answer-commitment direction.Its composite deviation score uses calibrated thresholds, interquartile-range normalization, and equal default weights; it is not a single causal factor.Source: §3.2 · pp. 5–6

3.3 Temporal-State Coordination

• Temporal-state analysis reconstructs transitions among support and constraint states, commitment, and verification, measuring onset, ordering, switching, and relative verification activity.Its process-deviation score combines early commitment, support–commitment inversion, frequent switching, late verification, and low verification relative to commitment; shifts can depend on model and failure type.Source: §3.3 · pp. 6–7

3.4 Reasoning Trajectory

• Trajectory features summarize support, commitment ordering, verification, state switching, constraint preservation, and continuation length across a generation.The expected organization is support before commitment, coordinated transitions, and post-commitment verification that can confirm or revise the candidate; topology distinguishes routes within one endpoint category.Source: §3.4 · p. 7

4.1 Configuration

• CauTrace compares successful and failed traces, matched prompt pairs, and model families using fixed development-set calibration, yielding process profiles rather than an endpoint classifier.The prompt-channel attacker can alter user prompts and task context but cannot change model weights, decoding, or diagnostic access; training-time poisoning and deployment-side compromise are excluded.Source: §4.1 · p. 7, §4.2 · p. 8

4.4 Temporal-State Level: Coordination of Failure-Reorganizing

• Pooled verification-to-commitment activity falls from 4.172 to 1.065 (∆= −3.107, q < 0.001), although the verification–commitment timing gap is not significant for 1.5B and 7B.Failed traces have higher aggregate process deviation at every Qwen2.5 size; the component changes are not uniform, with early-commitment and late-verification contributions increasing.Source: §4.4 · pp. 8–9, Table 3 · p. 10
• Correct and failed traces differ in transition probabilities: failures lose KR→FRT (−0.11) and PB→RC (−0.08), while gaining KR→AC (+0.08), SV→AC (+0.06), and RC→KR (+0.06).These shifts indicate selective rerouting of support construction toward commitment or reset, and more returns from verification to the committed candidate.Source: §4.4 · p. 9, Figure 4 · p. 11

4.5 Trajectory-Topology Level: One Output Label Contains Multiple Routes

• Trajectory regions are distinguishable to varying degrees: Early-Commitment and Verification without Overwrite have diagonal proportions of 0.99 and 0.84, versus 0.09 and 0.25 for Fragmented Support and Rule Failure.The first two projection axes explain 28.2% of trajectory-feature variance, and the confusion matrix shows overlapping rather than rigidly separated routes.Source: §4.5 · p. 9, Figure 5 · p. 11
• The trajectory patterns distinguish failure routes: premature commitment, uneven or decaying support, weak rule preservation, late checking without overwrite, and unsupported continuation.The figures show state-score curves and representative transition chains; unsupported continuation has the longest and most recurrent examples, with T = 10,R = 7 and T = 11,R = 8.Source: §4.5 · p. 9, Figure 6 · p. 12, Figure 7 · p. 12

4.6 Adversarial Test: Behavioral Flips Coincide with Larger Internal Changes

• Paired prompt attacks produce endpoint flips, but target-state accuracy is near chance: reasoning-interference PFR is 1.8%–6.2%, while TAR@1 is 15.2%–16.6% versus 14.3% chance.The 500 reasoning-interference pairs and 300 safety pairs per model are evaluated against matched clean traces under an unchanged task objective.Source: §4.6 · p. 10, Table 4 · p. 11
• Adversarial flips accompany broad process and representation changes, but reasoning and safety attacks emphasize different coordinates: support/constraint for reasoning and commitment organization for safety.Reported contrasts include OC effects for reasoning-interference flips and AC peak-layer shifts for safety flips; Q/K/V/O carriers also differ by setting.Source: §4.6 · pp. 10, 13, Table 5 · p. 13

4.7 Generalization

• Across four additional model families, failures show lower support-update agreement and verification relative to commitment, while representation-level carriers vary by architecture.The reported failure-minus-correct changes are −0.012 to −0.052 for support-update agreement and −1.102 to −15.519 for verification-to-commitment activity; other representation indicators change sign across models.Source: §4.7 · p. 13

5 Conclusion

• Across Qwen and LLaMA-family models, failed traces show lower support-state update agreement and lower verification activity relative to commitment.Samples with the same output label also exhibit different trajectory topologies, while successful adversarial perturbations are associated with larger internal changes.Source: §5 · p. 13
• CauTrace organizes hidden-state traces into seven functional-state coordinates linking representation changes, temporal coordination, and trajectory topology.These diagnostics connect support, commitment, and verification across the reasoning trajectory.Source: §5 · p. 13
• The evidence supports a unified diagnostic account of failure dynamics and provides a foundation for finer-grained causal localization.Source: §5 · p. 13

Ethics Considerations

• CauTrace is an offline analytical framework, and adversarial prompts serve only as controlled stress tests of whether ordinary and induced failures share internal process changes.Experiments use locally hosted open-weight models and public benchmarks, without human participants, private user data, unauthorized access, or attacks on deployed services.Source: Ethics Considerations · p. 14
• Diagnostics may inform future monitoring and mitigation, but are not a per-generation safety guarantee or a substitute for deployment-specific evaluation.Potential beneficiaries include LLM users, affected communities, developers, researchers, and benchmark maintainers.Source: Ethics Considerations · p. 14
• The threat model is limited to prompt-side perturbations against locally hosted models, with adversaries denied access to weights, hidden states, diagnostic outputs, evaluators, and deployed systems.The authors also exclude turnkey jailbreak pipelines, operational bypass instructions, unredacted harmful-prompt collections, and harmful completions from release.Source: Ethics Considerations · p. 14
• Harmful-content exposure is reduced through automated safety judges, predefined labels, structured filtering, and limited manual inspection for debugging and validation.Harmful instructions are not executed, and unsafe prompts or generations are not reproduced or redistributed.Source: Ethics Considerations · p. 14
• CauTrace may help distinguish failures sharing an output label but requiring different monitoring or mitigation strategies.The authors judge expected reliability and safety benefits to outweigh residual dual-use risks under the controlled setting, bounded threat model, and restricted release policy.Source: Ethics Considerations · p. 14

Open Science

• The anonymous artifact is planned to include materials needed to reproduce the diagnostic analyses, subject to the stated safety restrictions.Source: Open Science · p. 14
• Raw or harmful content will not be repackaged when redistribution is inappropriate; the artifact will instead provide source references, dataset manifests, version records, and sanitized instance identifiers.The experiments use public benchmarks and publicly released Qwen and LLaMA-family checkpoints, subject to their providers’ licenses and usage conditions.Source: Open Science · p. 14
• The artifact is planned to include implementation, environment specifications, scripts, fixed splits, configurations, and sanitized metadata, while excluding unsafe completions and operational jailbreak materials.The review artifact is identified at the supplied anonymous repository address, with a persistent archival repository planned after review.Source: Open Science · p. 14

A Experimental Protocol and Details

• The appendix provides details on benchmark composition, output evaluation, probe training, paired-adversary construction, and trajectory-topology evidence.Source: A Experimental Protocol and Details · p. 17

A.1 Evaluation Configuration

• Each sample is evaluated under concise and analysis prompt tracks, with process-level results generally using the analysis track.That track requests two to four short reasoning steps followed by a constrained final-answer line.Source: §A.1 · pp. 17–18
• Reasoning outputs receive one of five labels: Correct, Partial Hallucination, Full Hallucination, Abstain, or Invalid.Safety pairs are evaluated analogously as safe refusal versus unsafe or partially unsafe compliance.Source: §A.1 · p. 18
• Matched clean/adversarial evaluations report Benign Success Rate, Adversarial Failure Rate, Pair Flip Rate, and Target Alignment Rate.Source: §A.1 · p. 18

A.2 Benchmark Composition and Evaluation

• The clean reliability benchmark spans nine public datasets covering factual, multi-hop, numerical, mathematical, logic, implicit, and commonsense reasoning tasks.Output evaluation follows task-specific judge_type requirements; open-ended short answers use normalized token-level F1, while structured tasks are parsed before scoring and can receive distinct endpoint labels.Source: §A.2 · p. 18

A.3 State-Probe Training and Scale Selection

• Probe coordinates use causally pooled hidden states and weak process labels, with window size 32 selected for temporal resolution near validation-AUC saturation.The probes indicate evidence for operational roles under weak supervision, not ground-truth cognitive states; the framework computes diagnostic measurements offline without changing generation.Source: §A.3 · p. 18

A.4 Prompt Tracks and Matched Clean/Adversarial Construction

• The analysis track requests short reasoning steps and a final-answer line, while matched clean/adversarial pairs keep task semantics and evaluation targets fixed.Only the prompt condition differs; resulting changes are interpreted as perturbation-linked, not as proof of a uniquely localized cause.Source: §A.4 · p. 18

A.5 Adversarial Perturbations

• Adversarial prompts target one of seven CauTrace states, and the target annotation is reserved for analyzing whether the largest changes align with intended or downstream states.The evaluation also tracks behavior-level success, failure, pair flips, and target alignment; its adversarial set includes reasoning perturbations and safety pairs based on AdvBench and HarmBench [28] [37] [48].Source: §A.5 · pp. 19–20, §A.4 · p. 19

• Rstr rank stability remains 1.000 under ablation and random weighting, but ablation decision stability ranges from 0.250 to 1.000 and separation reaches 0.000.Thus, Claim I is robust at the ordering level, not as a universal binary rule.Source: §B.1 · p. 20
• Rproc median rank stability falls to 0.790 under ablation and 0.696 under random weights, while median decision stability is 0.827 and 0.811, respectively.Its aggregate score should be interpreted alongside timing, switching, rule-preservation, and verification components.Source: §B.1 · p. 20

• Across model families, shared directions of ∆¯Asup and ∆Rvc support the reported representation-propagation and coordination relations.The passage also notes other representation-level carriers, but does not specify their results here.Source: §B.2 · p. 20

B.3 Endpoint Outcomes and Distinct Trajectory Topologies

• Analysis-track correctness rises by 13.2, 20.1, 18.6, and 11.7 percentage points across Qwen2.5-1.5B/3B/7B/14B, while Full Hallucination rises for 3B and 7B.Full Hallucination increases by 6.1 points for 3B and 7.6 points for 7B, but decreases for 1.5B and 14B.Source: §B.3 · p. 21
• Failure is the largest category for formal mathematics (0.718), abstention for multi-hop QA (0.678), and correctness for commonsense multiple-choice QA (0.721).These task-dependent distributions motivate distinguishing operational and rule failures, missing support, and constraint handling.Source: §B.3 · p. 21

B.4 Trajectory Signatures

• Trajectory types are rule-defined regions of feature space, not output labels or mutually exclusive ground-truth classes.The illustrated state-transition chains are schematic, not token-level reconstructions, and each type may include multiple chain variants.Source: §B.4 · p. 21

B.5 CauTrace-Guided Repair and State Steering

• Repair was evaluated on 200 disjoint test pairs after calibration fixed a high-risk trigger and intervention configuration before testing.Triggered traces received a unified support-reconstruction, rule-checking, provisional-commitment, and final-verification sequence; topology-conditioned and deliberately mismatched protocols served as comparisons.Source: §B.5 · pp. 21–22, Table 17 · p. 23
• Steering combined positive support, rule, and verification directions with negative answer commitment, applied during the first eight generated tokens at a CauTrace-identified risk layer.Aligned steering was compared with reversed, random, and wrong-state controls using the same intervention strength, layers, duration, and decoding configuration.Source: §B.5 · pp. 22–23
• 5.0 percentage points: unified CauTrace repair reduced adversarial failure from 76.0% to 71.0%, the largest observed reduction among five conditions.The difference from generic review was not statistically significant, and the result does not establish a topology-specific repair effect.Source: §B.5 · pp. 23–24, Table 17 · p. 23
• On 7B safety, aligned steering alone lowered Rproc by 0.008, increased rule coverage by 0.011, and increased verification coverage by 0.021.All four steering directions reduced adversarial failure from 100.0% to 66.7%; with only 12 test pairs, endpoint recovery did not distinguish aligned steering from controls.Source: Table 18 · p. 23, §B.5 · p. 24

B.5.1 Discussion: Defense Implications and Future Directions

• The authors propose using structural retention, process coordinates, and Trajectory Topology to guide interventions, while calling for broader evaluation and stronger direction-specific controls.Future work would develop closed-loop monitoring and adaptive repair or steering while preserving clean behavior.Source: §B.5.1 · p. 24