Large Language Models and their Awareness of Mechanics and Spatial Geometry

1. Introduction

• MecEng’s pipeline generates exact 3D solid geometries and finite-element meshes from text, then compares multibody models with ground truth using graph, numerical, geometry, and mesh representations.The pipeline targets three complexity levels, from rigid to flexible multibody systems.Source: §1 · p. 2, §1.2 · p. 6
• MecEng evaluates more than 30 local open-weight models alongside Claude-Opus-4.8 and GPT-5.2 through their respective APIs.The paper identifies the proprietary models by their name and version.Source: §1 · pp. 2–3

1.1. State of the Art

• Engineering LLM evaluations use expert or human comparison, LLM judges, or automated execution and matching; this work validates generated models against parameterized ground truth using graph isomorphism and numerical time series.The paper positions its approach in the fully automated evaluation category.Source: §1.1 · p. 5
• No existing work combines spatial-reasoning evaluation, exact geometry generation, and automated engineering validation in one benchmark for mechanical systems.Graphics-oriented geometry tools and mesh generators target visual aesthetics, whereas flexible multibody simulation requires exact, closed geometry.Source: §1.1 · pp. 3, 5

1.2. Objectives and Novelties

• The study aims to measure local open-weight LLM performance and its dependence on temperature, reasoning, model size, and release date, and presents a text-to-FEM/MBS pipeline including reduced-order models.The introduction reports continued performance increases with model size and release date.Source: §1.2 · p. 6

2. Mechanical Tasks and Generation Pipeline

• The task set more than doubled and became substantially harder after LLMs approached 100% success on the previous tasks.The core task and pipeline design builds on the earlier paper.Source: §2 · p. 6
• The section introduces a pipeline for translating textual descriptions into multibody simulations, including a dedicated pipeline for systems with flexible parts.Source: §2 · p. 6

2.1. Mechanical Models and Tasks

• The benchmark includes three difficulty levels, with level 3 divided into part generation (3a) and assembly (3b); level 3b evaluates correct connections of flexible machine parts at their interfaces.Source: §2.1 · pp. 6–7
• Level 1 tests mass points and single rigid bodies with springs or distance constraints; level 2 adds classical joints, contact, vehicles, and scaffolds.The tasks probe basic connectivity and mass/inertia assignment at level 1, then greater topological complexity and multi-degree-of-freedom interfaces at level 2.Source: §2.1 · pp. 6–7
• Level 3a tests flexible parts generated from 3D geometry, tetrahedral finite-element meshes, joint interfaces, and model order reduction; level 3b tests their assembly.Part tasks assess geometry synthesis, interface assignment, and computational-tool use; assembly tasks assess correct connections at part interfaces.Source: §2.1 · p. 7
• The benchmark contains 30 level 1, 20 level 2, and 34 level 3 tasks, widely parameterized to produce thousands of unique instances per model.Parameterization can vary structural features, including body counts, coordinate alignments, and interchangeable flexible-part geometries.Source: §2.1 · p. 9
• Each task provides a generic text template with typically 5 to 10 parameters, and requires a self-contained Python simulation model in Exudyn.Parameters can include mass, gravity, dimensions, and stiffness; the target is a functional, mathematically accurate multibody simulation model.Source: §2.1 · p. 9

2.2. Simulation Model Workflow

• For each task, the pipeline selects randomized parameters, builds component-specific context, generates LLM code alongside parameter-matched ground truth, then independently executes and compares both models.The LLM selects simulation components before receiving the generated context; missing essential components often lead to failure, and level 3 uses a modified workflow.Source: §2.2 · pp. 9–11, Figure 2 · p. 10

2.2.1. Configuration and Settings

• Three settings groups control the model and inference, pipeline scope and evaluation limits, and interchangeable prompts for component selection and code generation.The prompt templates allow prompt-design ablations without changing other parts of the pipeline.Source: §2.2.1 · p. 11

2.2.2. Task-Level Tracking

• Per-task records retain prompts, LLM responses, selected components, generated and reference source files, and evaluation outcomes, supporting manual inspection and aggregate success-rate calculations.The component record flags selections that are surplus to or missing from the ground-truth reference set.Source: §2.2.2 · pp. 11–12

2.2.3. Logging and Outputs

• Each run’s self-contained JSON captures configuration, hardware telemetry, aggregated metrics, and task records, while retained source files enable independent simulation re-execution.Logs add prompts, responses, timing, token counts, and automatic error summaries; source files embed parameterized task descriptions, creating executable description–code pairs.Source: §2.2.3 · p. 12

2.3. Flexible MBS

• Required flexible parts are generated and evaluated before the workflow proceeds to their flexible MBS assembly.Parts may be reused across different systems; an assembly proceeds only when all its required parts are correct.Source: §2.3 · p. 13

2.3.1. Part Generation

• Part generation combines shared randomized master parameters with unambiguous descriptions specifying geometry, material, and named interfaces.Shared parameters keep connected features consistent, such as a tower bore and rotor shaft with the same radius.Source: §2.3.1 · p. 14
• The LLM selects geometry and finite-element components, then generates Python code for meshing, material and interface assignment, Exudyn import, and Hurty-Craig-Bampton reduction.A pre-execution check overrides overly fine mesh sizes, rejects inaccurate coarse meshes, and flags code that does not hard-code mesh size.Source: §2.3.1 · pp. 14–15

2.3.2. Flexible MBS Assembly

• Assembly tasks are attempted only when all required parts passed evaluation, and the generated and ground-truth models load the same saved LLM-generated parts.A high-level Exudyn function places markers at named interfaces, simplifying flexible-body connections and loads.Source: §2.3.2 · pp. 15–16

LLM Inference

• Most models run locally through Ollama with 4-bit quantization to support privacy, reproducibility, and a compromise among generation quality, VRAM use, and inference speed.Models are pulled and executed on workstations.Source: §2.4 · p. 17
• Batching prompts with hardware-matched batch sizes yields reported speedups of 2–5× for the tested models and hardware.The backend monitors token counts, timing, and execution faults during inference.Source: §2.4 · p. 17
• Context and generation limits start at 4096 and 2048 tokens, respectively, and are scaled by 1.5 for reasoning models or 3.75 when thinking is activated.With thinking activated, the maximum context size is 15,360; exceeded limits are monitored and tests are rerun if performance measures are significantly affected.Source: §2.4 · p. 17
• For structural retrieval tasks, XML-like responses are validated and converted into Python lists, while malformed outputs are logged so downstream tasks can continue.The passage notes that state-of-the-art LLMs readily output this format.Source: §2.4 · p. 17

2.5. Code Execution

• LLM output is post-processed by programmatically extracting the pure Python source from surrounding natural-language text before execution.The pipeline also adds lines and function calls to identify error stages, redirect solver outputs, and inject finite-element mesh settings.Source: §2.5 · pp. 17–18
• The pipeline injects ground-truth solver settings and runs each script in an isolated subprocess with hard timeouts to contain runtime errors and segmentation faults during meshing.The checks target illegal file-system writes and unauthorized access to sensitive packages; isolation also protects against infinite loops and excessive body instantiation.Source: §2.5 · p. 18
• After successful runs, the pipeline extracts metrics, status dictionaries, and mechanical states and stores them in task-specific data layouts for statistical evaluation.Source: §2.5 · p. 18

3. Model Comparison and Verification

• The pipeline gates evaluation from component selection and Exudyn assembly through solver completion, topology, and solution checks, adding part metrics before MBS checks for flexible tasks.A failed check terminates downstream evaluation. Overall success requires all required validation steps to pass.Source: §3 · pp. 18, 20, Figure 6 · p. 19

3.1. Numerical Solution Comparison

• Numerical matching compares total body-coordinate trajectories across all time steps using an unnormalized Frobenius norm, with differences declared above 10^-5.The comparison first checks coordinate and time-step dimensions; it assumes matching body order, enforced by the system prompt. A relatively large tolerance accommodates numerical variation from reordered components and nonlinear error propagation.Source: §3.1 · pp. 20–21

3.2. Graph-Based Comparison

• Graph verification converts Exudyn items and references into index-independent graphs, comparing counts, topology, then node annotations to separate connectivity errors from parameter errors.Graph representations are not unique, so the evaluation generally expects the simplest representation. Python user-functions and numerically computed matrices, including matrices from modally reduced flexible bodies, are excluded from graph comparison.Source: §3.2 · pp. 22–23

3.3. Part Metrics

• For flexible parts, validation checks component selection, interface number and names, exact Young’s modulus, Poisson’s ratio and density, mass within 1% of ground truth, and the first three eigenfrequencies using relative vector-norm difference with 5% tolerance.Source: §3.3 · pp. 24–25

4.1.1. Overall Performance

• 82.1% overall success was achieved by the best open-weight rigid-body model, qwen3.6:27b.The top eight open-weight models averaged 73%; their topology match was 88.6%, versus 77.5% for numerical solution and 76.0% for graph content match, with fine-grained parameter selection remaining a common failure source.Source: §4.1.1 · p. 26, Table 1 · p. 26

4.1.2. LLM Performance versus Release Date

• Models released in 2026 consistently outperformed earlier generations, while open-weight models showed a six-to-twelve-month delay in reaching comparable performance.The reported trend covered overall success and component selection; selected proprietary models outperformed their counterparts.Source: §4.1.2 · pp. 27–28, Figure 7 · p. 28

4.1.3. Temperature Effects

• 54% at temperature 0 fell to 42% at temperature 1 for ministral-3, while the two Gemma 4 models were essentially temperature-insensitive.Without reasoning, most other models declined as temperature rose; reasoning models could improve at higher temperatures, and low temperatures were generally preferable.Source: §4.1.3 · pp. 28–29, Figure 8 · p. 29

4.1.4. Thinking Effects

• Reasoning boosted overall success by 5% to 9% for gemma4:31b, gpt-oss:20b, and gemma4:26b, but could reduce performance for qwen3.5/3.6.It reduced kinematics, reference-condition, and inertia errors by factors of 3 to 5, but increased inference time by more than 10 times in some cases and qwen3.5:27b response tokens 21 times; most overall tests therefore omitted reasoning.Source: §4.1.4 · p. 30, Table 2 · p. 30, Table 3 · p. 31

4.1.5. Prompt Design Effects

• Prompt variations produced moderate changes in component-selection and overall success rates, indicating that the pipelines were insensitive to prompt formulation.The study varied six component-selection prompts and four code-generation prompts while holding other settings fixed.Source: §4.1.5 · pp. 30–31, Figure 9 · p. 32, Figure 10 · p. 33

4.2.1. Overall Performance

• Around 65% overall success was achieved by Claude-Opus-4.8 on flexible MBS simulations, with GPT-5.2 more than 5 percentage points behind.Gemma4:31b was the best open-weight model, achieving almost 77% part correctness; proprietary models averaged around 82% on parts versus around 51% for open-weight models. Gemma4:26b achieved only 5% overall, below the open-weight average of approximately 25%.Source: §4.2.1 · pp. 32–33, Table 4 · p. 34

4.2.2. Evaluation and Inference Time

• Around 2 to 3 hours was the typical runtime for an open-weight model on flexible MBS tasks, including roughly half an hour for part evaluation and two minutes for assembly checks.The 6144-token context limit was not a constraint for most workloads; proprietary inference times were not directly comparable because of network latency, server load, and rate limits.Source: §4.2.2 · p. 33, Table 5 · p. 35

4.2.3. Difficulty of Tasks

• Geometry-related metrics—mass, mesh, and eigenfrequencies match—showed the largest success-rate drops after component selection, making correct geometry a primary bottleneck.The mounting bracket and bearing block with pedestal were difficult because their chamfers had to be assigned to specific edges; the flexible table assembly averaged 32.5% success despite the table part alone exceeding 50%. Level 3’s sequential dependency means one part-generation error can prevent the entire assembly task from succeeding.Source: §4.2.3 · pp. 34–35, Figure 11 · p. 36

• 9.9% of tasks had no solver success and 17.1% had mechanics-related errors across the top 8 open-weight LLMs, supporting MecEng success rates as an awareness indicator.The latter is the difference between solver and overall success; coding-related errors did not dominate.Source: §5 · p. 36
• Comparing models remains challenging, especially for user functions, which require testing across randomized parameters.Large test runs offer detailed, reproducible statistics but require full automation.Source: §5 · p. 36
• Automated runs must carefully handle infinite loops, poor solver convergence, costly graph-isomorphism checks, and mesher crashes; the authors monitored and resolved these issues or revised parametrizations to avoid timeouts.Source: §5 · p. 37
• 94,565 files and 2.9 GB of text made human evaluation impossible, while the best proprietary LLM uncovered roughly as many level 1–2 task-definition and ground-truth errors as it made.The authors manually monitored tasks the best proprietary LLM did not complete and iterated on task definitions and ground-truth implementation.Source: §5 · p. 37
• Future agentic frameworks could iteratively improve models, detect modeling errors from simulation outputs or configuration images, or compare independently generated models across simulation codes.The self-validation approach is cited as prior work.Source: §5 · p. 37

Author contributions

• J.G., S.W., T.M., P.M., and M.P. contributed to research methods, software, and writing, with additional roles varying by author.J.G. and M.P. contributed to conceptualization; J.G. and T.M. to investigation and visualization; P.M. and M.P. to validation. All contributed to writing, review, and editing.Source: Declarations · p. 37

Funding

• The research received no specific grant from public, commercial, or not-for-profit funding agencies.Source: Declarations · p. 37

Usage of AI

• LLMs were used for code generation and as AI agents, as well as for spell-checking, text improvement, and translation.Source: Declarations · p. 37

Appendix A. Model List

• Table A.1 lists all 84 mechanical models used in the study, grouped by difficulty level.Source: Appendix A · p. 43, Table A.1 · p. 43

Appendix B. Complete LLM Evaluation Results

• The table covers all evaluated LLMs across 50 combined rigid-body task models.Results are reported as percentages for component choice, solver success, topology match, numerical solution match, graph content match, and overall success; proprietary and open-weight models are separated.Source: Table B.1 · p. 44, Appendix B · p. 44