PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models and their Awareness of Mechanics and Spatial Geometry

Johannes Gerstmayr, Sebastian Weyrer, Tobias Möltner, Peter Manzl, Michael Pieber

arXiv:2608.14615v1cs.AI

TL;DR

Mechanical engineering awareness in LLMs lacks systematic evaluation combining spatial reasoning, exact geometry, and engineering validation. MecEng benchmarks automated multibody-model creation across 84 tasks, finding substantial potential alongside persistent, hard-to-detect physical and kinematic errors.

  • Problem

    LLMs lack systematic evaluation of mechanical engineering awareness combining spatial reasoning, exact geometry, and automated engineering validation.

  • Method

    MecEng evaluates LLM-generated multibody simulation models across three difficulty levels using structured geometry generation and automated comparison with expert ground truth.

  • Results

    17.1% of tasks averaged over the top 8 open-weight LLMs produced executable models with hard-to-detect physical or kinematic errors, versus 9.9% with no solver success.

  • Takeaways & Limitations

    LLM mechanical engineering awareness is rapidly improving but remains error-prone, especially for complex mechanism and flexible-body simulation models.

  • Takeaways & Limitations

    Human evaluation was impossible for the enormous test output, and the best proprietary LLM uncovered roughly as many task-definition and ground-truth errors as it made on some tasks.

Abstract

from arXiv · show

Large Language Models (LLMs) perform well on established code-generation and mathematical-reasoning benchmarks, but their capabilities in mechanics and spatial geometry, here denoted as mechanical engineering awareness, has not been quantified systematically. We present MecEng, a fully automated benchmark that evaluates LLMs on the creation of multibody simulation models from parameterized textual descriptions. The benchmark comprises 84 generic tasks on three difficulty levels, ranging from rigid-body systems with joints and contact to flexible multibody systems that require exact 3D geometry generation, tetrahedral finite-element meshing, and Hurty-Craig-Bampton model order reduction of machine parts. A dedicated pipeline with LLMs generates simulation-ready geometry from text using Netgen, and builds multibody system models for the code Exudyn, which are then verified against expert ground truth on several levels: system-graph isomorphism including graph node annotations, numerical solutions, and part-specific measures such as mass, geometry, and eigenfrequencies. In total, 32 open-weight and two proprietary LLMs are evaluated. On rigid-body tasks, the best open-weight model obtains an overall success rate of 86.0%, compared to 91.4% for the strongest proprietary model, while flexible multibody tasks remain considerably harder. Additional studies quantify the influence of sampling temperature, reasoning, prompt design, model size, and LLM-release date. The results indicate rapidly improving, but still error-prone, mechanical engineering awareness of current LLMs.

1. Introduction

The introduction frames mechanical engineering awareness as an unresolved LLM capability involving spatial relationships, exact geometry, physical parameters, and multibody constraints. It presents MecEng as a fully automated benchmark that unifies text-to-model generation with engineering-grade validation across numerical, graph, geometric, and finite-element representations.

  • Motivation: Mechanical engineering awareness requires understanding spatial relationships, exact geometries, physical parameters, and constraints within multibody systems.The introduction contrasts this requirement with LLM strengths in language, code generation, and syntactic reasoning.
  • Evaluation domain: Multibody systems provide a broad testbed spanning rigid or flexible interconnected bodies and applications from simple mechanisms to robots, cranes, vehicles, and civil structures.This breadth motivates their use for assessing mechanical engineering awareness.
  • Research gap: Existing work does not combine spatial reasoning evaluation, exact geometry generation, and automated engineering validation in one benchmark for mechanical systems.Current geometry-generation methods prioritize visual aesthetics rather than deterministic engineering constraints.
  • Benchmark: MecEng is a fully automated engineering-grade benchmark for LLM-generated mechanical and multibody simulation models.It spans more than 30 local open-weight LLMs and proprietary models evaluated through application programming interfaces.
  • Evaluation framework: The benchmark compares generated models with parameterized ground truth using numerical results, system graphs, geometries, finite-element meshes, and reduced-order models.Its generation pipeline proceeds from text to finite-element models and multibody systems, including industry-standard reduced-order models.

2. Mechanical Tasks and Generation Pipeline · 2.1. Mechanical Models and Tasks

MecEng substantially expands earlier mechanical-model tasks into three difficulty levels and translates parameterized textual descriptions into Exudyn multibody simulations. The benchmark spans basic rigid-body models, contact-rich assemblies, and flexible parts requiring geometry generation, finite-element meshing, and model-order reduction.

  • 2. Mechanical Tasks and Generation Pipeline: The expanded MecEng task set increases both task count and difficulty because prior state-of-the-art LLMs approached 100% success on the earlier tasks.The task set was more than doubled relative to the previous paper.
  • 2. Mechanical Tasks and Generation Pipeline: The pipeline converts textual descriptions into multibody simulations and includes a dedicated process for systems containing flexible parts.The resulting simulations are generated for Exudyn through a text-to-model workflow.
  • 2.1. Mechanical Models and Tasks: Three difficulty levels are defined, with level 3 divided into flexible-part generation (3a) and flexible multibody assembly (3b).Level 3a generates parts, while level 3b assembles those parts through their interfaces.
  • 2.1. Mechanical Models and Tasks: Level 1 tests basic multibody modeling through mass points and rigid bodies connected by springs, distance constraints, forces, and torques.Examples include single-mass oscillators, freely flying rigid bodies, and slider-crank mechanisms, assessing spatial connectivity and mass/inertia assignments.
  • 2.1. Mechanical Models and Tasks: Level 2 introduces rigid-body systems with prismatic, revolute, and rolling joints, sphere-sphere and sphere-triangle contact, vehicles, and scaffolds.These tasks test increased topological complexity and multi-degree-of-freedom interfaces.
  • 2.1. Mechanical Models and Tasks: Level 3a generates flexible parts from 3D geometries through tetrahedral finite-element meshing, joint interfaces, and model-order reduction.Examples include flexible rotors, bearing blocks, and rods, testing geometry synthesis and computational modeling of flexible machine parts.
  • 2.1. Mechanical Models and Tasks: 30 level 1 tasks, 20 level 2 tasks and 34 level 3 tasks are widely parameterized, enabling thousands of unique tasks per mechanical model through randomized parameters.Parameterization can also vary structure, including body counts, coordinate alignments, and interchangeable parts.
  • 2.1. Mechanical Models and Tasks: Every task requires a self-contained Python program that builds a functional, mathematically accurate Exudyn multibody simulation from a generic text template containing typically 5 to 10 inserted parameters.Parameters include quantities such as mass, gravity, dimensions, and stiffness.

2.2. Simulation Model Workflow

For each randomized task, the workflow selects simulation components, generates specialized context and an LLM Python model alongside parameter-matched ground truth, then independently executes and evaluates both models. It records configuration, intermediate prompts and responses, source files, telemetry, and task-level metrics for reproducible analysis and inspection.

  • Workflow: Each task begins by selecting a level-dependent task and inserting randomized parameters.Level 3 tasks use a modified version of the workflow described later.
  • Context generation: The pipeline asks the LLM to choose simulation components, generates context from those choices, and warns that missing essential components usually cause failure.The component list contains 25 options for level 2, while complex examples require at most six different components.
  • Model generation and evaluation: Using the generated context and model description, the pipeline prompts Python-code generation while simultaneously creating parameterized ground truth, then independently executes and checks both models.The workflow applies several evaluation metrics after execution.
  • Tracking and logging: The pipeline stores prompts, LLM responses, selected components, extracted and reference source files, evaluation outcomes, and run settings to support task inspection, reproducibility, and aggregated success statistics.Run records also include hardware telemetry, simulator version, timing, token counts, and automatic error or warning summaries.

2.3. Flexible MBS

Flexible multibody-system tasks use a two-phase workflow that first generates and evaluates reusable finite-element machine parts, then assembles eligible parts into Exudyn simulations. The workflow preserves shared parameters and interfaces across parts while enabling LLM-generated geometry, meshing, reduction, and assembly.

  • Workflow structure: Level 3 flexible MBS tasks first load required level 3a flexible-part descriptions before addressing level 3b assembly, with parts reused across multiple systems.Examples include flexible wind-turbine rotor and tower parts used in flexible MBSs such as a simplified wind turbine.
  • Part generation: Master parameters are shared between a machine and its parts, enforcing cross-part consistency such as matching the tower’s nacelle-bore radius to the rotor’s shaft radius.Master parameters are randomly selected within predefined ranges.
  • Part generation: Part descriptions encode comprehensive geometric, material, and interface information, with some parameters randomized and others governed by master parameters.The descriptions follow a template structure intended to specify machine-part geometry unambiguously.
  • Part generation: The LLM selects geometry and finite-element components, then generates Python code intended to produce meshed parts with addressable interfaces for Exudyn’s FEM interface.Available operations include Netgen/NGSolve geometry construction, mesh transfer, material and interface assignment, and Hurty-Craig-Bampton reduction.
  • Assembly: Only level 3b tasks whose required parts were evaluated as correct proceed to assembly; otherwise, the task is skipped and rated as not successful.The assembly phase loads saved parts and generates an Exudyn simulation, including a high-level flexible-body function that places markers at predefined interfaces.
  • Assembly: The LLM and ground-truth assembly simulations use the same previously saved LLM-generated part, isolating assembly-stage differences in evaluation outcomes.This shared-part setup ensures discrepancies are not attributable to different underlying parts.

2.4. LLM Inference

The inference pipeline prioritizes private, reproducible local execution with mostly open-weight models, while tuning quantization, batching, context limits, and generation budgets for hardware-efficient performance. It post-processes model outputs into validated native structures and logs malformed responses for downstream handling.

  • Execution setup: Mostly open-weight models run locally through Ollama using default 4-bit quantization to balance generation quality, VRAM requirements, and inference speed.Local execution supports privacy, data security, and reproducibility.
  • Batching: Batch sizes are manually tuned to hardware VRAM limits, producing reported speedups between two and five for tested models and hardware.The pipeline aggregates raw prompts, applies standardized templates, and performs batched inference.
  • Token limits: The base context window of 4096 and maximum generation length of 2048 are scaled by 1.5 for reasoning-prone coding models and 3.75 when thinking is activated.Activated thinking yields a maximum context size of 15 360.
  • Output processing: For structural retrieval, the system requests XML-like outputs, validates them, converts them into native Python lists, and logs malformed responses within task structures.This post-processing is based on the target evaluation criteria and supports downstream pipeline handling.

2.5. Code Execution

The code-execution pipeline prepares LLM-generated simulation scripts for rigorous evaluation by extracting pure Python, enforcing runtime safety, and adding evaluation-supporting instrumentation.

  • A robust code-execution pipeline is required to rigorously evaluate LLMs’ synthesis of executable mechanical models.
  • The pipeline programmatically isolates and extracts pure Python source code from the LLM’s surrounding natural-language output.
  • Runtime safety checks block illegal filesystem writes and unauthorized access to sensitive system packages before execution.
  • The pipeline dynamically augments cleared code with special lines and function calls to identify error stages, redirect solver outputs, and inject settings.

3. Model Comparison and Verification

The evaluation compares LLM-generated Exudyn simulation scripts with expert ground truth through a sequential, gated pipeline of execution, numerical, topology, graph-content, and overall-success metrics. Level 3 tasks additionally evaluate part geometry, physical properties, and interfaces before multibody-system validation.

  • Evaluation metrics: The metrics assess component selection, assembly completion, solver success, topology match, solution match, graph-content match, and overall success.Assembly failures can indicate programming errors or inconsistent structures, while solver failures can arise from impossible constraints, invalid parameters, or integration errors.
  • Evaluation pipeline: Each evaluation step gates downstream checks, recording a failure metric and terminating evaluation when a check fails.The pipeline applies this sequential gating to level 1, level 2, and level 3 tasks.
  • Flexible multibody verification: Level 3 tasks evaluate part geometry, physical properties, and interface names before applying the same multibody-system metrics used for rigid-body models.The part-specific checks precede the multibody-system checks.
  • Numerical verification: Numerical validation compares generated and reference body-coordinate time series after checking matching coordinate and time-step dimensions.The comparison uses total coordinates q_total = q_ref + q_disp and requires coordinates to map to identical physical entities through deterministic body ordering.
  • Graph verification: Graph verification converts Exudyn items and their internal references into invariant graph nodes and edges, with physical and connector parameters stored as node annotations.This enables topology and parameter comparison independent of generation order, while Python user-functions and numerically computed reduced flexible-body matrices are excluded from graph comparison.

4. Results and Statistical Evaluation

LLM performance improves over time but remains uneven: models achieve strong component and topology selection, while fine-grained parameters, geometry, and flexible multibody simulations remain major bottlenecks. Reasoning can improve accuracy substantially, but increases computation substantially, and model behavior varies with temperature, size, and release generation.

  • Overall rigid-body results: 82.1% was the best open-weight overall success rate, while topology match reached 88.6%, compared with 77.5% for numerical solution and 76.0% for graph content match.The performance gap indicates that models capture broad mechanical concepts but often fail in fine-grained mechanical-parameter selection.
  • Trends and robustness: Six to twelve months was the estimated performance delay of open-weight models relative to selected proprietary counterparts, despite a clear improvement trend over release date.Prompt formulations caused only moderate variations, indicating that the evaluated pipelines were largely prompt-insensitive.
  • Temperature sensitivity: 12 percentage points was ministral-3’s success-rate loss as temperature increased from 0 to 1, whereas the two Gemma 4 models were essentially temperature-insensitive.Overall, low temperatures were preferable, and robustness to temperature was identified as desirable for mechanically aware LLMs.
  • Reasoning and efficiency: 5% to 9% was the thinking-related boost for gemma4:31b, gpt-oss:20b, and gemma4:26b, while reasoning increased inference time by a factor greater than 10 for some models.qwen3.5:27b used 21 times more response tokens with reasoning, so most overall tests were conducted without thinking.
  • Flexible multibody results: Around 65% was Claude-Opus-4.8’s highest flexible-MBS success rate, while gemma4:26b achieved only 5%, below the open-weight average of approximately 25%.Proprietary models averaged around 82% part correctness, compared with around 51% for open-weight models; gemma4:31b reached almost 77%.
  • Part-generation analysis: 12 parts were generated correctly every time by proprietary models, whereas no part was generated correctly every time by open-weight models.Both model classes solved at least some instances of every part-generation task, but geometry-related metrics showed the largest post-selection success-rate drops.

5. Discussion, Limitations and Outlook

The discussion highlights both the substantial potential and persistent errors of LLM-generated mechanical simulation models. It also identifies automation, evaluation-scale, and iterative validation challenges, alongside agentic approaches for improving model reliability.

  • Findings and limitations: 9.9% of tasks averaged over the top 8 open-weight models produced coding-related errors reported to the user.The paper frames LLMs as promising for mechanical tasks while emphasizing the severity of errors in engineering simulation-model creation.
  • Evaluation limitations: Comparison remains challenging for user functions, which require testing across randomized parameters, despite extensions to component, graph, assembly, and solver-error evaluation.Large automated test sets provide detailed statistics and excellent reproducibility, but some comparison classes remain difficult.
  • Evaluation limitations: 94 565 files and 2.9 GB of text made human evaluation impossible for the performance-plot data collected in Fig. 7.Manual monitoring was limited to tasks not completed by the best proprietary LLMs and required iterations to clarify tasks and ground-truth implementations.
  • Outlook: Future agentic frameworks could reduce simulation-model errors through iterative improvement, output analysis, image-based configuration checks, or comparison of independently generated models.The proposed approaches include self-validation and using different simulation codes to compare independently generated simulation models.

Declarations

The declarations specify contributions across conceptualization, methodology, software development, investigation, visualization, validation, writing, review, and editing.

  • Author contributions: The listed authors contributed across conceptualization, methodology, software development, investigation, visualization, validation, writing, review, and editing.J.G., S.W., and T.M. are credited with methodology and software development; P.M. is credited with software development and validation.

References7 · Appendix A. Model List

Appendix A catalogs all 84 mechanical models used in the study and groups them by difficulty level. Table A.1 provides the complete model list.

  • Appendix A. Model List: 84 mechanical models are included in the study.The appendix states that Table A.1 lists all models used in the study.
  • Appendix A. Model List: The models are organized by difficulty level.The model list is explicitly grouped according to difficulty.
  • Appendix A. Model List: The appendix documents the full set of mechanical models used in the study.This restates the appendix’s scope without adding model-specific claims.
  • Appendix A. Model List: The model inventory is presented in Table A.1.Both the appendix text and table caption identify Table A.1 as the model-listing table.
  • Appendix A. Model List: Table A.1 is the complete list of mechanical models.The table caption identifies it as the complete model list.
  • Appendix A. Model List: Table A.1 groups the mechanical models by difficulty level.The table caption specifies grouping by difficulty level.

Appendix B. Complete LLM Evaluation Results

Appendix B reports complete evaluation results for all LLMs across 50 rigid-body task models. Table B.1 defines the reported success measures and separates proprietary from open-weight models.

  • Table B.1 covers all evaluated LLMs on the combined set of 50 rigid-body task models.
  • The table reports component-selection, solver, topology, numerical-solution, graph-content, and overall success rates.Rates are given in percent.
  • A horizontal rule separates proprietary models from open-weight models, and † marks gpt-oss models using minimum reasoning effort (“low”).
Loading 2608.14615v1…