PaperlayerView exact revision on arXiv

Source-linked AI summary

Integrating Large Language Models for Automated Structural Analysis

Haoran Liang, Mohammad Talebi Kalaleh, Qipei Mei

arXiv:2504.09754v1cs.SEcs.LG

TL;DR

Traditional structural analysis workflows require manual effort and specialized knowledge, and no systematic LLM framework for automatic structural analysis is identified in the paper. The authors combine LLMs with OpenSeesPy and evaluate the framework on 20 word problems; GPT-4o performs best among the evaluated models, while the authors note dataset and run-count limits.

  • Problem

    Traditional FEM-based structural analysis requires substantial manual effort and specialized knowledge, while systematic evaluation of LLMs for structural analysis remains needed.

    Source: §1 · p. 2

  • Method

    The framework uses LLMs to turn natural-language structural descriptions into Python scripts for OpenSeesPy analysis, with structured prompts and staged code generation.

    Source: §1 · p. 2, §2.3 · p. 3, §5 · p. 16

  • Results

    GPT-4o demonstrates the best performance among the evaluated models on the 20-problem benchmark.

    Source: §5 · p. 16

  • Takeaways & Limitations

    The authors conclude that LLM-generated structural models from natural-language input could support structural-engineering workflows by streamlining repetitive modeling tasks.

    Source: §5 · p. 16

    The dataset and number of experimental runs were limited, and the dataset was manually designed for this project.

    Source: §5 · p. 16

Abstract

from arXiv · show

Automated analysis for engineering structures offers considerable potential for boosting efficiency by minimizing repetitive tasks. Although AI-driven methods are increasingly common, no systematic framework yet leverages Large Language Models (LLMs) for automatic structural analysis. To address this gap, we propose a novel framework that integrates LLMs with structural analysis software. LLMs serve as the core engine: they parse structural descriptions from text and translate them into executable Python scripts. Moreover, the framework integrates the generative capabilities of LLMs with code-based finite element (FE) tools like OpenSeesPy. It employs domain-specific prompt design and in-context learning strategies to enhance the LLM's problem-solving capabilities and generative stability, enabling fully automated structural analysis from descriptive text to model outputs. In our experiments, we introduce a well-curated small-scale benchmark dataset of 20 structural analysis word problems (SAWPs) with ground-truth solutions and evaluate the performance of different LLMs within our framework in solving these SAWPs. The role of system instructions, crafted by structural engineers, is also investigated to understand their impact on LLM-driven structural analysis. Additionally, the generative stability of our framework is examined. Through multiple validation experiments on the benchmark, our results demonstrate that the proposed framework can substantially increase the level of automation in solving SAWPs compared to traditional methods. Quantitatively, the framework, built on GPT-4o, achieved 100% accuracy, surpassing GPT-4 (85%), Gemini 1.5 Pro (80%), and Llama-3.3 (30%) on the test examples. Furthermore, integrating domain-specific instructions enhanced performance by 30% on problems with asymmetrical structural configurations.

1 Introduction

Traditional finite element methods still demand substantial manual effort and specialized knowledge, while systematic evaluation of LLMs for structural analysis remains an open need. The paper proposes an LLM–OpenSeesPy framework and evaluates it on a curated set of 20 structural analysis word problems.

Motivation

  • Traditional FEM-based methods require substantial manual effort and specialized knowledge, motivating further automation of structural analysis.Computational approaches are increasingly favored for efficiency and their ability to handle complex structural systems.Source: §1 · pp. 1–2

Related work

  • Civil engineering has received comparatively less attention in LLM applications than domains such as healthcare, finance, and data science.Existing civil-engineering examples include architectural flaw detection, construction writing, worker-risk assessment, and code-related question answering.Source: §1 · p. 2

Research gap

  • Prior LLM work in structural analysis includes code generation for a Poisson equation and a simple 2D plate, but the latter approach faces scalability issues.The paper identifies systematic assessment of model capabilities and reliable, comprehensive evaluation as two critical challenges.Source: §1 · p. 2

Contributions

  • 20 structural analysis word problems with ground-truth solutions form the benchmark for evaluating LLMs integrated with OpenSeesPy.The framework uses models including GPT-4, GPT-4o, Llama 3, and Gemini 1.5, with baseline and prompting-enhanced versions.Source: §1 · p. 2
  • The framework uses LLMs to extract information from descriptions and generate FEA Python scripts for 2D frames, with few-shot and in-context learning comparisons.The authors state that natural-language prompts can reduce the time required for structural modeling and results visualization.Source: §1 · p. 2

2 Methodology

The methodology connects user requirements and structured instructions to staged LLM code generation with OpenSeesPy, then formats analysis results into user-oriented outputs. Its components cover input preparation, model generation, and reporting.

2.1 General Workflow

  • The workflow connects user requirements, system instructions, LLM processing, and structured outputs for 2D-frame analysis.The framework's sections describe requirements and instructions, problem decomposition and model processing, then visualizations and output format.Source: §2.1 · p. 3, §2.3 · p. 3

2.2 Data Layer

  • System instructions pair a problem description with corresponding Python code and instruction-tuning elements, including intermediate reasoning steps for specific subtasks.SQLite structures these instructions, and the framework combines them with the user's problem description.Source: §2.2 · p. 3

2.3 Model Layer

  • Three API calls divide generation into parameter extraction, structural layout and loading setup, and visualization, isolating failures and simplifying debugging.OpenSeesPy performs the structural analysis, while OpsVis supports result visualization; the authors say staged generation mitigates errors from lengthy one-pass code generation.Source: §2.3 · p. 3
  • The ICL template combines general settings, task-specific constraints, a reference problem with code, and the question to be solved.The example includes instructions for writing OpenSeesPy and OpsVis scripts.Source: §2.3 · p. 6
  • Commonsense reasoning instructions target interpretation of problem descriptions and structural layouts, including reasoning about distributed-load direction.The figure examples are illustrative; the distributed-load direction reasoning is presented in Table 3.Source: §2.3 · p. 6

2.4 Output Layer

  • The output layer formats results as reports or spreadsheets, including generated code, result explanations, and internal forces at each node with maximum and minimum values.Outputs depend on user requirements and can include visualized results.Source: §2.4 · p. 7

3 Experiment Setup

The experiments use API-accessed LLMs and a manually designed benchmark of 20 simple 2D-frame problems with ground truth. The problems vary structure size, symmetry, and structural features to test different modeling demands.

3.1 Computational Setup

  • The study accessed GPT-4o, GPT-4 Turbo, Gemini 1.5 Pro, and Llama 3.3 through APIs; most tasks completed within a few seconds per request.Scripts were written in Python 3.12 and run on a Windows 11 laptop.Source: §3.1 · p. 7

3.2 Test Examples

  • 20 manually designed simple 2D-frame problems pair detailed word descriptions with ground-truth solutions for validating LLM outputs.The paper notes that no publicly available dataset systematically combines structural descriptions, clear layouts, and corresponding ground truth.Source: §3.2 · p. 7
  • The benchmark standardizes geometry, material properties, loads, boundary conditions, and visualization requirements, with schematics prepared as ground truth.Three generation patterns vary story and bay counts, introduce asymmetry, and add features such as diagonal members and cantilever beams.Source: §3.2 · p. 8

4 Results and Discussion

Across the benchmark, GPT-4o leads the model comparison, while one-example prompting generalizes across frame layouts. Asymmetrical cases and instruction design expose reliability and scalability trade-offs.

4.1 Comparative Analysis of LLM-Generated Structural Analysis

  • One frame example generalized to multi-bay, multi-story layouts and to diagonal members and cantilever beams absent from the reference.The reference modeled columns and girders; the framework extended the pattern to other configurations and components.Source: §4.1 · p. 10
  • 100% accuracy for GPT-4o, versus 85% for GPT-4 and 80% for Gemini 1.5 Pro, was reported on 20 SAWPs; Figure 8 presents best-of-3 results across four LLMs.Each problem was attempted three times, and a problem counted as solved if at least one attempt was correct.Source: §4.1 · pp. 9–10, Figure 8 · p. 10
  • Llama 3 failed all three problems, while Gemini 1.5 Pro struggled to scale patterns and GPT-4o followed distributed-load direction instructions better than GPT-4.On Example 10, Gemini produced non-executable code when scaling to a three-bay, two-story frame; on Example 20, GPT-4o improved with specific load-direction instructions while GPT-4 did not.Source: §4.1 · p. 10

4.2 Evaluation of Output Stability in the Proposed Framework

  • Performance declined on asymmetrical frames, and text-only generation produced layout and boundary-condition errors that GPT-4o could not visually inspect.The reported asymmetric examples are 11 and 13; errors also included incorrect nodes, elements, loads, and supports.Source: §4.2 · pp. 11–12
  • With one reference example and limited attempts, the GPT-4o framework generated complete, executable OpenSeesPy code for all 20 SAWPs; Figure 10 summarizes five-run accuracy.The stability experiment ran the framework five times per problem.Source: §4.2 · p. 11, Figure 10 · p. 12

4.3 Impact of Instructions on LLM Output Quality

  • On Problem 20, GPT-4o reached 100% accuracy from 70% with distributed-loading-direction instructions, while Gemini 1.5 Pro also improved.The models were tested ten times independently; the passage gives no post-instruction accuracy for Gemini.Source: §4.3 · p. 13, Figure 14 · p. 15
  • More system instructions increase token and computational costs, and excessive or ambiguous instructions can confuse models or degrade performance.The authors identify efficient instruction synthesis and compression as future research directions.Source: §4.3 · p. 15
  • 80% from 50% accuracy on Example 12 followed complete system instructions in ten GPT-4o runs; Figure 15 compares results with and without instructions.Without the instructions, outputs included layout, load, and support errors.Source: §4.3 · p. 14, Figure 15 · p. 15

5 Concluding Remarks

The paper presents an LLM–OpenSeesPy framework and a 20-problem benchmark for structural analysis from natural-language input. It reports GPT-4o as the best-performing evaluated model, while identifying dataset size, experimental runs, and manually designed tasks as limitations.

5 Concluding Remarks

  • The framework combines structured prompts and in-context learning with OpenSeesPy to generate structural-analysis models from natural-language input.The authors manually designed 20 structural-analysis word problems with ground truth, descriptions, and corresponding code for evaluating LLMs.Source: §5 · p. 16
  • GPT-4o performs best among the evaluated models on the benchmark, while reasoning instructions significantly enhance performance, especially for structural-engineering code generation and reasoning.The authors describe LLM-based structural analysis as having potential to enhance efficiency, reduce costs, and streamline repetitive modeling tasks.Source: §5 · p. 16
  • The benchmark and experimental runs were limited in number, and the tasks were manually designed rather than drawn from real-world applications.The authors also note that system instructions are not guaranteed to be correct and that expert evaluation may be costly.Source: §5 · p. 16

Appendix A Dataset

Appendix A presents problem descriptions and ground-truth schematics in Tables 4–8. Appendix B provides an ICL prompt example and four detail figures.

Appendix A Dataset

Appendix B ICL Template

Appendix C Baseline

Appendix C identifies a failure case involving GPT-4o’s response without the ICL template.

Appendix C Baseline

  • Table 9 is identified as failure case 1: GPT-4o’s response without the ICL template.Source: Table 9 · p. 29

Problem Statement

The example asks for deformations and internal forces in a fixed-support 2D frame under horizontal point and vertical distributed loads. Its stated effects include frame sway, girder bending, and associated force and moment responses.

Problem Statement

  • The problem specifies a 2D frame with two 4.0-meter columns and a 6.0-meter girder, subjected to a horizontal point load and a uniform vertical girder load.It asks for resulting deformations and axial, shear, and moment forces.Source: Problem Statement · p. 29

Effect of Horizontal Load

  • The horizontal load produces lateral sway, horizontal deflection at the loaded column top, and axial forces in the girder and columns.Source: Effect of Horizontal Load · p. 29

Effect of Vertical Uniform Load

  • The vertical uniform load bends the girder downward with maximum bending at midspan, induces shear and moment reactions at column bases, and creates column axial forces.Source: Effect of Vertical Uniform Load · p. 29

Horizontal Deflection at Top of Left Column

Vertical Deflection at Midspan of Girder

Girder

The reported results cover girder moment, support shear, and sway-related axial force, alongside column moments and forces.

Girder

  • The girder results report maximum bending moment and shear force at the supports.The supplied passage gives no numerical values for these two results.Source: Girder · p. 30
  • The girder has non-zero axial force due to sway.Source: Girder · p. 30

Columns

  • Columns have moments at the top and bottom, opposite axial-force signs, and shear from horizontal and vertical loading.One column is in compression and the other in tension.Source: Columns · p. 30

Summary Table

The summary gives approximate frame deformations and internal forces for a fixed-support frame with two columns, one girder, and horizontal and vertical loading.

Summary Table

  • 13.3 mm horizontal and 15.6 mm vertical deflections accompany a 45,000 Nm girder moment and 30,000 N support shear.The summary table also reports non-zero girder axial force from sway and opposite column axial forces.Source: Summary Table · p. 30
  • The listed results include significant column moments at the top and base, with one column in tension and the other in compression.Source: Summary Table · p. 30

Problem Description

Python Code for Structural Analysis

The Python code represents the frame with beam-column elements, assembles a global stiffness system, applies loads and fixed supports, and solves for free-degree-of-freedom displacements.

# Loads

  • The frame model uses four nodes and three elements, with column and girder section properties assigned to their respective elements.The nodes are located at the column bases and tops.Source: # Loads · pp. 31–32
  • Element stiffness is computed from material and section properties, element length, and orientation.The local stiffness matrix includes axial and bending terms.Source: # Loads · p. 32
  • Element stiffness matrices are assembled into a global matrix, while the horizontal point load and distributed girder load contribute to the global force vector.Source: # Loads · p. 32
  • After fixed degrees of freedom are removed, the code solves the reduced system and places the resulting displacements into the full vector.Source: # Loads · p. 32

Nodal Displacement Results

A figure and a table are labeled as the ground-truth solution for Example 1.

Nodal Displacement Results

Related papers

Loading 2504.09754v1…