PaperlayerView exact revision on arXiv

Source-linked AI summary

A Large Language Model-Empowered Agent for Reliable and Robust Structural Analysis

Jiachen Liu, Ziheng Geng, Ran Cao, Lu Cheng, Paolo Bocchini, Minghui Cheng

arXiv:2507.02938v1cs.CLcs.AI

TL;DR

The paper asks whether LLMs can reliably and robustly analyze beams, a structural-engineering task where errors may carry serious consequences. It reframes analysis as code generation and automatic execution, with the agent exceeding 99.0% reliability and 99.6% robustness across the benchmark.

  • Problem

    The study examines whether LLMs can reliably and robustly perform beam structural analysis, where errors can have larger consequences than in other fields.

    Source: §1 · p. 2

  • Method

    The agent reframes structural analysis as code generation, using chain-of-thought and few-shot prompting to generate and automatically execute OpenSeesPy code.

    Source: §1 · p. 2

  • Results

    Exceeding 99.0% reliability and 99.6% robustness across all benchmark problems, the agent significantly outperformed the baseline model.

    Source: §5 · p. 13

  • Takeaways & Limitations

    Complete examples and function-usage examples were the primary contributors to performance, with task-specific example portfolios offering potential for analogous tasks.

    Source: §5 · p. 13

    With structured text input, the agent sometimes hallucinates unspecified loads or boundary conditions, while complex configurations may be difficult to describe unambiguously.

    Source: §5 · p. 13

Abstract

from arXiv · show

Large language models (LLMs) have exhibited remarkable capabilities across diverse open-domain tasks, yet their application in specialized domains such as civil engineering remains largely unexplored. This paper starts bridging this gap by evaluating and enhancing the reliability and robustness of LLMs in structural analysis of beams. Reliability is assessed through the accuracy of correct outputs under repetitive runs of the same problems, whereas robustness is evaluated via the performance across varying load and boundary conditions. A benchmark dataset, comprising eight beam analysis problems, is created to test the Llama-3.3 70B Instruct model. Results show that, despite a qualitative understanding of structural mechanics, the LLM lacks the quantitative reliability and robustness for engineering applications. To address these limitations, a shift is proposed that reframes the structural analysis as code generation tasks. Accordingly, an LLM-empowered agent is developed that (a) integrates chain-of-thought and few-shot prompting to generate accurate OpeeSeesPy code, and (b) automatically executes the code to produce structural analysis results. Experimental results demonstrate that the agent achieves accuracy exceeding 99.0% on the benchmark dataset, exhibiting reliable and robust performance across diverse conditions. Ablation studies highlight the complete example and function usage examples as the primary contributors to the agent's enhanced performance.

1 Introduction

Although LLMs have shown promise in specialized applications, their reliability and robustness in structural analysis remain an open question. The study evaluates a beam-analysis benchmark and proposes an agent that generates and executes analysis code.

1 Introduction

  • Domain adaptation uses parameter-free prompting and retrieval as well as parameter updates such as fine-tuning and LoRA.Prompting strategies named include chain-of-thought, few-shot prompting, and prompt chaining.Source: §1 · p. 2
  • The study asks how reliably and robustly an LLM performs in structural analysis, where errors can carry larger consequences than in other fields.It evaluates Llama-3.3 70B Instruct on eight beam-analysis problems and proposes an agent that generates and executes OpenSeesPy code.Source: §1 · p. 2
  • Civil-engineering LLM work spans engineering-document processing, scientific code generation, and visual applications, but structural-analysis reliability and robustness remain unanswered.The paper situates its question amid reported hallucination challenges in LLMs.Source: §1 · p. 2
  • The study addresses this gap by testing a beam benchmark and reframing structural analysis as code generation for an automatically executing agent.The agent uses chain-of-thought and few-shot prompting to generate OpenSeesPy code.Source: §1 · p. 2

2 Benchmark

The benchmark evaluates beam-reaction analysis across eight problems and measures repeated-run reliability and robustness under changing load positions. The baseline LLM’s reliability varies with load complexity, load location, and beam configuration.

2.1 Dataset overview

  • The eight-problem benchmark targets support reactions for statically determinate beams, covering simply supported and mid-span-supported overhang beams under four load patterns.The load patterns include point loads, localized or full-span distributed loads, and combinations; load locations vary along the span.Source: §2.1 · pp. 2–3, Figure 1 · p. 3

2.2 Workflows to assess reliabity and robustness

  • Reliability is the proportion of correct outputs across 500 independent runs of the same problem; robustness tests performance under input variations.The robustness workflow shifts load location from right to left in 1-meter intervals and runs each variation independently 500 times.Source: §2.2 · pp. 3–4, Figure 3 · p. 4

2.3 Performance of Llama-3.3 70B Instruct model

  • For simply supported beams, average reliability was 87.1% for point loads and 93.6% for localized distributed loads, falling to 75.8% and 73.1% for combined-load cases.Reliability also dipped sharply when loads were positioned at beam ends.Source: §2.3 · p. 4
  • For overhang beams, reliability fell below 10.0% for combined point and full-length distributed loads applied in the cantilevered region.Reliability also declined with greater load complexity and when loads were placed at supports.Source: §2.3 · pp. 4–5
  • Three incorrect responses showed errors in equilibrium reasoning, algebra, or reaction direction despite signs of qualitative structural-mechanics understanding.The examples include omitting moment equilibrium and incorrectly assuming a load at a roller is shared equally by the supports.Source: §2.3 · p. 5

3 A large language model-empowered agent for structural analysis

The paper reframes beam analysis as code generation, using an LLM to produce OpenSeesPy code that domain-specific software executes. A structured prompt combines reasoning, examples, and constraints to guide the agent’s code generation.

3.1 Reframe the task: From text generation to code generation

  • Code generation constrains the output space relative to open-ended text, while integrating the LLM with engineering software avoids relying on it to internalize specialized knowledge and perform symbolic computations.The authors present this as a scalable approach that can break complex problems into programmable subtasks.Source: §3.1 · p. 5

3.2 Agent architecture

  • The agent combines a natural-language problem description with predefined prompts, generates OpenSeesPy code, then runs OpenSeesPy for simulation and OpsVis for visualization.Its modular workflow assigns code generation, numerical simulation, and result visualization to distinct stages.Source: §3.2 · pp. 5, 7

3.3 Prompt design

  • The five-part prompt template combines role specification, chain-of-thought reasoning, a complete example, function usage examples, and prescriptive constraints.The complete example demonstrates the reasoning-to-executable-code workflow; function examples address node, element, and complex-load modeling, while constraints specify mandatory code blocks.Source: §3.3 · pp. 7, 9

4 Results and discussion

The agent achieves reliability above 0.990 and robustness above 0.996 across varied loads and boundary conditions, outperforming the baseline model. It also maintains high reliability on extended beam tasks, while ablations identify complete examples and function usage examples as especially influential prompt components.

4.1 Performance of the proposed LLM-empowered agent

  • Reliability exceeds 0.990 and robustness exceeds 0.996 across load and boundary conditions, significantly outperforming baseline Llama-3.3 70B Instruct, whose reliability degrades for some load placements.The agent’s reliability curves remain flat; the authors attribute the improvement to the structured code-generation workflow and dedicated prompt templates.Source: §4.1 · p. 9
  • Remaining errors include hallucinating an extra roller support at a free cantilever end and extending a distributed-load range when a point load is also present.The latter occurs despite function usage examples in the prompt.Source: §4.1 · p. 9

4.2 Generalization of the performance analysis on extended beam tasks

  • Across three extended beam scenarios tested over 500 runs each, reliability is 100%, 99.8%, and 99.6% for tasks (a), (b), and (c), respectively.The cases vary loads and supports beyond the benchmark; the cantilever task asks for both reaction forces and bending moment.Source: §4.2 · pp. 9, 11
  • The agent executes generated OpenSeesPy code and visualizes finite-element models and axial-force, shear-force, and bending-moment diagrams to support validation and error checking.The authors also describe the graphical output as moving toward an explanation of results.Source: §4.2 · p. 11, Figure 10 · p. 12

4.3 Ablation experiments on prompt components

  • Removing the complete example drops reliability to 0% on all three extended tasks; removing function usage examples yields 2.6%, 29.6%, and 62.3%, respectively.The authors link these errors to missing or disordered code, incorrect node placement, and improper load application.Source: §4.3 · p. 12, Table 1 · p. 12
  • Without chain-of-thought reasoning, reliability is 100%, 99.8%, and 78.4%; without prescriptive constraints, it is 93.6%, 97.0%, and 55.6% across tasks (a), (b), and (c).The cantilever task is absent from the prompt template; constraint-removal errors primarily involve omitted code blocks.Source: §4.3 · p. 12, Table 1 · p. 12

5 Summary and conclusions

The paper evaluates LLM performance in beam static-equilibrium analysis and shifts the task from text generation to code generation with automatic execution. The proposed agent outperforms the baseline, while the study identifies input-scope limitations and potential roles for examples and visual feedback.

5 Summary and conclusions

  • The agent reframes structural analysis as code generation, automatically generating and executing OpenSeesPy and OpsVis code from natural-language input.The study focuses on static equilibrium analysis of beams.Source: §5 · p. 12
  • Reliability exceeds 99.0% and robustness exceeds 99.6% across all benchmark problems, significantly outperforming the baseline model.The agent generates and executes structural-analysis code.Source: §5 · pp. 12–13
  • Ablations identify complete examples and function-usage examples as central contributors to agent performance.The paper links these examples to interpreting semantic descriptions, reasoning, and coding syntax, and sees potential for tasks with similar logical and code patterns.Source: §5 · p. 13
  • Visual output can support analyst feedback and interactive human–LLM collaboration on structural configurations.The paper describes a progressive workflow in which experts use domain knowledge to identify and correct potential errors in generated code.Source: §5 · p. 13
  • With structured natural-language input, the agent sometimes hallucinates loads or boundary conditions absent from the problem description.The paper suggests vision-language models as a future direction for interpreting textual and graphical inputs, especially for more complex configurations where text can be ambiguous.Source: §5 · p. 13

Appendix: Execution examples of the proposed LLM-empowered agent

The appendix illustrates agent-generated OpenSeesPy examples for simply supported and overhang beams under varied loads. The scripts define beam models, perform static analysis, report support reactions, and plot internal-force diagrams.

Appendix: Execution examples of the proposed LLM-empowered agent

Loading 2507.02938v1…