Source-linked AI summary
DeepSeek vs. ChatGPT vs. Claude: A Comparative Study for Scientific Computing and Scientific Machine Learning Tasks
Qile Jiang, Zhiwei Gao, George Em Karniadakis
TL;DR
LLMs are increasingly applied to scientific computing, but their reliability and comparative capabilities on challenging computational tasks require domain-specific evaluation. This paper compares DeepSeek, ChatGPT, and Claude variants on numerical methods and scientific machine learning problems designed to require non-trivial decisions. Reasoning and hybrid-reasoning models consistently outperform non-reasoning models, while ChatGPT o3-mini-high generally offers the fastest reasoning speed.
Problem
LLMs show promise for scientific computing, but their reliability, reasoning depth, and generalization on research-level scientific problems remain limited and require domain-specific assessment.
Method
The study compares six DeepSeek, ChatGPT, and Claude variants on challenging numerical-method and scientific-machine-learning tasks requiring model choices and reasoning decisions.
Results
Reasoning and hybrid-reasoning models consistently performed better than non-reasoning models in recognizing problem characteristics and choosing appropriate solution strategies.
Takeaways & Limitations
The findings support the growing practicality of LLMs for scientific research while highlighting the risk that ambiguous or incorrect responses may confuse researchers unfamiliar with the subject.
Takeaways & Limitations
The study reports self-reported reasoning times only for reasoning models and calls for evaluation on more complex real-world challenges involving cascaded decisions.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have emerged as powerful tools for tackling a wide range of problems, including those in scientific computing, particularly in solving partial differential equations (PDEs). However, different models exhibit distinct strengths and preferences, resulting in varying levels of performance. In this paper, we compare the capabilities of the most advanced LLMs--DeepSeek, ChatGPT, and Claude--along with their reasoning-optimized versions in addressing computational challenges. Specifically, we evaluate their proficiency in solving traditional numerical problems in scientific computing as well as leveraging scientific machine learning techniques for PDE-based problems. We designed all our experiments so that a non-trivial decision is required, e.g. defining the proper space of input functions for neural operator learning. Our findings show that reasoning and hybrid-reasoning models consistently and significantly outperform non-reasoning ones in solving challenging problems, with ChatGPT o3-mini-high generally offering the fastest reasoning speed.
1 Introduction
LLMs are increasingly used in scientific research and computational problem-solving, but their reliability and reasoning depth remain concerns. This study therefore emphasizes domain-specific evaluation of DeepSeek, ChatGPT, and Claude on computational mathematics and scientific machine learning tasks.
- LLMs now support scientific information synthesis, coding, and domain-specific problem-solving across fields including materials science, genetics, medical imaging, and computational fluid dynamics.
- Despite these applications, LLM limitations raise concerns about reliability in scientific research and computing.
- Domain-specific testing is needed to assess correctness, reasoning depth, reliability, and generalization to research-level scientific problems.
- The study compares reasoning and non-reasoning models from DeepSeek, ChatGPT, and Claude on numerical methods and scientific machine learning tasks.Tasks include numerical integration, finite difference and finite element methods, image recognition, PINNs, and DeepONets.
2 Experiments
The experiments evaluate six LLMs from DeepSeek, OpenAI, and Anthropic on challenging numerical-method and scientific-machine-learning problems. Models differ in general-purpose, reasoning-optimized, and hybrid-reasoning designs, while the evaluation prioritizes solution quality and decision-making over response time.
- Six LLMs from DeepSeek, OpenAI, and Anthropic are tested on challenging numerical algorithms and scientific machine learning benchmark problems.
- Model variants: ChatGPT 4o and DeepSeek V3 are general-purpose models, whereas Claude 3.7 Sonnet is described as a hybrid reasoning model.
- Model variants: DeepSeek R1 and ChatGPT o3-mini-high are reasoning-oriented models, while Claude 3.7 Sonnet extended thinking adds an additional reasoning layer.
- Evaluation design: The evaluation disables chat memory and personalized settings, reports self-reported reasoning times only for reasoning models, and emphasizes solution quality and decision-making.
- Traditional numerical methods: The experiments cover traditional numerical methods for ordinary and partial differential equations, including finite difference and finite element methods.
- Scientific machine learning: Scientific machine learning tests include MNIST digit recognition, physics-informed neural networks, and DeepONets.
2.1 Traditional numerical methods
The study evaluates LLM choices and performance across stiff ODEs, Poisson finite differences, beam finite elements, and singular quadrature problems. Reasoning models generally make more appropriate methodological decisions, but finite-element accuracy remains limited.
- Robertson ODEs: Non-reasoning RK4 implementations with relatively large step sizes produced exponentially growing solutions for the Robertson problem.Claude 3.7 Sonnet instead used adaptive RK4, while ChatGPT o3-mini-high produced a satisfactory solution with the shortest reasoning time.
- Robertson ODEs: Reasoning models recognized the Robertson system’s stiffness and selected implicit methods, whereas non-reasoning models failed to do so.DeepSeek R1, ChatGPT o3-mini-high, and Claude 3.7 Sonnet with extended thinking identified stiffness; non-reasoning models used RK4.
- Poisson finite differences: Only Claude 3.7 Sonnet produced a correct Poisson solution on the L-shaped domain; other models made sign errors, and DeepSeek R1 also produced incorrect scales.DeepSeek used advanced iterative methods, while other implementations relied on conventional finite-difference discretization and linear-system solvers.
- Beam finite elements: None of the LLMs correctly applied finite elements to obtain a very accurate beam-equation solution.All models used Hermite basis functions and recognized the need for a weak form and Galerkin method, but selected the wrong test space.
- Beam finite elements: Reasoning models had smaller beam-equation L2 errors than non-reasoning models, with Claude 3.7 Sonnet extended thinking achieving the smallest L2 error.ChatGPT o3-mini-high and Claude 3.7 Sonnet with extended thinking also responded faster than DeepSeek R1 among reasoning models.
- Quadratures: Variable transformation achieved the fastest convergence for the singular integral, while Claude models compared multiple quadrature techniques.Most models identified the singularity and transformed the integral before applying quadrature; DeepSeek V3 instead used scipy.integrate directly.
2.2 Scientific Machine Learning
The study evaluates LLM-generated scientific machine-learning solutions across image recognition, PINNs, and DeepONets. Reasoning models generally make stronger decisions on challenging tasks, but implementation errors, inefficient computation, and insufficient training can limit performance.
- MNIST digits prediction: All models achieved comparable training times and high testing accuracy on the standard MNIST classification task.Claude models and DeepSeek R1 used early stopping; ChatGPT o3-mini-high had the fastest reasoning time among reasoning models.
- Physics-informed neural networks: Correcting JAX gradient usage enabled all PINN codes to run smoothly for the Poisson equation on an L-shaped domain.The original error arose because grad was incorrectly used with nonscalar outputs.
- Physics-informed neural networks: ChatGPT o3-mini-high and Claude 3.7 Sonnet achieved correct predictions and significantly lower L2 errors than the other PINN models.The results indicate stronger handling of the solution’s complex behavior and nontrivial domain modifications by reasoning models.
- Physics-informed neural networks: Reasoning speed varied substantially: DeepSeek R1 required 417 seconds, versus 15 seconds for ChatGPT o3-mini-high and 38 seconds for Claude 3.7 Sonnet.ChatGPT o3-mini-high required over 1000 seconds of training, partly because it generated new samples at every epoch.
- DeepONet operator learning: DeepONet data-generation choices strongly affected operator-learning generalization, with smaller GRF length scales being more difficult and DeepSeek R1 achieving the best reported result in that test.No model used Gaussian Random Fields for training, despite their use in the original DeepONet study.
- DeepONet operator learning: For the Caputo fractional derivative, fixed fractional orders reduced generalization, while Claude 3.7 Sonnet achieved the best result but was slowed by inefficient tensor manipulation.Most models also trained for no more than 100 epochs, whereas increasing training to 1000 epochs reduced relative error to single digits for the same model.
3 Summary
The study compares six DeepSeek, ChatGPT, and Claude models on challenging scientific-computing and scientific-machine-learning tasks. Reasoning-optimized models generally made more appropriate problem decisions, while general-purpose models sometimes produced incorrect solutions; the authors therefore emphasize both LLM practicality and remaining limitations.
- Study scope: Six models from DeepSeek, OpenAI, and Anthropic were evaluated on numerical methods and scientific machine-learning problems requiring advanced mathematical reasoning and implementation decisions.The assessment covered selecting numerical methods or neural-network architectures and implementing them correctly in Python.
- Findings: Reasoning-optimized models consistently performed better at recognizing problem characteristics and making appropriate decisions.The study identifies DeepSeek R1, ChatGPT o3-mini-high, and Claude with extended thinking as the stronger-performing models.
- Findings: General-purpose DeepSeek V3 and ChatGPT 4o sometimes ignored problem properties or user instructions, consequently generating incorrect solutions.Examples include failing to account for stiffness or to implement methods from scratch.
- Implications: The findings indicate practical potential for LLMs in scientific research while exposing ambiguous or incorrect responses that may confuse unfamiliar researchers.The authors call for continued improvements and broader evaluation on complex real-world computational challenges involving cascaded decisions.