FullStack Bench: Evaluating LLMs as Full Stack Coders

1. Introduction

• Existing benchmarks focus on limited application domains; DS-1000 has over 95% of its problems in data analysis and machine learning, while xCodeEval mainly covers advanced programming and mathematics.The paper’s analysis labels 500,000 StackOverflow questions and identifies 11 mainstream application domains covering about 88.1% of those questions.Source: §1 · p. 3
• In its analysis of StackOverflow, FullStack Bench reports that the ratios of its selected 11 application domains (excluding “Others”) are 94.3% for FullStack Bench and 88.1% for StackOverflow, respectively.Source: §1 · p. 3
• SandboxFusion is introduced to support evaluation across the benchmark’s varied formats, dependencies, and 16 programming languages, which existing sandboxes may not fully support.The paper identifies front-end browsers and deep-learning packages as examples of unsupported capabilities in existing sandboxes.Source: §1 · p. 3
• 3374 problems across 16 programming languages make FullStack Bench broader in application-domain coverage than existing code evaluation benchmarks.The benchmark is intended to assess models across real-world code-development scenarios.Source: §1 · p. 4
• Many models score high on HumanEval but relatively lower on FullStack Bench, indicating different performance across the two benchmarks.The comparison is shown in Figure 2.Source: §1 · p. 4, Figure 2 · p. 4

2.1. Data Overview

• FullStack Bench contains 3,374 problems, each with a question, unit tests, reference solution, and labels.The dataset has 15,168 unit tests overall and an average question length of 210.2 tokens.Source: §2.1 · p. 4

2.2. Data Construction and Quality Control

• Annotators assess difficulty, ambiguity, and solvability, then cross-refine one another’s work; disagreements are resolved by consensus or senior annotators.The construction process uses code snippets and documents alongside LLM and human verification, with programming experts also creating domain-specific questions.Source: §2.2 · pp. 5–6
• Six selected models classify problems as hard when only one answers correctly and easy when five or six do; remaining problems are medium.This voting method is used to improve the benchmark’s difficulty.Source: §2.2 · p. 6
• The benchmark samples 500k StackOverflow questions, prompts LLMs to label each application domain, and retains the top 11 domains, which together account for 88.1% of the questions.Other application domain types are named “Others”.Source: §2.2 · p. 6

2.3. Bilingual Benchmark Construction

• The Chinese and English versions each contain 1,687 problems, produced by translating collected questions between the two languages.Source: §2.3 · p. 7

2.4. Evaluation Metrics

• Pass@1 is the default evaluation metric, following HumanEval and MBPP.Source: §2.4 · p. 7

3. SandboxFusion

• SandboxFusion supports execution-based evaluation across datasets and is presented as applicable to data processing, model evaluation, and reinforcement learning.The paper motivates the environment by the need to execute complete model-generated code in compatible environments despite varying formats and dependencies.Source: §3 · p. 7
• The evaluation pipeline generates prompts, extracts executable code, synthesizes test programs, executes them, judges correctness, and calculates pass rates.Model completion is performed independently by users; the framework does not provide built-in inference.Source: §3 · p. 8
• SandboxFusion has a Dataset Module for implementing datasets and reusing common components, and a Sandbox Execution Module for language-specific execution, resource control, and safety.Source: §3 · p. 8

4.1. Experimental Setup

• The evaluation includes 27 popular language models, with open-source models grouped into five size categories and prominent closed-source models also tested.Open-source model inference used temperature 0 and a 2,048-token completion limit; close-sourced models were evaluated similarly via API.Source: §4.1 · pp. 8–9

4.2. Results and Analysis

• OpenAI o1-preview leads across the 11+ real-world domains, while some open-source models match or surpass closed-source models.DeepSeekCoder-v2-Instruct leads open-source models in AP, OS, and Others; several smaller open-source models top their size groups and outperform some models in the closest larger group.Source: Table 2 · p. 9, §4.2 · pp. 9–10
• Performance varies substantially across domains; mathematics shows the widest range, from OpenAI o1-mini at 81.47 to CodeLlama-34B-Instruct at 18.18.The authors note that mathematics combines math and coding demands, while ML, SC, and MM also require domain knowledge beyond coding.Source: §4.2 · p. 10, Figure 5 · p. 10

4.3. Analysis on the performance of different programming languages

• Language results vary significantly, with larger gaps in C#, D, PHP, Ruby, Rust, and Scala; compiler pass rates correlate positively with test pass rates but do not guarantee them.Prompt language also affects performance: some models perform better on Chinese than English questions, and vice versa.Source: §4.3 · p. 10, Table 3 · p. 11, Figure 7 · p. 12

4.4. Scaling Laws on FullStack Bench

• Performance improves as model size increases across all five model families, but gains diminish as size grows.The authors describe this pattern as a scaling law on FullStack Bench.Source: §4.4 · p. 11, Figure 8 · p. 11

4.5. Analysis on the performance of different difficulties

• On difficulty levels, 1B+ models and the CodeLlama series are less effective throughout; gaps emerge on medium problems, and closed-source models generally outperform open-source coders on hard problems.The problem difficulty labels were determined using votes from six AI coders.Source: §4.5 · p. 11

4.6. Analysis on the effect of feedback from SandboxFusion

• Reflection with SandboxFusion feedback performs much better than BoN, which infers repeatedly without refining answers using feedback.Reflection applies the self-refine strategy, refining answers N times with SandboxFusion feedback context.Source: §4.6 · p. 12

• Code LLMs have advanced across generation, debugging, and translation, while multilingual benchmarks such as MultiPL-E, McEval, and MdEval assess programming across languages.The cited work notes that many in-file benchmarks focus on a limited range of languages, such as Python and Java.Source: §5 · pp. 12–13
• Many code benchmarks target particular capabilities, including synthesis, translation, retrieval, vulnerability repair, structured data understanding, and repository-level tasks.McEval extends multilingual evaluation to 40 languages, while MdEval covers nearly 20 languages for code debugging.Source: §5 · p. 13
• The paper positions FullStack Bench as addressing the lack of evaluation of LLMs as program developers across varied real-world code development scenarios.Source: §5 · p. 14

6. Conclusion

• FullStack Bench contains 3,374 problems across mainstream application domains, each with corresponding unit test cases.The paper names basic programming, software engineering, and machine learning as examples.Source: §6 · p. 14
• SandboxFusion supports various languages, is easy to deploy, and provides a unified multi-dataset execution environment.Source: §6 · p. 14
• The authors hope FullStack Bench will help researchers understand existing LLMs’ code intelligence abilities and accelerate foundation-model growth.Source: §6 · p. 14

8. Contributions

• Siyao Liu implemented the dataset execution environment with SandboxFusion and performed quality assurance; other contributors implemented open-source datasets in SandboxFusion.He Zhu validated experimental results and made visualizations, and Shulin Xin conducted pass-rate and reflection experiments.Source: §8 · p. 15
• Aoyan Li and Yifan Sun implemented domain classification, while several contributors added minority-language and domain samples and revised them.The additional samples were contributed by Shukai Liu, Z.Y. Peng, Jinxiang Xia, and Jack Yang.Source: §8 · p. 15
• Li Chen, Aoyan Li, Siyao Liu, Rui Long, Yongsheng Xiao, and Shulin Xin contributed dataset samples and maintained an iterative curation process.Source: §8 · p. 15

A.2.1. Dataset Module

• AutoEvalDataset extracts generated code, runs it with prewritten tests, and checks the return value; CommonOJDataset tests language-agnostic problems by comparing standard output with expected output.FullStack Bench is supported in AutoEvalDataset mode.Source: §A.2.1 · pp. 23, 25

A.2.2. Sandbox Execution Module

• SandboxFusion supports 23 programming languages, integrates machine-learning packages and a front-end browser, and provides execution, version, task-specific, and isolation features.Its interface accepts code, language, and input context and returns execution information; additional capabilities include GPU execution, sequential Jupyter blocks, and resource, file-system, and network isolation.Source: §A.2.2 · p. 25

A.3. Comparison with Other Sandboxes

• Compared sandboxes differ in security and integration: SandboxFusion targets evaluation throughput and multi-dataset use, while DifySandbox, MultiPL-E, and MPLSandbox have distinct constraints.SandboxFusion can test PyQt desktop programs, Flask web applications, and PyTorch or TensorFlow neural networks, which the authors say other sandboxes cannot support.Source: §A.3 · pp. 25–26, Table 7 · p. 26