PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comparative Study of Programming Languages in Rosetta Code

Sebastian Nanz, Carlo A. Furia

arXiv:1409.0252v4cs.SE

TL;DR

Programming-language debates lack reliable, well-founded comparisons of properties such as speed and conciseness. This study compares solutions to common tasks in Rosetta Code, finding systematic differences in conciseness, performance, and runtime failures across language paradigms. These results inform developers and language designers about trade-offs among language features.

  • Problem

    Programming-language choices involve trade-offs, but reliable answers about properties such as speed and conciseness are not easily available.

  • Method

    The study compares solutions to clearly defined, comparable programming tasks in Rosetta Code, using task categories for lines-of-code analysis, compilation, and execution.

  • Results

    Functional and scripting languages are more concise; C is fastest on large inputs while moderate inputs narrow performance differences; compiled strongly typed languages have fewer runtime failures.

  • Takeaways & Limitations

    The findings identify trade-offs among language features and provide evidence relevant to software developers and language designers.

  • Takeaways & Limitations

    Rosetta Code favors small algorithmic tasks, so its programs may not represent complete real-world software projects.

Abstract

from arXiv · show

Sometimes debates on programming languages are more religious than scientific. Questions about which language is more succinct or efficient, or makes developers more productive are discussed with fervor, and their answers are too often based on anecdotes and unsubstantiated beliefs. In this study, we use the largely untapped research potential of Rosetta Code, a code repository of solutions to common programming tasks in various languages, to draw a fair and well-founded comparison. Rosetta Code offers a large data set for analysis. Our study is based on 7087 solution programs corresponding to 745 tasks in 8 widely used languages representing the major programming paradigms (procedural: C and Go; object-oriented: C# and Java; functional: F# and Haskell; scripting: Python and Ruby). Our statistical analysis reveals, most notably, that: functional and scripting languages are more concise than procedural and object-oriented languages; C is hard to beat when it comes to raw speed on large inputs, but performance differences over inputs of moderate size are less pronounced and allow even interpreted languages to be competitive; compiled strongly-typed languages, where more defects can be caught at compile time, are less prone to runtime failures than interpreted or weakly-typed languages. We discuss implications of these results for developers, language designers, and educators.

I. INTRODUCTION

The study uses Rosetta Code to replace anecdotal programming-language debates with empirical comparisons of comparable programs. Across eight languages and 7,087 solutions, it finds trade-offs in conciseness, performance, and runtime failure proneness.

  • Rosetta Code provides comparable solutions to common tasks, occupying a middle ground between controlled assignments and incomparable real-world repositories.
  • The study analyzes 7,087 solutions for 745 tasks in eight widely used languages spanning procedural, object-oriented, functional, and scripting paradigms.
  • Functional and scripting languages enable more concise code than procedural and object-oriented languages.
  • C is hard to beat for raw speed on large inputs, while Go is the runner-up and also uses memory economically.
  • Performance differences shrink on moderate inputs, allowing interpreted languages with lightweight runtimes to remain competitive.
  • Compiled strongly-typed languages are less prone to runtime failures than interpreted or weakly-typed languages.
  • The findings support practical implications for developers, language designers, and educators and motivate follow-up research using Rosetta Code.
  • The complete data and analysis scripts are released online to support repetition and replication studies.

A. The Rosetta Code repository

Rosetta Code is a large wiki-based repository of solutions to programming tasks in many languages. Its tasks and solutions provide a broad empirical source, but task descriptions vary substantially in precision and scope.

  • The analyzed repository snapshot was taken on 24 June 2014, and supporting data were cloned into the study’s repository.
  • Rosetta Code organizes 745 computational tasks and contains solutions in 379 languages, totaling 49,305 solutions and 3,513,262 lines.
  • Task descriptions range from precise algorithms with tests to vague themes that may be inapplicable to some languages or permit widely different solutions.

B. Task selection

The study selects tasks whose descriptions and inputs support meaningful cross-language comparisons. It progressively restricts the repository to tasks suitable for code-size, compilation, execution, and performance analyses.

  • Tasks are categorized as LOC, COMP, or EXEC using increasingly restrictive criteria for comparable solutions and meaningful execution.
  • Many tasks are too vague for analysis, although the differences among the three selection categories are limited.
  • PERF tasks provide consistent, well-defined inputs for everyday workloads, whereas SCAL tasks support larger computing-intensive inputs.

C. Language selection

The study selects eight languages by combining popularity with substantial Rosetta Code coverage, while representing four major programming paradigms. This procedure favors widely used languages with enough common tasks for statistical comparison.

  • Rosetta Code ranks languages by the number of tasks with at least one solution, while TIOBE supplies an external popularity ranking.
  • Languages must rank within TIOBE’s top 50 and implement at least one third of Rosetta Code’s tasks.
  • Twenty-four languages satisfy both inclusion criteria before the combined ranking is applied.
  • The combined ranking uses TIOBE and Rosetta Code scores to reduce the influence of languages overrepresented in either ranking.
  • The study selects the top two languages from each procedural, object-oriented, functional, and scripting category.

D. Experimental setup

The experiment standardizes solution files before measuring source-code, compilation, and runtime properties across languages. Scripts and makefiles automate merging, patching, LOC measurement, compilation, and execution under consistent procedures.

  • D. Experimental setup: Solutions are merged into self-contained file collections and patched only for simple errors that prevent compilation or execution.The procedure avoids altering functional correctness or highly solution-specific features.
  • D. Experimental setup: Performance tasks use comparable inputs, with demanding task-specific inputs supplied for scalability analyses.Sorting solutions, for example, process large integer arrays generated with a fixed-seed linear congruential generator.
  • D. Experimental setup: LOC, compilation, and execution are measured for each solution using language-specific scripts with a uniform command-line interface.Compilation targets native code or bytecode, while execution uses the appropriate virtual machines for bytecode languages.
  • D. Experimental setup: Compiler versions and optimization settings are selected to provide stable standard-library support without overly aggressive trade-offs.The listed toolchains include gcc 4.6.3 with -O2, Mono 3.2.1, ghc 7.4.1 with -O2, and Java javac 1.8.0_11.
  • D. Experimental setup: The workflow is orchestrated across every language, task, and action, applying patches or makefiles before invoking the relevant measurement script.Makefiles define self-contained source collections when a task solution spans multiple files.

E. Experiments

Experiments were conducted on a fixed Ubuntu Linux system, and logged measurements were extracted for statistical analysis in R.

  • E. Experiments: The experiments ran on Ubuntu 12.04 LTS 64-bit Linux with a 2.40 GHz Intel Quad Core2 processor and 4 GB of RAM.Logged data were subsequently extracted and analyzed using R.

F. Statistical analysis

The analysis compares languages pairwise across normalized task-level metrics using non-parametric tests, effect sizes, and median ratios. These measures distinguish statistical significance from the magnitude and direction of differences.

  • F. Statistical analysis: Language pairs are compared across lines of code, executable size, CPU time, memory usage, page faults, and runtime failures.Task subsets require both languages to have solutions and may impose metric-specific criteria such as successful compilation.
  • F. Statistical analysis: Measurements are aggregated per task using minimum or mean and normalized by the smallest positive value across both languages, or by one when that minimum is zero.This normalization keeps task-level comparison vectors defined despite zero-valued measurements.
  • F. Statistical analysis: The Wilcoxon signed-rank test assesses whether paired language measurements differ, with p-values indicating evidence against chance differences.The study normally uses this non-parametric test for the aggregated task vectors.
  • F. Statistical analysis: Cohen’s d standardizes the mean difference, while the signed ratio compares the largest and smallest medians to quantify direction and scale.For metrics where smaller is better, a positive ratio sign indicates that language Y has the smaller average value than language X.
  • F. Statistical analysis: Differences are called significant when p < 0.01 and tendencies when 0.01 ≤ p < 0.05, while effect sizes range from vanishing below d = 0.05 to large at d ≥ 0.7.The intermediate thresholds define small and medium effects as 0.05 ≤ d < 0.3 and 0.3 ≤ d < 0.7.

G. Visualizations of language comparisons

Language-comparison graphs encode pairwise effect direction, magnitude, and significance through node placement and arrow styling. Their layout simplifies transitive relationships by omitting redundant edges.

  • G. Visualizations of language comparisons: Nodes represent programming languages, and horizontal distance is roughly proportional to the absolute signed-ratio value.Exact proportional placement is not always possible because pairwise language ordering need not be total.
  • G. Visualizations of language comparisons: Solid arrows indicate significant superiority, dashed arrows indicate a tendency, and arrow thickness represents effect size.An arrow from X to Y means Y is better than X for the displayed metric.
  • G. Visualizations of language comparisons: Edges are omitted when an ordered relationship is subsumed by an intermediate path such as X → W → Y.This improves visual layout without displaying every implied pairwise relationship.

III. RESULTS

The results compare eight languages across conciseness, executable size, performance, memory usage, and runtime failures. Functional and scripting languages are generally more concise, while language advantages vary by workload and compiled strongly typed languages tend to fail less at runtime.

  • RQ1. Which programming languages make for more concise code?: Functional and scripting languages provide significantly more concise code than procedural and object-oriented languages.Java programs are 2.2–2.9 times longer on average than programs in functional and scripting languages.
  • RQ2. Which programming languages compile into smaller executables?: Bytecode languages have significantly smaller executables than languages compiling to native machine code.Go and Haskell executables average 153.3 and 111.2 times the size of C executables, respectively, largely because they use static linking by default.
  • RQ3. Which programming languages have better running-time performance?: C remains fastest on large inputs, but performance differences shrink on moderate workloads, where several languages can compete.On modest-running-time tasks, C is not significantly faster than Go and Haskell, while Python is fastest among the remaining languages.
  • Memory usage: Functional languages trigger significantly more page faults than other languages, indicating more non-locality of reference.The measured impact may depend on the machine architecture.
  • Memory usage: C and Go use the least RAM, with Go averaging only 1.8 times C’s usage despite including garbage collection.Other languages use 2.5–14.2 times more memory on average than C or Go.
  • Runtime failures: Compiled strongly typed languages are significantly less prone to runtime failures than interpreted or weakly typed languages, with Go the least failure-prone.The authors associate this pattern with more errors being caught at compile time.

IV. DISCUSSION

The study’s results inform choices by developers, language designers, and educators, while highlighting trade-offs between expressiveness, performance, and compile-time error detection. Its findings also suggest that language design can explore combinations of features and that programming-language comparisons should be interpreted within their measured scope.

  • The study’s results can help developers, language designers, and educators make better-informed choices about language usage and design.
  • Functional and scripting languages’ conciseness suggests that features such as list comprehensions and dynamic typing provide greater expressiveness.The paper links concise code to potentially fewer bugs, based on the claim that bug density is largely constant across languages when other factors are equal.
  • Interpreted languages such as Ruby trade away a visible binary by performing optimizations at runtime, illustrating a space–time and compilation trade-off.The passage contrasts this with native compilation, which can support more aggressive compile-time optimizations.
  • C dominates raw speed and efficient memory usage, but regular-size workloads often allow nearly any language to remain competitive.The discussion argues that most applications do not require better performance than Python offers.
  • Go combines respectable running times with good memory usage and compile-time error detection, illustrating room for language designs tailored to specialized requirements.The study did not consider Go’s concurrency mechanisms, which the paper identifies as another feature relevant to some software development.
  • Evidence supports compile-time error detection from strong static typing, but programmer productivity remains a trade-off between early testing and catching more errors during compilation.The paper presents this as an open area for selectively combining features that enhance compilation or execution.

V. THREATS TO VALIDITY

The study addresses validity threats through well-defined measures, careful execution, and explicit boundaries on generalizability. The main concerns are the use of lines of code for conciseness, standard tool settings, Rosetta Code’s algorithmic and small-program bias, and language-paradigm classification.

  • Lines of code is a widely used conciseness measure, but correlating it with other proposed measures would be useful.
  • The study’s constructs are generally well defined, with a partial exception for failure proneness because that concept is multifaceted.The authors note consistency with related empirical work using other approaches.
  • Standard tools with default settings reduce language-familiarity bias but may limit the scope of the findings.Exploring the best optimizations possible in each language remains an identified direction for future work.
  • Rosetta Code programs may not represent real-world software because tasks favor algorithmic problems and solutions are small, although they can indicate core functionalities.The algorithmic focus also provides a solid basis for comparing performance on comparable implementations.
  • Classifying languages by paradigm is harder for multi-paradigm languages, but paradigms still significantly influence how programs are written in this study.The authors associate languages with paradigms based on their Rosetta Code programs.

VI. RELATED WORK

Prior language-comparison research spans controlled experiments, concurrency studies, repository mining, and surveys. The paper positions its large Rosetta Code sample as addressing the limited task counts and generalizability of many earlier comparisons.

  • Prior work reports shorter programs in scripting languages than in C, C++, or Java, while performance findings are more mixed.
  • The literature also includes concurrency comparisons and language-adoption surveys, highlighting that different metrics and extrinsic factors shape language evaluation.
  • Earlier comparative studies often used few tasks and solutions, whereas this study’s large sample is intended to improve statistical significance and generalizability.
  • Repository studies have reported modest advantages for strong typing and functional languages over procedural languages.

VII. CONCLUSIONS

The conclusions argue that programming-language debates should be grounded in evidence about how language features relate to program properties. The study’s large Rosetta Code dataset supports significant comparisons and offers a resource for future research.

  • Large-scale analysis of Rosetta Code enabled highly statistically significant results about frequently discussed programming-language features.
  • Rosetta Code can support future programming-language studies that corroborate or complement these findings.
  • Researchers can increase Rosetta Code’s research value by correcting errors and maintaining accessible, up-to-date statistics.

VIII-O Fault proneness

The appendix compares programming languages using pairwise statistical analyses of fault proneness and related measures. It reports that compiled and strongly typed languages generally show fewer runtime failures, with Go standing out most clearly.

  • VIII-O Fault proneness: The appendix presents pairwise comparisons across metrics including code length, binary size, runtime, memory use, page faults, timeouts, and runtime failures.Comparisons aggregate solution-level measurements per task and apply statistical tests to language pairs.
  • VIII-O Fault proneness: The normalized comparison graphs use common tasks for their horizontal distances, which can make them sensitive to noise when few tasks are shared by all languages.For performance and scalability, the paper advises that unnormalized graphs may be more indicative and treats table data as authoritative.

X. APPENDIX: PLOTS

The appendix contains plots comparing language pairs and all languages across conciseness, comments, binary size, and related measures. The supplied plot passages identify graph subjects and metrics but do not provide cell values or additional outcomes.

  • X. APPENDIX: PLOTS: The appendix plots lines of code using minimum and mean aggregates for individual language comparisons and for all languages.The figures cover both all tasks and tasks compiling successfully.
  • X. APPENDIX: PLOTS: The plots compare conciseness across C, C#, F#, Go, Haskell, Java, Python, and all languages.Figure labels distinguish language-specific comparisons and aggregate views.
  • X. APPENDIX: PLOTS: The appendix includes figures for comments per line of code, using minimum and all-task views.The supplied passages identify comment-density plots but do not state comparative values.
Loading 1409.0252v4…