PaperlayerView exact revision on arXiv

Source-linked AI summary

DivAlgo: A systematic evaluation of algorithmic diversity for code generation models

Anubhav Jangra, Smaranda Muresan

arXiv:2610.19157v1cs.SEcs.CL

TL;DR

The paper examines whether code-generation models can produce algorithmically diverse solutions comparable to human solutions, a capability not well established despite widespread use. It introduces DivAlgo and a lightweight evaluation metric, and finds that model-generated diversity remains below the human level even with diverse generation strategies.

  • Problem

    How algorithmically diverse are model-generated solutions compared with human solutions, across models, and relative to human strategies?

    Source: §1 · p. 1

  • Method

    The paper builds DivAlgo and a lightweight similarity metric, then evaluates 24 code-generation models across uncontrolled, diverse-generation, and human-strategy steering settings.

    Source: §1 · pp. 1–2

  • Results

    Uncontrolled sampling by the best model reached 31% of human diversity, while diverse strategies reached up to 70.5% and steering still failed to reproduce overall human diversity.

    Source: §1 · p. 2, §5.1 · p. 7

  • Takeaways & Limitations

    Model-generated code can offer less algorithmic variety than human solutions, and overexposure to it may narrow the algorithmic vocabulary programmers encounter and learn.

    Source: §1 · p. 1, §6 · p. 9

    Exact human-diversity capture is defined as matching human solutions’ algorithmic-label set, a stricter condition than having more than one algorithm.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Code generation models are actively being used in professional and educational settings. However, the competence of large language models (LLMs) to generate algorithmically diverse code remains largely unknown. We propose the DivAlgo benchmark, a large-scale dataset constituting algorithmically diverse solutions to programming problems. We also introduce a lightweight similarity metric to enable large-scale evaluation. Our experiments across 24 modern code generation models found that uncontrolled model-generated code is significantly less diverse than human solutions (capturing ~31% of human diversity), and even the best diverse code generation strategy could achieve only ~70% of human diversity. Moreover, while it was possible to steer the models to generate solutions based on human strategies, they were unable to reproduce the diversity of the human solutions. These findings indicate the limitations of code generation models in capturing the algorithmic diversity humans exhibit, and that overexposure to model-generated code can narrow programmers' algorithmic vocabulary.

1 INTRODUCTION

The paper asks whether code generation models reproduce the algorithmic diversity of human solutions, an underexamined capability as LLM code use grows. It introduces DivAlgo and a lightweight metric, then reports substantial gaps between model-generated and human diversity.

1 INTRODUCTION

  • The paper frames three questions: model diversity versus humans, overlap across models, and whether models can follow human strategies.These correspond to within-model, across-model, and human-relative notions of algorithmic diversity.Source: §1 · p. 1
  • DivAlgo captures overlap at the data-structure, technique, and programming-paradigm levels, using distinct human solutions and a pool for population-level analysis.The benchmark was built through cascaded filtering with code-similarity metrics and LLM-judge ensembles.Source: §1 · pp. 1–2
  • 31%: the best model under uncontrolled sampling achieved this share of human diversity; diverse generation strategies reached up to 70.5%.The 70.5% result used an equal number of samples.Source: §1 · p. 2
  • Pooling unconditioned outputs from 24 models was less effective than one model with an appropriate prompt, and strategy steering still did not reproduce overall human diversity.Source: §1 · p. 2

2 RELATED WORK

Prior work has studied code diversity through varied metrics and generation strategies, but earlier investigations were limited in scale, task scope, or granularity. This paper positions DivAlgo as a large-scale, fine-grained evaluation against human solutions.

2 RELATED WORK

  • Earlier diversity research explored prompting, classifiers, clustering, incidental prompt cues, and execution behavior, with different constraints on tasks or methods.The cited studies include very simple problems, 11-task coverage, or approaches dependent on tractable intermediate execution structure.Source: §2 · p. 3
  • Prior studies examined algorithmic diversity on very simple problems, at small scale, for particular problem classes, or at coarse granularity.Source: §2 · p. 3
  • DivAlgo extends this work with a large-scale benchmark spanning varied problem difficulty and fine-grained control of algorithmic overlap, plus a lightweight metric for comparing generated code with human solutions.Source: §2 · p. 3

3 DI VAL G O BENCHMARK CONSTRUCTION

DivAlgo is built by filtering human C++ submissions to identify algorithmically distinct solutions, then assigning interpretable labels across algorithmic dimensions. Its construction combines lightweight similarity measures with LLM judgments.

3 DI VAL G O BENCHMARK CONSTRUCTION

  • 9,193 instances remained after filtering CodeContests+ for C++ solutions, sufficient submissions, problem statements, and duplicates.The starting collection contained 11,690 problems and 4.36 million correct submissions; the final set averaged 176.4 correct C++ submissions per instance.Source: §3 · p. 3
  • A two-stage selection process uses CodeXEmbed and program-dependency-graph similarity to shortlist diverse solutions, then LLM judges to select final dissimilar pairs.The lightweight measures align above chance with aggregate LLM judgments, while pairwise LLM evaluation is too costly at the dataset scale.Source: §3 · pp. 3–4
  • Medoid-seeded farthest-first traversal selected up to 10 PDG-space, 6 CodeXEmbed-space, and 5 random-control solutions, yielding a median of 19 per instance.Selection stopped when another pick reduced normalized covering radius by less than 2 percentage points.Source: §3 · p. 4
  • For LLM filtering, the authors ranked disjoint solution pairs by mean CodeXEmbed and PDG similarity, used Gemini-3.1-flash-lite and gpt-5.4-mini, and used Claude-Haiku-4.5 to break ties.Source: §3 · p. 4
  • The three-layer program ontology labels paradigms, techniques, and data structures, spanning 13 paradigms, 64 techniques, and 22 data structures.Solutions and pairs are categorized by label overlap, from different paradigms to identical ontology labels.Source: §3 · p. 4

4 EXPERIMENTAL SETTINGS

The experimental setup compares model-generated solution populations with human solutions, and separately tests coverage of representative human approaches. It defines population diversity and algorithm coverage, then specifies the metrics, models, and generation strategies used.

4.1 PROBLEM DEFINITION

  • A model is diverse relative to a population of human solutions when its solution set has more than one underlying algorithm; exact human-diversity capture requires matching the human set’s algorithmic labels.The exact-match criterion is strictly stronger than the population-diversity condition.Source: §4.1 · p. 5

4.2 EVALUATION METRICS

  • VendiCodeX-SFT estimates population diversity by embedding solutions with a distilled CodeXEmbed model and applying the Vendi score to their cosine Gram matrix.The trained similarity model achieved 0.727 AUC against a six-LLM ensemble, versus 0.686 for the base model; score outputs range from 1 for identical solutions to n for mutually dissimilar ones.Source: §4.2 · p. 5
  • AlgoCov measures whether a generated solution set overlaps the ontology labels of representative human solutions A and B, using the maximum coverage across them.Source: §4.2 · p. 5

4.3 CODE GENERATION MODELS AND GENERATION STRATEGIES

  • The evaluation covers 24 code-generation models, controlled diversity strategies, and steering toward human reference strategies; the uncontrolled comparison uses 50 solutions per model.Diverse strategies include temperature variation, verbalized sampling, and iterative constrained generation; steering provides ontology labels for references A and B.Source: §4.3 · pp. 5–6, Figure 2 · p. 6

5 RESULTS AND DISCUSSION

The strongest diverse-generation strategy reaches 70.5% of human diversity at a matched sample size, while unconditioned model outputs are less diverse than human solutions. Heterogeneous model ensembles and strategy steering improve diversity or coverage, but neither reproduces the full range of human solutions.

5.1 HOW DIVERSE ARE LLM-GENERATED SOLUTION?

  • Unconditioned sampling yielded a Vendi score of 4.84 for Gemini-flash-3.8, versus 14.27 for human solutions, despite sampling almost twice as many solutions as the human pool.Across the three tested diverse-generation strategies, verbalized sampling and iterative generation were more effective than temperature increases, though their higher scores partly reflected larger sample counts.Source: §5.1 · p. 6
  • 70.5% of the human Vendi score was reached by Gemma-4-31b with iterative generation at 25 matched solutions, still below human diversity.The comparison covers 412 problems with at least 25 solutions in both populations.Source: Figure 3 · p. 7, §5.1 · p. 7

5.2 DO MODELS CONVERGE ON THE SAME ALGORITHMS?

  • LLM preferences aligned with the more popular human reference approach (mean Pearson’s r = +0.695), and heterogeneous model unions raised diversity from 4.32 to 6.69.Combining diverse-generation outputs raised diversity from 8.83 to 9.53, a +0.70 gain over the best single-model approach despite a threefold increase in generation costs.Source: §5.2 · pp. 7–8

5.3 CAN LLM-GENERATED SOLUTIONS FOLLOW HUMAN SOLUTION STRATEGIES?

  • Iterative generation matched the oracle-style steer (A+B) in two of three ontology granularities, suggesting some algorithmic strategies are latent rather than absent.The comparison uses 10 solutions per strategy over 274 strong, 104 medium, and 106 weak cases.Source: §5.3 · pp. 8–9, Table 2 · p. 9
  • Human solutions exceeded every generation strategy across all three granularities, with weak-tier coverage of 79.6% versus 23.3% for models.The weak tier differs only in data structure, and AlgoCov here assumes two representative solutions rather than rewarding other valid strategies.Source: §5.3 · p. 9
  • Non-default strategy steering raised paradigm overlap from 15.4% to 34.7%, while default-strategy steering increased it from 91.5% to 93.8%.The non-default gain was consistent across ontology granularities (+19.8, +16.6, +19.2 pp).Source: §5.3 · p. 9

6 CONCLUSION

The paper presents DivAlgo and a lightweight embedding model to measure algorithmic diversity at scale. Its experiments find that model outputs underrepresent human variety, while diverse generation and steering improve diversity or coverage without reaching the human ceiling.

6 CONCLUSION

  • Unconditioned model sampling recovers only a fraction of human algorithmic variety, and models tend toward the more popular human approach.Diverse prompting closes some of the gap more effectively than ensembling heterogeneous models, but steering still falls below the human ceiling across all three granularities.Source: §6 · p. 9
  • DivAlgo provides a fine-grained benchmark, paired with a distilled lightweight embedding model and LLM-ensemble annotations for measuring population-level diversity at scale.Source: §6 · p. 9
  • The paper concludes that algorithmic diversity is latent in current LLMs, but they do not capture the full variety of human solutions.Source: §6 · p. 9

A.1 PRELIMINARY INVESTIGATION OF ALGORITHMIC SIMILARITY METRICS

The preliminary investigation compares LLM judges, CodeXEmbed, and PDG as measures of algorithmic similarity. LLM judges align strongly in ranking solution pairs, though they differ on similarity thresholds.

A.1 PRELIMINARY INVESTIGATION OF ALGORITHMIC SIMILARITY METRICS

  • Human annotation was difficult to scale: correlation fell from κ = 0.58 to 0.25 as edit distance increased, while CodeContests+ C++ pairs averaged ≥1,800 edits apart.The authors cite this as motivation for automatic similarity metrics.Source: §A.1 · p. 13
  • The study compared PDG, CodeXEmbed-2b, and six proprietary LLM judges, which classified pairs from identical to very distinct.PDGs were post-hoc edited to remove several stylistic attributes.Source: §A.1 · p. 13
  • LLM judges had Spearman’s ρ of ∼0.46−0.74 and AUC of ∼0.85; CodeXEmbed reached AUC 0.709 and PDG 0.657.Figure 5 compares these metrics on 100 uniformly sampled solution pairs.Source: Figure 5 · p. 13, §A.1 · p. 14
  • Figure 6 reports leave-one-out agreement among six proprietary LLMs on three-level ontology labeling.The caption does not provide agreement values.Source: Figure 6 · p. 14

A.2 SELECTING THE PROGRAM CLASSIFICATION LLM

The authors use a three-level algorithm ontology to label solutions, selecting Gemini-3.1-flash-lite for its agreement and cost.

A.2 SELECTING THE PROGRAM CLASSIFICATION LLM

  • The ontology labels programming paradigm, technique, and data structure; Gemini-3.1-flash-lite had the highest agreement with five other LLMs and was cheapest.It labeled 15,246 C++ solutions using predefined ontology labels; labels beyond the ontology were discarded.Source: §A.2 · p. 14
  • The three-level ontology contains 13 paradigms, 64 techniques, and 22 data structures.Source: §A.2 · p. 14
  • The authors selected Gemini-3.1-flash-lite because it had the highest agreement with the other five LLMs and was the cheapest.Source: §A.2 · p. 14

B CALIBRATING CO D EXEM BED AS AN EVALUATION METRIC

The section evaluates similarity metrics against six-LLM labels and calibrates embedding models on labeled pairs. CodeX-SFT performs best among the evaluated metrics and separates human-curated pair types better than the base CodeXEmbed model.

B CALIBRATING CO D EXEM BED AS AN EVALUATION METRIC

  • The evaluation subset contained 500 random pairs and 200 filtration-stage pairs, with six proprietary LLMs supplying majority-voted four-point ground-truth labels.The random pairs reflected the natural similarity distribution, while filtration-stage pairs sampled difficult near-threshold cases.Source: B CALIBRATING CO D EXEM BED AS AN… · p. 14
  • The authors evaluated CodeXEmbed, jina, PDG, and Llama-3.1-8B, then fine-tuned both embedding models on LLM-labeled pairs, excluding the 700 evaluation instances.The training data comprised 12,649 training and 1,546 validation solution pairs.Source: B CALIBRATING CO D EXEM BED AS AN… · p. 14
  • CodeX-SFT was trained for two epochs with mean-squared error on four ordinal labels, using a 10^-5 learning rate, 5% warmup, and batch size 64.The setup also used 8-bit AdamW.Source: B CALIBRATING CO D EXEM BED AS AN… · p. 15
  • CodeX-SFT was the best-performing metric; PDG showed the weakest alignment with the six-judge ensemble, with ROC 0.577.All evaluated metrics performed better than chance, and Figure 7 reports evaluation on 700 instances against the aggregated majority ground truth.Source: B CALIBRATING CO D EXEM BED AS AN… · p. 15, Figure 7 · p. 15
  • Fine-tuning improved both embedding models, and CodeX-SFT was chosen to compute Vendi Scores across pairwise similarities.PDG and CodeX-SFT self-diversity scores were also reported as complementary population measures.Source: B CALIBRATING CO D EXEM BED AS AN… · p. 15
  • On a small hand-curated evaluation dataset of Python solution pairs, CodeX-SFT separated the two types by 0.23, compared with a 0.08 difference for baseline CodeXEmbed.The embedding models were trained on C++ solutions.Source: B CALIBRATING CO D EXEM BED AS AN… · p. 15, Table 3 · p. 16
  • BehaveSim remained ahead by a 0.54 difference, but it requires tracing intermediate states and therefore applies only to problems with traceable steps.The authors note that this does not cover the spectrum of competitive-programming problems they investigate.Source: B CALIBRATING CO D EXEM BED AS AN… · pp. 15–16

C HYPERPARAMETERS AND REPRODUCIBILITY DETAILS

The section specifies sampling, execution, and hardware settings for reproducibility.

C HYPERPARAMETERS AND REPRODUCIBILITY DETAILS

D RESULTS (EXTENDED)

The extended results section identifies tables for uncontrolled zero-shot generation across 24 LLMs and controlled diverse generation across three open-weight models. The prompts appendix documents evaluation, ontology-labeling, and generation prompts, alongside illustrative examples of diversity levels.

D RESULTS (EXTENDED)

  • The extended results cover uncontrolled zero-shot generation for 24 LLMs and controlled diverse generation using three strategies across three open-weight models.Table 4 reports the uncontrolled results; Table 5 compares temperature variation, verbalized sampling, and iterative constrained generation.Source: D RESULTS (EXTENDED) · p. 16, Table 4 · p. 23, Table 5 · p. 23
  • Figure 8 presents the distribution of ontology labels for the two representative C++ solutions associated with each programming problem.Source: Figure 8 · p. 17

E PROMPTS

  • The appendix supplies prompts for four-way ordinal similarity judgments, ontology labeling, and i.i.d., verbalized, iterative, and strategy-steered generation.The verbalized and iterative approaches each produce 50 solutions per problem for comparison with the i.i.d. setting.Source: E PROMPTS · p. 17, Figure 13 · p. 20, Figure 14 · p. 21
  • Illustrations distinguish strong diversity across programming paradigms, medium diversity across techniques, weak diversity across data structures, and identical diversity across all three dimensions.The examples show code-similarity metrics and pairwise LLM judgments alongside the solutions.Source: Figure 9 · p. 18, Figure 10 · p. 18, Figure 11 · p. 19

D RESULTS (EXTENDED)

  • Table 4 reports pass rate, Pass@N, Pass≥2@N, generation statistics, self-diversity, and Vendi scores against human reference diversity.Its caption specifies PDG and CodeX-SFT distances, matched and all-solutions Vendi scores, and results over 1,000 test problems.Source: Table 4 · p. 23
  • Table 5 compares temperature variation, verbalized sampling, and iterative generation using the same 50 generated solutions per condition.Verbalized sampling uses 10 calls of five solutions per problem; iterative generation uses 10 chains of five constrained generations.Source: Table 5 · p. 23

Related papers

Loading 2610.19157v1…