PaperlayerView exact revision on arXiv

Source-linked AI summary

Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring

Indraneil Paul, Goran Glavaš, Iryna Gurevych

arXiv:2605.00754v4cs.SEcs.LG

TL;DR

Code reward modeling has largely centered on execution feedback and functional correctness, leaving multilingual, multi-criteria scoring underexplored. Themis builds multilingual evaluation and preference resources, trains Themis-RM models, and finds strong performance across criteria and languages, with current RMs weak beyond functional correctness.

  • Problem

    Code-generation post-training largely relies on execution feedback, leaving reward models’ multilingual scoring of diverse code-quality criteria underexplored.

  • Method

    Themis constructs an eight-language, five-criteria benchmark and preference datasets, then trains Themis-RM models through preference pre-training and modeling stages.

  • Results

    Current reward models are largely unusable beyond functional correctness, whereas Themis-RM performs strongly across axes and scales positively, with Themis-RM-32B reaching state-of-the-art.

  • Takeaways & Limitations

    The results support code reward models as a viable approach for multilingual, multi-criteria code preference scoring, including unseen criteria and programming languages.

  • Takeaways & Limitations

    Increasing the reward-variance constraint overly restricts variance and lowers accuracy, particularly for listwise re-ranking.

Abstract

from arXiv · show

Reward models (RMs) have become an indispensable fixture of the language model (LM) post-training playbook, enabling policy alignment and test-time scaling. Research on the application of RMs in code generation, however, has been comparatively sparse, with existing work largely focusing on execution feedback. This choice constrains post-training to optimizing functional correctness over self-contained executable code. In this work, we examine the training and evaluation of multilingual, multi-criteria code RMs. To this end, we first compile Themis-CodeRewardBench, a benchmark to evaluate code RMs across five preference dimensions (i.e., criteria) and eight programming languages, on which we profile 50+ code, math, and general-purpose RMs. Observing the limited proficiency of current RMs beyond scoring for functional correctness, we develop Themis-CodePreference, the largest open-source collection of code preferences to date (more than 350k preference pairs), and use it to train Themis-RM, a suite of multilingual code reward models for flexible multi-criteria scoring, ranging in size from 600M to 32B parameters. Our experiments and ablations demonstrate positive scaling trends, strong cross-lingual transfer when training on diverse preferences, and the importance of multi-criteria training for reliable code reward modeling.

1 Introduction

Code-generation post-training relies heavily on execution feedback, but this restricts optimization to self-contained executable code and functional correctness while introducing brittle, noisy, or hackable rewards. Themis addresses these gaps with multilingual, multi-criteria benchmarks, preference datasets, and flexible code reward models spanning five quality dimensions.

  • Limitations of execution feedback: Execution-based rewards dominate code-generation post-training but require procurable test cases and primarily support functional-correctness optimization.Synthetic test-case generation can rely on circular consensus and produce brittle or erroneous tests.
  • Limitations of execution feedback: Test-case feedback applies only to self-contained executable code, excluding code with external dependencies from post-training exposure.This restriction narrows the kinds of code that can be evaluated through execution feedback.
  • Alternative evaluation signals: Auxiliary runtime rewards and LM-based code evaluation remain problematic because they can be noisy, reward-hackable, or poorly calibrated in reference-free settings.Runtime signals may require tightly controlled emulated environments, while general-purpose LMs are poorly calibrated code evaluators.
  • Contributions: ≈8.9k preferences across eight programming languages and five scoring dimensions comprise Themis-CodeRewardBench, which is used to study 45 existing code RMs.The benchmark evaluates code reward models across diverse code preferences and scoring dimensions.
  • Contributions: 350k+ code-domain preferences support Themis-CodePreference, covering functional correctness, execution efficiency, memory efficiency, readability and maintainability, and security.Themis-GeneralPreference additionally contains 110k+ general-domain preferences.

2 Related Work

Prior work on code reward modeling has progressed from execution-based feedback toward pairwise preferences and extrinsic objectives beyond functional correctness. Related research also addresses multi-criteria reward modeling and benchmark coverage, motivating criteria-aware evaluation of code RMs.

  • Surrogate code execution modeling for alignment and quality estimation: Execution-based code quality methods require executable code and depend on user-provided or synthetic test cases, limiting their applicability.
  • Surrogate code execution modeling for alignment and quality estimation: Pairwise preference learning improves robustness to noisy signals, building on scalar reward models trained from execution or synthetic feedback.
  • Extrinsic rewards for code beyond functional correctness: Post-training has largely avoided non-functional code-quality objectives, despite security concerns from code smells and efficiency gaps relative to developer-written code.
  • Multi-criteria reward models: Naive multilingual, multi-task, and personalized multi-criteria reward modeling can suffer interference and distributional biases, while constituting principles enable scalar RMs to model multidimensional preferences with minimal interference.
  • Reward model evaluation benchmarks: Specialized reward-model benchmarks expose coverage and distributional shortcomings, but repurposed code-generation benchmarks risk evaluating code RMs on narrow settings.

3 Themis-CodeRewardBench: Multilingual And Multi-Criteria Code RM Evaluation

Themis-CodeRewardBench evaluates code reward models across five quality dimensions and eight programming languages, combining 13 existing and newly constructed preference datasets. It also broadens evaluation toward longer, more complex code responses and largely novel prompts.

  • Benchmark scope: Themis-CodeRewardBench comprises 13 code preference datasets spanning five quality dimensions and eight programming languages.The criteria are Functional Correctness, Execution Efficiency, Memory Efficiency, Readability And Maintainability, and Security Hardness.
  • Benchmark scope: The benchmark assembles preferences from human-annotated code changes and execution-validated metrics for human- or model-generated code.Its curated pairs cover functional correctness, execution efficiency, security, and code-quality-related preferences.
  • Dataset construction: The commit-mining workflow retrieves GitHub code changes, filters for non-reverted commits in merged pull requests, removes ambiguous changes through multi-model consensus, and generates diverse inverse instructions.The retained commits were authored between June 2019 and January 2021, while repository reputability was approximated using stars, contributors, and issue counts.
  • Benchmark scope: Themis-CodeRewardBench judges longer and more complex code responses over a largely novel prompt distribution than code subsets of popular existing RM benchmarks.This comparison is summarized in Figure 2.

4 Themis-RM: Training Multilingual Criteria-Following Code RMs

Themis-RM trains code-specialized reward models in two stages: general preference pre-training followed by multi-criteria code preference modeling. The suite combines diverse code preferences with flexible criteria prompts and regularized Bradley-Terry training.

  • Two-stage training: Themis-RM uses preference-model pre-training followed by preference modeling to establish diverse, code-specific reward modeling.The two stages are general preference pre-training and code preference modeling.
  • Preference Model Pre-Training (PT): 110k+ samples of natural-language and code preferences in Themis-GeneralPreference provide general notions such as relevance, helpfulness, and harmlessness.This dataset is used in the pre-training stage to support scoring beyond functional correctness and stylistic axes.
  • Preference Modeling (PM): 350k+ preferences in Themis-CodePreference cover five code-evaluation criteria across eight programming languages for the second training stage.The data combines GitHub commit preferences and synthetic prompts, with single-intent, single-file-changing commits reducing noise and reward-hacking risk.
  • Model suite and criteria control: Themis-RM spans 0.6B to 32B parameters and supports optional system prompts that specify custom evaluation criteria.Training samples may omit criteria, state all criteria generically, or specify criteria more directly.
  • Training objective: The models optimize a Bradley-Terry reward-modeling objective on preference tuples, with additional hidden-state regularization motivated by offline data and over-optimization risks.The objective is used in both training phases rather than introducing a bespoke pre-training objective.
  • Training setup: Themis-GeneralPreference is trained for two epochs and Themis-CodePreference for one epoch, using AdamW and cosine scheduling with 5% warmup in each phase.These choices define the reported implementation setup for the two-stage training procedure.

5 Experimental Research Questions And Results

The experiments find that Themis-RM substantially outperforms existing reward models on multilingual, multi-criteria code evaluation, while training ablations identify effective mechanisms for reducing cross-criteria interference. They also examine criteria prompting, model merging, cross-lingual behavior, and downstream listwise re-ranking and robustness.

  • RQ1: Existing RM competence: Themis-RM performs strongly across all evaluation axes and model sizes, while existing reward models are often unusable for non-functional criteria such as efficiency and security.Themis-RM-0.6B outscored multiple general-purpose reward models more than 100 times larger.
  • RQ1: Existing RM competence: Scalar reward modeling is well-suited to reference-free code evaluation, whereas generative and reasoning-enabled reward models often fail to use inference-time compute effectively.Generative models can suffer further when they self-propose evaluation rubrics, while text-based scoring has low resolution.
  • RQ2: Training ablations: Priming Themis-RM-8B with PT and adding auxiliary training losses improve performance across criteria, with PT helping models learn general notions of human preference.The ablations were designed to assess training-recipe components and criterion interactions while minimizing cross-criteria interference.
  • RQ2: Criteria prompting: Explicit criteria prompts markedly improve multi-dimensional preference modeling over a multi-task baseline without criteria-defining prompts.Multi-task learning remains viable without prompts, but criteria prompting has a markedly positive effect.
  • RQ2: Model merging: Merging criterion-specific models is ineffective: the merged reward model underperforms the multi-task baseline by over 10 points, likely because Bradley-Terry training leaves reward scales under-specified.Higher regularization values overly constrained reward variance and lowered accuracy, particularly for listwise re-ranking.
  • RQ3–RQ4: Cross-lingual transfer and downstream robustness: The study further evaluates multilingual interference and positive transfer across programming languages, alongside Themis-RM’s ability to retrieve correct solutions and correlate with ground truth in listwise re-ranking.The re-ranking evaluation uses Hits@10 and Rank Corr.@40 on 40 solutions per language for C++, Java, and Python.

6 Conclusion

Themis investigates model-based scoring of LM-generated code for multi-criteria alignment, quality estimation, and re-ranking. It introduces a multilingual preference benchmark and diverse multi-dimensional preference datasets for evaluating code-reward modeling.

  • Themis investigates model-based scoring of LM-generated code for multi-criteria alignment, quality estimation, and re-ranking.
  • Themis-CodeRewardBench evaluates reward models across eight typologically diverse programming languages and functional and non-functional criteria.
  • Themis constructs Themis-GeneralPreference and Themis-CodePreference as diverse multi-dimensional preference datasets.

A Architecture And Training Details

Table 5 summarizes the Themis-RM suite’s architectural heritage and training attributes across preference-model pre-training and preference-modeling stages. It directs readers to Section 4 for the training objective and Appendix B for training-data composition.

  • Architecture And Training Details: Table 5 covers the Themis-RM suite’s architectural heritage and training attributes.The table spans both preference-model pre-training and preference modeling.
  • Architecture And Training Details: The table covers preference-model pre-training and preference modeling stages.These are the two training stages identified in the table caption.
  • Architecture And Training Details: The training objective is described in Section 4, while training-data composition is detailed in Appendix B.The caption points to these sections for further methodological details.

B Data Collection · B.1 BigQuery SQL Query For Commit Mining

The data collection section uses a streamlined GoogleSQL query to mine single-file-changing commits from openly licensed GitHub repositories. The query joins commit, language, and license data, then filters by supported languages, licenses, and commit characteristics.

  • B.1 BigQuery SQL Query For Commit Mining: The query unnests repository names from the commits table and joins them with language and license tables.It retrieves commit metadata, file paths, licenses, and committer timestamps while linking repositories to language records.
  • B.1 BigQuery SQL Query For Commit Mining: The mining filters retain repositories using specified open-source licenses and code in Python, Java, JavaScript, C, C#, C++, TypeScript, Go, or Ruby.The license list includes MIT, Artistic-2.0, ISC, CC0-1.0, EPL-1.0, MPL-2.0, Unlicense, Apache-2.0, BSD-3-Clause, AGPL-3.0, LGPL-2.1, and BSD-2-Clause.
  • B.1 BigQuery SQL Query For Commit Mining: The query requires commit messages longer than 10 and shorter than 15000 characters while excluding common boilerplate messages and push-related patterns.It also excludes merge messages and several routine update or initialization messages.
  • B.1 BigQuery SQL Query For Commit Mining: The query keeps only modifications where old and new file paths are identical and non-null, grouping results by commit and file metadata.This excludes commits involving file renames or missing paths.
  • B.1 BigQuery SQL Query For Commit Mining: The final grouping requires exactly one distinct old file path per commit.This HAVING clause operationalizes the single-file-change constraint.
  • B Data Collection: The query mines single-file-changing commits from openly licensed repositories using the BigQuery GitHub dataset pipeline.The query is identified as modified from the BigQuery GitHub dataset query pipeline presented in Muennighoff et al. (2024).

B.2 Search Terms For Aspect-Specific Commit Classifier Training

Table 6 documents the per-criteria search terms used to recall positive examples for training the commit classifier, as part of the complete commit preference mining pipeline.

  • Search Terms For Aspect-Specific Commit Classifier Training: Table 6 lists per-criteria search terms for recalling positives during commit classifier training.The complete commit preference mining pipeline is described in Section 3.

B.3 Themis-RM Training Data Mixture And Filtering … D Detailed Results

Themis-GeneralPreference combines more than 110k natural-language preference samples, while Themis-CodePreference provides more than 350k code-focused pairs spanning functional and non-functional criteria. The collection combines mined, synthetic, and pre-existing datasets, then applies length, language, perplexity, deduplication, and contamination filtering.

  • B.3 Themis-RM Training Data Mixture And Filtering: More than 110k samples form Themis-GeneralPreference, combining helpfulness, harmlessness, and code-retrieval relevance preferences to teach general human-preference nuances.The collection is curated from pre-existing preference datasets and a small number of code-retrieval relevance preferences.
  • 2. Skywork-Preference: Themis includes preference sources from Skywork, Tulu, StackExchange, LMArena, Prometheus, HelpSteer3, Argilla-DPO, and Truthy-DPO.These sources contribute instruction-following, technical-answer, human-voting, helpfulness, harmlessness, stylistic, and truthfulness preferences.
  • B.3 Themis-RM Training Data Mixture And Filtering: More than 350k samples form Themis-CodePreference, centered on GitHub commits, synthetically bugged instruction-tuning data, and augmented code-preference and retrieval datasets.The collection targets functional and non-functional code-scoring axes.
  • 1. Commit-Preference: GitHub commit preferences retain non-reverted single-file changes from subsequently merged pull requests, with strength validated by consensus among multiple frontier LMs and single-intent selection.The commit data can target improvements such as runtime efficiency.
  • 2. CodeR-Pile: CodeR-Pile pairs use positive-document code as chosen responses and Zipfian samples from top-10 mined hard negatives as rejected responses.Themis uses code augmentation, exemplar, integration, refinement, simplification, pseudo-code, tutorial, and web query subsets.
  • 3. Bugged-Instruct: Synthetic bugged-instruction pairs introduce algorithmic or syntactic bugs into diverse instruction-tuning data, while generated bug-laden solutions create functional-correctness preferences.The generation prompt excludes new features, security vulnerabilities, memory leaks, and other non-functional bugs.
  • 4. ProSec: Additional code datasets target vulnerability fixes, efficiency, repair, runtime, correctness, and security, including ProSec, Venus, CodeNet, RunBugRun, ECCO, CodeScaleR, Pie4Perf, and CyberNative-DPO.Venus and CodeNet select pairs where the chosen completion is at least 5x as efficient as the rejected one; Pie4Perf uses successful submissions with highly divergent runtimes.
  • C Data Acquisition And Filtering Prompts: Filtering limits Themis-GeneralPreference and Themis-CodePreference samples to 2560 and 4096 tokens, removes shallow code, non-English or high-perplexity prompts, near-duplicates, and prompts overlapping evaluation benchmarks.The procedure uses GlotLID, KenLM with a perplexity threshold of 1200, MinHash with shingle size 20 and similarity threshold 0.75, and 13-gram overlap checks.

D.1 Functional Correctness (FC) Dataset-Level Accuracy

Table 7 compares extant reward models and Themis-RM on dataset-level preference accuracy for the Functional Correctness split. Extant models show marked performance drops on partially correct solutions and open-domain GitHub commit preferences.

  • Functional Correctness: Table 7 evaluates extant RMs and Themis-RM on dataset-level preference accuracy for the Functional Correctness split.The table focuses on the FC split of Themis-CodeRewardBench.
  • Functional Correctness: Extant RMs show a marked performance drop when judging partially correct solutions in MBPP+Fix (Hard).The caption identifies this as a specific weakness of existing models.
  • Functional Correctness: Extant RMs also exhibit a marked performance drop on open-domain GitHub commit preferences in CommitPref.The caption highlights CommitPref as another challenging evaluation setting.

D.2 Execution Efficiency (EE) And Memory Efficiency (ME) Dataset-Level Accuracy

The section evaluates dataset-level preference accuracy on the Execution Efficiency and Memory Efficiency splits, showing that most extant scalar RMs perform near randomly on non-functional criteria. It also highlights that generative RMs’ low-resolution scoring can make them unusable in these settings.

  • D.2 Execution Efficiency (EE) And Memory Efficiency (ME) Dataset-Level Accuracy: Themis-CodeRewardBench evaluates RMs on Execution Efficiency and Memory Efficiency dataset-level preference accuracy.The split-level averages and comparative discussion are presented in Section 5.1.
  • D.2 Execution Efficiency (EE) And Memory Efficiency (ME) Dataset-Level Accuracy: Most extant scalar RMs degenerate to near-random performance on the non-functional Execution Efficiency and Memory Efficiency criteria.
  • D.2 Execution Efficiency (EE) And Memory Efficiency (ME) Dataset-Level Accuracy: Generative RMs’ low-resolution scoring can render them unusable for Execution Efficiency and Memory Efficiency preference evaluation.

D.3 Readability And Maintainability (R&M) And Security Hardness (SH) Dataset-Level Accuracy · D.4 General-Domain Reward Modeling Performance · E Project Themis Artifacts

The section evaluates Themis-RM on non-functional code criteria and external general-domain reward-modeling benchmarks, while listing the publicly released Project Themis artifacts. It emphasizes extant models’ near-random non-functional scoring and Themis-RM’s broader evaluation scope.

  • D.3 Readability And Maintainability (R&M) And Security Hardness (SH) Dataset-Level Accuracy: Table 9 reports dataset-level preference accuracy on the Readability And Maintainability and Security Hardness splits of Themis-CodeRewardBench.The table compares extant reward models with the Themis-RM suite.
  • D.3 Readability And Maintainability (R&M) And Security Hardness (SH) Dataset-Level Accuracy: Most extant scalar reward models degenerate to near-random performance on non-functional criteria.This observation concerns the Readability And Maintainability and Security Hardness evaluations.
  • D.3 Readability And Maintainability (R&M) And Security Hardness (SH) Dataset-Level Accuracy: Low-resolution scoring by generative reward models can render them unusable for non-functional code-criteria evaluation.The passage identifies scoring resolution as the relevant limitation.
  • D.3 Readability And Maintainability (R&M) And Security Hardness (SH) Dataset-Level Accuracy: The section includes auxiliary training objectives as a component of the evaluation and training discussion.The supplied passage provides the subsection heading but no further findings.
  • D.4 General-Domain Reward Modeling Performance: Table 10 evaluates the Themis-RM suite on RewardBench V1, RewardBench V2, and JudgeBench for external general-domain reward modeling.The evaluation benchmarks against selected extant reward models with high Themis-CodeRewardBench scores.
  • E Project Themis Artifacts: Table 11 provides the complete list of publicly released artifacts accompanying Project Themis.The supplied passage does not enumerate the individual artifacts.
Loading 2605.00754v4…