Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring

Execution-feedback limitations

• Execution feedback is brittle, requires procured or synthetically generated tests, and restricts post-training to self-contained executable code, especially excluding much Rust, Go, and Swift code.Synthetic tests can be erroneous, while repository-level generation remains difficult and conventional verification can miss interface-contract violations.Source: §1 · pp. 1–2

Motivation for code reward models

• Robust code RMs can complement noisy runtime proxies by scoring non-functional quality across languages and underspecified scenarios, whereas general-purpose LM judges and surrogate executors remain poorly calibrated or narrow.Existing surrogate efforts primarily target self-contained Python, and low-resource programming languages cause notable performance drops.Source: §1 · p. 2

Benchmark

• Themis-CodeRewardBench contains ≈8.9k diverse code preferences spanning eight programming languages and five scoring dimensions, enabling failure analysis of 45 existing code RMs.The benchmark addresses limited prior evaluations dominated by valid-buggy Python pairs and functional correctness.Source: §1 · p. 2

Preference datasets

• Themis-GeneralPreference and Themis-CodePreference provide 110k+ general-domain and 350k+ code-domain preferences covering correctness, efficiency, readability, maintainability, and security.The code dataset spans five quality dimensions: functional correctness, execution efficiency, memory efficiency, readability and maintainability, and security.Source: §1 · p. 2

Themis-RM

• Themis-RM is an open-source multilingual code-RM suite ranging from 0.6B to 32B parameters that flexibly scores arbitrary user-defined subsets of five code-quality criteria.The models are trained using the paper’s general-domain and code-domain preference datasets.Source: §1 · pp. 2–3

Research agenda

• The study examines scaling trends, unseen criteria, and out-of-distribution programming languages, presenting the first test of code RMs for open-domain code under adversarial settings.Its research questions cover diverse real-world preferences, cross-criteria interference, transfer to unseen languages, adversarial robustness, and listwise reranking.Source: §1 · p. 3

Surrogate code execution modeling

• Surrogate code executors reduce executability requirements, but early quality estimators can be poorly calibrated and generation-based surrogates may have low scoring fidelity for fine-grained preferences.Pairwise and textual-feedback approaches improve robustness or semantic reasoning but can yield inconsistent preferences or insufficient fidelity for listwise ranking.Source: §2 · p. 3

Extrinsic rewards beyond functional correctness

• Non-functional code rewards increasingly target security, efficiency, memory, maintainability, and specification following, but existing methods often require executable code, tests, references, or specialized feedback.Themis-RM analyses indicate that non-functional requirements need not compromise functional correctness and can be modeled across multiple axes through training design.Source: §2 · pp. 3–4

Multi-criteria reward models

• Naive multilingual, multi-task, and personalized multi-criteria reward-model training risks interference and distributional bias, whereas textual criteria and constituting principles can help scalar models disentangle preferences.Themis-RM evaluations report effective multi-dimensional preference modeling with minimal interference.Source: §2 · p. 4

Reward model evaluation benchmarks

• Existing reward-model benchmarks can miss coverage gaps, distributional bias, contamination, and over-optimization failures because they often emphasize general preferences or narrow code distributions.Themis-CodeRewardBench provides diverse, multilingual, multi-criteria evaluation, uncovering gaps in current code RMs and testing robustness beyond preference accuracy.Source: §2 · pp. 4–5

Benchmark scope

• The benchmark covers five criteria—FC, EE, ME, R&M, and SH—across C, C#, C++, Go, Java, JavaScript, Python, and Ruby, using preference accuracy.It comprises 13 pre-existing and newly constructed code-preference datasets.Source: §3 · p. 5

Dataset curation

• Existing datasets provide preferences spanning functional correctness, efficiency, security, and maintainability, but leave sparse coverage across language–criterion combinations, especially for non-functional criteria.The collection includes human-annotated changes and execution-validated metrics, with inverse instructions synthetically generated when needed.Source: §3 · p. 5

Commit mining

• The commit-mining pipeline filters GitHub changes through criterion-specific classifiers, merged pull requests, non-reversion checks, consensus review, and synthetic inverse-instruction generation.It targets openly licensed, single-file changes and retains commits from June 2019 through January 2021 in reputable repositories.Source: §3 · pp. 6–7

Benchmark comparison

• Compared with existing code reward-model benchmarks, Themis-CodeRewardBench evaluates longer, more complex code responses over a largely novel prompt distribution.The benchmark’s complete dataset, criterion, and language composition is reported in Table 1.Source: Table 1 · p. 6, Figure 2 · p. 7, §3 · p. 7

4 Themis-RM: Training Multilingual Criteria-Following Code RMs

• Themis-RM trains in two stages: preference-model pre-training followed by preference modeling, establishing the suite’s overall training procedure.The section details the training data and exact setup for both stages.Source: §4 · p. 7

4.1 Training Data Mixture

• Themis-GeneralPreference provides 110k+ natural-language and code preference samples for pre-training common human-inspired notions of preference evaluation, including relevance, helpfulness, and harmlessness.Themis-CodePreference comprises 350k+ preferences across five code-evaluation dimensions and eight programming languages; its commits were pushed before March 2019 and came from repositories disjoint from benchmark commit data.Source: §4.1 · p. 7
• Single-intent, single-file commits reduce training noise and reward hacking while enabling criteria-specific preferences for otherwise rare non-functional code evaluations.The training mixture also includes functional, wall-time, memory-usage, and synthetic bug-injection preferences from code datasets and model-generated responses.Source: §4.1 · p. 8

4.2 Model Training

• Themis-RM comprises Qwen3-derived dense models from 0.6B to 32B parameters and uses optional criteria prompts to support flexible multi-criteria scoring.Training samples include no criterion prompt, a generic all-criteria prompt, or an instruction specifying exactly one criterion.Source: §4.2 · p. 8
• The models optimize Bradley-Terry preference modeling with conditional language-modeling and reward-magnitude regularizers, using the same objective across both training phases.Themis-GeneralPreference is used for two epochs in pre-training and Themis-CodePreference for one epoch in preference modeling, with AdamW and cosine scheduling with 5% warmup.Source: §4.2 · p. 8

5.2 RQ2: Training Choices To Minimize Multi-Criteria Interference In Code RMs

• Priming with PT, auxiliary losses, and criterion-specific prompts improve multi-criteria scoring, whereas merging criterion-specific models underperforms the multi-task baseline by over 10 points.Single-criterion training transfers across criteria, especially from functional correctness to non-functional requirements, but no criterion-specific model matches the full Themis-RM; higher regularization also lowers accuracy, particularly for listwise re-ranking.Source: §5.2 · pp. 9, 11

5.1 RQ1: Judging Reward Models On Multilingual Multi-Criteria Code Preferences

• Themis-RM-32B establishes state-of-the-art performance across all evaluated criteria, while Themis-RM-0.6B outscales much larger general-purpose models and existing RMs remain weak beyond functional correctness.Table 2 compares extant RMs with Themis-RM across criteria; existing models often score non-functional preferences near randomly, and dataset analyses expose weaknesses on commit preferences and partial correctness.Source: §5.1 · p. 9, Table 2 · p. 10

5.3 RQ3: Cross-Lingual Transfer In Code RMs

• Training on all programming languages yields the best cross-lingual performance, while Python and Java training transfer preferentially to dynamically and statically typed languages, respectively.The four controlled models show small performance differences across languages, indicating stable multilingual reward modeling from diverse, multi-criteria preferences.Source: Table 3 · p. 11, §5.3 · p. 12

5.4 RQ4: Downstream Robustness Evaluation

• Themis-RM achieves state-of-the-art adversarial robustness and matches the strongest correctness-specialized RMs on listwise re-ranking, with better downstream robustness scaling than pairwise accuracy.Evaluation uses Hits@10 and Rank Corr.@40 on generated C++, Java, and Python solutions, plus adversarial accuracy on Aletheia-Adv.Source: Table 4 · p. 12, §5.4 · p. 13

Contributions

• Themis examines model-based scoring of LM-generated code for multi-criteria alignment, quality estimation, and re-ranking.The project aims to expand reward-model use in post-training beyond executable code, alongside open-source artifacts.Source: §6 · p. 13
• Themis introduces Themis-CodeRewardBench, Themis-GeneralPreference, Themis-CodePreference, and Themis-RM for multilingual, multi-criteria code reward modeling.The benchmark covers eight typologically diverse programming languages and functional and non-functional criteria; the preference datasets train a suite of state-of-the-art multilingual code RMs.Source: §6 · p. 13

Findings

• Current reward models struggle beyond judging the functional correctness of code contest solutions.The opening evaluations identify a narrow operating mode for existing RMs when modeling code preferences.Source: §6 · p. 13
• Code RMs trained on diverse code preferences and criteria show positive scaling, minimal cross-criteria interference, cross-lingual transfer, and robust adversarial performance.These findings establish such models as a viable reward source for code post-training.Source: §6 · p. 13

Architecture And Training Details

• Table 5 summarizes Themis-RM architectural heritage and training attributes across preference-model pre-training and preference-modeling stages.The training objective is described in Section 4, while training-data composition for both stages is detailed in Appendix B.Source: Table 5 · p. 44

B.1 BigQuery SQL Query For Commit Mining

• It unnests repository names from commit records and joins them with language and license tables before expanding each commit’s file differences.The selected fields include commit metadata, repository names, licenses, paths, and commit timestamps.Source: §B.1 · p. 45
• It limits repositories to a specified set of permissive and open-source licenses, including MIT, Apache-2.0, BSD variants, GPL variants, and Unlicense.The license whitelist also includes Artistic-2.0, ISC, CC0-1.0, EPL-1.0, and MPL-2.0.Source: §B.1 · p. 45
• The query restricts results to Python, Java, JavaScript, C, C#, C++, TypeScript, Go, and Ruby files.These language constraints target multilingual code commits across nine programming languages.Source: §B.1 · p. 45
• Commit messages must contain 10–15,000 characters and are filtered to exclude generic updates, tests, placeholders, merges, and automated Pi-push messages.The query also excludes numerous low-information message templates and requires unchanged, non-null old and new paths.Source: §B.1 · p. 45
• Grouping by commit and file metadata, the query retains only commits with exactly one distinct changed file.This HAVING condition operationalizes the single-file-change requirement.Source: §B.1 · p. 45
• The query mines single-file-changing commits from the BigQuery GitHub dataset in openly licensed repositories, following the pipeline of Muennighoff et al. (2024).The query is presented as a streamlined GoogleSQL adaptation for commit mining.Source: §B.1 · p. 45

B.2 Search Terms For Aspect-Specific Commit Classifier Training

• Table 6 specifies per-criteria search terms for recalling positives during commit classifier training.The complete commit preference mining pipeline is described in Section 3.Source: Table 6 · p. 46

B.3 Themis-RM Training Data Mixture And Filtering

• Themis-GeneralPreference contains 110k+ samples curated from helpfulness, harmlessness, and code-retrieval preference datasets to teach general human-preference nuances.Source: §B.3 · p. 47
• Themis-CodePreference contains 350k+ samples centered on diverse GitHub-commit and synthetically bugged instruction-tuning scenarios, augmented with existing code-preference and retrieval data.Source: §9 · p. 48

1. Commit-Preference

• Commit-Preference uses non-reverted single-file commits from successfully merged pull requests, with multi-frontier-LM consensus validating preference strength and single-intent selection.Source: §1 · p. 48

2. CodeR-Pile

• CodeR-Pile pairs parse chosen code from positive documents against rejected code from Zipfian-sampled hard negatives among the top-10 ranked documents (Li et al., 2025a).Source: §2 · p. 48

3. Bugged-Instruct

• Bugged-Instruct creates rejected responses by introducing algorithmic or syntactic bugs into instruction-tuning data from OSS-Instruct, Inverse-Instruct, McEval-Instruct, and Package-Instruct.Source: §3 · p. 48

4. ProSec

• The mixture includes synthetic vulnerability-fix pairs from ProSec (Xu et al., 2025a), contest repair pairs from RunBugRun, runtime preferences from ECCO, and difficult correctness pairs from CodeScaleR.Source: §4 · p. 48, §7 · p. 49, §8 · p. 49

5. Venus

• Efficiency and runtime preferences select completions at least 5x more efficient than rejected responses from Venus and CodeNet, while Pie4Perf uses successful non-spam submissions with highly divergent runtimes.Source: §5 · p. 48, §6 · p. 49, §10 · p. 49

11. Cybernative-DPO

• Filtering caps general and code samples at 2560 and 4096 tokens, removes shallow code, restricts commits to no later than March 2019, and excludes non-English or high-perplexity prompts.The procedure uses GlotLID for language filtering and KenLM on OSCAR EN to remove prompts with perplexity greater than 1200.Source: §11 · p. 49
• Training data undergoes MinHash near-deduplication and 13-gram prompt-overlap decontamination against multiple reward-model benchmarks.The filter uses shingle size 20 and similarity threshold 0.75, removing overlaps with Themis-CodeRewardBench, RewardBench V1/V2, JudgeBench, and RM-Bench.Source: §11 · p. 49

C.1 Commit Saliency Rubric

• The commit saliency rubric asks reviewers to summarize specific functional and technical changes, then rate criterion-specific improvement from 1 to 5.Ratings range from no improvement or degradation to a well-implemented, highly specific must-have improvement, with unrelated edits and new bugs considered.Source: §C.1 · p. 50

C.2 Commit Inverse Instruction Generation

• The inverse-instruction prompt reconstructs realistic problem statements from two code solutions while requiring semantic consistency and equal plausibility of either solution.Generated prompts must match the requested content style and avoid references to the solutions or their specific code constructs.Source: §C.2 · p. 51

C.3 Themis-RM Training And Evaluation System Prompt

• Themis-RM evaluates general helpfulness and harmlessness alongside five code-only criteria: memory efficiency, functional correctness, readability and maintainability, runtime efficiency, and security hardness.The evaluation principles require higher scores for responses that better satisfy each applicable criterion.Source: §C.3 · p. 52

C.4 Synthetic Bug-Laden Solution Generation

• Synthetic buggy solutions modify validated reference solutions with functional, logical, or algorithmic bugs while preserving surface structure and excluding new features, security vulnerabilities, memory leaks, and other non-functional bugs.These generated samples create diverse functional-correctness preference pairs for improving Themis-RM robustness.Source: §C.4 · p. 53

• Table 7 compares extant RMs and Themis-RM across dataset-level preference accuracy on the Functional Correctness split.The split is part of Themis-CodeRewardBench.Source: Table 7 · p. 54
• Extant RMs show a marked performance drop when judging partially correct solutions in MBPP+Fix (Hard).Source: Table 7 · p. 54
• Extant RMs also show a marked performance drop on open-domain GitHub commit preferences in CommitPref.Source: Table 7 · p. 54

D.2 Execution Efficiency (EE) And Memory Efficiency (ME) Dataset-Level Accuracy

• Table 8 reports dataset-level preference accuracy for extant RMs and Themis-RM across the Execution Efficiency and Memory Efficiency splits.The table focuses on non-functional code-reward criteria.Source: Table 8 · p. 55
• Most extant scalar RMs perform near randomly on the non-functional Execution Efficiency and Memory Efficiency criteria.This indicates that scalar reward models generally fail to distinguish preferences on these splits.Source: Table 8 · p. 55
• Generative RMs can become unusable in these settings because their scoring resolution is too low for the non-functional criteria.The passage attributes this limitation to low-resolution scoring rather than reporting specific accuracy values.Source: Table 8 · p. 55

D.3 Readability And Maintainability (R&M) And Security Hardness (SH) Dataset-Level Accuracy

• Extant scalar reward models generally approach random performance on Readability And Maintainability and Security Hardness, while generative models’ low-resolution scores can be unusable.Table 9 covers detailed dataset-level preference accuracy for these two Themis-CodeRewardBench splits and compares extant RMs with Themis-RM.Source: Table 9 · p. 56
• The D.3 analysis also includes experiments on auxiliary training objectives.Source: §D.3 · p. 56

D.4 General-Domain Reward Modeling Performance

• Themis-RM is evaluated externally on RewardBench V1, RewardBench V2, and JudgeBench.The evaluation benchmarks the Themis-RM suite against selected extant reward models that achieved the highest scores on Themis-CodeRewardBench.Source: Table 10 · p. 57
• The general-domain comparison uses RewardBench V1 from Lambert et al. (2025), RewardBench V2 from Malik et al. (2025), and JudgeBench from Tan et al. (2025).Source: Table 10 · p. 57

E Project Themis Artifacts

• Project Themis publicly releases a collection of accompanying artifacts.Table 11 provides the complete list of publicly released artifacts.Source: Table 11 · p. 57
• The released materials are documented comprehensively in Table 11.Source: Table 11 · p. 57