PaperlayerView exact revision on arXiv

Source-linked AI summary

TenetBench: A Clause-Anchored Russian Corpus for Evaluating Language Models against Professional Codes of Ethics

Yegor Anichkov, Valery Vinogradov, Andrey Furov, Marina Polukoshko, Sergey Bolovtsov

arXiv:2610.29477v1cs.CL

TL;DR

Broad harm evaluations do not show whether assistants help specialists breach a named professional-code clause. TenetBench builds paired, clause-anchored tasks and evaluates model responses with a human-validated judge; most tested model and reasoning-mode combinations produce conflicting content on most conflicting requests. The paper also demonstrates the pipeline on English codes from other jurisdictions.

  • Problem

    Broad harm evaluations do not show how often an assistant helps breach a professional-code clause or identify which provision is implicated.

    Source: §1 · p. 2

  • Method

    TenetBench pairs clause-anchored conflicting requests with benign twins and uses a frozen model judge validated against adjudicated human labels.

    Source: §9 · p. 33

  • Results

    Eleven of fourteen model and reasoning-mode combinations produce clause-conflicting content on most conflicting requests, mostly without comment.

    Source: §9 · p. 33

  • Takeaways & Limitations

    The pipeline also produced tasks and human-validated verdicts from two English codes of other jurisdictions, demonstrating portability beyond Russian.

    Source: §9 · p. 33

    The ordering reflects domain-scoped caution rather than conformance with a professional code in a broad sense.

    Source: §7.1 · p. 27

Abstract

from arXiv · show

Codes of professional ethics guide and restrict specialists, who increasingly work with assistants built on large language models. Such an assistant can breach the user's code, or help the user breach it, without the user noticing. Evaluations reporting broad categories of harm cannot show how often this happens or which provision is breached. TenetBench pairs 3,433 requests conflicting with clauses of fifteen Russian-language professional codes with benign twins that differ only in the legitimacy of what is asked. An automated pipeline generates the tasks from the text of a code, so the corpus grows with every code added and a provision yields new tasks on every run. It also produced tasks and human-validated verdicts from two English codes of other jurisdictions. In human validation of the frozen pipeline, specialists rated 0.995 of the conflicting tasks plausible, and the model judge agreed with the adjudicated label at Gwet's AC1 of at least 0.894 on 1,860 elements. We evaluated fourteen model and reasoning-mode combinations on whether they produce the conflicting text, decline or warn. Eleven of the fourteen produce conflicting content on most such requests, mostly without a warning, and nearly three warnings in four accompany conflicting content. The models that conform more do so by declining while still serving the legitimate twin, and enabling reasoning halved the refusals of two models. We hope the dataset and pipeline will help evaluate how language models observe professional norms and let diagnostic datasets be built for particular professions and organisations.

1 Introduction

TenetBench evaluates whether assistants help specialists breach a named clause of a professional code, addressing a gap left by broad harm and refusal evaluations. It builds on prior components while combining clause-anchored tasks, benign twins on shared scenarios, and clause-specific judgments.

1 Introduction

  • TenetBench targets whether a model helps a specialist breach a named professional-code clause, rather than whether its answer falls under a broad harm category.The introduction describes the need for verdicts that can be carried back to the body that wrote the code.Source: §1 · p. 2
  • The pipeline generates tasks from code clauses, pairs each conflicting request with a benign twin on the same scenario, and releases the pipeline and corpus.The corpus contains 3,433 conflicting requests across fifteen Russian-language professional codes; text-only violations define its scope.Source: §1 · pp. 2–3

2.2 Known failure modes of judges and agreement coefficients

  • The judgment procedure separates action, clause conflict, and refusal or warning, and a fixed function treats warnings as no mitigation when content conflicts with a clause.The judge receives the clause, scenario, request, and answer; its verdict was frozen before human labels existed.Source: §1 · p. 3
  • The judge agrees with adjudicated human labels at Gwet’s AC1 of 0.894, 0.915, and 0.926 on its three observations.The paper uses Gwet’s AC1 in part because agreement coefficients such as Cohen’s κ can degenerate under class skew.Source: §1 · p. 3, §2.2 · p. 6

2.1 Prior work on the constituent features

  • Prior work supplies individual components, including code-derived tasks, clause anchoring, and paired benign controls, but does not combine them in this clause-anchored shared-scenario design.The related-work discussion describes these components across MedSafetyBench, TRIDENT, and paired-control resources.Source: §2.1 · p. 4

2.3 Positioning against named work

  • Unlike related methods that report categories, preferences, or scores, TenetBench returns judgments about named provisions of externally authored codes.The positioning contrasts this clause-level verdict with Constitutional AI, Statutory AI, and Russian-language legal benchmarks.Source: §2.1 · pp. 4–5, §2.3 · p. 7

3 The Resource

The resource filters professional codes for clauses whose violations can be localized in answer text, then generates clause-anchored conflicting requests and benign twins. Its accepted corpus contains 3,433 pairs drawn from fifteen codes, with ten documents released.

3.1 Eligibility and intake

  • Of 745 manually reviewed clauses, 155 (0.208) were judged localizable in text, and the second reader’s agreement on 162 blind items was 87.7%.A single reader labelled all 745 clauses; the subsample agreement was Cohen’s κ 0.643 and Gwet’s AC1 0.811.Source: §3.1 · p. 8
  • Of 745 reviewed clauses, 0.342 carried no norm and 0.145 were evaluative-categorical without stateable content, so raw clause counts do not measure evaluability.The eligibility criterion also requires a legally identifiable document, harm-domain coverage, clause-level distinctness, and a compatible source licence.Source: §3.1 · p. 8

3.2 Clause anchoring and the release mode

  • Clause identity uses a hash of document identity and parse-tree position, while the release publishes locators and a digest instead of copyrighted clause wording.The source reference identifies the issuer, address, and edition; a mismatched digest makes a locator’s failure visible.Source: §3.2 · pp. 8–9

3.3 Pair construction and acceptance

  • The pipeline extracts in-scope clauses, generates scenarios, then creates a conflicting request and benign twin together from each scenario.The twin shares setting, actors, trigger, constraints, framing, and request form, differing in the legitimacy of what is asked.Source: §3.3 · p. 9
  • Acceptance escalated 18 of 3,451 pair slots, removing each affected twin and leaving 3,433 complete pairs; the corpus has 3,433 conflicting tasks across 395 clauses.Seven validators cover five blocking requirement classes; acceptance reported zero errors and warnings across 6,866 records.Source: §3.3 · p. 11, §3.4 · p. 11

3.4 Corpus statistics

  • Across 3,433 pairs, conflicting requests were longer than their benign twins in 0.697 of cases, with median lengths of 70 versus 63 words.The median Jaccard overlap between paired word sets was 0.144, indicating shared scenario substrate rather than shared wording.Source: §3.3 · p. 11, §3.4 · p. 12
  • Document contributions ranged from 24 to 701 conflicting tasks, with the spread following how many clauses carried tasks.The corpus includes fifteen documents; ten are released, while five passed legal verification but were withheld because of its outcome.Source: §3.4 · p. 11
  • Eight codes with sanctions contributed 2,237 tasks, while the ten-domain resource’s data-security domain contributed 24 tasks and was excluded from analysis.The measured-value tables cover nine harm domains.Source: §3.4 · p. 11

4 The Evaluation Procedure

The evaluation procedure separates judgments of model action, clause conflict, and refusal, then deterministically maps them to outcome categories. A frozen judge, defined analysis set, and fourteen-row model panel support the comparison, with explicit safeguards and caveats.

4.1 Judgement axes and outcome categories

  • Three independent axes record whether an answer performs the requested action, conflicts with the clause, and refuses or warns; a deterministic function maps their values to outcome categories.Conflict outcomes remain violations even when accompanied by a warning or explicit refusal; 46 of 573 conflicting-task compliant non-violation verdicts involved no action, violation, or refusal.Source: §4.1 · pp. 12–13

4.2 Judge inputs and outputs

  • The judge receives the clause, scenario, request, and answer, while pipeline-generated partitioning labels are withheld to avoid label reconstruction or tautological judgments.The harm domain is byte-identical to the published harm type, and risk mode has one value across all 6,866 records, so those two withheld fields provide limited concealment.Source: §4.2 · pp. 13–14

4.3 The frozen judge

  • Judge and human annotator receive identical wording, including two construction-side fields that may cue both, limiting what their agreement can establish.The frozen judge was generated before human labels were collected, preventing label-informed changes but leaving no correction loop; the tasks and judge also share a base-model family.Source: §4.2 · p. 14, §4.3 · p. 14

4.4 Quantities, analysis set and panel

  • The leaderboard orders models by non-conflict rate on resolved conflicting tasks and reports instrument-health, benign-refusal, and benign-violation companions.The companion rates are not used to order the table; a high non-conflict rate alone could reward refusing every request.Source: §4.4 · pp. 14–15
  • Each of fourteen model-and-reasoning-mode rows receives the same 675-task analysis set, enabling paired tests while widening intervals for individual estimates.The set contains 279 conflicting tasks and 396 benign tasks, including 279 one-to-one twins and 117 orphans; human labels come from an earlier set.Source: §4.4 · p. 15

5 Validation of the Procedure

The validation separates agreement with adjudicated human labels from whether the judge follows the clause it is given. The frozen judge shows substantial agreement on the annotated set, while clause substitution tests reveal both sensitivity to the substituted norm and limits on what that test establishes.

5 Validation of the Procedure

  • The validation distinguishes agreement with people from clause sensitivity, and measures them separately on different sets.The judge was frozen before human labels existed; benign agreement covers 141 tasks, while 255 later-added twins were not annotated.Source: §5 · p. 17, §5.1 · p. 17, §5.2 · p. 19

5.1 Agreement with the adjudicated human label

  • Gwet’s AC1 was 0.894 on instrumental action, 0.915 on norm violation, and 0.926 on refusal type.On benign refusal type, AC1 was 0.977; Table 2 reports agreement by axis.Source: §5.1 · p. 17, Table 2 · p. 19
  • The lowest annotator-pair AC1 on norm violation was 0.731, below the declared 0.80 band.All three annotators agreed on 0.750 of elements; adjudicated agreement is a different, less noisy measure.Source: §5.1 · p. 18
  • Only 0.005 of conflicting-task elements were rated implausible or partly plausible, against a 0.30 cut-point.The share rated implausible was 0.000, against a 0.10 cut-point.Source: §5.1 · p. 18

5.2 Clause sensitivity under substitution

  • Substituting a clause from another profession yielded 0.9133 not-violated verdicts, versus 0.6867 for another clause from the same document.The different-profession arm passed its 0.8000 cut-point, including at the lower interval bound.Source: §5.2 · p. 19, Table 3 · p. 20
  • Clause sensitivity is a lower bound on tracking because unchanged verdicts may reflect an applicable norm, unread clauses, or construction-side fields.The answer-action axes barely moved under substitution, while the reverse direction was not tested.Source: §5.2 · p. 20

6 Results

Across fourteen model and reasoning-mode runs on 675 tasks, eleven usually produced clause-conflicting content; more-conforming models mainly refused, while warnings often accompanied violations.

6.2 Models differ in whether they decline

  • Safe refusal drove most between-row differences in non-conflict rates, while models largely agreed on which requests to refuse and rarely refused benign twins.Safe refusal accounted for 0.767 of between-row variance; over-refusal stayed at or below 0.029 in every row.Source: §6.2 · pp. 21–22

6.1 Most models complete the conflicting request

  • 0.671 of answers to conflicting tasks violated their clause; eleven rows violated on more than 0.6 of such tasks.Most violations were silent: 2,215 of 2,622 carried neither refusal nor warning.Source: §6.1 · p. 21, Table 4 · p. 22

6.3 A warning does not keep an answer within the clause

  • 0.726 of warnings without refusal accompanied clause-conflicting content, and twelve of fourteen rows mostly warned over violations.DeepSeek-V4-Flash warned on 100 tasks and violated in 79; gpt-oss-20b warned on six and refused outright on 72.Source: §6.3 · pp. 22–23

6.4 Where the models conform

  • The panel’s non-conflict rate was 0.611 on confidentiality clauses versus 0.279 on other domains, driven by more conflict-free fulfilment rather than more refusal.Procedural compliance and conflicts of interest had the lowest domain rates, at 0.232 and 0.252.Source: §6.4 · p. 23, Table 5 · p. 24
  • The eleven lower-ranked rows conformed more to prohibitions than obligations (0.287 vs. 0.143), while obligation breaches can look like ordinary omissions.The three leading rows treated the two clause types similarly, at 0.731 and 0.705.Source: §6.4 · pp. 23–24

6.5 What changes a model’s conformance

  • With reasoning enabled, two Qwen3.6 models’ conformance fell as safe refusals halved; role-play raised violations by 0.041, while hypothetical framing did not.Past-tense and academic framings were not tested on paired scenarios.Source: §6.5 · pp. 24–25
  • Generation and training mattered more than size: Qwen3.6 rows outperformed earlier Qwen3 rows, while Russian adaptation alone did not ensure conformance.Only one frontier model was evaluated, so its top rank cannot distinguish model, tier, or developer effects.Source: §6.5 · p. 25

6.6 What the findings rest on

  • The judge agreed with adjudicated labels at AC1 values of 0.894–0.926, and the ordering held on released documents; verdict sensitivity and truncation qualify some comparisons.Removing task-threat fields shifted verdicts by 0.060 against a declared 0.020 bound, though 58 of 59 separated pair directions persisted; task regeneration bounded generator-family advantage at 6.8 percentage points.Source: §6.6 · pp. 25–27

7 Discussion and Limitations

The discussion narrows what TenetBench’s ordering can establish: it reflects domain-scoped caution, while several checks and measurement boundaries qualify interpretation. The authors also report transfer to English codes and specify what the release permits readers to reproduce.

7.1 An ordering by caution

  • Paired discrimination is positive across all fourteen rows and survives length matching, but its magnitude varies widely and the released deposit cannot run the same measurement under the fixed rule.The median difference is +0.1111 across 279 pairs per row, versus +0.099 on 91 length-matched pairs; the weakest deposit row reaches only 24 pairs against the required 30.Source: §7.1 · pp. 27–28
  • The ordering is closer to domain-scoped caution than broad professional-code conformance, and the authors withdraw the broader claim.The discrimination direction is positive in every row, but its size differs across the panel by a factor of ninety.Source: §7.1 · p. 27

7.2 Checking a verdict against its evidence

  • Under the declared verbatim rule, cited evidence fragments occur in 0.623 of campaign verdicts and 0.627 of deposit verdicts, below the 0.95 threshold.With formatting set aside and partial matches counted, the rates are 0.937 and 0.951, respectively; most of the gap reflects markdown formatting and shortened or reordered quotes.Source: §7.2 · p. 28

7.3 A verdict that moves with construction-side fields

  • Removing construction-side fields shifts a net 0.060 of paired verdicts toward no violation, exceeding the 0.020 bound, while agreement with people falls from AC1 0.915 to 0.872.On the 83 adjudicated elements where configurations disagree, human labels favor the configuration retaining the fields 60 times to 17; row ordering changes little.Source: §7.3 · p. 28

7.4 Transfer to another jurisdiction

  • On two English codes from other jurisdictions, verdicts agree with adjudicated human labels at AC1 0.823 to 0.958; no Russian-versus-English quantity is compared.Transfer exposed twelve silent language-dependent defects, all fixed; the authors say this confirms pipeline portability to another language.Source: §7.4 · pp. 28–29

7.5 Threats no check was declared against

  • The verdict concerns answer text, not subsequent specialist actions or answer usefulness; benign twins are not checked against clauses, and 0.251 receive at least one violation verdict.The authors also note that task-purpose signaling is not verified, and five codes are advisory, so verdicts against them concern breaches of declarations.Source: §7.5 · pp. 29–30
  • Service route and replication constrain interpretation: the route comparison is too imprecise to exclude a route effect, and byte-for-byte repeated collection occurred for only 0.0071 of tasks.The corpus covers fifteen professions in one legal system and Russian; provider limits truncate the longest generations, so pairing does not protect absolute values.Source: §7.5 · pp. 29–30

7.6 What the release lets a reader redo

  • The deposit contains 4,470 of 6,866 records, and removing withheld documents raises non-conflict rates on all fourteen rows, so the published ordering is not claimed to survive that change.Readers can recompute some adjudication agreement and evidence-span traceability from the release, but cannot check the intake reading against withheld clause text; post-publication comparisons must account for possible training-data ingestion.Source: §7.6 · pp. 30–31

8 Ethics, Broader Impact and Release Risk

TenetBench is released despite dual-use risk because reproducible evaluation lets third parties check what was measured. The section also describes copyright and privacy boundaries, including limits that the screening does not resolve.

8 Ethics, Broader Impact and Release Risk

  • The corpus is dual-use: realistic requests can diagnose professional-code failures, but also provide phrasing that could help someone circumvent a norm.The release includes 7,303 verdicts; 2,621 identify full clause violations and 41 partial violations.Source: §8 · p. 31
  • The authors release the resource because reproducibility lets third parties check measurements, while stating that this benefit does not resolve the risk of misuse.The canary strings and held-out subset distinguish later comparisons from those involving a model that has already seen the corpus; they are not safeguards.Source: §8 · pp. 31–32
  • The release uses reference mode because the source codes are under third-party copyright, while derived deontic fields can be published under the stated legal grounds.The paper describes a full-text worked example for an official state act and scholarly quotation where derived fields coincide with wording.Source: §8 · p. 32
  • Across 1,898 clauses, published fields reproduce clause wording verbatim in 0.166 of conditions, 0.125 of actions, and 0.358 of actor roles.One clause is fully recoverable from published fields; the authors disclose it as a limit, and 53 clauses are recoverable beyond half from runs of at least ten words.Source: §8 · p. 32
  • The privacy screen covers verdict answers, evidence spans, and rationales, but does not establish that the corpus contains no personal data.Its recall is unmeasured; 144 distinct email addresses, some at live providers, remain unmasked, and one taxpayer number was unresolved without a register lookup.Source: §8 · pp. 32–33

9 Conclusion

The conclusion reports frequent production of clause-conflicting content across evaluated models, while more-conforming models tend to decline and still serve legitimate twins. It also summarizes benchmark scope, transfer to English codes, limitations, and release materials.

9 Conclusion

  • The benchmark contains 3,433 clause-anchored conflicting tasks with benign twins; 2,235 pairs from ten of fifteen documents are released with the pipeline and validated judge.The pipeline and judge were frozen before any human label on a measured answer existed and were not adjusted afterward.Source: §9 · p. 33
  • Eleven of fourteen model-and-reasoning combinations produce conflicting content on most breach requests, mostly without comment; nearly three warnings in four accompany conflicting content.More-conforming models decline the conflicting request while serving its legitimate twin; enabling reasoning lowered conformance in two models by turning refusals into completions.Source: §9 · p. 33
  • The conclusion identifies three limits: rankings largely reflect willingness to decline, evidence spans are locatable for 0.623 to 0.937 of prohibition-clause violation verdicts, and task-construction fields partly affect verdicts.Despite that dependence, 58 of 59 separated row pairs retain their order; the instrument was not adjusted in response to these results.Source: §9 · p. 33
  • Transfer to two English codes of other jurisdictions produced tasks and verdicts validated against human annotation, with defects exposed by transfer found and fixed.Source: §9 · p. 33

Use of AI tools in manuscript preparation.

  • The authors used assistants to search related literature, translate and edit the manuscript, and support software development, reviewing all outputs and taking responsibility for the published text.Source: Use of AI tools in manuscript… · p. 34

Data availability.

Supplementary Information for TenetBench: A Clause-Anchored Russian Corpus for Evaluating

The supplementary file provides enumerations supporting statements made in the article, rather than reporting results omitted from it.

Supplementary Information for TenetBench: A Clause-Anchored Russian Corpus for Evaluating

Online Resource 1

  • Online Resource 1 presents enumerations in the order the article invokes them so that article statements can be read against their supporting material.The resource states that it contains no result omitted from the article.Source: Online Resource 1 · p. 40

S1 Comparison with the nearest resources

The supplementary comparison table evaluates nearby resources along six stated axes and follows a source-by-source provenance rule. It records internal inconsistencies in several works rather than resolving them by inference.

S1 Comparison with the nearest resources

  • Table S1 compares resources by size, language, anchoring unit, benign counterpart, availability, and data licence, using only cells stated in the work or released files.Unreported cells are marked “not stated,” and a paper-only licence does not count as a data licence.Source: S1 Comparison with the nearest resources · p. 40
  • Three compared works contain conflicting counts across their text, versions, or releases, so the table records the provenance-supported figures rather than silently harmonizing them.Examples include CompliBench’s 318 dialogues versus 306 files, and TRIDENT counts that differ across versions and released files.Source: S1 Comparison with the nearest resources · p. 40

S2 Distribution of the declared fields

The declared fields describe a corpus dominated by obligations and prohibitions, with mostly high-severity tasks and a balanced schedule of surface styles. The section also reports domain coverage and measures how distinct paired requests are.

S2 Distribution of the declared fields

  • The figures report measurements for the conflicting half, and the benign half returns the same one.This is the stated mirroring invariant for the corpus figures.Source: S2 Distribution of the declared fields · p. 40
  • Obligations account for 0.475 of clauses, prohibitions 0.410, and permissions 0.115; high severity covers 0.627 of tasks.Ambiguity is clear for 0.554 and mixed for 0.446; the five surface styles each fall between 0.199 and 0.201.Source: S2 Distribution of the declared fields · p. 40
  • Conflict of interest is the largest domain at 0.212, followed by procedural compliance at 0.199 and misleading communication at 0.179.Data security has 24 conflicting tasks, or 0.007 of the corpus, all from one document, and is excluded from the analysis set.Source: S2 Distribution of the declared fields · pp. 40, 42
  • The conflicting and benign requests have a median Jaccard word-set overlap of 0.144 across 3,433 pairs, with quartiles at 0.118 and 0.177.Across 11,782,056 within-family pairs, maximum word-set similarity is 0.564 for conflicting tasks and 0.368 for benign twins; none exceeds 0.6.Source: S2 Distribution of the declared fields · p. 42

S3 Full text of the eight declared checks

The section documents how the eight instrument checks were declared and assessed against thresholds. It explains the visual encoding of Figure S1 and notes how agreement bands and pairwise corrections were set.

S3 Full text of the eight declared checks

  • Every outcome threshold was archived before its tested quantity was computed, but three of the eight checks did not meet their threshold.Six declarations predated answer collection; the other two were recorded during collection, each more than ten hours before its governed measurement.Source: S3 Full text of the eight declared… · p. 42
  • Figure S1 groups checks as held, returned a bound, or missed their threshold; five of eight were not refuted.Horizontal position shows the observed value relative to its threshold, while shading indicates the required side; plausibility has two rows and warning separability has no threshold of its own.Source: S3 Full text of the eight declared… · p. 42
  • Agreement bands of 0.80 for norm violation, 0.70 for each other axis, and 0.60 as the floor were Cohen’s κ thresholds carried over to Gwet’s AC1.The article prints κ and Brennan–Prediger alongside AC1 to make the transfer visible; the pairwise correction family was declared after the data existed and can only widen intervals.Source: S3 Full text of the eight declared… · p. 42

S4 Unpaired evidence on the length threat

The section presents unpaired evidence bearing on whether request length explains the difference between conflicting and benign branches. The reported branch gap persists across length buckets, while refusal does not rise with length within either branch.

S4 Unpaired evidence on the length threat

  • The section reports two additional unpaired measurements, but gives them less weight than the matched reanalysis.They are presented as a second check on a threat already bounded by paired measurement.Source: S4 Unpaired evidence on the length… · p. 42
  • Across four length buckets, the gap between branches stays positive, ranging from 0.177 to 0.224.A length-based account predicts a narrowing gap toward the middle of the range, which is not observed.Source: S4 Unpaired evidence on the length… · p. 44
  • Within branches, refusal does not rise with length: median gradients are −0.0125 for conflicting requests and 0.000 for benign ones.Rising refusal within a branch is another prediction of the length account, but neither branch shows it.Source: S4 Unpaired evidence on the length… · p. 44

S5 Codes and size of the transfer to another jurisdiction

The pipeline was tested on two English-language codes from other jurisdictions, producing 39 clauses with tasks and 78 records. Human-validated verdicts from a three-row campaign provide evidence of transfer beyond the Russian corpus.

S5 Codes and size of the transfer to another jurisdiction

  • The two English codes yielded 39 clauses carrying tasks and 78 records, evaluated in a campaign with three panel rows.The codes are 5 CFR Part 2635 and the Standards of Conduct for the International Civil Service.Source: S5 Codes and size of the transfer to… · p. 44
  • Across 117 conflicting-task verdict and adjudicated-label pairs, the English judge reached Gwet’s AC1 of 0.823 on instrumental action, 0.907 on norm violation, and 0.958 on refusal type.Source: S5 Codes and size of the transfer to… · p. 44
  • On 117 benign-branch verdict and adjudicated-label pairs, the English judge reached Gwet’s AC1 of 0.936 on refusal type.The campaign's verdicts were validated against human annotation.Source: S5 Codes and size of the transfer to… · p. 44

S6 Complete lists behind three statements of prior work

The section provides the full lists behind three statements of prior work.

S6 Complete lists behind three statements of prior work

  • Norm-extraction work covers policy and regulatory documents, runtime events, privacy law, and commercial contracts, while MedES uses professional norms (Datla et al., 2025; Guliani et al., 2026; Chung et al., 2025; Li et al., 2025; Singh et al., 2026; Jin, Ying, Ji, Xu, & Wu, 2026).The approaches include extracting norms, building a graph, and matching deontic and temporal entities.Source: S6 Complete lists behind three… · pp. 44–45
  • Prior work reports agreement by span, extracted field, annotator group, or rater pair, while validator-based acceptance checks use deterministic tests, satisfiability solving, or validator-derived labels (Datla et al., 2025; Zhang et al., 2026; Lee, On, Han, Cohan, & Hockenmaier, 2026; Malarkkan et al., 2026).The cited rater agreement assesses rubric reliability and judge fitness.Source: S6 Complete lists behind three… · p. 45
  • Listed Russian-language resources evaluate varied tasks, but none is anchored to professional-code provisions; LegalBench-RU uses article-level norms (Fenogenova et al., 2024; Martynov et al., 2025; Goloburda et al., 2025; Limonova & Studenikina, 2026; Grigoreva, Ivanova, Alimova, & Artemova, 2024; Arabov, 2026; Chetvergov et al., 2025; red_mad_robot R&D, 2026; Kozlov & contributors, 2026).The list includes safety, factuality, stereotypes, finance reasoning, and other evaluation targets; the closed Lexometrica ranking is excluded because its cases and prompts are unpublished (Lexometrica, 2026).Source: S6 Complete lists behind three… · p. 45

S7 Identifiers of the measured objects

Table S3 identifies measured objects by their deposit identifiers, and the section distinguishes objects and task samples that should not be conflated.

S7 Identifiers of the measured objects

  • Table S3 names each measured object by its deposit identifier and scope, making the object addressable; the judge identifier ends with a digest of its judging prompt after line endings are normalized to LF.Hashing the shipped template reproduces that identifier.Source: S7 Identifiers of the measured objects · p. 45
  • Analysis-set quantities use the paired set, whereas human labels come from an earlier annotation package; the two objects are not interchangeable.A figure from either object does not describe the other.Source: S7 Identifiers of the measured objects · p. 45
  • The task sample comprises two separately declared samples of 420 and 255 tasks, whose union has no declared-run fingerprint.Computing a union statistic after the fact would introduce a quantity no run declared.Source: S7 Identifiers of the measured objects · p. 45

Related papers

Loading 2610.29477v1…