FaithfulBench: Does AI Counsel Uphold or Undermine the User's Professed Faith?

1 Introduction

• FaithfulBench measures whether counsel moves a believer closer to or further from the life their tradition asks of them.Faithfulness is judged by the user’s own tradition, not the evaluator’s; knowing doctrine alone does not ensure faithful counsel.Source: §1 · p. 1
• Naming the user’s faith improves the first answer, but only the tradition’s guide helps models hold that answer under pressure.The figure compares post-pressure scores across eight traditions: one-line disclosure marks the recognition gap, and the guide marks the further instruction gap.Source: Figure 1 · p. 2, §1 · pp. 2–3
• Its authors direct tradition-specific modules from canonical sources, with expert review and a shared core that supports cross-tradition comparison.Eight modules exist so far, and each sitting includes a dilemma followed by a pushback; steadfastness is the post-pressure score minus the first-response score.Source: §1 · p. 2
• The benchmark reports 555 scenarios, 49,950 sittings, and 204,195 judgments, alongside a three-tier normativity taxonomy and dual-judge checks.The reported cross-tradition spread narrows from 0.55 unstated to 0.31 stated to 0.23 guided.Source: §1 · p. 3

Evaluating values and safety

• Moral-behavior benchmarks use fixed labels, while open-ended counsel is commonly assessed with model judges; FaithfulBench uses two providers to reduce dependence on one judge family.Religion has also been evaluated as subject knowledge, including IslamicMMLU, IslamicLegalBench, and IslamTrust.Source: §2 · p. 3

Virtue under pressure and scripture in context

• VirtueBench finds that models choose virtue less often under pressure than they identify it abstractly [Hwang, 2026c,d], while FaithfulBench studies counselors responding to user pressure across eight traditions.Related work also finds that scripture in context raises VirtueBench scores in larger models but not smaller ones.Source: §2 · p. 4

Religion, spirituality, and spiritual care in HCI

• HCI research documents religion’s marginalization and develops spiritual-care design requirements, but does not measure whether deployed models’ counsel upholds a believer’s faith.The reviewed design work centers individual spirituality, relational accompaniment, and not imposing a provider’s beliefs.Source: §2 · p. 4

AI behavior on contested social and cultural ground

• Audits find that model responses reflect some groups more than others, mention religion less than surveyed adults expect, and give asymmetric conversion advice.A Catholic case study reports models flatten distinct theological claims into generalized perspectives [Karr et al., 2026].Source: §2 · p. 4

Positioning

• FaithfulBench’s stated gap is a cross-tradition measure of whether AI counsel upholds the user’s professed faith.It distinguishes this from knowledge, fixed-position opinion audits, actor-centered virtue tests, representation and conversion-symmetry audits, and HCI design requirements.Source: §2 · p. 4

3 Method

• Tradition authors and reviewers develop modules, while the harness runs scenarios under three framings and two judges score transcripts against scenario-specific guidance.The study evaluates five frontier models, with each reported score averaging two judges from different providers.Source: §3 · p. 5, §3.3 · pp. 7–8

3.1 Corpus construction

• Each item is a first-person real-life dilemma, not a quiz; its judge guidance identifies respected textual passages and the direction of faithful counsel for that tradition.The eight modules are drop-in banks built around canonical sources, with tradition-specific criteria traveling with each scenario.Source: §3.1 · p. 6

3.2 Protocol

• The protocol scores the first response and the full post-pressure conversation on a −1…+1 scale; steadfastness is the post-pressure score minus the first-response score.The three framings are unstated, one-line stated, and guided; the difference between stated and unstated is the recognition gap, and guided versus stated is the instruction gap.Source: §3.2 · pp. 6–7
• In one Catholic medical-assistance-in-dying scenario, GPT-5.6 Terra’s first answer scored +1.0, but after a secularize push its full-scope score was −1.0.The resulting steadfastness was −2.0, the largest possible drop; the example illustrates how a single push can reverse initially endorsed counsel.Source: §3.2 · p. 7

3.3 Models and judges

• Two judges score each conversation at both scopes without seeing the framing given to the model, so they cannot reward it simply for being told the user’s faith.The scores are averaged across judges, and reported intervals use scenario-cluster bootstrap 95% confidence intervals.Source: §3.3 · pp. 7–8

4.1 Undeclared, counsel varies with the tradition’s normativity

• Unstated post-pressure scores vary by normativity tier: +0.45 for low, +0.10 for medium, and −0.03 for high; guided scores rise to +0.85, +0.84, and +0.71.The tier grouping is descriptive, emerged from the results grid, and was not pre-registered; the Protestant module is in the medium tier (+0.05 unstated, +0.82 guided).Source: §4.1 · p. 8, Figure 4 · p. 10

4.2 The difference is recognition, not capability

• The best-to-worst tradition spread falls from 0.55 unstated to 0.31 stated to 0.23 guided, indicating that disclosure and the guide narrow the gap.The authors attribute the undeclared gap to recognition rather than capability; the guided high-normativity tier still trails the low-normativity tier.Source: §4.2 · p. 9

4.3 The one-line declaration gets the first answer; the guide keeps it

• Under the guide, steadfastness is near zero for Sonnet 5 (−0.01), Inkling (0.00), and Gemini Flash (−0.02), but remains negative for Terra (−0.09) and Qwen (−0.15).Among conversations with correct first answers, stated conversations concede less often than unstated ones (17% against 21%); insistence is the most damaging push for every model.Source: §4.3 · pp. 9–10, Figure 6 · p. 12, Figure 7 · p. 12

4.4 Both judges see the same structure

• The two judges agree closely overall (r = 0.831), with 95.4% of paired verdicts within ±0.5; the high-normativity residual is similar under either judge.Scoring by either judge alone preserves the five-model order for unstated and stated framings; under guided, the top pair and third and fourth places are each within 0.01.Source: §4.4 · p. 10

4.5 Standings: a stable top pair, a changed middle order, and Qwen last everywhere

• The unstated and stated model orders are identical; guided, Gemini 3.6 Flash rises above GPT-5.6 Terra, while Qwen3-235B remains last in every framing.The unstated gap between Sonnet 5 and Inkling lies within their overlapping intervals.Source: §4.5 · p. 11, Figure 8 · p. 13, Table 3 · p. 13

• The eight default therapeutic priors are a qualitative reading of sampled failures, not a coded taxonomy; formal conflict and mediation tests remain future work.Source: §5 · pp. 14, 16
• Models may state obligations accurately yet soften them: Inkling called omitting Jumu’ah a strategic choice, while Claude reframed guilt as resolved after pushback.Source: §5 · pp. 14–15
• When a tradition agrees with the therapeutic default, counsel can uphold it; otherwise, the assistant may substitute its own normative register without either party noticing.Source: §5 · p. 15
• Measured normativity tracks how often binding counsel conflicts with the priors, including duties to parents, chastity, sacramental urgency, and obedience.Source: §5 · p. 15
• Each of the six pressures appeals to a therapeutic prior, which the authors connect to negative steadfastness under both unstated and stated framings.Source: §5 · p. 15
• The proposed account of the high-normativity residual is interpretive: those traditions may require prescriptive rather than formational counsel, though stricter Sunni guidance could also contribute.Source: §5 · p. 16
• The Protestant module scores 36 consensus questions, leaving substantive disagreement (16%) unscored and making its score an upper bound for any particular Protestant.Source: §5 · p. 16
• FaithfulBench frames omitted religious counsel as withheld competence: guided ceilings indicate that better counsel is within model capability, making recognition the missing element.Source: §5 · p. 17

6 Limitations

• Two language-model judges score the grid, one also evaluated; no human raters scored it, and judge agreement does not establish ground truth.Stated and guided grids used a different serving route from the unstated grid; route changes are bridged and disclosed.Source: §6 · p. 17
• The English-only corpus has one guide per tradition, unequal scenario-bank sizes and compositions, and no measured collection run-to-run variance.The guided ceiling is a point estimate, and tier groupings inherit the bank-composition confound.Source: §6 · p. 17
• Only Sunni Islam, Judaism, and Roman Catholicism received expert audits; the other five banks followed the same protocol without those audits.The audited sample covered ten stratified scenarios per bank.Source: §6 · p. 17
• About 3% of verdict signs flipped on the same judge’s rerun, although mean scores remained stable at +0.03.The paper addresses cell-level noise by averaging two judges over the full grid.Source: §6 · p. 17

7 Future work

• Guided high-normativity cells that remain negative are proposed for classification as model-counsel failures or rubric strictness.This is described as a direct test of the interpretation in section 5.Source: §7 · p. 17
• The companion FaithfulWeights experiment uses judge-filtered context distillation and on-policy DPO to raise unstated counsel toward the guided ceiling across traditions.On AllFaith’s cold condition, meaningful religious representation rises from 1% of items to 27% after distillation and 30% after preference tuning (mean 0.113 → 1.147).Source: §7 · p. 17
• FaithfulWeights gains transfer to randomly held-out scenarios within traditions, but do not establish cross-tradition transfer.The reported split is within each tradition; a leave-one-tradition-out ablation was not run, and a prompted guide loses effect as filler separates it from the dilemma.Source: §7 · p. 18

Use of generative AI

• Language models drafted tradition modules from canonical sources under author direction; modules were mechanically validated and reviewed by tradition-competent reviewers.The collection, judging, and analysis pipelines were written with AI assistance and are open source.Source: Use of generative AI · p. 18
• The authors wrote the manuscript’s structure and original content, used AI assistance in revision, and fact-checked the text against the data.The authors take full responsibility for its content.Source: Use of generative AI · p. 18
• Figures were generated programmatically from the data, with no generative imagery used.Source: Use of generative AI · p. 18

A.1 Tradition modules

• Traditions are drop-in directories in a human-first format, with manifests, companionship guides, scenario materials, and a validator; the same professed-faith measure is anchored in each tradition’s source.The core harness discovers modules without changing its core when a tradition is added.Source: §A.1 · p. 22, Table 4 · p. 22

A.2 Per-scenario judge guidance

• Each scenario’s guidance binds its proof texts, direction of sound counsel, and boundary rules, letting one universal rubric judge eight traditions against local criteria.Source: §A.2 · p. 22

A.3 Framings; A.4 The six pressures

• Framings are unstated, stated with a one-line adherence prompt, or guided by a tradition’s companionship guide; six universal pressure types challenge the counsel.The six types are secularizing, insistence, false authority, good cause, flattery, and personal appeal.Source: §A.3 · p. 23, §A.4 · p. 23

A.5 Scoring and scopes

• Responses receive −1 to +1 scores in half-point steps at first-response and full-conversation scopes; steadfastness is the latter score minus the former.Source: §A.5 · p. 23

A.6 Judge configuration and statistics

• Two judges score each cell, and reported scores average their verdicts; confidence intervals use scenario-cluster bootstrap resampling within traditions.Tier, tradition, and overall means average per-model tradition means, so bank size does not weight traditions.Source: §A.6 · p. 23

B Per-tradition sources and expert review

• Tradition modules anchor scenarios to canonical sources or documented source constellations, while each scenario’s guidance carries the binding ground truth locally.Most banks are marked draft and pre-scholar-review, with anchors paraphrased to classical sources for verification; Judaism completed a first expert pass.Source: B Per-tradition sources and expert… · pp. 23–24

B.1 The review tool

• The review interface lets reviewers inspect source documents, guides, sampled scenarios, pressure texts, guidance, and judges’ verdicts; three banks were audited as a construction check.The authors characterize these as team audits, not independent external review; broader multi-stream review remains future work.Source: §B.1 · p. 24

B Per-tradition sources and expert review

• The source choices encode differing scopes: the Sunni bank uses Riyāḍal-Ṣāliḥīn, while the derived Protestant bank includes only demonstrated cross-tradition agreement and excludes disputed matters.The secular-sage bank has no canonical book and acknowledges omissions; Judaism defers genuine disputes to the person’s rabbi.Source: §B.2 · p. 24, §B.6 · p. 27, §B.7 · p. 27

C Complete results tables

• Table 5 reports the five-model pooled post-pressure framing staircase by tradition, with recognition and instruction gaps and paired confidence intervals.Recognition is stated minus unstated; instruction is guided minus stated.Source: Table 5 · p. 29
• Tables 6–8 report post-pressure scores for each model and tradition separately under unstated, stated, and guided framings.Each table gives bootstrap 95% confidence-interval half-widths.Source: Table 6 · p. 29, Table 7 · p. 29, Table 8 · p. 29

D.1 Design

• The full-grid pass judges nearly all 99,900 cells with Opus, superseding the earlier stated-and-guided sample except where the full-grid pass lacked a verdict.Gemini coverage is complete; Opus returned verdicts for 99,898 cells, with two remaining cells scored on Gemini alone.Source: §D.1 · p. 30

D.2 Agreement

• Across 99,898 matched cells, judges correlate at r=0.831, with 93.9% of paired verdicts within ±0.5 and 75.4% exact agreement.The lower guided correlation (r=0.684) accompanies 95.4% within-±0.5 agreement, attributed to ceiling compression.Source: §D.2 · p. 30, Table 9 · p. 31
• The judges preserve the five-model ranking in unstated and stated framings, while guided close pairs are ties under both judges.The guided top pair and third/fourth pair are each within 0.01 under either judge; the judges order the latter pair differently.Source: §D.2 · p. 30
• On the full stated-and-guided grid, Opus confirms a 0.15 high-versus-medium normativity gap in guided scores, close to Gemini’s 0.17 gap on the same cells.Opus scores the high-normativity tier at +0.70 versus +0.85 for medium; the high-normativity guided shift is −0.01.Source: §D.2 · p. 30, Table 11 · p. 32

D.3 Disclosures

• Gemini’s self-scoring is slightly favorable: on matched unstated cells, Opus scores its counsel 0.03 lower, without changing its rank.The unstated and stated routes differ, but a pilot found router-versus-direct agreement of r=0.930 with bias +0.016.Source: §D.3 · pp. 30–31

E Cost and compute

• The programme covers 49,950 sittings and 204,195 judgments, combining Gemini and Opus scoring plus a router pilot.Each scenario expands to 90 sittings and 360 judgments; the Opus layers add 102,495 judgments and the router pilot adds 1,800.Source: E Cost and compute · p. 33
• Table 12 reports programme cost as run, with collection and Gemini judging priced at verified 2026-08-03 rates and Opus cost derived from recorded token use.Collection and Gemini figures are estimates rather than invoices because funded credits and cache accounting affect them; batch discounts are applied to Opus jobs.Source: Table 12 · p. 33

F Additional figures

• Figure 11 shows each model’s mean scores moving from unstated to stated to guided across tiers, with guided endpoints near ceiling for three models but short of it for Terra and Qwen.In the high-normativity tier, Gemini spans −0.16 to +0.83 and Qwen spans −0.62 to +0.16.Source: Figure 11 · p. 34
• Figure 12 plots post-pressure scores by tradition and model; guided open markers cluster at +1 except for the high-normativity pair and Qwen.The horizontal distance between filled unstated and open guided markers represents the per-cell cost of the model not knowing whom it serves.Source: Figure 12 · p. 35