ORQA: An Occupation-Realistic Question and Answer Framework for LLM Professional Knowledge

1 Introduction

• Existing economic studies typically measure AI exposure, adoption, or productivity, while available benchmarks do not provide scalable occupation-level model comparisons across the US economy.The desired benchmark is scalable, broadly applicable, occupation-level, and source-verifiable, and should assess professional knowledge beyond task completion.Source: §1 · pp. 2–3
• ORQA connects O*NET occupations to authoritative occupation-specific materials and turns them into source-traceable questions with automated checks and human review.The benchmark contains 480 questions from 187 source hosts, spanning 116 occupations across all 21 major SOC groups.Source: §1 · pp. 3–4
• 62.4%, 60.3%, and 58.7% are the closed-book multiple-choice accuracies for Claude Opus 4.6, GPT-5.4, and Claude Sonnet 4.6, respectively.The frontier models are closely grouped; smaller or older models score 33–41%.Source: §1 · p. 5
• 0% is GPT-5.4’s score for Actuaries, Sheet Metal Workers, and Fish and Game Wardens, while frontier models score 78–80% for Healthcare Practitioners and 36–43% for Office and Administrative Support.Occupation- and model-level results vary substantially, with cross-model differences reaching 100 points in some jobs.Source: §1 · p. 5
• ORQA’s rankings correlate with GDPval (Spearman +0.90) and GDPval-AA Elo (+0.92), and its occupation coverage spans different Anthropic Economic Index exposure bins.The authors state that ORQA does not appear to contradict external economic signals.Source: §1 · pp. 5–6
• The paper contributes a 15-model evaluation, wage-bill weighting, external-signal comparisons, and a community-thread version that uses public occupational discussions.The authors frame ORQA as both a resource and an example of extending automation to occupation-level understanding.Source: §1 · p. 6

2 Related Work

• General capability benchmarks measure areas such as reasoning and coding, but are not structured around jobs and do not establish whether a model can answer occupation-specific questions.A high score on an abstract capability benchmark need not imply assistance with concrete questions in a particular field.Source: §2 · pp. 6–7
• Web, enterprise, and workplace benchmarks make evaluation more realistic, yet tend to focus on digital work and underrepresent occupations beyond computer- and math-related fields.WorkArena, TheAgentCompany, WebArena, and SWE-bench are examples of these task-oriented benchmarks.Source: §2 · p. 7
• GDPval evaluates economically valuable occupation-based tasks with expert-created prompts, while ORQA tests whether model answers agree with authoritative occupational knowledge.Expert task creation and evaluation make repeated expansion to hundreds of occupations expensive; ORQA measures a narrower construct than end-to-end task performance or assistance value.Source: §2 · pp. 7–8

3 Methodology

• A two-annotator review informed the pipeline’s quality criteria, with about 67% agreement between human and automated “good” decisions; the authors do not present automation as replacing expert curation.Correctness is defined by entailment from the source text, independently of the scalable quality gate.Source: §3 · p. 8
• The pipeline allocates items across SOC groups according to each group’s share of the national wage bill, calculated from employment and average annual wages.Jobs are prioritized in decreasing wage-bill order within each occupational category.Source: §3 · pp. 8–9
• Occupation-specific whitelists restrict source discovery to trusted publishers, and research papers are excluded because they report study results rather than professional accountability.Search is constrained to each job’s whitelist, with no outside source allowed into the bank.Source: §3 · p. 9
• Documents are parsed into overlapping 2.5–4 KB windows, from which an extraction model creates evidence cards with source quotes, tasks or procedures, and occupations.Each card becomes a six-choice question whose correct answer must follow from the source quote; distractors come from the same document.Source: §3 · p. 10
• Automated filters check entailment, question and answer quality, uniqueness, single-answer validity, difficulty, and whether general knowledge can eliminate distractors.Failed items receive targeted regeneration and are rechecked; items solvable by all three specified smaller models are rejected as too easy.Source: §3 · p. 10
• Balancing limits source-domain concentration, preserves at least one verified item per occupation, and expands underrepresented jobs to at least three items where possible.The focused expansion raised the number of jobs with at least three items to 106 of 116.Source: §3 · p. 11
• 480 items cover 116 jobs across all 21 major SOC groups and 187 source hosts, with coverage uneven across groups.About 43% of items come from federal government sources, and osha.gov accounts for 9.8% of the bank.Source: §3 · p. 11
• Fifteen models are tested without browsing on each question three times, with accuracy averaged over question-model-seed combinations and uncertainty estimated by occupation-cluster bootstrap.The six-choice random-guessing reference is 1/6, or about 16.7%.Source: §3 · pp. 11–12

4 Results

• ORQA items are occupation-linked, have correct answers entailed by source excerpts, and undergo automated verification tuned using human annotation.Source: §4 · p. 12
• 62.4%, 60.3%, and 58.7%: Claude Opus 4.6, GPT-5.4, and Claude Sonnet 4.6 lead closed-book multiple-choice, within about four percentage points.All fifteen models exceed the 16.7% random-chance score, yet even the top models miss over a third of occupational questions.Source: §4 · pp. 12–13, Figure 2 · p. 13
• 54.2%, 53.1%, and 50.6%: GPT-5.4, Claude Opus 4.6, and Claude Sonnet 4.6 lead open-ended evaluation; web search raises frontier scores to 71–73% open-ended and 78–80% multiple-choice.Wage weighting preserves the same top-three order: Claude Opus 4.6 scores 66.8%, GPT-5.4 64.2%, and Claude Sonnet 4.6 63.4%.Source: §4 · pp. 13–14, Figure 2 · p. 13
• 78–80% for Healthcare Practitioners contrasts with 36–43% for Office and Administrative Support; GPT-5.4 scores 0% on Actuaries, Sheet Metal Workers, and Fish and Game Wardens.Some SOC groups have sparse coverage, so their group-level results should be treated provisionally.Source: §4 · p. 14, Figure 3 · p. 15, Figure 4 · p. 16
• Specific codes, numeric limits, and source facts challenge closed-book models; web search retrieves a Flight Attendants boarding-pay answer that a closed-book model misses.The examples illustrate a possible gap between closed-book and web-search performance.Source: §4 · pp. 15–17
• ORQA rankings correlate with GDPval (Spearman ρ = +0.90; Pearson r = +0.89) and GDPval-AA (Spearman ρ = +0.92; Pearson r = +0.93).The authors describe these checks as basic and say more in-depth evaluation against human-compiled gold answers across jobs would be advantageous.Source: §4 · p. 17

5 Discussion

• ORQA measures agreement with specified authoritative or professional sources, not whether a model would make the correct decision in a professional context.Relevant professional knowledge may not appear in the source materials, which may also differ in relevance or be incomplete.Source: §5 · pp. 17–18
• Coverage spans 116 occupations across all 21 SOC major groups, but thinly covered occupations and groups support exploratory rather than definite rankings.Some SOC major groups have only one or two occupations, while authoritative machine-readable sources are limited for some groups.Source: §5 · p. 18
• ORQA does not provide substantial human–LLM collaboration and should complement human-in-the-loop, task-based, and interactive assessments before practical use.The authors describe it as one element of occupational assessment and suggest future use for evaluating end-to-end occupational tasks with AI agents.Source: §5 · p. 19
• ORQA connects occupational knowledge to authoritative online resources and turns them into source-verifiable evaluation tasks at lower cost than expert-based evaluation.The same approach can also use non-authoritative resources.Source: §5 · p. 19

A Technical Appendices and Supplementary Material

• Appendix A.1 reports bank composition, occupation and source lists, source distributions, and SOC-group information.Appendix A.2 adds per-item diagnostics, including significance, difficulty, correctness correlation, and release-date trends.Source: A Technical Appendices and… · p. 23
• Appendices A.3–A.6 cover pretraining-contamination checks, prompts and inference parameters, external validation signals, and the interactive dashboard.The listed external signals are GDPval, GDPval-AA Elo, and Anthropic Economic Index exposure.Source: A Technical Appendices and… · p. 23
• Appendices A.7 and A.8 provide the complete occupation heatmap and a community-thread version of ORQA.Source: A Technical Appendices and… · p. 23

A.1 Composition of Bank and Lists

• 480 items cover 116 occupations across all 21 SOC major groups; 106 occupations have at least three items, while the rest have one or two.Source: §A.1 · p. 23
• 57.1% of items come from licensing boards, associations, and professional societies, while 42.5% come from federal and state government hosts.Source: §A.1 · p. 24
• The bank draws on 187 source hosts, with OSHA.gov contributing 9.8%; research publishers are excluded by the source whitelist.Educational sources account for 0.4% of items, including two from the Texas A&M Forest Service, which publishes forestry practice standards.Source: §A.1 · p. 24

• 38% of items are correct at least 80% of the time, 22% fall between 20% and 80%, and 40% are correct no more than 20% of the time.Median per-item accuracy is 51.1%; the authors attribute the heavy lower mode to the human-curated set’s higher proportion of difficult items.Source: §A.2 · p. 25, Figure 5 · p. 25
• Table 1 shows sharp occupation-level differences: some model spreads reach 75 percentage points, GPT-5.4 scores zero on some jobs, and Qwen 2.5 7B matches frontier models on others.The table restricts comparisons to occupations with at least three items and averages accuracy across three seeds.Source: §A.2 · p. 25, Table 1 · p. 26
• Higher-performing models correlate strongly on item correctness, while correlations involving the three difficulty-screening models are attenuated by construction.GPT-5.4 and Claude Opus 4.6 correlate at r = 0.94; GPT-4o mini and Claude Haiku 4.5 have the lowest correlation, r = 0.07.Source: §A.2 · pp. 26–27, Figure 6 · p. 27
• Frontier-model accuracy rises by about 27 percentage points from GPT-3.5 Turbo (33.3%) to GPT-5.4 (60.3%) across approximate release dates.Release date is a noisy indicator of performance effort, and some gains may reflect more focused post-training on professional material.Source: §A.2 · p. 28, Figure 7 · p. 28
• The 480-item bank draws 57.1% of its content from licensing and professional organizations and 42.5% from government sources, and covers all 21 SOC major groups.Coverage is uneven: several groups have only one or two occupations and few items, so their group-level accuracy is tentative.Source: §A.2 · pp. 28–29, Table 2 · p. 29, Table 3 · p. 30
• Employment-weighted accuracy is nearly unrelated to wage (r = −0.02), whereas unweighted accuracy has a weak positive correlation (r = +0.22, p = 0.02).The authors characterize this as a descriptive result, not a test of a specific hypothesis.Source: §A.2 · pp. 29, 31, Figure 8 · p. 31
• Digital and non-digital occupations have similar average accuracy across models: 48.1% for digital jobs versus 47.6% for non-digital jobs.Both categories include some of the most challenging occupations.Source: §A.2 · p. 32, Figure 9 · p. 32

• The diagnostic splits accuracy by whether a question’s source postdates the model’s pretraining cutoff; post-cutoff sources are treated as free from contamination for that model.If memorization inflated accuracy, the authors expect post-cutoff accuracy to be consistently lower than pre-cutoff accuracy.Source: §A.3 · pp. 32–33
• Source dates are assigned through URL patterns or conservative host-level priors; 261 undatable items are excluded, leaving 219 dated items for analysis.The cutoff dates come from model cards when available, or conservative dates inferred from release dates otherwise.Source: §A.3 · p. 33
• Fourteen of fifteen models have post-cutoff accuracy equal to or above pre-cutoff accuracy, suggesting the diagnostic is more consistent with professional-knowledge transfer than pretraining memorization.GPT-5.4 scores 77.8% on its post-cutoff stratum (n = 3) versus 59.3% pre-cutoff; GPT-3.5 Turbo is the sole exception, at −2.1 pp.Source: §A.3 · p. 33, Figure 10 · p. 34
• Post-cutoff samples are small for the newest frontier models, creating substantial uncertainty; a planned after-cutoff sub-bank would increase sample size and tighten per-model intervals.The passage notes n = 3 for GPT-5.4 and proposes using sources published after the most recently assessed model’s training cutoff.Source: §A.3 · p. 34

• Evidence-card extraction records a source excerpt, its task or procedure, an occupation domain, and a quality rating; item creation requires six choices and a source-entailed correct answer.The choices must include “All of the above” and “None of the above.”Source: §A.4 · p. 35
• Nine quality checks cover entailment, question and distractor quality, answer uniqueness, option structure, difficulty, eliminability, and concreteness, followed by categorization tuned using human labels.The difficulty pretest uses Claude Haiku 4.5, Gemini 2.5 Flash, and GPT-4o mini in parallel.Source: §A.4 · p. 35
• The evaluation includes fifteen models; API version, temperature, and random seed are recorded for three inferences per model, with open-weight models accessed through Together AI.The construction pipeline uses GPT-4o for item creation and evidence-card extraction, GPT-4o mini for search and planning, and a separate Claude Haiku 4.5 verifier.Source: §A.4 · pp. 35–36

• ORQA accuracy correlates with GDPval rankings at Spearman ρ = +0.90 and with GDPval-AA rankings at Spearman ρ = +0.92 among overlapping models.The corresponding Pearson correlations are +0.89 for GDPval and +0.93 for GDPval-AA.Source: §A.5 · p. 36, Figure 11 · p. 37
• Among nine occupations shared with GDPval, comparisons show directional rather than exact agreement, with notable disagreements for Nurse Practitioners, Real Estate Brokers, and Lawyers.The authors suggest differences in source use or expected response format may account for some mismatches; ORQA occupation scores use only 3–9 questions per cell and are intended as a qualitative check.Source: §A.5 · p. 37, Table 4 · p. 38

A.6 Interactive Dashboard

• Users can filter per-model and per-occupation results by source domain and category, then open individual items and source texts.The dashboard also provides access to the underlying data for further use without rerunning models.Source: §A.6 · p. 38
• The dashboard lets users examine the evaluation without rerunning any models.It also provides access to the underlying data for further use.Source: §A.6 · p. 38
• Users can click through from results to individual items and their source texts.Source: §A.6 · p. 38

A.7 Full Per-Occupation Heatmap

• 45 occupations are displayed from the 106 with at least three items; the released data covers 116 occupations, and the heatmap trends hold across the full set.Source: §A.7 · p. 38

A.8 Community-Thread Version

• The community-thread version uses upvoted answers from public occupational discussions, capturing what professionals ask in practice rather than what authoritative sources prescribe.It is created and reported separately from the 480-item authoritative dataset.Source: §A.8 · pp. 38–39
• The pipeline converts forum posts into six-option questions, using community top-voted answers as correct and four filtering stages for framing, relevance, distractors, and agreement.It samples one job per SOC group from O*NET and searches public discussion forums.Source: §A.8 · p. 39
• The community-thread bank contains 1,444 items across 758 occupations, averaging approximately 1.91 items per occupation.It prioritizes occupational coverage, allowing occupations with only two items.Source: §A.8 · p. 39
• The community-thread bank retains the frontier cluster seen on the authoritative bank and yields higher accuracy, as its questions are generally more answerable from general domain knowledge.This separate, forum-grounded stress test is not directly interchangeable with the authoritative-source bank.Source: §A.8 · pp. 39–40, Figure 12 · p. 40
• The model ordering remains unchanged in the SOC major-group leaderboard, which gives all major occupation groups equal weight.Figure 14 presents a cluster-level heatmap and a leaderboard that averages within groups before averaging across groups.Source: §A.8 · p. 40, Figure 14 · p. 42
• The ordering of the fifteen models remains the same under employment, annual mean wage, and employment-times-wage weighting.These variants change each occupation’s contribution to the final average, not the underlying model responses.Source: §A.8 · pp. 42–43, Figure 15 · p. 43
• The full community-thread heatmap includes occupations with at least three questions, excluding less stable one- or two-question cells.The stratified view samples occupations across the accuracy range; the full heatmap sorts eligible occupations by cross-model mean accuracy.Source: §A.8 · pp. 40, 42, 44