PaperlayerView exact revision on arXiv

Source-linked AI summary

NormasTCU --- A Brazilian Portuguese IR Dataset and an Evaluation of LLM-as-a-Judge for Relevance Assessment

Leandro Carísio Fernandes, Marcus Vinícius Borela de Castro, Leandro dos Santos Ribeiro, Leonardo Augusto da Silva Pacheco, Edans Flávius de Oliveira Sandes

arXiv:2608.27746v1cs.IR

TL;DR

Portuguese specialized IR lacks public datasets and affordable relevance assessment, while LLM reliability beyond English is uncertain. The paper introduces NormasTCU and evaluates LLM-generated relevance judgments against human assessments and downstream IR-system rankings. LLMs show positive scoring bias, yet rankings are often preserved for nDCG@10 and MRR but not P@10 or R@10.

  • Problem

    Evidence on LLM relevance assessment in non-English specialized domains is limited, while manual judging of large collections remains costly.

  • Method

    The paper releases NormasTCU and compares three LLMs using two prompting techniques with human judgments for query-document agreement and rankings of fifteen IR systems across four metrics.

  • Results

    LLMs show positive pair-level bias, while LLM-based system rankings remain highly correlated with human-reference rankings for nDCG@10 and MRR but diverge for P@10 and R@10.

  • Takeaways & Limitations

    LLM-generated judgments can support scalable Portuguese specialized-collection assessment when system evaluation uses nDCG@10 or MRR.

  • Takeaways & Limitations

    The experiment used six model-prompt combinations and proprietary APIs, so broader models or future model updates may produce different results and affect reproducibility.

Abstract

from arXiv · show

Portuguese Information Retrieval (IR) lacks public datasets, and relevance assessment for specialized collections remains costly. While Large Language Models (LLMs) increasingly support relevance assessment, their reliability in non-English specialized domains remains unclear. We introduce NormasTCU (https://huggingface.co/datasets/LeandroRibeiro/NormasTCU), a Brazilian Portuguese IR dataset with 14,469 legal documents, 46 queries, and 3,048 human judgments over 812 query-document pairs. Using NormasTCU, we evaluated LLM-as-a-judge for relevance assessment by prompting three models with two prompt techniques to grade these pairs. We then compared the rankings of 15 IR systems derived from LLM-generated and human reference qrels. LLMs consistently showed a positive scoring bias (mean absolute error: 0.46--0.66 on a 0-2 scale). Furthermore, pair-level agreement with human judgments achieved only fair to moderate levels, with Cohen's kappa ranging from 0.32 to 0.53. Despite this bias, LLM-generated judgments often yielded highly similar system rankings for nDCG@10 and MRR (observed Kendall's tau greater than or equal 0.90, although the bootstrap confidence intervals did not always remain above this threshold), but were less reliable for P@10 and R@10. Notably, LLM-based rankings were sometimes more strongly correlated with the reference ranking than individual human annotations were. As a practical implication, our results suggest that LLMs could effectively support scalable relevance assessment in specialized Portuguese corpora when evaluated using nDCG or MRR (rank-aware metrics), but they should be avoided when relying on precision or recall.

1 Introduction

Portuguese legal IR lacks sufficient public, domain-specific evaluation resources, while manual relevance assessment is costly and LLM reliability beyond English remains uncertain. The paper introduces NormasTCU and evaluates whether LLM-generated judgments align with humans and preserve IR-system rankings.

  • Dataset and motivation: Exhaustive manual judging is rarely feasible for modern corpora, and even judging document subsets remains time-consuming.Cranfield-style collections ideally assess relevance across the corpus using trained assessors and detailed guidelines.
  • Dataset and motivation: English-centered evidence leaves the consistency of LLM relevance assessment across languages and specialized domains unresolved.The paper targets this gap in Brazilian Portuguese legal retrieval.
  • Dataset and motivation: 14,469 legal documents, 46 queries, and four human annotators form the publicly released NormasTCU Brazilian Portuguese test collection.The release includes aggregated qrels and anonymized individual annotations.
  • Study objectives: The study compares human and LLM assessments at query-document and IR-system levels, including rankings from individual human annotators.The system-level analysis examines whether LLM-generated qrels preserve rankings based on aggregated human judgments.
  • Study objectives: The experiment finds positive pair-level scoring bias but high ranking correlation for nDCG@10 and MRR, unlike P@10 and R@10.The abstracted result supports evaluating LLM judgments with rank-aware metrics rather than precision or recall.

2 Related work

Portuguese IR resources include translated, general-purpose, and native legal collections, but they differ in corpus, query, and annotation design. NormasTCU extends prior work with four independent expert judgments and a broader LLM-based system-ranking evaluation.

  • Portuguese IR resources: Portuguese IR resources remain scarce, spanning CHAVE, translated mMARCO, and newer native Brazilian Portuguese datasets.CHAVE contains about 16,000 newspaper articles and 100 queries, while mMARCO offers about 8.8 million translated passages.
  • Portuguese IR resources: Quati provides approximately 10 million ClueWeb22 passages and 200 queries annotated through an LLM-based pipeline.Its labels were automatically generated and then sampled for manual assessment.
  • Legal collections: Ulysses-RFCorpus contains 105,681 legislative documents and 693 expert-annotated queries formulated as requests for legislative proposals.Its query formulation differs from keyword-based or question-based search queries.
  • Legal collections: JurisTCU contains 16,045 jurisprudential decisions and 150 keyword- or question-based queries, with LLM scores reviewed by one human annotator.Its hybrid annotation process contrasts with collections relying entirely on human experts.
  • LLM-based assessment: Prior LLM relevance studies report higher-than-human pair scores, while system rankings often remain correlated with human-based rankings.Portuguese system-level evidence previously covered two LLMs and four retrieval functions.
  • NormasTCU distinction: NormasTCU uses four independent expert judgments for identical pairs and evaluates three LLMs, two prompting strategies, and fifteen retrieval-system rankings.This broadens Portuguese system-level evidence beyond the previously reported two-model, four-function study.

3 NormasTCU – Dataset construction

NormasTCU is a Brazilian Portuguese legal IR test collection built from TCU normative documents, real user queries, pooled candidates, and independent expert judgments. Its construction balances scalable pooling with domain-informed annotation while acknowledging interpretive disagreement and unjudged documents.

  • 3.1 Document corpus: 14,469 normative documents from the Brazilian Federal Court of Accounts form the Brazilian Portuguese legal corpus.The collection uses the July 2024 public snapshot, with documents representing individual normative acts.
  • 3.2 Queries: Queries were drawn from frequent TCU searches over 12 months, then some were rewritten into semantically richer phrases, yielding 46 queries.The topics cover administration, governance, staff regulations, human resources, audit, and external control.
  • 3.3 Qrels (relevance judgments): Each query used a pool of 20 candidate documents selected from three two-stage retrieval pipelines and manual additions.The pipelines combined BM25 and semantic retrieval, reranked candidates with a Portuguese cross-encoder, and used list intersections before completing each pool.
  • 3.3 Qrels (relevance judgments): Four domain experts independently graded pooled query-document pairs as irrelevant, partially relevant, relevant, or cannot judge.Annotation used a custom application and took approximately one month; experts could add documents missing from the initial pool.
  • 3.3 Qrels (relevance judgments): Only seven additional query-document pairs were added, suggesting that pooling selected most potentially relevant documents.The added pairs were judged relevant or partially relevant by the suggesting annotator and were not assessed by the others.
  • 3.3 Qrels (relevance judgments): Annotators judged queries without an explicit search-intent statement, approximating real-world search but potentially increasing disagreement across interpretations.Different interpretations may also penalize documents that would be relevant under alternative intents.
  • 3.4 Quality of human annotations: 63% of 3,048 human judgments were irrelevant, while 54% of the 812 unique judged pairs reached full consensus.A further 28% showed weak disagreement and 18% strong disagreement, reflecting differing interpretations of query relevance.
  • 3.4 Quality of human annotations: Pairwise quadratic weighted Cohen’s κ ranged from moderate to substantial, with κ = 0.73 for A1–A2 and κ = 0.55 for A1–A3.The paper treats these levels as acceptable for graded relevance judgments, where subjectivity makes disagreement expected.

4 Evaluating LLM-as-a-judge for IR relevance assessment

The study compares human and LLM relevance judgments at the query-document and system-ranking levels, finding positive scoring bias but strong metric-dependent ranking agreement. LLM-derived rankings generally align with reference rankings for nDCG@10 and MRR, but diverge more for P@10 and R@10.

  • Method: Three models used simple and rationale prompts, with each model-prompt combination treated as an independent LLM annotator.The models were DeepSeek-V3.2, gpt-5-mini, and sabiazinho-4.
  • Query-document assessment: LLMs assigned fewer score-0 labels and more score-1 and score-2 labels than human annotators, indicating positive relevance bias.Bias was measured against the average human score using mean error and mean absolute error.
  • Query-document assessment: 0.46 was deepseek-3.2 simple’s MAE, the lowest bias among the tested models, while gpt-5-mini and sabiazinho-4 showed stronger positive bias.The deepseek-3.2 mean error was 0.08, indicating relatively balanced deviations despite the MAE.
  • Query-document assessment: 0.02 points was the maximum MAE difference between simple and rationale prompts across models, showing no clear prompting advantage for bias reduction.Rationale slightly improved gpt-5-mini but worsened deepseek-3.2 and sabiazinho-4.
  • Query-document assessment: 0.32 ≤κ≤0.53 characterized LLM-human agreement, ranging from fair to moderate across annotator pairs.Within-group LLM agreement was higher, with κ ranging from 0.63 to 0.91.

5 Discussion of the results, implications, and limitations of the study

Human disagreement was common in NormasTCU, while LLMs showed positive scoring bias whose usefulness depended strongly on the evaluation metric and prompting strategy. The study therefore supports cautious use of LLM-generated judgments for rank-aware system evaluation, subject to dataset, model, and prompt limitations.

  • Discussion of the results: 28% of query-document pairs showed weak human disagreement, while 18% showed strong disagreement involving scores of both 0 and 2.The authors attribute part of this variability to legitimate semantic differences rather than annotation noise or insufficient expertise.
  • Discussion of the results: LLMs consistently assigned higher relevance scores than humans, and deepseek-3.2 achieved the lowest MAE among evaluated models.The simple and rationale prompts did not differ significantly in pair-level bias.
  • Discussion of the results: nDCG@10 and MRR preserved highly correlated system rankings, whereas P@10 and R@10 produced substantial deviations.The rationale prompt was consistently functionally equivalent to the reference ranking for nDCG@10 and MRR.
  • Discussion of the results: The strongest end-to-end results came from gpt-5-mini and sabiazinho-4, showing that lower pair-level annotation error need not preserve system rankings best.This contrasts with deepseek-3.2's lowest MAE at the pair-annotation level.
  • Practical implications: In specialized domains with few relevant documents, LLM-generated qrels may support system comparison with nDCG@10 and MRR but are inadequate for P@10 or R@10.The discussion links the divergence to the shallow @10 cutoff and scarcity of relevant documents per query.
  • Practical implications: The rationale prompt should be preferred for system ranking, although exhaustive annotation by multiple domain experts remains the gold standard.With only one human annotator available, LLM judgments may provide a stable baseline under the study's examined conditions.
  • Limitations of the study: The experiment used only 46 queries, was conducted solely on NormasTCU, and covered six model-prompt combinations accessed through proprietary APIs.These constraints limit generalization, broader model coverage, and long-term reproducibility.
  • Limitations of the study: Only simple and rationale prompting techniques were evaluated, so other prompts could produce different bias patterns and system rankings.The authors identify prompt design as a possible mechanism for mitigating model bias.

6 Conclusion

The paper introduces NormasTCU as a released Brazilian Portuguese legal IR benchmark and evaluates LLM-generated relevance judgments for ranking retrieval systems. Results show metric-dependent consistency, motivating broader legal collections and further evaluation.

  • Conclusion: NormasTCU contains 14,469 legal documents, 46 queries, and graded judgments from four domain experts.The corpus comprises normative acts from Brazil's Federal Court of Accounts and includes aggregated qrels and anonymized individual annotations.
  • Conclusion: LLMs showed positive pair-level scoring bias but produced highly consistent system rankings for nDCG@10 and MRR, not for P@10 and R@10.The experiment compared three LLMs against human annotations when ranking fifteen IR systems.
  • Conclusion: Future work should build collections for other heterogeneous legal databases, including jurisprudence, contracts, legal opinions, and administrative decisions.The authors highlight legal search and case-law recommendation as applications involving a preliminary IR stage.

Appendix A Queries

Appendix A presents the NormasTCU queries in Portuguese alongside English translations, grouped into descriptive categories across continued table pages.

  • Appendix A Queries: Table A.1 lists original Portuguese queries and their English translations, grouped into categories for descriptive purposes only.The table continues across subsequent pages.
  • Appendix A Queries: Table A.1 continues from the previous page.
  • Appendix A Queries: Table A.1 continues from the previous page.

Appendix B Prompts

Appendix B documents the original Portuguese and translated English prompts used for relevance judgment, including simple and rationale-based variants.

  • Appendix B Prompts: Figures 8 and 9 show the original Portuguese simple and rationale relevance-judgment prompts.
  • Appendix B Prompts: Figures 10 and 11 show English translations of the simple and rationale relevance-judgment prompts.

Appendix C Evaluation metrics for across qrels

Figures 12–15 compare P@10, R@10, nDCG@10, and MRR across qrels for each IR system. The generally non-monotonic lines indicate that systems can change relative ranking depending on the qrels and metric used.

  • Evaluation metrics across qrels: Figures 12–15 report P@10, R@10, nDCG@10, and MRR for every IR system under different qrels.The systems are ordered on the x-axis according to their reference-qrels ranking.
  • Evaluation metrics across qrels: Under perfect correlation, the plotted lines would decrease monotonically across the reference-qrels system order.
  • Evaluation metrics across qrels: The lines generally decrease but are not monotonic, showing that some systems change relative ranking across qrels and metrics.
Loading 2608.27746v1…