PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System

Rubén Manrique, Michelle Castellanos, Jorge Morales, Juan David Gutiérrez, Antonio Barreto Rozo, Joaquín Vélez Navarro

arXiv:2610.03639v1cs.AIcs.CY

TL;DR

LLMs can produce fluent but incorrect or fabricated legal content. This paper builds an expert-validated benchmark for Colombian law and evaluates reliability across question formats; closed-question performance tracks model rankings but overstates free-text reliability.

  • Problem

    LLMs can produce fluent but incorrect or fabricated legal content, including citations to non-existent statutes and case law.

    Source: §1 · p. 2

  • Method

    The authors build an expert-validated Colombian legal benchmark and use reference-anchored, two-judge-averaged RAGAS correctness to assess free-text legal quality.

    Source: §1 · p. 2, §5.3 · p. 12

  • Results

    Closed-question accuracy tracks model ranking but overstates reliability, while free-text factual correctness never exceeds 0.45.

    Source: §9 · p. 33

  • Takeaways & Limitations

    The authors argue against unsupervised use and favor expert verification, with authoritative-source grounding as a promising direction for improvement.

    Source: §9 · p. 33

    The open-ended subset contains 55 items, and international and environmental law are absent.

    Source: §8.6 · p. 32

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used to support legal practice, education, and research, yet their reliability in national legal systems outside the United States remains largely undocumented. We introduce an expert-validated benchmark for evaluating LLM reliability on the Colombian legal system. The benchmark comprises 1,042 items spanning ten areas of law and three question formats (closed multiple-choice, semi-open, and open-ended IRAC), built through a human-in-the-loop pipeline with multi-stage expert review. We evaluate 15 contemporary proprietary and open-weight models with format-appropriate metrics. Accuracy on closed questions ranges widely, from 0.905 (Gemini 3.1 Pro) to 0.577, but on free-text legal answers factual correctness never exceeds 0.45 (on a 0-1 scale) for any model. We find a dissociation between answer relevancy and correctness (Spearman rho = -0.46): models reliably sound responsive while frequently being wrong, a pattern of particular concern for non-expert users. Closed-question accuracy and free-text correctness are strongly rank-correlated (rho = 0.94), so cheap multiple-choice screening predicts model ranking but overstates absolute reliability. An independent rubric-based LLM judge and blind human expert scoring both reproduce the free-text ranking (rho >= 0.88). The judge further reveals that only about half of the norms models cite are correct; the rest are wrong or non-existent. Reliability varies systematically by legal area and follows an inverted-U across question complexity. Our results indicate that current LLMs require expert supervision for Colombian legal tasks, and that grounding answers in authoritative sources is a promising path to higher reliability. We release the benchmark construction pipeline to support reproducible evaluation.

1 Introduction

LLM use in legal work is growing alongside documented risks of fluent but incorrect legal content, yet evidence on reliability in Colombian law remains scarce. The paper addresses this gap with an expert-validated benchmark and evaluation of contemporary models.

1 Introduction

  • LLMs can produce fluent but incorrect or fabricated legal content, including citations to non-existent statutes and case law.Studies cited in the introduction report hallucination rates above 50% for general-purpose models and error rates of 17–33% for commercial RAG legal tools.Source: §1 · p. 2
  • Empirical evidence on LLM legal reliability is concentrated in English-language U.S. common law, leaving Spanish-speaking civil-law jurisdictions and Colombian law particularly under-studied.The prior Colombian assessment covered limited areas and question types and predates the current frontier-model generation.Source: §1 · p. 2
  • The benchmark contains 1,042 expert-validated items across ten legal areas and three formats, built through a human-in-the-loop pipeline with multi-stage expert review.Source: §1 · p. 2
  • The paper evaluates 15 contemporary proprietary and open-weight models and analyzes failures, including the dissociation between answer relevancy and correctness.Source: §1 · p. 2

2 Related Work

Prior work documents legal hallucinations and AI adoption in judicial settings, while systematic Spanish-language evaluation remains limited. This paper builds on benchmark and evaluation precedents by focusing on Colombian law across question formats and expert validation.

2.1 Legal hallucinations and the reliability of LLMs

  • The paper adopts reference-anchored LLM judging and separates recognition on closed questions from generation on free-text answers in a non-U.S. civil-law jurisdiction.Earlier work includes studies of legal hallucinations, RAG-based tools, legal reasoning benchmarks, and LLM judges for open-ended answers.Source: §2.1 · pp. 2–3

2.2 AI in the Justice System in Latin America and Colombia

  • Research on AI in Latin American and Colombian justice maps adoption and governance but does not systematically measure how reliably current LLMs answer Colombian legal questions.Source: §2.2 · p. 3

2.3 Spanish-language legal QA in Latin America

  • Manrique et al. identify the lack of a dedicated Spanish-language legal QA benchmark and evaluate five models only on closed multiple-choice questions.The present benchmark responds with Colombian-law coverage, semi-open and IRAC formats, multi-stage expert validation, and a larger, more recent model set.Source: §2.3 · p. 3
  • Compared with the earlier Colombian assessment, the paper evaluates 15 models on a larger, broader benchmark and separates factual correctness from surface relevancy.The cited precedent evaluated three models across five areas and did not release a reusable benchmark.Source: §2.3 · p. 3

3 The Benchmark

The Colombian Legal Reliability Benchmark is designed to probe legal knowledge and reasoning across a progression of question formats, legal areas, and difficulty levels. Its item types use different scoring approaches suited to their response formats.

3 The Benchmark

  • The benchmark comprises 1,042 expert-validated Colombian legal questions designed to probe knowledge and reasoning at increasing difficulty.Source: §3 · p. 3

3.1 Question taxonomy

  • Closed multiple-choice items test foundational legal knowledge, semi-open items require concise answers at three complexity levels, and open-ended IRAC cases require synthesis of legal competencies.IRAC describes the authoring of open-ended items and reference answers; the model prompt does not name IRAC, and scoring uses reference-answer content.Source: §3.1 · p. 4
  • Semi-open questions target fixed catalogues of legal sub-tasks at each complexity level, adapted from Dahl et al. [4] to Colombian law.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 Legal areas

  • The benchmark spans ten areas, including constitutional, administrative, criminal, labor, civil, commercial and corporate, procedural, tax, family, and markets law.Contributors selected areas matching their expertise.Source: §3.2 · p. 5

3.3 Dataset composition

  • The evaluated benchmark contains 305 closed, 682 semi-open, and 55 open-ended items; semi-open items comprise 258 low-, 236 medium-, and 188 high-complexity questions.Figure 1 presents composition by legal area and question type; Figure 2 shows the complexity distribution.Source: §3.3 · p. 5, Figure 1 · p. 5, Table 2 · p. 5

3.4 A note on evaluation metrics

  • Closed questions are scored by accuracy, while free-text responses use text-generation, answer-quality, and Spanish-adapted readability metrics computed on different benchmark subsets.Readability indices characterize answer style rather than correctness.Source: §3.4 · p. 6

4 Dataset Construction Methodology

The dataset was assembled through a human-in-the-loop pipeline combining student-generated questions with multi-stage expert validation. A shared platform and approval protocol governed item creation, review, and inclusion.

4.1 The Horizontes Conectados platform

  • Horizontes Conectados organizes dataset construction through role-based work and modules for questions, prompts, model jobs, generated responses, and monitoring.Students generate questions, a legal assistant performs primary review, specialized evaluators validate legal rigor, and a technical team administers the system.Source: §4.1 · pp. 6–7

4.2 Workshops

  • Five workshops held between November 2025 and March 2026 involved volunteer undergraduate and graduate law students, with the two November in-person sessions producing the most questions.Source: §4.2 · p. 7
  • Of 1,084 questions received, 415 came from workshops and 669 from the legal team and individual students; 42 were archived, leaving 1,042 benchmark items.All sources entered the same repository and underwent the same validation protocol.Source: §4.2 · p. 7

4.3 Quality-assurance protocol

  • Every item followed reception, legal-assistant assignment, expert evaluation, and committee decision, with edits, requested adjustments, or archival possible along the way.The protocol processed 1,084 items and archived 42; 971 approved items (93%) were edited at least once, and three evaluators covered ten legal areas.Source: §4.3 · p. 8, Figure 4 · p. 9

5 Experimental Setup

The study evaluates 15 contemporary LLMs on shared Colombian-law questions using format-specific prompts and metrics. Responses are scored across the same items, with free-text legal quality assessed against expert references.

5 Experimental Setup

  • 15 models answered the same benchmark items without internet access or external tools, so comparisons reflect their parametric knowledge of Colombian law.Source: §5 · p. 10

5.1 Evaluated models

  • The evaluated models span proprietary and open-weight families and include general-purpose and reasoning-oriented systems.Source: §5.1 · p. 10, Table 4 · p. 10

5.2 Prompting strategy

  • Versioned prompts require Colombian legal grounding and structured JSON, with dedicated response formats for multiple-choice, semi-open, and open-ended questions.Source: §5.2 · pp. 10–11

5.3 Evaluation metrics

  • Question formats use different metrics over different benchmark subsets; closed items are scored by accuracy, while free-text answers are compared with expert references.Source: §5.3 · p. 11, Table 5 · p. 12
  • Reference-anchored RAGAS answer correctness, averaged across two independent judges, is the primary measure of free-text legal quality.BERTScore, BLEU, and ROUGE measure resemblance rather than legal validity and serve as supporting signals.Source: §5.3 · pp. 11–12
  • A separate rubric-based judge scores idea coverage, contradictions, cited norms, error types, and legal adequacy on a stratified subset, with blind human scoring used for validation.Source: §5.3 · p. 12
  • Spanish-adapted readability indices describe prose complexity but are excluded from the reliability ranking.Source: §5.3 · p. 12

5.4 Evaluation procedure

  • The same item set was presented to every model, and results were aggregated overall and by legal area and, for semi-open items, by complexity.Source: §5.4 · pp. 12–13

6 Results on closed questions

Closed-question performance varies substantially across models, with consistent differences across legal areas and substantial overlap in which items models answer correctly. Shared errors limit the benefit of majority voting.

6.1 Overall performance on closed questions

  • 0.905 (Gemini 3.1 Pro) to 0.577 (Command R) spans closed-question accuracy, with all models above the 0.25 random baseline.The 33-point gap separates the best and worst models.Source: §6.1 · p. 13, Figure 5 · p. 14

6.2 Performance across legal areas

  • Accuracy is highest in labor (0.81), tax (0.80), and civil law (0.79), and lowest in procedural and administrative law (0.71 each).Model rankings remain largely stable across areas; weaker models degrade more in harder areas.Source: §6.2 · pp. 13–14, Figure 6 · p. 15

6.3 Item difficulty and inter-model agreement

  • Of 305 closed items, 95 (31.1%) were answered correctly by all 15 models, 205 (67.2%) split models, and only 5 (1.6%) were missed by every model.Other reported difficulties included post-cutoff case law, systematically missed meta-options, and one internally inconsistent item flagged for correction.Source: §6.3 · pp. 14–15

6.4 Correlated errors and the limits of ensembling

  • 87% of the time on average, models that answered an item incorrectly selected the same distractor, indicating shared rather than idiosyncratic errors.Source: §6.4 · p. 17
  • 0.843 accuracy for a 15-model majority vote falls below the best single model’s 0.905, despite at least one model being correct on 98.4% of items.The authors attribute the gap to correlated errors and state that unsupervised aggregation does not recover the oracle result.Source: §6.4 · p. 17, Figure 8 · p. 17

6.5 Model similarity and clustering

Model agreement clusters align more with capability than provider, while statistical testing groups models into tiers rather than a strict ranking. Proprietary models lead on average, and item-response modeling adds a difficulty scale but has limited precision at the extremes.

6.5 Model similarity and clustering

  • The strongest models form a tight agreement cluster, while Command R and Nova Lite join only at the largest distance; clusters track capability more than provider.Source: §6.5 · p. 17

6.6 Statistical significance of model differences

  • Gemini 3.1 Pro is significantly more accurate than every other model, but several other models form statistically indistinguishable tiers.Within-tier orderings should not be over-interpreted.Source: §6.6 · pp. 17–18, Figure 10 · p. 19

6.7 The effect of model attributes

  • 0.804 vs. 0.716 is the average accuracy of proprietary versus open-weight models; reasoning-oriented models average 0.702 versus 0.752 for general-purpose models.The reasoning-model comparison is exploratory with only two such models; the authors characterize this benchmark as dominated by recall and recognition.Source: §6.7 · p. 18, §6.8 · p. 18, Figure 11 · p. 19

6.8 A latent difficulty scale via item response theory

  • The 2PL item-response model’s estimated abilities correlate with raw accuracy at Spearman ρ = 0.99, while its difficulty map places a long tail of hard, high-discrimination items.With only 15 model responses per item, parameter estimates are limited in precision and the map is interpreted qualitatively.Source: §6.8 · pp. 18, 20, Figure 12 · p. 20

7 Results on free-text generation

Free-text results show low, variable correctness despite high relevancy, with reliability also differing by question complexity, legal area, and model. Independent scoring supports the model ranking, while answer length and readability qualify comparisons.

7.1 Free-text answer quality

  • RAGAS free-text correctness peaked at 0.451 for GPT-5.4; most models scored 0.31–0.45 across 737 open and semi-open items.BERTScore was nearly constant across models (0.699–0.717), providing little discrimination.Source: §7.1 · p. 20
  • Across 11,055 answers, correctness averaged 0.373 (SD = 0.199), with 39.2% below 0.3; every model showed a wide spread.The ten hardest items were all semi-open, and seven hinged on exact recall; the hardest had mean correctness 0.04.Source: §7.1 · pp. 21–22, Figure 13 · p. 21, Table 8 · p. 23
  • Relevancy poorly tracked correctness (ρ = −0.46 across models; ρ = 0.08 per answer), while model rankings formed tiers and correlated across formats (ρ = 0.94).The leading models were statistically indistinguishable from each other, and adjacent model intervals often overlapped.Source: §7.1 · pp. 21–22, Figure 14 · p. 22, Figure 15 · p. 23

7.2 Effect of question complexity

  • Semi-open correctness followed an inverted-U across complexity, peaking at medium and falling at low and high levels for nearly all models.Exact legal facts such as deciding judge and normative existence were among the weakest sub-tasks, alongside reasoning-heavy tasks.Source: §7.2 · p. 24, Figure 16 · p. 24, Figure 17 · p. 25

7.3 Free-text quality by legal area

  • Free-text correctness varied by legal area: administrative law was lowest at 0.31, while markets law (0.45) and criminal law (0.42) were strongest.Area rankings only partly matched closed-question results; leading models performed consistently across areas.Source: §7.3 · pp. 24–25, Figure 18 · p. 26

7.4 Reliability of legal citations

  • Only 53.2% of cited norms were correct; 45.2% were wrong and 1.6% non-existent, while human verification agreed with the judge’s wrong-or-invented flag in 91% of cases.Citation accuracy ranged from 90.3% for GPT-5.4 to 23.1% for Nemotron 49B.Source: §7.4 · pp. 25–26, Figure 19 · p. 27

7.5 Convergent validity and human validation

  • The rubric judge and RAGAS ranked models similarly (ρ = 0.97), and blind human scoring reproduced the judge’s ranking (ρ = 0.88).The judge was systematically stricter than the human team by about half a point, so its absolute scores read low.Source: §7.5 · pp. 26–27, Figure 20 · p. 28, Figure 21 · p. 29

7.6 Answer length and readability

  • Answer length correlated with correctness (ρ = +0.78 across models), while models wrote denser-than-reference prose, especially on easy questions; gaps closed at high complexity.Length is a confounder in comparing open and semi-open answers; density was only weakly related to quality once length was accounted for.Source: §7.6 · pp. 28–29, Figure 22 · p. 30, Figure 23 · p. 30

8 Discussion

The discussion finds a gap between strong structured-question performance and reliable free-text legal analysis, where relevant-sounding answers can still be wrong. It recommends expert verification and source grounding while noting limits in coverage and evaluation.

8.1 Reliability remains below the threshold for unsupervised legal use

  • No model exceeds 0.451 RAGAS correctness on free-text answers, and only 53.2% of cited norms are correct, despite Gemini 3.1 Pro's 0.905 closed-question accuracy.Source: §8.1 · p. 31

8.2 Fluency does not imply correctness

  • Across models, relevancy (0.73–0.82) trends opposite correctness (ρ = −0.46), making responsiveness a poor correctness proxy for non-experts and high-stakes workflows.Source: §8.2 · p. 31

8.3 Closed-question accuracy overstates reliability but ranks models well

  • Closed-question accuracy and free-text correctness rank models similarly (ρ = 0.94), but a 0.86 multiple-choice score may mean fewer than half of free-text answers are correct.Source: §8.3 · p. 31

8.4 Where and why models fail

  • Accuracy is lowest in procedural and administrative law (0.71) and highest in labor, tax, and civil law (0.79–0.81); free-text quality is weakest in administrative law.Source: §8.4 · p. 32
  • Semi-open correctness peaks at medium complexity, falling at high reasoning demands and low citation-precision demands; authoritative-source grounding is a promising route to higher reliability.Source: §8.4 · p. 32

8.5 Implications

  • For Colombian practice, the authors recommend expert verification, especially for citation-precise and reasoning-intensive tasks; the benchmark separates recognition from generation using format-appropriate metrics.Source: §8.5 · p. 32

8.6 Limitations

  • Interpretation is bounded by 55 open-ended items, two absent legal areas, a time-specific benchmark, single-run prompting, and no formal inter-annotator agreement.Source: §8.6 · pp. 32–33

9 Conclusion

The paper presents an expert-validated benchmark and reproducible evaluation of LLM reliability on Colombian law. It finds that strong closed-question performance does not translate into reliable free-text legal answers, supporting expert verification and source grounding.

9 Conclusion

  • The benchmark contains 1,042 items across ten legal areas and three question formats, and the evaluation covers 15 contemporary models.Source: §9 · p. 33
  • Free-text correctness remains low and decoupled from answer relevance, while closed-question accuracy tracks model rankings but overstates practical reliability.Source: §9 · p. 33
  • The authors argue for expert verification, especially on citation-precise and reasoning-intensive tasks, and identify authoritative-source grounding as a promising improvement direction.Source: §9 · p. 33

A Example Questions

The examples span closed, semi-open, and open-ended questions, with semi-open items illustrating three complexity levels. The hardest examples expose failures of exact legal recall, alongside more complex questions requiring legal analysis.

A Example Questions

  • The examples also include a closed commercial-law question whose correct answer is absolute nullity for stipulating against mandatory rules.Source: A Example Questions · p. 33
  • A high-complexity commercial-law example asks whether the commercial agent’s termination payment conflicts with civil-law waiver rules; the cited court treats the rules as complementary and the payment as waivable subject to good faith and validity requirements.Source: A Example Questions · p. 34
  • An open-ended ransomware scenario calls for IRAC analysis of potentially concurrent cybercrime offenses and data-protection rules governing an international transfer of personal data.Source: A Example Questions · p. 34
  • Among the ten lowest-scoring benchmark items, the sharpest failures involved exact recall of an authority’s name or a precise article rather than legal reasoning.Source: A Example Questions · p. 34
  • Mean correctness was 0.07 for naming the reporting magistrate in ruling T-323 of 2024, 0.09 for identifying a statutory article, and 0.10 for recalling a recent tax ruling’s holding.The tax ruling post-dated the models’ training horizon.Source: A Example Questions · pp. 34–35

B Prompt Templates

The model prompts use separate, version-controlled Spanish templates for each question format and require structured JSON answers grounded in Colombian legal sources. The open-ended template’s four-part response structure corresponds to the Rule–Application–Conclusion structure of the IRAC reference answers.

B Prompt Templates

  • Each question format uses a dedicated, version-controlled Spanish prompt that casts the model as a Colombian-law expert and requires a structured JSON response grounded in applicable Colombian sources.Source: B Prompt Templates · p. 35
  • The multiple-choice template requires analysis of four options, selection and justification of the correct answer, and brief reasons for rejecting the others.Source: B Prompt Templates · p. 35
  • The semi-open template asks for a concise three-to-five-sentence answer, key legal concepts and terms, and a relevant Colombian legal citation.Source: B Prompt Templates · pp. 35–36
  • The open-ended template requests a four-part JSON response—normative framework, analysis, case law, and conclusion—that maps onto the Rule–Application–Conclusion structure of the IRAC reference answers.Source: B Prompt Templates · p. 36

Related papers

Loading 2610.03639v1…