PaperlayerView exact revision on arXiv

Source-linked AI summary

Legal Guidance in the Wild: How Users Seek Legal Help in Real-World LLM Conversations

Amirreza Mohseni, Gerasimos Spanakis

arXiv:2610.25159v1cs.CLcs.CY

TL;DR

People may discuss legal matters without seeking practical guidance, leaving unclear when real-world conversational AI users seek help and how they formulate requests. The paper builds a taxonomy, applies it to WildChat conversations, and compares classifiers on a human-reviewed evaluation set. Legal-content conversations often seek guidance, while model differences are not statistically significant and vary by task.

  • Problem

    Legal-themed conversation does not establish whether a person seeks practical guidance, and prominent legal NLP benchmarks reveal less about when and how people seek it in the wild.

    Source: §1 · p. 1

  • Method

    The paper applies a multi-axis taxonomy to 2,122 WildChat-1M conversations, creates a human-reviewed 200-conversation evaluation set, and compares a lexical cascade with ModernBERT models.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    ModernBERT-large performs best overall, but TF–IDF performs similarly on binary guidance detection while dense models lead on topic classification; no paired comparison reaches significance.

    Source: §7 · p. 9, §8 · p. 10

  • Takeaways & Limitations

    The taxonomy supports large-scale study of legal guidance requests, while the released annotations and evaluation artifacts support reproducible research and future model development.

    Source: §1 · p. 2, §8 · p. 10

    The enriched sampling design cannot estimate WildChat prevalence, and the gold set’s 84 guidance-positive cases cover only 12 observed topics.

    Source: §3.3 · p. 5, §7 · p. 10

Abstract

from arXiv · show

People ask conversational AI models about their legal rights, obligations, procedures, and personal situations that may have legal consequences. To study this behavior, we first need to distinguish requests for legal guidance from other legally themed discussions and unrelated conversations. We contribute a multi-axis taxonomy of legal queries capturing whether legal guidance is sought, how a request is formulated, and which legal domains it concerns; apply it to 2,122 naturally occurring WildChat-1M conversations; and create a human-reviewed gold evaluation set of 200 conversations. All annotations, model checkpoints, and evaluation code are publicly available. In this enriched sample, 71.1% of conversations containing legal content seek personalized or practical guidance. Rights-and-options advice is the most common primary request type among legal-content conversations, and 25.3% of guidance-positive conversations span multiple legal topics. The gold evaluation set supports a two-stage comparison of ModernBERT-base and ModernBERT-large against a TF-IDF cascade. Our framework supports large-scale studies of legal guidance in conversational AI, and the public release of the annotated data facilitates reproducible research and future model development.

1. Introduction

Legal content alone does not show that someone is seeking practical help, yet prominent legal NLP benchmarks reveal less about when people seek guidance in real conversations. The paper introduces a taxonomy and applies it to naturally occurring conversations to characterize guidance requests and compare classifiers.

1. Introduction

  • Prominent legal NLP benchmarks evaluate documents, retrieval, answer generation, or expert-defined reasoning, revealing less about when people seek guidance in the wild.The open question concerns how requests for legal guidance are formulated in real conversations.Source: §1 · p. 1
  • Legal content is not synonymous with guidance seeking: users may ask for translation or academic explanation, or seek concrete rights, options, or next steps.The classification task first identifies personalized or practical guidance and then assigns positive conversations to one of 14 primary legal topics.Source: §1 · pp. 1–2
  • A compact guidance detector could support empirical research or, after calibration and user testing, trigger a notice that generated output is not professional advice.The motivation is that public-facing LLMs can confidently produce unreliable legal information.Source: §1 · p. 2
  • The paper develops a multi-axis taxonomy and applies it to 2,122 WildChat-1M conversations, releasing a 200-conversation human-reviewed gold set and supporting artifacts.It compares a lexical cascade with ModernBERT-base and ModernBERT-large on the gold set after finalizing modeling decisions using separate silver-labeled development data.Source: §1 · p. 2

2. Related Work

Prior work examines legal-help AI, citizen legal questions, and legal NLP benchmarks, but this paper focuses upstream on identifying guidance seeking in naturally occurring, multi-turn conversations. It separates legal content from guidance seeking and adds hierarchical topic classification.

2. Related Work

  • Prior studies examine queries to a legal assistant, expectations for legal-help AI, and LLM support for civil legal-aid intake [1].These works distinguish legal information from advice, investigate desired safeguards, or support eligibility checks and follow-up questions.Source: §2 · p. 2
  • Health-information research provides a methodological analogue for studying how and why people seek help in large conversational datasets.This paper adopts a user-centered perspective for law while separating legal content from guidance seeking and adding hierarchical topic classification.Source: §2 · p. 2
  • Legal NLP benchmarks cover document understanding, statutory retrieval, and expert- or lawyer-annotated answers.These fixed-question resources inform question language, topic distributions, retrieval, and answer generation, whereas this study examines intent and topics in naturally occurring conversations.Source: §2 · p. 2
  • The classifier identifies conversations where safeguards, specialized routing, or human escalation may be warranted, but it does not assess answer correctness.Strong benchmark performance does not make a public model a reliable legal adviser; prior work documents frequent hallucinations and confident acceptance of false premises.Source: §2 · p. 3

3. Data and Task

The study constructs an enriched sample of English-language WildChat conversations, annotates them with a three-axis taxonomy, and reserves a human-reviewed subset for evaluation. It evaluates a two-stage classifier while noting limits on prevalence and topic-evaluation evidence.

3.1. Dataset construction, taxonomy, and silver annotation

  • The retained 2,122 English-language conversations were screened and sampled from WildChat-1M, then annotated by GPT-5.6 Sol under the taxonomy.The silver labels identify 1,367 conversations with legal content and 972 seeking guidance (45.8%); the enriched sample cannot estimate WildChat prevalence.Source: §3.1 · p. 3
  • The taxonomy records legal relevance, interaction context and request types, and legal domains; its domain axis has 13 substantive topics plus “OTHER.”Each conversation has one predominant primary topic and may have secondary topics; the classification pipeline predicts only the primary topic after guidance detection.Source: §3.1 · p. 3
  • The fixed 200-conversation gold subset contains 84 guidance-positive and 116 guidance-negative cases; the other 1,922 conversations form silver development data.The development set is split into 1,632 training and 290 validation conversations; Table 1 distinguishes silver labels from final human-reviewed guidance labels.Source: §3.1 · p. 4, Table 1 · p. 4

3.2. Human review and gold evaluation set

  • Two computer-science reviewers assessed silver guidance and topic labels in all 200 gold-set conversations, resolving disagreements by discussion into final labels.They agreed fully on 186 conversations (93.0%); guidance agreement was 96.5% with κ = .927, and topic agreement was 94.9% with κ = .561.Source: §3.2 · p. 4

3.3. Task and metrics

  • The two-stage task detects guidance first, then assigns positive conversations one of 14 topics; final labels are one of those topics or “NO GUIDANCE.”Accuracy and macro-F1 are reported for guidance detection, topic classification, and end-to-end evaluation; macro-F1 weights classes equally.Source: §3.3 · pp. 4–5
  • The gold topic evaluation covers only 12 of 14 topics among 84 guidance-positive conversations, with five topics represented by five or fewer examples.Topic macro-F1 is computed over the 12 observed classes.Source: §3.3 · p. 5

4. How Users Seek Legal Guidance

In the enriched sample, most legal-content conversations seek practical guidance, with rights-and-options advice the most common primary request type. Guidance spans multiple topics in about one quarter of positive conversations, and often begins in the opening turn.

4. How Users Seek Legal Guidance

  • 71.1% of legal-content conversations seek guidance (972 of 1,367); the enriched sampling design means these percentages are not WildChat prevalence estimates.The remaining 395 legal-content conversations do not seek guidance, while 755 conversations contain no legal content.Source: §4 · p. 5
  • 52.1% of legal-content conversations have rights-and-options advice as their primary request type, followed by information or explanation at 21.4%.Procedural how-to accounts for 13.2%, document drafting 10.0%, other requests 1.9%, and document review 1.4%.Source: §4 · p. 5
  • 25.3% of guidance-positive conversations receive more than one topic label; Immigration & Citizenship is the largest primary topic at 14.1%.Criminal Law accounts for 11.4%, Intellectual Property 11.3%, Business & Commercial 11.0%, and Tax Law 10.8%; secondary labels increase apparent coverage of cross-cutting domains.Source: §4 · p. 5, Table 2 · p. 6
  • 68.6% of guidance-seeking conversations begin with guidance in the opening turn; the onset annotations are not independently validated and are descriptive only.The annotator flags medium or high uncertainty for 20.7% of conversations labeled guidance-positive, versus 7.6% of the remaining conversations; predictive value was not tested.Source: §4 · p. 6

5. Classification Models and Early Prediction

The study compares lexical and contextual classifiers for full-conversation and prefix-based legal-guidance detection, using a locked gold evaluation and exploratory early-turn analysis.

5.1. Experimental setup

  • ModernBERT-base and ModernBERT-large provide contextual classifiers alongside a word- and character-level TF–IDF logistic-regression baseline.The comparison examines whether increased model size consistently improves performance.Source: §5.1 · p. 6
  • Class-weighted topic training and macro-averaged evaluation address the highly imbalanced topic distribution.Training and validation guidance rates are similar, at 46.0% and 46.2%, respectively.Source: §5.1 · p. 6
  • Prefix models use user messages through each turn, labeling prefixes before guidance begins negative and those at or after onset positive.Each conversation’s prefixes are weighted so their total weight is 1; no prefix topic classifier is trained.Source: §5.1 · pp. 6–7

5.2. Evaluation

  • Evaluation decisions were frozen before the locked gold-set evaluation; exploratory turn-1 and turn-2 analyses use frozen prefix models and thresholds.The study reports 95% bootstrap intervals from 2,000 conversation-level resamples and paired McNemar tests.Source: §5.2 · p. 7

6. Results

On the locked 200-conversation gold evaluation, ModernBERT-large leads full-conversation guidance detection and end-to-end routing, while dense models show their clearest advantage over TF–IDF in topic classification. Comparisons are qualified by exploratory prefix results, overlapping intervals, and sparse topic classes.

6.1. Stage 1: Guidance detection

  • ModernBERT-large reaches .845 accuracy and macro-F1 on full-conversation guidance detection; TF–IDF reaches .840 accuracy, a difference of one conversation in 200.Table 4 reports guidance detection on the 200-conversation gold evaluation set.Source: Table 4 · p. 8, §6.1 · p. 8
  • ModernBERT-large identifies 79 of 84 guidance-positive conversations (.940 recall), compared with 75 for TF–IDF and 71 for ModernBERT-base, while producing 26 false positives.Source: §6.1 · p. 8
  • In exploratory prefix results, TF–IDF has the highest turn-1 macro-F1 (.826), while ModernBERT-large leads turn 2 (.865); the columns use different conversation sets and eventual-guidance labels.Final-turn prefix ModernBERT-large scores are .860 accuracy and .859 macro-F1; these prefix results do not measure improvement over time or guidance onset.Source: §6 · p. 7, §6.1 · p. 8, Table 5 · p. 8

6.2. Topic classification

  • Using gold guidance labels, ModernBERT-large correctly classified 64 of 84 conversations, compared with ModernBERT-base’s 63 and TF–IDF’s 57; macro-F1 was .671 and .646 for the ModernBERT models.Topic-only evaluation used 84 guidance-positive conversations, representing 12 of 14 classes.Source: §6.2 · p. 8, Table 6 · p. 9
  • ModernBERT-large routes 151 of 200 conversations correctly end to end, compared with 148 for TF–IDF and 145 for ModernBERT-base.End-to-end scores use all 200 conversations and the 13 classification outcomes observed in the evaluation set.Source: §6.2 · p. 9, Table 6 · p. 9
  • Stage 1 accounts for 31–37 of the cascades’ 49–55 total errors; false positives are the largest single category, while wrong-topic predictions number 18–20.ModernBERT-base misses 13 guidance-positive conversations at Stage 1, whereas ModernBERT-large misses five, helping explain their end-to-end ranking.Source: §6.2 · p. 9
  • Five of the 12 observed topics have five examples or fewer, making per-class results descriptive rather than stable class-level estimates.Source: §6.2 · p. 9

6.3. Statistical reliability and silver-label effects

  • No exact paired comparison reaches p < .05, and bootstrap macro-F1 intervals overlap; dense models’ clearest advantage is topic classification, improving macro-F1 by 8.7 and 11.2 points.Model size has no consistent performance advantage: base leads the large model by 2.9 points on silver topic validation, while large leads by 2.5 on gold topic classification.Source: §6.3 · p. 9

7. Discussion

The discussion finds that ModernBERT helps most with legal-topic classification, while model trade-offs and multi-topic requests complicate guidance detection and routing. The findings are bounded by annotation, sampling, and deployment limitations.

7. Discussion

  • Both ModernBERT models perform better than TF–IDF on topic classification, while their differences across silver and gold datasets are not statistically significant.The authors suggest clear wording around options, forms, deadlines, and drafting may help explain TF–IDF's competitiveness on guidance detection.Source: §7 · p. 9
  • ModernBERT-large misses the fewest guidance requests but produces the most false positives, so the threshold should match whether flags trigger notices or specialist review.False positives increase workload when flagged conversations go to human reviewers or specialists.Source: §7 · p. 10
  • 25.3% of guidance-seeking conversations span multiple topics, although the classifier assigns only one primary topic.Some topic-classification errors may reflect multi-topic requests rather than incorrect predictions.Source: §7 · p. 10
  • The enriched sample does not support WildChat prevalence estimates, and the gold set includes only 84 guidance-positive conversations across 12 observed topics.Training and development labels came from one model; reviewers assessed pre-annotated labels, and topic labels were not independently checked by a legal expert.Source: §7 · p. 10
  • English-only, jurisdiction-agnostic taxonomy design and reducing multi-topic requests to one routing label limit transfer to other settings.Request types, secondary topics, and uncertainty flags were not human-validated; guidance onset was also not human-validated.Source: §7 · p. 10
  • The classifier identifies requests that may need safeguards but does not assess claim validity or answer correctness; deployment requires calibration, user testing, and human support.Conversations may describe sensitive legal circumstances despite anonymization procedures.Source: §7 · p. 10

8. Conclusion

The paper introduces a taxonomy and benchmark for legal guidance seeking in general-purpose LLM conversations. In the enriched sample, rights-and-options advice was common, while model performance varied by task; the authors point to future robustness testing and deployment research.

8. Conclusion

  • 45.8% of conversations in the enriched sample were labeled guidance-seeking; rights-and-options was the most common primary request, and 25.3% spanned multiple topic labels.The 45.8% figure refers to silver labels in the enriched sample.Source: §8 · p. 10
  • On the gold evaluation set, ModernBERT-large performed best overall, but TF–IDF performed similarly on binary guidance detection while dense models led on topic classification.No paired comparison reached significance at this sample size, so score differences warrant caution.Source: §8 · p. 10
  • Future work should test robustness across languages and legal systems and explore caution notices or routing; the public release is intended to support research and practical safeguards.The released materials include the annotated dataset, taxonomy, and evaluation code.Source: §8 · p. 10

Declaration of generative AI use

The authors used GPT-5.6 Terra during taxonomy development and GPT-5.6 Sol for final silver annotations; GPT-5.6 Sol also assisted with manuscript editing.

Declaration of generative AI use

  • GPT-5.6 Terra supported taxonomy development, while GPT-5.6 Sol generated final silver annotations and helped refine manuscript grammar and LaTeX.The authors reviewed and edited the resulting content and take responsibility for the final publication.Source: Declaration of generative AI use · p. 11

Related papers

Loading 2610.25159v1…