PaperlayerView exact revision on arXiv

Source-linked AI summary

QUEST: A Query and Extraction System for Topics in Asylum Law Application Decisions

Maria Vlachou, Anna Murphy Høgenhaug, Mohammad N. S. Jahromi, Galadrielle Humblot-Renaux, Thomas Gammeltoft-Hansen, Thomas B. Moeslund, Desmond Elliott

arXiv:2608.28555v1cs.IR

TL;DR

The paper asks how well automated methods identify credibility indicators in Danish asylum appeal materials, a task complicated by long and heterogeneous documents. QUEST combines synthetic query generation, topic extraction, retrieval, and standard or credibility-specific relevance assessment. The results show that credibility-based evaluation is more difficult, while the authors identify limitations from Danish-only data, restricted model choices, and predominantly machine-generated relevance judgments.

  • Problem

    The paper asks how well QUEST identifies credibility indicators in Appeals Board reports compared with human experts.

  • Method

    QUEST frames the task as information retrieval, combining synthetic query generation, topic extraction, retrieval, and qrels or credibility-specific crels labeling.

  • Results

    Credibility-based relevance assessments produce lower retrieval performance in most cases than qrels, indicating greater difficulty for the credibility-assessment task.

  • Takeaways & Limitations

    Domain-specific credibility judgments are critical for estimating whether retrieval systems address credibility assessment rather than general information needs.

  • Takeaways & Limitations

    The conclusions are based on Danish data and decision-makers’ perspectives, while most relevance assessments are not human-derived and the system is not deployed.

Abstract

from arXiv · show

Legal decisions on asylum applications consist of long, complex, and heterogeneous documents, covering narrative applicant interviews, original decisions, and additional supporting materials. If an application is rejected, a critical question in processing an appeal is whether the credibility of the information in the original application was a factor that determined the original decision. In this paper, we present the QUEST system (Query and Extraction System for Topics) to extract and identify factors relating to credibility assessments in two datasets of Danish asylum application appeals. QUEST frames this problem as an information retrieval task, combining synthetic query generation, topic extraction, and relevance assessment to identify information related to credibility indicators in appeals board application materials. In addition to standard retrieval evaluation metrics, we propose a new type of domain-specific assessments distinct from the traditional relevance to evaluate the performance of the tested systems with respect to credibility factors. In this way, we obtain insights about how well automatic methods can return answers for different types of indicators appearing in asylum appeals. Our results indicate that there is an increased challenge when estimating performance using credibility-based relevance assessments, thus pointing to the difficulty of the task.

1 Introduction

QUEST addresses the challenge of identifying credibility indicators in long, sensitive Danish asylum appeal materials. It frames the task as retrieval and introduces credibility-focused relevance assessments to compare automated systems with human experts.

  • Danish appeal materials are heterogeneous, lengthy narratives containing interviews, decisions, and supporting documents, making credibility assessment difficult without careful reading.
  • QUEST frames credibility-indicator identification as information retrieval over document chunks, using generated queries and topics linked to a credibility codebook.
  • The paper studies how well QUEST identifies credibility indicators in Appeals Board reports compared with human experts.
  • The findings report differing performance under standard and credibility-based assessments, limited retrieval tuning for credibility, and consistent gains from neural reranking.
  • The paper contributes QUEST, crels, and experiments varying expert-annotation strictness for credibility-indicator relevance.
  • The authors introduce crels, domain-specific relevance assessments that complement standard query-passage relevance judgments.

2 Related Work

Related work connects QUEST to credibility assessment, legal NLP, topic modeling, synthetic relevance judgments, and LLM-based evaluation. The paper distinguishes its short-passage retrieval setting from character-based legal-text retrieval.

  • Prior credibility-assessment research spans qualitative studies and more systematic manual annotation, while recent reviews identify conceptual fuzziness and methodological rifts.
  • Legal NLP has used LLM judges for evaluating retrieval-augmented responses and constructing domain-specific evaluation datasets.
  • Existing legal NLP systems retrieve exact character spans, whereas QUEST addresses short-passage retrieval.
  • Topic modeling traditionally identifies themes or labels in document collections, while QUEST uses a neural pipeline and an LLM for topic labeling.
  • Few-shot synthetic query-generation methods create task-specific relevance data or retrievers from relevant examples, including positive and negative query construction.

3 Methodology

QUEST is a six-step pipeline that segments asylum appeal documents, generates synthetic queries, extracts named topics, retrieves relevant chunks, and assigns standard or credibility-specific relevance labels.

  • The pipeline begins by splitting RAB documents into chunks and generating one query per chunk using few-shot query-passage examples.
  • Because the dataset lacks labeled query-answer pairs, few-shot examples come from Danish asylum-law documents and a credibility codebook.
  • BERTopic embeds, dimension-reduces, and clusters generated queries, after which Gemma3 assigns natural-language names to topic clusters.
  • The resulting topics retrieve top-ranked chunks using sparse and dense retrieval methods before relevance labeling.
  • An LLM judge assigns graded qrels for topic-passage relevance and crels for whether topic content supports credibility assessment.
  • The same topic-answer pairs support multiple label types, enabling comparison of rank-metric differences across tasks.

4 Experimental Setup

The experimental setup evaluates sparse and dense retrieval models with standard information-retrieval metrics. The tested systems include BM25, multilingual embedding retrievers, and SPLADE.

  • The first-stage systems include BM25, E5, Qwen3, and SPLADE, spanning sparse, dense, and hybrid-oriented representations.
  • The evaluation follows standard TREC practices and includes the official MS MARCO metric MRR@10.
  • Retrieval and reranking are evaluated with MAP@100 and NDCG@10, alongside MRR@10.

5 Dataset

The dataset comprises PUB and PRIV asylum-appeal samples processed into comparable short chunks, enabling topic analysis of heterogeneous case materials. Topic extraction reveals different distributions of country, risk, trauma, family-conflict, and credibility-related information across the datasets.

  • Dataset construction: The dataset samples PUB and PRIV cases and preprocesses their files into chunks for analysis.PRIV uses stratified random sampling of 15 cases from each year, while Table 1 reports sample statistics.
  • Dataset construction: Keeping chunks 2-3 sentences long produces similar average chunk lengths despite PRIV documents being much longer.
  • Topic distributions: The Topic Extractor identifies semantic themes covering country- and year-specific case information, credibility and risk assessments, trauma, and family conflict.PUB topics include Afghanistan- and time-focused information, while other topics address credibility, risk, trauma, and family conflict.
  • Topic distributions: PRIV contains a higher representation of credibility-based indicators, with its first topics reaching 34%.The paper relates this distribution to longer case files in which credibility is mentioned across documents and narrative discrepancies support outcome justifications.
  • Topic generation: Generated queries from sensitive case-file chunks are sufficiently generic to support topic generation with BERTopic.

6 Results

QUEST evaluates topic-based retrieval for credibility indicators using standard and domain-specific relevance assessments. Credibility-focused evaluation is harder than ordinary query relevance, while reranking and stricter judgments reveal distinct performance patterns.

  • 6.1 Estimating Performance with Different Types of Relevance Assessments: BM25 is slightly worse than more advanced retrieval methods, while E5 and SPLADE provide comparable, slightly higher results than the other systems.The reported comparison concerns first-stage retrieval across MAP@100 and NDCG@10.
  • 6.1 Estimating Performance with Different Types of Relevance Assessments: First-stage retrieval performs lower under credibility relevance than query relevance, reflecting the greater difficulty of identifying credibility indicators.Agreement between qrels and crels is low: Cohen’s κ is 0.04 for PUB and 0.02 for PRIV.
  • 6.1 Estimating Performance with Different Types of Relevance Assessments: MonoT5 reranking slightly improves NDCG@10 in most cases but does not improve MAP@100 after reranking the top 100 items.The authors attribute this pattern to MAP@100 covering exactly the reranked depth, whereas NDCG emphasizes higher-ranked items.
  • 6.1 Estimating Performance with Different Types of Relevance Assessments: MAP@100 decreases linearly as the relevance cutoff becomes stricter for both datasets.More conservative LLM-judge cutoffs produce lower performance estimates, with performance eventually collapsing under sufficiently strict criteria.
  • 6.2 Similarities and Differences between LLM Judge and Human Labels: LLM judges can annotate larger credibility-assessment samples, but they often assign partial relevance where human experts apply stricter judgments.Human experts better distinguish critical from less critical aspects of complex text, although the LLM sometimes matches human reasoning.
  • 6.3 Implications of Findings and System Use: QUEST returns relevant information from specific case-file chunks and is intended to support, rather than replace, expert credibility assessment.The proposed use is human-in-the-loop analysis of existing decisions, with no deployment in decision-making without human oversight.

7 Conclusions

The framework extracts credibility-related and general claimant information from Danish asylum appeals using topic generation and topic-based retrieval. Domain-specific relevance judgments help distinguish credibility-focused performance from general relevance, though the framework is less strict than human experts.

  • The framework combines topic generation and topic-based retrieval to identify credibility indicators and general claimant information in Danish appeals.
  • Compared with human experts, the framework is overall less strict when identifying credibility indicators.
  • Some generated topics produce answers more related to risk assessment than credibility assessment.
  • Domain-specific relevance judgments are critical for determining whether retrieved information addresses credibility assessment.
  • The topics provide data-derived indicators from Refugee Appeals Board materials, while crels distinguish credibility-relevant performance estimates from general labels.

Limitations

The approach relies extensively on LLMs and on Danish asylum data, while most relevance assessments are not human-derived. These constraints limit the breadth and direct human grounding of the conclusions.

  • LLMs perform query generation, topic naming, and relevance judging, with different models used for querying and judging.
  • The conclusions are based on Danish data and the perspective of its decision-makers, rather than datasets from multiple countries.
  • Most relevance assessments are not human-derived, although automated judging approximates human assessment across many query-passage pairs.

Ethics Statement

The study uses highly sensitive Danish-language Refugee Appeals Board data under ethical and data-governance controls. Because of bias and anonymization challenges, the system is not currently deployed.

  • The dataset consists of highly sensitive Refugee Appeals Board materials written in Danish, an under-represented language in language models and NLP datasets.
  • Researchers obtained ethical approval and signed a joint data controlling agreement governing the dataset’s processing purposes and means.
  • Bias and personal-identifiable-information anonymization challenges mean the described system is not currently deployed.

A Implementation Details

QUEST chunks asylum appeal documents, generates queries, extracts topics, retrieves passages, and applies LLM-based relevance assessment. Experiments use multiple retrieval models and distinguish general from credibility-specific label distributions.

  • Document Chunking: Documents are split into sliding windows of eight sentences, producing 4,028 PUB chunks and 20,912 PRIV chunks.
  • Few-Shot Query Generation: Three-shot query generation uses Gemma3:4b to produce one query per chunk for each dataset.
  • Topic Extraction: BERTopic embeds, reduces, clusters, and names generated queries, yielding 43 PUB topics and 151 PRIV topics.
  • Retrieval Methods: BM25, SPLADE, E5, and Qwen3 retrieve passages, followed by multilingual monoT5 reranking.
  • LLM Judge: The Llama3.1:8b judge scores pooled top-100 results for general and domain-specific relevance assessments.
  • Evaluation Metrics: MAP@100 and NDCG@10 evaluate retrieval and reranking, with MRR@10 added as the official MS MARCO metric.
  • Assessment Distributions: For qrels, labels 2 and 4 are most frequent while extreme values are rare; crels are more unbalanced in PRIV than PUB.

C LLM Judge Validation Details

The validation procedure compared LLM-judge assessments with domain-expert reasoning about credibility relevance. The appendix also documents QUEST’s query-generation and relevance-assessment prompts, including human-informed topic reformulations.

  • Validation procedure: Domain experts were briefed on QUEST’s topic derivation, prompts, and relevance-judgment procedure before annotation.Experts normally assess whole case files or summaries, whereas this validation provided topic–chunk pairs.
  • Validation findings: Human annotation examined whether text segments were relevant specifically because of credibility considerations, distinguishing this from general relevance.Experts reported cases that were generally relevant but not relevant in the domain-specific sense.
  • Validation findings: Some generated topics concerned risk assessment or claimant information rather than credibility.The domain expert identified these as topics that did not refer to credibility.
  • Topic reformulation: The expert found that some LLM-generated topic names did not match legal experts’ terminology and suggested alternative names.The proposed reformulations could make generic topics more suitable for identifying specific credibility indicators.
  • Prompt design: QUEST generated one query per chunk with few-shot examples derived from a credibility codebook, then used separate prompts for standard and domain-specific relevance assessment.The query-generation prompt asks for a query matching a passage’s semantic content; Llama3 was used for relevance assessment and crels.

E Additional Results

The additional results report privacy-preserving query examples and retrieval and reranking evaluations across query, credibility, and label relevance. They also caution that MRR@10 can overestimate first-stage effectiveness in the qrels setting.

  • Supplementary queries: The supplementary PRIV queries sometimes generalize city names or identifiable details to comply with privacy regulations.Table 8 lists representative queries within frequent identified topics and notes these privacy-related modifications.
  • Retrieval and reranking: Tables 9 and 10 evaluate first-stage retrieval and MonoT5 reranking for BM25 and Qwen3 using MRR@10.Table 9 covers qrels, crels, and label_rels; Table 10 marks improvements over corresponding first-stage results.
  • Retrieval and reranking: MRR@10 is described as inappropriate for qrels effectiveness because it mainly considers the top-ranked item and overestimates results.This caveat applies especially to the first-stage retrieval results.
  • Relevance assessment: The standard relevance prompt scores how well a system answer addresses a user query on a 0-to-5 scale.The prompt asks whether the question can be answered from the available context, with 1 meaning not helpful and 5 meaning completely helpful.
  • Relevance assessment: The domain-specific relevance prompt asks legal annotators to determine whether credibility assessment makes a text segment relevant to a query.It operationalizes credibility assessment as evaluating whether the asylum seeker’s account is believable.
Loading 2608.28555v1…