PaperlayerView exact revision on arXiv

Source-linked AI summary

BIT.UA at BioASQ 14B: Modular Retrieval with pg_textsearch and Qdrant, and Agent-Based Answer Generation

André Ribeiro, Rúben Garrido, Alexander Christiansen, Richard A. A. Jonker, Sérgio Matos

arXiv:2609.04999v1cs.CL

TL;DR

BioASQ biomedical question answering requires both retrieving evidence and generating answers, and prior BIT.UA work exposed weaknesses in generation and metric alignment. The paper presents a modular retrieval-and-generation overhaul using new indexing, reranking, and agentic methods, reporting competitive results and top Phase A rankings in multiple batches. Its findings also identify substantial cost and evaluation gaps that constrain deployment and interpretation.

  • Problem

    BioASQ requires accurate PubMed retrieval and evidence-based answer generation, while prior work revealed misalignment between automatic metrics and human judgments.

  • Method

    The team refactored the pipeline, replacing PISA with pg_textsearch and Qdrant, adding dense-negative reranker training, Context-1 retrieval, LLM judging, and agent quorum generation.

  • Results

    The systems achieved competitive results across batches, with Phase A systems reaching top rankings in multiple test batches.

  • Takeaways & Limitations

    The paper presents modular retrieval and multi-agent answer generation as a flexible BioASQ pipeline spanning document retrieval, snippet generation, and answer generation.

  • Takeaways & Limitations

    The agent quorum can be expensive in inference cost and wall-clock time, although lightweight local models were competitive.

Abstract

from arXiv · show

This paper describes the participation of the BIT.UA team from the University of Aveiro in the 14th edition of the BioASQ Task B challenge on biomedical question answering. Building on our previous submissions, we introduced a substantially refactored and modular codebase, and made significant changes to both the retrieval and generation components of the pipeline. For Phase~A document retrieval, we replaced the PyTerrier PISA index with PostgreSQL-based pg\_textsearch for BM25 retrieval and adopted Qdrant for dense embedding indexing, enabling more efficient storage and GPU-accelerated similarity search. We explored HyDE-based query expansion alongside a Context-1 retrieval strategy. A new reranker training pipeline was developed, incorporating dense retrieval for negative sampling. For Phases A+ and B answer generation, we introduced an LLM-as-a-judge framework and a novel agent quorum mechanism, where multiple agents with diverse prompts debate and iteratively converge on a consensus answer using adaptive document retention. We also participated in the snippets generation subtask for the first time. Our systems achieved competitive results across all batches, with Phase~A systems achieving MAP ranks of 5 (Batch~1,3). We discuss the impact of these architectural changes, lessons learned, and outline directions for future work including SPLADE and ColBERT integration. All code is openly available: https://github.com/bioinformatics-ua/BioASQ14b.

1. Introduction

BIT.UA rethought its BioASQ pipeline after finding misalignment between automatic metrics and human answer quality, introducing modular retrieval and agent-based generation changes for the 14th edition.

  • BioASQ Task B evaluates PubMed document retrieval in Phase A and answer generation in Phases A+ and B.
  • The 13th edition exposed persistent misalignment between automatic metrics and human-judged answer quality.
  • Dense Pseudo Relevance Feedback consistently improved retrieval, while generation remained the weakest component.
  • The 14th edition introduced a modular codebase, new retrieval infrastructure, HyDE and Context-1 exploration, reranker changes, and agent quorum generation.The team also entered the snippet-generation subtask for the first time.
  • The paper describes prior conclusions, methodological changes, validation and official results, lessons learned, and future directions.

2. Previous Work

The team’s previous BioASQ system combined hybrid retrieval, neural reranking, DPRF, and RAG generation, but prior evaluation exposed weaknesses in generation and metric alignment that shaped the new design.

  • The 13th-edition system used BM25 retrieval, neural reranking, DPRF, reciprocal rank fusion, and RAG-based answer generation.
  • DPRF produced consistent but modest retrieval improvements, while reducing reranked documents from 1,000 to 100 had negligible performance impact and accelerated inference.
  • The generation pipeline used retrieved documents as context for multiple large language models, including a fine-tuned Gemma 3 27B model.
  • Ensembling and summarization were unified into one generation step, and inference moved from Ollama to LMDeploy for faster generation.
  • Automatic metrics repeatedly diverged from human judgments, motivating greater attention to robustness and cross-batch consistency.
  • Snippets produced high recall but did not always yield high human-evaluation rankings, while generation remained the weakest component.
  • These findings informed the 14th-edition effort to address prior architectural and methodological limitations while retaining effective elements.

3. Methodology

The 14th-edition methodology modularizes retrieval, combining BM25 and dense search with reranking, while also evaluating HyDE and Context-1 as alternative retrieval strategies.

  • The refactored codebase separates retrieval, reranking, and generation into independently testable modules.This design supports faster experimentation and reduces coupling between pipeline stages.
  • The traditional retrieval pipeline fuses BM25 and dense retrieval, applies cross-encoder reranking, and extracts snippets, while Context-1 provides an alternative agentic path.
  • Parallel lexical and dense searches are fused into one candidate list before reranking, extending the previous BM25-only first stage.
  • The migration from PISA reduced dependence on separately loaded index artifacts, which had required repeated disk-to-memory loading.
  • Dense retrieval uses BGE-M3 embeddings served through TEI and indexed in Qdrant for vector similarity search.
  • At query time, the top 100 dense and top 100 BM25 documents are fused, using either the question or its HyDE expansion for dense search.
  • Context-1 iteratively decomposes questions, searches with BM25 and dense tools, evaluates relevance, and refines its strategy across turns.

3.3. Re-Ranking

The reranking methodology adds flexible training and dense-retrieval negatives, then fuses heterogeneous reranker outputs with RRF; the team also introduces snippet generation as text generation.

  • The new reranker training pipeline supports multiple base models, training strategies, data configurations, and negative-sampling approaches.
  • Dense retrieval supplies harder and more diverse negative examples than BM25-only sampling, improving discrimination across retrieval paradigms.
  • The team trained 29 reranker models spanning architectures such as BGE-reranker-v2-m3, PubMedBERT, BioLinkBERT, BioBERT, MedCPT, and LLaMA Nemotron.
  • BasicSampler selects positive and irrelevant documents to construct training pairs, while ShifterSampler progressively introduces harder negatives during training.
  • RRF combines reranker rankings using reciprocal positions rather than raw scores, avoiding normalization and accommodating heterogeneous score scales.
  • RRF consistently outperformed weighted-sum fusion for rerankers in the reported experiments.
  • For the first time, the team treated snippet generation as text generation from a question plus document title and abstract.A Gemma 4 31B model was fine-tuned with QLoRA using BioASQ snippet annotations.

3.5. Answer Generation

Answer generation combines LLM-based candidate judging with a multi-agent quorum that debates evidence and synthesizes a final answer. The quorum uses diverse agent focuses, adaptive document retention, iterative agreement, and explicit stopping criteria.

  • LLM-as-a-judge: LLM-as-a-judge replaces fixed majority voting by selecting or synthesizing candidate answers using the question, context, and available reference answers.The judge scores correctness, faithfulness, completeness, and overall quality.
  • LLM-as-a-judge: The judge is particularly effective for yes/no, factoid, and list questions, whose answers are more objectively verifiable against context.
  • Agent quorum: The agent quorum instantiates agents with different models and cognitive focuses, including analytical, evidence-based, and skeptical perspectives.
  • Agent quorum: Adaptive retention initially gives each agent three documents, then replaces discarded documents with randomly selected documents while retaining selected evidence.
  • Agent quorum: Agents debate through opinions and agreement levels, with randomized ordering, until consensus, a round limit, or stable disagreement stops the quorum.
  • Final synthesis: A designated synthesizer converts the debate and full context into one definitive answer grounded solely in the context documents.

4. Results

The results section reports internal validation and official evaluation results for each phase across four BioASQ test batches.

  • Internal validation and official evaluation results are presented for each phase across the four BioASQ test batches.

4.1. Validation

Internal validation compares retrieval strategies, reranker models, and negative-sampling methods, while qualitative analysis examines answer-generation behavior. Hybrid retrieval and ShifterSampler improve selected internal metrics, but answer-generation validation remains difficult to align with human judgments.

  • Retrieval validation: Internal testing measures the marginal contribution of retrieval components on 13B Batch 1–4 golden datasets.
  • Retrieval validation: RRF fusion improves MRR from 0.549 to 0.696 and R@100 from 0.373 to 0.406 over BM25, while RRF outperforms weighted sum on MRR and Recall.
  • Retrieval validation: Weighted-sum retrieval followed by BGE-reranker-v2-m3 achieves the best MRR overall at 0.738, whereas reranking RRF results lowers MRR to 0.649.
  • Retrieval validation: Context-1 achieves the highest M@100 at 14.16 and R@100 at 0.453, but its MRR of 0.508 is lower than the best fusion configurations.
  • Reranker validation: LLaMA Nemotron rerankers achieve outstanding validation performance, although their unusually high scores raise concerns about data leakage.
  • Reranker validation: ShifterSampler outperforms BasicSampler across all four architectures, with absolute MAP improvements ranging from 0.0052 to 0.0445.
  • Answer-generation validation: Answer-generation validation is considered unreliable because no direct targeted metrics correlate well with human judgments, so analysis focuses on qualitative quorum consistency and adaptive retention.

4.2. Official Results

Official results show mixed retrieval performance but stronger answer-generation outcomes in selected batches. LLM-as-judge, ensemble selection, and lightweight or open-source agent configurations were recurrently competitive, while computational cost remained a constraint.

  • Document retrieval: Weighted-sum hybrid retrieval outperformed hybrid RRF, while query expansion and Context-1 fell short of the BM25 baseline.The absence of strict control tests prevents isolating query expansion’s intrinsic benefit from added computational overhead.
  • Snippet retrieval: BM25 with the best rerankers achieved the strongest snippet ranks: 25th in Batch 3 and 28th in Batch 4.In Batch 4, this snippet advantage did not simply follow document-retrieval rank, because the same configuration ranked 13th for document retrieval.
  • Exact answers: In Phase A+, judge-based runs lifted list F1 to 0.3145 (rank 15) and 0.3227 (rank 13), while two Batch 3 quorums achieved perfect yes/no F1 of 1.0 (rank 1).The open-source bioinfo-3 quorum also placed 3rd on factoid (0.4706) and 6th on list (0.3138) in Batch 3.
  • Exact answers: In Phase B, widening LLM-as-judge selection from top-k=3 to top-k=5 improved factoid MRR from 0.4130 to 0.4565 (rank 8), while Batch 4 produced factoid 0.5000 (rank 5) and list F1 0.5671 (rank 9).Two Batch 4 systems also reached 3rd on yes/no with a score of 0.9352.
  • Cross-phase patterns: Factoid and list metrics separated systems more clearly than yes/no F1, and LLM-as-judge or ensemble selection produced the largest gains on list F1.Open-source quorums matched or exceeded mixed proprietary-API configurations in the strongest runs.
  • Efficiency and model scale: Small agents remained competitive: Agents v0 [S] achieved joint-best factoid MRR of 0.4118 (rank 9) in Batch 3 Phase A+ and matched larger systems on several metrics.The full agents ensemble required approximately 17 model calls per question, whereas small agents required 4 calls, representing an approximately 10× cost reduction.
  • Ideal answers: LLM-as-judge systems led recall-oriented ideal-answer results but often had low F1, whereas agent quorums produced more balanced outcomes and sometimes competitive lightweight configurations.In Phase A+, judge-based systems achieved recall ranks 3 and 4 but F1 values of 0.0963–0.0983, while in Phase B lightweight Agents v4 [S] achieved the strongest F1 of 0.1903 in Batch 2.

5. Discussion

The discussion finds that the modular retrieval redesign improved operational reliability, while several retrieval changes did not translate into stronger official results. Generation advances, especially agent quorum, produced the strongest answer results, but evaluation incompleteness and computational cost constrain conclusions and deployment.

  • Retrieval: The PostgreSQL-centric migration improved operational reliability by eliminating index-corruption and synchronization issues across four batches.Documents, BM25 indexes, and metadata were co-located, reducing configuration errors and simplifying submissions.
  • Retrieval: Internal hybrid retrieval improved MRR from 0.549 to 0.696 (+27%) and R@100 from 0.373 to 0.406 (+9%), but official results favored BM25-only systems across multiple batches.RRF also outperformed weighted sum internally on MAP without producing a clear official ranking advantage, highlighting the limits of internal validation.
  • Retrieval: The best reranker reached an internal-validation MRR of 0.738, yet configurations using all rerankers outperformed the single best configuration in Batches 1 and 2.Dense-retrieval negatives and curriculum-based hard-negative mining supported the training pipeline, although the generalization hypothesis awaits ablation testing.
  • Retrieval: Context-1 achieved the highest internal R@100 of 0.453 but had lower MRR of 0.508 and weak official performance, limiting its competitiveness as a standalone method.The internal–official discrepancy may reflect question distributions, batch conditions, or its competition-pipeline implementation.
  • Future Work: Automated results remain provisional because human evaluation is incomplete, and future work targets broader retrieval testing, dynamic prompting, judge–quorum integration, SPLADE, ColBERT, and better snippet extraction.The paper specifically identifies official and human-based evaluation as necessary for resolving conflicting evidence about Context-1 and other retrieval changes.
  • Generation: Agent quorum produced the strongest Phase A+ and Phase B results, including rank 1 for yes/no accuracy and factoid strict accuracy in Phase A+ Batch 3.Adaptive document retention is presented as balancing iterative reasoning depth with computational efficiency.
  • Generation: Smaller models were competitive with larger models on most metrics, while quorum deployment remained costly when using multiple large models and debate rounds.List questions favored larger models, and lightweight local models made iterative debate more economically viable.

6. Conclusion

The 14th-edition system combined modular retrieval infrastructure with new answer-generation mechanisms and achieved competitive results while reinforcing the value of robust, cross-batch biomedical QA design.

  • Retrieval: The refactored system replaced prior retrieval infrastructure with pg_textsearch for BM25, Qdrant for dense indexing, TEI embeddings, HyDE and Context-1 expansion, and redesigned reranker training.These changes were intended to make retrieval more flexible and efficient while maintaining competitive performance.
  • Answer generation: LLM-as-a-judge and agent quorum mechanisms enabled diverse models to debate, evaluate context, and converge on consensus answers in Phases A+ and B.Snippet-level evidence was incorporated to accelerate the quorum process and improve answer precision.
  • Answer generation: The team participated in snippet generation for the first time, establishing a foundation for future improvements.
  • Lessons learned: The challenge’s retrieval, exact-answering, and summarization structure encouraged holistic system design rather than narrow metric optimization.The authors identify architectural robustness, cross-batch consistency, and methodological innovation as complementary pillars of competitive biomedical QA systems.
  • Future work: The authors position the 14th-edition system as a foundation for continued development in future BioASQ editions.

Declaration on Generative AI

The authors used generative models for grammar correction, spelling checks, and paraphrasing, then reviewed and edited the resulting text.

  • Generative models were used to correct grammar, check spelling, and paraphrase text in the document.
  • The authors state that they reviewed and edited the generated content and take responsibility for the publication.

A. Trained Reranker Models

The appendix catalogs 29 Phase A reranker models and explains the architectures, training configurations, losses, and negative-sampling strategies used to distinguish them.

  • 29 reranker models were trained for Phase A document retrieval.
  • Models are identified by base architecture, training epochs, sampler, loss type, and data configuration.FullData denotes training on the full available dataset, while shifter denotes a negative-selection strategy.
  • All models used pairwise ranking loss unless otherwise noted, with multi-negative pairwise training and InfoNCE providing alternative configurations.InfoNCE is described as a contrastive loss variant applied to the LLaMA Nemotron model.
  • The Shifter sampler dynamically adjusts negative sampling during training.

B. Prompt Templates for Answer Generation

The appendix documents prompt templates for exact, ideal, typed, and agent-quorum answer generation, with variants designed for different question types and reasoning strategies.

  • B. Prompt Templates for Answer Generation: The prompt-template appendix describes each variant’s design rationale, while full template text is available in the project repository.
  • B.1. Exact Answer Prompts: Exact-answer prompts target yes/no, factoid, and list questions with multiple variants using distinct reasoning strategies.The strategies include evidence weighing, null-hypothesis framing, coverage, precision, and reformulation.
  • B.2. Ideal Answer Prompts: Ideal-answer prompts support all question types through seven increasingly structured variants for paragraph-length responses.The variants produce 50–150 word paragraph answers.
  • B.3. Typed Generation Prompts: Typed-generation prompts combine ideal and exact answering in one call, producing both a paragraph summary and typed JSON.
  • B.4. Agent Quorum Prompts: The agent-quorum debate prompt gives each agent a role, document sample, adaptive-retention instructions, prior conversation history, and a thinking-focus description.
  • B.4. Agent Quorum Prompts: The agent-quorum prompt components are assembled into debate turns, while the final-answer prompt uses a simpler synthesis-oriented structure.
Loading 2609.04999v1…