BITEM at the NTCIR-19 R2C2 Task: Predicting Confidence from Agentic RAG Pipeline Signals

• The task comprises the English Passage Retrieval (PR) and Answering with Confidence (AC) subtasks.Source: Subtasks · p. 1

1 Introduction

• The orchestrator checks each recorded evidence claim against its cited passage and computes answer confidence from the run’s record.The model cannot rate itself, and an answer is submitted only after enough evidence passes the entailment check.Source: §1 · p. 1
• 0.0709 nDCG@20 is the retrieval gain from assembling passes, while the organisers’ test does not establish a difference.The two retrieval runs placed 4th and 5th of 22; for answering, pass assembly left accuracy unchanged while improving confidence calibration.Source: §1 · p. 1
• Simple rules over already-computed artefacts achieve HMR close to 0.70 while improving accuracy, without further model calls or retrieval.The rules require only a few programming changes.Source: §1 · p. 1

2 Related Work

• Agentic retrieval research studies live-web search loops and ways to diversify queries or parallel retrieval paths.One cited study reports 32 per cent of multi-turn agent steps near-duplicating the previous query [10].Source: §2 · p. 2
• Evidence-checking work evaluates entailment against cited passages or verifies draft statements before delivery.Honovich et al. [3] report 81.5 ROC AUC for the best entailment model versus 63.8 for token-level F1; Liu et al. [8] find 51.5% of audited statements fully supported by their citations.Source: §2 · p. 2
• Supervised confidence methods use labelled examples, including a trained value head or a regression over question and retrieved context.The cited regression predicts hallucination from context sufficiency and self-rated answer probability.Source: §2 · p. 2
• Some confidence methods ask models to judge their own outputs, while others estimate uncertainty from groups of semantically equivalent answers.Kuhn et al. [6] use entropy over answer meanings rather than wordings.Source: §2 · p. 2
• Unsupervised approaches read model logits, compare repeated samples, or ask the model to verbalise confidence, with reported calibration varying by method and task.A study across 11 datasets recommends information-based methods for short answers and sample diversity for longer generations.Source: §2 · p. 2

3 Method

• Confidence is computed outside the model from recorded, checked evidence, using by-products of the agentic run rather than a self-rating.The method is unfitted for the task, and sparse evidence yields low confidence by construction.Source: §3 · p. 2

3.1 Preprocessing

• Passages are sentence-aligned, greedily packed to a 200-word target under a 300-word ceiling, and stored without overlap.Tables become separate passages; the indexed corpus contains 2,091,427 passages.Source: §3.1.1 · p. 2

3.1.1 Chunking.

• Entity annotation adds names to passages so retrieval can match questions whose answering passage refers to an entity obliquely.A second document-level call assigns names to passages that do not spell them out; just under half the indexed annotations arrive this way.Source: §3.1.2 · p. 2

3.1.2 Named entities.

• The sparse inverted index serves questions naming an entity and stores passage text alongside title, section heading, and entity fields.Source: §3.1.3 · pp. 2–3

3.1.3 Indexing.

• The dense index serves descriptions that omit an entity’s name, embedding each passage with its first ten annotated names, title, section, and contents.The separately built co-occurrence knowledge graph matched only about one search in 60, and none of its results reached a filed run.Source: §3.1.3 · p. 3

3.2.1 Retrieval.

• Search combines lexical and dense rankings with reciprocal rank fusion, reranks the merged list, and can return whole passages with neighbours.Entity search also returns annotated names carried by matched passages, most frequent first.Source: §3.2.1 · p. 3, §3.2.2 · p. 3

3.2.2 Reading.

• Reading locates useful sentences within a passage by lexical scoring, falling back to entailment scoring when passage words differ from the question.The returned sentences supply words the model can later quote.Source: §3.2.2 · p. 3

3.2.3 Recording evidence and submitting an answer.

• Recording attaches quoted spans and claims to named passages; the orchestrator checks quotation and entailment, and answers require supporting passages or a refusal.A recorded nugget can be retracted if later reading contradicts it; no pass used that tool.Source: §3.2.3 · p. 3

3.3.1 The question.

• The model is assigned one of six question types as advice on answer form, then its tool proposals pass through the orchestrator for execution or rejection.The pass record tracks retrieved passages, nuggets, answer and confidence, search counts, and returned answers.Source: §3.3.1 · p. 3

3.3.2 The loop.

• The orchestrator rejects evidence calls naming passages absent from its record, and the loop ends on acceptance, no tool call, or an overlong transcript.Source: §3.3.2 · p. 4
• An answer must meet the file format, evidence-count or strong-entailment requirement, and confidence floor before acceptance; refusals return feedback for another attempt.Confidence is compared with recorded claims using answer-word overlap, while overlap never removes a claim from the evidence count.Source: §3.3.3 · p. 4

3.3.4 Confidence.

• The orchestrator computes filed confidence from the run's record, not from a model-proposed rating, using pipeline by-products without extra retrieval or model calls.Available signals include searches, turned-back answers, entailment scores, and reasoning-turn length.Source: §3.3.4 · p. 4

• Each question receives three passes and a fourth unless the last two are both unsure; fewer passes are therefore used on questions the system is least certain about.Source: §3.4 · p. 4
• Later passes search after removing passages recorded by earlier passes, serving both passage retrieval and cross-pass answer comparison.Source: §3.4 · p. 4
• Passes are not independent: when a pass cannot answer, a fallback may file the most confident earlier answer if a passage entails it.Source: §3.4 · p. 4

3.5 Runs

• Each pass trace contains both transcript and record; PR runs draw from the record, while AC runs use record nuggets and transcript answer attempts.A PR run returns up to 20 ranked passages per question; an AC run returns an answer, confidence, and supporting nuggets.Source: §3.5 · p. 4, §3.5.1 · p. 4, §3.5.2 · p. 4

3.5.1 Passage retrieval.

• BITEM-PG-2 pools passages from the answer group selected by strongest pass confidence and most pass agreement, whereas BITEM-PG-1 starts with the first pass.Both order evidence-bearing passages by entailment strength before other retrieved passages.Source: §3.5.1 · p. 4

3.5.2 Answering with confidence.

• BITEM-AC-1 pairs the first-pass answer and confidence with BITEM-PG-1, using transcript attempts if the pass committed a refusal.It files verified nuggets from that pass only when their passages appear in BITEM-PG-1.Source: §3.5.2 · p. 4
• BITEM-AC-2 files the highest-confidence submission unchanged, while BITEM-AC-3 groups normalized matching answers and files the group with the highest summed confidence, capped at 100.Both pool non-refusal transcript submissions, including answers the orchestrator turned back.Source: §3.5.2 · pp. 4–5
• The two pooled-answer runs normally file the same nuggets on all 65 questions because nugget matching ignores the chosen answer and cited passages.A deadline fix caused BITEM-AC-3 to be filed twice and scored twice.Source: §3.5.2 · p. 5

4.1 Passage retrieval

• 0.0709 nDCG@20 and 0.0905 Q@20 separated the pooled run from the single-pass run; it won 42 questions and lost 15.The organisers’ test did not separate the runs, so the gain is an observed difference rather than an established one.Source: §4.1 · p. 5
• 0.1824 and 0.0998 were the pooled-run gains on post-processing-heavy and multi-hop questions, versus 0.0223 on comparison; it ranked top of the field on the first two types.Those two types require information beyond a single passage; the pooled run held 5.3 more relevant passages per question on post-processing-heavy and 3.1 more on multi-hop, versus 0.4 on comparison.Source: §4.1 · p. 5, Table 2 · p. 5

4.2 Answering with confidence

• 0.9219 accuracy for most confident ranked 6th, while its HMR was 0.4915 (13th) and accHMR 0.4531 (7th).The official evaluation scored 64 questions, with question 0025 reissued after release; the four reported runs represented three distinct systems.Source: Table 3 · p. 5, §4.2 · pp. 5–6, §4 · p. 5

4.1 Passage retrieval

• The overall retrieval gain is qualified by short first-pass lists and the search loop’s stopping behavior, so the authors treat per-type ordering as more informative.The single-pass run fell short of 20 slots on 37 questions; no significance test was reported, and the per-type scores are descriptive trends.Source: §4.1 · p. 5

4.2 Answering with confidence

• Five of 64 most-confident answers were wrong; three were answer-writing errors despite the required passages being retrieved.Summed agreement could count submissions refused by the evidence gate, while four disputed counts had a correct number proposed by another pass but not filed.Source: §4.2 · p. 6

5 Additional experiments

• The experiments label recovered answers using judged matches or manual review, and most of the 192 labels came from answers already filed by BITEM runs.For 175 answers, the matching judged answer was one of BITEM's own filed answers; 15 unmatched answers were read by hand.Source: §5 · p. 6

5.1 Recovered model as judge run

• The recovered model-as-judge run resubmits recorded answers with their confidence, citations, verified nuggets, and filed passages, retrying if it cites an unfiled passage.Unlike the other BITEM runs, it files the model's own confidence.Source: §5.1 · p. 6
• The judge run would have achieved the best accuracy, but it still failed to report low confidence for wrong answers and retained calibration problems.The organisers did not judge this run; its answers were labelled by matching them to judged answers or by hand.Source: §5.1 · p. 6, §5 · p. 6

5.2 From signal to confidence

• For the revised single pass, the longest reasoning turn determines confidence: it receives none at or above the median and full confidence otherwise.Its turn length separated wrong from right answers at 0.707, compared with 0.559 for distinct queries and 0.586 for tool calls; the authors report this as a direction, not a result, on six errors.Source: §5.2 · p. 7
• The four-pass rule groups proposals by answer, weights accepted submissions, discounts fallback answers, and uses verified entailment only to break ties.Its filed confidence is 100 when two passes independently agree, 50 when they divide, and 0 when only one answers; entailment-based confidence performed poorly.Source: §5.2 · p. 7
• 60 of 64 questions were answered correctly by the four-pass rule, which scored 0.6985 HMR and 0.6549 accHMR, ranking 9th and 5th among 25 filed runs.The single-pass rule answered 58 correctly and scored 0.6639 HMR and 0.6017 accHMR, ranking 11th and 7th.Source: §5.2 · p. 7

6 Discussion and Conclusions

• 0.4531 to 0.6549 accHMR: rules over recorded passes improved confidence while leaving answering accuracy unchanged, without further model calls or retrieval.The four-pass runs cost about four times as much as a single pass.Source: §6 · p. 7
• Pooling passes helped retrieval, but close to half the gain came from list length rather than better material.In answering, the reported benefit was improved confidence rather than accuracy.Source: §6 · p. 7
• The hand-written rules remain unvalidated on unseen questions; 64 questions are a small set, and resampling within it may capture properties of that set.The authors say their claim is limited to signal already present in their traces, not that these particular rules should be carried forward.Source: §6 · p. 7
• A next step is fitting a regression on existing trace quantities, such as query count, entailment evidence, and deliberation length, then evaluating it on held-out R2C2 data.The proposed comparison is against the hand-written rules and filed confidence on the same questions and measure.Source: §6 · p. 7
• accHMR multiplies HMR by accuracy to make the confidence reward proportional to accuracy; the figure illustrates that HMR alone can be high near one-half accuracy.Figure 3 plots accuracy against HMR for filed runs and traces rule scores from the point where both equal 0.5.Source: §6 · pp. 7–8, Figure 3 · p. 7
• The authors propose adding corpus-unsupported questions to the measure, since HMR's wrong-answer-only R_O defaults to one when every answer is correct.Such questions would prevent that position from being reached for free and make low confidence and high accuracy jointly attainable.Source: §6 · p. 8