PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Agent Retrieval-Augmented Generation for Efficient Cloud Knowledge Base Search in Telecom SNOC Environment

Harish Saragadam, Sudhanshu Sharma, Ipsha Routray

arXiv:2609.01618v1cs.IRcs.LG

TL;DR

SNOC engineers need reliable search across large, heterogeneous cloud-document collections during critical incidents. The paper presents a fully offline multi-agent hybrid RAG pipeline with retrieval fusion, reranking, diversity selection, and explicit attribution verification, achieving 78.4% EM and a 14.6-point improvement over single-stage dense retrieval.

  • Problem

    SNOC search tools can miss semantic matches, exact telecom identifiers, or unsupported claims during time-critical incidents.

  • Method

    The paper combines dense, BM25, and knowledge-graph retrieval with CombSUM, reranking, MMR, per-chunk attribution verification, and multi-chunk fallback in a fully offline pipeline.

  • Results

    78.4% Exact Match accuracy, a 14.6-point improvement over single-stage dense retrieval, is reported on a 312,000-chunk SNOC corpus.

  • Takeaways & Limitations

    Explicit span-level attribution checking is identified as essential for safety-critical telecom operational workflows.

  • Takeaways & Limitations

    The 2.15 s end-to-end latency may be unsuitable for real-time alert pipelines requiring sub-second responses.

Abstract

from arXiv · show

Telecom Service and Network Operations Centers (SNOCs) rely on large collections of cloud documents, including Standard Operating Procedures (SOPs), vendor technical manuals, incident reports, and configuration guides, to maintain uninterrupted network operations. During critical incidents, engineers must quickly retrieve accurate information, yet traditional keyword based and single stage retrieval approaches often struggle to provide precise results. This paper presents Athena for Cloud Knowledge Base, a fully offline, multi agent Retrieval Augmented Generation (RAG) framework designed for enterprise cloud document search in Vodafone Idea's SNOC environment. The system integrates dense retrieval using E5 Large V2 embeddings, BM25 sparse retrieval, and Knowledge Graph expansion within a LangGraph based orchestration framework. Retrieved candidates are fused using Weighted CombSUM, followed by cross encoder reranking and Maximal Marginal Relevance (MMR) to obtain a diverse and relevant evidence set. To improve answer reliability, the framework performs per chunk LLM evaluation with explicit attribution verification, assessing each MMR selected chunk independently before generating a response. Unsupported or weak evidence is discarded, and if no chunk satisfies the verification criteria, the system automatically evaluates multiple chunks together as a fallback. Experiments on a corpus of 4200 SNOC cloud documents containing 312000 indexed chunks show that the proposed approach achieves an MRR at 10 of 0.910 and an Exact Match (EM) score of 78.4 percent, outperforming single stage dense retrieval by 14.6 percentage points. The entire pipeline operates in a fully offline environment, satisfying enterprise data sovereignty requirements while delivering accurate and grounded responses for cloud document search.

I. INTRODUCTION

SNOC engineers must search heterogeneous cloud documents quickly during outages, but existing keyword, dense-retrieval, and unattributed-generation tools have important limitations. The paper proposes a multi-agent hybrid retrieval system with attribution verification and template-specific answers for this setting.

  • Motivation: SNOCs contain diverse operational documents, and engineers must rapidly locate authoritative information during active outages.The corpus includes vendor datasheets, SOP manuals, infrastructure runbooks, and historical incident tickets.
  • Problem: Existing tools miss semantic matches, exact telecom identifiers, or unsupported claims during time-critical incidents.The stated limitations affect keyword search, single-stage dense retrieval, and LLM generation without explicit attribution verification.
  • Approach: The system combines dense, BM25, and knowledge-graph retrieval through weighted CombSUM fusion.The contribution specifies three retrieval sources and weights ωv=0.50, ωb=0.35, and ωk=0.15.
  • Approach: A cascaded domain-aware agent detects telecom vendors using regex and LLM methods.The contribution targets telecom network, cloud, hardware, and operational repositories.
  • Reliability: Per-chunk evaluation checks evidence attribution before answering, with multi-chunk fallback when needed.The loop includes neighbor-need detection, sufficiency checking, generation, and attribution verification.
  • Answering: Five answer templates tailor outputs to procedures, concepts, comparisons, parameter lookups, and troubleshooting.The system also evaluates component contributions through an ablation study on 4,200 SNOC documents.

II. RELATED WORK

Prior approaches provide useful lexical, semantic, fusion, RAG, and workflow capabilities, but the paper identifies a gap in combining them for offline telecom cloud-document search. Its proposed scope addresses hybrid retrieval, vendor awareness, attribution verification, and answer-type templates.

  • Traditional IR: BM25 performs strongly on keyword queries but is less effective for semantic or paraphrased queries and does not generate synthesized answers.Its reliance on exact word matching limits semantic coverage.
  • Dense Retrieval: Dense retrieval captures paraphrases effectively but can underperform BM25 on exact-match telecom patterns.E5-Large-V2 produces 1,024-dimensional embeddings and supports asymmetric retrieval.
  • Score Fusion: CombSUM fuses normalized scores by weighted summation, using absolute score magnitudes across complementary lexical and semantic signals.This differs from rank-based fusion.
  • RAG: RAG methods improve grounding through retrieved evidence, while the cited prior methods do not combine multi-source retrieval with iterative chunk-level attribution verification.The comparison includes Fusion-in-Decoder and Self-RAG.
  • Multi-Agent Systems: LangGraph enables conditional multi-agent workflows, but prior work largely targets open-domain QA rather than offline enterprise search with strict constraints.The paper positions its setting as telecom enterprise document search.
  • Research Gap: The stated research gap covers three-source telecom retrieval, cascaded vendor detection, per-chunk attribution verification, and fully offline five-template deployment.These are the four explicitly listed areas not addressed by prior work.

C. Chunking and Metadata

The system ingests heterogeneous PDFs into section-aware, metadata-rich chunks and orchestrates retrieval and post-retrieval evaluation as separate LangGraph phases. Neighbor pointers support contextual evaluation, while templates govern answer structure.

  • C. Chunking and Metadata: The corpus is chunked hierarchically by section into approximately 512-token chunks with 10% overlap.Chunks inherit section heading, document ID, and page number metadata.
  • C. Chunking and Metadata: Each chunk stores previous- and next-chunk pointers so the evaluator can retrieve neighboring context.This metadata supports the per-chunk evaluation stage.
  • Pipeline Orchestration: The LangGraph StateGraph separates offline ingestion from real-time querying and divides querying into pre-MMR and post-MMR phases.The pre-MMR phase handles retrieval and evidence selection; post-MMR handles evaluation and answer generation.
  • User Interface Layer: The UI exposes retrieved evidence, source metadata, extracted tables, images, latency traces, state logs, and follow-up questions.Engineers can also query arbitrary PDFs through ad-hoc upload mode.
  • Query Handling: Queries are classified into five answer templates that control BM25 extraction and generation prompt formats.A regex fallback handles unavailable or unrecognized LLM classifications.

D. BM25 Keyword Extraction Agent

The retrieval stage combines query-term extraction, vendor detection, three parallel search sources, score fusion, reranking, and diversity selection. This design uses both lexical exactness and semantic or graph-based evidence before answer generation.

  • BM25 Keyword Extraction Agent: A dedicated LLM extracts high-signal telecom query terms for BM25, with vocabulary filtering and deterministic regex fallback.The fallback activates when LLM extraction fails.
  • Vendor Detection: Vendor detection merges regex and LLM results to scope vector search, BM25 retrieval, and knowledge-graph expansion.The cascade always runs a gazetteer sweep and adds LLM-guided detection when available.
  • Parallel Retrieval: Three retrieval streams run concurrently: Qdrant vector search, SQLite FTS5 BM25 retrieval, and knowledge-graph expansion.The streams return top-40, top-40, and top-15 results respectively, with approximate latencies of 8 ms, 12 ms, and 5 ms.
  • Score Fusion: Weighted CombSUM min-max normalizes source scores and assigns weights ωv=0.50, ωb=0.35, and ωk=0.15.The method uses absolute score magnitudes rather than ranks.
  • Ranking Agent: Cross-encoder reranking reduces candidates to 12, and MMR with λ=0.65 selects the final 10 evidence chunks.Reranking is the dominant pre-MMR latency component at approximately 380 ms.

H. Per-Chunk Evaluation Agent

The per-chunk evaluation agent independently tests each MMR-selected chunk, expands context when needed, generates a template-specific answer, and verifies sentence-level attribution before delivery.

  • Independent evaluation replaces naive joint-context generation by processing each MMR-selected chunk separately.
  • Neighbor-need detection fetches preceding or following chunks for up to two iterations, while off-topic chunks are skipped.
  • Expanded anchor-and-neighbor context is checked for sufficiency before answer generation; insufficient chunks are discarded.
  • Mistral-7B-Instruct-v0.2 generates low-variance, template-specific answers from expanded context with inline source citations.Generation uses vLLM at temperature 0.1.
  • Each answer sentence receives a binary attribution label, and answers scoring below 0.70 are rejected.The system falls back to deduplicated merged contexts after exhausting all 10 MMR chunks.

A. E5-Large-V2 Embeddings

E5-Large-V2 creates asymmetric, unit-normalized query and passage embeddings for semantic retrieval, while BM25 supplies lexical scoring based on term frequency and inverse document frequency.

  • E5-Large-V2 uses asymmetric query: and passage: prefixes to preserve its learned directional relevance signal.
  • Query and passage vectors are mapped to unit-norm embeddings in R1024, with passage vectors computed offline and query vectors at search time.
  • Under ℓ2-normalisation, cosine similarity becomes the dot product between query and passage embeddings.
  • BM25 uses term frequency, document frequency, corpus size, chunk length, and average chunk length to compute lexical relevance.
  • BM25Okapi uses k1=1.5, b=0.75, and a Robertson–Spärck Jones IDF variant with a +1 constant ensuring non-negative IDF scores.

C. Weighted CombSUM Fusion

Weighted CombSUM combines vector, BM25, and knowledge-graph retrieval scores after source-wise min-max normalization, then passes candidates to reranking and diverse evidence selection.

  • Vector, BM25, and KG sources return candidate sets of 40, 40, and 15 chunks respectively before score fusion.
  • Each source score is min-max normalized over its own candidate set, and absent-source chunks receive a score of zero.
  • Zero scores for absent-source chunks preserve additive rewards for candidates retrieved by multiple sources.
  • The fused candidates are cross-encoder reranked, producing a top-12 set for subsequent MMR selection.
  • MMR selects 10 evidence chunks by balancing cross-encoder relevance against similarity to already-selected chunks, using λ=0.65.The first chunk is selected by relevance alone because the diversity term is zero when the selected set is empty.

E. Attribution Scoring

Attribution scoring evaluates whether generated answer sentences have explicit supporting spans in expanded evidence, using a threshold to reject insufficiently grounded answers.

  • The attribution score averages the fraction of atomic answer sentences linked to at least one supporting evidence span.
  • Attr(σ, e)=1 when the LLM identifies a supporting span for sentence σ in evidence e, and 0 otherwise.
  • Answers with attribution score below τ=0.70 are rejected and the system proceeds to another MMR chunk.
  • Unlike soft NLI entailment, explicit evidence pointers create directly auditable answer–source links.
  • Evaluation uses offline experiments and reports MRR@10, NDCG@10, EM, F1, Precision@5, Recall@5, and wall-clock latency.

D. Ablation Analysis

The ablation shows that attribution-focused evaluation contributes most to performance, while retrieval and orchestration components support different query types and evidence needs.

  • Query-type behavior: KG expansion retrieves related SOPs, configuration guides, troubleshooting documents, and historical incidents for operational and fault-related queries.The incident example describes BGP neighbor-down retrieval, while troubleshooting can begin from a generic symptom.
  • Query-type behavior: BM25 supports exact model matching for hardware manuals, while template classification generates procedures and comparison-oriented responses for relevant queries.Deployment checklist queries use procedure_sop, and comparative version queries use comparison_decision.
  • Ablation findings: +12.9 EM points come from the per-chunk evaluation loop and attribution verifier over CombSUM+CE alone.The ablation identifies these as the most impactful components for safety-relevant SNOC workflows.
  • Ablation findings: +3.5 EM points come from KG expansion, with the largest impact on entity-dense queries.
  • Ablation findings: +2.6 EM points come from vendor detection, which addresses vendor-specific queries comprising 47% of the test set.
  • Verification design: Explicit span-level attribution requires evidence linking and produces binary per-sentence judgments that engineers can interpret directly.This design is presented as more precise and interpretable than NLI-based groundedness methods.

C. Limitations

The system’s main boundaries are latency, rule-based knowledge-graph coverage, score-fusion sensitivity, and the need for future extensions to multimodal retrieval and continuous learning.

  • Limitations: 2.15 s end-to-end latency may be unsuitable for real-time alert pipelines requiring sub-second responses.The cross-encoder and LLM together account for 78% of total latency.
  • Limitations: Rule-based knowledge-graph construction may miss implicit relations in informal incident post-mortems.A fine-tuned relation-extraction model is proposed to improve recall and enable richer multi-hop traversal.
  • Limitations: CombSUM is sensitive to score-distribution mismatches, because outlier scores can dominate rankings after min-max normalization over small result sets.
  • Future extensions: Future multimodal retrieval could add diagram pages alongside text chunks using a CLIP-based image encoder.The proposed extension targets topology diagrams, interface tables, and KPI trend charts in SNOC documents.
  • Future extensions: Engineer feedback could support nightly fine-tuning of the template classifier and attribution verifier.
  • Future extensions: SNOC-specific contrastive fine-tuning of E5-Large-V2 could improve synonym handling across vendors and calibration for CombSUM fusion.
Loading 2609.01618v1…