Loom: Weaving Diagnostic Strands into Free-Text Consensus via Embedding-Space Reweighting

Ron Begleiter; Katya Egert Berg; Gilad Saban; Gil Shabat

Source-linked AI summary

Loom: Weaving Diagnostic Strands into Free-Text Consensus via Embedding-Space Reweighting

Ron Begleiter, Katya Egert Berg, Gilad Saban, Gil Shabat

arXiv:2609.02649v1Sep 2, 2026cs.AIcs.CLcs.LG

TL;DR

Industrial NLP systems need reliable aggregation of conflicting free-text hypotheses without the latency and fragility of iterative LLM agents or the discrete limits of traditional weak supervision. Loom addresses this gap by embedding and centroid-reweighting Diagnostic Strand outputs before one lightweight synthesis call, achieving Pareto-frontier accuracy–efficiency on OpenRCA. Its deployment experience also identifies dataset-dependent accuracy limits and trade-offs from single-shot synthesis.

Problem
Aggregating noisy, conflicting free-text hypotheses is difficult because autonomous LLM agents face context, hallucination, latency, and cost limitations, while weak supervision is restricted to discrete classes.
Method
Loom uses modular Diagnostic Strands to emit episode-specific templated hypotheses, embeds them in continuous space, reweights them around a centroid, and passes the ordered consensus to one synthesis LLM.
Results
Loom occupies the OpenRCA accuracy–efficiency Pareto frontier, matching the agent on Bank and Market-2, trailing on Market-1 and Telecom, and using one LLM call per incident.
Takeaways & Limitations
Loom provides a practical, deterministic, auditable blueprint for real-world automated diagnostics and supports SME trust under industrial latency and cost constraints.
Takeaways & Limitations
Loom is materially less accurate than the iterative RCA-Agent baseline on Market-1 and Telecom, and single-shot synthesis has limits on highly complex datasets.

Abstract

from arXiv · show

Aggregating noisy, conflicting textual hypotheses into a reliable consensus is a fundamental challenge when deploying NLP systems in real-world industrial settings. While monolithic Large Language Model (LLM) agents offer unbounded expressivity for tasks like Root Cause Analysis (RCA), they suffer from context limits, compounding hallucinations, and prohibitive inference latency. Traditional weak supervision offers statistical rigor but is mathematically restricted to discrete classes. We present Loom, a generative consensus framework deployed for real-world RCA that bridges these paradigms. Loom aggregates open-form hypotheses emitted by modular heuristics (diagnostic templates dynamically populated with episode-specific entities, times, and metrics) by projecting them into a continuous embedding space, and resolves conflicting signals with an iterative centroid-based reweighting algorithm. The resulting consensus weights ground a single lightweight LLM synthesis step. Evaluated on the OpenRCA benchmark, Loom occupies the accuracy--efficiency Pareto frontier: it matches a state-of-the-art autonomous agent on Bank and Market-2 and trails on Market-1 and Telecom, while using a single LLM call per incident on all four datasets ($\sim$26$\times$ faster; $\sim$33$\times$ with an 8B-parameter synthesizer). We discuss our deployment experience, highlighting lessons learned regarding the trade-offs between agentic depth and inference latency, negative results in redundancy detection, and how deterministic consensus fosters trust among Subject Matter Experts~(SMEs).

1 Introduction

Loom addresses the difficulty of aggregating open-form, noisy RCA hypotheses by combining modular heuristics, continuous-space reweighting, and lightweight LLM synthesis. Its deterministic design targets industrial latency, cost, and auditability constraints while retaining semantic expressivity.

Open-form RCA hypotheses are difficult to aggregate because autonomous LLM agents incur context, hallucination, latency, and cost limitations, while weak supervision is restricted to discrete classes.
Loom uses Diagnostic Strands to emit episode-specific templated hypotheses, projects them into continuous space, and applies iterative centroid-based reweighting before synthesis.
Deterministic consensus makes Loom auditable and supports SME trust while reducing reliance on costly iterative agentic loops.
Loom’s contributions include a real-world RCA framework, deterministic conflict resolution, deployment case studies, OpenRCA evaluation, and analysis of redundancy detection and single-shot synthesis limits.

2 Background and Related Work

Prior approaches provide efficient extraction, discrete weak-supervision aggregation, or expressive iterative reasoning, but each leaves a gap for scalable free-text consensus. Loom instead pre-aggregates structured hypotheses in embedding space before LLM synthesis.

AIOps and Automated Log Analysis: Automated log analysis efficiently extracts symptoms and matches spatial-temporal patterns but lacks generative semantic capabilities for cohesive RCA descriptions.
Traditional Weak Supervision: Traditional weak supervision aggregates noisy heuristic rules for discrete labels but treats semantically similar textual explanations as completely disagreeing.
Autonomous RCA Agents: Autonomous RCA agents use expressive multistep reasoning but suffer from unbounded search spaces, token costs, latency, and compounding hallucinations.
LLM Consensus and Multi-Agent Debate: Iterative consensus and debate methods resolve conflicts through repeated LLM generation, which scales poorly to large evidence spaces and real-world latency requirements.
Loom’s Position: Loom pre-aggregates structured hypotheses in continuous space, delivering a denoised ranked slate to the synthesis LLM without aggregation cost scaling with hypothesis count or repeated invocations.

3 The Loom Framework

Loom evaluates modular Diagnostic Strands, embeds their episode-specific outputs, and iteratively reweights them around a semantic centroid. The resulting ordered evidence is passed to a synthesis LLM for a coherent RCA report.

3 The Loom Framework: At inference time, fired strands produce textual RCA outputs that Loom embeds, reweights through centroid alignment, and supplies to a synthesis LLM.
3.1 Diagnostic Strands (DSs): Diagnostic Strands are programmatic heuristics that populate diagnostic templates with telemetry such as hostnames, KPI names, counts, and timestamps, or abstain.
3.2 Iterative Embedding-Centroid Reweighting: Static redundancy detection groups highly similar strands and divides their effective weights by group size to prevent redundant rules from dominating.
3.2 Iterative Embedding-Centroid Reweighting: Initial strand weights come from expert-curated reliability metadata, while runtime embeddings represent the fired textual outputs in a continuous vector space.
3.2 Iterative Embedding-Centroid Reweighting: The iterative procedure computes a weighted embedding centroid and updates each strand’s weight according to its alignment with that centroid.
3.3 Textual Resolution and LLM Synthesis: Consensus weights impose a strict ordering on textual outputs, which the synthesis LLM uses to produce a coherent RCA from a small denoised context.
3.3 Textual Resolution and LLM Synthesis: The pipeline combines diverse data sources, programmatic strands, semantic embedding, centroid reweighting, and final LLM synthesis into one inference flow.

4 Real-World Deployment and Evaluation

Loom was deployed in large-scale computing environments and evaluated on OpenRCA as a deterministic, efficient alternative to iterative RCA agents. Its deployment examples show coherent RCA synthesis from weighted diagnostic evidence, while evaluation reveals strong efficiency, benchmark-dependent accuracy, and limitations in redundancy handling and single-shot disambiguation.

Real-World Deployment: Loom provides operators with a deterministic, auditable evidence trail across distributed failures and silent networking degradations, supporting SME adoption.Its conflict-resolution burden is shifted to the continuous consensus stage rather than the synthesis LLM.
Real-World Deployment: Loom identifies localized hardware causes and their blast radius in distributed training failures despite hundreds of secondary timeout errors.The case study attributes failure to hosts node-011-T07 and node-011-T15 and reports a 507-host impact.
Real-World Deployment: A composite strand receives weight 0.94 by detecting switch-side ECN marking alongside absent host-side CNP response, enabling synthesis of a broken feedback-loop RCA.The resulting narrative connects congestion, missing throttling, and likely configuration or routing issues preventing CNP delivery.
Evaluation: On every OpenRCA dataset, Loom uses 1 LLM call and approximately 22 seconds per incident versus approximately 62 calls and nearly 10 minutes for RCA-Agent, a approximately 26x speedup.It matches the agent on Bank and Market-2, exceeds partial accuracy on Bank at 51.22% versus 49.15%, and trails on Market-1 and Telecom.
Evaluation: Replacing Claude 4.6 with Llama-3.1-8B reduces overall strict accuracy by approximately 3.7 percentage points, while enabling approximately 33x speedup on Bank.The judges tie on single-element queries, and the smaller model ties Claude 4.6 on easy queries.
Lessons Learned and Ablations: Static docstring redundancy grouping improves neither reliability nor accuracy in the Bank setting, where removing it coincidentally adds 5.88 percentage points strict accuracy.The authors diagnose docstring grouping as too coarse for true redundancy because it can remove entire reason families from the top-K candidates.

5 Conclusion

Loom provides a deployable framework for aggregating templated, episode-specific diagnostic hypotheses in continuous embedding space. Its deterministic reweighting and lightweight synthesis achieve substantial efficiency gains while supporting industrial diagnostic operations.

Loom aggregates templated, episode-specific diagnostic hypotheses in continuous embedding space.
∼26–33× inference speedup on OpenRCA enables small 8B local models under industrial latency and cost constraints.
Deterministic, auditable processing fosters Subject Matter Expert trust in automated diagnostics.

6 Limitations

Loom’s scalability and inference-latency advantages come with strategic trade-offs, including bounded diagnostic coverage, cold-start requirements, embedding sensitivity, and weaker accuracy on Market-1 and Telecom. Its specified, deterministic framework supports reproducibility in principle, subject to backend API nondeterminism.

Accuracy limitations: 11.43 pp strict on Market-1 and 11.77 pp strict on Telecom separate Loom from the iterative RCA-Agent baseline.Loom scores 28.57% versus 40.00% on Market-1 and 29.41% versus 41.18% on Telecom.
Coverage and governance: Loom’s diagnostic coverage is bounded by its predefined Diagnostic Strand catalog, unlike bottom-up agents’ theoretically unbounded coverage.The catalog boundary makes the system’s solvable and unsolvable cases explicit, enabling operators to address blind spots systematically.
Coverage and governance: New Black Swan failures require a human expert or offline LLM extraction pipeline to define and validate a Diagnostic Strand before diagnosis.This delays immediate zero-shot resolution but enforces stricter governance than exploratory zero-shot diagnosis.
Representation dependence: Embedding-space aggregation is sensitive to whether textual embeddings preserve distinctions between subtle technical root causes.Poor semantic geometry can cause the iterative reweighting algorithm to conflate distinct causes.
Inference trade-offs: Single-shot synthesis reduces inference latency but can struggle to weigh competing high-confidence candidates with saturating anomaly scores.The paper terms this failure mode Judge Overfocus and observes it on Telecom and Market benchmarks.
Reproducibility: The specified framework and deterministic temperature-0 pipeline make independent reimplementations reproducible in principle, apart from backend LLM-API nondeterminism.The paper states that OpenRCA strict and partial accuracy figures should be numerically identical up to that nondeterminism.

A Example Diagnostic Strand

The example Diagnostic Strand combines unstructured syslog signatures with structured fabric counters to populate an episode-specific failure hypothesis. The final synthesis prompt then restricts the LLM to mathematically denoised evidence rather than independent investigation.

Diagnostic Strand: The example Diagnostic Strand diagnoses non-fatal accelerator-fabric link failures using syslog signatures and structured fabric counters.It fills a diagnostic template with episode-specific hosts and counts.
Diagnostic Strand: Episode-specific hosts and counts are inserted into the strand’s diagnostic template.This connects heuristic detection to the concrete entities and measurements in an incident.
Synthesis prompt: The final synthesis prompt forces the LLM to reason over ordered, mathematically denoised evidence instead of acting as an independent investigator.The stated goal is to keep the final narrative grounded in raw telemetry.

C Offline Learning Pipeline

Loom constructs its Diagnostic Strand catalog offline by converting unstructured operational knowledge into executable Python functions. This pre-compilation avoids blind diagnosis of massive telemetry dumps during inference.

Offline extraction: Generative AI agents extract Diagnostic Strands from system documentation and incident tickets.The offline pipeline processes unstructured knowledge bases before deployment.
Offline extraction: The extracted strands are compiled into discrete, executable Python functions.This turns operational knowledge into reusable diagnostic heuristics.
Inference-time role: Pre-compilation prevents Loom from blindly diagnosing massive telemetry dumps at inference time.The catalog is constructed before incidents are processed online.

D InfiniBand Switch Environmental Factor

Correlated degradation across most active InfiniBand switch ports indicates a common-mode environmental cause rather than independent port faults. Thermal telemetry strengthens a thermal attribution, but without it the strand leaves the specific environmental factor unresolved.

Failure pattern: Correlated degradation of many ports on one switch is uncommon for isolated cable or optic failures.The diagnostic rule treats chassis-wide degradation as evidence against independent per-port faults.
Detection rule: The strand marks ports degraded when effective bit-error rate exceeds 10^-12 or packet-loss-retry ratio is at least 10^-2.It groups InfiniBand telemetry by switch before applying these thresholds.
Decision rule: The environmental hypothesis fires when more than 50% of a switch’s active ports, with at least three ports, are degraded.This majority condition makes a chassis-level explanation appropriate.
Corroboration: Temperature of at least 80°C for the device, 70°C for a module, or thermal flags names an environmental/thermal factor.Without thermal corroboration, the strand reports a common-mode environmental factor but leaves heat versus dust, smoke, or airflow ungrounded.

E OpenRCA Diagnostic Strand Catalog

The OpenRCA evaluation uses Diagnostic Strand catalogs, with one shared catalog for Market-1 and Market-2. Raw hypothesis generation can greatly exceed the number of fired strand functions, and synthesis receives only ranked top-K hypotheses.

Hypothesis volume: Bank averages 340.93 raw hypotheses but only 7.92 fired functions because one strand function can instantiate many entity-, time-, and metric-specific hypotheses.Only the ranked top-K hypotheses are passed to synthesis.
Catalog structure: Market-1 and Market-2 share a single Diagnostic Strand catalog.The catalog listing also records approximately one engineer-week for construction per dataset.
Construction cost: Catalog construction required about one engineer-week per dataset for schema integration, authoring, testing, and evaluation.Market-1 and Market-2 use one shared catalog despite being separate benchmarks.

F Worked OpenRCA Example

The worked Bank example shows Loom ranking many diagnostic hypotheses before a constrained synthesizer produces the correct root cause. The highest-ranked CPU hypothesis matches the ground truth and yields an official strict score of 1.0.

Worked result: The Bank query’s ground truth is Tomcat03, high CPU usage, at 19:42, with an official strict score of 1.0.The synthesizer emits the same component and reason in OpenRCA JSON.
Diagnostic strands: Figure 2’s diagnostic strand cross-references a syslog signature with two fabric counters, fills episode-specific slots, and returns ABSTAIN when conditions do not fire.This illustrates how modular heuristics produce structured, telemetry-backed textual hypotheses.
Synthesis: The synthesis prompt constrains the LLM to reason over ordered Diagnostic Strand outputs rather than investigate independently.The prompt is intended to keep the final narrative grounded in telemetry-backed heuristics.
Consensus inputs: Loom reduces 206 instantiated Bank hypotheses to one hypothesis per component-reason pair and retains the top K=8 for synthesis.The fired strands include CPU evidence plus disk, JVM, network, trace, and GC-related distractors.
Evidence: The CPU template identifies a sustained anomaly on Tomcat03 lasting four consecutive minutes with high-direction peak z = 236.67.The surrounding window includes CPU saturation and corroborating GC allocation-failure logs and trace spans.
Ranking: The true hypothesis ranks first with score 0.898: Tomcat03, high CPU usage, 19:43.The strand’s 19:43 time hint reflects the first sustained minute, while the synthesizer reports 19:42 to match the record label.