Too many clients spoil the logs: distributed parsing, aggregation techniques and metrics

1. Introduction

• It examines how client counts and log-data volumes affect parser effectiveness, defining effectiveness by equivalence to ground-truth templates.It also evaluates how aggregation techniques transfer parsing knowledge among local parsers and may improve their performance.Source: §1 · p. 1
• The paper reviews metrics and introduces a taxonomy to explain their relevance and application in distributed parser deployments.Source: §1 · p. 2
• The paper benchmarks multiple parsers in distributed environments, both with and without federated techniques.The study is motivated by the limited focus on distributed parsing itself, as prior work often assumes logs are parsed before distribution.Source: §1 · pp. 1–2
• It proposes lessons and recommendations for designing and deploying parsers in federated environments.Source: §1 · p. 2

2. Background

• Log parsing extracts templates from logs to create structured representations for downstream tasks such as anomaly detection.Templates identify event types and remain static for each type, while parameters vary.Source: §2 · p. 2

2.1. Types of Parsers

• Parser families include frequency-, similarity-, heuristic-, semantic-, and LLM-based approaches.They respectively use frequent tokens, clustering, heuristics, neural models, or large language models to derive templates.Source: §2.1 · pp. 2–3

2.2. Parsing Caches

• Parsing caches store and reuse previously generated templates to reduce repeated LLM queries and improve consistency.On a cache miss, the system queries an LLM and either inserts a template or refines an existing one by replacing differing tokens with a wildcard.Source: §2.2 · p. 3

3. Related work

• Fu et al. (2023) and Khan et al. (2024) recommend the distinguishability concept to tackle the mismatch between log parsers and log anomaly metrics.The comparisons cover detectors using only log event IDs; LogRobust is the exception, also leveraging log templates.Source: §3 · p. 3
• Parsing errors can degrade anomaly-detection performance, and detectors using templates tend to be more robust to those errors.Sedki et al. (2024) study how metric choice influences reported outcomes; Beck et al. (2025) assess 12 parser algorithms and identify influential components.Source: §3 · p. 4
• Prior studies often assume logs are parsed before distribution, leaving how parsing would work in real-world federated deployments unaddressed.The paper identifies federated parsing in distributed systems as an unexplored aspect, despite its practical relevance to proposed federated anomaly detection.Source: §3 · p. 4

4. Methodology

• The methodology introduces the pipeline, possible federated-parsing scenarios, aggregation strategies, and metrics for the experiments.Source: §4 · p. 4

4.1. Federated pipeline

• Federated parsing can jointly use parsers across organizations, addressing the challenge that local parsers may not encounter every log type.This matters as services may appear on some nodes but rarely on others; the example pipeline routes logs from applications to parsers and then downstream tasks.Source: §4.1 · p. 4
• The study also considers streaming, in which each message must be parsed close to real time.Real-time template updates can change how previously seen log types are interpreted, potentially affecting downstream tasks.Source: §4.1 · p. 4

4.2.1. Mathematical definition

• A parser maps an unstructured log to a structured log using a template set that represents inferred constant message portions.Log statements combine constant text with variables, such as a user name or application name.Source: §4.2.1 · p. 4, §4.2 · p. 4
• When updates are allowed, new log types can change the template set; with a fixed parser, unseen types may be parsed incorrectly or left empty.A representative log set should yield a template set close to ground truth, while aggregation can bring together templates from less representative local sets without sharing raw logs.Source: §4.2.1 · pp. 4–5

4.2.2. Federated parsing

• Federated parsing processes local log batches, merges the resulting templates into a global set, and updates each client’s local templates.Communication can be centralized through a server or decentralized through a Paxos-based voting protocol.Source: §4.2.2 · p. 5
• Distributed parsers need a unified template-to-Event-ID mapping, since inconsistent IDs across clients complicate downstream federated anomaly detection.Centralized communication sends new templates to a server for redistribution; decentralized communication uses a Paxos-based vote.Source: §4.2.2 · p. 5

4.3. Aggregation strategies

• Aggregation techniques create a unified agreement on which template corresponds to each event or Event ID across participating clients.Prior federated anomaly-detection work often assumes templates and consistent Event IDs are already available.Source: §4.3 · p. 5

4.3.1. Concatenation

• Concatenation merges unique templates into one set and assigns them corresponding Event IDs.It is presented as the most straightforward aggregation method.Source: §4.3.1 · p. 6

4.3.2. Parsing Caches

• Parsing caches store extracted templates, merge similar ones, prune incorrect ones, and match logs; the experiments use LILAC and LogBatcher caches.When a log does not match an existing template, template extraction can run separately from the cache and distribute templates to a centralized or client-specific cache.Source: §4.3.2 · p. 6

4.4. Metrics analysis

• The paper groups parsing metrics into message, template, and distribution categories, and separately distinguishes continuous from discrete measures.Distribution metrics address both template quality and volume; the paper defines quality in terms of templates usable to match multiple logs.Source: §4.4 · p. 6
• Group accuracy checks whether logs from the same template are assigned to the same group, and is classified as template-continuous.The paper excludes Precision and Recall template accuracy because it says they add no further insight.Source: §4.4 · pp. 6–7
• Template Set (TS) adds a discrete template-based measure by checking whether original templates are present in the parsed logs.The generated template set is compared with the ground-truth template set.Source: §4.4 · p. 7
• The proposed Ratio of templates (RT) measures the number of produced templates relative to the expected total, independent of template assignments.Found ratio (FR) measures how many logs receive a template, without assessing whether the assignment is correct.Source: §4.4 · p. 7
• Parsing Score (PS) combines previously presented metrics in a weighted composite intended to provide a more informative representation.Its weights α, β, and γ sum to approximately 1; the paper positions it as a more conventional approach to combining message- and template-level scores.Source: §4.4 · p. 7

5. Experiment design

• The experimental code is open source.Source: §5 · p. 7

5.1. Experiment setup

• Training-data volume and its division among clients are varied, while parsers process the full dataset at prediction time without changing templates.Each setup emulates streaming by training on a subset; the resulting templates remain fixed for downstream tasks.Source: §5.1 · p. 7
• The study restricts experiments to an IID configuration to simplify the setup.The authors contrast this choice with studies that also evaluate non-IID conditions.Source: §5.1 · pp. 7–8

5.2. Data analysis

• The benchmark covers 6,720 distributed parsing setups and analyzes 40,320 metric data points, with high-level plots used to highlight principal trends.Results were logged and released in a public dataset because all outcomes could not be displayed in the paper.Source: §5.2 · p. 8
• The analysis uses high-level plots to compare continuous and discrete versions of metric types, and to compare aggregation techniques across data distributions.The accompanying interpretation explains how differences between selected metric pairs relate to template matching, similarity, and generalization.Source: §5.2 · p. 8

5.3. Parsers used

• The parser selection spans multiple parsing strategies but excludes supervised semantic-based parsers, which require templates for training.The study includes frequency-, similarity-, heuristic-, and LLM-based approaches.Source: §5.3 · p. 8

5.4. Dataset selection

• HDFS and BGL were selected as widely used datasets, and DetectMate’s tree matcher standardized template matching across the experiments.DetectMate failed to match 6% of BGL templates, partly because some logs lacked correct templates and the ground truth lacked a generic template.Source: §5.4 · p. 9

5.5. Setup

• Experiments ran on a laptop with 16 GB RAM and an Intel Core Ultra 7 165Ux14 CPU; LogBatcher used gpt-5.4-mini.The system ran Ubuntu 24.04.3 LTS.Source: §5.5 · p. 9

6.1. Local performance

• LogBatcher performance declines as client count and data volume decrease, while Drain and Brain are more resilient and sometimes perform better with less training data.The compared configurations range from three nodes with all streaming data available for training to 100 clients trained on 1% of the data.Source: §6.1 · pp. 9–10, Figure 6 · p. 10, Figure 7 · p. 10
• With Brain on HDFS and only three clients, more data does not always yield better performance; LogBatcher instead gains predictably as client counts fall and dataset size grows.The authors hypothesize that some logs included during training may harm performance.Source: §6.1 · p. 10, Figure 8 · p. 11, Figure 9 · p. 11
• LogBatcher is more sensitive to data distribution and performs worse on BGL than the simpler, older AEL method.The reported performance degradation is more abrupt for BGL than for HDFS; BGL has a larger and more evenly distributed set of event types, while HDFS has fewer types with a small, frequent subset.Source: §6.1 · p. 10

6.2. Federated setup

• The aggregation comparison contrasts three clients using 100% of the data with 100 clients using 1%, using the metric introduced in Equation 7.Additional aggregation results across configurations are available in the code repository.Source: §6.2 · p. 10
• Aggregation generally enhances parser performance, but not in every case; the methods have broadly comparable performance, with concatenation the least computationally complex.The authors consider concatenation a more suitable candidate for future work; aggregation is especially beneficial when many templates are distributed across clients.Source: §6.2 · p. 10
• Concatenation benefits distribution-sensitive LogBatcher on HDFS, while Brain’s NED improvement is debatable; TS improves in both cases.The authors interpret the TS results as improved recovery of original templates, while Brain can identify similar templates that serve a comparable role without strictly requiring concatenation.Source: §6.2 · pp. 10–11

• More clients or less data per client reduces parser performance, especially for LogBatcher; degradation is sharper on BGL than HDFS, whose event distribution is more concentrated.Some parsers can perform better on smaller HDFS datasets, possibly because rare log messages harm accuracy.Source: §7 · pp. 11–12
• Aggregation improves performance overall, but gains vary by metric and task; benefits are strongest for BGL, especially LogBatcher, and limited or absent for Drain and Brain on HDFS.Source: §7 · pp. 12–13
• Aggregation adds system complexity and privacy risks, including possible exposure of names or passwords embedded in generated templates.Source: §7 · p. 13
• Aggregation is essential when federated anomaly detection needs aligned event IDs, but may be impractical when few event types and clients yield gains below its added complexity.Source: §7 · p. 13
• Updating templates across nodes when new log types appear can create high communication costs; aggregating after clients discover a set number of templates can reduce overhead.Source: §7 · pp. 13–14
• Organizations can share established template sets federatively so new systems with similar infrastructure can benefit from accumulated parsing knowledge.Source: §7 · p. 14

8. Threat to validity

• The study identifies threats that may jeopardize the validity of its experiments.Source: §8 · p. 14

8.1. Ground truth templates

• Ground-truth templates can contain human errors, and closer template matches do not necessarily yield better anomaly-detection performance, so some task results may mislead.Source: §8.1 · p. 15

8.2. Parser variety

• The study includes diverse parsers to reduce single-parser bias, but conclusions may change as parser algorithms evolve.Source: §8.2 · p. 15

8.3. Data leakage

• LogBatcher pretraining may have exposed it to the evaluated log data, potentially giving it an advantage, although cited studies question exact memorization.Source: §8.3 · p. 15

9. Conclusion

• The paper evaluates aggregation strategies for federated parsing and introduces a taxonomy of metrics to interpret distributed-parsing results.Source: §9 · p. 15
• 6,720 distributed parsing experiments produced 40,320 individual results, analyzed for insights and practical recommendations for parser deployment.Source: §9 · p. 15
• Future work will add newly published datasets and develop aggregation techniques for validation against this study’s baselines.Source: §9 · p. 15