Source-linked AI summary
Hue: A User-Adaptive Parser for Hybrid Logs
Junjielong Xu, Qiuai Fu, Zhouruixing Zhu, Yutong Cheng, Zhijing Li, Yuchi Ma, Pinjia He
TL;DR
Existing parsers do not natively handle mixed single- and multi-line logs and have difficulty incorporating domain knowledge about ambiguous tokens. Hue combines typed key casting, line aggregation, pattern extraction, and user feedback; it reports higher average grouping accuracy than existing parsers on hybrid logs and deployment for daily parsing in Huawei Cloud.
Problem
Existing parsers assume single-line logs and have difficulty distinguishing constants from variables when logging statements are inaccessible.
Source: §1 · pp. 1–2Method
Hue parses hybrid logs online using key casting, line aggregating, pattern extracting, and a merge-reject strategy for user feedback.
Source: §1 · p. 2Results
0.845 average grouping accuracy on hybrid logs exceeds the best existing-parsers result of 0.563 on average.
Source: §1 · p. 2Takeaways & Limitations
Hue has been deployed in Huawei Cloud services for daily hybrid log parsing.
Source: §1 · p. 2Hue’s performance may degrade on logs without headers, which the authors identify as future work for hybrid-log parsing.
Source: §7 · p. 11
Abstract
from arXiv · showhide
Log parsing, which extracts log templates from semi-structured logs and produces structured logs, is the first and the most critical step in automated log analysis. While existing log parsers have achieved decent results, they suffer from two major limitations by design. First, they do not natively support hybrid logs that consist of both single-line logs and multi-line logs (\eg Java Exception and Hadoop Counters). Second, they fall short in integrating domain knowledge in parsing, making it hard to identify ambiguous tokens in logs. This paper defines a new research problem, \textit{hybrid log parsing}, as a superset of traditional log parsing tasks, and proposes \textit{Hue}, the first attempt for hybrid log parsing via a user-adaptive manner. Specifically, Hue converts each log message to a sequence of special wildcards using a key casting table and determines the log types via line aggregating and pattern extracting. In addition, Hue can effectively utilize user feedback via a novel merge-reject strategy, making it possible to quickly adapt to complex and changing log templates. We evaluated Hue on three hybrid log datasets and sixteen widely-used single-line log datasets (\ie Loghub). The results show that Hue achieves an average grouping accuracy of 0.845 on hybrid logs, which largely outperforms the best results (0.563 on average) obtained by existing parsers. Hue also exhibits SOTA performance on single-line log datasets. Furthermore, Hue has been successfully deployed in a real production environment for daily hybrid log parsing.
1 INTRODUCTION
Log parsing structures semi-structured messages for downstream analysis, but existing parsers focus on single-line logs and struggle with ambiguous tokens. Hue introduces online hybrid log parsing with user feedback, outperforming existing parsers on hybrid logs and achieving strong single-line results.
1 INTRODUCTION
- Existing parsers assume single-line inputs, although logs can mix single-line messages with multi-line exceptions, counters, and other content.Log parsing extracts templates and structures messages for automated downstream log analysis.Source: §1 · pp. 1–2
- When logging statements are inaccessible, distinguishing constants from variables is difficult, and downstream tasks can require different valid templates for the same logs.The paper describes differing templates for “display ipv6” in root cause analysis and user profile construction.Source: §1 · p. 2
- 0.845 average grouping accuracy on hybrid logs outperforms the best existing-parsers result of 0.563 on average.Hue also achieves the highest grouping accuracy on 8 of 16 single-line datasets, the most among compared parsers.Source: §1 · p. 2
- Hue combines key casting, line aggregating, and pattern extracting with a merge-reject strategy that lets users reject potentially ambiguous template updates.It is evaluated on three hybrid-log datasets and sixteen Loghub single-line datasets.Source: §1 · p. 2
- Hue has been deployed in Huawei Cloud services to parse daily hybrid logs.The paper also reports SOTA single-line parsing accuracy without human feedback.Source: §1 · p. 2
2 MOTIVATION
Hybrid logs combine message types that existing parsers may split or mishandle, despite their value in operational and analytical tasks. Ambiguous tokens also make templates task-dependent, motivating a manual feedback mechanism.
2.1 Neglected Hybrid Logs
- Hybrid logs mix single-line and multi-line messages, including event logs, console echoes, tracebacks, and KPI tables, and occur in open- and closed-source systems.Centralized collection across service components can bring these varied outputs together.Source: §2.1 · p. 2
- Treating each line of a multiline message as a separate log can produce incorrect parsing, while flattening with special regex may still mishandle tables with variable row counts.The paper proposes key casting and line aggregating to address these issues.Source: §2.1 · p. 3
2.2 Ambiguous Template Tokens
- Ambiguous tokens can yield multiple plausible templates: analysts may differ on merging logs or prioritize actions versus objects for different downstream tasks.The paper introduces a manual feedback mechanism within the parse tree to improve template compatibility and generalizability.Source: §2.2 · p. 3
3 PRELIMINARY
The paper treats hybrid log parsing as parsing single-line logs, multi-line logs, or their mix, with outputs that include templates and structured variable data. It distinguishes event, table, and text logs and frames parsing as classification.
3.1 Hybrid Logs
- Hybrid logs may comprise event logs, table logs, and text logs, corresponding respectively to single-line messages and two kinds of multiline messages.The paper mainly considers these three types based on its hybrid-log analysis experience in Huawei Cloud.Source: §3.1 · p. 3
3.1.1 Event Log.
- An event-log template contains message constants and wildcards marking variables, following existing single-line parsing definitions.Event logs record a service or component operation or status and contain a header and message content.Source: §3.1.1 · p. 3
3.1.2 Table Log.
- A table-log template is its header, which captures column count and types; table content becomes parameters, and row counts may vary.Table logs contain a header and multiple parameter lines, potentially including table rows.Source: §3.1.2 · p. 3
3.1.3 Text Log.
- A text-log template consists of tokens that remain constant across lines, such as traceback error types or keys in key-value messages.Text logs are multiline messages outside the table-log category; parsing extracts those constant lines.Source: §3.1.3 · p. 3
3.2 Hybrid Log Parsing
- Hybrid log parsing aims to cover single-line logs, multiline logs, and their mix, producing a meta-file of templates and original locations plus structured variable lists.Its scope is presented as a superset of research targeting single-line logs.Source: §3.2 · pp. 3–4
- The parser predicts a message’s log type, template group within that type, and whether each token is constant or variable.The paper frames the task as classification and identifies mixed log types and constant-variable distinctions as challenges.Source: §3.2 · p. 4
4 APPROACH
Hue processes logs through key casting, line aggregating, pattern extracting, and online updating. Its key-casting preprocessing uses typed wildcards to retain distinctions among common variable types in hybrid logs.
4.1 Overview
- Hue’s pipeline comprises key casting, line aggregating, pattern extracting, and online updating, with optional human feedback guiding template revision.It reads logs line by line, splits on spaces, and passes token sequences through the parsing components.Source: §4.1 · p. 4
- Because hybrid logs include single-line messages, Hue can also parse single-line logs as existing parsers do.Source: §4.1 · p. 4
4.2 Key Casting
- Unlike generic wildcard preprocessing, which can collapse variable-only table lines, key casting maps common variable tokens to distinct typed wildcards using a general casting table.For example, the IP address in an authentication message becomes “<*ip>” while a numeric uid becomes “<*int>”; users can customize the table.Source: §4.2 · p. 4
4.3 Line Aggregating
Hue aggregates adjacent lines to distinguish event, table, and text logs, using sequence similarity and indentation to guide grouping and type assignment.
4.3 Line Aggregating
- Adjacent lines are aggregated when sequence similarity exceeds ε_a or indentation matches; a counter then classifies similarity-driven aggregation as TABLE and indentation-driven aggregation as TEXT.The similarity score is the proportion of identical tokens at corresponding positions, using average token length as the denominator.Source: §4.3 · p. 4
- Hue first treats single-line messages as EVENT logs, while multi-line messages undergo adjacent-line aggregation into blocks for later pattern extraction.Lines containing only delimiters or null characters are skipped, and log headers can delimit logs.Source: §4.3 · p. 4
4.4 Pattern Extracting
Hue uses aggregated blocks and log type to group logs by identifiers rather than clustering entire messages. Coarse grouping narrows candidate groups, and a search tree refines them.
4.4 Pattern Extracting
- Traditional message-level clustering is inefficient for long flattened hybrid logs, and variable multi-line lengths can reduce clustering accuracy.Hue instead uses block queues and log type from line aggregation to group logs sharing templates.Source: §4.4 · p. 5
- Hue forms log identifiers from event sequences, text-log first and last block common sequences, or a table log’s first single-capacity block sequence.The design assumes text templates often appear at the beginning or end and table headers are distinctive.Source: §4.4 · p. 5
4.4.1 Coarse-grained Grouping.
- Hue’s coarse grouping assigns logs using identifier token and key counts, assuming same-template identifiers share both counts.The passage notes the equal-token-count assumption is commonly used and verified in prior research.Source: §4.4.1 · p. 5
4.4.2 Fine-grained Grouping.
- Within each count-based group, Hue uses a multinomial search tree that branches first on keys and then on non-key prefix tokens.Tree depth and identifier-length limits guard against incorrect rules from overly long or short logs; identifiers outside the length criteria go to a similarity-matching group.Source: §4.4.2 · pp. 5–6, Figure 6 · p. 6
4.5 Online Parsing
Hue updates templates online as identifiers arrive, using automatic merging for regular logs and a user-guided merge-reject process for potentially ambiguous tokens.
4.5 Online Parsing
- When an identifier reaches a leaf, Hue assigns its group template to the current log and updates that template with incoming identifiers.The group template is the common sequence of identifiers in the group; Hue retains automatic updating for regular log files.Source: §4.5 · p. 6, §4.5.2 · p. 6
4.5.2 Guided Updating.
- For potentially ambiguous non-key tokens, Hue prompts users to accept or reject a proposed merge; rejection creates a new group, while acceptance updates the template.The merge-reject check triggers a query when tokens requiring updates in the candidate template include non-key tokens.Source: §4.5.2 · p. 6
- Hue reduces feedback-query frequency by skipping key updates, since cast parameter tokens are keys and ambiguous tokens are non-keys.Guided mode also avoids numerical computation to decide whether to trigger a query.Source: §4.5.2 · p. 6
5 EXPERIMENTS
Experiments compare Hue with established parsers on hybrid and single-line logs, then test its feedback mechanism. Hue achieves the strongest reported average accuracy on both log types, while feedback improves results on most evaluated datasets.
5.1.2 Datasets.
- The evaluation covers 19 datasets: 16 LogPAI single-line datasets and three manually labeled hybrid datasets, with ground-truth labels for all.Hybrid datasets are HiBench runtime logs, CTS multi-source logs, and Paas multi-terminal logs; the first two were released for replication.Source: §5.1.2 · p. 7, Table 1 · p. 7
5.1.3 Metrics.
- Hue is compared with six parsers using grouping accuracy, template F1-score, and execution time; hybrid-log comparisons flatten multi-line entries with "/n" delimiters.The hybrid comparison is not apples-to-apples because existing parsers were not designed for hybrid logs; the same regexes are used for all parsers.Source: §5.1.3 · p. 7, §5.1.4 · p. 7, §5.2 · p. 7
5.2.1 Accuracy.
- 0.845 average grouping accuracy and 0.825 average F1-score on hybrid logs exceed Drain’s 0.563 and 0.692, respectively.Hue leads on all three hybrid datasets; it is designed to preserve structure across multiple lines.Source: Table 2 · p. 7, Table 3 · p. 8, §5.2.1 · p. 8
5.2.2 Efficiency.
- Hue’s hybrid-log execution time increases linearly with log volume, while Drain times out at 4.0 million messages.The authors report Hue as the strongest efficiency result in this comparison; some flattened multi-line logs are very long.Source: Figure 9 · p. 8, §5.2.2 · p. 8
5.3.1 Accuracy.
- 0.927 grouping accuracy and 0.870 template F1-score on single-line logs are the highest averages, above the best traditional parser’s 0.901 and 0.749.Against SPINE, Hue performs better on 8 datasets and comparably on 4 at message level, and achieves template-level SOTA on 9 datasets.Source: Table 4 · p. 8, §5.3.1 · p. 8, Table 5 · p. 9
5.3.2 Efficiency.
- Hue’s single-line processing speed is comparable with the best baseline, though it is slightly slower than the fastest baseline in some cases.The comparison averages five runs on HDFS, Spark, and BGL.Source: Figure 10 · p. 9, §5.3.2 · p. 9
5.4 RQ3: How Effective is Hue’s Human Feedback Integration Mechanism?
- With fewer than 20 feedback queries, grouping accuracy exceeds 0.92 and template F1-score exceeds 0.87 on all datasets except CTS.The experiments assume all user feedback is correct; the authors suggest CTS results may reflect many templates but few corresponding logs.Source: §5.4 · p. 9
- Hue’s feedback mechanism improves accuracy with fewer queries than SPINE and reaches a higher final grouping accuracy after tens of queries.Hue queries before a potentially incorrect merge; SPINE queries while partitioning groups after merges.Source: §5.4 · p. 9
6 INDUSTRIAL DEPLOYMENT
Hue ran in Huawei Cloud for two months, where engineers used guided updating to tailor parsing to downstream tasks. The deployment processed logs with a low query rate and acceptable accuracy, and parsed hybrid logs were integrated into experimental models.
6 INDUSTRIAL DEPLOYMENT
- Hue ran smoothly in Huawei Cloud for two months, from October to December 2022, with 1,000 logs annotated per downstream task each week.Operators did not have access to the source code of the logging statements.Source: §6 · p. 9
- Nearly 70% of parsed templates were multi-line; event, text, and table logs accounted for 32.3%, 26.1%, and 41.6%, respectively.These proportions were reported for root-cause-analysis logs over the two-month period.Source: §6 · pp. 9–10, Figure 14 · p. 10
- Guided updating let engineers define templates differently for root cause analysis and user profile construction, reflecting each task’s focus on objects or actions.Engineers used different human feedback when parsing collected log samples for the two tasks.Source: §6 · p. 10
- In Huawei Cloud, fewer than 30 feedback queries per 1,000 logs coincided with grouping accuracy above 0.9, and parsed hybrid logs entered experimental models.Engineers judged performance acceptable for their respective tasks.Source: §6 · p. 11
7 THREATS TO VALIDITY
The evaluation faces threats from dataset labeling errors, assumptions about log headers and key settings, and the possibility of incorrect user feedback.
7 THREATS TO VALIDITY
- Labeling errors in the evaluated datasets can negatively impact parser performance.The authors report finding such errors across multiple log datasets.Source: §7 · p. 11
- Hue segments logs based on headers, so its performance may degrade on logs without headers; parsing such logs is future work.Source: §7 · p. 11
- The experiments use the most common key setting, which may not fit some logs; users can customize key-casting configurations to mitigate this threat.Source: §7 · p. 11
- The evaluation assumes user feedback is correct, while incorrect feedback may introduce parsing errors; the authors recommend discussion when users are unsure of a template.Source: §7 · p. 11
8 RELATED WORKS
Prior work includes unsupervised pattern-mining, clustering, and heuristic parsers, alongside less common supervised approaches. Hue is positioned as an unsupervised parser for hybrid logs that also integrates user feedback.
8 RELATED WORKS
- Unsupervised parsers are grouped into frequent-pattern-mining, clustering-based, and heuristic-based approaches.These methods respectively treat frequent patterns as templates, group similar logs before extracting templates, or encode expert knowledge as rules.Source: §8 · p. 11
- Drain is described as an online heuristic parser with state-of-the-art performance among open-source traditional parsers; POP and SPINE extend this line of work.POP adds parallel Spark implementation, while SPINE adds progressive clustering and manual correction.Source: §8 · p. 11
- Hue is unsupervised and does not rely on labeled log datasets, unlike supervised approaches such as UniParser and SemParser.Source: §8 · p. 11
- Hue is presented as the first attempt at hybrid log parsing and adds a merge-reject feedback mechanism beyond the hyperparameter tuning available in most existing parsers.Source: §8 · p. 11
9 CONCLUSION
The paper frames hybrid log parsing as a superset of single-line parsing and presents Hue as an online approach that combines log patterns with domain knowledge. Evaluations and production deployment support its reported accuracy, efficiency, and practical value.
9 CONCLUSION
- Hue uses incoming log patterns and expert domain knowledge to parse hybrid and single-line logs online.Source: §9 · p. 11
- A key-casting table and merge-reject strategy let Hue use user feedback to adapt to complex, changing log templates.Source: §9 · p. 11
- Evaluations on three hybrid and sixteen single-line datasets demonstrate Hue's superiority in accuracy and efficiency.Source: §9 · p. 11
- Hue's successful deployment in a real production environment further highlights its practical value.Source: §9 · p. 11