PaperlayerView exact revision on arXiv

Source-linked AI summary

Lev-3B: A Data Recipe for Compact Probabilistic Log Decision Models

Leo Betthauser, Yingqi Liu, Uday Shah, Jishang Wei, Sainyam Galhotra, Liang Gou, Vedant Dharnidharka, Hao Yang

arXiv:2610.38915v1cs.LGcs.AI

TL;DR

Log decisions need to transfer across heterogeneous sources while providing probabilities, but prompted frontier models fit high-volume, sensitive triage poorly. Lev-3B uses a staged training recipe and a shared decision model for log tasks. General log data improves excluded-source transfer, while performance differs between anomaly detection and ATT&CK classification and HDFS confidence remains problematic.

  • Problem

    The open question is how to train a compact decision model that makes accurate, calibrated choices across heterogeneous logs, including sources excluded from adaptation.

    Source: §1 · pp. 1–2

  • Method

    Lev-3B combines natural-language decision training with a joint mixture of anomaly data, synthetically augmented ATT&CK data, and general log data behind one shared pointer readout.

    Source: §1 · p. 3

  • Results

    General log data offsets most interference and improves both excluded-source results; Lev-3B outperforms prompted frontier models on anomaly detection but trails them on ATT&CK, while confident HDFS errors are mostly missed anomalies.

    Source: §1 · p. 2, §7 · p. 21

  • Takeaways & Limitations

    The recipe yields one compact open checkpoint for both tasks, with stronger transfer on anomaly detection than on attack-technique classification.

    Source: §7 · p. 21

    One excluded source does not characterize all out-of-domain logs, and its exclusion from adaptation cannot establish exclusion from backbone pretraining; synthetic-data generators may also have seen public log corpora.

    Source: §6 · p. 21

Abstract

from arXiv · show

Many log-analysis tasks involve choosing among known options, such as detecting anomalous sessions or identifying attack techniques, and benefit from calibrated probabilities. We present Lev-3B, an open 3B-parameter decision model built on Granite-4.1-3B-Base with a shared pointer readout, and study its training recipe. The recipe has two key parts. First, training follows a curriculum: the model learns bounded natural-language decisions, using no log data, before it sees any logs. Second, the log stage mixes the target tasks with general log data spanning 35 auxiliary tasks unrelated to either evaluation task. We evaluate one shared checkpoint on sources seen in training and on sources excluded from training, calibration and model selection. The two-stage recipe transfers substantially better than training on log data alone. Adding general log data improves performance on excluded sources. At a fraction of their serving cost, Lev-3B outperforms prompted frontier models on anomaly detection, including on an excluded logging system, but trails them on attack-technique classification. Its probabilities are well calibrated on familiar sources, but on the excluded logging system it confidently misses anomalies. We release the checkpoint, recipe, data manifests, and prediction-level artifacts.

1 Introduction

Log decisions must transfer across heterogeneous sources while providing probabilities, yet prompted frontier models fit high-volume, sensitive triage poorly. The introduction presents Lev-3B’s data recipe and evaluates its performance, transfer, and probability reliability on represented and excluded sources.

1 Introduction

  • The recipe trains bounded natural-language decisions without logs before mixing anomaly data, augmented ATT&CK data, and general log data behind one shared pointer readout.General log data is unrelated to either evaluation task and is intended to expose the model to varied formats and support transfer.Source: §1 · pp. 2–3
  • Evaluation compares one checkpoint on represented sources and on HDFS and 18 withheld ATT&CK sourcetypes excluded from training, calibration, and model selection.The study compares decision and prompted models, examines raw and temperature-scaled probabilities and confident errors, and reports serving cost.Source: §1 · pp. 2–3
  • Lev-3B reaches 88–99 anomaly F1 on represented HPC sources and the best excluded-HDFS F1 (43.3 vs. at most 25.9), but trails prompted GPT-5.6 on ATT&CK, especially on withheld sourcetypes.On ATT&CK, it outperforms Jev, Microsoft Decision-1, and OpenAI Decisions.Source: §1 · p. 2
  • Adding general log data offsets most task interference and improves both excluded-source results: HDFS F1 rises from 25.3 to 43.3, AUPRC from 0.25 to 0.41, and withheld ATT&CK accuracy from 34.3 to 42.1.Adding ATT&CK supervision alone lowers Spirit F1 from 91.8 to 77.5; adding general log data recovers it to 88.3.Source: §1 · pp. 2–3
  • Lev-3B probabilities are close to calibrated on represented anomaly sources, but on HDFS confident errors are mostly missed anomalies (2,105 vs. 84 false alarms), which temperature scaling largely leaves unresolved.Raw Lev-3B probabilities have lower reported NLL than all three hosted decision models on both excluded sources.Source: §1 · p. 2
  • The contribution is the log-domain adaptation recipe, source-exclusion evaluation, and open release—not a new pointer architecture or temperature-fitting method.The release includes the checkpoint, adaptation recipe, data manifests, and prediction-level artifacts.Source: §1 · p. 3

2 Related Work

Prior log methods commonly target one task and return scores or generated text, while this work studies probabilistic decisions across tasks and sources. Related work situates its data recipe, typed decision interface, reliability evaluation, and bounded ATT&CK task against prior approaches.

Log anomaly detection

  • Log anomaly detectors progressed from parsing, statistical and sequence methods to transformers, language models, and prompting, typically targeting one task and returning a score or generated text.The paper instead trains one compact model for two log tasks with probability vectors over described options.Source: §2 · p. 3

Cross-source generalization

  • The evaluation excludes HDFS and withheld ATT&CK sourcetypes from every adaptation step, unlike target-side approaches such as demonstration retrieval.The paper changes the source-side training mixture to improve transfer.Source: §2 · p. 3

Domain data, data mixtures, and synthetic data

  • The recipe orders general natural-language decisions before log tasks and uses ablations to test data contributions, finding that general log data offsets most interference from security supervision and improves excluded-source transfer.Skill-It studies data order and mixtures [10], while joint training can cause negative transfer [50].Source: §2 · pp. 3–4

Typed decision models

  • Jev maps unstructured state and typed questions to probabilistic decisions, while Kev provides an open compatible implementation; the paper’s contribution is the log recipe, checkpoint, and source-exclusion reliability evaluation.A decision interface for logs is not itself claimed as new because hosted Jev has been applied to log triage.Source: §2 · p. 4

Probability reliability

  • Because calibration can fail under distribution shift [39] and confident errors may persist despite favorable aggregate metrics, the paper reports proper scores, reliability diagrams, and confident false alarms and misses separately.A single fitted temperature is treated as a diagnostic, not part of the method.Source: §2 · p. 4

Small models for security classification

  • The ATT&CK task is a closed set of 130 parent techniques from Splunk data, not benign-versus-attack detection or coverage of the full ATT&CK ontology.The full catalog has 227 techniques and 479 sub-techniques.Source: §2 · p. 4

3 Model and Adaptation

Lev-3B uses one bounded-option interface across natural-language decisions and log tasks, trained first without logs and then on a joint mixture of target and general log data. A shared probabilistic readout supports both tasks, with a single fitted temperature used as a post-hoc diagnostic.

3 Model and Adaptation

  • The released Lev-3B checkpoint follows D→A+L+T: natural-language decision training followed by joint adaptation on anomaly, general-log, and ATT&CK data.Ablated variants share the stage-one checkpoint and remove one or more stage-two sources.Source: §3 · p. 4, §3.4 · p. 7

3.1 Decision interface

  • Each example supplies a log state, question, and ordered set of described alternatives, and the model returns probabilities conditional on that question and candidate set.The alternatives represent ordinary versus anomalous behavior or ATT&CK parent techniques; changing the candidates changes the decision problem.Source: §3.1 · p. 5

3.2 Backbone and probabilistic readout

  • A shared pointer readout scores candidate representations without generating label text, using the Granite-4.1-3B-Base backbone with LoRA adapters and pointer projections trained.Questions are isolated from other questions, while candidate order is permuted during training because candidates interact causally.Source: §3.2 · p. 5

3.3 Stage one: natural-language decision adaptation

  • Stage one trains on bounded natural-language binary, categorical, and ordinal decisions with no log data, and measures rather than assumes probability calibration.Training permutes options and can augment choices with “none of the above” or a distractor; cross-entropy does not guarantee calibration under finite samples or source shift.Source: §3.3 · pp. 5–6

3.4 Stage two: joint log adaptation

  • Stage two jointly trains on anomaly data, ATT&CK data, and general log data; the latter adds 28,740 requests spanning 35 task definitions without using withheld-source records as seeds or templates.The ATT&CK training pool has 544 unique real records, with synthetic examples generated and audited for underrepresented techniques using training records as templates.Source: §3.4 · p. 6

3.5 Post-hoc temperature

  • A single scalar temperature is fitted by minimizing calibration-pool NLL and applied unchanged across both tasks and every source.A positive temperature preserves the predicted option and, for binary tasks, ranking and the decision at 0.5; it changes probabilities and decisions at other thresholds.Source: §3.5 · p. 7

4 Data and Evaluation Setup

The evaluation covers binary anomaly detection and closed-set classification over 130 ATT&CK parent techniques, using represented and excluded sources. The setup defines source partitions and task-specific metrics, while withholding excluded sources from fitting and model selection.

4 Data and Evaluation Setup

  • Evaluation covers anomaly detection over log windows and sessions and classification over 130 ATT&CK parent techniques; excluded sources enter no project-controlled adaptation, calibration, or model-selection step.The excluded-source results test transfer to unseen logging systems, not general out-of-distribution performance.Source: §4 · p. 7

4.1 Task 1: Anomaly detection

  • Anomaly sessions in BGL, Spirit, and Thunderbird group up to 100 timestamp-ordered messages by node, while HDFS is evaluated at block-session level.Sessions are anomalous when at least one constituent message is labeled anomalous; HDFS is excluded from supervised tasks, auxiliary corpora, replay streams, and derived views.Source: §4.1 · pp. 7–8

4.2 Task 2: ATT&CK technique classification

  • ATT&CK questions choose among 130 parent techniques under a pinned ontology, and the withheld split contains 178 records from 18 excluded sourcetypes.Model-visible state omits repository paths and annotation headers that reveal answers; records with multiple annotations follow a single-label selection rule.Source: §4.2 · p. 8

4.3 Source roles

  • Training, development, calibration, and final evaluation use separate source roles, with HDFS and withheld sourcetypes reserved for final evaluation only.The source-role separation is by parent group.Source: §4.3 · p. 8

4.4 Evaluation Metrics

  • Anomaly detection reports AUROC, AUPRC, and F1 at a fixed 0.5 threshold; ATT&CK reports accuracy and macro-F1, while probability quality uses NLL, ECE, and confident-error coverage.Anomaly thresholds are not tuned on HDFS, and AUPRC and F1 comparisons are limited to matching populations because they depend on prevalence.Source: §4.4 · pp. 8–9

5 Experiments

The experiments assess Lev-3B’s performance, transfer, and probability reliability across familiar and unseen log sources, alongside serving cost and latency. They examine the effects of its staged recipe and general log data while documenting task-specific limitations, including missed anomalies under high confidence.

5.1 Systems and evaluation setup

  • The evaluation compares systems on identical inputs and separates represented sources from excluded HDFS and withheld ATT&CK sourcetypes.For Lev-3B, probability quality is measured both raw and after a single fitted temperature; hosted decision models use their native probabilities.Source: §5.1 · pp. 9–10, §5.6 · p. 16

5.2 Comparison with hosted decision models and frontier models

  • Lev-3B far outperforms prompted models on adapted log sources and remains ahead on unseen HDFS, but trails frontier models on ATT&CK.On Spirit and Thunderbird, prompted models reach at most about 28% and 10% F1, while Lev-3B exceeds 85% on both; on HDFS its absolute performance drops sharply. On ATT&CK, it beats the hosted decision models but not GPT-5.6.Source: §5.2 · pp. 11–12, Table 2 · p. 11, Table 3 · p. 12

5.3 Comparison with prior log anomaly detectors

  • Lev variants achieve the best F1 on all four HPC targets, 9–24 points above the strongest baseline on each.On HDFS, Lev is close to LogICL (33.14 vs. 34.67 F1), with perfect precision but low recall.Source: §5.3 · p. 13, Table 4 · p. 13

5.4 Effect of training recipe

  • Adding ATT&CK data raises represented ATT&CK accuracy from 4.48 to 35.8, but reduces Spirit F1 from 91.82 to 77.47.The full recipe’s general log data partly recovers anomaly performance and improves excluded-source metrics; starting from the decision checkpoint also helps transfer versus log-data-only training. On HDFS, Lev-3B exceeds the log-data-only variant in AUPRC (0.10 vs. 0.41) and F1 (11.8 vs. 43.3).Source: §5.4 · pp. 13–14, Table 5 · p. 14

5.5 RQ2: General data and transfer to excluded sources

  • General log data improves withheld ATT&CK accuracy from 34.3 to 42.1 and macro-F1 from 19.8 to 29.5, with only about a point or less lost on represented sourcetypes.On HDFS, it raises AUPRC from 0.2485 to 0.4116 and F1 from 25.31 to 43.32, with nearly unchanged AUROC (0.7509 to 0.7519); the withheld ATT&CK gain reflects about 14 of 178 records from a single run.Source: Table 5 · p. 14, §5.5 · p. 15, Table 6 · p. 15

5.6 Raw and calibrated probability quality

  • Raw Lev-3B anomaly probabilities are close to calibrated on adapted sources but overstate anomaly likelihood in HDFS’s upper probability bins.Temperature fitting moves HDFS upper-bin points closer to the calibration diagonal but does not remove the gap; ATT&CK probabilities carry some signal but remain unreliable.Source: Table 7 · p. 16, §5.6 · p. 17

5.7 Confidence on errors

  • The confidence-on-errors analysis accepts predictions with top-label confidence of at least τ = 0.9 and measures coverage and accepted-prediction error rate.The supplied passage introduces this evaluation but does not provide its findings.Source: §5.7 · p. 17
  • On HDFS, Lev-3B is confident on about 98% of sessions, with 2,105 confident misses versus 84 confident false alarms; unequal coverage makes error rates incomparable.Its confidence keeps the alert stream clean but hides missed anomalies.Source: §5.7 · pp. 17–18, Table 8 · p. 19
  • On HDFS, temperature fitting reduces confident false alarms from 84 to 1 and confident misses from 2,105 to 1,912, while coverage falls from 98.3% to 96.3%.About 1,900 sessions remain confidently missed, so calibration alone does not address this failure mode.Source: §5.7 · p. 18
  • On unseen ATT&CK records, 37.1% of Lev-3B’s 70 predictions at confidence ≥0.9 are wrong; after fitting, 33.3% of 18 confident predictions are wrong.The fitted confident set has 10.1% coverage; native Jev has 45.5% wrong zero-shot and 34.6% wrong five-shot, at 12.4% and 14.6% coverage.Source: §5.7 · p. 18

5.8 Inference latency and cost

  • The latency and cost comparison favors hosted infrastructure: hosted systems use production stacks, while Lev-3B is served one request at a time in standard precision without a serving framework.The reported Lev-3B figures are an upper bound for its cost and latency, while hosted-system figures are a best case.Source: §5.8 · pp. 19–20
  • On reserved instances Lev-3B is the cheapest option; on demand only Jev and Microsoft Decision-1 cost less, while GPT-5.6 Luna costs 3–8× and Sol 60–180× as much.Decision APIs are faster than Lev-3B on an A10G; OpenAI Decisions costs more on both tasks even on demand.Source: §5.8 · p. 20, Table 9 · p. 20
  • At batch size 1, moving Lev-3B from an A10G to an H100 cuts anomaly-detection latency from 924 ms to about 172 ms, roughly 5× faster and within about 30 ms of Jev.The model and serving code are unchanged; H100 costs about $0.33 per 1,000 requests on demand and $0.14 with three-year reserved pricing.Source: §5.8 · p. 20

6 Limitations and Release Scope

The evaluation has narrow transfer scope: public anomaly corpora reflect specific systems, and the study tests only one excluded logging source. The security task and experimental comparisons also have data and design constraints.

6 Limitations and Release Scope

  • One excluded source cannot characterize all out-of-domain logs, and its exclusion from project training does not establish exclusion from backbone pretraining.Public anomaly corpora represent specific systems and historical fault distributions, where familiar templates and correlated incidents may support strong in-domain performance without establishing generalization to unseen failures.Source: §6 · pp. 20–21
  • Generated log data may reproduce excluded-source templates, and same-family generators may transfer prior knowledge to Lev-3B.No HDFS or withheld-sourcetype records were used as seeds or templates, but the generators may have encountered public log corpora during pretraining.Source: §6 · p. 21
  • The 130-way security task uses potentially incomplete or ambiguous labels; sparse classes limit per-class analysis, and the withheld split is easier for most systems.Its single-label score does not establish complete attack understanding; withheld-split scores measure transfer to new sourcetypes, not harder examples.Source: §6 · p. 21
  • Each variant is a single run, leaving no variance estimate, and the study tests only one 3B backbone.Small differences on the 178-record withheld split correspond to only a few records.Source: §6 · p. 21
  • The release supplies manifests and prediction-level outputs, while raw-data redistribution follows each source’s terms.When raw log text cannot be redistributed, reproducible builders and hashes identify the inputs.Source: §6 · p. 21

7 Conclusion

General log data unrelated to either evaluation task offsets interference from security supervision and improves transfer to excluded sources. The release includes prediction-level artifacts and a ledger documenting data and split provenance.

7 Conclusion

  • General log data offsets most interference from security supervision and improves results on both excluded sources.With general log data, Lev-3B outperforms prompted frontier models and hosted decision models on anomaly detection, including an unseen logging system, at a fraction of their serving cost; it still confidently misses HDFS anomalies and trails prompted frontier models on security classification.Source: §7 · p. 21

A Prediction Artifacts and Access Ledger

  • Prediction files record labels, option sets, probability vectors, model revisions, rendering hashes, and applied temperatures; the release ledger tracks source checksums, split builders, and record provenance.The paper’s tables are generated from these prediction files and manifests.Source: A Prediction Artifacts and Access Ledger · p. 26

B Threshold Transfer under Domain Shift

On HDFS, threshold transfer does not improve Lev-3B’s fixed-0.5 F1, while threshold adaptation roughly doubles Jev’s F1 but remains limited. The withheld ATT&CK split excludes 18 sourcetypes from training.

B Threshold Transfer under Domain Shift

  • Threshold rules include a source-derived threshold and alternatives adapted using only the unlabeled HDFS score distribution; all were frozen before HDFS labels were accessed.The source threshold maximizes equal-source mean anomaly F1 across BGL, Spirit, and Thunderbird development sets.Source: B Threshold Transfer under Domain Shift · p. 26
  • For Lev-3B, fixed 0.5 beats all transfer rules: the unchanged source threshold drops F1 from 43.32 to 28.37, and mean-percentile transport reaches 18.20.The mildest interpolation and robust rule come within 0.3 points of fixed 0.5; for Jev, adaptation roughly doubles F1, to at most 11.87.Source: Table 10 · p. 26, B Threshold Transfer under Domain Shift · p. 26

C.1 Split construction

  • The withheld ATT&CK split holds out all records from 18 sourcetypes, and 30% of techniques have no real training records.Some withheld-split techniques therefore receive synthetic-only training examples or none.Source: §C.1 · pp. 26–27

C.2 Effect of synthetic augmentation

Synthetic ATT&CK augmentation improves represented-sourcetype metrics but leaves withheld-split performance essentially unchanged. The comparison is limited because the rows are single runs with different total training exposure.

C.2 Effect of synthetic augmentation

  • Represented-sourcetype accuracy rises 4.5 points and macro-F1 7.0, while withheld accuracy changes −0.5 and macro-F1 +0.1.Both checkpoints are D→A+T and omit general log data.Source: Table 11 · p. 27, §C.2 · p. 27
  • The represented-split gain cannot be attributed to synthetic content alone because the single-run rows also differ in total training exposure.Synthetic augmentation does not by itself improve withheld-split transfer; the passage attributes the full recipe’s withheld-split gain to general log data.Source: §C.2 · p. 27

C.3 Performance by real training support

Performance on withheld techniques varies with real training support: Lev-3B’s advantage over Jev is concentrated among high-support techniques, while the support-floor comparison with frontier models has important limits.

C.3 Performance by real training support

  • On the withheld split, Lev’s advantage over Jev comes entirely from high-support techniques; Jev is more accurate in every lower bucket, especially for techniques with zero or one real record.Synthetic examples may exist for zero-real-support techniques, but the passage says they do not substitute for real examples.Source: Table 12 · p. 27, §C.3 · p. 27
  • At ≥5 and ≥20 real records, Lev-3B’s macro-F1 exceeds Jev’s, but at ≥10 it trails; its accuracy is 19.7–24.0 percentage points higher than Jev’s at all three nonzero support floors.Lev-3B’s accuracy rises from 41.9% at ≥5 to 52.0% at ≥20.Source: Table 13 · p. 28, §C.3 · p. 28
  • As the support floor rises, Luna’s accuracy also increases from 45.3% to 60.0%, while Lev-3B’s accuracy gap to Luna does not shrink.The ≥20 row covers four techniques and 25 records, so the authors describe it as a directional signal; the comparison does not show that limited training exposure causes the frontier gap.Source: Table 13 · p. 28, §C.3 · p. 28

D Decision and General Log Data Details

The section reports serialized-data counts before online option augmentation, epoch repetition, or replay, and cautions that requests need not represent distinct underlying text or log states.

D Decision and General Log Data Details

  • Counts refer to serialized data before online option augmentation, epoch repetition, or replay, and do not mean each request has a distinct underlying text or log state.The section distinguishes a request, which contains a shared input state and one or more questions, from a labeled question.Source: D Decision and General Log Data Details · p. 28

D.1 Decision data (D)

The decision data combines a frozen natural-language decision suite with generated supplementary examples, spanning varied source datasets and question formats without using system logs.

D.1 Decision data (D)

  • The Decision-v7 training partition is combined with 1,425 programmatically generated supplementary examples to form decision data D.Decision-v7 is a frozen natural-language decision suite distributed with Kev.Source: §D.1 · pp. 28–29
  • Decision-v7 converts ten public NLP datasets into binary, categorical, or ordinal decisions and also includes generated policy decisions and compositional rule-following examples; none contains system logs.The public datasets include AG News, Amazon reviews, Banking77, BoolQ, DBpedia14, IMDB, MNLI, SST5, TREC, and Yelp.Source: §D.1 · p. 28
  • The 12,576-request training partition includes 1,000 requests from each public dataset, accounting together for 79.5% of requests, plus legacy-policy and compositional examples.Legacy-policy examples contribute 896 requests (7.1%), and compositional examples contribute 1,680 (13.4%).Source: §D.1 · pp. 28–29, Table 14 · p. 29
  • The 15,576 labeled questions comprise 5,224 binary (33.5%), 6,904 categorical (44.3%), and 3,448 ordinal (22.1%) questions, with class balance varying by source.The counts are from the frozen training sample, not the full upstream datasets.Source: §D.1 · p. 29
  • The 1,425 supplementary examples include 900 date-based policy decisions and 525 missing-evidence examples, followed by training mixed with 2,000 replayed Decision-v7 examples.In 256 missing-evidence examples the deciding evidence is removed and the target is uniform over offered alternatives; the remaining 269 are intact controls.Source: §D.1 · p. 29

D.2 General log data: provenance and distribution

The general log slice combines heterogeneous task data from multiple provenance routes, with an uneven task mix that excludes HDFS and Hadoop logs.

D.2 General log data: provenance and distribution

  • The slice combines converted public benchmark records, generated examples or questions, and mechanically derived labels or contrasts; its task families reuse some underlying logging systems.Its sources include LogGLUE-, Loghub-, LogQA-style, software-build, microservice, insider-threat, root-cause, and trace or incident-derived tasks.Source: §D.2 · p. 30
  • The general log slice contains 28,740 requests across 35 task definitions grouped into 14 task families.Each request has one labeled question and a task-dependent state, such as a log line, session, build log, incident, or trace-derived context.Source: §D.2 · pp. 29–30, Table 15 · p. 30
  • LogGLUE-derived, LogQA-style, and Trace LogQA requests account for 56.8% of the slice, which contains 24,760 categorical (86.2%) and 3,980 binary (13.8%) questions.The unequal request shares should not be interpreted as natural prevalence of operational events; class balance varies by task.Source: §D.2 · p. 30
  • The 28,740-request general log slice excludes HDFS logs and Hadoop logs in general.Source: §D.2 · p. 30
Loading 2610.38915v1…