Source-linked AI summary
Ranked by the Matcher: A Reproducibility Audit of Knowledge Graph Extraction from Threat Reports
Safayat Bin Hakim, Houbing Herbert Song
TL;DR
CTI knowledge-graph rankings depend strongly on matching protocols that are often unspecified, limiting the comparability of published triple-F1 scores. The paper audits matcher sensitivity and uses CTIForge’s fixed-extraction ablations to examine deterministic validation. It finds substantial ranking instability and a backbone-dependent precision split, while documenting scope limits and releasing auditable artifacts.
Problem
Published CTI extraction scores may be incomparable because papers often fail to specify how predicted triples are matched to gold annotations.
Method
The paper re-scores published outputs under multiple matching protocols and uses CTIForge to toggle deterministic validation while extraction remains byte-identical.
Results
Eleven of forty-five pairwise orderings reversed across eight protocols, while validation raised precision for all four hosted backbones and lowered it for all three offline backbones.
Takeaways & Limitations
Matching rules should be explicit, document-scoped, and reported as a spread rather than a single unexplained F1.
Takeaways & Limitations
The comparison is descriptive because backbone, decoding, and backend-specific prompting covary, and CTIForge trails CTI-Nexus in recall by 0.057 triplet F1 at matched backbone.
Abstract
from arXiv · showhide
Security teams and researchers choose knowledge-graph extraction tooling for threat reports on the strength of published triple-F1 scores, yet those scores depend on how predicted triples are matched to gold annotations. We could reimplement the stated matching rule for only five of twelve inspected systems. Re-scoring ten system outputs on shared documents under eight protocols reverses eleven of forty-five pairwise orderings; one fixed prediction set spans 0.16-0.70 F1. On GRID's external 378-item calibration set, no mechanical matcher (lexical, embedding, or entailment) agrees with multi-reviewer adjudication above 71%, whereas an LLM judge reaches 86%. To separate component effects from matcher rewards, we build CTIForge, whose deterministic validation layer can vary while extraction is held byte-identical. Across seven tested deployment configurations, validation raises precision for all four hosted backbones and lowers it for all three offline backbones. Because backbone, decoding, and backend-specific prompting covary, this is a descriptive split rather than an isolated serving effect. It coincides with a roughly 2.8-fold increase in actions explicitly disputing entity type, consistent with hand-written rules encoding the conventions of the extractor against which they were developed. We release the pipeline, protocol suite, and per-triple audit records.
1. Introduction
Published CTI knowledge-graph scores are not directly comparable because triple-F1 depends on under-specified matching rules. The paper audits this problem and uses CTIForge to separate downstream validation effects from fixed extraction outputs.
- Triple-F1 comparisons depend on how predicted triples match gold annotations, including name, relation, argument-order, and corpus-scope rules.
- Five of twelve inspected systems had reimplementable matching rules, while three left criteria partly unspecified and four stated none.
- Eleven of forty-five pairwise orderings reversed across eight protocols, and one fixed prediction set ranged from 0.16 to 0.70 triplet F1.
- No mechanical protocol exceeded 71% agreement with multi-reviewer judgment on GRID’s external 378-item calibration set.
- CTIForge holds extraction byte-identical while toggling deterministic validation, revealing precision increases for hosted backbones and decreases for offline backbones.
- The paper releases the pipeline, evaluation protocols, configurations, logs, and per-triple audit records.
2. Related Work
Prior CTI extraction work largely ranks systems with triple-F1, although matching criteria are often unspecified and therefore make reported results difficult to compare. CTIForge positions itself as an audit apparatus focused on whether those comparisons are supported.
- CTIBench and SEvenLLM broaden CTI evaluation, but graph-construction systems still rely mainly on triple-F1 against gold annotations.
- Among twelve inspected systems and benchmarks, five matching rules were reimplementable, three left choices unresolved, and four stated no criterion.
- The paper contributes the CTI knowledge-graph case by treating matching-rule bias as a correctable reporting artifact rather than a property of the data.
- Related work adds interpretability, ontology conformance, verifier agents, reasoning traces, and canonicalization around neural extraction.
- GRID calibrates an LLM judge to 86.0% agreement with three reviewers, while this paper evaluates ten mechanical protocols against that calibration set.
- CTIForge combines deterministic, categorized, per-triple logging with a standard LLM extraction call, a combination not found together among inspected artifacts.
3. Problem Formulation: Scoring as a Free Parameter
The paper formalizes scoring as a free parameter: a matching protocol specifies both when a prediction matches a gold triple and where eligible gold candidates are drawn from. This makes ranking sensitivity measurable rather than implicit.
- A matching protocol M = (µ, σ) combines a binary match predicate µ with a scoping rule σ over candidate gold triples.
- True positives are computed through greedy one-to-one assignment, so each gold triple can be consumed at most once.
- A rank reversal occurs when two systems change ordering between protocols M1 and M2, even with unchanged predictions.
- The match predicate varies by name equality, relation equality, and argument-order tolerance, while scope is per-document or corpus-pooled.
- Comparative metrics must be independent of a participant’s ontology, whereas metrics enforced by that participant’s own type-pair table are definitional.
4. Apparatus: The CTIForge Pipeline
CTIForge is a modular apparatus that separates probabilistic extraction from deterministic downstream processing. This shared-extraction design enables controlled validation and canonicalization ablations with provenance retained for every graph edge.
- CTIForge makes each pipeline stage independently toggleable while keeping extraction separable from downstream processing.
- Every stage after extraction is deterministic, so ablation arms can be derived from one extraction pass and differ only by the tested module.
- Extraction uses one schema-constrained LLM call per paragraph chunk, with an optional second call only for disconnected-graph link prediction.
- Validation checks fields, indicators, types, relations, self-loops, evidence alignment, placeholders, and ATT&CK identifiers, logging rejected, repaired, or flagged actions per triple.
- Canonicalization uses aliases, type-gated embedding alignment at τ = 0.85, and ATT&CK grounding, while fusion preserves per-triple origin and confidence.
- Canonicalization may improve graph usability while reducing gold-matched F1 because gold annotations preserve rewritten surface forms.
5. Experimental Design
The design separates matcher sensitivity from downstream module effects by standardizing evaluation and holding extraction fixed during ablations. It also defines distinct metric families and tracks deployment-specific implementation choices.
- Experimental design: The study re-scores systems under varied matching protocols, then holds extraction fixed to isolate validation and canonicalization effects.All downstream stages are deterministic in the module-level ablation, so arms differ only by the module under test.
- Datasets and separation: The evaluation uses CTI-Nexus reports and CTIKG sentences, with CTI-Nexus few-shot retrieval restricted to training data.The primary corpus contains 149 reports, while the secondary benchmark contains 255 sentences and 693 gold triples.
- Evaluation protocol: Matching is scoped per document, preventing cross-report credits that otherwise inflate true positives by 6.4%.All reported figures use per-document scoping.
- Evaluation metrics: The paper separates gold-matched, schema-independent, and schema-dependent metrics because they support different claims.Schema-dependent scores are reported only as evidence that CTIForge enforces its own type-pair contract, not as evidence of superiority.
- Deployment configurations: The study compares GPT-4o defaults with hosted and offline backbone substitutions, while temperature, serving stack, decoding, and wrappers covary.Results therefore measure transfer across deployment configurations rather than an isolated serving effect.
- Matcher sensitivity: Figure 2 counts, for each system pair, how many of eight protocols rank the row system above the column system.Shading encodes the count, and outlined cells identify eleven unstable pairwise orderings.
6. Results and Analysis
The results show that both evaluation matching and deterministic validation materially shape reported extraction performance. Matcher choice can reorder systems and alter F1 substantially, while validation effects differ across tested hosted and offline configurations and canonicalization is invisible to the metric.
- Human calibration: 0.598–0.706 agreement is achieved by ten mechanical protocols on GRID’s 378 adjudicated items, versus 0.860 for an LLM judge.No mechanical protocol exceeds 0.71, and the strict protocol misses 40.2% of matches credited by humans.
- Matcher sensitivity: 11 of 45 pairwise orderings reverse when ten published systems are rescored under eight matching protocols.The comparison uses shared documents and a common extraction backbone, so the measured variation concerns matcher sensitivity rather than backbone differences.
- Matcher sensitivity: 0.1577–0.6994 triplet F1 is reported for one fixed CTIForge prediction set when only the matcher changes.The spread is 0.5417 on identical predictions; the reported protocol yields 0.4877.
- Validation effects: Validation raises precision for all four tested hosted backbones and lowers it for all three tested offline backbones.The split is descriptive because backbone, decoding, and backend-specific prompting covary; seven configurations are an observation, not a law.
- Validation effects: Roughly one in eight hosted validation actions and one in three offline actions dispute entity typing, consistent with schema-conformance mismatch.The rules were developed against frontier-model output, so deterministic constraints can reject structurally noisy output together with correct signal.
- Metric visibility: 54 of 1,571 MiniMax-M3 triples differ after canonicalization even though B+C+D and B+C receive identical scores to four decimals.The matcher’s soft name comparison absorbs the surface-form rewrites, so the metric does not register canonicalization.
7. Discussion
The discussion shows that deterministic validation is useful but configuration-dependent: it behaves differently across hosted and offline backbones, while primarily grading rather than rejecting triples. Its effects support provenance-aware analyst workflows, but the study’s comparisons remain conditional and bounded by several untested assumptions.
- Roughly four in five validator actions are soft flags, so the layer predominantly annotates doubt rather than discarding triples.Confidence-lowering preserves partially supported triples that binary gating would remove.
- The shared-extraction ablation is conditional on fixed prediction sets and does not estimate between-generation variance or calibrate soft flags as correctness probabilities.The external calibration set bounds matcher families rather than measuring in-domain CTI triple equivalence.
- Per-edge provenance and validation status support triage between evidence-aligned edges when confidence matters and flagged edges when coverage matters.The graph can also be exported to STIX without removing the edge’s source information.
8. Recommendations
The recommendations target reproducibility defects in matching and reporting: evaluate within documents, expose protocol spread, separate schema-dependent metrics, specify matchers precisely, and retain per-edge provenance.
- R1. Scope matching within a document: Pooling predictions and gold annotations across documents inflates true positives by 6.4% on the benchmark.Matching within each document prevents crediting a prediction from one report against another report’s annotations.
- R2. Report a spread, not a point: Eleven of forty-five system orderings flip under a different protocol, while one fixed prediction set spans 0.54 F1 across protocols.The recommendation is to report a spread rather than a single score.
- R3. Separate metrics by what they license: Schema-dependent metrics cannot rank systems lacking the same enforced schema, so they should remain separate from schema-independent measures such as STIX 2.1 compliance.This keeps conformance results from being read as competitive comparisons.
- R4. State the matching rule precisely enough to reimplement: Only five of twelve inspected matching rules were reimplementable, making “semantic matching” without an encoder and threshold an incomplete specification.A multidimensional protocol is preferable when exact matching is unsuitable for generative output.
- R5. Treat a single published F1 as a lower bound on uncertainty: A single published F1 should be treated as a lower bound on uncertainty because within-system protocol gaps can exceed between-system differences.For procurement comparisons, the paper recommends rescoring candidates on local documents under at least two protocols.
- R6. Prefer artifacts that carry per-edge provenance: Per-edge provenance lets analysts check the source sentence behind an edge, a property that remains independent of the matcher.Artifacts carrying provenance are therefore more inspectable for downstream action.
9. Conclusion and Future Work
The conclusion is that CTI knowledge-graph results depend on evaluation protocols, while deterministic validation effects vary across deployment configurations. The paper releases artifacts for inspection and rescoring, but its calibration and ablation designs leave important scope boundaries.
- Matching protocols can change both reported score magnitude and system ordering, so matching rules should be explicit, document-scoped, and reported as spreads.The recommendation follows from observed ordering reversals and protocol-dependent scores.
- Deterministic validation does not transfer uniformly across seven configurations, and the observed sign split does not isolate serving as its cause.Differing entity-type action shares are consistent with schema-convention mismatch.
- The external calibration set bounds matcher families rather than measuring in-domain CTI triple equivalence.The authors identify a multiply annotated CTI set as still necessary.
- The released pipeline, configurations, scoring suite, ablation harness, and per-triple records enable inspection and rescoring of the reported measurements.
10. Ethical Considerations
The paper describes CTIForge’s data practices, compact symbolic schema, extraction prompt, and deterministic validation pipeline. Its design favors tractable constraints, selective extraction, provenance, normalization, and logged repairs or rejections.
- The corpora are public vendor reports and published annotations, and the produced graphs contain nothing beyond the source reports.
- CTIForge uses 14 entity types and 12 relation types to keep the symbolic schema expressive enough for CTI while remaining tractable for validation.IOC subtypes and overlapping relations are intentionally collapsed into smaller inventories.
- The validator processes raw triples through repair, type-pair checks, self-loop rejection, evidence alignment, ATT&CK validation, status assignment, and action logging.Identifier and type repair precede validation so recoverable defects can be corrected before rejection.
- Normalization covers CVEs, hashes, domains, and ATT&CK IDs, while evidence checks, placeholder handling, and auto-swap repair address additional triple defects.
- Several auxiliary modules were disabled because they over-generated triples, rejected valid additions, or failed to align confidence ranking with gold-match status.These modules include supplement extraction, type-guided re-extraction, and multi-signal confidence rescoring.
- The extraction prompt combines entity and relation inventories, a JSON output schema, and fifteen extraction rules, with optional few-shot examples.Relation definitions emphasize commonly confused labels, and Rule 1 limits each chunk to 8–15 triples.
Appendix E. Evaluation Protocol
The evaluation protocol defines document-scoped matching across entity names, types, relations, and argument order, with deterministic implementation and configuration-controlled validation. The appendix also records validator actions and distinguishes reached categories from unreached specifications.
- Matching scope: 6.4% inflation for CTIForge and 5.9% for CTI-Nexus results when corpus pooling credits predictions against annotations from other reports.The corresponding increase is roughly +0.03 F1 for both systems.
- Matching rules: Entity mentions match through normalized equality, containment, or at least 50% stemmed-token overlap, with agent-noun equivalences canonicalized.Leading determiners are stripped before comparison.
- Matching rules: Three type groups are treated as equivalent during type-aware matching: Tool–Software–Malware, Technique–Tactic, and IOC–File–Infrastructure.The equivalences avoid penalizing distinctions that gold annotations apply inconsistently.
- Matching rules: The strict matcher requires normalized identical relations, while the relaxed matcher adds fixed compatibility pairs, inverse perspectives, and swap-aware matching passes.Matching proceeds from exact matches to name-soft and then compatible-relation comparisons.
- Reproducibility: ±0.0012 F1 variation arose from hash-dependent greedy match order before sorted iteration made repeated executions deterministic.The released configurations also fix software versions, model identifiers, and decoding environments.
- Validator audit: Seven of fourteen declared validator categories fire on either benchmark, while others are parser-resolved, alias-resolved, or retained for future work.Validator actions count rule firings rather than triples, so one triple may trigger multiple actions.
I.1. Schema-Dependent Metrics
Schema-dependent metrics require careful separation from uniform and schema-independent comparisons. The appendix reports internal contract-enforcement metrics, matched-backbone comparisons, runtime differences, and the methodological basis for interpreting these results.
- Internal schema metrics: 27.7 points is the semantic-validity margin under CTIForge’s internal constraint schema, versus 13.3 points under the external STIX 2.1 vocabulary.These definitional metrics demonstrate contract enforcement rather than superiority over a system using a different schema.
- Matched-backbone comparison: GPT-4o comparisons on 149 documents use CTI-Nexus’s public matcher, so their F1 values are controlled relative-comparison results rather than uniform-matcher counterparts.Gold-matched metrics depend on the stated matcher, whereas schema-independent metrics do not use either system’s ontology.
- Interpretation: The gold-matched rows describe a precision–recall trade-off under one protocol, while STIX 2.1 and evidence-presence rows represent schema-independent contract differences.The two kinds of rows should not be interpreted as the same type of comparison.
- Runtime: 26.4 min versus 38.7 min is CTIForge’s runtime compared with CTI-Nexus on the identical 149-document benchmark and inference endpoint.CTIForge uses one extraction call per chunk, with optional link prediction, while its symbolic stages run deterministically and locally.
- Uniform evaluation: A single matcher evaluates both CTIForge and CTIKG outputs on the 255-sentence benchmark so reported differences reflect extraction behavior rather than evaluator choice.The uniform matcher uses substring containment and token-overlap ≥0.5 after normalization.
- Interpretation: The appendix retains underlying values while interpreting the precision–recall trade-off and gap between uniform and originally reported scores in Section 6.4.These values support comparison across evaluation protocols without treating them as interchangeable metrics.