PaperlayerView exact revision on arXiv

Source-linked AI summary

SeriCrypt: An LLM-Driven Context-Aware Serialization Framework for Cryptographic Protocols

Maosong Chen, Xi Chen, Mengcheng Ju, Dongliang Zhao, Chunxiang Gu

arXiv:2608.24498v1cs.CR

TL;DR

Cryptographic protocol message construction requires both cross-message state consistency and cryptographic correctness, but existing automation has largely focused on plaintext protocols. SeriCrypt uses an LLM to extract protocol knowledge into CDSL/PAST and a protocol-independent engine to evaluate attributes and serialize messages. It generated accepted sequences, completed every evaluated handshake, found five security violations, and achieved higher fuzzing coverage within the same time budget.

Abstract

from arXiv · show

Constructing syntactically correct and cryptographically valid message sequences is essential for protocol state machine learning, conformance testing, and fuzzing. Unlike plaintext protocols, cryptographic protocols involve complex cross-message state dependencies and cryptographic computation constraints. Existing automated approaches predominantly target text-based or plaintext protocols, leaving cryptographic message construction largely manual. We present SeriCrypt, an LLM-driven, context-aware serialization framework for cryptographic protocols. It employs a large language model to extract field constraints, state dependencies, and cryptographic computation rules from unstructured protocol specifications into a unified structured intermediate representation, formally characterized by a domain-specific language for cryptographic protocols (CDSL). A protocol-agnostic execution engine parses CDSL declarations, automating field value resolution, cryptographic primitive invocation, and byte-stream serialization. As case studies in protocol security testing, we use the framework to construct violation messages targeting specification-defined security constraints and to support protocol fuzzing, evaluating it on mainstream implementations of TLS 1.2/1.3, IKEv1/v2, SSH, and TLCP. SeriCrypt generated message sequences accepted by all evaluated implementations and completed handshakes in every scenario. Security constraint testing revealed five specification violations, and fuzzing reached deeper protocol states with higher code coverage than mainstream fuzzers under the same time budget, demonstrating the framework's practical value for cryptographic protocol security testing.

I. INTRODUCTION

SeriCrypt addresses the difficulty of constructing cryptographically valid protocol messages by combining LLM-based specification extraction with a structured representation and protocol-independent execution engine. It supports automated serialization and security testing across cryptographic protocols.

Motivation

  • Implementation deviations from thoroughly verified specifications can still create severe vulnerabilities, making implementation security testing essential.The cited example is state machine bypass.Source: §I · p. 1
  • Cryptographic message construction must preserve byte-level encoding, cross-message state dependencies, and cryptographic computation constraints.These dependencies include prior random values, negotiated parameters, ephemeral keys, key derivation, authentication codes, and signatures.Source: §I · p. 1, §II · p. 2

Approach

  • SeriCrypt uses an LLM-based extraction chain to convert unstructured specifications into a Protocol Abstract Syntax Tree constrained by CDSL.The approach reduces manual organization of protocol messages and cryptographic dependency rules.Source: §I · pp. 1–2
  • CDSL uniformly describes message structures, field attributes, cross-message references, and cryptographic rules, while the execution engine evaluates attributes and serializes messages.The engine invokes cryptographic primitives and separates protocol-specific descriptions from reusable execution mechanisms.Source: §I · p. 2

Design requirements

  • The design decomposes automation into representation and execution problems, requiring structural, dependency, cryptographic, executable, and protocol-independent attribute-aware capabilities.These requirements guide both the formal language and the general execution engine.Source: §II · p. 2

III. FORMAL MODEL AND UNIFIED ATTRIBUTE SYSTEM

CA-AGCP extends attribute-grammar foundations with session knowledge, runtime context, and cryptographic primitives, providing the formal basis for representing and evaluating cryptographic protocol messages.

A. From CFG+AG to CA-AGCP

B. Formal Definition of CA-AGCP

  • The CA-AGCP formalism is a 9-tuple divided into a syntactic skeleton, a basic attribute system, and a dynamic semantic extension.The first layer defines protocol topology, the second supports node-level evaluation, and the third captures cryptographic-protocol-specific dynamics.Source: B. Formal Definition of CA-AGCP · p. 3
  • The knowledge space K stores specification constants, local security policies, and current test-configuration values.Examples include algorithm identifiers, version numbers, fixed labels, byte sequences, and certificates.Source: B. Formal Definition of CA-AGCP · p. 3
  • The runtime context Γ stores session values addressed by structured paths or simple identifiers and supports reading and writing state.It locates exchanged-message fields and names intermediate key-derivation results.Source: B. Formal Definition of CA-AGCP · p. 3
  • The cryptographic primitive space Ω contains atomic library operations and composite operations formed from ordered rule chains.Composite operations represent procedures such as key derivation and encryption or decryption encapsulation.Source: B. Formal Definition of CA-AGCP · p. 3
  • PAST is the protocol abstract syntax tree produced from a CA-AGCP grammar for protocol message representation.Its definition is introduced as a formal object within the model.Source: B. Formal Definition of CA-AGCP · p. 3
  • Message structure trees and session computation blocks are linked through the dynamic context Γ.Blocks contain ordered chains of primitives, auxiliary functions, and intermediate variables used to generate keys or composite cryptographic operations.Source: B. Formal Definition of CA-AGCP · p. 3

C. Unified Attribute System (UAS)

The Unified Attribute System classifies protocol-field semantics into intrinsic, relational, and transformational attributes, enabling independent values, context-based references, and cryptographic computation.

C. Unified Attribute System (UAS)

  • The UAS divides attributes into three categories according to semantic function and computation method.The categories distinguish context-free data, historical field dependencies, and cryptographic transformations.Source: C. Unified Attribute System (UAS) · p. 3

Intrinsic Attributes

  • Intrinsic attributes describe context-free atomic data whose values depend on symbol semantics or local encoding rules.Their evaluation does not depend on the runtime context Γ or cryptographic operation space Ω.Source: C. Unified Attribute System (UAS) · p. 3

Relational Attributes

  • Relational attributes represent dependencies between fields formed through historical protocol interactions.Their values are obtained by projecting data from runtime context Γ through declared reference paths.Source: C. Unified Attribute System (UAS) · p. 3

Transformational Attributes

  • Transformational attributes encode higher-order semantics driven by cryptographic primitives, including ciphertext generation, hash verification, and key derivation.Their operation consumes inputs extracted from Γ and the knowledge space K according to the attribute declaration.Source: C. Unified Attribute System (UAS) · p. 4

IV. CDSL: DOMAIN-SPECIFIC LANGUAGE FOR CRYPTOGRAPHIC PROTOCOLS

CDSL instantiates CA-AGCP as an unambiguous, machine-parsable language that jointly describes protocol message structure and cryptographic attribute semantics.

Language design

  • CDSL carries both structural and semantic dimensions of CA-AGCP through three coordinated subsystems.The supplied passage introduces these subsystems as the language’s organizing design.Source: §IV · p. 4

Message Structure Grammar

  • CDSL’s structural grammar represents CA-AGCP’s syntactic skeleton through named production rules defined in BNF.This provides the formal syntax for protocol message descriptions.Source: A. Message Structure Grammar · p. 4
  • Atomic nodes represent indivisible data units with length constraints and attribute annotations.Reference sequences concatenate substructures by pointing to productions through rule names.Source: A. Message Structure Grammar · p. 4
  • Alternatives model variable-length homogeneous sequences using recursive production and locally governed termination.The construction corresponds to recursive expansion ending at an empty production.Source: A. Message Structure Grammar · p. 4

B. Unified Attribute Evaluation System

CDSL organizes protocol field semantics into intrinsic, relational, and transformational attributes, each with deterministic evaluation behavior for values, dependencies, lengths, and cryptographic computations.

Intrinsic Attributes

Relational Attributes

Transformational Attributes

  • Transformational attributes wrap cryptographic computation and support auxiliary functions, primitive algorithms, composite computation-block functions, and traced argument references.Source: B. Unified Attribute Evaluation System · p. 4

C. Session-Level Cryptographic Computation Grammar

CDSL expresses session-level cryptographic computations as ordered or parameterized blocks that the execution engine evaluates alongside message attributes. The TLS 1.3 Finished example shows these blocks resolving dependencies, deriving keys, computing verification data, and serializing the result.

Flow-type computation blocks

  • Flow-type blocks arrange assignment rules into dependency-ordered chains evaluated sequentially, updating context so later rules can reference earlier results.Source: C. Session-Level Cryptographic… · p. 5

Parameter-type computation blocks

  • Parameter-type blocks declaratively specify algorithm configurations through key-value pairs, including dimensions for operations such as signature algorithms.Source: C. Session-Level Cryptographic… · p. 5

TLS 1.3 Finished example

  • In the TLS 1.3 Finished example, the engine triggers key-derivation and PRF blocks when verify data depends on handshake messages and the client handshake traffic secret.Source: D. Comprehensive Example: TLS 1.3… · p. 5
  • The engine derives the finished key with HKDF-Expand-Label, computes verify data with HMAC, then performs encryption encapsulation, length backfilling, and byte-level serialization.Source: D. Comprehensive Example: TLS 1.3… · p. 5

V. SYSTEM DESIGN AND IMPLEMENTATION

SeriCrypt separates LLM-based PAST construction from attribute-driven execution through CDSL, enabling protocol-independent message generation and PAST-level security testing mutations.

  • SeriCrypt uses two phases: LLM-driven PAST construction and attribute-driven PAST execution, with CDSL connecting them.Source: A. System Overview · p. 5

PAST construction

PAST execution

  • The execution engine instantiates the tree, evaluates independent and dependent fields with cryptographic computations, and completes encryption encapsulation and serialization.Source: C. Phase2: Attribute-Driven PAST… · p. 6

Protocol independence

VI. EXPERIMENTAL EVALUATION

The evaluation examines SeriCrypt’s interoperability, expressiveness, LLM-generated PAST accuracy, direct LLM generation, protocol testing, and fuzzing across six cryptographic protocols and six mainstream implementations.

RQ4–RQ6

  • The evaluation separately investigates whether direct LLM generation can complete handshakes and how SeriCrypt performs in protocol testing and fuzzing.These questions compare direct client generation with SeriCrypt’s structured approach and assess targeted security testing and fuzzing applications.Source: §VI · p. 7

Evaluation Setup

  • Six mainstream implementations were tested across TLS 1.2/1.3, IKEv1/v2, SSH, and TLCP in 10 protocol configurations.The configurations included IKEv1 PSK/certificate modes, IKEv2 PSK/certificate, TLS, SSH password authentication, and TLCP dual-certificate mutual authentication.Source: A. Experimental Setup · p. 7

RQ1: Interoperability

  • 10 configurations completed full handshakes, with generated messages passing Wireshark field-level parsing and server verification.Accepted messages contained valid negotiation parameters, key materials, authentication fields, and encrypted payloads.Source: B. RQ1: Interoperability · p. 7

RQ2: Expressiveness

  • 52 client messages were described across 10 configurations, with every field mapped to one of CDSL’s six attribute types.The messages covered six protocols, and no field pattern fell outside CDSL’s attribute-system coverage.Source: C. RQ2: Expressiveness · p. 7

RQ3: LLM Generation Quality

  • PAST generation averaged above 94% accuracy, although TLS 1.3 structure generation and SSH computation-block extraction were comparatively difficult.Reported errors involved conditional field presence, cross-message state and transcript boundaries, and protocol-specific computation details; all were discoverable and correctable through verification.Source: D. RQ3: LLM Generation Accuracy for PAST · p. 7

E. RQ4: Comparison with Direct LLM Generation

Directly prompting an LLM to generate complete cryptographic protocol clients failed to produce full handshakes, while SeriCrypt’s division between specification extraction and deterministic computation addresses the observed failure modes.

E. RQ4: Comparison with Direct LLM Generation

  • 30 direct-generation experiments produced no L4 result, meaning none achieved a full handshake.The baseline used Gemini-3-Pro-Preview with five repetitions for each of six target protocols.Source: E. RQ4: Comparison with Direct LLM… · p. 8
  • Direct LLM failures arose from protocol constants and encoding, unstable historical negotiation-state acquisition, and broken cryptographic computation chains.Examples include incorrect byte order or length encoding, omitted server-selected values or random numbers, and confusion over HKDF labels, PRF input order, and AEAD additional authenticated data.Source: E. RQ4: Comparison with Direct LLM… · p. 8
  • SeriCrypt assigns specification understanding and knowledge extraction to the LLM while the execution engine performs precise cryptographic computation and byte-level encoding.The authors report that this division matches the LLM’s stronger extraction performance and mitigates its instability on cryptographic computation.Source: E. RQ4: Comparison with Direct LLM… · p. 8

F. RQ5: Protocol Testing Application

  • 125 extracted security constraints yielded 364 test cases and five discovered violations, including four new findings.PAST mutations were applied after establishing the normal session context, while the execution engine preserved syntactically correct encoding.Source: F. RQ5: Protocol Testing Application · pp. 8–9

G. RQ6: Fuzzing Application

  • SeriCrypt achieved the highest code coverage on all six evaluated protocols within the same four-hour budget.PAST-based structured mutations preserved cryptographic validity, allowing test cases to reach deeper protocol states than blind byte-level mutations.Source: G. RQ6: Fuzzing Application · p. 9

VII. DISCUSSION

SeriCrypt’s declarative CDSL and reusable execution engine provide broad protocol coverage and lower mutation effort, while current scope remains centered on client-message serialization.

Extensibility

  • CDSL’s six attribute types and computation blocks express differences across TLS, IKE, SSH, and TLCP protocols.The execution engine is reused when extending to new protocols.Source: B. Extensibility · p. 9
  • Extending SeriCrypt requires configuration, an approximately 120-line response parser, and manual verification or correction of the generated PAST.The configuration specifies session parameters and includes a 5–8-field output example.Source: B. Extensibility · p. 9

Future Directions

  • SeriCrypt currently focuses on serializing client messages, with automatic response-parser generation and broader protocol validation identified as future work.Planned targets include QUIC, WireGuard, and post-quantum hybrid handshakes.Source: C. Future Directions · p. 9
  • Combining PAST mutation strategies with structure-aware fuzzing and replacing manual PAST verification with iterative agents are proposed future directions.The paper reports preliminary results supporting the fuzzing direction.Source: C. Future Directions · p. 10

Comparison

  • SeriCrypt’s capability boundary includes automated specification extraction, cross-message dependency expression, cryptographic computation expression, and a protocol-independent execution engine.These dimensions are compared with representative approaches in Table VIII.Source: §VIII · p. 10, Table VIII · p. 10

APPENDIX

The appendix documents TLS 1.3 message and computation templates alongside concrete specification-violation tests and their observed implementation behaviors.

TLS 1.3 Construction

Prompt Templates

  • TLS 1.3 handshake key derivation and signature-algorithm handling are represented through dedicated computation and parameter-configuration templates.Figures 7–14 cover structure construction, attribute annotation, and computation-block extraction.Source: Figure 5 · p. 12, Figure 6 · p. 12, Appendix B · p. 12

Security Violations

  • Five semantic violations were summarized, including missing responses to IKEv2 INFORMATIONAL requests and TLS 1.3 certificate, extension, and signature-algorithm violations.The appendix states that each violation includes an RFC basis, test construction, and observation.Source: Appendix C · pp. 12–13
  • TLSE accepted an MD5-signed certificate without sending the required bad-certificate alert.SeriCrypt presented TLSE with a certificate signed using MD5.Source: Appendix C · p. 13
  • TLSE continued handshakes after receiving ClientHello messages missing required extensions and a server-selected signature algorithm not offered by the client.The reported outcomes were absence of missing-extension or abort behavior and acceptance of the undeclared signature algorithm.Source: Appendix C · p. 13
  • TLSE ignored an illegal ChangeCipherSpec payload value of 0x02 instead of aborting with an unexpected-message alert.The legal payload value was 0x01.Source: Appendix C · pp. 13–14

Mutation Tasks

  • The appendix includes the implementation-cost table for nine mutation tasks, reporting lines of code and modified-function counts.Table IX compares implementation cost across the evaluated tools.Source: Table IX · p. 14

Appendix D

Appendix D compares mutation implementation effort across nine tasks using lines of code and modified-function counts, showing that SeriCrypt achieves the same mutations with substantially less code.

Experimental Setup

  • The evaluation uses TLS 1.2 with mutual authentication and OpenSSL’s s_server as the server under test.The cipher suite is TLS DHE RSA WITH AES 128 GCM SHA256.Source: Appendix D · p. 14

Task Set

  • Nine tasks cover structural fields, cross-message dependencies, and cryptographic computations, with identical target deviations across the three tools.The tasks are labeled S1–S3, D1–D3, and C1–C3.Source: Appendix D · p. 14

Metrics

  • Mutation cost is measured by lines of code and the number of functions written or modified, excluding baseline-client construction.Each table cell reports LoC / number of functions.Source: Appendix D · p. 14

Results

  • 10 lines and 0 function changes let SeriCrypt complete all mutations, versus totals 4.1 and 8.8 times larger for TLS-Attacker and Scapy, respectively.Each SeriCrypt task edits 1–2 declaration lines; dependency-affecting tasks require 3 lines versus 32 and 31.Source: Appendix D · p. 14
Loading 2608.24498v1…