PaperlayerView exact revision on arXiv

Source-linked AI summary

SeriCrypt: An LLM-Driven Context-Aware Serialization Framework for Cryptographic Protocols

Maosong Chen, Xi Chen, Mengcheng Ju, Dongliang Zhao, Chunxiang Gu

arXiv:2608.24498v1cs.CR

TL;DR

Cryptographic protocol message construction requires both cross-message state consistency and cryptographic correctness, but existing automation has largely focused on plaintext protocols. SeriCrypt uses an LLM to extract protocol knowledge into CDSL/PAST and a protocol-independent engine to evaluate attributes and serialize messages. It generated accepted sequences, completed every evaluated handshake, found five security violations, and achieved higher fuzzing coverage within the same time budget.

  • Problem

    Automating cryptographic protocol messages is difficult because fields depend on prior session state and cryptographic computations, while existing automated approaches largely target plaintext protocols.

  • Method

    SeriCrypt combines LLM-based extraction into a CDSL-constrained Protocol Abstract Syntax Tree with a protocol-independent engine for field evaluation, cryptographic operations, and serialization.

  • Results

    SeriCrypt completed handshakes in all 10 test configurations, discovered five security violations, and achieved the highest code coverage across six protocols under the same time budget.

  • Takeaways & Limitations

    The framework supports interoperability, targeted security-constraint testing, and structured cryptographic-protocol fuzzing.

Abstract

from arXiv · show

Constructing syntactically correct and cryptographically valid message sequences is essential for protocol state machine learning, conformance testing, and fuzzing. Unlike plaintext protocols, cryptographic protocols involve complex cross-message state dependencies and cryptographic computation constraints. Existing automated approaches predominantly target text-based or plaintext protocols, leaving cryptographic message construction largely manual. We present SeriCrypt, an LLM-driven, context-aware serialization framework for cryptographic protocols. It employs a large language model to extract field constraints, state dependencies, and cryptographic computation rules from unstructured protocol specifications into a unified structured intermediate representation, formally characterized by a domain-specific language for cryptographic protocols (CDSL). A protocol-agnostic execution engine parses CDSL declarations, automating field value resolution, cryptographic primitive invocation, and byte-stream serialization. As case studies in protocol security testing, we use the framework to construct violation messages targeting specification-defined security constraints and to support protocol fuzzing, evaluating it on mainstream implementations of TLS 1.2/1.3, IKEv1/v2, SSH, and TLCP. SeriCrypt generated message sequences accepted by all evaluated implementations and completed handshakes in every scenario. Security constraint testing revealed five specification violations, and fuzzing reached deeper protocol states with higher code coverage than mainstream fuzzers under the same time budget, demonstrating the framework's practical value for cryptographic protocol security testing.

I. INTRODUCTION

SeriCrypt addresses the difficulty of constructing cryptographically valid protocol messages by combining LLM-based specification extraction with a structured representation and protocol-independent execution engine. It supports automated serialization and security testing across cryptographic protocols.

  • Motivation: Implementation deviations from thoroughly verified specifications can still create severe vulnerabilities, making implementation security testing essential.The cited example is state machine bypass.
  • Motivation: Cryptographic message construction must preserve byte-level encoding, cross-message state dependencies, and cryptographic computation constraints.These dependencies include prior random values, negotiated parameters, ephemeral keys, key derivation, authentication codes, and signatures.
  • Approach: SeriCrypt uses an LLM-based extraction chain to convert unstructured specifications into a Protocol Abstract Syntax Tree constrained by CDSL.The approach reduces manual organization of protocol messages and cryptographic dependency rules.
  • Approach: CDSL uniformly describes message structures, field attributes, cross-message references, and cryptographic rules, while the execution engine evaluates attributes and serializes messages.The engine invokes cryptographic primitives and separates protocol-specific descriptions from reusable execution mechanisms.
  • Design requirements: The design decomposes automation into representation and execution problems, requiring structural, dependency, cryptographic, executable, and protocol-independent attribute-aware capabilities.These requirements guide both the formal language and the general execution engine.

III. FORMAL MODEL AND UNIFIED ATTRIBUTE SYSTEM

CA-AGCP extends attribute-grammar foundations with session knowledge, runtime context, and cryptographic primitives, providing the formal basis for representing and evaluating cryptographic protocol messages.

  • A. From CFG+AG to CA-AGCP: CA-AGCP extends classical context-free and attribute grammars to model cryptographic protocols with structural and semantic dependencies.Classical grammars capture hierarchy, while attribute grammars add node evaluation through synthesized and inherited attributes.
  • B. Formal Definition of CA-AGCP: The CA-AGCP formalism is a 9-tuple divided into a syntactic skeleton, a basic attribute system, and a dynamic semantic extension.The first layer defines protocol topology, the second supports node-level evaluation, and the third captures cryptographic-protocol-specific dynamics.
  • B. Formal Definition of CA-AGCP: The knowledge space K stores specification constants, local security policies, and current test-configuration values.Examples include algorithm identifiers, version numbers, fixed labels, byte sequences, and certificates.
  • B. Formal Definition of CA-AGCP: The runtime context Γ stores session values addressed by structured paths or simple identifiers and supports reading and writing state.It locates exchanged-message fields and names intermediate key-derivation results.
  • B. Formal Definition of CA-AGCP: The cryptographic primitive space Ω contains atomic library operations and composite operations formed from ordered rule chains.Composite operations represent procedures such as key derivation and encryption or decryption encapsulation.
  • B. Formal Definition of CA-AGCP: PAST is the protocol abstract syntax tree produced from a CA-AGCP grammar for protocol message representation.Its definition is introduced as a formal object within the model.
  • B. Formal Definition of CA-AGCP: Message structure trees and session computation blocks are linked through the dynamic context Γ.Blocks contain ordered chains of primitives, auxiliary functions, and intermediate variables used to generate keys or composite cryptographic operations.

C. Unified Attribute System (UAS)

The Unified Attribute System classifies protocol-field semantics into intrinsic, relational, and transformational attributes, enabling independent values, context-based references, and cryptographic computation.

  • C. Unified Attribute System (UAS): The UAS divides attributes into three categories according to semantic function and computation method.The categories distinguish context-free data, historical field dependencies, and cryptographic transformations.
  • Intrinsic Attributes: Intrinsic attributes describe context-free atomic data whose values depend on symbol semantics or local encoding rules.Their evaluation does not depend on the runtime context Γ or cryptographic operation space Ω.
  • Relational Attributes: Relational attributes represent dependencies between fields formed through historical protocol interactions.Their values are obtained by projecting data from runtime context Γ through declared reference paths.
  • Transformational Attributes: Transformational attributes encode higher-order semantics driven by cryptographic primitives, including ciphertext generation, hash verification, and key derivation.Their operation consumes inputs extracted from Γ and the knowledge space K according to the attribute declaration.

IV. CDSL: DOMAIN-SPECIFIC LANGUAGE FOR CRYPTOGRAPHIC PROTOCOLS

CDSL instantiates CA-AGCP as an unambiguous, machine-parsable language that jointly describes protocol message structure and cryptographic attribute semantics.

  • Language design: CDSL carries both structural and semantic dimensions of CA-AGCP through three coordinated subsystems.The supplied passage introduces these subsystems as the language’s organizing design.
  • Message Structure Grammar: CDSL’s structural grammar represents CA-AGCP’s syntactic skeleton through named production rules defined in BNF.This provides the formal syntax for protocol message descriptions.
  • Message Structure Grammar: Atomic nodes represent indivisible data units with length constraints and attribute annotations.Reference sequences concatenate substructures by pointing to productions through rule names.
  • Message Structure Grammar: Alternatives model variable-length homogeneous sequences using recursive production and locally governed termination.The construction corresponds to recursive expansion ending at an empty production.

B. Unified Attribute Evaluation System

CDSL organizes protocol field semantics into intrinsic, relational, and transformational attributes, each with deterministic evaluation behavior for values, dependencies, lengths, and cryptographic computations.

  • CDSL instantiates three theoretical attribute categories into six concrete types with deterministic evaluation semantics.
  • Intrinsic Attributes: Intrinsic attributes bind fields to specified values, cryptographically secure randomness, or locally imported data such as certificates and timestamps.
  • Relational Attributes: Relational attributes reference prior fields or session state, while length attributes are backfilled from serialized subtree sizes.
  • Transformational Attributes: Transformational attributes wrap cryptographic computation and support auxiliary functions, primitive algorithms, composite computation-block functions, and traced argument references.

C. Session-Level Cryptographic Computation Grammar

CDSL expresses session-level cryptographic computations as ordered or parameterized blocks that the execution engine evaluates alongside message attributes. The TLS 1.3 Finished example shows these blocks resolving dependencies, deriving keys, computing verification data, and serializing the result.

  • Session-level derivations such as TLS 1.3 key schedules and IKEv2 prf+ require multi-step computation blocks beyond single-field expressions.
  • Flow-type computation blocks: Flow-type blocks arrange assignment rules into dependency-ordered chains evaluated sequentially, updating context so later rules can reference earlier results.
  • Parameter-type computation blocks: Parameter-type blocks declaratively specify algorithm configurations through key-value pairs, including dimensions for operations such as signature algorithms.
  • TLS 1.3 Finished example: In the TLS 1.3 Finished example, the engine triggers key-derivation and PRF blocks when verify data depends on handshake messages and the client handshake traffic secret.
  • TLS 1.3 Finished example: The engine derives the finished key with HKDF-Expand-Label, computes verify data with HMAC, then performs encryption encapsulation, length backfilling, and byte-level serialization.

V. SYSTEM DESIGN AND IMPLEMENTATION

SeriCrypt separates LLM-based PAST construction from attribute-driven execution through CDSL, enabling protocol-independent message generation and PAST-level security testing mutations.

  • SeriCrypt uses two phases: LLM-driven PAST construction and attribute-driven PAST execution, with CDSL connecting them.
  • PAST construction: PAST construction decomposes specification understanding into three stages and eight sequential steps using a chain-of-thought strategy.
  • PAST construction: The stages construct message structures, annotate fields with CDSL attributes and dependencies, and extract cryptographic computation blocks.
  • PAST execution: The execution engine instantiates the tree, evaluates independent and dependent fields with cryptographic computations, and completes encryption encapsulation and serialization.
  • Protocol independence: Protocol independence comes from encoding composite algorithms as parameterized computation blocks, allowing new protocols without modifying execution-engine code.

VI. EXPERIMENTAL EVALUATION

The evaluation examines SeriCrypt’s interoperability, expressiveness, LLM-generated PAST accuracy, direct LLM generation, protocol testing, and fuzzing across six cryptographic protocols and six mainstream implementations.

  • RQ4–RQ6: The evaluation separately investigates whether direct LLM generation can complete handshakes and how SeriCrypt performs in protocol testing and fuzzing.These questions compare direct client generation with SeriCrypt’s structured approach and assess targeted security testing and fuzzing applications.
  • Evaluation Setup: Six mainstream implementations were tested across TLS 1.2/1.3, IKEv1/v2, SSH, and TLCP in 10 protocol configurations.The configurations included IKEv1 PSK/certificate modes, IKEv2 PSK/certificate, TLS, SSH password authentication, and TLCP dual-certificate mutual authentication.
  • RQ1: Interoperability: 10 configurations completed full handshakes, with generated messages passing Wireshark field-level parsing and server verification.Accepted messages contained valid negotiation parameters, key materials, authentication fields, and encrypted payloads.
  • RQ2: Expressiveness: 52 client messages were described across 10 configurations, with every field mapped to one of CDSL’s six attribute types.The messages covered six protocols, and no field pattern fell outside CDSL’s attribute-system coverage.
  • RQ3: LLM Generation Quality: PAST generation averaged above 94% accuracy, although TLS 1.3 structure generation and SSH computation-block extraction were comparatively difficult.Reported errors involved conditional field presence, cross-message state and transcript boundaries, and protocol-specific computation details; all were discoverable and correctable through verification.

E. RQ4: Comparison with Direct LLM Generation

Directly prompting an LLM to generate complete cryptographic protocol clients failed to produce full handshakes, while SeriCrypt’s division between specification extraction and deterministic computation addresses the observed failure modes.

  • E. RQ4: Comparison with Direct LLM Generation: 30 direct-generation experiments produced no L4 result, meaning none achieved a full handshake.The baseline used Gemini-3-Pro-Preview with five repetitions for each of six target protocols.
  • E. RQ4: Comparison with Direct LLM Generation: Direct LLM failures arose from protocol constants and encoding, unstable historical negotiation-state acquisition, and broken cryptographic computation chains.Examples include incorrect byte order or length encoding, omitted server-selected values or random numbers, and confusion over HKDF labels, PRF input order, and AEAD additional authenticated data.
  • E. RQ4: Comparison with Direct LLM Generation: SeriCrypt assigns specification understanding and knowledge extraction to the LLM while the execution engine performs precise cryptographic computation and byte-level encoding.The authors report that this division matches the LLM’s stronger extraction performance and mitigates its instability on cryptographic computation.
  • F. RQ5: Protocol Testing Application: 125 extracted security constraints yielded 364 test cases and five discovered violations, including four new findings.PAST mutations were applied after establishing the normal session context, while the execution engine preserved syntactically correct encoding.
  • G. RQ6: Fuzzing Application: SeriCrypt achieved the highest code coverage on all six evaluated protocols within the same four-hour budget.PAST-based structured mutations preserved cryptographic validity, allowing test cases to reach deeper protocol states than blind byte-level mutations.

VII. DISCUSSION

SeriCrypt’s declarative CDSL and reusable execution engine provide broad protocol coverage and lower mutation effort, while current scope remains centered on client-message serialization.

  • Extensibility: CDSL’s six attribute types and computation blocks express differences across TLS, IKE, SSH, and TLCP protocols.The execution engine is reused when extending to new protocols.
  • Extensibility: Extending SeriCrypt requires configuration, an approximately 120-line response parser, and manual verification or correction of the generated PAST.The configuration specifies session parameters and includes a 5–8-field output example.
  • Future Directions: SeriCrypt currently focuses on serializing client messages, with automatic response-parser generation and broader protocol validation identified as future work.Planned targets include QUIC, WireGuard, and post-quantum hybrid handshakes.
  • Future Directions: Combining PAST mutation strategies with structure-aware fuzzing and replacing manual PAST verification with iterative agents are proposed future directions.The paper reports preliminary results supporting the fuzzing direction.
  • Comparison: SeriCrypt’s capability boundary includes automated specification extraction, cross-message dependency expression, cryptographic computation expression, and a protocol-independent execution engine.These dimensions are compared with representative approaches in Table VIII.

APPENDIX

The appendix documents TLS 1.3 message and computation templates alongside concrete specification-violation tests and their observed implementation behaviors.

  • TLS 1.3 Construction: TLS 1.3 Finished Client comprises record, encrypted-block, handshake, message-body, and verify-data fields with specified field lengths.The verify-data field depends on transcript hashing and derived cryptographic values.
  • Prompt Templates: TLS 1.3 handshake key derivation and signature-algorithm handling are represented through dedicated computation and parameter-configuration templates.Figures 7–14 cover structure construction, attribute annotation, and computation-block extraction.
  • Security Violations: Five semantic violations were summarized, including missing responses to IKEv2 INFORMATIONAL requests and TLS 1.3 certificate, extension, and signature-algorithm violations.The appendix states that each violation includes an RFC basis, test construction, and observation.
  • Security Violations: TLSE accepted an MD5-signed certificate without sending the required bad-certificate alert.SeriCrypt presented TLSE with a certificate signed using MD5.
  • Security Violations: TLSE continued handshakes after receiving ClientHello messages missing required extensions and a server-selected signature algorithm not offered by the client.The reported outcomes were absence of missing-extension or abort behavior and acceptance of the undeclared signature algorithm.
  • Security Violations: TLSE ignored an illegal ChangeCipherSpec payload value of 0x02 instead of aborting with an unexpected-message alert.The legal payload value was 0x01.
  • Mutation Tasks: The appendix includes the implementation-cost table for nine mutation tasks, reporting lines of code and modified-function counts.Table IX compares implementation cost across the evaluated tools.

Appendix D

Appendix D compares mutation implementation effort across nine tasks using lines of code and modified-function counts, showing that SeriCrypt achieves the same mutations with substantially less code.

  • Experimental Setup: The evaluation uses TLS 1.2 with mutual authentication and OpenSSL’s s_server as the server under test.The cipher suite is TLS DHE RSA WITH AES 128 GCM SHA256.
  • Task Set: Nine tasks cover structural fields, cross-message dependencies, and cryptographic computations, with identical target deviations across the three tools.The tasks are labeled S1–S3, D1–D3, and C1–C3.
  • Metrics: Mutation cost is measured by lines of code and the number of functions written or modified, excluding baseline-client construction.Each table cell reports LoC / number of functions.
  • Results: 10 lines and 0 function changes let SeriCrypt complete all mutations, versus totals 4.1 and 8.8 times larger for TLS-Attacker and Scapy, respectively.Each SeriCrypt task edits 1–2 declaration lines; dependency-affecting tasks require 3 lines versus 32 and 31.
Loading 2608.24498v1…