PaperlayerView exact revision on arXiv

Source-linked AI summary

Using Hyper-V Sockets for Real-time Data Extraction from a Malware Analysis Sandbox

István-Attila Császár, Radu-Marian Portase, Adrian Coleşa, Adrian Groza

arXiv:2608.30383v1cs.CR

TL;DR

Ransomware can compromise sandbox trace extraction by encrypting files or disrupting transfer mechanisms, motivating a non-network, real-time channel. The paper evaluates Hyper-V sockets as that channel, comparing them with WinSock TCP sockets across throughput, visibility, and resilience. Hyper-V sockets provide lower but sufficient real-time throughput, reduced visibility to common tools, and immunity to the studied TCP/IP-layer disruption methods.

  • Problem

    Ransomware can encrypt sandbox trace files or stop transfer services, limiting the reliability of post-execution extraction.

  • Method

    The paper evaluates Hyper-V sockets for guest–host sandbox communication and compares them with WinSock TCP sockets across throughput, visibility, and resilience.

  • Results

    Hyper-V sockets reach 700 MB/sec versus 6GB/sec for TCP sockets, remain sufficient for real-time transfer, are less visible to common tools, and are unaffected by studied TCP/IP-layer attacks.

  • Takeaways & Limitations

    Hyper-V sockets are a feasible out-of-band communication method for real-time malware-sandbox data extraction.

Abstract

from arXiv · show

We present how Hyper-V sockets can be used as a real-time communication channel for a malware analysis sandbox. We show that, compared to WinSock TCP sockets, Hyper-V sockets are not subject to TCP/IP-layer blocking and are not enumerated by common TCP connection listing tools. We compare the throughput of the two communication channels as a function of buffer size.

I. INTRODUCTION

The paper addresses ransomware-related failures in sandbox trace extraction by investigating Hyper-V sockets as a real-time, non-disk communication channel. It evaluates throughput, visibility, and resilience against WinSock TCP sockets.

  • Motivation: Ransomware can encrypt trace files or stop transfer services, causing sandbox data loss or preventing retrieval.These failures motivate extracting events without relying on files stored inside the guest VM.
  • Motivation: Hyper-V sockets provide a host–guest communication channel through which applications can communicate without relying on a network connection.The paper identifies Powershell Direct as using Hyper-V sockets and investigates their reuse for sandbox communication.
  • Research questions: The paper evaluates whether guest–host communication can sustain real-time transfer, how Hyper-V socket connections can be detected, and whether they can be blocked or redirected.These questions cover throughput, visibility, and resilience.
  • Contributions: The proposed sandbox avoids traditional disk-based and network-based data extraction methods.This design targets ransomware scenarios in which traces or transfer mechanisms may be tampered with.
  • Contributions: The study measures Hyper-V socket and WinSock TCP throughput as a function of buffer size and examines their visibility and resilience.It also investigates how Protected Process Light affects socket visibility and systematizes disruption methods targeting WinSock TCP communication.

II. BACKGROUND

The background contrasts agent-based and agentless sandbox monitoring and explains why real-time extraction requires a carefully protected communication path between the malware VM and a trusted collector.

  • Sandbox monitoring: Agent-based sandboxes deploy monitoring programs inside the malware VM, whereas agentless approaches often use hypervisor-level Virtual Machine Introspection.Agentless monitoring is transparent to malware but must interpret low-level virtualized hardware activity.
  • Sandbox monitoring: Virtual Machine Introspection faces a semantic gap because high-level behavior must be inferred from CPU and memory activity.This complicates extracting events such as file, registry, and network actions.
  • Sandbox monitoring: Agent-based systems can use commodity hypervisors and monitoring techniques including function hooking, dynamic binary instrumentation, ETW, and kernel drivers.The background identifies the choice between post-execution and real-time data extraction as an important design issue.
  • Data extraction: Post-execution extraction writes collected data to files inside the VM and copies them afterward through network, hypervisor-specific, or virtual-disk mechanisms.This approach leaves traces inside the analysis environment before transfer.
  • Data extraction: Real-time extraction requires an active agent-to-collector connection, commonly implemented as a network link with firewall isolation.Hypervisor-based alternatives require hypervisor support and careful handling of VM-escape vulnerabilities.

III. SYSTEM DESIGN AND IMPLEMENTATION

The system combines an in-VM behavior sensor and communication service with a host-side data collector, using Hyper-V sockets for out-of-band guest–host transfer.

  • Architecture: The architecture contains a behavior sensor, communication service, and host-side data collector server.The collector receives traces and can send commands back to the communication service.
  • Architecture: A kernel-mode filesystem minifilter monitors malware actions and batches multiple events into messages based on total message size.The sensor forwards these messages to the communication service through filesystem minifilter communication ports.
  • Communication path: A user-mode Antimalware PPL communication service forwards sensor messages to the host collector through a Hyper-V socket.The service runs in each virtual machine while the collector listens for connections on the host.
  • Communication path: Out-of-band communication lets malware retain network access without sharing the network adapter with the agent’s communication channel.This avoids filtering agent-generated traffic and reduces the risk that malware or a firewall modifies or blocks it.
  • Hyper-V socket design: Hyper-V socket addresses use a VM identifier GUID and a service identifier GUID, analogous to an IP address and port in TCP.Hyper-V sockets are restricted to communication between the host operating system and guest virtual machines.
  • Protection and monitoring: The communication service is run as an Antimalware Protected Process Light to mitigate user-mode termination or code-injection risks.The paper also notes that malware could still tamper with the agent through a vulnerable driver or by removing kernel callbacks.

IV. EVALUATION

The evaluation assesses the proposed system along three dimensions: throughput, visibility, and resilience.

  • Throughput: The evaluation measures throughput between the guest and host.This corresponds to the paper’s real-time communication research question.
  • Visibility: The evaluation examines the visibility of the communication channel.This addresses whether sandbox communication can be detected.
  • Resilience: The evaluation tests the resilience of the connection.This covers whether the communication can be blocked or redirected.

A. Real time guest-host communication

The study evaluates whether Hyper-V sockets can support real-time guest-host communication, focusing on throughput, buffer size, and the latency-throughput trade-off. Hyper-V and WinSock perform similarly with small buffers, but WinSock is faster at larger sizes while Hyper-V remains more stable.

  • A. Real time guest-host communication: Throughput measurements determine whether Hyper-V sockets can support real-time communication and assess sensitivity to message buffer size.The benchmarks compare host-to-VM data transfer across buffer sizes from 64 bytes to 32 kilobytes on Windows 10 and Windows 11 hosts.
  • A. Real time guest-host communication: Up to 1024 bytes, Hyper-V and WinSock throughput is almost identical; above 4096 bytes, WinSock reaches 6GB/sec versus 700MB/sec for Hyper-V.WinSock peak throughput differs across systems: 4GB/sec on System 1 and 6GB/sec on System 2.
  • A. Real time guest-host communication: Small messages reduce throughput, whereas batching increases throughput but also increases latency and the amount of data lost if the agent terminates unexpectedly.An ideal buffer size appears to be above 4096 bytes and should be selected according to latency requirements.
  • A. Real time guest-host communication: Above 4096 bytes, Hyper-V socket performance is much more stable than Windows socket performance, especially on System 2.The reported stability difference accompanies the larger-buffer throughput comparison.

B. Detecting Hyper-V socket connections

The paper examines whether Hyper-V and WinSock connections can be detected by common Windows tools, including when the client runs as a Protected Process Light. It finds that WinSock is broadly visible, while Hyper-V sockets evade tools limited to conventional TCP/UDP enumeration.

  • B. Detecting Hyper-V socket connections: The evaluation uses netstat, System Informer, and AfdSocketViewer to test visibility of host-guest communication connections.The setup runs a host server and VM client, then repeats the inspection with the client as an Antimalware PPL.
  • B. Detecting Hyper-V socket connections: netstat enumerates TCP connections through the undocumented NsiAllocateAndGetTable function, while System Informer uses Windows APIs for TCP/UDP and undocumented Hyper-V socket controls.System Informer accesses Hyper-V socket listeners and connections through the HvSocketSystem device.
  • B. Detecting Hyper-V socket connections: AfdSocketViewer investigates Windows sockets by duplicating process handles and querying handles associated with the AFD device.It is designed for forensic investigation and debugging of Windows sockets.
  • B. Detecting Hyper-V socket connections: WinSock connections are visible with all three tools, but PPL protection limits netstat to the owning PID and prevents AfdSocketViewer from obtaining socket information.PPL prevents other processes from duplicating handles from the protected client process.
  • B. Detecting Hyper-V socket connections: Hyper-V socket visibility requires host-only Hyper-V APIs because netstat is limited to TCP and UDP connections.The table summarizes visibility by socket type and tool.

C. Blocking, redirecting or closing the connection

The paper compares methods for blocking, redirecting, or terminating WinSock TCP communication with Hyper-V sockets. Because Hyper-V sockets do not use TCP/IP, the examined TCP/IP-layer attacks do not affect them.

  • C. Blocking, redirecting or closing the connection: The resilience evaluation collects known Windows attacks against WinSock TCP connections, many developed to evade or tamper with endpoint security software.Table III organizes the blocking methods by the layer where each operates.
  • C. Blocking, redirecting or closing the connection: WinSock TCP connections can be forcefully terminated by changing their TCP state to MIB_TCP_STATE_DELETE_TCB through SetTcpEntry or equivalent undocumented controls.TcpNsiKill and System Informer implement this technique.
  • C. Blocking, redirecting or closing the connection: Null routing, name resolution policy tables, and hosts-file blocking prevent communication by redirecting or overriding endpoint resolution.The listed methods target traffic associated with EDR communication domains or endpoints.
  • C. Blocking, redirecting or closing the connection: Windows Firewall and WFP block traffic through rules or programmable filters based on connection attributes or process binary paths.These methods can operate without opening or duplicating handles from the targeted processes, including protected processes.
  • C. Blocking, redirecting or closing the connection: Because Hyper-V sockets do not use the TCP/IP stack, they are implicitly unaffected by the enumerated WinSock TCP blocking attacks despite sharing the WinSock programming interface.This separates the communication channel from the TCP/IP-layer mechanisms used by the evaluated attacks.

V. DISCUSSION AND RELATED WORK

The paper situates its sandbox against established VMI- and agent-based systems, highlighting differences in monitoring, communication, and trace extraction timing. Related systems may risk data loss when extraction depends on post-execution transfer or vulnerable communication paths.

  • V. DISCUSSION AND RELATED WORK: VMI- and agent-based sandboxes include open-source and commercial systems such as DRAKVUF, VMRAY, Cuckoo, ANY.RUN, and Hybrid Analysis.
  • V. DISCUSSION AND RELATED WORK: Cuckoo’s original version used injected-DLL function hooking, whereas Cuckoo 3 uses a closed-source kernel-mode driver.
  • V. DISCUSSION AND RELATED WORK: Cuckoo communicates with a host result server through a network adapter and sockets, unlike the paper’s approach.
  • V. DISCUSSION AND RELATED WORK: Curator uses a kernel-mode driver and extracts behavioral traces in stages through a shared folder hosted by a storage-server virtual machine.
  • V. DISCUSSION AND RELATED WORK: Curator extracts results only post-execution, risking data loss after crashes or unexpected shutdowns despite basic self-protection mechanisms.

VI. CONCLUSIONS

The paper evaluates Hyper-V sockets for real-time sandbox data extraction against WinSock TCP sockets across throughput, visibility, and resilience. Hyper-V sockets provide lower but more stable throughput, limited visibility, and resistance to attacks targeting WinSock connections.

  • VI. CONCLUSIONS: 700 MB/sec peak throughput for Hyper-V sockets is lower than TCP sockets’ 6GB/sec, but Hyper-V throughput is more stable and sufficient for real-time transfer.
  • VI. CONCLUSIONS: Hyper-V sockets have more limited visibility than TCP sockets, making them harder to detect.
  • VI. CONCLUSIONS: WinSock-based connection attacks studied in the paper do not apply to Hyper-V sockets because their addressing scheme is GUID based.
Loading 2608.30383v1…