PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Tokens: NumAnchor for Address-Grounded Numeric Operand Representation in Assembly Language Models

Kun Liu, Keith Mitchell, Xin Hu, Jiangcheng Yu, Chenke Luo, Zhengming Ding, Haotian Zhang, Jiang Ming

arXiv:2610.29027v1cs.CR

TL;DR

Numeric-like operands carry structural information that token-level ALM interfaces may obscure, and existing strategies can lose entity identity and locality. NumAnchor represents recoverable operands as typed, canonicalized entities with compact structural descriptors. Across evaluated tasks, it improves retrieval and type recovery, with gains in 10k-candidate function retrieval reaching 35.0% Recall@1.

  • Problem

    Numeric-like operands can encode control-flow, data, and stack structure, but raw values are unstable and generic normalization can discard identity and locality.

    Source: §I · p. 1

  • Method

    NumAnchor maps recoverable numeric-like operands to typed, canonicalized entities and fuses compact structural descriptors into ALM input embeddings.

    Source: §I · p. 1

  • Results

    Across multiple ALM families, NumAnchor improves function retrieval and execution-aware type recovery, with 10k-candidate retrieval gains reaching 35.0% Recall@1.

    Source: §I · p. 2

  • Takeaways & Limitations

    Code, data, and stack descriptors provide complementary benefits with modest overhead across the evaluated downstream tasks.

    Source: §I · p. 2, D. Ablation Study of NumAnchor (RQ3) · p. 9

    Evaluation covers x86/x64 binaries from standard benchmark corpora; other ISAs remain unevaluated, and strong obfuscation may disrupt the static topography used by NumAnchor.

    Source: §VI · pp. 10–11

Abstract

from arXiv · show

Assembly language models (ALMs) are becoming important for automated binary analysis, yet their token-level interfaces often obscure numeric-like operands. In two corpora, such operands account for 45.7% and 40.9% of operand occurrences and may denote branch targets, global data, stack slots, constants, or unresolved pointer-derived values. Existing normalization strategies often lose their identities and structural relationships. We present NumAnchor, an input-boundary adapter that represents recoverable numeric-like operands as typed, canonicalized program entities. NumAnchor encodes code, data, and stack context in compact structural side descriptors, preserving entity identity and locality without increasing the host sequence length. These descriptors are fused into input embeddings before the first Transformer layer, while the host architecture, pretraining objectives, and downstream protocols remain unchanged. Probing frozen instruction-level embeddings yields 98.7% hierarchical-location accuracy and AUROC scores of 0.932 and 0.907 for jump and call target alignment, respectively, showing that the encoded structural information remains recoverable. Across multiple ALM families, NumAnchor achieves relative improvements of up to 35.0% in Recall@1 and 31.7% in MRR on 10k-candidate function retrieval, and improves execution-aware type recovery by up to 5.2 F1 points. Ablations show complementary gains from code, data, and stack descriptors with modest overhead.

I. INTRODUCTION

NumAnchor addresses the difficulty of representing numeric-like operands, which carry structural meaning that token-level interfaces can obscure. It exposes recoverable structure to ALMs and improves results across the evaluated tasks.

I. INTRODUCTION

  • Numeric-like operands make up 45.7% and 40.9% of operand occurrences in two ALM corpora, yet raw values are unstable and generic placeholders discard identity and locality.The operands can encode control-flow anchors, data, or stack structure, making their meaning difficult to infer from surface form alone.Source: §I · p. 1
  • Existing approaches expose only parts of numeric structure, while NumAnchor provides a unified interface for code, data, and stack signals.Evaluation across ALM families and downstream tasks reports consistent gains over baseline models and simpler numeric encodings, with complementary ablation benefits and modest overhead.Source: §I · pp. 1–2
  • NumAnchor maps recoverable numeric-like operands to typed, canonicalized entities and compact descriptors, falling back to generic numeric representations when grounding is unavailable or ambiguous.It fuses the signals at the input embedding layer without changing the host architecture, pretraining objectives, or downstream protocols.Source: §I · p. 2
  • Up to 35.0% Recall@1 and 31.7% MRR gains on 10k-candidate function retrieval accompany type-recovery gains of up to 5.2 F1 points.Frozen-embedding probes also reach 98.7% hierarchical-location accuracy and 0.932/0.907 AUROC for jump/call target alignment.Source: §I · p. 2

A. Why Numeric-like Operands Are Hard to Model

Numeric-like operands are difficult to model because their values are large and unstable, while their meaning depends on executable context. Existing ALM interfaces preserve only subsets of their entity-level structure.

A. Why Numeric-like Operands Are Hard to Model

  • Numeric operands span large address spaces, change with layout transformations, and can denote different things depending on executable context.For example, 0x4d0 can be a stack-frame offset or an arithmetic immediate; raw literals expose unstable values, while coarse placeholders lose entity identity and locality.Source: A. Why Numeric-like Operands Are Hard… · p. 2

B. How Existing ALMs Handle Numeric-like Operands

  • Table I compares which address/entity signals prior interfaces expose, including entity role, co-reference, targets, instruction location, and structural locality.Its indicators distinguish explicit, partial or indirect, and absent support, as well as signals used only in pretraining or auxiliary channels.Source: B. How Existing ALMs Handle… · p. 3, Table I · p. 3
  • Prior interfaces trade off lexical abstraction, jump-aware rewriting, and runtime channels, without uniformly exposing layout-normalized code, data, and stack structure.Abstraction can collapse repeated entities; jump-aware methods focus on branch targets; execution-aware signals may be unavailable in static deployment.Source: B. How Existing ALMs Handle… · p. 3

III. MOTIVATION AND OVERVIEW

NumAnchor addresses the gap between static-analysis artifacts and ALM inputs by exposing recovered numeric-operand structure as model-facing signals. Its workflow types and canonicalizes operands, adds structural descriptors, serializes them into the host format, and fuses them before the first Transformer layer.

III. MOTIVATION AND OVERVIEW

  • NumAnchor is a backbone-agnostic input-boundary adapter that maps recoverable numeric-like operands to typed, canonicalized entities and falls back to generic numeric representations when grounding is unavailable or ambiguous.It adds compact descriptors for shared identity and hierarchical locality without changing the Transformer backbone.Source: §III · p. 4

A. NumAnchor Workflow

  • The workflow types operands and instruction anchors, constructs type-specific descriptors, serializes them into the host carrier, and fuses the signals before the first Transformer layer.The same interface is used during pretraining and downstream fine-tuning, preserving structural cues for static deployment.Source: A. NumAnchor Workflow · p. 4, §IV · p. 4

IV. SYSTEM DESIGN

  • The host-compatible record includes a lexical token, routing path, optional code or data descriptor, optional signed stack offset, and host-specific auxiliary fields.An internal grounded entity identifier supports co-reference but is never emitted as a standalone lexical token.Source: §IV · p. 4

A. Analysis-Driven Typing and Canonicalization

Analysis-driven typing assigns numeric-like operands semantic routes using recovered static-analysis artifacts, then canonicalization separates model-facing tokens from grounded entity identity. Repeated code and data references retain shared identifiers, while stack references retain signed frame-relative offsets.

A. Analysis-Driven Typing and Canonicalization

  • Typing routes operands using recovered cross-references, section membership, function and basic-block boundaries, and stack-frame layouts; opcode tokens can also carry the current instruction address as an anchor.The prototype uses IDA Pro, though the interface can use other disassembly frontends that provide comparable artifacts.Source: A. Analysis-Driven Typing and… · p. 5
  • Canonicalization maps typed occurrences to compact model-facing tokens while retaining internal grounding metadata for resolved code, data, and instruction-anchor entities.Internal identifiers build structural descriptors and are not emitted as tokens.Source: A. Analysis-Driven Typing and… · p. 5
  • Co-referent code and data references reuse the same internal identifier, while stack-relative references reuse the same signed displacement across instructions.Constants default to imm and unresolved pointer candidates to num; non-numeric tokens have no grounded identifier or stack offset.Source: A. Analysis-Driven Typing and… · p. 5

B. Structural Descriptor Construction

NumAnchor constructs normalized hierarchical descriptors for grounded code and data entities, while preserving stack structure through signed frame-relative offsets. It masks structural routes when the required layout information is invalid or unavailable.

B. Structural Descriptor Construction

  • Code descriptors locate an address across basic-block, function, and binary-span boundaries, placing source instruction anchors and branch targets in a shared coordinate system.This supports modeling their relative spatial relation even when they are distant in the token sequence.Source: B. Structural Descriptor Construction · p. 6
  • Descriptors are generated only when enclosing spans are valid and non-degenerate; otherwise, the structural route is masked..Source: B. Structural Descriptor Construction · p. 6
  • Data descriptors encode normalized locality within a section, adding finer-grained information than the canonical token’s coarse section-permission label.If section attributes are unavailable or unreliable, NumAnchor sets the descriptor to ⊥.Source: B. Structural Descriptor Construction · p. 6
  • Stack entities retain their frame-relative signed displacement rather than being mapped to global binary coordinates, with no layout descriptor.This preserves their frame-relative geometry.Source: B. Structural Descriptor Construction · p. 6

C. Host-Compatible Serialization

NumAnchor serializes host-compatible inputs with an address-aware tuple and parallel structural channels. Input-side fusion adds route-specific structure while keeping the host sequence compact.

C. Host-Compatible Serialization

  • Each serialized element retains an address-aware tuple combining a lexical token, semantic route, structural coordinate, stack offset, and host-specific fields.The lexical stream follows the host vocabulary while parallel side channels carry route-specific geometry and mask inactive routes.Source: C. Host-Compatible Serialization · p. 6
  • NumAnchor offloads raw numeric values into parallel coordinate descriptors, avoiding added lexical tokens and mitigating vocabulary sparsity.For execution-aware backbones, augmentation affects static token-side channels, not dynamic-trace or state channels.Source: C. Host-Compatible Serialization · p. 6

D. Input-Side Fusion and Structural Lifting

  • Code and data coordinates receive multi-scale Fourier features, allowing projections to represent both coarse layout and fine-grained locality.Separate code and data projections reflect their distinct structural behaviors.Source: D. Input-Side Fusion and Structural… · p. 7
  • Stack offsets use a hybrid discrete-continuous encoding that combines precise local buckets with coarse magnitude markers for extreme displacements.The design supports exact matching for frequent local slots while limiting embedding-table growth for rare, large offsets.Source: D. Input-Side Fusion and Structural… · p. 7

E. Portability and Fallback Design

NumAnchor adds structural signals only at the input boundary, before the Transformer layers. It leaves the host backbone, prediction heads, and training objectives unchanged.

E. Portability and Fallback Design

  • Structural signals are fused before the Transformer layers, leaving the host backbone, prediction heads, and training objectives unchanged.The adapter can support different ALM input formats without auxiliary pre-training.Source: E. Portability and Fallback Design · p. 7
  • When static analysis cannot reliably ground an operand as code, data, or stack-related, NumAnchor masks its structural side fields.It falls back to generic immediate or numeric channels to avoid unsupported structural assumptions while preserving frontend compatibility.Source: E. Portability and Fallback Design · p. 7
  • The adapter's fallback preserves frontend compatibility when structural grounding is unavailable or ambiguous.Source: E. Portability and Fallback Design · p. 7

V. EVALUATION

The evaluation tables cover operand-level statistics in stripped binaries and the evaluated models with their pre-training tasks.

V. EVALUATION

  • Table II reports operand-level statistics in stripped binaries, with percentages measured over all operand occurrences.Source: Table II · p. 7
  • Table II groups operand statistics into Code, Data, Stack, Imm., and Other, which sum to Num./All.Source: Table II · p. 7
  • Table III summarizes evaluated models and identifies their pre-training tasks, including MLM, CWP, DUP, JTP, and GSM.These abbreviations denote Masked Language Modeling, Context Window Prediction, Def-Use Prediction, Jump Target Prediction, and Generative State Modeling.Source: Table III · p. 7

A. Experimental Setup

The evaluation uses controlled within-model comparisons across ALM backbones and tasks. Probes of frozen PalmTree embeddings show that full NumAnchor exposes hierarchical locality and control-flow target alignment.

A. Experimental Setup

  • The controlled protocol compares each baseline only with its NumAnchor-enhanced variant, keeping architecture, training hyperparameters, and task formulation unchanged.The evaluated benchmarks include BinaryCorp-3M for the jTrans lineage and the StateFormer dataset for other baselines.Source: A. Experimental Setup · p. 7
  • Numeric operands account for 45.73% of StateFormer operand occurrences and 40.93% in jTrans; Figure 4 depicts PalmTree retrieval scores across pool sizes.Its bars show means across repeated sampling trials and error bars show standard deviations.Source: A. Experimental Setup · p. 8, Figure 4 · p. 8

B. Probing Address Structure (RQ1)

  • 98.7% accuracy at block-level location prediction and 0.932/0.907 AUROC for jump/call target alignment show structure recoverable in frozen embeddings.Full NumAnchor also reaches 91.3% binary-level and 95.7% function-level location accuracy.Source: B. Probing Address Structure (RQ1) · p. 8
  • The hierarchical-location probe predicts an instruction’s relative bucket within the binary, function, and basic block.Full NumAnchor outperforms the NAU variant at function- and block-level locality, where NAU reaches only 31.2% and 44.1%.Source: B. Probing Address Structure (RQ1) · p. 8

C. Downstream Evidence Across ALM Families (RQ2)

Within-host comparisons show NumAnchor improving retrieval across static-code, address-rebasing, and execution-aware ALMs. Ablations attribute gains to complementary structural descriptors and encoding locality beyond operand type alone.

C. Downstream Evidence Across ALM Families (RQ2)

  • At 10k candidates, PalmTree gains 35.0% Recall@1 and 31.7% MRR, while Vanilla BERT gains approximately 21% and 20%, respectively.Both carriers improve across pool sizes, and their NumAnchor variants degrade more slowly as distractors increase.Source: C. Downstream Evidence Across ALM… · p. 9, Figure 5 · p. 9
  • Across six cross-optimization retrieval pairs, jTrans average MRR/Recall@1 rises from 0.620/0.537 to 0.706/0.619, and JTRANS-CLAP rises from 0.569/0.495 to 0.693/0.607.JTRANS-CLAP recovers much of its gap after adding structural descriptors.Source: C. Downstream Evidence Across ALM… · p. 9
  • NumAnchor improves ALMs across instruction- and function-level settings, spanning static-code, address-rebasing, and execution-aware tasks under within-backbone comparisons.The results are comparisons against each host model’s baseline, not direct rankings across different models.Source: C. Downstream Evidence Across ALM… · pp. 8–9
  • For TREX, 10k-candidate Recall@1/MRR improves from 0.28/0.40 to 0.31/0.43; StateFormer type-recovery F1 gains are +5.2, +3.6, +2.9, and +2.0 points from O0 to O3.These are complementary gains alongside execution-aware components.Source: C. Downstream Evidence Across ALM… · p. 9

D. Ablation Study of NumAnchor (RQ3)

  • In PalmTree ablations, CODE descriptors improve Recall@1 by +0.042, while adding DATA and STACK raises the gain to +0.071.The passages attribute complementary gains to code-location information, memory references, and stack-frame layouts.Source: D. Ablation Study of NumAnchor (RQ3) · p. 9
  • ROLE-ONLY and continuous numeric encoding trail the full design; Fourier lifting raises MRR from 0.346 to 0.386 beyond the no-Fourier variant.The full structural encoder combines Fourier lifting for code/data coordinates with type-specific stack encoding.Source: D. Ablation Study of NumAnchor (RQ3) · pp. 9–10
  • On jTrans, code descriptors improve the baseline, data descriptors add further gains, and full NumAnchor raises average MRR/Recall@1 from 0.620/0.537 to 0.706/0.619.The passage notes this is additive even though jTrans already uses jump-aware symbolic rebasing.Source: D. Ablation Study of NumAnchor (RQ3) · p. 10

E. Overhead of NumAnchor (RQ4)

NumAnchor adds side-channel encoders without lengthening the host sequence, with modest parameter and inference overhead. The evidence is bounded to static information and evaluated x86/x64 settings, with cross-ISA and stronger-obfuscation robustness left for future work.

E. Overhead of NumAnchor (RQ4)

  • For PalmTree, NumAnchor adds 4.73M parameters over an 86.31M-parameter baseline, a 5.48% increase, without increasing host sequence length.Across groups of 20 functions, added inference time is 0.15s for PalmTree, 0.11s for jTrans, and 0.08s for TREX, at most about 7.5ms per function.Source: E. Overhead of NumAnchor (RQ4) · p. 10

VI. THREATS TO VALIDITY AND CONCLUSION

  • The evaluation covers x86/x64 binaries under standard compiler optimizations; other ISAs and robustness under stronger obfuscation remain unevaluated.The authors note that strong obfuscation may disrupt the static topography NumAnchor uses.Source: §VI · pp. 10–11
  • NumAnchor is limited to statically observable information and maps unresolved heap offsets or complex pointer-derived addresses to generic numeric tokens.The prototype also relies on IDA Pro to recover function boundaries and related artifacts.Source: §VI · p. 10

Related papers

Loading 2610.29027v1…