PaperlayerView exact revision on arXiv

Source-linked AI summary

An Observational Investigation of Reverse Engineers' Processes

Daniel Votipka, Seth M. Rabin, Kristopher Micinski, Jeffrey S. Foster, Michelle L. Mazurek

arXiv:1912.00317v1cs.CRcs.HC

TL;DR

Reverse engineering is important to security work, but the processes of reverse engineers are not well understood, leaving tool designers with limited guidance. This paper studies 16 professionals and models their process in three phases. The authors report phase-specific analysis patterns and propose design guidelines for RE tools.

  • Problem

    Reverse-engineering processes are not well understood, despite the task’s role in security work and the availability of many tools.

    Source: §1 · p. 1, §7 · p. 13

  • Method

    The authors conducted semi-structured observational interviews with 16 professional reverse engineers and developed a process model from their accounts.

    Source: §8 · p. 14

  • Results

    The model identifies overview, sub-component scanning, and focused experimentation, with typically static methods in the first two phases and dynamic methods in the last.

    Source: §1 · p. 2, §8 · p. 14

  • Takeaways & Limitations

    The model and five proposed guidelines are intended to help RE tools better match reverse engineers’ processes.

    Source: §7 · p. 12, §8 · p. 14

    Participants may have omitted confidential process details, although the authors believed this did not affect results because general processes were still described.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Reverse engineering is a complex process essential to software-security tasks such as vulnerability discovery and malware analysis. Significant research and engineering effort has gone into developing tools to support reverse engineers. However, little work has been done to understand the way reverse engineers think when analyzing programs, leaving tool developers to make interface design decisions based only on intuition. This paper takes a first step toward a better understanding of reverse engineers' processes, with the goal of producing insights for improving interaction design for reverse engineering tools. We present the results of a semi-structured, observational interview study of reverse engineers (N=16). Each observation investigated the questions reverse engineers ask as they probe a program, how they answer these questions, and the decisions they make throughout the reverse engineering process. From the interview responses, we distill a model of the reverse engineering process, divided into three phases: overview, sub-component scanning, and focused experimentation. Each analysis phase's results feed the next as reverse engineers' mental representations become more concrete. We find that reverse engineers typically use static methods in the first two phases, but dynamic methods in the final phase, with experience playing large, but varying, roles in each phase. % and the role of experience varies between phases. Based on these results, we provide five interaction design guidelines for reverse engineering tools.

1 Introduction

The paper addresses the limited understanding of reverse engineers’ processes that leaves tool design ad hoc, using an observational study to model those processes. It identifies three phases and cross-phase trends that inform five design guidelines.

1 Introduction

  • The study examines REs’ questions and hypotheses, investigative steps, and decisions about which code and analyses to pursue.Because prior work offered limited process models and no theoretical basis for quantitative assessment, the authors chose an exploratory qualitative approach to characterize behavior and inform tool design.Source: §1 · pp. 1–2
  • The authors conducted a semi-structured observational study with 16 participants, asking them to recreate recent RE experiences while researchers tracked their decisions and reasoning.The study tracked mental simulation methods, questions, hypotheses, and recognized beacons.Source: §1 · p. 2
  • The RE process has three phases—overview, sub-component scanning, and focused experimentation—with the last two forming an iterative loop.REs use the overview to prioritize sub-components, then test questions and hypotheses through execution or in-depth static analysis until their goal is achieved.Source: §1 · p. 2
  • REs typically use static analysis in the first two phases and dynamic methods in focused experimentation; experience shapes decisions throughout.Experience informs search priorities, recognition of functionality and potential vulnerabilities, and method selection.Source: §1 · p. 2
  • The paper identifies five interaction-design guidelines for RE tools based on its findings.The motivation is that tool design has been ad hoc despite extensive tool development and limited theoretical understanding of RE processes; RE also differs from traditional program comprehension because source, developers, and documentation may be unavailable and code may be obfuscated.Source: §1 · pp. 1–2

2 Background and Related Work

The paper draws on naturalistic decision-making, program-comprehension research, and prior RE usability and process studies. It adapts these ideas to expert reverse engineering, where analysts often work with binaries or obfuscated code and pursue different goals from developers.

2.1 Naturalistic Decision-Making

  • Naturalistic decision-making research informed the study’s focus on expert decisions, and the authors applied the Critical Decision Method to investigate RE processes.Prior applications of these methods have informed automation and interaction design in several domains.Source: §2.1 · p. 2

2.2 Program Comprehension

  • Program-comprehension studies describe non-linear hypothesis testing, beacon recognition, and switching among mental simulation, execution, and searches for contradictory evidence.The authors expected some overlap with RE, but anticipated divergence because RE often involves obfuscated code or raw binaries and focuses on identifying or exploiting flaws.Source: §2.2 · pp. 2–3

2.3 Improving Usability for RE Tools

  • Prior RE usability work includes incremental static analysis for developers, human-assisted vulnerability discovery for non-experts, and usability-oriented decompilation.Other work for experts proposed a cryptographic API specification language that still required learning a potentially complicated language.Source: §2.3 · p. 3
  • DREAM++ evaluations found that code transformations improved students’ and professionals’ ability to reverse engineer malware.The authors hope their broader investigation of RE processes will spur further usability improvements.Source: §2.3 · p. 3

2.4 The Vulnerability Discovery Process

  • Compared with Bryant’s smaller mixed-methods study, this work offers a more detailed model based on 16 interviews and observed RE on different real-world programs.The model expands on prior findings by describing specific approaches and how RE behaviors change through the process.Source: §2.4 · p. 3
  • An earlier study of white-hat hackers identified RE as part of vulnerability discovery but had a broader focus, prompting this more focused investigation.The earlier study also addressed process, skill development, and community interaction.Source: §2.4 · p. 3

3 Method

The authors used semi-structured observational interviews and iterative coding to develop a theoretical model of expert RE processes. They note limits involving recall, confidential steps, and the focus on experienced practitioners.

3 Method

  • The study used a semi-structured, observation-based interview protocol to develop a theoretical model of RE strategy and techniques.Interviews averaged 70 minutes and were conducted by an interviewer with six years of professional RE experience.Source: §3 · pp. 3–4

3.1 Interview Protocol

  • Participants recreated a recent RE experience while sharing their screen, allowing the interviewer to observe actions and probe their reasoning.Follow-up questions covered decisions, questions and hypotheses, simulation methods, beacons, and external information sources.Source: §3.1 · pp. 4–5

3.2 Data Analysis

  • The authors developed the model by coding interview data, grouping codes into three analysis phases, and using axial coding to identify relationships and cross-phase trends.The coders repeated coding and reconciliation until Krippendorff’s Alpha reached 0.8.Source: §3.2 · p. 5

3.3 Limitations

  • Recall may omit task details, especially for expert work, although recreating the task under observation allowed probing of actions that might otherwise be skipped.Participants may also have withheld confidential process details, but the authors believed this did not affect results because participants still described their general process.Source: §3.3 · p. 5
  • The study focused on experienced REs, so the authors identify newer REs’ struggles and development as a topic for future work.Source: §3.3 · p. 5

4 Recruitment and Participants

The researchers recruited experienced reverse engineers through online communities, relevant organizations, and conferences. They interviewed 16 participants after screening and stopping when new themes no longer emerged.

4 Recruitment and Participants

  • Participants were recruited through online RE forums, vulnerability-discovery organizations, and relevant conferences.Forum recruitment included communities for RE tools and Reddit; organizations included ranked CTF teams and bug-bounty companies.Source: §4 · p. 5
  • The study selected volunteers who self-rated at least three on a five-point expertise scale and had at least three years of RE experience.The screening questionnaire also asked about years of experience and demographics.Source: §4 · p. 6
  • Of 68 screening responses, 42 met the expertise criteria and 16 were interviewed; recruitment stopped when interviews yielded no new themes.The authors describe this as saturation and note that 16 participants falls within a recommended range for qualitative studies.Source: §4 · p. 6
  • Interviews covered different program types and recorded participants’ self-reported skill, experience, and recruitment method.Three participants used a challenge binary because they could not show examples from confidential or proprietary work.Source: §4 · p. 6
  • Participants were mostly male (94%), under 30 (63%), and held a bachelor’s degree (75%), similar to reported bug-bounty demographics.The comparison cited reported similar gender, age, and education distributions among Bugcrowd hunters.Source: §4 · p. 6

5 Results: An RE Process Model

Reverse engineers move from a broad program overview to prioritized sub-component scanning, then test focused questions through execution or in-depth static analysis; results feed back iteratively. Their methods and beacons vary by phase, and the study reports broadly similar high-level processes for malware analysis and vulnerability discovery.

5 Results: An RE Process Model

  • The process has three phases—overview, sub-component scanning, and focused experimentation—with results feeding back to refine questions and hypotheses until the overall goals are achieved.Each phase has its own questions, methods, and beacons.Source: §5 · p. 6

5.1 Overview (RQ1)

  • In overview, REs seek a high-level view to prioritize code, commonly listing strings and APIs, running the program, and reviewing metadata.For malware, they first unpack binaries when packing makes strings and imported APIs unintelligible.Source: §5.1 · p. 7, Malware analysts perform overview… · p. 7

Malware analysts perform overview after unpacking

  • Malware analysts perform these overview analyses after unpacking packed binaries so strings and imported APIs become intelligible.The study reports that they then use the same overview analyses as other reverse engineers.Source: Malware analysts perform overview… · p. 7

5.2 Sub-component Scanning (RQ1)

  • During sub-component scanning, REs prioritize functions and scan for APIs, strings, constants, and variable names, inferring behavior from individual beacons and their sequences.They also use control flow, compiler artifacts, and program flow, which can help when semantic information is obfuscated.Source: §5.2 · pp. 7–9
  • REs also inspect control-flow patterns, compiler instructions, function prototypes, and program flow, or trace data- and control-flow paths to test whether a specific behavior is possible.One path analysis found no possible undefined-memory read because the remaining paths returned -1.Source: §5.2 · pp. 8–9

5.3 Focused Experimentation (RQ1)

  • In focused experimentation, REs test hypotheses under specific inputs and conditions, often using debuggers, monitoring tools, or execution-environment changes; some fuzz inputs or compare behavior with another implementation.Comparisons include rewriting code or using a public implementation and checking whether outputs match.Source: §5.3 · pp. 9–10
  • REs resort to line-by-line reading when code is simple or execution is costly to set up; some track conditions manually and use a solver when possibilities become too complex.They may also notice a beacon that supplies enough information to stop an experiment and return to scanning.Source: §5.3 · p. 10

6 Results: Cross-phase Trends

The cross-phase findings show REs shifting from static representations to dynamic execution, with experience and strategy shaping where they look and which analysis methods they choose.

6 Results: Cross-phase Trends

  • REs mainly use static representations in the first two phases and dynamic execution during focused experimentation, though switching can feel daunting.Manually transferring information between static and dynamic tools was difficult for 14 participants; four placed both tools side by side.Source: §6 · p. 11
  • In the first two phases, REs use prior experience to prioritize outputs and paths, falling back on simple heuristics when they lack a useful precedent.Examples include inspecting larger functions first or avoiding functions with many cross-references; during sub-component scanning, hypotheses often draw on prior experience, or else on more labor-intensive execution or code review.Source: §6 · p. 11

Experience used to select analysis method throughout

  • REs choose among analysis methods based on context, sometimes skipping overview or trading precision for a more readable representation.Five participants described weighing an approximation’s benefits against its inaccuracies and switching granularity to verify results.Source: Experience used to select analysis… · p. 11
  • REs favored tools that connect results to code, including disassemblers that display strings and API calls alongside references.Participants reported using disassembler plugins for fuzzing or symbolic execution when available; one used Z3 separately because she did not know a suitable code-context tool for the binary.Source: Experience used to select analysis… · pp. 11–12
  • Readability work included decompiling code, renaming variables, taking notes, and reconstructing data structures.Fourteen participants renamed variables and took notes; eight reconstructed data structures.Source: Experience used to select analysis… · p. 12
  • REs consulted documentation and online resources to resolve details about opcodes, APIs, poorly documented system functions, and distinctive constants.Ten referenced system or API documentation, four used online articles, four used question-answering sites, and seven searched specific constants or strings.Source: Experience used to select analysis… · p. 12

7 Discussion

The discussion presents a three-phase model of reverse engineering and derives interaction guidelines from observed behaviors. It also identifies tool-design challenges and notes that the guidelines still need validation.

7 Discussion

  • The model describes hypothesis generation and refinement across phases, alongside the questions, actions, and decisions through which REs expand their program knowledge.The authors say the model confirms and extends prior work describing increasingly refined hypotheses.Source: §7 · p. 12
  • The five guidelines are grounded in the findings, but further work is needed to validate their effectiveness.Table 2 summarizes each guideline with an example application and adoption challenges.Source: §7 · p. 12, Table 2 · p. 13
  • G1 recommends matching tool interaction to the phases: overview, highlighted program slices during scanning, and detailed analysis on demand.The authors note that current tools often handle only one part of the process, leaving REs to stitch results together.Source: §7 · p. 12
  • G2 recommends presenting both inputs and outputs in code context; Lighthouse highlights covered blocks but does not let users provide inputs in that context.The authors use Lighthouse to illustrate both an existing capability and a remaining interaction gap.Source: §7 · p. 13

G3. Allow data transfer between static and dynamic con-

  • G3 recommends transferring information between static and dynamic contexts, which nearly all participants switched between at least once.The authors identify visualization and performance as challenges for more complex interactive analyses.Source: G3. Allow data transfer between static… · p. 13
  • G4 recommends letting REs select analysis methods and tune analyses to their needs, including choices that trade precision for readability.The authors describe the decompiler’s binary choice between an easier-to-read decompiled view and a more complex disassembled view as a minimal implementation.Source: G3. Allow data transfer between static… · p. 13
  • G5 recommends supporting readability edits such as notes and renaming, and propagating annotations to similar outputs.The authors identify semantic learning from user input as an implementation challenge because it may require attention to program context.Source: G3. Allow data transfer between static… · p. 14
  • The authors suggest that exploratory visual analysis may offer useful design ideas, while cautioning that its techniques may not transfer as-is to reverse engineering.They point to similarities in overview, hypothesis refinement, scanning, and detailed analysis.Source: G3. Allow data transfer between static… · p. 14

8 Conclusion

The section summarizes a study of 16 professional reverse engineers, modeling their work in three phases and cataloging the questions and cues they use to interpret programs. It also presents five tool-design guidelines intended to help tools better match the reverse-engineering process.

8 Conclusion

  • Reverse engineering proceeds through overview, sub-component scanning, and focused experimentation, with static techniques typical in the first two phases and dynamic techniques in the last.Engineers use varied manual and automated approaches, sometimes combining methods for a task.Source: §8 · p. 14
  • The authors developed this process model through semi-structured observational interviews with 16 professional reverse engineers.The study aimed to model reverse engineers’ processes to support better tool design.Source: §8 · p. 14
  • The authors propose five design guidelines and expect the process model to help tools better match reverse engineers’ work.Source: §8 · p. 14

C.2 Question

  • Reverse engineers ask whether code is reachable, how the program changed, and which control-flow path an input follows.They also ask about variable or register types, function inputs and outputs, callers and callees, and code behavior.Source: §C.2 · p. 20
  • Their questions also cover how function outputs are used, what calls or uses an item, and what a function calls.These questions address the purpose and connections of functions and other items of interest.Source: §C.2 · pp. 20–21

C.3 Beacon

  • Beacon cues include strings, API calls, low-level operations, constants, variable names, and operation sequences that suggest behavior at a glance.Comments and program metadata, such as lines of code, are also cues.Source: §C.3 · p. 21
  • Other beacons include function prototypes, UI elements, control-flow paths, and a code segment’s position within broader program behavior.Developer comments may also provide cues when source code is available.Source: §C.3 · p. 21

C.4 Simulation Method

The study grouped simulation methods into dynamic analysis, static analysis, and metadata review, and also coded how participants interacted with these methods.

C.4 Simulation Method

  • Simulation methods were classified into dynamic analysis, static analysis, and metadata review.The researchers also coded interactions involving the use of these methods.Source: §C.4 · p. 21

C.4.1 Dynamic Analysis

Dynamic analysis methods include running programs under selected conditions, observing their behavior, and comparing outcomes with known implementations.

C.4.1 Dynamic Analysis

  • Reverse engineers execute programs with specific inputs, set debugger breakpoints, or alter the environment while observing effects.Environment changes can include network state or files on disk.Source: §C.4.1 · p. 21
  • They can monitor program interactions with the environment using tools such as packet capture or file monitoring.Source: §C.4.1 · p. 21
  • Other methods include editing and recompiling source code, fuzzing with varied inputs, and comparing results against a known implementation.Fuzzing inputs may be selected manually or automatically.Source: §C.4.1 · p. 21

C.4.2 Static Analysis

Static analysis methods range from quick scans and searches to detailed control- and data-flow analysis; metadata review includes checking security mitigations.

C.4.2 Static Analysis

  • They may read code line by line, use symbolic execution, or cross-reference function calls.Source: §C.4.2 · pp. 21–22
  • Reverse engineers scan code for beacons, list imported functions or strings, and search for specific strings to locate relevant information.Source: §C.4.2 · p. 21
  • Control-flow and data-flow analyses examine paths through code, including paths under specific control-flow inputs and data movement between variables and memory.Source: §C.4.2 · p. 22
  • Other static methods include comparing code with a known implementation, reimplementing suspected behavior, and reconstructing data structures from binaries.Source: §C.4.2 · pp. 21–22

C.4.3 Metadata Review

  • Metadata review includes checking the program's security mitigations.Source: §C.4.3 · p. 22

C.4.4 Method Interactions

Reverse engineers move between static and dynamic methods in both directions or use them in concert, sometimes viewing both representations simultaneously. Decision coding records both the reasoning behind a decision and its type.

C.4.4 Method Interactions

  • Static-to-dynamic and dynamic-to-static interactions use information from the first method to inform the second.Source: §C.4.4 · p. 22
  • In combined use, reverse engineers continually pass information between static and dynamic methods, and may view both representations side by side.Source: §C.4.4 · p. 22

C.5 Decision

  • For each decision point, the researchers coded both the reasoning behind the decision and the type of decision made.Source: §C.5 · p. 22

C.5.1 Reason

Decision reasoning includes investigation state, strategy, and task needs, as well as cues from observed behavior, code paths, function prototypes, proximity, experience, and program metadata.

C.5.1 Reason

  • Decisions may reflect investigation state, an overarching strategy, or a specific sub-goal that must be completed.Source: §C.5.1 · p. 22
  • A function’s prototype can guide investigation; for example, many function-pointer arguments may suggest that it starts threads.Source: §C.5.1 · p. 22
  • Other decision cues include proximity to interesting code, prior reverse-engineering experience, and program metadata.Source: §C.5.1 · p. 22
  • Reverse engineers also base decisions on observed behavior and the control-flow or data-flow path being followed.Source: §C.5.1 · p. 22

C.5.2 Type

The coded decision types concern what code or functions to analyze, which simulation inputs or methods to use, and the order of analysis.

C.5.2 Type

  • One decision type is selecting which code segments or functions to analyze.Source: §C.5.2 · p. 22
  • Reverse engineers decide which inputs to use when performing a simulation method.Source: §C.5.2 · p. 22
  • They also decide the order in which to analyze portions of code or functions.Source: §C.5.2 · p. 22
  • Another decision type is choosing which simulation method to employ at a given moment.Source: §C.5.2 · p. 22

Related papers

Loading 1912.00317v1…