PaperlayerView exact revision on arXiv

Source-linked AI summary

Structural Inference in Undocumented Mobile Databases: A Reproducible Benchmark for Evaluating Agentic Reasoning in Digital Forensics

Jeel Piyushkumar Khatiwala, Divyangkumar Patel, Weifeng Xu

arXiv:2608.21470v1cs.CR

TL;DR

Undocumented mobile databases make relational inference difficult, yet structurally incorrect joins can execute and appear plausible. This paper isolates structural reasoning in a deterministic benchmark across contrasting SQLite schemas, finding reliable inference in regular schemas but sharp degradation under ambiguity. The results support explicit verification before inferred relationships are treated as forensic evidence.

  • Problem

    The study addresses limited understanding of whether agents can infer reliable relational structure in undocumented mobile databases when structurally plausible but incorrect queries may execute successfully.

  • Method

    The paper evaluates a fixed agent configuration on two contrasting SQLite repositories using expert-validated ground truth, separating structural correctness, execution coherence, and robustness.

  • Results

    Structural inference is reliable when identifiers are explicit and consistently propagated but degrades under schema ambiguity, where successful execution can conceal divergent joins and systematic failure modes.

  • Takeaways & Limitations

    Execution success alone is insufficient for evidential reliability, so inferred relational pathways require explicit structural verification in irregular databases.

  • Takeaways & Limitations

    The study scopes its conclusions to structural soundness rather than end-to-end forensic accuracy under a controlled experimental design.

Abstract

from arXiv · show

Agentic large language models are increasingly used in digital forensic analysis, yet their ability to infer relational structure inside undocumented mobile application databases remains poorly understood. In forensic contexts, structurally incorrect inferences can yield results that appear plausible while remaining evidentially unsound. This work evaluates agentic structural inference as an isolated capability, treating execution success and structural correctness as distinct evaluation axes. It examines how an agent reconstructs table relationships, linking attributes, and executable join paths when given only a raw database and a natural-language investigative prompt. We apply a fixed, deterministic evaluation pipeline to two contrasting SQLite repositories: Android's SMS database with stable identifier propagation, and Snapchat's database with irregular schemas, ephemeral identifiers, and polymorphic relationships. Using expert-verified SQL ground truth, we evaluate (i) structural correctness of inferred relational links, (ii) execution coherence under multi-table reasoning, and (iii) robustness and failure modes of inferred structure when execution succeeds but relational interpretation diverges from expert ground truth. Evaluation is performed independently of semantic interpretation, with full queries and execution traces provided in the Appendix. Results show that structural inference remains reliable in regular schemas but degrades sharply as schema ambiguity increases, frequently producing structurally plausible yet incorrect joins that execute successfully. These findings clarify where schema-agnostic agentic reasoning can support forensic analysis, how its robustness degrades under realistic schema irregularities, and why additional verification remains essential before inferred relationships can be treated as reliable evidence.

I. INTRODUCTION

Undocumented mobile databases force forensic analysts to infer table roles, linking fields, and join paths because schemas evolve without documentation. This study isolates structural inference and evaluates execution success separately from agreement with expert-validated relational pathways across regular and irregular repositories.

  • Mobile application databases often evolve without documentation, requiring analysts to infer table roles, linking fields, and implicit join paths through inspection and iterative querying.
  • Template-driven forensic parsers degrade when schemas drift or linking attributes become indirect, while manual reconstruction is slow and inconsistent for unstable identifiers.
  • The study evaluates structural inference as an isolated capability, focusing on table selection, linking attributes, join paths, and executable SQL rather than semantic retrieval.
  • Execution success is evaluated separately from whether an inferred relational structure matches expert-validated pathways, exposing executable but structurally incorrect queries.
  • The benchmark applies identical prompts, configuration, constraints, and scoring across Android’s regular mmssms.db and Snapchat’s irregular, polymorphic main.db.
  • The evaluation covers join-path construction, linking-attribute inference, structural filtering, and failure modes including key mismatch, join-path drift, auxiliary-table interference, and overconstraint.

II. RELATED WORK

Prior forensic research addresses parser stability, dataset diversity, storage ambiguity, and automated reasoning largely in isolation. This study targets the missing systematic evaluation of relational inference, execution survival, and robustness under schema ambiguity in undocumented mobile databases.

  • Parser-based extraction relies on predefined schemas and degrades when application layouts evolve or even minor schema changes break tool compatibility.
  • Mobile forensic repositories exhibit substantial structural divergence, while synthetic and standardized datasets still require manual identification of relational patterns and linking attributes.
  • Damaged storage states, partial records, irregular naming, and weak relational cues complicate manual reconstruction of schema knowledge and join paths.
  • Large language models perform strongly with well-defined schemas but degrade when field names are ambiguous or foreign-key relationships are absent.
  • Adjacent anomaly-detection methods generally presume known schemas and do not generalize to undocumented relational structure.
  • No existing study systematically evaluates how relational structure is inferred, survives execution, and degrades under ambiguity in undocumented mobile databases.
  • The framework defines structural inference as reconstructing tables, linking attributes, join paths, and ordering semantics from schema-visible signals in a raw SQLite repository.

A. Terminology and Agent Configuration

The framework uses explicit terminology and a controlled multi-stage agent loop to transform a raw SQLite schema and investigative question into validated, read-only SQL execution. Structural cues make relational assumptions explicit and testable.

  • Terminology: Execution coherence means a generated query executes without structural contradictions such as invalid joins, unresolved attributes, or empty relational paths.
  • Terminology: The agent is the complete controlled reasoning loop, while the model denotes only the underlying large language model used as a bounded reasoning component.
  • Agent Configuration: Each task follows schema snapshot acquisition, structural hypothesis formation, SQL generation, and supervised execution under a fixed configuration.
  • Structural Inference: Without foreign keys or documentation, the agent uses repeated identifiers, substring overlaps, prefix alignment, timestamp co-occurrence, and parallel table geometry as structural cues.
  • SQL and Execution: The pipeline converts hypotheses into one constrained SQL query using only schema-defined tables and columns, then executes it in a sandboxed read-only environment while recording traces.

E. Evaluation Alignment

The evaluation aligns structural hypothesis formation, supervised execution, and divergence analysis across twelve investigative questions and two contrasting mobile databases. It measures structural reasoning through linking attributes, join paths, projected evidentiary units, and expert-validated ground truth.

  • The benchmark applies twelve investigative questions across increasing relational complexity, including implicit key recovery and multi-table reconstruction.
  • The evaluation targets structural correctness through Precision, Recall, and F1 while excluding natural-language explanations and intermediate reasoning traces.
  • The regular mmssms.db repository provides explicit identifiers and stable thread semantics, while Snapchat’s main.db contains ephemeral identifiers, polymorphic layouts, and indirect relationships.
  • Ground truth is independently constructed for each task by experienced mobile forensics analysts who identify correct joins, linking attributes, constraints, and complete expected row sets.
  • Scoring compares agent-generated and ground-truth queries after projection onto task-specific key columns, isolating agreement on evidentiary units rather than full rows.

1) Row Level Agreement Metrics:

Row-level agreement metrics quantify overlap between projected agent and ground-truth evidentiary sets, while explicit edge cases define how empty, missing, or spurious results are scored. The worked Snapchat example shows that executable retrieval can still include unrelated records and reduce precision.

  • Precision and Recall compare projected agent results with projected ground-truth sets, and F1 summarizes their balance as the harmonic mean.
  • False positives represent spurious relationships from incorrect joins or filtering, whereas false negatives represent valid relationships the agent failed to recover.
  • When both result sets are empty, agreement is structurally valid; empty agent results against non-empty ground truth yield zero Recall, and spurious agent results yield zero Precision.
  • For Snapchat Question 7, retrieving three correct records plus two unrelated rows produces Precision 0.60, Recall 1.00, and F1 0.75.

D. RQ1: How well does an agentic system recover structural

RQ1 tests whether an agent can infer relational structure from schema-visible evidence and recover intended records through executable queries. Structural recovery is reliable in regular SMS data but degrades in Snapchat as identifier ambiguity and relational distance increase.

  • RQ1 requires the agent to infer linking attributes, construct valid join paths, and produce executable queries without foreign keys, naming templates, domain semantics, or application-specific assumptions.
  • Structural evaluation combines row-level agreement with expert ground truth and Precision, Recall, and F1 over projected evidentiary units.
  • In mmssms.db, the agent reliably recovers intended structural links for all G1 and G2 tasks, with failures concentrated in G3 multi-hop or absence-based tasks.
  • Snapchat performance declines when cross-table inference is required because weak UUID propagation, inconsistent fields, and similar tables produce systematic key mismatches and more false positives and negatives.
  • Overall, structural-link recovery and execution stability depend strongly on schema regularity, identifier propagation, relational distance, and multi-table ambiguity.
  • Execution coherence isolates whether inferred structure becomes operational SQL, but successful execution does not establish ground-truth correctness.

F. RQ3: How robust is structural inference when execution

RQ3 evaluates robustness only after generated queries execute coherently, separating relational correctness from executability. Robustness remains high for simple tasks but declines as relational complexity and schema ambiguity increase, especially when multiple plausible join paths exist.

  • RQ3 measures robustness only among queries that satisfy execution coherence, comparing inferred links and join paths with expert ground truth.The evaluation uses row-level Precision, Recall, and F1 metrics.
  • Execution coherence means generated SQL runs without invalid joins, missing linkage attributes, or contradictory execution paths.
  • Simple single-table and direct-filtering tasks show consistently high robustness across both repositories.Successful execution almost always coincides with correct structural inference for these low-complexity tasks.
  • As relational complexity increases, mmssms.db retains ground-truth alignment through stable thread identifiers and uniform table geometry.
  • G3 multi-stage joins, implicit key recovery, and temporal alignment produce executable but potentially divergent join paths when several structural interpretations are valid.These divergences reflect schema ambiguity rather than execution instability.

2) Platform-Level Robustness Differences:

Robustness differs substantially between the regular Android database and Snapchat’s irregular database. Successful execution can conceal systematic structural errors, including wrong keys, substituted paths, irrelevant auxiliary relationships, and overconstrained results.

  • Platform-Level Robustness Differences: Android’s mmssms.db maintains relatively stable robustness as relational depth increases, while Snapchat’s main.db shows substantially lower robustness under comparable conditions.Explicit identifier propagation and limited polymorphism support Android; polymorphic tables, caches, and loosely coupled identifiers complicate Snapchat.
  • Observed Robustness Failure Modes: Four recurring failure modes are ambiguous key selection, join path substitution, auxiliary table interference, and structural overconstraint.
  • Observed Robustness Failure Modes: These failures can preserve executability while selecting incorrect relationships, introducing irrelevant joins, or reducing recall without validation rejection.
  • Sources of Robustness Failure: Schema ambiguity drives key-selection and auxiliary-table failures, weak identifier propagation contributes to path substitution, and prompting influences overconstraint.The evaluation protocol also contributes when alternative valid decompositions are judged against a single expert path.
  • Discussion: Successful execution alone is insufficient for evidential reliability in irregular databases, so multi-hop and temporal inferences require additional verification.

A. Structural Robustness and Evidential Risk

Executable queries can still encode incorrect evidential relationships when schema ambiguity is high. The paper therefore supports graduated trust: reliable provisionally under regular conditions, but requiring corroboration as irregularity, relational depth, or identifier ambiguity increases.

  • Structural Failure Modes: The four characterized failure modes occur without invalid joins, schema errors, or validation rejection, allowing executable queries to encode incorrect evidential relationships.This risk is especially pronounced under multi-stage joins, implicit key recovery, and cross-table temporal alignment.
  • Verification: Each failure mode has a corresponding verification action that can be performed within the same controlled, read-only, reproducible execution pipeline.These actions are intended to precede treating inferred relationships as evidentiary.
  • Graduated Trust Model: When schema regularity is high, identifier propagation is consistent, and relational distance is short, inferred structure can be treated as provisionally reliable subject to sanity checks.
  • Graduated Trust Model: As schema irregularity, relational depth, or identifier ambiguity increases, inferred structure should be treated as a hypothesis requiring corroboration.Recommended corroboration includes verification actions, manual schema inspection, or comparison with independent extraction tools.
  • Scope: The study isolates structural inference from end-to-end forensic accuracy, so its results characterize structural soundness rather than complete forensic performance.
  • Evaluation Constraints: Using one model and fixed deterministic decoding limits direct generalization across model families, parameter scales, prompting designs, and stochastic variance.The protocol prioritizes reproducibility and per-query attribution over confidence intervals or repeated-sampling statistics.

3) Benchmark Scope:

The benchmark probes structural inference using two repositories and twelve tasks under controlled conditions, while recognizing that expert-defined scoring and isolated datastore analysis constrain interpretation. Its model-agnostic framework is designed for expansion and comparative evaluation.

  • Benchmark Scope: The benchmark uses two real mobile repositories and twelve investigative tasks across three difficulty tiers.The repositories expose contrasting regular and irregular structural conditions rather than representing the full diversity of mobile schemas.
  • Ground Truth: Ground truth reflects one expert-validated relational interpretation per task, although alternative join paths may return evidentially equivalent records.
  • Ground Truth: Precision, Recall, and F1 use expert-defined key-column projections, so structurally different but evidentially equivalent outputs may receive lower scores than their forensic value warrants.This qualifies interpretation of low agreement scores in irregular schemas.
  • Scope Boundaries: Each datastore is analyzed independently, excluding cross-database structural correlation and multi-source reconstruction common in real forensic workflows.Results are also sensitive to prompt formulation, although prompts were held fixed for comparability.
  • Extensibility: The framework is model-agnostic, prompt-agnostic, and benchmark-extensible, enabling controlled comparative expansion without changing its core methodology.
  • Future Extensions: Future work includes comparisons across models and prompts, non-agentic Text-to-SQL baselines, and lightweight semantic or consistency checks.These extensions are intended to test generalization and constrain inference when structural cues are weak or inconsistent.
  • Benchmark Findings: The study identifies conditions under which schema-agnostic reasoning remains reliable and points at which it degrades by separating execution success from structural correctness.

IX. APPENDIX

The Appendix repository provides the complete materials needed to reproduce the benchmark, including code, schemas, prompts, traces, ground-truth SQL, structural definitions, and projected results.

  • Reproducibility: The public repository contains all code, schema snapshots, prompts, execution traces, and evaluation artefacts used in the study.
  • Reproducibility: Executable notebooks preserve the workflows for both mmssms.db and Snapchat main.db, including profiling, plan generation, SQL translation, validation, and execution.
  • Ground Truth: Ground-truth SQL, structural key definitions, and projected result sets are included for every task.
Loading 2608.21470v1…