Who Is Behind the Harness? Fingerprinting LLMs through Agentic Behavior

• Harnesses mediate visible responses and subsequent actions through controller context, tools, and feedback, so direct-interface fingerprints may not transfer unchanged.Source: §1 · p. 1
• The approach addresses the need for repeatable probes, comparable representations of stochastic executions, and a lightweight identifier when enrollment is small.Source: §1 · p. 2
• Under the two harnesses, LIDAR reaches 95.13% and 88.36% accuracy, with MRR 0.9736 and 0.9336, and outperforms four fingerprinting and API-auditing baselines.Source: §1 · p. 2
• Three paired coding probes hold the repair objective and repository context fixed while varying conditions that expose verification, recovery, or conflict-handling decisions.Source: §1 · p. 2
• LIDAR combines instance-level decisions with recurring distribution-level tendencies in trajectories and compares them with clean, harness-specific references using a lightweight probabilistic identifier.Source: §1 · p. 2

• The verifier observes CLI-boundary messages, controller events, actions, tool results, repository changes, and final responses, but cannot inspect weights, logits, provider logs, or routing state.Source: §2 · p. 2
• A provider, aggregator, relay, or compromised router may remap the claimed identity or transform remote instructions, context, messages, actions, or presentation, but cannot alter verifier-controlled host components.Source: §2 · p. 3
• For each claimed identity and harness, the verifier freezes clean fingerprint samples collected under a declared protocol, then scores later queries against registered references.Source: §2 · p. 3
• The current closed-set decision always maps a query to a registered label, so an unknown model may be assigned one of those labels.Source: §2 · p. 3

3 Related Work

• The review frames fingerprinting by evidence access, claim scope, watermarking, and interactive-agent behavior.It distinguishes LIDAR’s remote-model verification focus from prior work on forensics, capability, safety, or procedural characterization.Source: §3 · p. 3, §3.3 · p. 4

3.1 LLM Fingerprinting

• White-box fingerprints use model internals or local references, while black-box methods use queries and returned text or tokens; access level alone does not determine accuracy.Examples include Instructional Fingerprinting [34], LLMPrint [14], TRAP [13], LLMmap [26], ZeroPrint [30], CoTSRF [28], and One Token [3].Source: §3.1 · p. 3
• Model identification selects the likeliest registered identity, but does not establish weight sharing or authenticate the full remote service path.A service audit instead asks whether a live endpoint remains consistent with a claim or reference.Source: §3.1 · p. 3

3.2 Watermarking for LLMs and Agents

• Watermarking inserts owner-chosen evidence, whereas LIDAR leaves the service unchanged and matches naturally occurring coding-agent behavior to black-box enrollment references.The two approaches are complementary: one tests for an inserted signal; the other identifies the registered model that best matches behavior.Source: §3.2 · pp. 3–4

3.3 Agent Behavior and Trajectory Analysis

• Agent trajectories provide richer evidence than single prompt–response outputs, but harness and environment effects can be mistaken for model identity.ProcGrep reports substantial same-model divergence across harnesses [23]; LIDAR maps native events from two CLI harnesses to shared semantics and excludes raw harness labels from the fingerprint.Source: §3.3 · p. 4

4 Overview

• Six trajectories from the A/B variants of VERIFY, RECOVER, and RESOLVE form each fingerprint sample, alongside recorded native events, repository outcomes, and controlled local checks.The verifier takes a claimed identity, a coding-agent CLI, and registered model identities; the behavioral semantics are shared across supported harnesses.Source: §4 · p. 4
• A common representation maps harness-native records to shared action and outcome semantics, retaining per-variant decision structure and sample-wide behavioral distributions.These are the instance-level and distribution-level feature sets.Source: §4 · p. 4
• Repeated clean samples establish frozen feature processing and one count-based reference per registered model and harness; the later query is scored against every reference.The highest-scoring identity determines the closed-set verdict, without updating enrollment-fitted quantities.Source: §4 · p. 4
• Provider-side routing, instructions, and output wrappers can affect query fingerprints, while CLI records capture local execution events and outcomes on the user’s host.The overview’s threat-model callout distinguishes remote influence from locally recorded execution.Source: §4 · p. 4

5.3 Model Identification

• LIDAR compares query behavior with clean, harness-specific model references using a lightweight probabilistic identifier designed for limited enrollment data.The score uses enrollment frequencies with a symmetric half-count; the evaluated decision selects the registered model with the largest query-to-reference score.Source: §5 · p. 4, §5.3 · p. 8

5 Design

• The design shifts fingerprint evidence from final text alone to coding-agent execution, where actions, feedback responses, and repository outcomes are observable.The design goals also require consistent feature meaning across supported harnesses and identification with only a small number of clean samples.Source: §5 · p. 4, §5.1 · p. 5

5.1 Probe Design

• Three paired coding probes expose post-edit checking, recovery from temporary test failures, and resolution of specification–test conflicts.Each pair holds the repair goal fixed while changing one condition; the six variant trajectories together form one fingerprint sample.Source: §5.1 · pp. 5–6

5.2 Feature Extraction

• The feature representation combines detailed per-variant decisions and action paths with summaries of recurring behavior across the probe suite.The instance-level representation has 756 fields; distribution-level means and standard deviations over 101 descriptors produce 202 fields, for 958 combined fields.Source: §5.2 · pp. 6–7

6.1.1 Paired Probe Suite

• Each of three probe pairs holds the repair goal fixed while variant B introduces one controlled change; both variants together yield six trajectories per fingerprint sample.Each variant runs on an isolated fixture copy.Source: §6.1.1 · p. 9

6 Evaluation

• The study evaluates identification across 36 LLMs from seven families and tests each through OpenCode and mini-swe-agent.The same probe pairs and behavioral definitions are applied to both harnesses, with harness-native event parsing adapted to each implementation.Source: §6.1.2 · p. 9

6.1.2 Models and Harnesses

• Enrollment and query are collected separately, with six complete fingerprint samples per model–harness combination and 36 trajectories in each set.Each sample contains all six probe-variant trajectories, collected with a distinct seed.Source: §6.1.2 · p. 9

6.1.3 Baselines

• LIDAR is compared with LLMmap, MET, ZeroPrint, and One Token while preserving each baseline’s published probe contract, feature extractor, statistic, and ranking rule.Baselines are fitted and evaluated separately for each harness; sampling units are preserved with balanced enrollment and query allocation.Source: §6.1.3 · pp. 9–10

6.1.4 Evaluation Metrics

• Top-1 measures exact identification, while Top-3 and MRR measure the rank of the true model, with MRR rewarding higher ranks more.Scalar metrics are computed separately for each harness.Source: §6.1.4 · p. 10

6.2 Overall Identification Performance

• LIDAR has the highest mean in all six comparisons; under mini-swe-agent it exceeds the strongest baselines by 35.96 Top-1, 23.08 Top-3, and 28.09 MRR points.The respective strongest baselines are MET, LLMmap, and MET.Source: §6.2 · p. 10
• Under OpenCode, LIDAR improves on MET by 1.69 Top-1, 2.64 Top-3, and 1.76 MRR percentage points.MET is the strongest baseline for all three OpenCode metrics.Source: §6.2 · p. 10
• Baseline rankings change across harnesses, and the comparison measures transfer to 36-way agent identification rather than each baseline’s native task.The evaluation is performed within each harness, without pooled training or cross-harness transfer; repeated configurations test whether results persist across data assignments.Source: §6.2 · p. 10

6.3 Ablation Study

• The ablation uses fixed enrollment/query roles and a locked query set; the full method retains 958 fields, all three probe pairs and their variants, and Jeffreys.The two harnesses are fitted separately, then predictions are pooled for reporting.Source: §6.3 · pp. 10–11
• Combining instance- and distribution-level features reaches 93.52% Top-1 and 96.31% MRR, versus 89.81% Top-1 for distribution-level features alone and 75.00% for instance-level features alone.Keeping instance-level fields with only the distribution mean or standard deviation reaches 91.90% and 90.97% Top-1, respectively.Source: Table 5 · p. 11, §6.3 · p. 11
• Removing VERIFY, RECOVER, or RESOLVE decreases Top-1 by 5.32, 5.09, or 8.33 percentage points, respectively; retaining only A or only B yields 84.26% or 85.88% Top-1.RESOLVE removal causes the largest loss, and neither side alone retains all evidence from the ordinary condition and controlled change.Source: §6.3 · p. 11
• Jeffreys has the highest query Top-1 and MRR point estimates and remains competitive with limited enrollment using an auditable count-based estimator.The identifier comparison uses the full feature representation and all A/B variants; Figure 3 compares alternative classifiers with Jeffreys.Source: §6.3 · p. 11, Figure 3 · p. 11

6.4 Robustness to Provider-Side Manipulation

• A1 asks models to avoid or alter explicit identity claims, while A2 requires a provider-specified notice before natural-language responses; coding tasks and tools remain unchanged.Enrollment stays clean, and only the condition-specific system instruction changes across query conditions.Source: §6.4 · p. 12
• Under A1, LIDAR’s largest Top-1 and MRR drops are 11.11 and 5.56 points; under A2, both harnesses retain Clean accuracy and MRR exactly.Every baseline loses accuracy in at least one manipulated condition, with Top-1 drops reaching 33.33 points under A1 and 83.33 under A2.Source: §6.4 · p. 12, Table 6 · p. 12
• LIDAR aggregates paired-trajectory tool use, action order, verification, recovery, and repository outcomes, and its few A1 errors involve adjacent Qwen3.6-Plus and Qwen3.7-Plus identities.The reported contrast supports execution-level behavioral evidence as more resilient to these surface manipulations than returned-text or token-statistic evidence.Source: §6.4 · p. 12

6.5 What Makes an Agentic Fingerprint?

• The analysis sums signed log-likelihood-margin contributions by mechanism, then groups them into trajectory organization, command/action use, response form, probe-pair decisions and outcomes, and other task behavior.The frozen identifier is fitted on clean enrollment data; query data do not affect its feature handling or model probabilities.Source: §6.5 · pp. 12–13
• Under the same task, provider, harness, and seed, three models produce valid one-line fixes in 5, 8, and 14 steps.Their trajectories differ in inspection, reproduction, editing, testing, and review.Source: Figure 4 · p. 12, §6.5 · p. 13
• Trajectory organization contributes 50.6–53.6% of absolute margin evidence under mini-swe-agent and 67.8–70.5% under OpenCode.Source: Figure 5 · p. 13, §6.5 · p. 13
• Unique-transition fraction ranks among top-20% contributors for 50.9–69.4% of queries and covers 97.2% of models; five other mechanisms cover at least 91.7%.Source: Figure 5 · p. 13, §6.5 · p. 13
• The largest mechanism contributes a median 4.6%/4.4% under mini-swe-agent/OpenCode, while reaching 80% requires 45/43 mechanisms.The analysis concludes that identification draws on many signals rather than any single action or response.Source: Figure 5 · p. 13, §6.5 · p. 13

7 Conclusion

• LIDAR combines paired coding probes, instance- and distribution-level trajectory evidence, and a lightweight probabilistic scorer to audit LLM identity in coding-agent services.Source: §7 · p. 13
• Across 36 models, seven families, and two agent harnesses, LIDAR achieves the strongest mean result in every reported metric–harness comparison.Source: §7 · p. 13
• Ablation, robustness, and feature analyses show identity evidence distributed across execution behavior rather than isolated outputs or actions.The paper presents active behavioral probing as a practical basis for auditing remotely served model identity.Source: §7 · p. 13

A Evaluated Model Panel

• Table 7 enumerates the model identities used in the evaluation.Source: A Evaluated Model Panel · p. 17, Table 7 · p. 17
• The table lists the complete set of model identities used in the evaluation.Source: A Evaluated Model Panel · p. 17
• The evaluated LLMs are grouped by family in Table 7.Source: Table 7 · p. 17