Source-linked AI summary
Adoption Telemetry: Measuring Enterprise AI Adoption from Production Signals
Damon A. Young
TL;DR
Enterprise AI adoption is widely deployed yet often fails to produce measurable organizational value, while existing tools measure system quality or activity rather than changed work behavior. The paper introduces adoption telemetry, unifies evaluation gates, production signals, and change-management staging, and operationalizes the approach through NANTE. Its reference implementation separates healthy and failure-mode cohorts on synthetic data, but the thresholds remain proposed constructs requiring validation against real outcomes.
Problem
Existing measurement traditions do not instrument enterprise adoption as a population’s progression from access to changed behavior.
Method
The paper defines adoption telemetry and NANTE, mapping production event streams to explicit five-stage change predicates within a system combining evaluation gates, telemetry, and change management.
Results
The reference implementation distinguishes a healthy cohort from five characteristic adoption-failure modes on synthetic populations with known ground truth.
Takeaways & Limitations
Adoption telemetry makes stage progression, stall location, and failure signatures computable from production telemetry, enabling organizations to see where adoption is stopping.
Takeaways & Limitations
NANTE’s thresholds are proposed defaults co-developed with synthetic pathologies, so the evaluation establishes computability rather than correctness or real-world calibration.
Abstract
from arXiv · showhide
We introduce adoption telemetry: a method for measuring enterprise AI adoption by computing change-management stage-progression directly from production usage signals. We contribute (1) a framework unifying pre-deployment evaluation gates, production telemetry, and change-management staging into one instrumented system; (2) NANTE, a concrete five-stage operationalization with defined telemetry thresholds, published openly so they can be tested and disproven; and (3) an open-source reference implementation that distinguishes a healthy cohort from five characteristic adoption-failure modes on synthetic populations with known ground truth. We are explicit that the thresholds are proposed constructs requiring empirical validation against real outcomes -- a research agenda we outline -- not a calibrated model.
1 Introduction
Enterprise AI is widely deployed but frequently disappoints because organizations lack an instrument for measuring the organizational change that adoption requires. The paper introduces adoption telemetry to map production behavior to change-management stages and diagnose where adoption stalls.
- Motivation: 95% of generative-AI pilots delivered no measurable P&L impact, while 42% of companies abandoned most enterprise AI initiatives, illustrating widespread disappointment across distinct populations.The paper treats these figures as directional evidence because the underlying populations and methods differ.
- Motivation: Enterprise AI failure is attributed primarily to organizational factors such as workflow integration, sponsorship, skills, and behavior change rather than model capability.
- Measurement gap: Existing instruments measure system quality or user activity but do not measure progression from access to changed behavior.Observability tracks traces, latency, cost, and output quality, while usage dashboards count activity without interpreting behavioral depth.
- Adoption telemetry: Adoption telemetry continuously measures behavior change from production signals through an explicit model of change, mapping logged behavior to computable milestones and intervention locations.
- Contributions: The paper contributes a unified framework, the five-stage NANTE operationalization, and an open-source implementation that distinguishes healthy cohorts from five synthetic failure modes.NANTE publishes thresholds that can be tested and disproven.
- Scope: The reference implementation demonstrates computability, not validity, because its proposed thresholds are unvalidated constructs requiring calibration against real adoption outcomes.
2 Related Work
Related work measures adjacent objects: agent quality, activity, product engagement, or acceptance and change theory. The paper argues that these traditions remain fragmented and do not unify production telemetry with population-level change stages and mapped interventions.
- Agent evaluation: Agent evaluation and observability measure system capability, reliability, and operational properties rather than a population’s progression through organizational change.
- Enterprise usage analytics: Enterprise usage dashboards provide rich activity data and habit-formation tiers, but their measures remain activity gradients without change milestones, stall diagnoses, or mapped interventions.Microsoft’s active-user construct requires an intentional action in a supported application within the preceding 28-day window.
- Behavioral analytics: Behavioral workforce analytics can reveal a breadth-versus-depth gap: 82% of AI users sustained usage quarter over quarter, but only 2% reached consistently embedded workflow use.
- Product analytics: Product analytics contributes activation events, retention curves, and cohort methods, but primarily optimizes opted-in users for vendor retention and revenue.
- Change management: Acceptance and change-management research supplies theories and staged models, yet logged behavior is generally treated as a coarse outcome while the main instruments remain surveys.
- Structure of the gap: The four traditions are developed by separate communities, and the proposed unification adds production stage progression with explicit intervention mapping.
3 Adoption Telemetry
Adoption telemetry treats organizational adoption as population-level behavior change inferred from production signals under an explicit change model. It combines readiness gates, behavioral telemetry, and interpretation while defining boundaries for which systems the method can validly measure.
- Definition: Adoption telemetry measures a working population’s progression through organizational change using production signals such as invocations, sessions, turns, and task outcomes.
- Design properties: Its four required properties are computability, explicit stages and thresholds, population-level classification, and intervention mapping.
- Population scope: The framework classifies cohorts rather than individuals, preserving organizational focus and supporting deployment within privacy, works-council, and security constraints.
- System scope: The method fits assistants and human-in-the-loop agents but not autonomous agents, where successful delegation can reduce human-initiated invocation and mimic abandonment.
- Architecture: Architecturally, adoption telemetry unifies readiness evaluation gates, a production behavioral signal layer, and an interpretation layer that converts existing data into diagnosis.
- Operational question: Unlike dashboards that report usage and evaluation platforms that report whether systems work, adoption telemetry asks where a population stopped changing how work is done and what problem type that represents.
4 NANTE
NANTE operationalizes adoption telemetry as five computable stages over production event streams, then locates cohort stalls and flags characteristic failure signatures. Its thresholds and executive score are proposed diagnostic constructs, not validated measures.
- 4.1 The staged model.: NANTE defines five stages—Notice, Attempt, Navigate, Transform, and Embed—as computable predicates over a deployed agent’s event stream.The model tracks awareness, trial, recurring use, integrated multi-step work, and continuity of integration.
- 4.2 Telemetry signatures.: The telemetry substrate joins invocation events, session identifiers, turn counts, task outcomes, and a provisioned-user roster.The roster makes provisioned users with no events visible, allowing Notice to include people who never arrived.
- 4.2 Telemetry signatures.: Navigate requires recurring activity in at least 3 distinct weeks, while Transform proposes at least 6 active weeks, 60% multi-step sessions, and 55% task success.Embed additionally requires activity in at least 90% of weeks since first use; Embed therefore implies Transform, while the stages above Navigate split one qualifying pool by continuity.
- 4.3 Stall detection and honest ceilings.: A cohort’s stall point is the first stage boundary where progression falls below its proposed floor, with early boundaries using 90% healthy and 80% failing thresholds.For depth boundaries, NANTE deliberately sets an unreachable healthy floor and a 0.01 failing floor because progression beyond Navigate is near-universally rare.
- 4.4 Failure signatures.: Four flags characterize post-stall behavior: shallow plateau, champion dependency, usage regression, and low task success.Proposed examples include a divergence index above 0.3, Gini activity concentration above 0.5, late usage below 40% of early usage, and a Navigate-specific success-gate failure rule.
- 4.5 From diagnosis to intervention.: Each stall maps to a targeted response with an explicit not-this contrast, while the 0–100 ordinal score is reported only alongside richer diagnostics.NANTE warns that the scalar can assign similar scores to cohorts with opposite diagnoses and marks values unavailable below proposed observation-window or cohort-size floors.
5 Implementation
The reference implementation operationalizes NANTE over synthetic event streams and distinguishes a healthy cohort from five characteristic adoption-failure modes. Its results demonstrate computability and discrimination, not validity on real populations.
- Implementation: The open-source agent-adoption-kit implements event-schema ingestion, synthetic-population simulation, NANTE analytics, stall detection, failure flags, interventions, and composite scoring.The schema joins invocation events and task outcomes to a provisioned-user roster in SQLite.
- Evaluation: Six synthetic profiles—one healthy and five pathological—provide known ground truth for testing whether classifications and diagnoses can be derived mechanically from event streams.The evaluation tests computability with discrimination rather than calibrated real-world validity.
- Results: 98% or more of both headline cohorts reach recurring use, yet the healthy cohort reaches 73.7% Navigate and 25.1% Transform or Embed while the shallow-plateau cohort has 99.4% at Navigate and none beyond.The healthy cohort scores 60.5 with no stall; the shallow-plateau cohort scores 49.9, stalls at Navigate, and raises the shallow-plateau flag with divergence index 0.67 versus the proposed 0.30 threshold.
- Results: The same five-stage axis and usage data can yield opposite adoption diagnoses because NANTE distinguishes depth and integration from recurring activity.The comparison targets the gap between dashboards that report active users and diagnoses that locate adoption failure.
- Interpretation: The evaluation shows that stage progression, stall location, and failure signatures are computable and discriminative, but synthetic co-development of thresholds and pathologies prevents claims of correctness.The reported demonstration uses a single seed and proposed default thresholds; real-outcome validation remains necessary.
6 Limitations
The paper’s limitations concern construct validity, synthetic evidence, measurement proxies, scope boundaries, and strategic behavioral effects. NANTE should therefore be treated as a structured hypothesis about a cohort until validated against independent outcomes.
- Construct validity: NANTE’s thresholds are proposed defaults, not calibrated values, and the evaluation establishes computability rather than correctness because thresholds and pathologies were co-developed.Independent outcomes must test whether classifications correspond to sustained changed work practices and whether mapped interventions work differentially.
- Evidence fidelity: Synthetic cohorts isolate one pathology, whereas real populations may combine several failure modes amid seasonality, reorganizations, tool changes, and attrition.The simulator mirrors published aggregate patterns without claiming to reproduce real usage microstructure.
- Measurement schema: Turn count is an ambiguous proxy for workflow depth because long sessions can indicate either deep integration or user struggle.The success-rate gate helps separate these cases only when deployment-reported outcome labels exist, and those labels vary in quality.
- Scope boundaries: NANTE classifies one cohort’s relationship to one deployed capability, leaving cross-tool aggregation and portfolio-level adoption unresolved.The framework does not yet specify how to aggregate simultaneous adoption or substitution across multiple AI systems.
- Scope boundaries: Minimum floors make small teams and young rollouts unmeasurable, while thresholds may vary by function, industry, and task type.Single-organization findings therefore require cross-organizational baselines before transfer.
- Measurement effects: Measured adoption can become performative: users may generate invocations or long sessions without genuine reliance or integration.NANTE’s depth-weighted constructs are harder to game than activity counts but remain vulnerable when scores become targets rather than diagnostics.
7 Why Existing Systems Have Not Bridged This
Existing measurement traditions have the relevant pieces but lack incentives, expertise, or neutrality to unify system telemetry, behavioral analytics, and change-management staging. Professional and commercial separation therefore preserves the interpretation gap.
- Institutional positions: Engineering vendors prioritize system fidelity—traces, evaluations, and cost attribution—while population-level behavior change lies outside their mandate and organizational-psychology expertise.Their category and teams rarely include the disciplines needed to interpret adoption as change.
- Institutional positions: Platform vendors increasingly publish adoption benchmarks, but they cannot credibly provide neutral diagnoses of their own products.Their recent behavior indicates shallow deployment threatens expansion, yet product-specific constructs limit neutrality.
- Institutional positions: Product analytics measures employed populations sophisticatedly but serves vendor renewal objectives rather than the adopting organization’s transformation layer.Sponsorship, incentives, workflow authority, and related organizational factors are absent from its models.
- Institutional positions: Change-management firms possess staged-change theory and client relationships, but telemetry instruments sit outside their expertise and can partially substitute for licensed assessment work.Their established survey paradigm and delivery economics do not naturally reward computable instrumentation.
- Institutional positions: Consultancies build bespoke adoption dashboards inside billable transformation engagements, discouraging open, reusable instruments with contestable thresholds.Their delivery model rewards instrumentation as a per-client service rather than a productized system.
- Professional separation: The four traditions train different specialists and reward different skills, making fluency across evaluation, telemetry, behavioral analytics, and change theory rare.This professional separation compounds the commercial barriers to unification.
- Emerging convergence: The boundaries are beginning to erode as each tradition adopts pieces of behavioral or end-user measurement, but the explicit interpretation layer remains the missing bridge.The raw materials increasingly exist across traditions without yet forming a shared change model.
8 Research Agenda and Future Work
Future work centers on validating NANTE against real outcomes, replacing proxies, testing mixed and shifting behaviors, and adapting the instrument to heterogeneous production sources. The agenda also extends adoption telemetry toward human-agent disposition and complementary perception signals.
- Validity: Real deployments should be classified prospectively and compared with independently observed outcomes, including sustained changed work practices and differential intervention response.This is the primary test of whether NANTE’s classifications mean what they claim.
- Validity: Multi-organization telemetry can test whether the Navigate-to-Transform cliff is near-universal or should be replaced by percentile comparisons among comparable deployments.The current stall design hard-codes the near-universal-cliff belief.
- Engineering agenda: Near-term engineering gaps include direct task-type fields, depth-regression constructs, mixed-pathology simulation, and threshold-sensitivity analysis.These additions target proxy limitations and the narrowness of the current synthetic evaluation.
- Deployment: Adapters must consume actual platform metrics, agent analytics, and observability exports because production sources differ in which NANTE stages they can populate.The framework should compute supported stages and report unsupported outputs as unavailable rather than infer them.
- Human-agent systems: Human-in-the-loop agents require disposition telemetry that records what happened to agent work, not only whether the agent was invoked or whether nominal outcomes succeeded.The proposed specification concerns review, revision, acceptance, escalation, and related work outcomes.
- Human-agent systems: Override rate and override accuracy provide established behavioral constructs for measuring calibrated trust, but production joins between agent outputs and later human actions remain operationally difficult.The paper identifies this as a specification requiring event linkage and deployment-specific predicates.
- Human-agent systems: The current schema cannot detect four disposition-related failure signatures, including systematic override despite good output and deep sessions.Usage, depth, and nominal success can therefore coexist with trust failure or other misdiagnoses.
- Complementary evidence: Perception instruments may precede behavioral telemetry, but their lead time and disagreement patterns require empirical study.Confidence hedges and permission-seeking are proposed as candidate precursors to within-team behavioral divergence.
9 Conclusion
The paper frames adoption telemetry as a first instrument for measuring where organizational behavior change stalls, using production events and an explicit staged model. Its evidence demonstrates computability and discrimination on synthetic populations, while real-world validity remains untested.
- Adoption telemetry measures behavior change from production signals through an explicit model, addressing the absence of an instrument that reports where populations stop changing work practices.
- NANTE expresses five change milestones as defined predicates over event streams and pairs diagnosable stalls with response classes.
- Given only events and a roster, the reference implementation separates a healthy cohort from five characteristic adoption-failure modes, including a stalled cohort indistinguishable by activity metrics.
- The demonstrated evidence is computability with discrimination on synthetic populations, not validity on real populations.
- The instrument is offered openly with exposed assumptions so it can be disputed, calibrated, and improved by communities whose expertise it unifies.