Source-linked AI summary
Harbormaster: Evidence-Gated, Replay-Safe Maritime Anomaly Detection on AWS
Arun Sharma
TL;DR
Harbormaster addresses how to make alerts from untrusted AIS reports attributable, reviewable, and recoverable when messages repeat and models change. It combines physics-first checks, replay-safe state projection, and evidence-gated model promotion. Synthetic drills found that shadowing caught a skewed candidate that passed holdout, while canary checks reverted a planted regression.
Problem
Harbormaster asks how a cloud pipeline can turn untrusted AIS reports into alerts that remain attributable, reviewable, and recoverable despite repeated messages and model changes.
Source: §1 · p. 1Method
The system runs physics checks before learned models, projects PostgreSQL changes idempotently into DynamoDB, and requires evidence-producing gates before a candidate receives traffic.
Source: §1 · pp. 1–2Results
In synthetic drills, shadowing caught a skewed candidate that passed holdout, and the canary reverted a planted regression.
Source: §10 · p. 14Takeaways & Limitations
The design keeps guarantees small and checkable, with each reported result scoped to where it ran.
Source: §10 · p. 14AWS evidence remains bounded: no trained model served, the live-endpoint canary was traffic-inert, and drift monitoring was not reported on AWS.
Source: §9 · p. 13
Abstract
from arXiv · showhide
Ships broadcast their positions through the Automatic Identification System (AIS), and those reports can be false or missing. An operator who acts on an anomaly alert needs that alert to be attributable, reviewable, and recoverable after a failure. This paper describes Harbormaster, a production-shaped system on AWS that follows three rules. Physics checks run on every report before any learned model runs. The DynamoDB read store is an idempotent projection of PostgreSQL, and a guard on the log sequence number (LSN) of each change protects every write to it. A candidate model must pass a holdout gate and a shadow comparison before a canary gives it traffic, and a burn-rate check guards each canary step. The paper proves that replaying any prefix or suffix of the change log leaves each projected key at the value of its highest applied LSN. It also notes effects this result does not cover, such as repeated cache invalidations and repeated audit rows. In a bounded AWS window, a one-hour soak returned 35,999 HTTP 200 responses to 36,000 requests, with a 95th-percentile (p95) client latency of 142.751 ms. In a separate bounded AWS run of 900 s, the stream path received a burst of 400 records/s, and its consumer lag later drained to zero. Every number in the paper carries a label that says where it was measured, and the paper lists the parts of the design that were never built.
1 Introduction
Harbormaster asks how to make alerts from untrusted AIS reports attributable, reviewable, and recoverable despite message repeats and model changes. It combines physics-first scoring, replay-safe state projection, and evidence-gated model promotion, with results and reruns explicitly scoped.
1 Introduction
- Harbormaster targets alerts that operators can act on safely by making them attributable, reviewable, and recoverable after failures, duplicate messages, or model changes.Two read-only historical cases illustrate impossible-speed reports and prolonged silence; neither was scored by Harbormaster.Source: §1 · p. 1
- The system runs a fast physics check before any learned model, projects PostgreSQL changes idempotently into DynamoDB, and gates candidate models before traffic.Its AWS components ran only in attended, time-boxed windows, and that environment is no longer running.Source: §1 · p. 1
- Each reported number carries a scope label, and only the public test suite, local CDC tests, and public drill scripts can be rerun by readers.Appendix A states this rerun boundary.Source: §1 · p. 2
- The paper contributes a physics-first path, an LSN-guarded replay-safe CDC path, evidence-gated model promotion, endpoint scaling analysis, and a scoped evaluation with unbuilt elements identified.Synthetic drills test holdout, shadow, and canary steps; two metric-dimension bugs each disabled an endpoint scaling path without an error.Source: §1 · p. 2
2 Related Work
Harbormaster draws on established maritime anomaly, trajectory-gap, cloud-processing, stream-processing, and ML-operations work. Its distinction is an operational serving system centered on evidence gates and replay-safe state updates, rather than a new detection method.
Maritime anomaly detection and trajectory gaps
- Harbormaster uses space-time prisms and trajectory-gap detection from prior work, including Sharma et al.’s STAGD, within a serving system and proposes no new detection method.It also builds on route-learning ideas such as TREAD and uses Douglas-Peucker [15] and HDBSCAN for lane building.Source: §2 · p. 2
Learned spatial models and physical knowledge
- Prior work connects learned spatial models to domain knowledge across prediction and generation tasks, while Harbormaster focuses on serving gates and replay-safe updates around model artifacts.Its reported AWS runs use a numpy surrogate for the data term, and diffusion training remains design only.Source: §2 · p. 2
Cloud geospatial analysis and parallel mining
- Cloud geospatial systems and parallel-mining work address infrastructure and spatial analysis, whereas Harbormaster studies operational correctness for a narrower maritime serving path.The path centers on replay handling, artifact gates, and review records.Source: §2 · p. 2
Stream processing and change data capture
Machine learning operations and gating
Drift and reward hacking
- Harbormaster pairs population-stability and Kolmogorov-Smirnov input checks, and probes one narrow reward-hacking form: higher preference reward alongside less respect for physics.The related work frames these choices against dataset shift, concept drift, and reward hacking.Source: §2 · p. 3
3 System Overview
Harbormaster separates off-cloud model training from AWS serving, with deployed components and design-only workflows explicitly distinguished. The AWS serving path used a synthetic replay fixture and surrogate data term, while several planned integrations remained unbuilt or local-only.
3 System Overview
- Training belongs on an off-cloud research GPU cluster and remained design only; no trained checkpoint crossed into AWS serving, where the endpoint served a labeled stand-in image.The figure marks deployed components and flows with solid styling and components or flows lacking AWS evidence with dashed styling.Source: §3 · p. 3, Figure 1 · p. 4
- The design has five triggered workflows, but Workflow 5’s job service, dispatch worker, and result collector were not built.The ingestor replayed a synthetic fixture of 2,709 events for 8 vessels over 6 hours, with no real-time AIS source connected.Source: §3 · p. 3
- Provisioned Kinesis carries vessel reports to Managed Flink, which keeps five fixes per MMSI, applies a physics gate, and forwards passing reports to DynamoDB and the scorer.Kafka carries Debezium change events; AWS windows used MSK Serverless.Source: §3 · p. 3
- The scorer is a FastAPI service on ECS Fargate reached through an IAM-authorized API Gateway and VPC Link, with no standing load balancer.The reviewed Terraform runs it on Fargate Spot capacity.Source: §3 · p. 3
- On AWS, the scorer’s data term came from a numpy surrogate because its ECS task set no endpoint, and the review queue fell back to process memory while the console ran locally.The SageMaker endpoint served a labeled demo stand-in and scaled from zero to one instance and back.Source: §3 · p. 4, Figure 1 · p. 4
- The batch design writes lane tables to Iceberg, but publication to the serving reference is assumed integration and the scorer reads a small synthetic lane reference.A bounded AWS window succeeded on a valid backfill and rejected invalid data before any write.Source: §3 · p. 4
4 Physics-First Scoring
The scoring path uses closed-form rules grounded in reported fixes and reference data, with physics checks preceding heavier model use. Its gap detector uses prism feasibility and a blended severity, while the stream gate can delay or drop some fast jumps.
Physics-First Scoring
- The scorer uses closed-form rules over reported fixes and reference data, making each score recomputable from constants in the public code.Physics is its first source of evidence.Source: §4 · p. 4
Space-time prism feasibility
- A feasible pair has dAB ≤ L and an elliptical prism footprint; when dAB > L, no path fits under the speed cap.The reach length is L = vmax(tB − tA), with vmax set to 25 knots.Source: Figure 2 · p. 5, §4 · p. 5
Required speed and the stream gate
- A jump requiring more than vmax/0.3, about 83 knots, is dropped at arrival; jumps from about 83 to 300 knots can be scored only when a later passing report preserves both fixes.If the next four reports also fail the gate or never arrive, the jump goes unscored; above 12 vmax, or 300 knots, later requests carrying both fixes receive HTTP 422.Source: §4 · p. 5
Space-time prism feasibility
- The prism test for the latest pair matches the speed test up to tolerance, so it does not affect the single-vessel fused score; prism geometry instead helps merge candidate gaps.The gap detector merges candidates whose prism bounding rectangles overlap.Source: §4 · p. 5
Gap severity
- A silence longer than 600 s opens a candidate gap, and the detector merges overlapping prisms before scoring the earliest gap in each group.It keeps the group with the largest abnormal gap measure (AGM), which blends kinematic implausibility with a data term at w = 0.6; AWS used a numpy surrogate for that term.Source: §4 · pp. 5–6
- The gap-duration term is clipped, and the gap score combines duration with AGM.The target design replaces the AWS numpy surrogate for Pdata with a diffusion model trained off-cloud.Source: §4 · p. 6
Corridor severity and fusion
- The corridor detector measures distance to the nearest lane segment, while watchlist or sanctions hits add fixed-severity reasons; noisy-OR fuses reason severities.A strong reason can by itself raise the score close to 1, and analyst labels are stored without overwriting the automated score.Source: §4 · p. 6
The deterministic planner
- The deterministic planner always runs the corridor detector, adds prism, speed, and validation with one prior fix, and adds the gap detector with at least three fixes.The validation gate rejects anchors implying more than 1.05 vmax, but the single-vessel path passes no anchors, so it has no effect there.Source: §4 · p. 6
5 Replay-Safe Change Data Capture
Harbormaster projects committed PostgreSQL changes into DynamoDB with an LSN guard that makes the per-key projection converge under replay. The guarantee depends on log retention and worker progress, and it does not extend to every sink or cross-key atomicity.
CDC path
- PostgreSQL changes flow through Debezium and Kafka to a DynamoDB read store, while the worker also invalidates Redis and appends Iceberg audit rows.The bounded AWS path used MSK Serverless; the local path used Strimzi on kind.Source: §5 · p. 6, Figure 3 · p. 7
Commit protocol
- The worker flushes every sink before committing Kafka offsets, so a transient sink failure leaves the batch uncommitted for redelivery.Kafka tombstones are skipped, while source deletes become soft-delete items written at the delete’s LSN.Source: §5 · p. 6
LSN guard
- The worker writes an item only when its incoming LSN is newer than the stored LSN or the item has no stored LSN.Snapshot rows use LSN 0 and apply only to empty keys, so streamed changes outrank them.Source: §5 · p. 7
Replay convergence
- Replaying any finite delivery sequence leaves each projected key at the value with its highest delivered LSN, under the proposition’s stated assumptions.The assumptions include atomic check-and-write, whole-item replacement, and a valid key for each delivered event.Source: §5 · p. 7
Guarantee boundaries
- Recovery depends on worker progress and complete log retention; the per-key guard does not prevent repeated cache or audit effects or cross-key partial visibility.If the replication slot is lost, a re-snapshot at LSN 0 may not repair keys that already hold streamed LSNs.Source: §5 · p. 7
Slot-lag monitoring
- The slot-lag alarm fires at 200 MiB after three consecutive one-minute periods at or above the threshold.A heartbeat upsert handles lag accumulating while captured tables are quiet.Source: §5 · p. 8
6 Evidence-Gated Promotion
Harbormaster gates candidate models through holdout, optional reward-hacking, shadow, and canary checks, with rejection stopping promotion. These gates and checks are exercised in synthetic drills; the live endpoint moved no AWS canary traffic.
Promotion ladder
- A candidate must pass holdout, shadow, and weighted canary checks; the reward-hacking probe runs only when a probe result is supplied.Every gate has a reject path, and a rejected candidate does not reach the canary.Source: §6 · p. 8, Figure 4 · p. 9
Holdout gate
- The holdout gate requires AUC ≥ 0.85, mean CRPS ≤ 1.0, and calibration ratio in [0.8, 1.2], with every threshold required to pass.A strong AUC cannot compensate for poor calibration.Source: §6 · p. 8
Reward-hacking probe
- The reward-hacking probe blocks candidates whose reward rises while physical consistency worsens, including cases with unchanged violation counts but greater severity.It compares a preference-tuned candidate with the baseline on held-out data.Source: §6 · p. 8
Shadow comparison
- The shadow step passes when the mean absolute score difference across the same N inputs is at most the caller-set limit, set to 0.05 in the drills.Source: §6 · p. 8
Canary and burn rate
- Canary weights advance through 5, 25, 50, and 100; a page at any weight returns the full weight to the champion, while a warning does not revert.The paging tiers require burn rates above 14.4 over 1 h and 5 min or above 6 over 6 h and 30 min; the warning tier is above 3 over 1 day and 2 h.Source: §6 · p. 9
Drift and operational scope
- Three post-promotion drift detectors cover input shift, calibration, and proxy-based concept drift, but the live endpoint’s single variant made the canary traffic-inert.No model retraining ran, so shadow and canary findings came from synthetic drills.Source: §6 · p. 9
7 Scale-to-Zero Serving
Harbormaster uses separate tracking and wake policies to scale an asynchronous endpoint between zero and one instance. AWS checks exposed two metric-dimension bugs, while a bounded drill demonstrated waking the endpoint and returning it to zero.
7 Scale-to-Zero Serving
- Target tracking steers backlog per instance toward 5 and scales in to zero when backlog empties; the wake policy sets capacity to one after two 60 s periods with backlog-without-capacity at least 1.Because the target caps capacity at one, target tracking in practice only scales in.Source: §7 · p. 9
- The target-tracking metric lacked EndpointName, so its alarms received no data and scale-in to zero could not fire.The code-only audit found this dimension bug before any live symptom appeared; the fix added the dimension.Source: §7 · p. 10
- A live check found the wake alarm's extra VariantName dimension prevented datapoints for over 15 minutes while backlog waited; removing it fixed the bug.This was P26, distinct from the 19 audit findings, and neither bug raised an error or left resources absent.Source: §7 · p. 10
- A bounded AWS drill woke the endpoint from zero, returned a score, and later returned capacity to zero; the endpoint remained single-variant.The endpoint served a labeled demo stand-in image, and AWS rejected a second variant.Source: §7 · p. 10
- The separate KEDA on EKS scale-from-zero path ran once in a bounded AWS window, was then torn down, and did not replace ECS as the front door.The design's asynchronous SageMaker endpoint can queue requests while it has no instance and serve them after waking.Source: §7 · pp. 9–10
8 Evaluation
The evaluation reports results with scope labels and covers stream load, serving capacity, change-data-capture behavior, and synthetic promotion-gate drills. The runs show bounded operational behaviors, but do not measure detection accuracy.
8 Evaluation
- The evaluation reports scope labels for bounded AWS windows, local runs, synthetic drills, and public-repository results; no experiment measures detection accuracy.The AWS evidence is private; the public suite, local CDC tests, and listed public drills are rerunnable, while the soak is restated in the public README but cannot be rerun.Source: §8 · p. 10, Table 1 · p. 11
8.1 Stream run under load
- Source and key-by backpressure rose above 0.1 at 79.5 s, peaked at 0.998 about 160 s after load start, fell below 0.1 around 694 s, and ended at 0.0.Kinesis iterator age peaked at 139,000 ms in a single one-minute bucket, while Flink millisBehindLatest held that maximum for four consecutive buckets; both drained to 0.Source: §8.1 · p. 10
8.2 One-hour soak through the ECS front door
- 35,999 of 36,000 requests returned HTTP 200 in the 3,600 s soak, with a 0.99997 success ratio against a 0.999 gate and client p95 latency of 142.751 ms.The prewarmed soak repeated one fixed request whose history held one fix; it measured serving capacity, not detection quality or cold-start behavior.Source: §8.2 · pp. 11–12
8.3 Change data capture
- A bounded AWS CDC window passed five drills; replaying the topic left three DynamoDB items and their content hash unchanged after the LSN guard rejected three events.A Debezium heartbeat query reduced slot lag from 1,006,637,152 bytes to 4,328 bytes.Source: §8.3 · p. 12
- In a local synthetic duplicate drill, an LSN guard kept severity at 0.95 after a stale event arrived and rejected all five redeliveries after a crash.The unguarded sink regressed severity from 0.95 to 0.9; after redelivery, the guarded state hash matched the baseline.Source: §8.3 · pp. 12–13
8.4 Promotion gate drills
- A synthetic skew drill's holdout passed a candidate with an online feature-encoding bug, but shadow comparison failed it at mean absolute difference 0.6056 against a 0.05 limit.The holdout labels and observations were constructed from the candidate's own scores, so those holdout values did not measure a model.Source: §8.4 · p. 13
- In a synthetic canary drill, a candidate advanced to weight 5, met a planted regression at weight 25, and rolled back to the champion.The drill's injected burn check reported a page at weight 25; it tested the revert path, not Equation (19).Source: §8.4 · p. 13
9 Lessons and Limitations
The evidence shows why live behavior checks matter: quiet alarms and successful HTTP responses can coexist with mechanisms that do not work. The paper also bounds its claims to tested paths and artifacts.
What is not built
- The paper does not claim a deployed candidate model, reinforcement-learning fine-tune, AWS drift monitoring, or a traffic-moving live canary.The AWS endpoint served a labeled demo stand-in; the stream used synthetic load, and the ingestor replayed only a synthetic fixture.Source: §9 · p. 13
- The Workflow 5 job service, dispatch worker, and result collector were not built, while the lane publisher remained assumed integration and the review console ran only locally.The Flink job also lacked event-time windows and watermarks, so its gate could score fast jumps late or drop them.Source: §9 · p. 13
Lessons from the evidence
- The P25 and P26 alarms were present and quiet even though scale-in and wake-up could never fire, showing that a live test of the resource’s behavior is the useful check.Source: §9 · p. 13
- One AWS streaming window returned HTTP 200 responses even though Flink’s one-fix history prevented the planner from adding the gap detector.Keeping five fixes clears the gaps step’s three-fix threshold; a code review also found the AWS task definition omitted the database credential, leaving the review queue to fall back to process memory.Source: §9 · p. 13
- An AWS replication slot held 1,006,637,152 bytes while the connector reported RUNNING, showing why slot lag needs its own alarm and quiet tables need a heartbeat.A replication slot holds disk until a consumer drains it.Source: §9 · p. 13
- 0.927 was the burst-window maximum, but the raw capture showed a run maximum of 0.998 after the burst, illustrating how summaries can misstate their artifact.The paper takes numbers from the raw artifact and labels their scope.Source: §9 · p. 13
10 Conclusion
Harbormaster keeps its guarantees small and checkable, with closed-form scoring, replay-safe change handling, and evidence-gated model promotion. Synthetic drills caught a skewed candidate at shadow comparison and reverted a planted canary regression, while measured results remain scoped and the next steps remain unbuilt.
Conclusion
- Harbormaster makes each guarantee small and checkable so alerts remain attributable, reviewable, and recoverable.Closed-form rules produce scores, and the paper names which jumps the stream gate scores late or drops.Source: §10 · p. 14
- In synthetic drills, shadow comparison caught a skewed candidate that passed holdout, and canary reverted a planted regression.The promotion ladder turns model trust into gates that produce evidence.Source: §10 · p. 14
- A trained model on the endpoint, a traffic-moving canary, and CDC outside attended windows remain next steps.The measured results are described as small and scoped, with each one naming where it ran.Source: §10 · p. 14
A Reproducibility and Disclosure Boundary
The paper separates reproducible public artifacts from private AWS evidence and distinguishes the integrated target design from what was built. It also supplies algorithmic details, a replay proof, and an input-drift rule.
A Reproducibility and Disclosure Boundary
- The public snapshot reported 821 passed and 12 skipped tests, while AWS evidence and several other records remain private; the paper makes no claims of real-case accuracy or trained-model traffic.The public repository includes code, tests, Terraform, a synthetic replay fixture, load-test scripts, and drill scripts; the skipped tests require live PostgreSQL, and coverage was measured locally.Source: A Reproducibility and Disclosure… · p. 17
B The Full System Board
- Figure 6 is an integrated target-design map, not a deployed-system diagram; its Workflow 5 jobs service, dispatch worker, and result collector are unbuilt.The AIS Provider box is a placeholder, and only the synthetic replay fixture ran.Source: Figure 6 · p. 18
C Method Details
- For one report, Algorithm 1 rejects tracks requiring speed above 12 vmax, selects detectors by history length, then returns scores, reasons, and a trace identifier.It runs the gaps detector only when n ≥ 3 and queues review according to Equation (11).Source: C Method Details · p. 19
C.1 Proof of the replay proposition
- The replay proof shows each key retains the value associated with its highest delivered LSN, provided the stated delivery and write assumptions hold.Replaying a prefix or suffix adds no new event to the delivered set, so the projected state remains unchanged.Source: §C.1 · p. 19
C.2 The input drift check
- The input drift check compares reference and current feature samples using PSI and a Kolmogorov-distribution p-value, flagging PSI ≥ 0.25 or PSI ≥ 0.1 with p < 0.05.It uses at most 10 equal-frequency bins whose edges are distinct reference deciles, with a floor of 10^-6 on each fraction.Source: §C.2 · p. 19
D.1 Local change data capture runs
Local CDC tests passed their five end-to-end checks, and the paper reports separate kernel, audit, test-suite, and Kubernetes scaling results. Each result is tied to its run setting and scope.
D.1 Local change data capture runs
- 0.57 s after a PostgreSQL insert, a new watchlist row reached DynamoDB Local in the first local run, against a 5 s target; all five end-to-end tests passed in 33.50 s.On a fresh stack after a registry-key schema change, the smoke test took 4.28 s and the five tests passed in 36.41 s.Source: §D.1 · p. 20
D.2 Kernel latency, audit, and test suites
- 0.6109 ms p95 and 0.7483 ms p99 were the local single-event scoring-kernel latencies, with throughput of 1,760.2 scores/s over 1,000 iterations.The single-thread benchmark used a 121-fix history, disabled the watchlist, and supplied no Pdata.Source: §D.2 · p. 20
- The AI audit returned 19 findings, including two required fixes, seven hardening items, six decision records, and four requiring no change.A later commit provisioned the SageMaker Model Package Group identified by the second required fix.Source: §D.2 · p. 20
- The public snapshot reported 821 passed and 12 skipped tests with no failures; local coverage was 83.72%, while the private regression gate passed 1,633 tests and skipped 21.The public figures are reproducible; CI does not run coverage, and the paper does not add the private and isolated-suite counts.Source: §D.2 · p. 20
D.3 The Kubernetes scaling test
- 1.89 s elapsed from the scale request to the first HTTP 200 when KEDA moved one EKS deployment from zero replicas to one ready replica in a bounded AWS window.This measured a warm cluster scaling one pod, not an SLO or cold-start time; the deployment later returned to zero replicas.Source: §D.3 · p. 20