PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Instinct, Slow Reason: A Dual-Speed Agentic Framework for Self-Configuring Multi-Object Tracking in Satellite Video

Fan He, Baining Wu, Yan Li, Xiaoyang Zeng

arXiv:2610.33396v1cs.CV

TL;DR

Satellite trackers need to adapt within short observations, but retraining, target-evaluated search, and language-agent decisions do not fit that routine onboard choice. The paper pairs offline-compiled AUSPEX selection with CORVUS’s verified skill and language-agent runtime. AUSPEX improves benchmark accuracy without target labels, while CORVUS commits early and keeps routine choices off the language-model path.

  • Problem

    Satellite observations last tens of seconds, while retraining, target-evaluated search, and language-agent decisions pose constraints on timely, label-free onboard adaptation.

    Source: §I · pp. 1–2

  • Method

    AUSPEX compiles a covering shortlist and label-free pairwise ranker from labelled source tasks, while CORVUS runs the verified skill for routine configuration and reserves the agent for missions.

    Source: §I · p. 2

  • Results

    Under LOTO and LODO, AUSPEX gains 2.84 and 2.94 points of IDF1 and 3.21 and 3.29 points of MOTA with no target label, three passes per sequence, and no token.

    Source: B. Overall Performance · p. 8

  • Takeaways & Limitations

    CORVUS commits 2.5 s into each observation, while its runtime hosts fifteen language models without altering any committed configuration.

    Source: E. Space-Ground Communication Load… · p. 7, §VI · p. 12

Abstract

from arXiv · show

Multi-object tracking in satellite video underpins persistent monitoring, yet offline-tuned trackers lack the onboard intelligence and efficiency to adapt within an observation of tens of seconds. The root is that adaptation still means relabelling and retraining the detector, which cannot run on board in time. Tuning the tracker configuration strongly controls accuracy without touching the network, yet search needs labelled or proxy evaluations, and a language agent costs seconds and tokens per decision. We propose a dual-speed agentic framework pairing a fast instinct compiled offline with a slow reason serving the mission. Its algorithm, Arbitration of Unlabelled Shortlists from Prior Experience, or AUSPEX, learns a covering shortlist and a label-free ranker from labelled source tasks and commits one configuration per sequence. Its system, a Cascaded Onboard Runtime with Verified Uplinked Skills, or CORVUS, cascades reflex, skill, and agent layers, so that a verified skill commits 2.5 s into each observation while a language agent serves natural- language missions. On SatMTB and SatVideoDT with three recent detectors, AUSPEX raises IDF1 by 2.84 to 2.94 points and MOTA by 3.21 to 3.29 points on average without target labels, balances accuracy and labels best among sixteen methods, and loses at most 0.28 IDF1 points at 1.25 tracking passes. CORVUS executes four missions with execution accuracy up to 0.99 and target recall at the perception ceiling, spends no token on the routine decision, runs on Jetson Xavier NX, and hosts fifteen language models from six families without altering any commitment, demonstrating its extensibility and generalization.

I. INTRODUCTION

The paper argues that tracker configurations offer an onboard adaptation lever, then pairs offline-compiled selection with a runtime that commits without waiting for a language agent. It situates this approach against retraining, search, and agent-based alternatives.

I. INTRODUCTION

  • Onboard adaptation is needed within observations lasting a median 20 to 30 s, whereas an offboard loop responds after minutes to hours.Existing searches rely on labelled evaluations or proxies and have target-dependent evaluation counts; language-agent decisions take seconds and consume tokens.Source: §I · pp. 1–2
  • Configuration is an onboard lever: on frozen SatVideoDT detections, IDF1 ranges from below 13 to about 80 percent across configurations for each of three detectors.Changing the detector instead requires annotation, fine-tuning, and validation that cannot run within an observation.Source: §I · p. 1
  • AUSPEX compiles complementary candidates and a label-free ranker from labelled tasks, then commits one configuration per sequence; CORVUS keeps this routine choice off the language-model path.The runtime assigns routine decisions to a fast skill and leaves missions and exceptions to the slower language agent.Source: §I · p. 2
  • AUSPEX raises IDF1 by 2.84 to 2.94 points and MOTA by 3.21 to 3.29 points on average without target labels, while CORVUS commits 2.5 s into each observation.AUSPEX has the best accuracy-label balance among sixteen methods; CORVUS executes missions with accuracy up to 0.99 and spends no token on routine decisions.Source: §I · p. 2

A. Multi-Object Tracking in Satellite Video

  • Satellite MOT methods improve accuracy through learned detection and association, but network methods keep association thresholds fixed and adaptive frame-based rules do not use experience from other scenes.AUSPEX instead keeps the detector frozen and selects among discrete post-detection configurations once per sequence.Source: A. Multi-Object Tracking in Satellite… · p. 2

B. Hyperparameter Optimization with Labels

Prior approaches either search with target evaluations, infer or optimize from proxies, or use language agents whose decisions incur latency. AUSPEX completes candidate construction offline and selects among candidates using a short unlabeled sequence prefix.

B. Hyperparameter Optimization with Labels

  • Transfer optimization carries prior experience into online search, but still ranks configurations using labelled target evaluations until convergence.Random search and related optimizers start from scratch, while transfer methods bring portfolios, search regions, or surrogates from source tasks.Source: B. Hyperparameter Optimization with… · pp. 2–3

C. Label-Free Configuration Selection

  • Label-free selectors use target meta-features or output proxies, but their choices may be uncorrectable, encode hand-set notions of good tracks, or depend on source-target similarity.Online expert switching also faces identity handover in multi-object tracking; AUSPEX instead compares shortlisted candidates and commits once per sequence.Source: C. Label-Free Configuration Selection · p. 3

D. Language Agents and Onboard Autonomy

  • Language-agent approaches incur model inference at each decision, while CORVUS runs routine configuration as an uplinked skill and leaves goals and exceptions to a bounded agent.The supported autonomy claim for this decision is event-based autonomy at level E3.Source: D. Language Agents and Onboard Autonomy · p. 3

III. AUSPEX: ARBITRATION OF UNLABELLED SHORTLISTS FROM PRIOR EXPERIENCE

  • AUSPEX completes the search offline, then uses a covering shortlist and label-free arbiter to choose a configuration from a short prefix at a fixed cost.Commitment from a 25-frame prefix costs 1.17 to 1.25 tracking passes.Source: §III · p. 3

A. Problem Formulation

  • AUSPEX adapts only the post-detection tracker configuration, selecting from a fixed candidate head using the first W frames and applying the choice to the whole sequence.Regret separates into a shortlist coverage term and an arbitration term; increasing the head can reduce coverage regret while increasing competition for the arbiter.Source: A. Problem Formulation · pp. 3–4

B. Coverage-Based Shortlist Construction

AUSPEX constructs a shortlist to cover varied source tasks, then ranks shortlisted candidates using label-free detection and track statistics. The two stages address candidate coverage and sequence-specific choice, respectively.

B. Coverage-Based Shortlist Construction

  • Greedy rank clipping builds the shortlist by crediting tasks the existing entries have not yet served well.Clipping each task's ranks at its best rank so far removes credit from already-served tasks and makes the next choice target neglected tasks.Source: B. Coverage-Based Shortlist Construction · p. 4
  • The greedy head achieves at least a 1 − 1/e fraction of the best size-M coverage under the monotone submodular coverage function.After every task reaches its best configuration, the method reranks configurations to form a backup round; wider heads retain the guarantee.Source: B. Coverage-Based Shortlist Construction · p. 4

C. Label-Free Pairwise Arbitration

  • The ranker combines 24 detection statistics with 16 track-output statistics, and is trained on within-sequence source-task pairs whose ground-truth ordering is never used as a feature.On target sequences it standardizes candidate scores and commits the highest-scoring configuration; relative ordering is intended to transfer more reliably than absolute IDF1 prediction.Source: C. Label-Free Pairwise Arbitration · p. 4

D. Prefix-Based Online Commitment

AUSPEX runs the shortlist head on a sequence prefix, ranks the resulting candidates, and commits one configuration for the remainder of the sequence.

D. Prefix-Based Online Commitment

  • The head candidates track the first W frames in parallel, after which the arbiter scores their prefix statistics and selects one configuration.The selected tracker reuses the tracks already produced, while the other trackers stop.Source: D. Prefix-Based Online Commitment · pp. 4–5
  • Only the selected configuration continues after the prefix, so one configuration supplies the whole sequence and no identity is handed over.The decision is made once per sequence because later switching severs identities rewarded by IDF1.Source: D. Prefix-Based Online Commitment · p. 5
  • A 25-frame prefix at ten frames per second delays the decision by 2.5 s; using a longer prefix trades greater evidence for greater delay.The prefix can extend to the whole sequence.Source: D. Prefix-Based Online Commitment · p. 5

E. Complexity and Cost Analysis

CORVUS assigns decisions to the cheapest layer that meets their deadline, with the tracker configuration committed by the skill layer at the sequence prefix. Its onboard selection cost is bounded by that prefix, while ground-side compilation carries configuration-space costs.

E. Complexity and Cost Analysis

  • Ground-side costs that scale with the configuration space include the labelled sweep, greedy shortlist construction, and ranker fitting; these costs do not reach the satellite.The shortlist construction costs O(K|S||Θ|), and the labelled sweep runs across source tasks and configurations.Source: E. Complexity and Cost Analysis · p. 5
  • At W = 25, M = 3, and 200–300 frames, AUSPEX uses 1.17–1.25 tracking passes instead of M passes, and commits at a fixed frame without labels, gradients, or tokens.A search with n evaluations runs n passes, and its settling time depends on the target.Source: E. Complexity and Cost Analysis · p. 5

IV. CORVUS: CASCADED ONBOARD RUNTIME WITH VERIFIED UPLINKED SKILLS

  • CORVUS commits the configuration 2.5 s into each observation through a skill, while the language-model agent handles goals and exceptions over seconds.The skill and reflex layers act within milliseconds; the agent operates on the slower timescale.Source: §IV · p. 5

A. Requirements and Decision Placement

  • The configuration has a one-prefix deadline and needs no language, so the decision-placement rule assigns it to the skill layer.CORVUS assigns each decision to the cheapest layer that meets its deadline and requirements.Source: A. Requirements and Decision Placement · p. 5

B. System Architecture and Interfaces

CORVUS separates perception, runtime, skill registry, and agent responsibilities, keeping the routine configuration commitment on a path that does not pass through the agent. The runtime triggers selection at the prefix and wakes the agent only for specified exceptional events.

B. System Architecture and Interfaces

  • Perception holds the frozen detector and tracker, the skill registry stores packages such as tracker_config, and the bounded agent holds a language model and its tools.The runtime holds the clock, event bus, output gate, and downlink queue; the skill contract maps prefix detections and statistics to a configuration without labels or language-model calls.Source: B. System Architecture and Interfaces · p. 5
  • The routine path commits θ through the skill registry without passing the agent; the sequence timing diagram uses a logarithmic axis and ground segments are not to scale.Agent latencies in the figure come from one measured run.Source: Figure 3 · p. 6

C. Runtime Workflow

  • The runtime fires the skill at prefix_ready, commits θ* and opens the output gate; low margin or empty candidate outputs wake the agent, which cannot overrule the commitment.The agent reads tracks only after commitment, so identities it has seen are not later withdrawn.Source: C. Runtime Workflow · p. 6

D. Onboard Computational Load Analysis

  • The skill makes one prefix-level burst of candidate statistics and ranking, while the agent's duty cycle is zero on the routine path and grows only with exceptions and mission actions.Routing commitment through the seconds-scale agent would delay track release by at least twice the skill delay when λA ≥ W/ν.Source: D. Onboard Computational Load Analysis · p. 7

E. Space-Ground Communication Load Analysis

CORVUS limits routine downlink to compact sequence summaries and sends clips only for flagged sequences. Ground-compiled skills are far smaller than detector weights and take effect within the observation that prompted adaptation.

E. Space-Ground Communication Load Analysis

  • On the downlink, every sequence yields a summary of a few hundred bytes or less than 1/5,000 of one uncompressed 1080 by 1080 colour frame, plus a clip only when flagged.Source: E. Space-Ground Communication Load… · p. 7
  • A compressed skill package is 244–332 KiB versus 91–112 MiB for detector weights, transferring 282–400 times faster.The registry verifies the package on installation and refuses one whose sources include the task it is to serve.Source: E. Space-Ground Communication Load… · p. 7
  • The skill commits after W/ν = 2.5 s, within the first 8–13% of the observation, whereas a ground loop cannot improve the observation that revealed the need.The ground updates what the satellite knows; the satellite applies it to every sequence without waiting for the ground.Source: E. Space-Ground Communication Load… · p. 7

V. EXPERIMENTS

The experiments compare per-sequence configuration selection across tracker spaces, detector–dataset tasks, and transfer protocols. They also define a reference deployment configuration and describe the observation missions posed to the agent.

V. EXPERIMENTS

  • The experiments separate gains from choosing a configuration on each sequence from gains due to the layer that commits it, and compare AUSPEX with alternative selectors and runtimes.The evaluation also dismantles the selector and carries the runtime onto an embedded board.Source: §V · p. 7

1) Configuration Spaces:

  • ByteTrack is evaluated over 178 configurations of seven parameters and SORT over 250 configurations; each also has a fixed deployment setting tuned on one scenario as the gain reference.The ByteTrack grid includes detection thresholds, overlaps, lost-track retention, tentative period, and direction-consistency weight; the SORT grid includes confidence, track age, minimum hits, and matching overlap.Source: 1) Configuration Spaces · p. 7

2) Detectors, Datasets, and Tasks:

  • The evaluation uses frozen SPAX, FMA-Net, and MP2Net detections on SatMTB and SatVideoDT, organized into seven tasks with 70–134 sequences each and 682 sequences total.MP2Net association is replaced by SORT to make its configuration space explicit.Source: 2) Detectors, Datasets, and Tasks · p. 7

3) Transfer Protocols:

  • LOTO trains on other tasks sharing the held-out task's tracker, while LODO further excludes sources sharing its dataset; each fold's choices are fixed using only its remaining source tasks.The two SORT tasks receive identical folds under both protocols because their single source already comes from the other dataset.Source: 3) Transfer Protocols · p. 7
  • The reported ceiling for any method committing one configuration per task is 3.10 points of IDF1; shifts below half a point on one task are not treated as effects.The stated reason is that numerical-library load order alone can move a single task by up to half a point.Source: 3) Transfer Protocols · p. 7

V. EXPERIMENTS

  • The agent missions add, in turn, a sort, a filter, a filter conflicting with the sort, and a comparison over time.Each mission grants three detection calls and three tracking slots and ends when locked targets are submitted.Source: 3) Transfer Protocols · p. 8

4) Metrics and Cost:

The section defines how tracking accuracy and labeling or computation cost are assessed, then describes the language-model mission tests and their evaluation criteria.

4) Metrics and Cost:

  • IDF1 is primary because it matches whole trajectories; cost combines the fraction of labelled sequences, tracking passes, and language-model tokens.The normalized hypervolume summarizes accuracy and labeling together, with zero when the whole corpus is labelled or performance falls below the fixed configuration.Source: 4) Metrics and Cost · p. 8

5) Agent Missions and Language Models:

  • The agent scans a SatMTB clip, reads runtime tracks, and submits three targets while the runtime independently commits its configuration.Four missions add a sort, a filter, a conflicting filter, and a comparison over time.Source: 5) Agent Missions and Language Models · p. 8
  • Fifteen models run four missions on nine sequences spanning cars, aircraft, and ships; EX measures mission execution, while R@3 measures area-based target recall.Applying the fixed mission rule to reference tracks gives an R@3 perception ceiling of 0.67; results are from one server version, whose numerical kernels can shift small-model mean EX by up to 0.47.Source: 5) Agent Missions and Language Models · p. 8

6) Baselines and Implementation:

AUSPEX is compared with label-free selectors and labelled searches using accuracy and adaptation cost. It achieves the strongest accuracy-label balance overall, with gains concentrated where the fixed configuration leaves room for improvement.

6) Baselines and Implementation:

  • AUSPEX uses detection and track statistics as inputs, scores the whole sequence in baseline comparisons, and uses a three-candidate shortlist with an XGBoost ranker.Labelled searches stop at twenty evaluations; deployed prefix-window evaluation is reported separately.Source: 6) Baselines and Implementation · p. 8

B. Overall Performance

  • Under LOTO and LODO, AUSPEX gains 2.84 and 2.94 IDF1 points and 3.21 and 3.29 MOTA points, with no target labels, three passes, and no tokens.It recovers 92% and 95% of the ceiling and attains the highest hypervolume; no other method exceeds its MOTA gain.Source: B. Overall Performance · p. 8
  • Predictors without target observations can fall below the fixed configuration under LODO, while proxy-guided searches can worsen tracking and learned-evaluator searches benefit from more passes.With about sixteen labelled sequences and twenty evaluations, BoundingBox has the highest IDF1, followed by CTS and ZeroShot; labelled methods’ hypervolume is at most 0.86.Source: B. Overall Performance · pp. 8–9
  • AUSPEX’s gains correlate with the room above the fixed configuration at 0.998 under both protocols and remain positive on every task in both metrics.Without SPAX on SatVideoDT, its IDF1 gains are 1.47 points under LOTO and 1.59 under LODO; the mean is dominated by the task where the fixed configuration fails badly.Source: B. Overall Performance · p. 9
  • On SatVideoDT video 08 under SPAX, AUSPEX reaches 60.07% IDF1 versus 41.67% for the fixed configuration by lowering the threshold from 0.30 to 0.25.Missed targets fall by about a third, outweighing the added false tracks and identity switches.Source: B. Overall Performance · p. 9

1) Structure of the Selector:

  • Withdrawing arbitration costs 1.33 and 1.58 IDF1 points, while source-rank ordering instead of coverage costs 1.67 under LODO and nothing under LOTO.A random shortlist lowers all seven tasks; replacing the learned ranker with a most-boxes rule changes mean IDF1 by at most 0.13 points.Source: 1) Structure of the Selector · p. 9

2) Width of the Shortlist:

The ablations examine shortlist width and runtime commitment choices. They identify a narrow shortlist and early, once-per-sequence skill commitment as effective, while language-model interchangeability chiefly affects mission execution rather than configuration choice.

2) Width of the Shortlist:

  • At shortlist width three, coverage regret is 0.03 points under both protocols; widening further only increases arbitration regret.Increasing width from one to two reduces coverage regret from 1.59 and 1.75 points to 0.03 and 0.17, making three the narrowest width that covers the target under both protocols.Source: 2) Width of the Shortlist · p. 9, Figure 4 · p. 10

D. System-Level Ablations

  • The system-level ablations hold AUSPEX fixed and vary when commitment occurs, which runtime layer commits, and whether the language model can be exchanged.These tests address framework behavior when selection operates within an agent on a live stream.Source: D. System-Level Ablations · p. 9

1) Granularity of Commitment:

  • A 25-frame prefix matches full-sequence IDF1 under LOTO and trails by 0.28 points under LODO at 1.25 passes; switching every 50 frames loses about a point.Once-per-sequence commitment avoids identity-severing switches and releases tracks 2.5 seconds into the observation.Source: 1) Granularity of Commitment · p. 9, Table III · p. 11

2) Layer of Commitment:

  • Moving commitment from perception to the skill layer produces the adaptation; moving it to the agent adds tokens without adding accuracy.A hand-written skill-layer rule nearly matches the ranker, whereas the agent-layer call uses about 1,600 tokens per sequence and source examples help under LOTO but hurt under LODO.Source: 2) Layer of Commitment · pp. 9–10, Table III · p. 11

3) Interchangeability of the Language Model:

  • Across fifteen models, the committed configuration is identical whenever commitment is reached; execution accuracy varies with model, while R@3 remains at the perception ceiling.Within Qwen3.5, EX rises from 0.14 at 0.9 billion parameters to 0.99 at 27 billion; the paper cautions this is a trend of the tested suite and prompt, not a law of scale.Source: 3) Interchangeability of the Language… · pp. 10–11
  • In the tool-call persistence test, Qwen3.5 models invoke the configuration tool on at most three of nine sequences, and their R@3 is 0.19 to 0.44 below the skill layer at every scale.Across 36 paired runs, the skill layer wins 21, ties 13, and loses 2; the example mission reaches the sequence’s perception ceiling despite detector box errors.Source: 3) Interchangeability of the Language… · p. 11

E. Onboard Deployment

On Jetson Xavier NX, CORVUS commits within the 2.5-second prefix window while language-model turns take much longer. The conclusion reports broad selector and runtime results, while limiting the evidence to two datasets and seven tasks.

E. Onboard Deployment

  • On the workstation, a nine-billion-parameter model turn takes 2.8–3.5 s, exceeding the 25-frame prefix and preventing the agent from committing in time.The extra trackers add at most about 13 ms to the 100 ms frame interval, and the skill burst stays below about 22 ms.Source: E. Onboard Deployment · p. 11
  • On Jetson Xavier NX, three trackers process the 25-frame prefix in at most 2.07 s, and the skill burst takes at most 61 ms.The 25 frames arrive in 2.5 s; a four-billion-parameter model turn takes 15.6–18.4 s on the board GPU.Source: E. Onboard Deployment · p. 12

VI. CONCLUSION

  • On two benchmarks with three detectors, AUSPEX achieves the best accuracy–label balance among sixteen methods at no more than 1.25 tracking passes.CORVUS uses no language-model token for routine decisions and hosts fifteen language models without changing committed configurations.Source: §VI · p. 12
  • The evidence spans two datasets and seven tasks, one carrying most of the shift; adaptation covers only association, and ECSS E4 goal-level autonomy is outside the claims.Source: §VI · p. 12

Related papers

Loading 2610.33396v1…