PaperlayerView exact revision on arXiv

Source-linked AI summary

Predictive audio representations for early detection and tracking of hidden dynamic objects

Katerina Vinciguerra, Moritz Brandes, Danilo Hollosi, Letizia Marchegiani

arXiv:2609.13595v1cs.SDcs.AIeess.AS

TL;DR

Occluded traffic agents may be detected too late, and previous acoustic systems handled either single-vehicle direction or type rather than simultaneous multi-vehicle estimates. This paper combines self-supervised temporal prediction on raw multichannel audio with supervised direction, count, and type prediction. It reports improvements over prior systems and transfer to an unseen driving scenario, while its proof-of-concept handles only two simultaneous vehicles.

  • Problem

    Prior acoustic studies addressed single-vehicle direction or vehicle type, while roadside multi-vehicle monitoring uses fixed infrastructure and minute-level estimates rather than onboard, timely sensing.

    Source: §I · p. 1

  • Method

    A JEPA-inspired self-supervised stage predicts future latent representations from past raw multichannel audio, followed by supervised multi-task fine-tuning for direction, count, and type.

    Source: §I · p. 2

  • Results

    Against on the TJ dataset, the model improves by 21.2% in Direction, 40.1% in Type, and 5.6% in Count; it also reports a 9.1% Direction improvement over the strongest prior result.

    Source: B. Experiment 1 · pp. 4–5

  • Takeaways & Limitations

    The paper presents an onboard acoustic system jointly estimating direction, number, and class of multiple NLOS vehicles at one-second latency, with reported generalization to unseen scenarios.

    Source: §II · p. 2

    The proof-of-concept handles simultaneous presence of only two vehicles.

    Source: §I · p. 2

Abstract

from arXiv · show

Predicting potential dangers is core to safety. Forecasting the presence of other traffic agents is core to danger prediction. Occluded traffic agents challenge detection systems as they might become visible too late, leaving the autonomous vehicle too little time to identify, plan and act accordingly in a robust and safe way. Previous works proved that auditory perception, being omnidirectional and not constrained by a field-of-view, provides fundamental cues for early spotting of different road users, even when hidden by other vehicles or infrastructures. Yet, those contributions deal with scenarios with only one vehicle present, and they either identify the type of vehicle or estimate its direction of arrival. In this work, we move forward, and propose a multi-task system which, simultaneously, estimates the number of vehicles present, their type, and their direction of arrival. Our methodological contribution is a two-stage pipeline: a self-supervised pre-training stage inspired by the Joint- Embedding Predictive Architecture (JEPA) applied directly to multichannel raw waveforms, followed by supervised multi-task fine-tuning with a bidirectional LSTM and three classification heads. The pre-training stage trains the encoder without labels, pushing it to predict the latent representation of a future audio segment from its past context. To train and test our framework, since no suitable dataset was publicly available, we collected an ad-hoc one covering Non-Line-Of-Sight scenarios with multiple traffic agents simultaneously operating. Experimental evaluation shows that our method outperforms the state of the art; it also proves that our design choice allows the model to learn robust representations, which can be transferred to an unseen driving scenario, maintaining reasonable and stable performance.

I. INTRODUCTION

Acoustic perception can detect vehicles hidden from view, but earlier systems addressed single-vehicle direction or type separately, while roadside monitoring is too slow for immediate onboard use. This work proposes an onboard, multi-task system and a JEPA-inspired raw-audio training pipeline.

I. INTRODUCTION

  • Vehicle type and location jointly matter for safe planning because different vehicle classes have different kinematics and movement limits.For example, a city bus has a wider turning radius and slower manoeuvrability than a car.Source: §I · p. 1
  • Earlier acoustic studies estimated approaching-vehicle presence or direction in single-vehicle settings, while roadside monitoring reports multi-agent estimates only at minute-level granularity.Those roadside systems use fixed infrastructure and are not directly usable by an autonomous vehicle.Source: §I · p. 1
  • The system predicts direction, vehicle count, and class at one-second resolution from raw audio captured by an eight-microphone circular array.It targets outside-field-of-view vehicles and avoids sound-source separation.Source: §I · p. 2
  • A two-stage pipeline predicts future latent audio representations from past multichannel raw waveforms, then fine-tunes a bidirectional LSTM with three classification heads.The self-supervised pre-training is inspired by JEPA and uses no spectral preprocessing.Source: §I · p. 2

II. RELATED WORK

Prior acoustic work covers single-vehicle direction estimation and fixed-roadside multi-vehicle monitoring, but not the onboard, low-latency combination needed here. The paper introduces a JEPA-inspired temporal-prediction objective on raw multichannel waveforms.

II. RELATED WORK

  • Prior acoustic systems estimate direction for single vehicles, whereas roadside monitoring covers multi-vehicle localization and classification but uses stationary sensors and minute-level updates.The paper positions these systems as valuable for traffic monitoring but not for immediate planner interventions.Source: §II · p. 2
  • The paper adapts JEPA temporal prediction to raw multichannel waveforms, forecasting a future latent representation from context rather than reconstructing audio detail.It presents this as part of the first onboard acoustic system jointly estimating direction, number, and class of multiple NLOS vehicles at one-second latency.Source: §II · p. 2

III. METHOD

  • The framework is organized around input windowing, self-supervised encoder pre-training, and supervised multi-task fine-tuning.The full pipeline is shown in Fig. 2.Source: §III · p. 2

A. Temporal Windowing of the Input Signal

The input is divided into one-second raw waveform frames and grouped into three-second sliding windows. The encoder maps each frame to a latent vector using strided convolutions, pooling, and a linear projection.

A. Temporal Windowing of the Input Signal

  • Each frame is a channel-wise normalized raw waveform from eight microphones sampled at 48 kHz, with 48,000 samples per channel and no spectral transform.Consecutive frames are grouped into three-second sliding windows centered on a frame; windows do not cross scenario boundaries.Source: A. Temporal Windowing of the Input… · p. 2

1) Architecture:

  • The encoder maps a one-second waveform to a 256-dimensional latent using three strided one-dimensional convolutional blocks, global average pooling, and a linear projection.The blocks use kernel size 9 and stride 2, with 64, 128, and 256 output channels; the latent dimension matches the final block depth.Source: 1) Architecture · pp. 2–3
  • A lightweight two-layer MLP predictor approximates the mapping from a past latent representation to a future one.Its fully connected layers map from the latent space back to the same dimension.Source: 1) Architecture · p. 3

2) Training objective:

Self-supervised pre-training uses an online encoder to predict a target encoder’s representation of a future audio frame from the past frame, with the target encoder updated by EMA.

2) Training objective:

  • The target encoder follows the online encoder through EMA, with momentum m following the BYOL convention adopted by I-JEPA.The target receives no gradient and is updated after each online-weight gradient step.Source: 2) Training objective · p. 3
  • The online encoder processes the past frame while a predictor forecasts the target encoder’s representation of the future frame.The target encoder receives no gradient and is updated by exponential moving average (EMA) of the online weights, providing stable targets without negative pairs.Source: 2) Training objective · p. 3
  • The intermediate frame is excluded from pre-training to encourage the encoder to capture longer-range acoustic dynamics.After pre-training, the online encoder weights are retained and the predictor is discarded.Source: 2) Training objective · p. 3

C. Supervised Multi-Task Fine-tuning

The fine-tuning architecture encodes three consecutive frames, aggregates their latents with a bidirectional LSTM, and uses a shared MLP representation for three classification heads.

C. Supervised Multi-Task Fine-tuning

  • Three frames are encoded independently by a shared encoder, then their latent sequence is aggregated by a single-layer bidirectional LSTM.Only the final-step hidden state is propagated to the shared module.Source: 1) Model architecture · p. 3
  • A shared dropout-regularized MLP projects the final LSTM state to 128 dimensions before the direction, count, and vehicle-type heads.The heads predict five direction classes, three count classes, and four vehicle-type classes.Source: 1) Model architecture · p. 3

2) Training objective and protocol:

The section describes the multi-task training protocol, the experiment plan, and the collected datasets, labels, and evaluation split.

2) Training objective and protocol:

  • Each task head uses cross-entropy, and fine-tuning begins with five frozen-encoder warm-up epochs before joint encoder optimization.The model trains for 20 epochs at batch size 32, retaining the checkpoint with highest validation direction accuracy.Source: 2) Training objective and protocol · p. 3

IV. EXPERIMENTS

  • The experiments compare the multi-task model with prior work, test design choices through ablations, and assess generalization to unseen scenarios.The three tasks are Direction, Count, and Type.Source: §IV · p. 3

A. The Dataset

  • The dataset covers six scenarios grouped into stationary T-Junction and moving Take-Over settings, with 6,827 and 3,005 one-second labeled frames, respectively.Data collection used three roof-mounted cameras above an eight-microphone array, plus GPS.Source: A. The Dataset · p. 3
  • Labels represent vehicle count with three classes, vehicle type with four classes, and direction with five classes.Direction labels include left, front, right, none, and both; the driving scenarios also assign back to vehicles not yet at either lateral camera.Source: A. The Dataset · p. 4
  • All training and validation use the T-Junction dataset, while Take-Over is held out from supervised training and self-supervised pre-training for cross-scenario evaluation.The T-Junction split has 5,458 training frames and 1,369 validation frames.Source: A. The Dataset · p. 4

B. Experiment 1

On the T-Junction dataset, the proposed model improves direction accuracy over prior work, including a comparison using the extended five-class taxonomy and a multi-task comparison with.

B. Experiment 1

  • 9.1% improvement in Direction over the strongest prior result, despite using an additional class in the five-class taxonomy.The comparison uses the T-Junction dataset and the same split as prior direction models.Source: B. Experiment 1 · p. 4
  • The reference scenarios include static T-junction approaches with one or two vehicles and driving overtakes with one or two vehicles approaching from behind.Some static cases include one vehicle occluding another, while one case has vehicles approaching from different directions.Source: Figure 4 · p. 5
  • 21.2% in Direction, 40.1% in Type, and 5.6% in Count improvement over on the T-Junction dataset.The authors identify as the only other work performing all three tasks.Source: B. Experiment 1 · p. 5

C. Experiment 2

The ablations favor a two-step temporal-prediction horizon, while the unseen-scenario experiment tests head-only, full-model, and SSL re-pretraining adaptation after holding out TO.

C. Experiment 2

  • The LSTM-only baseline performs worst; unfreezing the encoder after stage 2 changes performance little but helps the Direction task.Source: C. Experiment 2 · p. 5
  • 81.8% Direction accuracy for two-step prediction exceeds one-step (80.8%) and interpolation (79.3%), making two-step the default.The variants differ in which frames serve as context and target.Source: C. Experiment 2 · p. 5

D. Experiment 3

  • TO is held out during training; after training on TJ, adaptation options are head-only tuning, full-model tuning, or SSL re-pretraining followed by head tuning.The TO direction head is extended to include a new back class before fine-tuning.Source: D. Experiment 3 · p. 5
  • Each K-shot option is evaluated over 10 random draws, and Direction recall excludes the none and both classes.Fine-tuning runs for 15 epochs.Source: D. Experiment 3 · p. 6

1) Zero-shot transfer:

Zero-shot transfer to the dynamic TO setting yields near-zero Direction accuracy and low Count and Type accuracy. With support data, accuracy rises, and the three adaptation strategies perform similarly at the tested shot counts.

1) Zero-shot transfer:

  • At K=0, overall Direction accuracy is 2.3%, while Count and Type accuracy are 8.2% and 6.5%, respectively.The back class cannot be predicted before the direction head receives gradient updates; the authors attribute the Count and Type collapse to the shift from stationary to dynamic conditions.Source: 1) Zero-shot transfer · p. 6

2) Few-shot adaptation:

  • At K=5, overall Direction accuracy is 54.5–55.8% across options, rising to 69.2–70.1% at K=10; option rankings are unstable and differences are small relative to variation across draws.The authors report similar near-equivalence among options for Count and Type.Source: 2) Few-shot adaptation · p. 6

3) Per-class behaviour under class imbalance:

The reported findings include advance detection before FoV entry, uneven per-class recall under class imbalance, and front-direction errors concentrated at class transitions.

3) Per-class behaviour under class imbalance:

  • Right-class recall stays at or below 13% at both 5- and 10-shot; under Option A, left recall falls from 32.4% to 15.6%.The authors identify diverse data collection and better zero-shot understanding of long-tailed scenarios as future directions.Source: 3) Per-class behaviour under class… · p. 6

1) Detection horizon outside the Field-of-View (FoV):

  • 4.6 s mean detection horizon (median 5.0 s); 77% of 38 scenarios provide at least 1 s of advance prediction and 62% at least 3 s.The horizon is the longest run of correctly classified 1-second frames ending immediately before FoV entry.Source: 1) Detection horizon outside the… · p. 6

2) Front-class error pathology:

  • Fig. 5 shows most front-class errors at transitions between left/right and front, consistent with the authors’ view that direction boundaries are arbitrary.The figure breaks down predictions by frame position within the front visibility cone.Source: Figure 5 · p. 6, 2) Front-class error pathology · p. 6

3) Dynamics and Types Representations:

The paper reports that its multi-task acoustic framework outperforms the state of the art, with robust predictions every second. Frozen-encoder latents reveal structure across silence, vehicle count, type, and direction, while the conclusion identifies extensions to more vehicles and continuous-angle localization as future work.

3) Dynamics and Types Representations:

  • Frozen latents separate silence from activity and organize active scenes by count, while direction regions are recognizable but front overlaps and car clusters less tightly than van.Two-vehicle scenarios are car–van pairs by recording protocol; the UMAP panels use the same projection coordinates and show direction, count, and type labels.Source: 3) Dynamics and Types Representations · p. 6, Figure 6 · p. 7

V. CONCLUSION

  • The framework outperforms the state of the art and provides robust predictions every second.The conclusion describes estimates of direction, number, and type of occluded vehicles.Source: §V · p. 6
  • Table II compares per-class recall on static T-junction validation data with driving data across three adaptation strategies.For K>0, cells report means and standard deviations over 10 random support-set draws; at K=0, strategies are identical.Source: Table II · p. 7
  • Future work targets scenarios with more than two vehicles and localization by continuous angles rather than classes.The conclusion also identifies integration into a multimodal system as a future direction.Source: §V · p. 7
Loading 2609.13595v1…