PaperlayerView exact revision on arXiv

Source-linked AI summary

V2TATC: A Joint Voice-Trajectory Embedding Framework and Dataset for Air Traffic Controller Situational Awareness

Louis Brusset, Mathurin Petit, Jordan Kam, Alexandre Bayen

arXiv:2608.28981v1cs.LGeess.AS

TL;DR

ATC decision-support tools need to reason over voice instructions and aircraft trajectories as traffic grows, but these modalities are often handled separately. V2TATC learns a shared voice–trajectory space with bidirectional querying and bijective lifting, then evaluates it on a San Francisco Bay Area case study. The framework supports cross-modal retrieval and downstream operations, while voice-conditioned trajectory forecasting remains substantially weaker.

  • Problem

    ATC systems commonly process speech and trajectories in isolation, leaving the correspondence between a controller's instruction and the resulting aircraft maneuver implicit.

  • Method

    V2TATC combines self-supervised trajectory encoding, speech encoding, contrastive joint embedding, and normalizing-flow-based bijective lifting.

  • Results

    The joint space supports cross-modal retrieval well above random and carries enough structure to drive simple downstream operations in a congested terminal-area case study.

  • Takeaways & Limitations

    A shared voice–trajectory representation can provide a building block for ATC situational-awareness and decision-support tools.

  • Takeaways & Limitations

    Voice-conditioned trajectory forecasting remains an order of magnitude weaker than the other evaluated path.

Abstract

from arXiv · show

As air traffic volumes in the National Airspace System continue to expand, in particular in the low altitude airspaces, the need for scalable decision support tools used by air traffic controllers will also require more development. This article introduces Voice-to-Trajectory for Air Traffic Control, a joint voice communication-flight trajectory data embedding framework, that can be a component of situational awareness in congested airspaces, and assist the development of tools for ATC as they reason in real-time over Automatic Dependent Surveillance-Broadcast trajectories, or the intent expressed by pilots in natural language. We show that these data modalities are not independent and represent a common physical referent: an aircraft flying through the airspace. V2TATC maps a voice instruction and the trajectory of the addressed aircraft to nearby points in a single latent space that can be queried in both directions. It combines a self-supervised trajectory encoder, a frozen large-scale speech encoder, a contrastive joint embedding, and a bijective lifting via normalizing flows. We demonstrate V2TATC's effectiveness on the San Francisco Bay Area, for its concentration of major airports, and its mix of commercial and general aviation low altitude traffic. Lastly, we release a novel paired voice-trajectory dataset, and report experiments on cross-modal retrieval, ablations, and latent-space analysis.

Nomenclature

V2TATC addresses the need for scalable ATC decision-support tools by aligning voice transmissions with aircraft trajectories in a shared, bidirectionally queryable representation. The framework is instantiated over the dense, multimodal San Francisco Bay Area and supports retrieval, forecasting, and other downstream operations.

  • Motivation: V2TATC aligns voice transmissions and ADS-B trajectories because both refer to the same aircraft moving through airspace.Existing systems commonly process speech, trajectories, and weather separately, leaving the correspondence between instructions and maneuvers implicit.
  • Framework: The framework maps an instruction and the addressed aircraft's trajectory to nearby points in one latent space, enabling queries in both directions.Trajectory-to-voice retrieval can support context recovery, while voice-to-trajectory retrieval can help identify the addressed aircraft.
  • Framework: Its pipeline combines a self-supervised trajectory encoder, a frozen speech encoder, contrastive alignment, and normalizing flows for bijective lifting.The trajectory modality is a structured time series, motivating a motion-aware encoder; the flows extend contrastive retrieval toward invertible generation.
  • Applications: The joint space is designed as a building block for decision-support tools, including anomaly detection, short-horizon forecasting, simulator instruction synthesis, and counterfactual maneuver probes.These uses arise from retrieval, cross-modal inconsistency signals, and the invertible lifting above the joint space.
  • Dataset and case study: The San Francisco Bay Area provides a dense case study combining major airports, commercial and general-aviation traffic, overlapping controlled airspace, and potential future eVTOL operations.The concentrated region supports collection of a rich multimodal dataset within small geographic boundaries.
  • Dataset and evaluation: The authors release a paired voice–trajectory dataset and processing pipeline, addressing the scarcity of publicly released datasets containing both modalities.They report quantitative experiments on alignment, cross-modal retrieval, and downstream operations such as short-horizon forecasting.

II. Problem Formulation

The paper frames controller situational awareness as linking voice intent with observed aircraft trajectories, an alignment not directly available from raw operational signals. It learns a shared space where matched voice–trajectory pairs are nearby and unrelated pairs are distant, enabling bidirectional lookup.

  • A. Situational Awareness in Congested Airspaces: Situational awareness comprises perceiving, comprehending, and projecting the state of an airspace.
  • A. Situational Awareness in Congested Airspaces: Busy terminal areas impose high cognitive load because controllers track many aircraft, each with dynamic state, identity, and verbal instructions.
  • A. Situational Awareness in Congested Airspaces: Existing displays provide detailed radar and ADS-B information, while radio communications remain largely raw audio and are rarely aligned with surveillance data.
  • A. Situational Awareness in Congested Airspaces: The missing alignment prevents standard decision-support tools from directly connecting an aircraft on the surveillance display with what was said about it.
  • A. Situational Awareness in Congested Airspaces: V2TATC represents voice transmissions and trajectory windows as observations of the same airspace through different modalities.
  • A. Situational Awareness in Congested Airspaces: The trajectory representation uses consecutive differences rather than absolute positions, exposing relative maneuvers that may encode intent in both directions.
  • A. Situational Awareness in Congested Airspaces: The learned space Z places a voice phrase and its addressed aircraft trajectory nearby while unrelated pairs remain distant.
  • A. Situational Awareness in Congested Airspaces: Nearest-neighbor lookup in Z can retrieve a trajectory from voice or voice from trajectory, making Z a learned cartography of the airspace.

B. Two-Tower Joint Embedding Approach

V2TATC uses separate voice and trajectory towers that communicate through a shared embedding space. Contrastive alignment supports retrieval, while invertible flows extend the system to bidirectional latent translation.

  • B. Two-Tower Joint Embedding Approach: The two-tower design processes voice and trajectory data with separate modality-specialized encoders that communicate through one shared output space.
  • B. Two-Tower Joint Embedding Approach: Candidate embeddings can be pre-computed for efficient online retrieval, and either tower can be replaced without retraining the other.
  • B. Two-Tower Joint Embedding Approach: Contrastive training needs only whether paired observations describe the same aircraft at the same time, rather than semantic labels or predefined classes.
  • C. Bijectivity Beyond Retrieval: Because contrastive projectors are forward-only, they support retrieval but cannot recover a modality embedding from a joint-space point.
  • B. Two-Tower Joint Embedding Approach: Contrastive projectors map both modalities into a common space, where matched aircraft representations become close after training.
  • C. Bijectivity Beyond Retrieval: Two normalizing flows add a learned bijection between each modality space and the joint space, enabling latent voice-to-trajectory and trajectory-to-voice translation.
  • C. Bijectivity Beyond Retrieval: The paper uses bijection geometrically between low-dimensional modality submanifolds rather than claiming a strict one-to-one mapping between raw spaces.
  • A. Trajectory Tower: The trajectory tower uses masked reconstruction on ADS-B windows, with normalized feature differences and time-gap encoding to represent irregular sampling.

B. Voice Tower

The voice tower freezes a large pretrained ASR encoder and temporally mean-pools its frame representations into a compact voice embedding. This reduces storage and supports latent retrieval, but loses invertibility to raw audio.

  • B. Voice Tower: The voice tower applies a frozen large-scale ASR encoder to variable-length waveforms and mean-pools frame-level hidden states into one embedding.
  • B. Voice Tower: Freezing the speech encoder preserves pretrained robustness, avoids catastrophic forgetting risks, and limits trainable parameters for the available data.
  • B. Voice Tower: Approximately 1,500 Whisper frames per 15-second clip produce a 7.7 MB single-precision representation before pooling.
  • B. Voice Tower: Mean pooling reduces the full 83,000-sample corpus to under 500 MB, three orders of magnitude lighter than storing full frame-level outputs.
  • B. Voice Tower: Mean pooling is selected because every frame contributes and the arithmetic mean avoids discarding components through an argmax.
  • B. Voice Tower: Temporal pooling creates a strict information bottleneck because the pooled vector cannot be inverted to the original frame sequence or raw audio.
  • B. Voice Tower: A lookup table can reuse associated training audio for nearby joint-space queries, while the embedding itself supports latent retrieval and cross-modal generation.
  • B. Voice Tower: The voice and trajectory projections are normalized before symmetric InfoNCE training, converting their inner products into bounded cosine similarities.

D. Bijective Lifting via Normalizing Flows

V2TATC adds normalizing flows after contrastive alignment to obtain bidirectional, dimension-compatible mappings between modality embeddings and the joint space. The staged training procedure combines projector matching, inverse reconstruction, and cross-modal consistency.

  • D. Bijective Lifting via Normalizing Flows: After contrastive training, voice and trajectory embeddings describing the same aircraft are close in the joint space.
  • D. Bijective Lifting via Normalizing Flows: The projectors are not invertible, so two flows learn bidirectional maps between each padded modality space and the joint space.
  • D. Bijective Lifting via Normalizing Flows: Normalizing flows are sequences of invertible transformations with tractable Jacobians, supporting exact differentiable change-of-variables likelihoods.
  • D. Bijective Lifting via Normalizing Flows: RealNVP affine coupling splits the input, conditions scale and translation on one half, and provides an analytically available inverse with triangular Jacobian.
  • D. Bijective Lifting via Normalizing Flows: The implementation uses width 1792, zero-padding smaller embeddings and retaining padded dimensions in reconstruction so surplus coordinates contract toward zero.
  • D. Bijective Lifting via Normalizing Flows: The triple loss combines forward projector matching, backward embedding recovery, and cross-modal alignment of paired samples to one joint point.
  • D. Bijective Lifting via Normalizing Flows: Synthetic encoder-space samples enlarge supervision for the forward term, while the cross term remains restricted to real paired data.
  • D. Bijective Lifting via Normalizing Flows: The framework trains stagewise: trajectory encoding, contrastive projectors, and flows are learned sequentially over frozen upstream components.

IV. Implementation

The implementation builds trajectory and voice streams concurrently over the San Francisco Bay Area, transforms ADS-B data into fixed six-dimensional windows, and pre-trains a Transformer-based trajectory encoder. The resulting latent representation is flattened and passed to the contrastive projector.

  • Data collection: Trajectory and audio are collected concurrently, enabling temporal alignment between the two modalities.ADS-B and voice sources are polled on separate schedules over overlapping time windows.
  • Trajectory preprocessing: ADS-B samples are converted into a local Cartesian frame with six features measured in meters or meters per second.The transformation addresses longitude nonlinearity, incompatible spatial units, and circular track values.
  • Trajectory preprocessing: Flight segments are divided into fixed windows of T=14 consecutive samples, with boundaries detected from callsign changes or gaps exceeding two minutes.A 13-point continuity buffer prevents artificial truncation at raw-file boundaries.
  • Trajectory encoder: The trajectory encoder is pre-trained as a masked autoencoder using two stages that move from spherical differences to Cartesian features and deltas.Stage 2 removes track wrap-around and angular-versus-linear unit mismatch from the representation.
  • Trajectory encoder: Among four tested backbones, the Transformer is retained for the framework, producing a flattened trajectory embedding of d_t=1792.The embedding is obtained from 14 timesteps and hidden dimension 128, then fed to the contrastive projector.

B. Voice Data

The voice-data pipeline archives Bay Area ATC audio, detects and transcribes short transmissions with Whisper, extracts callsigns through prioritized matching rules, and prepares paired speech data. A VRAM-aware model pool makes large-scale transcription feasible under GPU-memory constraints.

  • Audio collection: ATC audio covers eight Bay Area frequencies spanning ground, tower, approach, and en-route sectors, with 12 hours per day over four consecutive days.Audio is delivered as 30-minute MP3 chunks with UTC metadata for later alignment.
  • Transcription: Whisper large-v3 transcribes each MP3 chunk, while Silero VAD is tuned to retain short ATC clearances before transcription.The VAD uses a 0.3 threshold, 50 ms minimum speech duration, and 800 ms padding.
  • Voice representation: The same voice encoder supplies both transcription features for callsign matching and the downstream voice embedding.This keeps the embedding used for alignment tied to the network from which the transcription is decoded.
  • Transcription: A VRAM-aware pool loads as many Whisper instances as fit below a configurable memory headroom and dispatches chunks through worker threads.This addresses Whisper’s runtime and GPU-memory bottlenecks while allowing resumable output writes.
  • Callsign extraction: Callsigns are extracted with prioritized patterns ranging from full identifiers to telephony designators, while normalization handles separators, split transmissions, and digit words.Only the highest-priority match is retained when multiple patterns match a line.

C. Linking Modalities

The linking pipeline matches ADS-B flight windows to timestamped transcriptions, packages paired audio and trajectory samples, and trains the joint architecture in two stages. Pairing substantially reduces and biases the effective corpus, while synthetic embeddings augment flow training.

  • Pair construction: Each ADS-B window is matched to transcription lines with the same callsign within a ±60-second temporal window.Searching from flights to speech reduces false positives from similar callsigns operating elsewhere.
  • Pair construction: Each matched sample contains an audio clip, a centered 14-step trajectory window, and metadata describing callsign, timestamps, frequency, and source file.The audio includes a 750 ms tail pad, while the trajectory centers on the speech event.
  • Dataset construction: The paired corpus is serialized into HDF5 shards containing variable-length 16 kHz audio and float32 trajectory arrays with timestamps.Dataset statistics and frequency distributions are reported in Tables 2 and 3.
  • Dataset construction: Linking drastically reduces the effective dataset and biases it toward transcribable commercial operations on instrumented sectors.The altitude distribution retains its general shape while becoming increasingly selective through the pairing stages.
  • Joint training: The projectors contain roughly 10^7 trainable parameters each despite a validation set of only about 5×10^3 paired samples.Dropout, weight decay, and early stopping are used to regularize this capacity relative to the data scale.

V. Results

V2TATC supports cross-modal retrieval and short-horizon forecasting from aligned voice and trajectory representations. Retrieval substantially exceeds random performance, while voice-conditioned forecasting preserves qualitative motion trends but remains less accurate than direct trajectory forecasting.

  • Cross-Modal Retrieval: The model achieves R@1=0.235 and R@10=0.726 over 5,229 validation candidates, with R@1 roughly three orders of magnitude above random retrieval.The gap between R@1 and R@10 indicates frequent confusion among similar trajectories, while usually recovering the correct region.
  • Cross-Modal Retrieval: The joint representation establishes a bijection between modality submanifolds rather than directly between individual paired samples.This supports querying the shared space across modalities without implying one-to-one sample identity.
  • Trajectory Forecast: Direct trajectory forecasting reaches global MSE 0.071, compared with 0.710 for the voice-to-trajectory chain in normalized delta z-score space.The cross-modal path preserves qualitative position trends but is offset on heading and vertical rate.
  • Trajectory Forecast: The zero-shot forecasting path reaches MSE 2.0, confirming the importance of the task-specific StepPredictor head.The zero-shot variant also produces substantially larger physical errors, including approximately 149° heading error and 2.1 km altitude error.
  • Trajectory Forecast: Single-shot prediction keeps per-step MSE roughly flat across four forecast steps instead of compounding errors autoregressively.Longer horizons obtained by repeating successive windows would reintroduce compounding behavior in the outer loop.

C. Next Sentence Prediction

The joint space supports bidirectional voice–trajectory retrieval and organizes paired data by tower, flight phase, and broader instruction semantics. Among alignment designs, the contrastive baseline produces the most compact and coherent representation.

  • Cross-modal retrieval: Bidirectional retrieval ranks likely voice phrases from trajectory windows by cosine similarity in the joint space.Retrieved phrases are compared with ground-truth transcriptions.
  • Cross-modal retrieval: Retrieval errors show that similar flight phases often recover phrases from the same controller using the same phraseology, even when addressees differ.This indicates coarse instruction semantics accompany fine-grained identity matching.
  • Cross-modal retrieval: Tower-level R@1 is higher for stereotyped ground and delivery exchanges than for longer, more diverse NorCal departure instructions.The breakdown agrees with the cluster structure observed in the joint space.
  • Alignment ablation: Arch. 0 concentrates more variance than Arch. 2, carrying 42% in its first five directions versus 27%, and 78% in its first twenty versus 62%.Its effective rank is 203, compared with 564 for Arch. 2 and 806 for Arch. 1.
  • Alignment ablation: Contrastive supervision produces the compact alignment, whereas removing it or folding it into the flow spreads information across more directions.Voice reconstruction converges roughly 35× faster than trajectory reconstruction.
  • Latent-space analysis: UMAP projections share tower-based clusters and secondary flight-phase structure across voice and trajectory modalities without supervision on those labels.The joint space remains effectively low-dimensional despite operating in R1024, but this concentration motivates regularization against dimensional collapse.

F. Continuity in the Trajectory Latent Space

Trajectory representations are locally smooth within flight-phase clusters and transition between clusters when the aircraft changes regime. The framework uses this structure for downstream forecasting, retrieval, and prospective generative decision support.

  • Continuity: Small changes in motion produce small latent displacements, while flight-phase transitions cause jumps between trajectory clusters.A single inbound KSFO flight moves continuously through descent and landing clusters, with a jump at the phase change.
  • Continuity: This piecewise-continuous structure mirrors sequences of continuous flight regimes separated by discrete maneuver transitions.The cited example follows descent to touchdown and then continues within the landing cluster.
  • Framework role: The framework aligns ATC voice transmissions and aircraft trajectories in a single learned latent space using contrastive alignment followed by bijective lifting through normalizing flows.The two-stage design supports both shared representation learning and invertible cross-modal mappings.
  • Supported tasks: The resulting joint space supports cross-modal retrieval above random and downstream tasks such as short-horizon trajectory forecasting from voice instructions.The paper reports these capabilities for a congested terminal area using public data only.
  • Applications: Potential controller-facing uses include retrieving recent voice exchanges from an aircraft selection and flagging voice–trajectory pairs that are too far apart in the joint space.The latter is proposed as an anomaly-detection use case.
  • Limitations and future work: Path B voice-conditioned trajectory forecasting remains an order of magnitude less precise than Path A because of residual distribution shift from the bijective stage.The authors propose adversarial alignment, end-to-end predictor fine-tuning, and broader terminal-area data as possible remedies.

Appendix

The appendix documents implementation choices and evaluates trajectory and voice encoders. Trajectory backbones perform similarly, while Whisper provides the preferred voice representation and the Transformer is retained as the default trajectory encoder.

  • Implementation: The appendix specifies contrastive projectors, affine coupling flows, trajectory-encoder training, and InfoNCE optimization settings for reproduction.The documented choices include SELU projectors, eight coupling layers, AdamW optimization, and early stopping on validation loss.
  • Trajectory backbones: All four trajectory backbones achieve validation MSE values within 4 × 10^-4, indicating an architecture-agnostic MAE objective at this scale.The candidates are Transformer, BiLSTM, sLSTM, and xLSTM.
  • Trajectory backbones: The Transformer is retained because it has the lowest validation MSE and the most cleanly organized flight-phase representation, although BiLSTM is about four times cheaper per epoch.The four latent spaces otherwise show near-identical qualitative organization.
  • Trajectory latent structure: Trajectory latent spaces organize general-aviation and commercial traffic differently, with continuous general-aviation structure and tighter commercial clusters.Neighboring clusters follow flight chronology, while separate airport geography creates multiple clusters for the same phase.
  • Voice encoder: Whisper outperforms Wav2Vec 2.0 at every projector size, with validation loss becoming essentially flat at projector dimension d_j = 1024.This motivates the working configuration.

D. Affine Coupling Layer in RealNVP

The RealNVP joining stage uses repeated affine coupling layers with alternating masks to construct invertible mappings between modality representations. The associated release provides paired voice–trajectory data and metadata for training and evaluation.

  • RealNVP coupling: An affine coupling layer splits its input, leaves one half unchanged, and uses it to condition scale and translation networks acting on the other half.The inverse is analytically computable in closed form.
  • RealNVP coupling: Repeating coupling layers with alternating masks spreads the flow transformation across all dimensions and forms the two normalizing flows.The mask is swapped by half between layers.
  • Dataset release: The released dataset is organized as sharded HDF5 files per control tower with parallel audio, trajectory, metadata, and sample-index datasets.Trajectory records contain six kinematic features plus an absolute timestamp.
  • Dataset release: A HuggingFace subset contains approximately 83,223 paired audio–trajectory samples with metadata describing how each pair was linked.The release also includes repositories for preprocessing, analysis, model training, and downstream evaluation.
Loading 2608.28981v1…