Source-linked AI summary
Towards Efficient and Effective Multi-Camera Encoding for End-to-End Driving
Jiawei Yang, Ziyu Chen, Yurong You, Yan Wang, Yiming Li, Yuxiao Chen, Boyi Li, Boris Ivanovic, Marco Pavone, Yue Wang
TL;DR
Multi-camera driving inputs contain spatial and temporal redundancy, while geometry-structured compression imposes fixed representations. Flex jointly compresses images across cameras and time into learned scene tokens; on a 20,000-hour dataset it delivers higher inference throughput and improved minADE6 versus the in-house VLA baseline.
Problem
Multi-camera and multi-timestep images contain substantial spatial and temporal redundancy, while geometry-grounded representations impose predefined structure.
Source: §I · p. 1Method
Flex jointly encodes images across cameras and timesteps into compact learnable scene tokens, reducing the policy model's token budget without explicit 3D priors.
Source: §I · pp. 1–2Results
2.2× increase in inference throughput and minADE6 improves from 0.798 to 0.761 versus the in-house VLA baseline.
Source: B. Comparisons with Prior Systems · p. 5Takeaways & Limitations
Flex's scene tokens develop an emergent semantically meaningful decomposition, and the encoder is flexible across model sizes and sensor configurations.
Source: §V · p. 8
Abstract
from arXiv · showhide
We present Flex, an efficient and effective scene encoder that addresses the computational bottleneck of processing high-volume multi-camera data in end-to-end autonomous driving. Flex employs a small set of learnable scene tokens to jointly encode information from all image tokens across different cameras and timesteps. By design, our approach is geometry-agnostic, learning a compact scene representation directly from data without relying on the explicit 3D inductive biases, such as Bird-Eye-View (BEV), occupancy or tri-plane representations, which are common in prior work. This holistic encoding strategy aggressively compresses the visual input for the downstream Large Language Model (LLM) based policy model. Evaluated on a large-scale proprietary dataset of 20,000 driving hours, our Flex achieves 2.2x greater inference throughput while improving driving performance by a large margin compared to state-of-the-art methods. Furthermore, we show that these compact scene tokens develop an emergent capability for scene decomposition without any explicit supervision. Our findings challenge the prevailing assumption that 3D priors are necessary, demonstrating that a data-driven, joint encoding strategy offers a more scalable, efficient and effective path for future autonomous driving systems.
I. INTRODUCTION
Flex addresses the visual-token bottleneck in multi-camera driving by jointly compressing images across views and time into learned scene tokens. It reports higher throughput and better driving accuracy than the baseline, without explicit 3D priors.
I. INTRODUCTION
- Thousands or tens of thousands of visual tokens per forward pass impose a significant computational load on LLM-based policy models.Multi-camera rigs collect tens of images per second, creating the high-bandwidth input stream.Source: §I · p. 1
- Spatial overlap between adjacent cameras and temporal continuity between frames create redundancy that per-image encoding fails to exploit.Naive per-image encoding also increases latency and memory.Source: §I · p. 1
- Flex reaches 41.08 vs. 18.60 clips/s and 0.76 vs. 0.80 minADE6, improving throughput and driving accuracy over the baseline.The introduction reports 2.2× inference throughput and improved driving performance on a proprietary dataset of 20,000 driving hours.Source: Figure 1 · p. 1, §I · p. 2
- Prior approaches use structured 2D or 3D representations, but their rigid geometry and fixed granularity may constrain performance.The paper motivates a simpler, data-driven alternative to explicit 3D inductive biases.Source: Figure 1 · p. 1, §I · p. 1
- Flex concatenates learnable scene tokens with image tokens from all cameras and timesteps, then jointly processes them with Transformer self-attention.It discards the image tokens afterward and passes only the updated scene tokens to the policy model.Source: §I · p. 2
- The design reduces the policy-model token budget by about 3× to 20× and coordinates redundancy suppression across the whole scene.The authors also report that the compressed tokens emerge as attending to destinations, lane markers, and safety-critical areas.Source: §I · p. 2
II. RELATED WORK
Prior end-to-end driving systems either use explicit 3D structures or leave token-level compression across cameras and time unaddressed. Flex focuses on camera encoding with jointly learned scene tokens and no fixed geometric scaffold.
II. RELATED WORK
- Planning-oriented systems such as UniAD and ST-P3 use shared queries, BEV features, or spatial-temporal features but retain explicit 3D structures and require precise camera poses.These approaches fuse perception and planning within unified networks.Source: A. End-to-end driving and camera… · p. 2
- Vectorized end-to-end approaches such as VADv2 [15] consume multi-view image sequences directly and output actions, but do not address aggressive token compression across cameras and timesteps.Source: A. End-to-end driving and camera… · p. 2
- Flex complements these systems by learning compact scene tokens jointly across views and time without imposing a geometric scaffold.Its focus is the camera-encoding stage.Source: A. End-to-end driving and camera… · p. 2
B. Scene representations for autonomous driving
Geometry-grounded representations compress camera inputs but rely on predefined structure, pose information, and fixed granularity. Flex instead uses learned queries over the full image set, while its scene-token design follows the latent-query paradigm with joint, driving-oriented encoding.
B. Scene representations for autonomous driving
- The volume of camera inputs creates a computation bottleneck, prompting methods to use geometry-grounded representations for compression.Source: B. Scene representations for… · p. 2
- BEV, occupancy, and neural-field methods predefine information structure and require accurate camera poses; fixed grid or voxel granularity cannot adapt to perspective-dependent information density.Near objects occupy many pixels while distant objects may collapse into very few.Source: B. Scene representations for… · p. 2
- Flex minimizes geometry-specific priors and uses joint attention over the full image set to discover structure and adapt to non-uniform image density.Its learned scene queries do not commit to a fixed 3D decomposition.Source: B. Scene representations for… · p. 2
C. Learned representations
- Unlike prior latent-query methods, Flex jointly attends across all cameras and timesteps with self-attention, and its overview shows only the compact scene-token set passed onward as the scene representation.Special tokens encode modality boundaries, camera type, and timestamps.Source: C. Learned representations · p. 2, Figure 2 · p. 3
III. METHOD
Flex is a compact scene encoder for VLA driving policies, intended to improve training and inference efficiency while enhancing driving performance. The described baseline feeds processed multi-camera, multi-timestep image tokens and ego-history information to an autoregressive policy model that predicts a future trajectory.
III. METHOD
- Flex produces a compact visual scene representation for VLA policies, with the stated aim of improving training and inference efficiency and driving performance.Source: §III · p. 3
A. Vision-Language-Action Model
- The baseline VLA model processes RGB-image sequences and past trajectory data to autoregressively predict a future ego-vehicle trajectory.The observations include images from multiple cameras and past timesteps, alongside recent ego-state history.Source: A. Vision-Language-Action Model · p. 3
- The baseline patchifies each image, downsamples its tokens, projects them to the LLM hidden dimension, and concatenates tokens across cameras and timesteps.The long sequence imposes significant computational load on the policy head; trajectory history is encoded into one embedding, while future waypoints become discrete tokens predicted autoregressively.Source: A. Vision-Language-Action Model · p. 3
B. Flex Scene Encoder
Flex addresses spatial and temporal redundancy by jointly encoding image tokens across cameras and timesteps into a compact scene representation. A lightweight Transformer updates learnable scene tokens through full self-attention, and only those tokens are passed to the policy model.
B. Flex Scene Encoder
- Jointly encoding all views and timesteps lets Flex address redundancy from camera overlap and temporal continuity in a compact scene representation.The representation is designed to promote efficiency and a more effective scene representation.Source: B. Flex Scene Encoder · p. 3
- Timestep and camera embeddings are added to image tokens to encode their temporal and camera-type positions.Camera embeddings are learnable vectors indexed by camera type.Source: B. Flex Scene Encoder · p. 4
- Learnable scene tokens are prepended to all image tokens, and an 8-layer Transformer computes full self-attention over the combined sequence.The scene tokens act as queries.Source: B. Flex Scene Encoder · p. 4
- After the final layer, Flex discards the image tokens and uses the updated scene tokens as the representation passed to the policy model.The scene tokens are linearly projected to the policy model's embedding dimension.Source: B. Flex Scene Encoder · p. 4
C. Interleaved Prediction
Interleaved prediction increases supervision by training on prefixes of varying lengths rather than applying loss only at the final timestep. Flex partitions its holistic scene representation into sequential chunks, whose specialization emerges during end-to-end training.
C. Interleaved Prediction
- Training only on the final timestep provides limited supervision, so interleaved prediction supervises prefixes of varying lengths.Each prefix is paired with its trajectory history and used to predict the next H future steps, producing T supervision signals per sequence.Source: C. Interleaved Prediction · p. 4
- The interleaved strategy is implemented by adjusting the attention mask without splitting the sequence.Past history and future trajectory tokens in the context are masked from contributing to the final prediction.Source: C. Interleaved Prediction · p. 4
- Flex evenly partitions its holistic scene tokens into T sequential, non-overlapping chunks, conditioning step k on the first k chunks and the most recent history token.The jointly encoded representation has no explicit mapping to a specific camera or time.Source: C. Interleaved Prediction · p. 4
- End-to-end training yields emergent chunk specialization, with different chunks implicitly capturing information needed for decision-making at each step.The partition provides no explicit guidance to the encoder.Source: C. Interleaved Prediction · p. 4
IV. EXPERIMENTS
The experiments evaluate Flex on a geographically held-out portion of a large, diverse driving-log dataset and compare efficiency and driving performance against prior systems. The system-level comparison reports a superior effectiveness–efficiency trade-off.
IV. EXPERIMENTS
- The internal dataset contains 20,000 driving hours from over 1,700 cities in 25 countries, with a geographically separate 90% training/validation and 10% test split.Although vehicles have seven cameras, the experiments primarily use front-wide and front-telephoto cameras; evaluation uses nine timesteps.Source: A. Setup · pp. 4–5
- Flex achieves the best effectiveness–efficiency trade-off in the system-level comparison across error, policy input length, training cost, and inference throughput.The table caption describes lower error, fewer tokens, faster training, and higher throughput for Flex.Source: Table I · p. 5
- Effectiveness is measured using standard minADEk, the minimum Average Displacement Error over k predicted trajectories (k = 6), while inference efficiency is measured in clips processed per second.Source: A. Setup · p. 5
- The in-house baseline feeds the full flattened multi-view, multi-timestep image-token sequence to the policy model and otherwise matches Flex's policy head, trajectory tokenizers, and training schedule unless noted.The experiments also include variations in prediction strategy and camera-by-timestep choices.Source: A. Setup · p. 5
B. Comparisons with Prior Systems
On the held-out test set, Flex improves inference throughput and driving performance over the in-house VLA model while using less training time.
B. Comparisons with Prior Systems
- 2.2× higher inference throughput and minADE6 improved from 0.798 to 0.761 versus the in-house VLA model; Flex uses about 60% of its total training time.The paper attributes the comparison to the model's compression bottleneck for policy inputs.Source: B. Comparisons with Prior Systems · p. 5
C. Ablation Study
The ablations test Flex’s encoder and training choices, revealing trade-offs among accuracy, throughput, and design. Joint scene encoding and interleaved prediction outperform alternatives, while the default configuration lies near the accuracy–throughput frontier.
Default configuration
- DINOv2-Base balances performance and latency; the default uses 900 tokens, eight joint-self-attention layers, and two front cameras, matching the baseline setup.The patchifier comparison also found further accuracy gains when moving from Base to Large, at additional computational cost.Source: C. Ablation Study · p. 5
Token count and encoder depth
- minADE6 improves nearly monotonically with token count, saturating around K ≈900, while throughput falls inversely with K; performance gains from encoder depth level off beyond eight layers.The study tests 1, 2, 4, 8, and 16 layers and adopts eight by default.Source: C. Ablation Study · p. 5
Overall ablation results
- 2×–3×: Across ablation settings, Flex improves both accuracy and throughput versus the baseline.Flex data points require roughly 650 to 1,000 A100 GPU hours to train, compared with around 1,300 to 1,800 GPU hours for baselines.Source: Figure 4 · p. 6
Scene encoder design
- Joint scene-token designs outperform per-image tokens, and self-attention variants outperform cross-attention within both design families.The comparisons suggest that image-token interaction mediated through scene tokens is important for effective representations.Source: C. Ablation Study · p. 6
Interleaved prediction
- Interleaving lowers baseline minADE6 from 0.894 to 0.860 and Flex minADE6 from 0.991 to 0.833, with Flex at 41 clips/s.The strategy predicts from partial contexts and uses intermediate ground truth for supervision.Source: C. Ablation Study · p. 6
Camera count
- 2.2× to 3.4×: Flex’s throughput advantage over baselines grows from 2 to 7 cameras while maintaining strong driving performance.Baseline performance degrades significantly as camera count increases, while Flex remains stable; the test split does not specifically emphasize rare scenarios requiring side or rear cameras.Source: C. Ablation Study · p. 6
D. Emerging Scene Decomposition and Intention Tokens
Flex’s scene tokens show specialization across driving-scene elements, and these behaviors emerge without handcrafted priors or explicit supervision. The analysis ranks tokens by their strongest attention to image tokens and inspects representative patterns.
Token-response analysis
- Final-layer scene-to-image attention is averaged over heads, then each token’s maximum image-token response ranks tokens for analysis.The ranked responses are used for quantitative analysis and visualization of representative examples.Source: D. Emerging Scene Decomposition and… · p. 6
Emergent token specialization
- Ranks 1–3 focus on the destination, mid-ranked tokens scan ahead, Rank 822 highlights lane markings, and Rank 900 appears to capture positional bias.The caption notes that stronger average token response indicates stronger attention to image tokens.Source: D. Emerging Scene Decomposition and… · pp. 6–7, Figure 5 · p. 7
- Without handcrafted priors or explicit supervision, joint attention yields tokens for destination, motion anticipation, road layout, and positional context.The paper describes this emergent decomposition as partially explaining why the representation remains compact yet effective.Source: D. Emerging Scene Decomposition and… · pp. 7–8
V. CONCLUSION
Flex is a lightweight, geometry-agnostic Transformer that jointly encodes multi-view, multi-timestep images into compact learnable scene tokens. The paper reports that holistic encoding is critical to its success and that the tokens develop an emergent semantic decomposition.
Conclusion
- Flex jointly encodes multi-view, multi-timestep images into compact learnable scene tokens using a lightweight, geometry-agnostic Transformer.The conclusion describes Flex as simple, efficient, and data-driven for end-to-end autonomous driving.Source: §V · p. 8
- The studies confirm that holistic joint encoding is critical to Flex’s success and that its scene tokens learn an emergent, semantically meaningful decomposition.The conclusion also reports flexibility with respect to model size and sensor configuration.Source: §V · p. 8
Limitations and future directions
- The proprietary dataset may underrepresent rare long-tail scenarios; broader evaluation and consideration of safety and fairness across geographies remain future directions.The conclusion also identifies interpretability and token specialization as directions for future study.Source: §V · p. 8