Dual Variational Autoencoders for Efficient Sim-to-Real Transfer in Low-Cost Robotic Navigation

• A common representation space for simulation and limited real-world data is proposed to mitigate domain discrepancies and improve real-world generalization.The method targets vision-based tasks requiring accurate, efficient image classification, including autonomous navigation.Source: §1 · p. 2
• Domain randomization first trains on meaningful features from extensive simulated data collected in a high-fidelity indoor physics simulator.The approach then applies feature-level adaptation using two convolutional VAEs with a shared decoder, without requiring one-to-one paired samples.Source: §1 · p. 2
• The shared-decoder architecture generates similar synthetic and pseudo-synthetic data to support policy generalization to novel real scenarios.Source: §1 · p. 2
• Memory-efficient image representations are intended to support policy transfer to low-cost robots using a single cost-effective camera.Source: §1 · p. 2
• The proposed policy is validated on a low-cost physical robot in a real-time, collision-free, map-less visual exploration task in an indoor corridor.Source: §1 · p. 2

2.1. Domain Randomization

• Domain randomization varies simulated object placement, illumination, and surface textures; this work combines that source-domain training with offline model-centric adaptation using a small batch of real data.Static randomization samples from a fixed parameter distribution, while adaptive approaches refine the simulation distribution during learning.Source: §2.1 · pp. 2–3

2.2. Domain Adaptation

• Domain adaptation aligns source and target representations, and this method uses generative feature-level adaptation rather than pixel translation.Pixel-level GAN translation may alter task-critical content, whereas the proposed feature-level approach is described as a more stable way to align structural information.Source: §2.2 · p. 3

2.3. Zero-Shot Generalization

• Zero-shot approaches seek direct deployment without target-domain adaptation, using strategies such as system identification or low-fidelity simulation, each with distinct brittleness or expressivity trade-offs.The proposed hybrid strategy instead fine-tunes a pretrained VAE with a small batch of unlabeled real images.Source: §2.3 · p. 4
• Foundation models offer broad semantic performance but have memory and processing requirements described as prohibitive for low-cost, resource-constrained robots.Source: §2.3 · p. 5

2.4. Real-World Autonomous Navigation Systems

• Related navigation systems span LiDAR-based obstacle avoidance and monocular-camera doorway detection, illustrating differing sensor requirements from this work’s minimalist visual focus.Source: §2.4 · p. 5

3. Sim-to-Real Transfer with Dual Variational Autoencoders

• The dual-VAE framework shares a decoder to learn a latent space where simulated and real-image feature distributions can converge.The method combines domain randomization with feature-level domain adaptation and uses two data-augmentation techniques to refine alignment.Source: §3 · p. 5

3.1. Data Acquisition for Hybrid Sim-to-Real Transfer

• The source dataset contains 45,225 labeled synthetic images; the target dataset contains 4,556 images captured in the corresponding real-world indoor corridor for fine-tuning the simulation-pretrained model.During collision-free exploration, the simulator varies fixed obstacles, illumination, and realistic textures. The real images support domain adaptation by fine-tuning the model pre-trained on simulated data.Source: §3.1 · p. 5

3.2. Dual Encoder-Decoder Architecture for Shared Latent Space Learning

• The two VAEs use compact convolutional encoders to extract low-dimensional features, and the shared decoder is trained on simulated images before real-domain encoder adaptation.The decoder is initialized from simulation training and frozen while the second encoder is fine-tuned on real images to map them into the simulation-learned latent space.Source: §3.2 · pp. 6–7
• The VAE loss combines image-reconstruction error with KL-divergence regularization toward a standard normal latent prior.The reconstruction term compares input and reconstructed images; the KL term is described as helping prevent overfitting and promote generalization.Source: §3.2 · p. 6

3.3. Enhanced Domain Alignment via Data Augmentation

• Standard image-space transformations—random cropping and brightness, contrast, and saturation changes—expand the real-image set for intermediate encoder fine-tuning.The augmented set is used to fine-tune the encoder of the pseudo-synthetic autoencoder.Source: §3.3 · p. 8
• A temporary VAE encodes real images, after which perturbing random latent-vector elements and decoding them creates additional plausible real-image variations for final fine-tuning.The synthesized set is larger than the original real dataset and is used for final fine-tuning of the pseudo-synthetic encoder.Source: §3.3 · p. 8

4. Experiments

• The evaluation compares three policies on real test images: real-only training, simulation-only training, and simulation pretraining followed by real-data domain adaptation.The test datasets comprise labeled real images unseen during training; the evaluation also includes latent-space analysis, computational estimation, and a physical-robot exploration case study.Source: §4 · p. 8

4.1. Experimental Protocols and Architectures

• The real-world baseline learns features from 4,556 unlabeled images and trains an MLP classifier on a labeled subset of 186 images using stratified 10-fold cross-validation.The classifier has one 100-neuron hidden layer and a four-neuron output layer.Source: §4.1 · p. 8
• The simulation-only policy learns from 45,225 synthetic images and is evaluated on real scenarios without target-domain adaptation.Its classifier structure matches the real-world baseline.Source: §4.1 · p. 9
• The proposed hybrid model uses a shared-decoder dual VAE and two augmentation methods, scaling the approach to 45,225 synthetic and 4,556 real images.Adaptation fine-tunes the encoder on real images while freezing the decoder; the MLP is first trained on synthetic latent vectors and then fine-tuned on representations of 18,500 latent-augmented samples.Source: §4.1 · pp. 9–10

4.2. Training and Evaluation Datasets

• The training sets contain 45,225 simulated images for domain randomization and 4,556 labeled physical-robot images for domain adaptation.Both sets use four navigation-command labels: forward, right, left, and stop.Source: §4.2 · p. 10
• Two complementary augmentation techniques expand the real-world training data through image-space transformations and latent-space manipulation.The augmented samples support fine-tuning.Source: §4.2 · p. 10
• Evaluation uses three balanced, labeled test sets unseen during training and fine-tuning, including non-augmented real images and realistically transformed images.The non-augmented set uses 10-fold cross-validation; transformed sets apply slight random changes such as brightness, contrast, and saturation.Source: §4.2 · p. 10, Table 1 · p. 10

4.3. Training Configuration and Optimizer Selection

• RMSProp was selected for stable convergence and adaptive step sizes during fine-tuning, with a conservative learning rate supporting adaptation without catastrophic forgetting.The configured hyperparameters for initial training and fine-tuning are presented in Table 2.Source: §4.3 · p. 10

4.4. Qualitative Analysis of Reconstruction Capabilities

• The real-only and pseudo-synthetic VAEs reconstruct global structure, obstacles, and general lighting, while the simulation-only VAE produces blurrier images but preserves identifiable geometry.The first and third models have similar reconstruction quality and struggle mainly with high-frequency details such as floor texture.Source: §4.4 · p. 12, Figure 4 · p. 13

4.5. Quantitative Analysis of Classification Performance

• 90.88% mean accuracy on the (CV) test set was achieved by the pseudo-synthetic model, versus 67.16% for real-only and below 35% for simulation-only on every real test set.The pseudo-synthetic model achieved approximately 60% on the two transformed test sets, showing lower adaptation there than the real-only model.Source: §4.5 · p. 12
• The shared decoder is described as a generative constraint that anchors the latent space to simulated geometry and regularizes adaptation to the 4,556-image real dataset.The authors report 90.88% classification performance in this setting, but did not conduct a direct shared-versus-independent-decoder ablation.Source: §4.5 · pp. 12–13

4.6. Ablation Study: Impact of Data Augmentation Strategy

• Removing both augmentation stages reduced primary (CV) mean accuracy from 90.88% to 47.06%, a drop of 43.82 percentage points.Accuracy on transformed sets T1 and T2 fell to 39.15% and 34.20%, respectively.Source: §4.6 · p. 14
• Without augmentation, the shared-decoder dual VAE alone is insufficient to bridge the reality gap for limited, unpaired real-world data.The authors characterize the shared decoder as a necessary structural constraint, while attributing most high generalization capability to the augmentation stages.Source: §4.6 · p. 14

4.7. Policy Analysis for Reactive Collision-Free Navigation

• The policy predicted a turning action in a column-and-window scenario, which the authors treat as successful evasive behavior despite predicting the opposite direction.The task-oriented analysis groups right and left turns because initiating an evasive turn is more critical than its direction.Source: §4.7 · p. 15
• The policy also supported collision-free exploration in open spaces and stopping, including three consecutive stop commands when a person stood in front of the camera.The scenario used a low-cost robot exploring a real indoor corridor with a single monocular RGB camera.Source: §4.7 · pp. 14–15
• 90.88% primary (CV) test-set accuracy was maintained by the pseudo-synthetic policy under the simplified three-category evaluation.A prediction of either turn direction counted as correct when the ground truth called for a right or left turn.Source: §4.7 · p. 15

4.8.1. Qualitative Evaluation of Latent Space Organization

• t-SNE maps the autoencoders’ 32-dimensional latent vectors to 2D to visualize local relationships and clusters for the simplified forward, turn, and stop actions.Stop generally forms a clearer cluster, but turning and stopping overlap, which could cause classification errors and suboptimal exploration decisions.Source: §4.8.1 · pp. 15–18
• 90.88% mean classification accuracy accompanies the pseudo-synthetic model’s clearly separated clusters on the non-augmented real-world set, versus 72.57% for real-only and 32.92% for sim-only.The sim-only latent space is largely unstructured with substantial class overlap.Source: §4.8.1 · p. 15, Figure 7 · p. 17
• For the augmented T1 and T2 sets, real-only accuracy reaches 81.76% and 83.91%, while pseudo-synthetic clusters are less defined than on CV and classification performance is lower.The sim-only model remains poorly organized, especially on T2.Source: §4.8.1 · p. 15, Figure 8 · p. 18, Figure 9 · p. 19

4.8.2. Quantitative Evaluation of Cluster Validity

• On CV, the pseudo-synthetic autoencoder scores best across all three cluster-validity indices, while sim-only scores worst, including a negative Silhouette Index.The comparison averages 10 independent t-SNE test runs.Source: §4.8.2 · p. 20
• On T2, the pseudo-synthetic model combines exceptional cluster-validity scores with 60.78% classification accuracy, showing that geometric cluster purity need not track downstream task performance.The authors suggest augmentation-induced low-level features may form geometrically pure but semantically confused clusters.Source: §4.8.2 · p. 20

4.9. Computational Efficiency Analysis for Low-Cost Robotic Deployment

• The computational estimate is intended as a conservative, practical upper bound rather than measured hardware performance.The analysis covers a Linux laptop, Raspberry Pi 3B+, Raspberry Pi 4, and NVIDIA Jetson Nano 2 GB.Source: §4.9 · p. 20

4.9.1. Theoretical Complexity of the Proposed Architecture

• The inference model uses the encoder and MLP classifier, totaling 2,176,568 parameters, with an 8.31 MiB FP32 footprint and ~169.4 MFLOPs per 64x64 RGB image.The decoder is excluded during inference.Source: §4.9.1 · p. 21

4.9.2. Performance Estimation via the Roofline Model

• The Roofline analysis uses operational intensity and hardware performance ceilings, including intermediate ceilings for sub-optimal conditions, to estimate attainable performance.The estimates are model-based; precise ceiling values require exhaustive empirical micro-benchmarking on physical hardware.Source: §4.9 · p. 20, §4.9.2 · p. 21
• ~46 FPS on Raspberry Pi 4 and ~833 FPS on Jetson Nano are estimated, whereas Raspberry Pi 3B+ reaches ~4 FPS, below the cited 30–60 FPS range for smooth navigation.The authors identify the Pi 3B+ as unsuitable without a dedicated accelerator and the Pi 4 and Jetson as viable platforms.Source: §4.9.2 · pp. 21–22, Table 7 · p. 22

4.10.1. Experimental Setup

• The available Raspberry Pi 3B+ was estimated at ~4 FPS, so a distributed setup sent camera images to a Linux laptop for inference and returned commands to the robot.This separated policy decision-making from the robot’s hardware limitations.Source: §4.10.1 · pp. 22–23

4.10.2. Navigational Trajectory Analysis

• The robot successfully explored a significant portion of the indoor corridor while reactively avoiding varied obstacles, with a trajectory reconstructed from executed motor commands rather than pre-planned.The path included six checkpoints for examining real-time decisions.Source: §4.10.2 · p. 23

4.10.3. Reactive Behavior at Critical Checkpoints

• At six checkpoints, the robot avoided doors, columns, a bin, a chair, and another robot through turns, forward movement, and pauses without collision.The reported behaviors include entering an office and maneuvering around obstacles in a corridor.Source: §4.10.3 · pp. 23–24

4.10.4. Robustness to Environmental Variations

• Across different days, lighting, object presence, and people entering the camera view, the policy produced coherent actions, including stopping for sudden motion and turning to avoid obstacles.A predicted left turn was treated as correct when the expert indicated right because either evasive turn avoided the imminent obstacle.Source: §4.10.4 · p. 24

5. Conclusions

• A proof-of-concept physical-robot deployment performed reactive collision-free exploration, while qualitative trials report adaptation to lighting changes, novel objects, and dynamic obstacles.The qualitative evaluation compared teleoperated and autonomous navigation on two days with varying conditions.Source: §5 · p. 26, Figure 13 · p. 26
• A computational performance estimation supports the model’s suitability for resource-constrained embedded platforms.The conclusions propose future empirical inference benchmarking on Raspberry Pi 4 Model B and NVIDIA Jetson Nano 2 GB hardware.Source: §5 · p. 26
• Approximately 70% average success rate came from real-only training, while simulation-only training did not surpass a 41% average success rate on real test images.The conclusions attribute the real-only limitation to difficulties acquiring extensive, balanced labeled data, and note persistent simulation–reality discrepancies despite a high-fidelity simulator.Source: §5 · p. 26
• Nearly 91% average success rate on unseen real-world data was achieved with sequential domain randomization and feature-level adaptation using two augmentations and a shared-decoder dual VAE.The shared decoder aligns feature distributions between domains.Source: §5 · pp. 25–26
• Future real-world evaluations will test more complex, dynamic indoor environments with greater illumination variation, clutter, and dynamic obstacles.The authors also plan to benchmark inference on target low-cost hardware to obtain latency and throughput metrics.Source: §5 · pp. 26–27
• Future work proposes extending the VAE-based perception model to multiple low-cost agents for decentralized visual processing in collaborative navigation tasks.Planned tasks include coordinated exploration, target tracking, and surveillance in large-scale real-world scenarios.Source: §5 · p. 27

CRediT author statement

• Álvaro Díez is credited with conceptualization, methodology, software, validation, analysis, investigation, data curation, writing, and visualization; Fidel Aznar with conceptualization, methodology, data curation, validation, resources, supervision, and administration.Source: CRediT author statement · p. 27