Source-linked AI summary
Coastal Environment Generation with HoloOcean
Abigail Austin, Brady Moon, Joshua G. Mangelson
TL;DR
Marine robotics simulators need high-fidelity environments, but manual environment creation requires substantial time, data, and expertise. This paper presents a UE5 pipeline that generates coastal HoloOcean levels from one overhead image, combining terrain generation with automated asset selection and placement. Asset placement averaged less than 0.5 meters xy location RMSE and less than 2 degrees orientation RMSE, while the terrain results were non-ideal.
Problem
Creating high-fidelity simulation environments often requires substantial time, data, and specialized environment-system knowledge.
Method
The pipeline uses one overhead coastal image to generate terrain and semantic information, then detects and matches assets for placement in a HoloOcean environment.
Results
Asset placement averaged less than 0.5 meters xy location RMSE and less than 2 degrees orientation RMSE.
Takeaways & Limitations
The pipeline produces customized, HoloOcean-compatible coastal levels while reducing reliance on expert Unreal Engine environment-development knowledge.
Takeaways & Limitations
The Seabed-Net-generated height maps were non-ideal, particularly because they lacked gradual terrain changes between land and water.
Abstract
from arXiv · showhide
Marine robotic simulation provides a safe and inexpensive method of developing and testing algorithms for unmanned underwater vehicle (UUV) and unmanned surface vessel (USV) autonomy and perception before full field deployment. However, these simulations are often limited by the availability of simulated environments. Current marine robotics simulation suites offer manual ways to edit or create environments, but they require existing data or specialized knowledge of the environment system. To address these issues, we introduce a novel Unreal Engine 5 level generation pipeline that enables automatic creation of coastal environments for HoloOcean. Our pipeline relies on a user-provided overhead image of a coastal scene. The pipeline then uses the image to generate height map data, as well as automatically select assets and place them in the environment.
I. INTRODUCTION
Marine simulation environments support safe and inexpensive UUV and USV autonomy and perception testing, but creating high-fidelity environments often requires substantial time, data, and expertise. This work introduces an automated UE5 pipeline that builds coastal HoloOcean environments from overhead imagery.
- Environment fidelity can limit large-scale, high-fidelity marine robotics testing because environment creation is often time-, data-, and expertise-intensive.
- The proposed UE5 pipeline automatically creates coastal simulation environments for HoloOcean from a user-provided overhead image.
- The pipeline combines automatic terrain generation using semantic imagery and height-map estimation with asset selection and placement from an asset library.
- The resulting levels support customized HoloOcean environments without requiring expert Unreal Engine environment-development knowledge and remain compatible with HoloOcean sonar implementations and robotic agents.
II. RELATED WORK
HoloOcean builds on Unreal Engine tools, assets, and rendering, while related simulators offer different mechanisms for creating or customizing environments. These alternatives vary in realism, required input data, and flexibility.
- HoloOcean leverages Unreal Engine 5 tools, assets, rendering pipelines, and procedural content generation for customizable marine environments.
- Project DAVE creates heightmaps from user-provided bathymetry but lacks Unreal Engine’s visual capabilities and asset access.
- OceanSim supports scanned real-world locations, whereas Stonefish loads assets or alters environmental parameters within a predefined level.
- UNav-Sim also uses Unreal Engine 5 and provides access to Unreal Engine tools for level design.
C. Environment Generation
Procedural and language-guided systems can generate diverse simulated scenes, but replicating specific real-world coastal locations remains difficult. The proposed context centers on image-based coastal environment generation and sensor-compatible outputs.
- Infinigen generates diverse Blender scenes through procedural templates for assets, materials, and animals, with automatic object annotations for computer vision.
- SceneX and UnrealLLM use text prompts and language models to control procedural scene-generation parameters in Blender or Unreal Engine.
- Text-prompted procedural approaches are not adequate for replicating real-world locations, while other systems retrieve landscape data from GPS coordinates.
- The generated environment can be evaluated with Raycast Semantic LiDAR material identification and Imaging Sonar results.
D. DEM and Bathymetry Generation
DEM and bathymetry prediction remains challenging, with existing models often specialized to particular terrain or coastal regions. The proposed pipeline uses a base Unreal level in which generated landscapes and assets are later spawned.
- DEM and bathymetry prediction remains an ongoing challenge, and existing methods target specific terrain types such as mountains or lunar regions.
- Seabed-Net provides bathymetry and pixel-classification prediction models trained on coastal data, but its models struggle to generalize beyond their corresponding locations.
- The pipeline requires only a true or generated overhead image and produces a functional HoloOcean level through landscape generation and asset selection and placement systems.
- A mostly empty Unreal base level supplies HoloOcean weather, water, lighting, and underwater post-processing controls before landscapes and assets are spawned.
B. Image Input and Semantic Processing
Users provide a scaled aerial coastline image, which is cleaned and semantically segmented into terrain classes for landscape texturing and vegetation placement.
- A single aerial coastline image and its dimensions in meters are the pipeline’s required inputs.
- Gemini removes objects such as boats and cars before creating a semantic segmentation map.
- The map labels eight terrain types, including Asphalt, Water, Grass, Trees, Dirt, Rocks, Sand, and Sidewalk/Parking Lines.
- Semantic labels drive automatic landscape texturing and serve as a biome map for vegetation.
- Seabed-Net height maps are applied to an Unreal landscape scaled to the input image, with water spawned at z = 0.
D. Asset Library Generation
The system builds an asset library from rendered user assets, matches detected manmade objects to that library, and uses biome rules to place natural objects procedurally.
- Asset Library Generation: User-provided assets are rendered under three lighting conditions and two background colors before DINOv3 creates their embedding database.
- Asset Selection and Placement: The asset system supports manmade objects such as boats and piers and natural objects such as vegetation, trees, and rocks.
- Asset Selection and Placement: YOLO detects manmade-object locations, DINOv3 embeds object crops, and FAISS selects a random close asset match by cosine similarity.
- Asset Selection and Placement: Pixel locations are converted to Unreal coordinates, while bounding boxes determine asset scale and rotation.
- Asset Selection and Placement: Two additional YOLO passes detect pier walkways and smaller parts, which are filled by chaining a selected pier asset within the detected bounds.
- Procedural Natural Placement: The semantic map supplies a biome map whose PCG rules control vegetation location, density, type, elevation, and water-level placement.
A. Synthetic Environment Regeneration Experiments
Synthetic tests compare regenerated HoloOcean environments with manually created ground truth across resolutions, measuring placement accuracy, matching, segmentation, terrain quality, and runtime.
- Experimental Setup: Five manually created Unreal environments served as ground truth, with aerial imagery captured at two pixel resolutions for regeneration tests.
- Asset Accuracy: Asset placement averaged less than 0.5 meters xy location RMSE, while orientation RMSE stayed below 2 degrees despite high rotation RMSE.
- Resolution Effects: Increasing input resolution improved precision from 0.859 to 0.992 and recall from 0.965 to 0.978, while location error remained comparable.
- Limitations: YOLO sometimes missed complete pier bounding boxes, leaving some generated pier parts disconnected from one another or the land.
- Limitations: Semantic maps identified broad vegetation regions but lacked fine detail, causing grass to cover an entire dirt area in one example.
- Limitations: Seabed-Net produced non-ideal height maps with stark land-water transitions and no gradual terrain changes, although users could edit them.
- Runtime: Generation took approximately 20.13 seconds for a three-asset level and 5 minutes 39.2 seconds for a 96-asset level.
B. Real World Imagery Test
A real-world Hale‘iwa Boat Harbor image was hand-labeled for evaluation; the generated level matched the scene visually but struggled with occluded objects and fine object identity.
- Evaluation Setup: Aerial drone imagery of Hale‘iwa Boat Harbor was hand-labeled as ground truth for the generated level.
- Results: The generated harbor matched the real-world image closely in visuals, texture placement, and vegetation locations.
- Results: Precision and recall were slightly lower than in the simulated environments.
- Limitations: The pipeline struggled with partially obscured or closely spaced objects and sometimes selected the wrong object type despite identifying the correct class.
V. CONCLUSION
The pipeline generates coastal HoloOcean environments by combining height-map modeling, semantic segmentation, and automated asset matching. Generation takes seconds rather than the days required for manually creating complicated coastal levels, while future work targets improved terrain modeling and customization.
- The pipeline generates coastal HoloOcean environments using height maps, semantic segmentation, and automated asset matching.Semantic segmentation supports texturing and vegetation placement, while YOLO detects objects and DINOv3 selects similar assets.
- Future work will replace Seabed-Net with a custom DEM and bathymetry model trained on paired RGB imagery and DEM data.
- Generation time scales linearly with detected assets but remains on the order of seconds rather than days of manual coastal-level creation.
- Future work will also improve user customization of generated levels.