One of the major topics of developing an open world game is the terrain. It is the foundation on which all the beautiful details of the world are placed. In a streaming 3D globe, the terrain data is not fixed; raw terrain is streamed and loaded on-the-fly. With the exception of 3D Tiles, which I'll talk about later. This fundamentally changes the approach and makes the development of the 3D Earth unlike that of a typical game.
The general steps for rendering terrain (without vegetation) are:
Loading Terrain Data From Disk
Terrain data (or digital elevation data as its known in the geomatics world) is used across many different industries and comes in a variety of formats. Each format has its own pros and cons for the given purpose.
Digital Elevation Models (DEMs) are the simplest format of elevation data. NASA developed the elevation data sets from the Shuttle Radar Topography Mission (SRTM) in 2000 that still have tremendous value today. The file extension that the NASA data managers use is a simple binary raster format in a standardized grid encoded in a file with extension HGT. This data can be used directly in a game environment but custom code is required to load it and push it to the GPU. Once the data is on the GPU, it’s up to the rendering pipeline to render the terrain on the screen. However, it’s more typical to see heightmaps in games.
A heightmap is a raster grid of heights relative to some height datum. In the world of geomatics, the reference height datum is typically an ellipsoid the approximate shape of Earth. When rendering, the height is added/subtracted from the ellipsoid height to get the final height of the terrain at that particular XYZ coordinate.
The general steps for rendering terrain (without vegetation) are:
- Load the terrain data from disk
- Arrange the data in memory
- Push the data to the GPU
- Generate geometry and refine
- Push the data into the rendering pipeline
- Geo morph in the vector shader
- Render using Physically Based Rendering (PBR) in the pixel shader
Loading Terrain Data From Disk
Terrain data (or digital elevation data as its known in the geomatics world) is used across many different industries and comes in a variety of formats. Each format has its own pros and cons for the given purpose.
- DEM
- heightmap
- quantized mesh
Digital Elevation Models (DEMs) are the simplest format of elevation data. NASA developed the elevation data sets from the Shuttle Radar Topography Mission (SRTM) in 2000 that still have tremendous value today. The file extension that the NASA data managers use is a simple binary raster format in a standardized grid encoded in a file with extension HGT. This data can be used directly in a game environment but custom code is required to load it and push it to the GPU. Once the data is on the GPU, it’s up to the rendering pipeline to render the terrain on the screen. However, it’s more typical to see heightmaps in games.
A heightmap is a raster grid of heights relative to some height datum. In the world of geomatics, the reference height datum is typically an ellipsoid the approximate shape of Earth. When rendering, the height is added/subtracted from the ellipsoid height to get the final height of the terrain at that particular XYZ coordinate.
Example Heightmap
One can think of rendering a heightmap as extruding the height from the flat map. This has a drawback in that you can’t model caves, holes or other interesting terrain phenomenon. However, for the majority of cases, heightmaps work just fine and developers can use other techniques to model these features (e.g., clipping a volume of the terrain and inserting your own 3D model in its place).
Heightmaps are often encoded as images, though as mentioned, they don’t have to be as we saw with DEMs. When encoded as an image, a heightmap’s colours must be decoded to an actual height. There are two main methods for this encoding scheme:
RGB heightmaps typically encode the data in a weighted fashion where the major height distance is encoded in the red channel, the medium height distance is encoded in the green channel and the fine height distance is represented in the blue channel. A formula to decode RGB heightmaps is typically provided by the data supplier. For example, Mapbox’s RGB heightmaps use an encoding/decoding formula, which can be found at https://docs.mapbox.com/data/tilesets/reference/mapbox-terrain-rgb-v1/.
Heightmaps are often encoded as images, though as mentioned, they don’t have to be as we saw with DEMs. When encoded as an image, a heightmap’s colours must be decoded to an actual height. There are two main methods for this encoding scheme:
- RGB
- Grayscale
RGB heightmaps typically encode the data in a weighted fashion where the major height distance is encoded in the red channel, the medium height distance is encoded in the green channel and the fine height distance is represented in the blue channel. A formula to decode RGB heightmaps is typically provided by the data supplier. For example, Mapbox’s RGB heightmaps use an encoding/decoding formula, which can be found at https://docs.mapbox.com/data/tilesets/reference/mapbox-terrain-rgb-v1/.
RGB heightmap
Grayscale heightmaps are similar to RGB but they effectively only use one channel, which is usually encoded using 8 or 16 bits. The heightmap designer trades off height resolution with total size of the file depending on their needs.
Grayscale Heightmap
Another downside to heightmaps is they are raster. I.e., they are represented in a grid format where every row/column has a height value. Well, if you have a very flat section of terrain, the heightmap will contain redundant information. This is where Triangulated Irregular Networks (TINs) come in. TINs only contain data where there are actual changes as described at https://gistbok.ucgis.org/bok-topics/triangular-irregular-network-tin-models.
Triangulated Irregular Network (TIN)
Cesium uses TINs in their quantized mesh format. The data is encoded using ints. This format of terrain also supports the encoding of normals in line with their modus operandi to encode data such that it is ready for rendering.
Another popular format of terrain data is point clouds, in particular Light Detection and Ranging (LiDAR). A point cloud is just like it sounds, a bunch of 3D points floating around in air. Once these points are processed, they are translated into a geodetic 3D point and can even be encoded using heightmaps (though they often stay as point clouds). When they are encoded as heightmaps, LiDAR data is often encoded in a format that holds more resolution than an image heightmap, e.g., grid float. Grid float files are similar to digital elevation models but with a finer resolution because 32 bit floats are used to encode the height value instead of an 8 or 16 bit integer.
Another popular format of terrain data is point clouds, in particular Light Detection and Ranging (LiDAR). A point cloud is just like it sounds, a bunch of 3D points floating around in air. Once these points are processed, they are translated into a geodetic 3D point and can even be encoded using heightmaps (though they often stay as point clouds). When they are encoded as heightmaps, LiDAR data is often encoded in a format that holds more resolution than an image heightmap, e.g., grid float. Grid float files are similar to digital elevation models but with a finer resolution because 32 bit floats are used to encode the height value instead of an 8 or 16 bit integer.
LiDAR Point Cloud
One could imagine using LiDAR data in a game could be really cool because you get a super high-res scan of the real world. If you add an optical camera to this environment and you texture the object then, absolutely, that scanned object (with a little pre-processing) can be used in a game engine environment. Somewhat obvious though, LiDAR data is quite large so you are sacrificing space on disk and loading time for the sake of high resolution.
LiDAR is great for capturing key objects in the world whose original form should be preserved in electronic format. However, for generic features like a forest on a mountainside, it doesn’t make sense esp. if the forest is only ever seen at a distance.
Imagery
Normally at this point we would talk about texturing the terrain. But if we take a step back to 3D globes, this is usually discussed as imagery. When satellite imagery first became available in the 1960s, it revolutionized the way we look at our planet. And then when Google Earth hit the streets in 2005 and put satellite imagery at the fingertips of the average Joe, it was mind blowing. Nowadays we refer to this imagery as “aerial” because a lot of the higher resolution data is collected by planes flying tandem LiDAR-optical camera sensor systems often over cities.
Once the data is collected, it must be processed as I touched on in my It’s Your Planet blog post. This includes reprojecting the data into a map projection suitable to the end use. In Google’s case, they popularized the use of the Web Mercator projection because it represents the imagery in linear units as well as being quick and easy to render. Due to their widespread availability, Web Mercator imagery and maps can be useful to render in a 3D globe. The trick is globe coordinates are geodetic by nature so the Web Mercator images must be “warped” to fit the 3D terrain tile. For simple broad-area views from space, it is fine to do the reprojection from Web Mercator to geographic on the GPU in a shader. However, for more detailed scenes at the surface, Web Mercator will contain visible oddities so is not appropriate. To achieve the reprojection, UVs are calculated at each ECEF XYZ vertex of a “template” tile mesh.
LiDAR is great for capturing key objects in the world whose original form should be preserved in electronic format. However, for generic features like a forest on a mountainside, it doesn’t make sense esp. if the forest is only ever seen at a distance.
Imagery
Normally at this point we would talk about texturing the terrain. But if we take a step back to 3D globes, this is usually discussed as imagery. When satellite imagery first became available in the 1960s, it revolutionized the way we look at our planet. And then when Google Earth hit the streets in 2005 and put satellite imagery at the fingertips of the average Joe, it was mind blowing. Nowadays we refer to this imagery as “aerial” because a lot of the higher resolution data is collected by planes flying tandem LiDAR-optical camera sensor systems often over cities.
Once the data is collected, it must be processed as I touched on in my It’s Your Planet blog post. This includes reprojecting the data into a map projection suitable to the end use. In Google’s case, they popularized the use of the Web Mercator projection because it represents the imagery in linear units as well as being quick and easy to render. Due to their widespread availability, Web Mercator imagery and maps can be useful to render in a 3D globe. The trick is globe coordinates are geodetic by nature so the Web Mercator images must be “warped” to fit the 3D terrain tile. For simple broad-area views from space, it is fine to do the reprojection from Web Mercator to geographic on the GPU in a shader. However, for more detailed scenes at the surface, Web Mercator will contain visible oddities so is not appropriate. To achieve the reprojection, UVs are calculated at each ECEF XYZ vertex of a “template” tile mesh.
256x256 Tile Mesh with Web Mercator Image Warped Onto It
The [default] rendering pipeline will then render the mesh using these UVs.
Having said all that, if we always plan to render our world in 3D, there’s no sense processing our data in Web Mercator. It would be much more effective to simply encode the imagery in geographic coordinates, which is what Bending Time does.
One special case to note is the polar regions of Earth. Geographic coordinates are not very useful in the polar regions. As mentioned in my It’s Your Planet blog post, Bending Time may end up using polar stereographic coordinates for the polar regions. For now, let’s ignore the poles.
Features
At risk of diverging from the subject matter of terrain, I want to talk about feature data briefly. In a 3D globe application, it is very typical to load and render map data in layers starting with terrain, then draping on imagery followed by rendering vector features in order of polygons, polylines and points using a styling of choice. A feature in this case could be a lake (polygon), road (polyline) or the location of a manhole cover in a city (point).
Having said all that, if we always plan to render our world in 3D, there’s no sense processing our data in Web Mercator. It would be much more effective to simply encode the imagery in geographic coordinates, which is what Bending Time does.
One special case to note is the polar regions of Earth. Geographic coordinates are not very useful in the polar regions. As mentioned in my It’s Your Planet blog post, Bending Time may end up using polar stereographic coordinates for the polar regions. For now, let’s ignore the poles.
Features
At risk of diverging from the subject matter of terrain, I want to talk about feature data briefly. In a 3D globe application, it is very typical to load and render map data in layers starting with terrain, then draping on imagery followed by rendering vector features in order of polygons, polylines and points using a styling of choice. A feature in this case could be a lake (polygon), road (polyline) or the location of a manhole cover in a city (point).
Vector Tile Features
There are a variety of formats for feature files. A GIS standard that has seen heavy usage for decades in the ArcGIS community is the shapefile. More recently, Google’s protocol buffers are very efficient and are used as part of Mapbox’s vector tile format.
The reason for bringing vector features into the discussion is they are a staple of 3D globes but they can also be very useful for rendering things on top of our terrain like streets for example. Imagine loading a vector tile containing a country road. The data is loaded, sent to the graphics card and then with some heuristics (or actual data depending on the source) a 3D road is rendered on top of terrain that has been automatically flattened to avoid z-fighting. I will talk about this more in future blog posts.
3D Tiles
An emerging file format is the 3D Tiles specification championed by Patrick Cozzi and the folks at Cesium. In layman terms, 3D Tiles essentially contain the terrain data, imagery and features (3D objects encoded in glTF format in this case) all in one tile. Their raison d’être is to contain everything you need to render the data in a 3D engine quickly without any preprocessing steps. 3D tiles are quite effective but Bending Time will not use them for the relative near term. Let me explain.
To build 3D tiles for a virtual Earth, you need to acquire all the data that is required for your virtual world. This includes terrain data, LiDAR scans of the surface and imagery, which is then packaged up and copied to the cloud for streaming by 3D virtual world clients. Now imagine you want to change something in one of the tiles or worse, you have a new collection that you want brought in. The single tile or the entire set must be re-generated and re-deployed to the cloud. This is time consuming and expensive, and we’re still early in the whole 3D-model-of-the-Earth-down-to-the-sidewalk-level epic journey.
This is not just a casual observation. This is a painful lesson I learned during the CP 140 Aurora trainer simulator program. The overall training simulator was required to train all the surveillance operator roles onboard the Aurora aircraft. Instead of developing redundant sets of terrain data for each sensor simulator, one “terrain database” was developed. The database was generated by taking input data in the format of DEMs, imagery and vector features and then terrain generation software was used to procedurally generate the 3D data from the 2D vector features. E.g., a 3D road would be generated from the 2D road data. TerraTools is an application from the military simulation world to do this work. The problem arose trying to deal with changes. As you could imagine, 20 or so years ago the process of generating a terrain database took a long time to complete. I believe it started out taking about a week to generate and then was “optimized” to 2 days. When changes would come in (whether from the customer, source map data or whatever), the whole database had to be re-generated. You can try to minimize the effects but the kicker is this problem has a snowball effect. The more you add to it, the bigger it grows but also the faster the growth rate. For the Aurora program, this had significant cost and schedule effects. It was a lesson that stuck with me.
I call the process of encoding features into the final 3D tiles “baking.” The source data is baked into the final product. One alternative to baking is try to do the procedural generation work at runtime. This is not an easy task made obvious by the fact it used to take up to a week to generate a terrain database 20 years ago! However, proc gen at runtime has the benefit of only working on the data that is loaded, which is a very small subset of the total data available for the world. In addition, technology is so advanced today that the latest powerful GPUs can do a huge amount of processing in real-time.
For Bending Time, early on we did some experimenting on procedurally generating buildings and roads, which is an area I hope to continue once the base, natural world is “complete.”
Circling back to 3D Tiles, I’m not saying they’re no good. It’s more the opposite, they are an excellent step towards an open 3D Earth. The point is to avoid “going all in” on 3D Tiles at such an early stage of a multi-decade effort.
Arrange Data in Main Memory
Once tiles are loaded in memory, they must be organized for quick and easy retrieval. A common spatial index is the quadtree, where each level of tiles is divided into four quadrants. This works well for tiles in geographic coordinates esp. if you view the typical map of Earth where the x-axis represents the longitude and the y-axis represents the latitude (notwithstanding the irregularity at the poles).
The reason for bringing vector features into the discussion is they are a staple of 3D globes but they can also be very useful for rendering things on top of our terrain like streets for example. Imagine loading a vector tile containing a country road. The data is loaded, sent to the graphics card and then with some heuristics (or actual data depending on the source) a 3D road is rendered on top of terrain that has been automatically flattened to avoid z-fighting. I will talk about this more in future blog posts.
3D Tiles
An emerging file format is the 3D Tiles specification championed by Patrick Cozzi and the folks at Cesium. In layman terms, 3D Tiles essentially contain the terrain data, imagery and features (3D objects encoded in glTF format in this case) all in one tile. Their raison d’être is to contain everything you need to render the data in a 3D engine quickly without any preprocessing steps. 3D tiles are quite effective but Bending Time will not use them for the relative near term. Let me explain.
To build 3D tiles for a virtual Earth, you need to acquire all the data that is required for your virtual world. This includes terrain data, LiDAR scans of the surface and imagery, which is then packaged up and copied to the cloud for streaming by 3D virtual world clients. Now imagine you want to change something in one of the tiles or worse, you have a new collection that you want brought in. The single tile or the entire set must be re-generated and re-deployed to the cloud. This is time consuming and expensive, and we’re still early in the whole 3D-model-of-the-Earth-down-to-the-sidewalk-level epic journey.
This is not just a casual observation. This is a painful lesson I learned during the CP 140 Aurora trainer simulator program. The overall training simulator was required to train all the surveillance operator roles onboard the Aurora aircraft. Instead of developing redundant sets of terrain data for each sensor simulator, one “terrain database” was developed. The database was generated by taking input data in the format of DEMs, imagery and vector features and then terrain generation software was used to procedurally generate the 3D data from the 2D vector features. E.g., a 3D road would be generated from the 2D road data. TerraTools is an application from the military simulation world to do this work. The problem arose trying to deal with changes. As you could imagine, 20 or so years ago the process of generating a terrain database took a long time to complete. I believe it started out taking about a week to generate and then was “optimized” to 2 days. When changes would come in (whether from the customer, source map data or whatever), the whole database had to be re-generated. You can try to minimize the effects but the kicker is this problem has a snowball effect. The more you add to it, the bigger it grows but also the faster the growth rate. For the Aurora program, this had significant cost and schedule effects. It was a lesson that stuck with me.
I call the process of encoding features into the final 3D tiles “baking.” The source data is baked into the final product. One alternative to baking is try to do the procedural generation work at runtime. This is not an easy task made obvious by the fact it used to take up to a week to generate a terrain database 20 years ago! However, proc gen at runtime has the benefit of only working on the data that is loaded, which is a very small subset of the total data available for the world. In addition, technology is so advanced today that the latest powerful GPUs can do a huge amount of processing in real-time.
For Bending Time, early on we did some experimenting on procedurally generating buildings and roads, which is an area I hope to continue once the base, natural world is “complete.”
Circling back to 3D Tiles, I’m not saying they’re no good. It’s more the opposite, they are an excellent step towards an open 3D Earth. The point is to avoid “going all in” on 3D Tiles at such an early stage of a multi-decade effort.
Arrange Data in Main Memory
Once tiles are loaded in memory, they must be organized for quick and easy retrieval. A common spatial index is the quadtree, where each level of tiles is divided into four quadrants. This works well for tiles in geographic coordinates esp. if you view the typical map of Earth where the x-axis represents the longitude and the y-axis represents the latitude (notwithstanding the irregularity at the poles).
Quadtree Spatial Structure
One downside to the quadtree is it doesn’t balance the tree based on its contents. I.e., if the loaded data is all in one particular region of the world, the tree will be very dense for that region but sparse in other areas. This causes an unbalanced tree, which can result in slower data retrieval. This is where R-trees come in. They organize the data in rectangles based on the density of data per region. The Wikipedia page at https://en.wikipedia.org/wiki/R-tree describes R-trees clearly and concisely.
R-tree Spatial Structure
The downside, at least in terms of a virtual world, is the rectangles in play at runtime can change based on the data that is loaded. Some algorithms (e.g., terrain geo morphing) rely on the grid structure being known ahead of time.
In a 3D virtual world, if retrieving a tile takes 500 microseconds instead of 200, it’s not a big deal. To avoid dropping frames at render time, tile rendering may be spread across many frames. In this case, the extra microseconds can be easily handled.
Now that we can easily identify tiles, we need to figure out when to load new tiles and when to drop old tiles; the key function of a 3D globe.
An easy way to conceptualize the process is the virtual world app maintains the World around the Observer regardless of scale. So if the Observer is on the surface of the Earth, the app loads fine data in the immediate vicinity around the Observer, and then increasingly coarser data the farther away from the Observer you go. As long as what the Observer looks at appears like it would in real life. Nothing is pixelated.
In order to preserve memory, tiles that are no longer needed for the current Observer view must be removed from main memory. In a nutshell, once a tile is far enough away from the Observer such that it no longer contributes to their view, it is removed from memory. The 3D Engine Design for Virtual Globes book refers to this as screen-space error. However, I prefer to think of it as an Observer’s view resolution. I.e., the finest resolution object an Observer can see at their current location. If a tile’s resolution is smaller than the view resolution, it is removed from memory.
Push Data to GPU
Now that we are loading and removing tiles in main memory, we need to get the data to the graphics card for rendering.
A typical flow for getting geometry from main memory to video memory is to make a draw call with a buffer as described at https://www.khronos.org/opengl/wiki/Shader_Storage_Buffer_Object.
In a 3D virtual world, if retrieving a tile takes 500 microseconds instead of 200, it’s not a big deal. To avoid dropping frames at render time, tile rendering may be spread across many frames. In this case, the extra microseconds can be easily handled.
Now that we can easily identify tiles, we need to figure out when to load new tiles and when to drop old tiles; the key function of a 3D globe.
An easy way to conceptualize the process is the virtual world app maintains the World around the Observer regardless of scale. So if the Observer is on the surface of the Earth, the app loads fine data in the immediate vicinity around the Observer, and then increasingly coarser data the farther away from the Observer you go. As long as what the Observer looks at appears like it would in real life. Nothing is pixelated.
In order to preserve memory, tiles that are no longer needed for the current Observer view must be removed from main memory. In a nutshell, once a tile is far enough away from the Observer such that it no longer contributes to their view, it is removed from memory. The 3D Engine Design for Virtual Globes book refers to this as screen-space error. However, I prefer to think of it as an Observer’s view resolution. I.e., the finest resolution object an Observer can see at their current location. If a tile’s resolution is smaller than the view resolution, it is removed from memory.
Push Data to GPU
Now that we are loading and removing tiles in main memory, we need to get the data to the graphics card for rendering.
A typical flow for getting geometry from main memory to video memory is to make a draw call with a buffer as described at https://www.khronos.org/opengl/wiki/Shader_Storage_Buffer_Object.
OpenGL’s Shader Storage Buffer Object (SSBO)
In Unity, this is done using the ComputeBuffer object passed to a call to a compute shader’s SetBuffer() function. The whole usage of compute shaders in Unity is described well at https://catlikecoding.com/unity/tutorials/basics/compute-shaders/.
Compute Shader for Geometry and Refinement
Once the terrain data is loaded onto the GPU, we execute a compute shader to refine the terrain data to fill in the gaps between what the Observer sees and the original resolution of the DEM tile. Brano Kemen of Outerra describes this process quite well at https://outerra.blogspot.com/2009/02/procedural-terrain-algorithm.html.
This is equivalent to the Tessellation render stage however, this stage typically applies a smoothing algorithm whereas we generally want the terrain to look rough, i.e., rocky, depending on the landform being rendered.
Bending Time has implemented compute shaders to receive the terrain data and output the corresponding mesh buffers, which are then read in latter stages of the rendering pipeline. The fractal refinement has been experimented with but a final GPU-based solution has not been implemented yet. This will be one of the first tasks when I get back to development.
There are alternatives to using Compute shaders. Bending Time originally experimented with Geometry shaders to generate the terrain mesh. This was achievable and looked promising however, since then, the increasing flexibility of Compute shaders and buffers makes them the clear choice to transform terrain data from disk into mesh data on the GPU.
Push Data Through Pipeline
Pushing the data through the pipeline using Unity is as simple as creating a MeshRenderer with a shader that reads the compute buffers from before. Unity executes the shader, which normally contains vertex and fragment sections to output the vertex position and pixel colour respectively.
As the terrain rendering in Bending Time gets more advanced, shaders at different stages will be investigated.
Geo Morph in the Vertex Shader
In a 3D virtual globe application, tiles are loaded and dropped all the time as the user moves around the world. A well-known issue with this is the terrain can “pop” into or out of the scene, which can be somewhat abrasive to the Observer. To ease these transitions, a technique called geo-morphing was created.
Bending Time has not implemented geo-morphing yet but has investigated the solution space and determined the vertex shader is the best place to implement this transition. The Continuous Distance-Dependent Level of Detail paper found at https://hhoppe.com/svdlod.pdf describes the approach.
Physically-Based Rendering
One of the last stages of the render pipeline is the pixel shader. As described in Microsoft’s Direct3d documentation, the pixel-shader stage (PS) enables rich shading techniques such as per-pixel lighting and post-processing. The most common approach these days is to use Physically-Based Rendering (PBR) shading techniques.
Bending Time implemented a PBR pipeline for its 3D ocean but this has not been applied to terrain yet. However, I’ll spend the remainder of this last section describing the planned approach.
One of the starting points for rendering an object using PBR is the albedo. From Wikipedia, albedo is the fraction of sunlight that is diffusely reflected by a body. In games, the albedo texture is often just the base colours excluding normals, etc. However, NASA measures albedo on the Earth’s surface as just the amount of light being reflected, not its colour. For our purposes here, I will think of the albedo for Bending Time’s terrain incorporating both the colour and [base] intensity. Maybe this is how everyone thinks of it, I dunno.
The obvious choice for albedo is from the aerial imagery. This is a good starting point but I suspect the solution for albedo will involve more steps.
One of those steps could be the incorporation of land use imagery, also known as land cover classification.
Compute Shader for Geometry and Refinement
Once the terrain data is loaded onto the GPU, we execute a compute shader to refine the terrain data to fill in the gaps between what the Observer sees and the original resolution of the DEM tile. Brano Kemen of Outerra describes this process quite well at https://outerra.blogspot.com/2009/02/procedural-terrain-algorithm.html.
This is equivalent to the Tessellation render stage however, this stage typically applies a smoothing algorithm whereas we generally want the terrain to look rough, i.e., rocky, depending on the landform being rendered.
Bending Time has implemented compute shaders to receive the terrain data and output the corresponding mesh buffers, which are then read in latter stages of the rendering pipeline. The fractal refinement has been experimented with but a final GPU-based solution has not been implemented yet. This will be one of the first tasks when I get back to development.
There are alternatives to using Compute shaders. Bending Time originally experimented with Geometry shaders to generate the terrain mesh. This was achievable and looked promising however, since then, the increasing flexibility of Compute shaders and buffers makes them the clear choice to transform terrain data from disk into mesh data on the GPU.
Push Data Through Pipeline
Pushing the data through the pipeline using Unity is as simple as creating a MeshRenderer with a shader that reads the compute buffers from before. Unity executes the shader, which normally contains vertex and fragment sections to output the vertex position and pixel colour respectively.
As the terrain rendering in Bending Time gets more advanced, shaders at different stages will be investigated.
Geo Morph in the Vertex Shader
In a 3D virtual globe application, tiles are loaded and dropped all the time as the user moves around the world. A well-known issue with this is the terrain can “pop” into or out of the scene, which can be somewhat abrasive to the Observer. To ease these transitions, a technique called geo-morphing was created.
Bending Time has not implemented geo-morphing yet but has investigated the solution space and determined the vertex shader is the best place to implement this transition. The Continuous Distance-Dependent Level of Detail paper found at https://hhoppe.com/svdlod.pdf describes the approach.
Physically-Based Rendering
One of the last stages of the render pipeline is the pixel shader. As described in Microsoft’s Direct3d documentation, the pixel-shader stage (PS) enables rich shading techniques such as per-pixel lighting and post-processing. The most common approach these days is to use Physically-Based Rendering (PBR) shading techniques.
Bending Time implemented a PBR pipeline for its 3D ocean but this has not been applied to terrain yet. However, I’ll spend the remainder of this last section describing the planned approach.
One of the starting points for rendering an object using PBR is the albedo. From Wikipedia, albedo is the fraction of sunlight that is diffusely reflected by a body. In games, the albedo texture is often just the base colours excluding normals, etc. However, NASA measures albedo on the Earth’s surface as just the amount of light being reflected, not its colour. For our purposes here, I will think of the albedo for Bending Time’s terrain incorporating both the colour and [base] intensity. Maybe this is how everyone thinks of it, I dunno.
The obvious choice for albedo is from the aerial imagery. This is a good starting point but I suspect the solution for albedo will involve more steps.
One of those steps could be the incorporation of land use imagery, also known as land cover classification.
Land Cover Classification Imagery
Looking at the image above one might consider using the LCC image by looking up a texture based on the pixel classification. E.g., red is urban, green is forests. And this idea has merit but it’s only part of the story. It would seem a combination of the optical satellite imagery and the LCC imagery is the best approach. One of the considerations is the typical resolution of each type of data. Aerial [optical] imagery taken from a plane is often down to 1m resolution. Whereas LCC imagery, most often derived from satellite imagery, is lower resolution. Bending Time plans to primarily use the optical imagery and use the land cover imagery to “correct” any pixels (actually materials). For example, if there is a green building top, the pixel shader might interpret the material as forest and render it accordingly. The land cover classification would say it’s urban and could correct the reflectance in this case. This will need to be experimented with and surely there will be an ongoing effort to improve and tune the albedo portion of the shading.
There are other factors to the albedo as well, which are seasonal changes. Specifically, snow and ice can come and go throughout the seasons depending on the environment/biome of the region. For mountain views, the snow level can be calculated based on altitude and average temperature of the region. This could enable automatic identification and implementation of snow that would be rendered with a snow material. This is just another example of the plethora of solutions/implementations for procedural material identification and rendering.
The next aspect of PBR shading is the normal vectors. These will be computed in the vertex shader on the post-fractal-refined terrain so the time that we’re in the pixel shader, the normal vectors will be available.
Another aspect of PBR rendering is the “metallicness” of the material. Metal has unique properties when it comes to reflecting light and the effect is often implemented in a PBR solution. I won’t talk about the metallic component at this time because I’m not sure how it will come into play for the natural features on our planet.
Now we have the material properties of the object, we need to discuss the light. In Bending Time, the true Sun position is calculated and the sunlight vector is passed to the terrain shaders. The Sun and Moon actually. In addition to the direction, the sunlight colour is calculated using the atmospheric rendering shaders. I’ll discuss atmospheric rendering in a future blog post. From here, we are ready to render the final pixels on the screen.
In Bending Time, this involves the summation of the main types of reflectance, which include:
This is a simplistic PBR solution that is currently only implemented in BT’s ocean rendering solution, which is still not complete. However, the core pieces are all there and I will look at implementing fractal refinement and PBR terrain rendering as one of my first tasks when I return to development of Bending Time’s lifelike virtual world.
Terra firma.
--Sean
There are other factors to the albedo as well, which are seasonal changes. Specifically, snow and ice can come and go throughout the seasons depending on the environment/biome of the region. For mountain views, the snow level can be calculated based on altitude and average temperature of the region. This could enable automatic identification and implementation of snow that would be rendered with a snow material. This is just another example of the plethora of solutions/implementations for procedural material identification and rendering.
The next aspect of PBR shading is the normal vectors. These will be computed in the vertex shader on the post-fractal-refined terrain so the time that we’re in the pixel shader, the normal vectors will be available.
Another aspect of PBR rendering is the “metallicness” of the material. Metal has unique properties when it comes to reflecting light and the effect is often implemented in a PBR solution. I won’t talk about the metallic component at this time because I’m not sure how it will come into play for the natural features on our planet.
Now we have the material properties of the object, we need to discuss the light. In Bending Time, the true Sun position is calculated and the sunlight vector is passed to the terrain shaders. The Sun and Moon actually. In addition to the direction, the sunlight colour is calculated using the atmospheric rendering shaders. I’ll discuss atmospheric rendering in a future blog post. From here, we are ready to render the final pixels on the screen.
In Bending Time, this involves the summation of the main types of reflectance, which include:
- Ambient
- Specular
- Diffuse
This is a simplistic PBR solution that is currently only implemented in BT’s ocean rendering solution, which is still not complete. However, the core pieces are all there and I will look at implementing fractal refinement and PBR terrain rendering as one of my first tasks when I return to development of Bending Time’s lifelike virtual world.
Terra firma.
--Sean
RSS Feed