Written by Jesper Tingvall, Product Expert, Simplygon

Disclaimer: The code in this post is written using version 10.2.5200.0 of Simplygon. If you encounter this post at a later stage, some of the API calls might have changed. However, the core concepts should still remain valid.

Introduction

In this blog we will cover how to generate meshes using NVIDIAs new micro-mesh techology with Simplygon. We will create a batch processor for kit-bashed 3d scanned content intented to save disc space and optimize render performance.

This blog is related to Accelerated remeshing using tessellated attributes. That blog was written for an older version of Simplygon where displacement casting was not done automatically during remeshing. It also did not cover micro-meshes.

Prerequisites

This example will use the Simplygon Python API. Exporting tesselated attributes as *.bary files is currently an API only feature.

To get the most out of this blog you should have an understanding of what micro-meshes are. We will use NVIDIA Displacement Micro-Map Toolkit to view the output data.

Problem to solve

The assets we will optimize is from a church pack. We will just have a look at a subset of the assets. The assets uses two 4k textures; base color and normal map.

From left to right; Church_Wall_008, Church_Wall_007, Church_Wall_002 with Church_Door_Var01.

The original files are saved as fbx. But in order to not compare apples to oranges lets convert them to glb when comparing file size to optimized assets.

If we look on the inside of the meshes we can see that a lot of geometry on the hiding in there. Geometry on the insides as well as a unconnected outside shell makes these assets a prime candidate for remeshing.

Use cases for NVIDIA micro-meshes

Before we delve into solution to our problem and let us briefly talk about NVIDIA micro-meshes.

Micro-meshes is a very memory efficient way of storing displacement data that is then applied to a base mesh. While it is aimed at NVIDIA GeForce RTX 40 Series GPUs that has hardware acceleration for this technique, it is open source and cross platform. It can be used both for ray tracing but also traditional rasterization.

It can either be used in the rendering pipeline or as just a way of storing memory on disc for very dense meshes. Upon loading the meshes into the game the meshes can be tessellated and rendered using any traditional render pipeline. We can specify the level of tessellation upon loading. That means we could potentially use the same micro-mesh data at different tessellation for different LOD levels, and use the low poly base mesh for distant LODs.

The cost of storing a micro-mesh triangle is around 6-1 bits depending on asset and tessellation level. As we only store displacement data for micro-meshes we get better compression then other storage techniques for dense geometries which stores coordinates, UVs and normals for all triangles.

As we do not store normals per micro-mesh triangle it is good to use micro-meshes in conjunction with a high resolution normal map. This enables us to have a less dense base mesh and correct the normals with the texture.

Solution

We are going to use a number of concepts in the solution. First we are going to use the remeshing pipeline which creates a new watertight mesh from the input. This is a good way of cleaning up assets that has lots of faces on the inside, or holes which makes reduction tricky. To the remeshing we are going to apply attribute tessellation. This allows us to store data more frequently then per vertex. In our case we are going to store displacement data. That allows us to have a fairly low poly base mesh and to it apply tessellation to get more details. Lastly we are going to save the attribute tessellation data with the micro-mesh format *.bary. Bary files are a very efficient way of storing displacement data.

It is possible to use tessellated attributes for all pipelines in Simplygon. In most cases you need to use a displacement caster to cast the data into tessellation. In the case of remeshing one of the new features in Simplygon 10.2 is that this is done automatically.

It is important to point out that attribute tessellation is something internal in Simplygon and does not only work with NVIDIA micro-meshes. You can implement your own exporter for the tessellation data, or apply tessellation before saving the file to disc.

Screen size as tessellation density mode

Let us start by using screen size to determine when to tessellate a triangle. First let us create a standard remeshing pipeline and setup mapping image used for material casting as well as setting a target screen size.

def create_remeshing_pipeline_screen_size(sg, screen_size, tesselation_screen_size):
    sgRemeshingProcessor = sg.CreateRemeshingPipeline()
    
    sgRemeshingSettings = sgRemeshingProcessor.GetRemeshingSettings()
    mapping_image_settings = sgRemeshingProcessor.GetMappingImageSettings()
    material_output_settings = mapping_image_settings.GetOutputMaterialSettings(0)

    # Set quality of material casting.
    material_output_settings.SetTextureHeight(TEXTURE_SIZE)
    material_output_settings.SetTextureWidth(TEXTURE_SIZE)
    mapping_image_settings.SetGenerateMappingImage(True)
    mapping_image_settings.SetTexCoordName("MaterialLOD")

    # Set quality of remeshing.
    sgRemeshingSettings.SetOnScreenSize( screen_size ) 
    sgRemeshingSettings.SetSurfaceTransferMode( Simplygon.ESurfaceTransferMode_Accurate )
    sgRemeshingSettings.SetHoleFilling(Simplygon.EHoleFilling_Low)

Now comes the interesting part. First we'll set so the remeshed automatically cast displacement into tessellated attributes via PopulateAttributeTessellationDisplacement. We also need to enable attribute tessellation via EnableAttributeTessellation on the remesh pipeline's AttributeTessellationSettings.

With those settings enabled we can set options for our tessellated attributes. As AttributeTessellationDensityMode we are going to pick OnScreenSize. This will make the tessellated triangle take up roughly one pixel at the given OnScreenSize. This tessellation density mode is good for when you want to control triangle density per pixel on screen.

We are also setting OnlyAllowOneLevelOfDifference to True. This is required for exporting tessellated attributes into a *.bary file. This setting tells the processor to only allow one level of tessellation difference from one base triangle to its neighboring triangles. Larger differences would make the topology not connect, as micro-meshes can only handle one level of difference.

    # Populate the attribute tesselation with displacement during remeshing.
    sgRemeshingSettings.SetPopulateAttributeTessellationDisplacement( True )

    sgAttributeTessellationSettings = sgRemeshingProcessor.GetAttributeTessellationSettings()
    
    # Attribute tessellation settings
    sgAttributeTessellationSettings.SetEnableAttributeTessellation( True )
    sgAttributeTessellationSettings.SetAttributeTessellationDensityMode( Simplygon.EAttributeTessellationDensityMode_OnScreenSize )
    sgAttributeTessellationSettings.SetOnScreenSize( tesselation_screen_size )
    sgAttributeTessellationSettings.SetOnlyAllowOneLevelOfDifference( True )

    return sgRemeshingProcessor

Save micro-mesh

When saving the file as glTF data saved in tessellated attributes will automatically be saved in a .bary file in same output folder. The glTF file contains the base mesh and the .bary file contains the attribute tessellation displacement data.

def export_output(sg, scene, path):
    exporter = sg.CreateSceneExporter()
    exporter.SetScene(scene)
    print("Saving " + path)
    exporter.SetExportFilePath( path )
    exporter.Run()

Let us process one of the meshes with our remesher function and save to disc. We process SM_Church_Wall_008 at a remeshing screen size of 500 pixels and tessellation density mode of 500 pixels.

The result is saved as SM_Church_Wall_008_micromesh.glb. Since our mesh now contains displacement data in tessellated attributes we also get a SM_Church_Wall_008_micromesh_displacement0.bary. Let us load up SM_Church_Wall_008_micromesh.glb in micromesh_toolbox so we can inspect the result. The first image is how the base model look, and second is the base model with tessellation applied.

Absolute length as tessellation density mode

Another way to define the quality of our remeshing is to think about it in terms of deviation. As we want to set these attributes in metrics we have introduced a global constant for scene scale.

SCENE_SCALE = 100

First we are going to introduce a helper function for our remeshing pipeline which calculates a max deviation from original mesh to screen size.

def MaxDeviationToScreenSize(maxDeviation, sgScene):
    sceneDiameter = sgScene.GetRadius() * 2
    screenSize = int(sceneDiameter / maxDeviation)
    return screenSize

As above let us create a remeshing pipeline, but use the calculated screen size as OnScreenSize target, as well as taking SCENE_SCALE into account.

def create_remeshing_pipeline_deviation_length(sg, scene, max_deviation, tesselation_length):
    sgRemeshingProcessor = sg.CreateRemeshingPipeline()
    
    sgRemeshingSettings = sgRemeshingProcessor.GetRemeshingSettings()
    mapping_image_settings = sgRemeshingProcessor.GetMappingImageSettings()
    material_output_settings = mapping_image_settings.GetOutputMaterialSettings(0)

    # Set quality of material casting.
    material_output_settings.SetTextureHeight(TEXTURE_SIZE)
    material_output_settings.SetTextureWidth(TEXTURE_SIZE)
    mapping_image_settings.SetGenerateMappingImage(True)
    mapping_image_settings.SetTexCoordName("MaterialLOD")
    
    # Set quality of remeshing.
    sgRemeshingSettings.SetOnScreenSize( MaxDeviationToScreenSize(max_deviation * SCENE_SCALE, scene)  ) 
    sgRemeshingSettings.SetHoleFilling(Simplygon.EHoleFilling_Low)
    sgRemeshingSettings.SetSurfaceTransferMode( Simplygon.ESurfaceTransferMode_Accurate )

Now we enable attribute tessellation and set AttributeTessellationDensityMode to AbsoluteLength. In this mode we control the tessellation amount by setting a max length of the tessellated triangles edges. We can specify this via SetMaxLengthOfTessellatedValue. We also need to take SCENE_SCALE into account here. In this mode one thing to look out for is that more slivery triangles will be relatively more tessellated than equal area isotropic triangles, since their length to area ratio is higher.

It is worth pointing out that we can only use one mode to control tessellation density. So we can not for example use a combination of screen size and max length.

    # Populate the attribute tesselation with displacement during remeshing.
    sgRemeshingSettings.SetPopulateAttributeTessellationDisplacement( True )
    sgAttributeTessellationSettings = sgRemeshingProcessor.GetAttributeTessellationSettings()
    sgAttributeTessellationSettings.SetEnableAttributeTessellation( True )

    # Attribute tessellation settings
    sgAttributeTessellationSettings.SetAttributeTessellationDensityMode( Simplygon.EAttributeTessellationDensityMode_AbsoluteLength )
    sgAttributeTessellationSettings.SetMaxLengthOfTessellatedValue( tesselation_length * SCENE_SCALE)
    sgAttributeTessellationSettings.SetOnlyAllowOneLevelOfDifference( True )

    return sgRemeshingProcessor

Let us process SM_Church_Wall_008 with max deviation of 5 mm and a max triangle length of 1 mm. We get a very dense asset of 33 M triangles. But the total geometry data only takes up 20 MB.

Absolute area as tessellation density mode

Lastly we will look at absolute area as tessellation density mode. The first section is identical to absolute length part. We create a remeshing pipeline and use max deviation to control it.

def create_remeshing_pipeline_deviation_area(sg, scene, max_deviation, tesselation_area):
    sgRemeshingProcessor = sg.CreateRemeshingPipeline()
    
    sgRemeshingSettings = sgRemeshingProcessor.GetRemeshingSettings()
    mapping_image_settings = sgRemeshingProcessor.GetMappingImageSettings()
    material_output_settings = mapping_image_settings.GetOutputMaterialSettings(0)

    # Set quality of material casting.
    material_output_settings.SetTextureHeight(TEXTURE_SIZE)
    material_output_settings.SetTextureWidth(TEXTURE_SIZE)
    mapping_image_settings.SetGenerateMappingImage(True)
    mapping_image_settings.SetTexCoordName("MaterialLOD")

    # Set quality of remeshing.
    sgRemeshingSettings.SetOnScreenSize( MaxDeviationToScreenSize(max_deviation * SCENE_SCALE, scene)  ) 
    sgRemeshingSettings.SetHoleFilling(Simplygon.EHoleFilling_Low)
    sgRemeshingSettings.SetSurfaceTransferMode( Simplygon.ESurfaceTransferMode_Accurate )
    
    # Populate the attribute tesselation with displacement during remeshing.
    sgRemeshingSettings.SetPopulateAttributeTessellationDisplacement( True )
    sgAttributeTessellationSettings = sgRemeshingProcessor.GetAttributeTessellationSettings()
    sgAttributeTessellationSettings.SetEnableAttributeTessellation( True )

We set AttributeTessellationDensityMode to AbsoluteArea. In this mode the area our tessellated triangle covers determine the tessellation level. We can specify maximum area it can cover with MaxAreaOfTessellatedValue. As we are calculating area we need to apply the SCENE_SCALE twice to get it in correct metric format.

With mode set to absolute area we can not directly control the length of our tessellated edges directly, but we get the benefit of slivery triangles will not be as tessellated as with absolute length in example above.

    # Attribute tessellation settings
    sgAttributeTessellationSettings.SetAttributeTessellationDensityMode( Simplygon.EAttributeTessellationDensityMode_AbsoluteArea )
    sgAttributeTessellationSettings.SetMaxAreaOfTessellatedValue( tesselation_area * SCENE_SCALE * SCENE_SCALE)
    sgAttributeTessellationSettings.SetOnlyAllowOneLevelOfDifference( True )

    return sgRemeshingProcessor

To get around the same quality as absolute length we can use the triangle's area formula area = side_length * side_length / 2. So if we would want to use same quality metric as before, 1 mm we would want a target area of 1 mm * 1mm / 2 = 0.0000005.

Apply tessellation to scene

Instead of saving the file with tessellated data in a .bary file we can apply the tessellation to the scene. This gives us a new scene where all data previously saved in tessellated attributes instead are saved in geometry. This can be useful if main reason for us using tessellated attributes is speeding up remesher, rater then wanting a bary output file.

def tessellate_scene(sg, scene):
    # We will now create an attribute tessellation tool object, and generate a scene with the 
    # tessellated attribute displacement data generated into real tessellated mesh data, which is 
    # stored into a new scene. 
    sgAttributeTessellation = sg.CreateAttributeTessellation()
    sgTessellatedScene = sgAttributeTessellation.NewTessellatedScene(scene)
    return sgTessellatedScene

With that function we can create a tessellated scene then export it.

tesselated_scene = tessellate_scene(sg, scene)
path = without_fileending_name + "_tesselated.fbx"
export_output(sg, tesselated_scene, path)

Result

Let us experiment with different remeshing and attribute tessellation settings.

Church_Door_Var01

At low resolution we can see some artifacts around the handles. At high resolution we can see that the screw holes are well preserved. In the original mesh each plank was separated so we have removed a lot of internal geometry as well.

Church_Wall_002

Church_Wall_007

Church_Wall_008

Notice on low resolution base mesh

It is important to point out that if we have a low resolution we will get bad result no matter how dense the tessellation is. We can see this clearly on SM_Church_Wall_007.

What can mitigate this is to use lower or no hole filling using by changing SetHoleFilling. But it is likely that the only way to get good result is to increase the screen size of the base mesh. In our case the 300 screen size remesh only contains 1.5k triangles.

Let's try to increase base mesh to a screen size of 900 pixels. That gives us 5.9k triangles and much better looking result.

Conclusions

Micro-meshes can be very useful when it comes to compress down geometry, but care need to be taked when using them. Here are some pointers for your journey into it.

• While it is very tempting to go very low poly for base mesh it can cause artifacts that can not be solved with tessellation.
• Avoid aggressive hole filling.
• Use a high quality normal map.
• It is worth pointing out that even when we over-tessellated some of our assets, made them contain more triangles then the original geometry, we did not go over the original assets file size.
• Our take away from experimenting with micro-meshes is that uniform tessellated meshes, scan meshes, can usually have a lower resolution cage meshes while modeled meshes either from DCC tools or CAD require higher resolution base mesh.

Complete script

# Copyright (c) Microsoft Corporation. 
# Licensed under the MIT license. 
 
from simplygon10 import simplygon_loader
from simplygon10 import Simplygon
import os

TEXTURE_SIZE = 4096

INPUT_FILE_ENDING = ".FBX"
SCENE_SCALE = 100


def process_asset(sg, asset):
    print("Optimizing " + asset)
    
    importer = sg.CreateSceneImporter()
    scene = None
    
    importer.SetImportFilePath(asset)
    if importer.Run() == Simplygon.EErrorCodes_NoError:
        without_fileending_name = asset.replace(INPUT_FILE_ENDING, '')
        scene = importer.GetScene()

        #pipeline = create_remeshing_pipeline_screen_size(sg, 300, 300)
        #pipeline = create_remeshing_pipeline_screen_size(sg, 500, 500)
        pipeline = create_remeshing_pipeline_screen_size(sg, 1080, 1080)
        #pipeline = create_remeshing_pipeline_deviation_length(sg, scene, 0.005, 0.001)
        #pipeline = create_remeshing_pipeline_deviation_area(sg, scene, 0.005, 0.0000005) # 0.5 cm deviation, 1 mm * 1mm / 2 area = 0.0000005

        add_default_casters(sg, pipeline, scene)
        pipeline.RunScene(scene, Simplygon.EPipelineRunMode_RunInThisProcess)

        path = without_fileending_name + "_micromesh.glb"
        export_output(sg, scene, path)

        tessellated_scene = tessellate_scene(sg, scene)
        path = without_fileending_name + "_tessellated.glb"
        export_output(sg, tessellated_scene, path)

    else:
        print("Error: Failed to import " + asset)



def create_remeshing_pipeline_screen_size(sg, screen_size, tessellation_screen_size):
    sgRemeshingProcessor = sg.CreateRemeshingPipeline()

    sgRemeshingSettings = sgRemeshingProcessor.GetRemeshingSettings()
    mapping_image_settings = sgRemeshingProcessor.GetMappingImageSettings()
    material_output_settings = mapping_image_settings.GetOutputMaterialSettings(0)

    # Set quality of material casting.
    material_output_settings.SetTextureHeight(TEXTURE_SIZE)
    material_output_settings.SetTextureWidth(TEXTURE_SIZE)
    mapping_image_settings.SetGenerateMappingImage(True)
    mapping_image_settings.SetTexCoordName("MaterialLOD")

    # Set quality of remeshing.
    sgRemeshingSettings.SetOnScreenSize( screen_size ) 
    sgRemeshingSettings.SetSurfaceTransferMode( Simplygon.ESurfaceTransferMode_Accurate )
    sgRemeshingSettings.SetHoleFilling(Simplygon.EHoleFilling_Low)

    # Populate the attribute tesselation with displacement during remeshing.
    sgRemeshingSettings.SetPopulateAttributeTessellationDisplacement( True )
    sgAttributeTessellationSettings = sgRemeshingProcessor.GetAttributeTessellationSettings()
    sgAttributeTessellationSettings.SetEnableAttributeTessellation( True )

    # Attribute tessellation settings
    sgAttributeTessellationSettings.SetAttributeTessellationDensityMode( Simplygon.EAttributeTessellationDensityMode_OnScreenSize )
    sgAttributeTessellationSettings.SetOnScreenSize( tessellation_screen_size )
    sgAttributeTessellationSettings.SetOnlyAllowOneLevelOfDifference( True )

    return sgRemeshingProcessor


def MaxDeviationToScreenSize(maxDeviation, sgScene):
    sceneDiameter = sgScene.GetRadius() * 2
    screenSize = int(sceneDiameter / maxDeviation)
    print("Calculated screen size: " + str(screenSize))
    return screenSize


def create_remeshing_pipeline_deviation_length(sg, scene, max_deviation, tessellation_length):
    sgRemeshingProcessor = sg.CreateRemeshingPipeline()
    
    sgRemeshingSettings = sgRemeshingProcessor.GetRemeshingSettings()
    mapping_image_settings = sgRemeshingProcessor.GetMappingImageSettings()
    material_output_settings = mapping_image_settings.GetOutputMaterialSettings(0)

    # Set quality of material casting.
    material_output_settings.SetTextureHeight(TEXTURE_SIZE)
    material_output_settings.SetTextureWidth(TEXTURE_SIZE)
    mapping_image_settings.SetGenerateMappingImage(True)
    mapping_image_settings.SetTexCoordName("MaterialLOD")
    
    # Set quality of remeshing.
    sgRemeshingSettings.SetOnScreenSize( MaxDeviationToScreenSize(max_deviation * SCENE_SCALE, scene)  ) 
    sgRemeshingSettings.SetHoleFilling(Simplygon.EHoleFilling_Low)
    sgRemeshingSettings.SetSurfaceTransferMode( Simplygon.ESurfaceTransferMode_Accurate )
    
    # Populate the attribute tesselation with displacement during remeshing.
    sgRemeshingSettings.SetPopulateAttributeTessellationDisplacement( True )
    sgAttributeTessellationSettings = sgRemeshingProcessor.GetAttributeTessellationSettings()
    sgAttributeTessellationSettings.SetEnableAttributeTessellation( True )

    # Attribute tessellation settings
    sgAttributeTessellationSettings.SetAttributeTessellationDensityMode( Simplygon.EAttributeTessellationDensityMode_AbsoluteLength )
    sgAttributeTessellationSettings.SetMaxLengthOfTessellatedValue( tessellation_length * SCENE_SCALE)
    sgAttributeTessellationSettings.SetOnlyAllowOneLevelOfDifference( True )

    return sgRemeshingProcessor


def create_remeshing_pipeline_deviation_area(sg, scene, max_deviation, tessellation_area):
    sgRemeshingProcessor = sg.CreateRemeshingPipeline()
    
    sgRemeshingSettings = sgRemeshingProcessor.GetRemeshingSettings()
    mapping_image_settings = sgRemeshingProcessor.GetMappingImageSettings()
    material_output_settings = mapping_image_settings.GetOutputMaterialSettings(0)

    # Set quality of material casting.
    material_output_settings.SetTextureHeight(TEXTURE_SIZE)
    material_output_settings.SetTextureWidth(TEXTURE_SIZE)
    mapping_image_settings.SetGenerateMappingImage(True)
    mapping_image_settings.SetTexCoordName("MaterialLOD")

    # Set quality of remeshing.
    sgRemeshingSettings.SetOnScreenSize( MaxDeviationToScreenSize(max_deviation * SCENE_SCALE, scene)  ) 
    sgRemeshingSettings.SetHoleFilling(Simplygon.EHoleFilling_Low)
    sgRemeshingSettings.SetSurfaceTransferMode( Simplygon.ESurfaceTransferMode_Accurate )
    
    # Populate the attribute tesselation with displacement during remeshing.
    sgRemeshingSettings.SetPopulateAttributeTessellationDisplacement( True )
    sgAttributeTessellationSettings = sgRemeshingProcessor.GetAttributeTessellationSettings()
    sgAttributeTessellationSettings.SetEnableAttributeTessellation( True )

    # Attribute tessellation settings
    sgAttributeTessellationSettings.SetAttributeTessellationDensityMode( Simplygon.EAttributeTessellationDensityMode_AbsoluteArea )
    sgAttributeTessellationSettings.SetMaxAreaOfTessellatedValue( tessellation_area * SCENE_SCALE * SCENE_SCALE)
    sgAttributeTessellationSettings.SetOnlyAllowOneLevelOfDifference( True )

    return sgRemeshingProcessor


def add_default_casters(sg: Simplygon.ISimplygon, pipeline: Simplygon.spPipeline, scene: Simplygon.spScene) -> None:
    caster = sg.CreateColorCaster()
    caster_settings = caster.GetColorCasterSettings()
    caster_settings.SetMaterialChannel("Diffuse") 
    pipeline.AddMaterialCaster( caster, 0 )

    caster = sg.CreateNormalCaster()
    caster_settings = caster.GetNormalCasterSettings()
    caster_settings.SetGenerateTangentSpaceNormals(True)
    caster_settings.SetMaterialChannel("Normals") 
    pipeline.AddMaterialCaster( caster, 0 )


def process_assets(sg: Simplygon.ISimplygon, input_dir: str):
    for asset in os.listdir(input_dir):
        if INPUT_FILE_ENDING in asset:
            process_asset(sg, input_dir + "/" + asset)


def export_output(sg, scene, path):
    exporter = sg.CreateSceneExporter()
    exporter.SetScene(scene)
    print("Saving " + path)
    exporter.SetExportFilePath( path )
    exporter.Run()


def tessellate_scene(sg, scene):
    sgAttributeTessellation = sg.CreateAttributeTessellation()
    sgTessellatedScene = sgAttributeTessellation.NewTessellatedScene(scene)
    return sgTessellatedScene


def main():
    print("Starting batch processing")
    sg = simplygon_loader.init_simplygon()
    process_assets(sg, "E:/Git/simplygon.blogs/Micromesh/Models")


if __name__== "__main__":
    main()