Written by Jesper Tingvall, Product Expert, Simplygon

Disclaimer: The code in this post is written using version 10.4.199.0 of Simplygon. If you encounter this post at a later stage, some of the API calls might have changed. However, the core concepts should still remain valid.

Introduction

In this post we'll showcase how to merge the texture of a remeshed proxy into the original texture. This allows you to use the remesher without requirement of introducing a new material and texture set.

Prerequisites

This example will use the Simplygon Python API, but the same concepts can be applied to all other integrations of the Simplygon API.

Problem to solve

We want to optimize a character model down to a really low triangle count. We start by trying the triangle reducer. Below we can see the result. It has a number of issues; we get holes in our output model and surfaces intersects each other. The disconnected topology of the model prevents us from going further with the triangle reducer.

When this happens we suggest to switch to the remesher. The remesher creates a new watertight low poly model. But it also creates a new set of textures. This is sometimes desired as it allows us to optimize draw calls for the distant model. In some other cases this is just a hassle. It can be that our engine are not able to handle different materials for different LOD levels, or that the extra texture sets is not desired.

Solution

The remesher creates a new model with new topology that does not have a connection to the original model. Thus the new model's UVs can not use the original textures and we need to bake new textures.

So how can we render both the original model and the remeshed model with the same textures? The trick is to reserve part of the original model's texture space for our proxy model. Our original model uses two 4096x4096 textures, diffuse and normal. We want to bake our proxy at at texture size of 128x128. We can easily fit the proxy texture into our original texture without impact to visual quality. We introduce directive to our artists that the lowest 130x130 (we add some margin) left corner of the texture chart can never be used. In this image purple is our reserved texture space. We can see that it is a very small portion of the entire texture atlas.

Remeshing and material casting

We create a remeshing pipeline and enables generation of a mapping image. A mapping image is required for material casting. We set the output texture size to the reserved texture size in our original texture. With SetGutterSpace we can control how tightly we should pack our texture charts. Since we generate a very small texture, we set this to be packed tightly.

def create_remeshing_pipeline(sg: Simplygon.ISimplygon, on_screen_size: int, texture_size: int):
    """Create a remeshing pipeline with material casters for diffuse and normal textures."""

    pipeline = sg.CreateRemeshingPipeline()
    settings = pipeline.GetRemeshingSettings()

    settings.SetOnScreenSize(on_screen_size)
    
    mapping_image_settings = pipeline.GetMappingImageSettings()
    mapping_image_settings.SetGenerateMappingImage(True)
    mapping_image_settings.SetGenerateTangents(True)
    mapping_image_settings.SetGenerateTexCoords(True)
    
    material_output_settings = mapping_image_settings.GetOutputMaterialSettings(0)
    material_output_settings.SetTextureHeight(texture_size)
    material_output_settings.SetTextureWidth(texture_size)
    material_output_settings.SetGutterSpace(1) # Pack charts for proxy really close
    

We will now add material casters for our two material channels. The name of these depends on file format and integration. In our case we have a usd file so the material channels are diffuseColor and normal. We also need to set caster settings in accordance with our game engine.

    # Color caster
    caster = sg.CreateColorCaster()
    caster.SetOutputFilePath(diffuse_temp_texture)
    caster_settings = caster.GetColorCasterSettings()
    caster_settings.SetMaterialChannel("diffuseColor") # name of diffuse channel in usd format.
    caster_settings.SetOutputSRGB(True)
    pipeline.AddMaterialCaster( caster, 0 )
    
    # Normal caster
    caster = sg.CreateNormalCaster()
    caster.SetOutputFilePath(normal_temp_texture)
    caster_settings = caster.GetNormalCasterSettings()
    caster_settings.SetGenerateTangentSpaceNormals(True)
    caster_settings.SetMaterialChannel("normal") # name of normal channel in usd format.
    
    # Normal caster settings. Change these settings to correspond to your engine
    caster_settings.SetFlipGreen(False)
    caster_settings.SetCalculateBitangentPerFragment(True)
    caster_settings.SetNormalizeInterpolatedTangentSpace(False)
    
    pipeline.AddMaterialCaster( caster, 0 )
    return pipeline

After remeshing and casting we get this 128x128 output texture. We also get a normal map.

Scale UVs

We want to use a part of our original's texture for our proxy LOD. Currently our proxy model's UVs covers the entire texture. Now it is time to scale down our proxy model's UVs so it fits into the corner we allocated to it's textures. We start by iterating through all models in our scene. We do this by creating a selection set containing all SceneMesh nodes in our scene. We can then iterate through it and access the geometry data.

def scale_uvs(scene: Simplygon.spScene, scale: float):
    """Scales UV field 0 down to scale."""

    scene_meshes_selection_set_id = scene.SelectNodes("SceneMesh")
    scene_meshes_selection_set = scene.GetSelectionSetTable().GetSelectionSet(scene_meshes_selection_set_id)
    # Loop through all meshes in the scene
    for node_id in range(scene_meshes_selection_set.GetItemCount()):
        scene_mesh = Simplygon.spSceneMesh.SafeCast(scene.GetNodeByGUID( scene_meshes_selection_set.GetItem(node_id)))
        geometry = scene_mesh.GetGeometry()

        if not geometry: 
            print("Scene without geometry, skipping")
            continue

        if not geometry.GetMaterialIds():
            print("Geometry with no material IDs, skipping")
            continue

        if not geometry.GetTexCoords(0):
            print("Geometry with no TexCoords, skipping")
            continue

Once we have a geometry we iterate through its TexCoords field and scale every UV coordinate so it fits into our corner.

        current_uvs = geometry.GetTexCoords(0)
        if current_uvs != None:
            for i in range(0, current_uvs.GetItemCount()):
                current_uvs.SetRealItem(i, current_uvs.GetRealItem(i) * scale)

After scaling the UVs we now have them in a corner of our UV map.

Merging

To merge textures we are going to use the Python Pillow package. With this package we can put our proxy LOD's texture into a corner of our original texture.

def merge_textures(base_texture: str, overlay_texture: str, output_texture: str):
    """Puts overlay_texture on top of base_texture in a corner."""

    print (f"Loading {base_texture}...")
    base_img = Image.open(base_texture)
    base_width, base_height = base_img.size

    print (f"Loading {overlay_texture}...")
    overlay_img = Image.open(overlay_texture)
    overlay_width, overlay_height = overlay_img.size
    overlay_img = overlay_img.convert("RGBA") # In case overlay_texture does not have an alpha channel we need to add it to make paste function below work.

    base_img.paste(overlay_img, (0,base_height-overlay_height), overlay_img)

    print (f"Saving {output_texture}...")
    base_img.save(output_texture)

This is how it looks after we merged the textures. Here we have zoomed in to left corner as well as highlighted it so we can see what is going on more easily.

Putting it all together

Now it is time to put all of our functions together. We start with a function that loads our original model, performs remeshing on it and then scales it's UVs to a corner.

def create_proxy_mesh(input_file: str, on_screen_size: int, original_texture_size: int, remeshing_texture_size: int, output_folder: str):
    """Creates a remeshed proxy model for input file and scale UVs so it fits into one corner of original texture."""

    # Change these settings to correspond to your engine
    sg.SetGlobalDefaultTangentCalculatorTypeSetting(Simplygon.ETangentSpaceMethod_MikkTSpace)
    
    scene = load_scene(sg, input_file)

    pipeline_opaque = create_remeshing_pipeline(sg, on_screen_size, remeshing_texture_size)

    print("Performs remeshing on " + input_file + "...")
    pipeline_opaque.RunScene(scene, Simplygon.EPipelineRunMode_RunInThisProcess)

    scale_uvs(scene, remeshing_texture_size / original_texture_size)

    save_scene(sg, scene, output_folder + input_file)

We can then call merge textures to put our proxy model's output into a corner of our original texture. We do this both for the diffuse and normal texture.

def process_file(input_file: str, on_screen_size: int, original_texture_size: int, remeshing_texture_size:int, texture_folder: str, diffuse_texture: str, normal_texture: str, output_folder: str):
    """Create a proxy model and merges casted texture into original texture files."""

    create_proxy_mesh(input_file, on_screen_size, original_texture_size, remeshing_texture_size, output_folder)

    merge_textures(texture_folder + diffuse_texture, diffuse_temp_texture, output_folder + diffuse_texture)
    merge_textures(texture_folder + normal_texture, normal_temp_texture, output_folder + normal_texture)

Result

We can now process our model using the following function.

process_file(input_file="ork.usdc", 
             on_screen_size= 50,
             original_texture_size= 4096,
             remeshing_texture_size= 128,
             texture_folder="textures/",
             diffuse_texture = "T_OrcShaman_1_Albedo_Skin1.png",
             normal_texture = "T_OrcShaman_1_Normal.tga",
             output_folder="output/")

This give us following output. Our model is intended for far away viewing and we are inspecting it way to close. But we can see that we do not suffer from the same issues where different surfaces intersect; causing the skin to be outside of the clothes. We are also capturing the silhouette of the model better.

Since we use the same textures for original model and proxy model we can now render them using the same material. This is useful if our game engines LOD system can not handle different materials for different LOD levels. If this is not a limitation we suggest to render the proxy LOD asset using a simpler shader. There is often a lot of lighting and skin effects we can skip when the asset is far away. We are just interested in the color of some pixels on the screen.

Complete script

# Copyright (c) Microsoft Corporation. 
# Licensed under the MIT license. 

from simplygon10 import simplygon_loader
from simplygon10 import Simplygon
from PIL import Image


diffuse_temp_texture = "casted_diffuse.png"
normal_temp_texture = "casted_normal.png"


def load_scene(sg: Simplygon.ISimplygon, path: str) -> Simplygon.spScene:
    """Load scene from path and return a Simplygon scene."""

    scene_importer = sg.CreateSceneImporter()
    scene_importer.SetImportFilePath(path)

    print (f"Loading {path}...")
    
    import_result = scene_importer.Run()
    if Simplygon.Failed(import_result):
        raise Exception('Failed to load scene.')

    scene = scene_importer.GetScene()
    return scene


def save_scene(sg: Simplygon.ISimplygon, sgScene: Simplygon.spScene, path: str):
    """Save scene to path."""

    scene_exporter = sg.CreateSceneExporter()
    scene_exporter.SetExportFilePath(path)
    scene_exporter.SetScene(sgScene)

    print (f"Saving {path}...")
    
    export_result = scene_exporter.Run()
    if Simplygon.Failed(export_result):
        raise Exception('Failed to save scene.')
    

def merge_textures(base_texture: str, overlay_texture: str, output_texture: str):
    """Puts overlay_texture on top of base_texture in a corner."""

    print (f"Loading {base_texture}...")
    base_img = Image.open(base_texture)
    base_width, base_height = base_img.size

    print (f"Loading {overlay_texture}...")
    overlay_img = Image.open(overlay_texture)
    overlay_width, overlay_height = overlay_img.size
    overlay_img = overlay_img.convert("RGBA") # In case overlay_texture does not have an alpha channel we need to add it to make paste function below work.

    base_img.paste(overlay_img, (0,base_height-overlay_height), overlay_img)

    print (f"Saving {output_texture}...")
    base_img.save(output_texture)


def scale_uvs(scene: Simplygon.spScene, scale: float):
    """Scales UV field 0 down to scale."""

    scene_meshes_selection_set_id = scene.SelectNodes("SceneMesh")
    scene_meshes_selection_set = scene.GetSelectionSetTable().GetSelectionSet(scene_meshes_selection_set_id)
    # Loop through all meshes in the scene
    for node_id in range(scene_meshes_selection_set.GetItemCount()):
        scene_mesh = Simplygon.spSceneMesh.SafeCast(scene.GetNodeByGUID( scene_meshes_selection_set.GetItem(node_id)))
        geometry = scene_mesh.GetGeometry()

        if not geometry: 
            print("Scene without geometry, skipping")
            continue

        if not geometry.GetMaterialIds():
            print("Geometry with no material IDs, skipping")
            continue

        if not geometry.GetTexCoords(0):
            print("Geometry with no TexCoords, skipping")
            continue

        current_uvs = geometry.GetTexCoords(0)
        if current_uvs != None:
            for i in range(0, current_uvs.GetItemCount()):
                current_uvs.SetRealItem(i, current_uvs.GetRealItem(i) * scale)
                    

def create_remeshing_pipeline(sg: Simplygon.ISimplygon, on_screen_size: int, texture_size: int):
    """Create a remeshing pipeline with material casters for diffuse and normal textures."""

    pipeline = sg.CreateRemeshingPipeline()
    settings = pipeline.GetRemeshingSettings()

    settings.SetOnScreenSize(on_screen_size)
    
    mapping_image_settings = pipeline.GetMappingImageSettings()
    mapping_image_settings.SetGenerateMappingImage(True)
    mapping_image_settings.SetGenerateTangents(True)
    mapping_image_settings.SetGenerateTexCoords(True)
    
    material_output_settings = mapping_image_settings.GetOutputMaterialSettings(0)
    material_output_settings.SetTextureHeight(texture_size)
    material_output_settings.SetTextureWidth(texture_size)
    material_output_settings.SetGutterSpace(1) # Pack charts for proxy really close
    
    # Color caster
    caster = sg.CreateColorCaster()
    caster.SetOutputFilePath(diffuse_temp_texture)
    caster_settings = caster.GetColorCasterSettings()
    caster_settings.SetMaterialChannel("diffuseColor") # name of diffuse channel in usd format.
    caster_settings.SetOutputSRGB(True)
    pipeline.AddMaterialCaster( caster, 0 )
    
    # Normal caster
    caster = sg.CreateNormalCaster()
    caster.SetOutputFilePath(normal_temp_texture)
    caster_settings = caster.GetNormalCasterSettings()
    caster_settings.SetGenerateTangentSpaceNormals(True)
    caster_settings.SetMaterialChannel("normal") # name of normal channel in usd format.
    
    # Normal caster settings. Change these settings to correspond to your engine
    caster_settings.SetFlipGreen(False)
    caster_settings.SetCalculateBitangentPerFragment(True)
    caster_settings.SetNormalizeInterpolatedTangentSpace(False)
    
    pipeline.AddMaterialCaster( caster, 0 )
    return pipeline
    

def create_proxy_mesh(input_file: str, on_screen_size: int, original_texture_size: int, remeshing_texture_size: int, output_folder: str):
    """Creates a remeshed proxy model for input file and scale UVs so it fits into one corner of original texture."""

    # Change these settings to correspond to your engine
    sg.SetGlobalDefaultTangentCalculatorTypeSetting(Simplygon.ETangentSpaceMethod_MikkTSpace)
    
    scene = load_scene(sg, input_file)

    pipeline_opaque = create_remeshing_pipeline(sg, on_screen_size, remeshing_texture_size)

    print("Performs remeshing on " + input_file + "...")
    pipeline_opaque.RunScene(scene, Simplygon.EPipelineRunMode_RunInThisProcess)

    scale_uvs(scene, remeshing_texture_size / original_texture_size)

    save_scene(sg, scene, output_folder + input_file)


def process_file(input_file: str, on_screen_size: int, original_texture_size: int, remeshing_texture_size:int, texture_folder: str, diffuse_texture: str, normal_texture: str, output_folder: str):
    """Create a proxy model and merges casted texture into original texture files."""

    create_proxy_mesh(input_file, on_screen_size, original_texture_size, remeshing_texture_size, output_folder)

    merge_textures(texture_folder + diffuse_texture, diffuse_temp_texture, output_folder + diffuse_texture)
    merge_textures(texture_folder + normal_texture, normal_temp_texture, output_folder + normal_texture)
        

sg = simplygon_loader.init_simplygon()

process_file(input_file="ork.usdc", 
             on_screen_size= 50,
             original_texture_size= 4096,
             remeshing_texture_size= 128,
             texture_folder="textures/",
             diffuse_texture = "T_OrcShaman_1_Albedo_Skin1.png",
             normal_texture = "T_OrcShaman_1_Normal.tga",
             output_folder="output/")