Custom material casting using mapping image
Disclaimer: The code in this post is written using version 9.1.36100 of Simplygon. If you encounter this post at a later stage, some of the API calls might have changed. However, the core concepts should still remain valid.
Introduction
In Simplygon to optimize material complexity you use material casters. However sometimes you need full control of the material casting process and in these cases, you can use the mapping image to transfer data from the original asset to the optimized asset.
In this post we’ll demonstrate how you can use the mapping image to cast static lightmaps into the diffuse map of a remeshed asset to save texture lookups in the shader.
Prerequisites
This example will use the Simplygon integration in Unity, but the same concepts can be applied to all game engines using the Simplygon API.
Problem to solve
We want to create an optimized asset we can use as a faraway LOD. Due to performance requirements, we can only use a single texture so everything from material color to lighting information must be in the same texture.
We’ll use the following input asset with lightmapping enabled to demonstrate our approach. All input meshes should be combined into one and the optimized asset should only use a single texture.
Remeshing
Let us start optimizing the input asset using the remeshing processor with an attached material caster for the diffuse channel. This will reduce the number of triangles and create a single mesh we can render with just one draw call.
Here you can see the newly create diffuse texture for the remeshed input asset (left) and the asset rendered with a unlit shader (right).
As you can see the lighting information is lost so we need to find a way to map the lightmap created by Unity and blend it with our newly created diffuse texture. This is where the mapping image comes in.
Mapping image
The mapping image is a data structure you can use to map a point on the optimized asset back to the original asset. We can use this to map the texcoord space of the optimized asset back to the original asset and sample the lightmap created by Unity. You can find more info in the mapping image section in the documentation.
NOTE
In Unity the lightmaps are by default not accessible using a script. To be able to sample the lightmap you need to enable Read/Write in the properties for the lightmap textures.
We'll use the mapping image to transfer the lighting data from the original lightmap to the newly created diffuse texture. You can see how this is done using the Simplygon API at the end of this blog post.
Here you can see the result of mapping the original lightmap (left) to the texcoord space in the optimized asset (middle) and blended with the diffuse texture (right).
Result
With the newly blended texture we can apply this to the unlit shader and compare the result with the original asset. As you can see we are very close to the original but with fewer triangles and a much simpler shader.
Original asset (left) and optimized asset rendered with a unlit shader with only one texture (right).
Complete script
using Simplygon;
using Simplygon.Unity.EditorPlugin;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.IO;
using UnityEditor;
using UnityEngine;
using System;
public class CustomShader : EditorWindow
{
private SimplygonUI.ThreadDispatcher mainThreadDispatcher;
private List<GameObject> selectedGameObjects = new List<GameObject>();
public CustomShader()
{
mainThreadDispatcher = new SimplygonUI.ThreadDispatcher();
}
[MenuItem("Window/SimplygonWithCustomerShader")]
static void Init()
{
var window = EditorWindow.GetWindow(typeof(CustomShader), false, "SimplygonWithCustomShader");
}
int GetPixelIndex(int x, int y, int chunkWidth, int samplingX, int samplingY, int multiSamplingLevels)
{
return (x * multiSamplingLevels + samplingX) + (y * multiSamplingLevels + samplingY) * (chunkWidth * multiSamplingLevels);
}
bool GetOriginalTriangleIndex(spMappingImageMeshData mappingImageMeshData, int globalTriangleIndex, out int geometryIndex, out int localTriangleIndex)
{
geometryIndex = -1;
localTriangleIndex = -1;
if(globalTriangleIndex < 0)
{
return false;
}
// Loop through all geometries in the mapping image to map the global triangle index to a geometry index and local triangle index.
int geometryCount = (int)mappingImageMeshData.GetMappedGeometriesCount();
for (int i = 0; i < geometryCount; ++i)
{
int nextGeometryTriangleIndexOffset = -1;
if (i < geometryCount - 1)
{
nextGeometryTriangleIndexOffset = mappingImageMeshData.GetStartTriangleIdOfGeometry(i + 1);
}
if (globalTriangleIndex < nextGeometryTriangleIndexOffset || nextGeometryTriangleIndexOffset == -1)
{
geometryIndex = i;
localTriangleIndex = globalTriangleIndex - mappingImageMeshData.GetStartTriangleIdOfGeometry(i);
return true;
}
}
return false;
}
Vector3 GetNormalizedBarycentricCoordinates(Vector2 baryCentricCoordinate)
{
// Not a typo, converting coordinates from Simplygon to Unity.
return new Vector3( baryCentricCoordinate.y / (float)UInt16.MaxValue,
baryCentricCoordinate.x / (float)UInt16.MaxValue,
((float)UInt16.MaxValue - baryCentricCoordinate.x - baryCentricCoordinate.y) / (float)UInt16.MaxValue);
}
void AddClippingPlane(ISimplygon simplygon, spRemeshingPipeline sgRemeshingPipeline, spScene sgScene)
{
using spSelectionSetTable sgSelectionSetTable = sgScene.GetSelectionSetTable();
using spGeometryCullingSettings sgGeometryCullingSettings = sgRemeshingPipeline.GetGeometryCullingSettings();
using spScenePlane sgPlane = simplygon.CreateScenePlane();
sgPlane.SetNormal(new float[] { 0.0f, 1.0f, 0.0f });
sgScene.GetRootNode().AddChild(sgPlane);
var selectionSet = simplygon.CreateSelectionSet();
selectionSet.AddItem(sgPlane.GetNodeGUID());
int selectionSetId = sgSelectionSetTable.AddSelectionSet(selectionSet);
sgGeometryCullingSettings.SetUseClippingPlanes(true);
sgGeometryCullingSettings.SetClippingPlaneSelectionSetID(selectionSetId);
}
public void OnGUI()
{
try
{
List<GameObject> selectedGameObjects = new List<GameObject>();
foreach (var o in Selection.objects)
{
GameObject go = o as GameObject;
if (go != null)
{
selectedGameObjects.Add(go);
}
}
if (selectedGameObjects.Count == 0)
{
EditorGUILayout.HelpBox("Please select game object(s) to process.", MessageType.Info);
}
else
{
if (GUILayout.Button("Run Simplygon"))
{
Simplygon.EErrorCodes simplygonErrorCode = 0;
string simplygonErrorMessage = string.Empty;
using (ISimplygon simplygon = global::Simplygon.Loader.InitSimplygon(out simplygonErrorCode, out simplygonErrorMessage))
{
if (simplygon != null)
{
var progressObserver = new SimplygonObserver(mainThreadDispatcher);
EditorUtility.DisplayProgressBar("Simplygon", "Please wait while the asset is optimized.", 0.0f);
string exportTempDirectory = SimplygonUtils.GetNewTempDirectory();
using spScene sgScene = SimplygonExporter.Export(simplygon, exportTempDirectory, selectedGameObjects);
string processingName = string.Join("_", selectedGameObjects.Select(p => p.name).Distinct().ToArray());
if (sgScene != null && !sgScene.IsNull())
{
if (!AssetDatabase.IsValidFolder("Assets/LOD"))
{
AssetDatabase.CreateFolder("Assets", "LOD");
}
string assetFolderGuid = AssetDatabase.CreateFolder("Assets/LOD", processingName);
string assetFolderPath = AssetDatabase.GUIDToAssetPath(assetFolderGuid);
using spMaterialTable sgMaterialTable = sgScene.GetMaterialTable();
using spTextureTable sgTextureTable = sgScene.GetTextureTable();
using spRemeshingPipeline sgRemeshingPipeline = simplygon.CreateRemeshingPipeline();
using spRemeshingSettings sgRemeshingSettings = sgRemeshingPipeline.GetRemeshingSettings();
using spMappingImageSettings sgMappingImageSettings = sgRemeshingPipeline.GetMappingImageSettings();
sgRemeshingSettings.SetOnScreenSize(600);
// Enable the mapping image generation, will be used both by the color caster and our own lightmap casting.
sgMappingImageSettings.SetGenerateMappingImage(true);
// Add color caster for the diffuse channe
using spColorCaster sgColorCaster = simplygon.CreateColorCaster();
using spColorCasterSettings sgColorCasterSettings = sgColorCaster.GetColorCasterSettings();
sgColorCasterSettings.SetMaterialChannel("diffuseColor");
sgRemeshingPipeline.AddMaterialCaster(sgColorCaster, 0);
// Add clipping plane to the remeshing to optimize UV space usage.
AddClippingPlane(simplygon, sgRemeshingPipeline, sgScene);
int textureWidth = (int)sgMappingImageSettings.GetOutputMaterialSettings(0).GetTextureWidth();
int textureHeight = (int)sgMappingImageSettings.GetOutputMaterialSettings(0).GetTextureHeight();
sgRemeshingPipeline.AddObserver(progressObserver);
// Run the remeshing process which will create the new mesh and the diffuse texture.
sgRemeshingPipeline.RunScene(sgScene, Simplygon.EPipelineRunMode.RunInThisProcess);
int lodIndex = 1;
List<GameObject> importedGameObjects = new List<GameObject>();
SimplygonImporter.Import(simplygon, sgRemeshingPipeline, ref lodIndex, assetFolderPath, processingName, importedGameObjects);
List<MeshFilter> meshFilters = null;
if (selectedGameObjects.Count > 1)
{
meshFilters = new List<MeshFilter>();
foreach (var selectedGameObject in selectedGameObjects)
{
meshFilters.AddRange(selectedGameObject.GetComponentsInChildren<MeshFilter>().ToList());
}
}
else
{
meshFilters = selectedGameObjects.First().GetComponentsInChildren<MeshFilter>().ToList();
}
// Create cached versions of triangle indices and UVs. This will speed up things when we sample the lightmap using the mapping image.
var cachedOriginalTriangleIndices = meshFilters.Select(i => i.sharedMesh.triangles).ToList();
var cachedOriginalLightmapUVs = meshFilters.Select(i => i.sharedMesh.uv2).ToList();
var castedLightmapTexture = new Texture2D(textureWidth, textureHeight, TextureFormat.ARGB32, false);
using spMappingImage mappingImage = sgRemeshingPipeline.GetMappingImage(0);
using spMappingImageMeshData mappingImageMeshData = mappingImage.GetMappingMeshData();
using spChunkedImageData chunkedImageData = mappingImage.GetImageData();
int multiSamplingLevels = (int)sgMappingImageSettings.GetOutputMaterialSettings(0).GetMultisamplingLevel();
// Go through all chunks in the mapping image and sample the original lightmap.
for (int chunkY = 0; chunkY < chunkedImageData.GetYSize(); ++chunkY)
{
for (int chunkX = 0; chunkX < chunkedImageData.GetXSize(); ++chunkX)
{
using spImageData currentChunk = chunkedImageData.LockChunk2D(chunkX, chunkY);
using spRidArray triangleIndices = spRidArray.SafeCast(currentChunk.GetField("TriangleIds"));
using spUnsignedShortArray barycentricCoords = spUnsignedShortArray.SafeCast(currentChunk.GetField("BarycentricCoords"));
if (triangleIndices.IsNull() || barycentricCoords.IsNull())
continue;
int chunkWidth = (int)currentChunk.GetXSize() / multiSamplingLevels;
int chunkHeight = (int)currentChunk.GetYSize() / multiSamplingLevels;
for (int y = 0; y < chunkHeight; ++y)
{
for (int x = 0; x < chunkWidth; ++x)
{
Color pixel = new Color(0.0f, 0.0f, 0.0f, 0.0f);
int samplesPerPixelCount = 0;
// Loop through all sampling levels.
for (int samplingY = 0; samplingY < multiSamplingLevels; ++samplingY)
{
for (int samplingX = 0; samplingX < multiSamplingLevels; ++samplingX)
{
int pixelIndex = GetPixelIndex(x, y, chunkWidth, samplingX, samplingY, multiSamplingLevels);
int geometryIndex = -1;
int localTriangleIndex = -1;
// Get original triangle index for the current pixel in the mapping image
if (!GetOriginalTriangleIndex(mappingImageMeshData, triangleIndices.GetItem(pixelIndex), out geometryIndex, out localTriangleIndex))
{
continue;
}
// Get original barycentric coordinates for the current pixel in the mapping image
Vector3 normalizedBarycentricCoordinates = GetNormalizedBarycentricCoordinates(new Vector2(barycentricCoords.GetItem(pixelIndex * 2 + 0), barycentricCoords.GetItem(pixelIndex * 2 + 1)));
if (geometryIndex >= 0 && geometryIndex < meshFilters.Count)
{
var meshFilter = meshFilters[geometryIndex];
var meshRenderer = meshFilter.gameObject.GetComponent<MeshRenderer>();
var mesh = meshFilter.sharedMesh;
int cornerIndex1 = cachedOriginalTriangleIndices[geometryIndex][localTriangleIndex * 3 + 0];
int cornerIndex2 = cachedOriginalTriangleIndices[geometryIndex][localTriangleIndex * 3 + 1];
int cornerIndex3 = cachedOriginalTriangleIndices[geometryIndex][localTriangleIndex * 3 + 2];
Vector2 cornerUV1 = cachedOriginalLightmapUVs[geometryIndex][cornerIndex1];
Vector2 cornerUV2 = cachedOriginalLightmapUVs[geometryIndex][cornerIndex2];
Vector2 cornerUV3 = cachedOriginalLightmapUVs[geometryIndex][cornerIndex3];
// Get the original UV for the current pixel in the mapping image using the triangle index and the barycentric coordinates.
Vector2 uv = new Vector2(normalizedBarycentricCoordinates.x * cornerUV1.x + normalizedBarycentricCoordinates.y * cornerUV2.x + normalizedBarycentricCoordinates.z * cornerUV3.x,
normalizedBarycentricCoordinates.x * cornerUV1.y + normalizedBarycentricCoordinates.y * cornerUV2.y + normalizedBarycentricCoordinates.z * cornerUV3.y);
Texture2D lightMapTexture = LightmapSettings.lightmaps[meshRenderer.lightmapIndex].lightmapColor;
Vector2 lightmapScale = new Vector2(meshRenderer.lightmapScaleOffset.x, meshRenderer.lightmapScaleOffset.y);
Vector2 lightmapOffset = new Vector2(meshRenderer.lightmapScaleOffset.z, meshRenderer.lightmapScaleOffset.w);
Vector2 lightmapUV = uv * lightmapScale + lightmapOffset;
// Bilinear sample the original lightmap using normalized uvs.
pixel += lightMapTexture.GetPixelBilinear(lightmapUV.x - 1.0f / (2.0f * (float)lightMapTexture.width), lightmapUV.y - 1.0f / (2.0f * (float)lightMapTexture.height));
samplesPerPixelCount++;
}
}
}
if (samplesPerPixelCount > 0)
{
pixel /= samplesPerPixelCount;
}
castedLightmapTexture.SetPixel(chunkX * chunkWidth + x, chunkY * chunkHeight + y, pixel);
}
}
chunkedImageData.UnlockChunk2D(chunkX, chunkY);
}
}
castedLightmapTexture.Apply();
var diffuseTexture = new Texture2D(textureWidth, textureHeight);
diffuseTexture.LoadImage(File.ReadAllBytes(Path.Combine(assetFolderPath, "LOD1", "diffuseColor_0.png")));
// Blend lightmap with diffuse map
for (int y = 0; y < textureHeight; ++y)
{
for (int x = 0; x < textureWidth; ++x)
{
Color light = castedLightmapTexture.GetPixel(x, y);
Color diffuse = diffuseTexture.GetPixel(x, y);
diffuseTexture.SetPixel(x, y, diffuse * light);
}
}
diffuseTexture.Apply();
System.IO.File.WriteAllBytes(Path.Combine(assetFolderPath, "LOD1", "lightmap.png"), castedLightmapTexture.EncodeToPNG());
System.IO.File.WriteAllBytes(Path.Combine(assetFolderPath, "LOD1", "diffuseWithLightmap.png"), diffuseTexture.EncodeToPNG());
UnityEditor.AssetDatabase.Refresh();
}
else
{
Debug.LogError("Failed to export selection for Simplygon processing");
}
EditorUtility.ClearProgressBar();
Directory.Delete(exportTempDirectory, true);
}
else
{
Debug.LogError(simplygonErrorMessage);
}
}
}
}
}
catch (System.Exception ex)
{
Debug.LogException(ex);
EditorUtility.ClearProgressBar();
}
}
}