Unreal Engine 4 material baking plugin

By Erik Vogelzang

Figure 1: A screenshot form the infiltrator [1] demo scene.

Introduction

Motivation

My original research subject was to port and optimize a graphics technology demo called “Infiltrator” [1], to run on an Android [2] phone. This demo was created in Unreal Engine 4.22 (UE4) [3] which is a videogame development environment. After running the demo on my phone it turned out that none of the materials [4] were compatible with the hardware in the phone. The biggest problem was the amount of texture samplers [5] that were used. You can think of a texture [6] as a picture that can be displayed on screen. Multiple textures can be combined to create the look of objects in a game. A texture sampler in UE4 is used to load a texture and use it to create visual effects. The Android [2] build platform in UE4 only supports a max total of four texture samplers at a time for a single material. This is a quarter of what a PC is allowed to use. This is a problem since the materials for the demo are made with PCs in mind. I tried to recreate a material manually to reduce the amount of texture samplers while keeping the output material the same. I was able to reduce the number of samplers by packing the Roughness, Metallic and Specular textures into a single texture. This can be done using any image editing tool like Photoshop [7]. This helped but was not enough for many materials. I was able to reduce the number of samplers even further, by rendering each material property into a render texture [8] and using the result in the new material. Thus skipping complicated blending logic in many materials. This worked and clearly it was possible to keep the sampler count to a maximum of four, for almost all of the materials used in the demo. Some materials did not look correct because they use external variables like position in the world or time passed to determine the result. They are dynamic and cannot be baked.

The problem

The UE4 Infiltrator [1] demo has over 900 different materials. Replacing these materials manually wouldn’t be realistically feasible nor time efficient. For reference it takes about five minutes for me to do the process manually so it would be roughly 75 hours of material baking by hand excluding the baking/loading times.

The solution

I have been working on an UE4 plugin that can automatically bake materials without the need for external image editor tools like Photoshop. In this blog post I will explain how my plugin uses C++ [19] (the programming language used to create UE4) to extend the UE4 source code to automate the baking process and give control over the resulting material. The contrast of the blueprint [9] system vs C++ for my scope will also be explained.

Optimizations for Android ES2

Possible optimizations

Android devices in UE4 use the OpenGL ES2 renderer [10] which has a limitation of a maximum of four texture samplers per material. In UE4 a texture sampler is a node in the UE4 material editor which allows designers to add textures to the material. They can also be used in combination with other nodes to change the look of the output material. If I take the M_Infil1_Courthouse_Roof_Copper_Trim material from the demo than I can see that it uses twelve texture samplers to achieve its look. UE4 also needs three texture samplers for lighting calculations. This means that I have room for one extra sampler. Figure 2 and figure 3 show the material graph and the sampler count respectively.

Figure 2: The graph for the M_Infil1_Courthouse_Roof_Copper_Trim material.
Figure 3: The shader information for the M_Infil1_Courthouse_Roof_Copper_Trim material.

To get the texture sampler count from twelve to four or below I will have to bake the output of the material [4]. The right most node is the final node of the material which takes in different properties. In this case it takes in Base Color, Normal, Metallic and Roughness. This means that I only need an input for each of these properties. My goal is to render each property separately and store each output in a unique texture. Afterwards I can use these resulting four textures to achieve the same output result in a new material. Gallery 1 shows the resulting textures.

Some of the material properties like Metallic, Roughness and Specular are greyscale [11] textures. That means the color information per pixel is stored as a single value between 0 (black) and 255 (white). A texture sampler can take in 4 values between 0 and 255 for each pixel color for Red, Green, Blue and Alpha transparency. This means that the values for Roughness, Metallic and Specular can be combined in a single sampler. Gallery 2 shows the results, the combined texture is green with red because the Metallic is stored in the green channel and the Roughness is stored in the red channel.

With these optimizations the texture sampler count can be reduced to six because there are three texture samplers and UE4 also requires three more samplers for lighting. I can further decrease the amount of texture samplers by using the “Shared: Wrap Sampler Source” property [11], its functionality is described by UE4 as: “Shared sampler source that does not consume a sampler slot”. This functionality uses a single sampler slot by combining all textures in a huge texture. At the same time it registers which part of the huge texture should be used for each sampler. Gallery 3 shows the difference between the old and new material, on the left and right respectively.

Gallery 3: Shows the results of baking the material with the old and new material left and right respectively.

The limitations of my optimizations

The optimizations that are described in the previous section are shown in a scenario where the result looks almost the same as the original material. Some information might have to be discarded to stay under the limit of four texture samplers which will visibly affect the quality, thus resulting in information loss. One of the causes of loss can be that the number of material properties using a RGBA value is higher than four. One of the available strategies to compensate information loss is to store multiple greyscale properties as explained previously. It is the responsibility of the designer and/or programmer to choose which properties will be excluded or retained by a smart selection. Clearly, the final material, will look visibly different from the original.

Another challenge is that some materials use functions which require world position, camera vectors or other external variables. These cannot be baked since they are not constant values. Possible solutions are:

  1. Remove the use of external variables and have a simpler material, but resulting in a different looking material.
  2. Process the external variables after the baking of materials is finished. This requires that one material property is reserved for the external variables, which can only be processed after the baking, thus sacrificing other properties. So if an external variable is used in combination with a texture to create an effect then that means that I would have one less sampler to use for other layers and/or effects.
  3. Implement a custom solution for the current material that best matches the original. This would required that the external variable must be simulated prior to the baking process.

Since my project focuses on automation, I will only focus on solution 1 and 2 since these can be achieved through automation whilst solution 3 requires a per basis-analysis, which is fully dependent on each unique material, thus it is not fit for automation in my tool.

Automation with UE4 blueprints

Rendering to textures

UE4 has a blueprint node [12] called “Draw Material to Render Target” [13] which takes the Emissive property of a material and renders it on a given render target. After the rendering is finished I can manually create a texture [6] from the render target [8] with the “Create Static Texture” option in the right-click menu of a render target. The difficult part is looping through all the properties, attaching one property to the Emissive output, rendering it and then going to the next one. It is not possible in UE4 to edit materials directly via a blueprint [9]. This means that the user has to manually modify the old material with a “baking material function” (BMF). This is the material baking function that I created thus facilitating the baking of material properties. It uses a value from a “Material Parameter Collection” (MPC) [15] as input. Values in a MPC can be changed in blueprints and can be accessed in material functions [20]. I need a single value in the MPC to determine which property should be rendered. For instance the selection Base Color = 1 or Metallic = 2, etc. The BMF needs to be manually inserted before the output node of the material you want to bake. Figure 4 shows the position of the BMF in the material with the baking node highlighted in orange. Figure 5 displays the complexity of the logic graph of the “baking material function” (BMF).

Figure 4: The orange colored “baking material function” (BMF) is inserted as the last node before the output.
Figure 5: The logic graph of the “baking material function” (BMF).

The BMF takes in all the material properties, checks the value of the MPC against a constant value for each property. The outcomes are as follows. If the MPC doesn’t match the constant for a property it will output black otherwise it will output the corresponding property value. The output values are blended together using an additive blend. Figure 6 visualizes the baking logic.

Figure 6: Explains how the baking node works using the MPC.

This process is efficient because it allows different properties to be rendered without having to change the material thus preventing a time costly UE4 recompilation.

How the blueprint tool was used

First the user needs to load a scene with a blueprint actor [7] for each of the properties that need to be rendered. Figure 7 shows what that looks like in the UE4 editor.

Figure 7: The white spheres are objects that are used render different material properties the different render targets.

The white spheres are blueprints that will switch the MPC value for each property and start rendering the texture process. They take in a render target which must be unique for each property it represents. It takes in the material that you want to bake with the BMF inserted before the final output. Each one also has the unique constant value that should match the MPC for the corresponding property.

Now a user opens the plugin, selects a material and clicks “Setup”. This will load the material information. Subsequently the blueprint objects need to be prepared according to the material as described in the previous paragraph. The user can then click “Bake” which will trigger the render process and create new textures. Figure 8 shows an example of resulting textures.

Figure 8: Resulting texture assets.

After the baking process the greyscale textures, Roughness, Specular and/or Metallic, need to be manually packed into a single RGBA texture. This can be achieved by selecting specific image editor programs, because the texture format is .hdr which means the files can only be opened in Photoshop [7], GIMP [15] or a programming language. I used Python [16] with the OpenCV [17] module which allows for programmatically editing of images. In Figure 9 an example is shown of a python script for greyscale packing.

Figure 9: Python script running to pack multiple greyscale textures into a single RGBA texture.

After running the Python script I imported the packed texture in the UE4 project. Next action is duplicating the original material and then add the textures to texture samplers. The user has to make sure that the sampler source is set to “shared: wrap” [11].

Limitations of the blueprint tool and possibilities for the C++ tool

Material Instances

UE4 has normal materials and material instances [18] which are configurable materials. In the blueprint process it is required to manually insert the baking function which is time consuming and prone to human errors. This is time consuming because one has to copy the original “root” material and adjust every setting manually within the new material. The C++ tool can automate this process which saves a lot of time.

Optimizations

The blueprint tool required the user to manually pack greyscale textures and do other optimizations while the C++ tool can automate that process.

Managing files

The blueprint tool requires the user to manually manage files while the C++ tool can handle this in a clean way by grouping materials and their textures together in a new folder with the “_mobile” suffix.

User choice

The new tool can give the user more choice in which material properties to use and which resolution the output textures should be. In addition the C++ tool can, theoretically, provide options to handle external variables that cannot be baked.

To summarize, the C++ version of the tool is significantly more automated than the blueprint tool.

The first demo

The first version of the C++ tool contains the basic functionality as are present in the blueprint tool, for comparison. The user interface looks the same as the old tool but now it is fully automated. In this section I will explain what I did to make it work and what limitations are still present.

The Initialization

The C++ tool uses a custom object [25] called a RenderChannel to store information for each material property, this includes texture information and related output node information. It also has functions to clear the data and store new data, this means that RenderChannel objects only have to be constructed once. For the demo they will be limited to the Base Color, Metallic, Roughness, Specular, Emissive, Normal and Opacity Mask properties, which is in line with the blueprint tool.

Figure 10: The code for initializing the information needed to bake a material.

The first argument in the Init function (Figure 10) will determine the tag of the RenderChannel which is used to determine which material property it corresponds to. The value of second argument is matched with the MPC value while rendering.

Setting up the material

The first thing the user has to do is select a material and click the “Setup” button. This will create a new name and path for the new material using the name and path of the old material. It will also duplicate the original material so it can make changes without altering the original. It takes some time for the temporary material to compile by UE4, subsequently the C++ tool loads the material property data.

Baking expressions

The blueprint tool uses the BMF which is able to render properties individually to the Emissive output. Recreating the BMF with C++ required me to make my own material function class. This class contains functions that allow me to create new material function nodes and create a new material function. The C++ tool partially uses UE4 source code from the material editor to achieve this. It handles creating nodes for inputs, outputs and the basic building blocks for complex functions. Basic blocks are like subtract, if, const and blending expressions. It also has functions to bind the resulting material function to a material and to connect expressions together. Since the custom material functions are build outside of the UE4 material editor, memory needs to be managed separately.

Material blending and shading

Two important properties that I have to take into account when baking a material are the blend mode [21] and the shading model [22]. The blend mode setting lets the user choose how the material will be blended with other materials in a game world. There are 6 possible blending modes (figure 11):

  1. Opaque: this is the default blending mode and it does not blend with other materials in a game world.
  2. Masked: this is the same as opaque but a developer can set a mask which tells the engine to skip rendering certain parts of the material, useful for rendering leaves for instance.
  3. Translucent: this allows a developer to change the opacity of the entire material, this is useful for materials like glass.
  4. Additive: this adds the color of the material to the color of the background.
  5. Modulate: this multiplies the color of the material with the color of the background.
  6. Alpha composite: this lets the developer manually set which parts of a material are blended additively, and which parts are blended translucently using the opacity of a material.
Figure 11: The visual difference between blending modes.

For the first iteration of the tool I only supported the Opaque blend mode, this is because it is the default blending mode. Blend modes are combined with shading models. A shading model changes how the material will be lit. Some shading models also add extra properties to or remove properties from the material. UE4 offers 10 different shading models:

  1. Default lit: this is the default shading model that is used for most materials.
  2. Unlit: this is used for materials that should not be lit by the environment. But instead uses an emissive property to light the environment. Can be used for fire or other light sources.
  3. Subsurface: this model is used to emulate the look of light penetrating a surface and diffusing throughout it. This is most useful on skin and ice for instance.
  4. Preintegrated skin: Lower cost version of Subsurface shading model.
  5. Clear coat: this model allows for rendering a thin clear or translucent surface over another surface.
  6. Subsurface profile: this is a heavier and more percise version of the Subsurface model.
  7. Two sided foliage: this emulates light passing through uniform natural materials like leaves.
  8. Hair
  9. Cloth
  10. Eye

Below is a spreadsheet that shows compatibility with different properties and shading models (Figure 12).

Figure 12: Spreadsheet displaying the differences between the UE4 shading models.

Because of the complexity combining lighting and blending modes, which have different properties, I have focused on only the default lit shading model and the opaque blending mode for the first demo.

Result of the first demo

The result is a tool that allows the user to bake materials with the opaque blend mode and default lit shading model. Next I will be supporting all of the shading models and blending modes.

User interface improvements

For the next step I worked on supporting all shading models and blending modes offered by UE4. I improved the user interface by giving the user the option to toggle for each available property. A property is only available if it was present in the original material. I also added the option to change the output resolution down to a minimum of 1/16 of the original, to lower the render time on mobiles. Figure 13 shows the new interface.

Figure 13: The new user interface.

Resolution

When clicking the setup button the tool will retrieve all the textures that are used in the original material and check which has the highest resolution. The highest resolution is shown in the interface as “Source Resolution”. The user is now able to select the “Target Resolution” to be a factor of the original. Factors of the original resolution are used instead of a custom resolution input field because using factors means that the resulting texture will always fit correctly on the new material.

Fetching material properties

The tool has to detect which layers are used in the original material. This is can be complicated because some properties are used differently based on the shading model that is used. To resolve this complication, I have done extensive research to analyse the relationship between properties and shading models. This in-depth knowledge is essential to identify and separate properties using the same input node although they are not identical. This information is summarized in Figure 14.

Figure 14: The differences between the UE4 shading models again.

For example, for property “Base color”, all green colored cells, represent Shading models with identical properties. Here, the single red colored cell indicates that the corresponding property is not available for the corresponding Shading model. Consequently the corresponding checkbox is disabled. In contrast, for property “Custom data 0” the blue colored cells all represent different and unique behavior. The C++ tool must be able to separate them make them available as separate options in the checkbox menu. The memory address of the final node that connects to the property output is saved by the tool and is linked to a checkbox based on the table above (Gallery 5).

By linking the checkbox data and the output node information, it was possible to take all the possible combinations into account.

Material instances

Differences between materials

UE4 has two different types of materials as mentioned earlier in chapter 3. The normal materials are build using a graph of nodes. The second, material instances, are a special kind of configurable material that takes a normal material as a root material (Figure 15). A designer adds configurable properties to the root material from which new material instances can be created (Gallery 16). These material instances allows the user to change any configurable property defined in the root material without affecting the root material. The material instance in Gallery 6 looks completely different from the root material in figure 15.

Figure 15: root material

Reading material instances

Using material instances with the tool requires extra steps.

  1. Finding the root material
  2. Read material property settings from the selected instance
  3. Apply the property settings to the duplicated material

Finding the root material

The tool searches for the root material recursively as shown in figure 16.

Figure 16: Logic for finding the root parent of a material instance.

The recursive check (Figure 16) is needed because a material instance can also be created from another material instance, it will always have a root material as an origin. The root material is then duplicated so that the tool can edit it without potentially breaking the original root material.

Reading and applying material property settings

There are six different types of configurable parameters that have to be read.

  1. Scalar – a single float value per scalar
  2. Font – a text font
  3. Static component mask – these are used to remove color channels from an image. The channels for red, green, blue and transparency can be toggled by the developer
  4. Static switch – if else logic branches
  5. Texture
  6. Vector – four float values per vector and outputs them as r,g,b,a

For the found root material a temporary array is created for each type of configurable parameter and filled with new information from the originally selected material instance. The C++ tool loops through each array, fetches the values from the temporary arrays for the specific parameter type and the unique name of the parameter. The name is used to look up the corresponding parameter in the previously duplicated root material. Once a corresponding parameter is found in the root material it is overwritten with the new value. After applying the property settings the resulting material will be a normal material which means that the conversion is completed.

Texture optimizations in C++

The automated tool is using two methods to reduce the amount of textures needed for the final material.

  1. texture packing
  2. texture to constant optimization

The first technique is explained in this section of the introduction. The difference between the previously explained method and the final method is that it is no longer a manual process.

The texture to constant optimization (TTCO) is a technique that checks if a texture can be converted to a constant value. If the texture is a greyscale texture than it is converted to an integer value otherwise it is converted to a vector value (Figure 17).

Figure 17: Textures with a solid color can be converted to constants or vectors.

When TTCO is combined with texture packing it will result in a constant vector but only if all layers are convertible to constant. Figure 18 visualizes the applied logic.

Figure 18: The logic for converting Roughness, Metallic and Specular textures.

Modes of operation

The material baking plugin has a manual mode which gives more control to the user and an automatic mode which can convert many materials in a row (Galery 7).

Automatic mode

In automatic mode the user can select which material properties should be allowed and which resolution should be used for the materials. After this the user can select multiple assets in the content browser and click on start.

Figure 19: A selection of multiple assets including some materials.

Selected material assets are stored in a buffer ignoring assets that are not identified as a material. Afterwards the baking process will begin and all materials in the buffer will be baked. The C++ tool will show a progress indicator during the baking process (figure 20).

Figure 20: The progress indicator of the automation mode.

Manual mode

The manual mode gives the user the option to specifically select material properties for baking. In addition a resolution can be defined per material basis. Loading the assets works the same as with the automatic mode but only the first material in the asset buffer is loaded and the rest is discarded.

Conclusion

The goal of the C++ tool was to easily convert multiple materials from PC to mobile in UE4. The tool gives the user options to change resolution and material properties to finetune the results. These are the features of the final product.

  1. Support for all of the shading models in UE4.
  2. Support for all blending modes in UE4.
  3. Support for baking material instances.
  4. Texture packing for Metallic, Roughness and Specular properties.
  5. Support for changing output resolution.
  6. Converting properties to constant values if possible.
  7. Reducing the material complexity.
  8. Integration with the UE4 content browser and version control systems.
  9. An automated mode for converting many materials with limited control over individual materials.
  10. A manual mode which gives the user more control over a single material.

I believe it is possible to create a automated system to handle materials with external variables, however this would require a follow-up project, because it is very complex and would take multiple extra months to finish. I am proud of the results and I will be developing this tool further so I can publish it in the UE4 asset store.

Research Sources

[1] Epic Games, “Infiltrator Demo” https://www.unrealengine.com/marketplace/en-US/product/infiltrator-demo

[2] Google LLC, “Android” https://source.android.com/

[3] Epic Games, “Unreal Engine 4.22 released” https://www.unrealengine.com/en-US/blog/unreal-engine-4-22-released

[4] Epic Games, “Materials” https://docs.unrealengine.com/4.26/en-US/RenderingAndGraphics/Materials/

[5] Epic Games, “What counts as a texture sampler?” https://forums.unrealengine.com/t/what-counts-as-a-texture-sampler/285029

[6] Epic Games, “Textures” https://docs.unrealengine.com/4.26/en-US/RenderingAndGraphics/Textures/

[7] Adobe Inc., “Photoshop” https://www.adobe.com/products/photoshop.html

[8] Epic Games, “Render Targets” https://docs.unrealengine.com/4.27/en-US/RenderingAndGraphics/RenderTargets/

[9] Epic Games, “Introduction to Blueprints” https://docs.unrealengine.com/4.27/en-US/ProgrammingAndScripting/Blueprints/GettingStarted/

[10] Aaftab Munshi, “OpenGL ES Common Profile Specification 2.0” https://www.khronos.org/files/opengles_spec_2_0.pdf

[11] Epic Games, “SamplerSource” https://docs.unrealengine.com/4.26/en-US/API/Runtime/Engine/Materials/UMaterialExpressionTextureSample/SamplerSource/

[12] Epic Games, “Blueprint node” https://docs.unrealengine.com/4.27/en-US/ProgrammingAndScripting/Blueprints/UserGuide/Nodes/

[13] Epic Games, “Creating Textures Using Blueprints and Render Targets” https://docs.unrealengine.com/4.26/en-US/RenderingAndGraphics/RenderTargets/BlueprintRenderTargets/HowTo/CreatingTextures/

[14] Epic Games, “Material Parameter Collections” https://docs.unrealengine.com/4.27/en-US/RenderingAndGraphics/Materials/ParameterCollections/

[15] The GIMP Development Team, “GIMP” https://www.gimp.org

[16] Guido van Rossum, “Python” https://www.python.org/

[17] OpenCV Team, “opencv-python” https://pypi.org/project/opencv-python/

[18] Epic Games, “Instanced Materials” https://docs.unrealengine.com/4.27/en-US/RenderingAndGraphics/Materials/MaterialInstances/

[19] Bjarne Stroustrup, “C++” https://cplusplus.com/

[20] Epic Games, “Material functions” https://docs.unrealengine.com/4.27/en-US/RenderingAndGraphics/Materials/Functions/

[21] Epic Games, “Material blend modes” https://docs.unrealengine.com/4.26/en-US/RenderingAndGraphics/Materials/MaterialProperties/BlendModes/

[22] Epic Games, “Shading models” https://docs.unrealengine.com/4.27/en-US/RenderingAndGraphics/Materials/MaterialProperties/LightingModels/

A large part of my research was done by researching UE4 source code. The files below are the most important. They are all from the source cited directly below.

Epic Games (4-9-2019) Unreal Engine (Version 4.23) [Source code]. https://www.unrealengine.com.

The following files are used for researching how I can integrate the default content browser into my tool for moving and packaging assets and retrieving assets data.

  1. AssetRegistryModule.cpp
  2. AssetToolsModule.cpp
  3. ContentBrowserModule.cpp
  4. IContentBrowserSingleton.cpp
  5. ObjectTools.cpp
  6. SContentBrowser.cpp

The following files are used for researching how materials are made and how I can customise a material using C++ code.

  1. Material.cpp
  2. MaterialEditingLibrary.cpp
  3. MaterialExpression.cpp
  4. MaterialFunction.cpp
  5. MaterialInstance.cpp
  6. MaterialInstanceDynamic.cpp
  7. MaterialParameterCollection.cpp

The following files are used for researching how a material can be baked.

  1. KismetMaterialLibrary.cpp
  2. KismetRenderingLibrary.cpp
  3. PixelFormat.cpp
  4. RenderingThread.cpp
  5. Texture.cpp
  6. TextureRenderTarget2D.cpp
  7. TextureResource.cpp

Leave a Reply

Your email address will not be published. Required fields are marked *