THE FX FILE FORMAT - PART TWO - CARTOON SHADING IN DBPRO
The goal of this tutorial is two-fold. The first goal is to provide the reader with a firmer grasp on the cartoon shading abilities of DBPro so each developer can create a more personalized look to their cartoon shaded models. The second goal is to explore the concept of assigning media to a shader dynamically.
To that end, I present to you the cartoon shader that DBPro uses internally:
//
// FX Effect file - Cartoon
//
// Constants
matrix wvp : WorldViewProjection;
matrix world : World;
matrix worldIT : WorldIT;
float4 eyePos : EyePosition;
float4 lhtDir < string UIDirectional = "Light Direction"; >;
// Model and Texture Names
string XFile = "default.x";
texture toonshade < string name = "shade.bmp"; >;
texture toonedge < string name = "edge.dds"; >;
technique Toon
{
pass p0
{
VertexShaderConstant[0] = <wvp>;
VertexShaderConstant[4] = <worldIT>;
VertexShaderConstant[8] = <world>;
VertexShaderConstant[12] = <eyePos>;
VertexShaderConstant[13] = <lhtDir>;
VertexShader =
asm
{
// v0 -- position
// v3 -- normal
vs.1.1
dcl_position v0
dcl_normal v3
// transform position
dp4 oPos.x, v0, c0
dp4 oPos.y, v0, c1
dp4 oPos.z, v0, c2
dp4 oPos.w, v0, c3
// transform normal
dp3 r0.x, v3, c8
dp3 r0.y, v3, c9
dp3 r0.z, v3, c10
// normalize normal
dp3 r0.w, r0, r0
rsq r0.w, r0.w
mul r0, r0, r0.w
// compute world space position
dp4 r1.x, v0, c8
dp4 r1.y, v0, c9
dp4 r1.z, v0, c10
dp4 r1.w, v0, c11
// vector from point to eye
add r2, c12, -r1
// normalize e
dp3 r2.w, r2, r2
rsq r2.w, r2.w
mul r2, r2, r2.w
// e dot n
dp3 oT1.x, r0, r2
// l dot n
dp3 oT0.x, r0, -c13
};
Texture[0] = <toonshade>;
Texture[1] = <toonedge>;
MinFilter[0] = Linear;
MagFilter[0] = Linear;
MipFilter[0] = Linear;
MinFilter[1] = Linear;
MagFilter[1] = Linear;
MipFilter[1] = Linear;
AddressU[0] = Clamp;
AddressV[0] = Clamp;
AddressU[1] = Clamp;
AddressV[1] = Clamp;
ColorOp[0] = SelectArg1;
ColorArg1[0] = Texture;
ColorOp[1] = Modulate;
ColorArg1[1] = Texture;
ColorArg2[1] = Current;
}
}
This shader and its media can be found under Help->examples->basic3D.
Now to highlight the important bits.
Since we want to use whatever media we would like with this shader lets take a look at the media required by this shader.
string XFile = "default.x";
texture toonshade < string name = "shade.bmp"; >;
texture toonedge < string name = "edge.dds"; >;
This code can be found at the top of the .fx file below the constants.
The "string XFILE = "default.x" means that it will use the model "default.x". The model isn't really all that important for our purposes so we'll ignore it for now.
The next two lines are the most important. The textures toonshade and toonedge determine how the model will be shaded. You can find these in the same folder that you can find the cartoon.fx file. Lets examine them in detail.
The toonshade texture is shade.bmp and opening it up in paint you can see it is a single pixel in height and 16 pixels in width.
Note: You are going to have to zoom in a bit on it to get a good view.
Now you are probably wondering why the image is only 1 pixel high and the answer is of course that it doesn't need to be any higher. What happens is that the shader takes the dot product of the light vector and the normal of the vertex and the result is a single value. That value is used to look up a pixel in the shade.bmp.
So the shade.bmp is basically a one dimensional look up table. The closer to the light a paticular vertex is the further to the right in the texture the value is looked up.
You can change shade.bmp to hold whatever color ramp that you want. Just keep in mind that all pixels below the first line will be ignored.
The next texture is the toonedge. As you could probably have guessed this texture determines the color of the edges of the model. Its listed as edge.dds but that is incorrect. The edge.dds image is a 1 by 128 image that is just a single color: white. The example that came with DBPro that showcases the Set Cartoon Shading on command uses the edge.bmp texture instead.
The edge.bmp texture is 32 by 32 image. It consists of a gradient that starts on the right with the color white and goes to the left to the color black. Now it doesn't need to be 32 by 32. It can be just 1 by 32 as edge.bmp is a look up table just like Shade.bmp. The values for it are determined by taking the dot product of the eye vector with the normal of the vertex. The result is a single value used to look up which pixel should be multipled with the current pixel.
You are probably not going to want to modify the edge image other than to maybe make it a 1 by 32 image. However, there is nothing stoping you from trying out different values if you want the edge of your image to be shaded differently.
Now that we have covered the needed textures, lets examine how to load and apply them seperately from the shader.
Note: Before we begin, make sure that the shader and its media are in the same folder as your project. The following code will be based upon this assumption.
The first line of our source will be this:
load effect "cartoon.fx", 1, 0
The zero at the end of the command tells DBPro to use the existing textures on the model and not the ones specified in the fx file. This is different from our usual approach of leaving a one at the end of the command seeming how we want to assign the media seperately from the shader.
After that we will load in our images and create our test model:
load image "edge.bmp", 1
load image "shade.bmp", 2
make object sphere 1, 100,100,100
At this point I'd like to point out the advantage of loading in media seperately from the .fx file. If we loaded media through the .fx file, i.e. left a one at the end of the load effect command, we would be stuck with that media for whatever we applied it to. Loading the media in seperately from the .fx file would allow us to use whatever media we wanted to. We could even use different media for different objects all with the same shader.
To use our current shader as an example, we could load in multiple shade images, that may or may not be named "shade.bmp", to give a different color to a variety of objects. We wouldn't be able to do that if we loaded in the hardcoded media. The only way we could achieve an effect like that is to have seperate shader files each with a different shade image and that could be pretty wasteful of both space and time.
For simple demos, the hardcoded way is probably easier, but for games that have varying texture requirements the dynamic way is the way to go.
Now we have the most important part to do and that is assigning the loaded textures to the shader. Its surprisingly simple to do and it involves the texture object command. Insert these two lines into your DBPro code that you have so far:
texture object 1, 0, 2
texture object 1, 1, 1
The first number, 1, is the object to be textured. The second number is the stage index to which the texture is being assigned. After that is the image number.
Now in order to determine the proper texture stage index we need to look back at the shader code.
Under the vertex shader assembly is a series of render states. This section of code looks like this:
Texture[0] = <toonshade>;
Texture[1] = <toonedge>;
MinFilter[0] = Linear;
MagFilter[0] = Linear;
MipFilter[0] = Linear;
MinFilter[1] = Linear;
MagFilter[1] = Linear;
MipFilter[1] = Linear;
AddressU[0] = Clamp;
AddressV[0] = Clamp;
AddressU[1] = Clamp;
AddressV[1] = Clamp;
ColorOp[0] = SelectArg1;
ColorArg1[0] = Texture;
ColorOp[1] = Modulate;
ColorArg1[1] = Texture;
ColorArg2[1] = Current;
The two that we are interested in are the Texture[0] = <toonshade>; and the Texture[1] = <toonedge>;.
The number in brackets is the texture stage index. So for the toonshade texture we would assign a 0 and for the toonedge texture we would assign a 1. Its as simple as that.
Unfortunately, not all shaders use the Texture[n] state command. Only the shaders that use assembly usually do. So for HLSL shaders most of the time you'll need to do a bit more work. For the shader being used, you'll have to look in its input parameters for a variable or a variable of a structure that contains the TEXCOORD
n semantic. From there it will take a little investigative work to figure out which texture is assigned to which. Usually you'll need to look for the tex2D function or its derivitives in the pixel shader. You'll see a variable that contains the tex coords to be used and a sampler state name. With the name of the variable and the specific texcoord semantic assigned to it all that is left to do is find the sampler state specified by the sampler state name.
Sampler states are usually declared above the function somewhere. They are usually easy to find because they start with a keyword like sampler2D or samplerCUBE followed by the name of the sampler. In them you'll find a Texture = <whatever>; and the "whatever" is the name of your texture.
Now that we have assigned the textures to their appropriate stages, we can assign the shader to the object.
After that we have a little while-loop that exits when the space bar is pressed. The code from here on out is pretty standard fare so I'll just list it as it needs no real explanation.
rem Vector for our lhtDir constant
null = make vector4(1)
while spacekey() = 0
rem Control light position using mousex
position light 0,mousex()-320,240-mousey(),-300
rem Make sure our light is constantly pointing at the object
point light 0, object position x(1), object position y(1), object position z(1)
rem Update the light direction
set vector4 1, light direction x(0), light direction y(0), light direction z(0), 1.0
rem Make sure the direction constant is constantly updated
set effect constant vector 1,"lhtDir",1
endwhile
But the cartoon shader that comes with DBPro is rather limiting. You can only shade the object one color and there is no option to apply a texture to it. So you end up with a rather uninteresting monochrome object. We can do better.
As an added bonus, I will add some additional code to DBPro's cartoon shader improving it. It will now take an additional texture in as a base texture so you can detail your model even more.
Add this code beneath the first technique:
technique ToonWithTex
{
pass p0
{
VertexShaderConstant[0] = <wvp>;
VertexShaderConstant[4] = <worldIT>;
VertexShaderConstant[8] = <world>;
VertexShaderConstant[12] = <eyePos>;
VertexShaderConstant[13] = <lhtDir>;
VertexShader =
asm
{
// v0 -- position
// v3 -- normal
vs.1.1
dcl_position v0
dcl_normal v3
dcl_texcoord0 v6
// transform position
dp4 oPos.x, v0, c0
dp4 oPos.y, v0, c1
dp4 oPos.z, v0, c2
dp4 oPos.w, v0, c3
// transform normal
dp3 r0.x, v3, c8
dp3 r0.y, v3, c9
dp3 r0.z, v3, c10
// normalize normal
dp3 r0.w, r0, r0
rsq r0.w, r0.w
mul r0, r0, r0.w
// compute world space position
dp4 r1.x, v0, c8
dp4 r1.y, v0, c9
dp4 r1.z, v0, c10
dp4 r1.w, v0, c11
// vector from point to eye
add r2, c12, -r1
// normalize e
dp3 r2.w, r2, r2
rsq r2.w, r2.w
mul r2, r2, r2.w
// e dot n
dp3 oT1.x, r0, r2
// l dot n
dp3 oT2.x, r0, -c13
//Dump our basetex's coords into oT0.
mov oT0, v6
};
Texture[2] = <toonshade>;
Texture[1] = <toonedge>;
Texture[0] = <basetex>;
MinFilter[0] = Linear;
MagFilter[0] = Linear;
MipFilter[0] = Linear;
MinFilter[1] = Linear;
MagFilter[1] = Linear;
MipFilter[1] = Linear;
MinFilter[2] = Linear;
MagFilter[2] = Linear;
MipFilter[2] = Linear;
AddressU[2] = Clamp;
AddressV[2] = Clamp;
AddressU[1] = Clamp;
AddressV[1] = Clamp;
AddressU[0] = Wrap;
AddressV[0] = Wrap;
ColorOp[0] = SelectArg1;
ColorArg1[0] = Texture;
ColorOp[1] = Modulate;
ColorArg1[1] = Texture;
ColorArg2[1] = Current;
ColorOp[2] = Lerp;
ColorArg1[2]= Texture;
ColorArg2[2] = Current;
}
}
And add this code up at the top beneath the two texture declarations for toonshade and toonedge:
//Our base texture
texture basetex < string name = "base.bmp"; >;
Now we only have to add a few lines of code to our existing DBPro source to get it to work. The first line will of course be a load image command after all of the other load image commands. It will load in our texture that will be used as base. For this example we will be using the marble.bmp texture that came with DBPro. It can be found in the same directory as all of the other media that we have used so far.
load image "marble.bmp", 3
Next we will assign the texture to the object. We will assign a texture stage index of 2 to our base texture. Place this code just below all of the other texture object commands:
The last line we will need to add will switch the current technique, which doesn't accept a base texture, to the new technique that will. Place this under the set object effect command:
Set Effect Technique 1, "ToonWithTex"
The final version of the source can be found below. Note: This source will not run the old version of the shader but the new one only. Also, the new version of the cartoon shader is still named cartoon.fx in the source though you can change that if you want.
load effect "cartoon.fx", 1, 0
rem Load our images
load image "edge.bmp", 1
load image "shade.bmp", 2
load image "marble.bmp", 3
make object sphere 1, 100,100,100
rem Assign our images to texture stage indexs
texture object 1, 0, 2
texture object 1, 1, 1
texture object 1, 2, 3
set object effect 1, 1
rem New and improved version of the Cartoon.fx effect
set effect technique 1, "ToonWithTex"
rem Vector for our lhtDir constant
null = make vector4(1)
while spacekey() = 0
rem Control light position using mousex
position light 0,mousex()-320,240-mousey(),-300
rem Make sure our light is constantly pointing at the object
point light 0, object position x(1), object position y(1), object position z(1)
rem Update the light direction
set vector4 1, light direction x(0), light direction y(0), light direction z(0), 1.0
rem Make sure the direction constant is constantly updated
set effect constant vector 1,"lhtDir",1
endwhile
For future work, you can always modify the existing code to switch back and forth between shader techniques or other shaders. Since applying the texture object command to an object will wipe whatever shading is being done to it, you can simply set up the texture stages for the new shader or technique with the texture object command and apply whatever shader you want or none at all. There is no special command needed to stop an object from being shaded other than using a texture object command on it.
That concludes the second tutorial of the FX FILE FORMAT series. I hope that this has been an informative experience that answers many of the questions surrounding cartoon shading in DBPro.
