Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Professional Discussion / How to get the vertex colour in a shader?

Author
Message
Fallout
23
Years of Service
User Offline
Joined: 1st Sep 2002
Location: Basingstoke, England
Posted: 11th May 2012 17:32
I'm trying to make a shader which modifies a colour in the VertexShader. To do this, I want access to a vertexes diffuse/specular or whatever colour. I'm not too fussed, so long as I can access a colour value. Anyone know how to do this?

What I have so far:

My VertexShader in structure looks like this:


For the value Diffuse I have tried COLOR, COLOR0 and COLOR1 in combination with the following modifications in my program code:

SetObjectDiffuse()
SetObjectSpecular()
ConvertObjectFVF() <- To ensure it has specular/diffuse values
Lock model, loop through all vertices and SetVertexDataDiffuse() to a colour.

In all these combinations, I only get black (0) into the diffuse variable. The documentation for vertexshader semantics says COLOR/0/1 specifically refer to DIFFUSE and SPECULAR. So does anyone know how to get those values passed into my vertexshader?

The actual problem I'm trying to solve is, I want to set an overall brightness for my model, but I want to do it on a per model basis. So I can't pass in a constant(tweakable) diffuse, as this would apply to all models using this effect. I want to be able to set each overall brightness for each model individually, and using the diffuse/specular colours makes the most sense, but I just can't figure out how to access it.

Any clue?!

Brendy boy
21
Years of Service
User Offline
Joined: 17th Jul 2005
Location: Croatia
Posted: 11th May 2012 17:49
Quote: "SetObjectDiffuse()
SetObjectSpecular()"

i'm quite sure that commands don't affect vertex diffuse/specular components (only material components which has nothing to do with vertices)

First, you need to convert object to fvf format which has diffuse component, then set each vertex color with set vertexdata diffuse command.
the way you access the color in a shader seems ok to me.

Fallout
23
Years of Service
User Offline
Joined: 1st Sep 2002
Location: Basingstoke, England
Posted: 11th May 2012 19:33
That would make sense that the diffuse/specular commands don't directly affect the vertex data. However, I've already tried adding a diffuse and specular component to the FVF and looping through every limb/every vertex and setting the diffuse value (mentioned above), but no luck.

I'll have another go at that and see whether I can rejig things. Perhaps the order of setting up the mesh and applying the shader effect matters. Cheers.

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 11th May 2012 20:22
There are problems reading diffuse with shaders in DBPro - but they mainly affect bumpmapping which needs tangents, etc. I vaguely recall Cash Curtis II posted a question along similar lines a few years back.

I believe there's also a problem if you have more than one set of UV data - as your example seems to do (used for the blend constants).

I'll try to dig out some relevant examples later today.

I usually end up removing the diffuse component from any models I use. There is a rather cumbersome workaround which worked for one user a while back - add two new sets of UV coords and store the vertex colour in those. That worked - but whether it works for boned models I don't know (I've had little success changing the fvf format of boned models - the animation seems to get screwed up).
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 11th May 2012 21:57 Edited at: 11th May 2012 21:58
Quote: "Lock model"

AFAIR there is several lock modes with different results on original object

«It's the Shader, shader me this, shader me that»
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 11th May 2012 22:39
That's certainly worth trying.
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 12th May 2012 01:27
I'm not sure whether this helps. It's an old, incomplete, demo of mine which may work. It doesn't fail the way I expected it to and I don't have time to check through it at the moment. Nevertheless it seems to show DIFFUSE working with bump mapping so may be helpful.
Fallout
23
Years of Service
User Offline
Joined: 1st Sep 2002
Location: Basingstoke, England
Posted: 14th May 2012 10:20
Awesome. Thanks for that Mr Gandalf. I checked through the code and it was basically doing the same thing. The only difference was the FVF format of 338, where as I'd set the FVF by adding together the HEX flags in the documentation. Using 338 worked a treat. The only weirdness that it has caused is, the original model has lost control of his arms ... .. he animates like a seal. All the other models that have been cloned from him are fine.

Weird, but diffuse/color problem solved. Cheers.

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 14th May 2012 12:54 Edited at: 14th May 2012 12:56
Quote: "he animates like a seal"


I had similar problems when I tried to change the FVF format of an animated boned model the other day.

Quote: "All the other models that have been cloned from him are fine."


That's interesting. Could you give more details?

1. Did you clone before or after changing the FVF format of the parent object?

2. Are you saying the cloned models had the new FVF format and animated correctly with the shader?

It would be interesting to see if I can replicate the symptoms. I was trying to go the other way, i.e. going from 338 to 274, and resorted to manually removing the diffuse colours from the X file.

Sounds like you've unearthed one of those very rare DBPro bugs that keep us on our toes.

Quote: "The only difference was the FVF format of 338, where as I'd set the FVF by adding together the HEX flags in the documentation."


They should be doing the same thing. Which flags did you use? What was your code exactly? Perhaps there was a typo in there or something?
Fallout
23
Years of Service
User Offline
Joined: 1st Sep 2002
Location: Basingstoke, England
Posted: 14th May 2012 14:43 Edited at: 14th May 2012 14:45
Let's see ... Order of events was:


So I cloned all the models before changing the FVF. Also I tried a load of different flag combinations, and none of them worked, but I used this method to add the flags together:

0x002 + 0x010 + 0x040 + 0x080 + 0x020

That was the last combination I tried that was commented out in my code (since replaced with 338! )

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 14th May 2012 15:13
Quote: "0x002 + 0x010 + 0x040 + 0x080 + 0x020"


That gives an FVF of 242 - and no UV coordinates. From the Help file those codes are

XYZ
NORMALS
DIFFUSE
SPECULAR
PSIZE

What on earth is PSIZE? I'm not sure about SPECULAR - haven't tried that yet.

Format 338 is

0x002 + 0x010 + 0x040 + 0x100

i.e. XYZ + NORMAL + DIFFUSE + TEX1
Fallout
23
Years of Service
User Offline
Joined: 1st Sep 2002
Location: Basingstoke, England
Posted: 14th May 2012 16:30 Edited at: 14th May 2012 16:37
At that point I was trying out any old random rubbish in an effort to get it to do something different, but without luck.

However, the help file specifies FVF_TEX0=0x000 - which I assume meant it had UV coordinates as default. Or do you think that is specifically saying "no textures coords". In which case, it's very misleading.

Either way, it never changed the model. I always had UV coords.

Edit: Ahh, now I look at it again and looking at the number sequences, I see you specify 1 value for the number of texture layers you want, as well as 1 value for the XYZ format, and then a few flags for psize,diffuse etc. That caught me out. I didn't notice the sequences and assumed they all referred to individual bits. A fairly confusing jumbled mass of values in there.

Just replaced 338 with "0x002 + 0x010 + 0x040 + 0x100" for my own readability/clarity and works as expected.

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 14th May 2012 16:49
Quote: "I see you specify 1 value for the number of texture layers you want. That caught me out. It appears you have to add some items together, but some are sequences that you just choose one value from, but they're all jumbled together in a nice confusing mass."


Yes - but those values are defined by DirectX.

Quote: "FVF_TEX0=0x000"


TEX0 is just the name DirectX uses for 0 UV coordinates, TEX1 for 1 set and so on up to TEX8 for 8 sets. I found it confusing at first as well and thouht TEX0 referred to texture stage 0.

Quote: "Either way, it never changed the model. I always had UV coords."


Interesting. Something else to look into.
Fallout
23
Years of Service
User Offline
Joined: 1st Sep 2002
Location: Basingstoke, England
Posted: 15th May 2012 12:37
Quote: "Interesting. Something else to look into."


Thinking about it, my shader was attempting to read the diffuse component and setting the overall object brightness accordingly. When I was receiving black through, my model was black, so if the UV coordinates had vanished, I wouldn't have known anyway! So it probably was zapping all the texture data correctly.

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 15th May 2012 12:54
Quote: "So it probably was zapping all the texture data correctly."


It probably was. I did a few tests as well and couldn't see any evidence of UV data being retained.
Fallout
23
Years of Service
User Offline
Joined: 1st Sep 2002
Location: Basingstoke, England
Posted: 15th May 2012 13:00
Btw, I can confirm that converting FVF on an animated model after a clone works. I've just been reorganising my code so I have a library I can clone all my objects from.

Doing this:
- Load base object to clone others from
- Set it's FVF
- Exclude it
- Clone an object from the base object

At this point, it is messed up and is in seal flipper mode.

Doing this instead resolves the problem:
- Load base object to clone from
- Exclude it
- Clone an object from the base object
- Set clone's FVF

Strange, but true! It must be something to do with how DBP organises the model data after a load, verses how it organises it after a clone. The clone code path must handle the bones/animation slightly differently, such that it survives the FVF command.

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 15th May 2012 13:21
I'll do some tests now and report back.
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 15th May 2012 13:45
Yes, I can confirm your workaround works for me too. Good find. I sense a bug report coming ...

Wish I'd known about it when I was working on this:

Shadow Mapping for bone animated models

It would have saved a lot of messing about with the object's structure.
Fallout
23
Years of Service
User Offline
Joined: 1st Sep 2002
Location: Basingstoke, England
Posted: 15th May 2012 14:50
I saw that thread. Nice work. I cracked that a few days ago using my own basic shadow mapping system, and had the same issues you had with running out of instructions and registers. I've made do with a simpler shader, so I've not packed in the content you have managed to, but it's good enough for my needs.

It's games that drive these discoveries I think. It's not until you actually NEED to do something that you start figuring out the work arounds.

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 15th May 2012 15:51
Very true.
david w
20
Years of Service
User Offline
Joined: 18th Dec 2005
Location: U.S.A. Michigan
Posted: 17th May 2012 00:36
@Fallout can you post a sample of the problem, with the shader you are using. I really would like to see this because I am not totally convinced that it is a bug. Thank you.
Fallout
23
Years of Service
User Offline
Joined: 1st Sep 2002
Location: Basingstoke, England
Posted: 17th May 2012 01:33
Which problem David? The vertex color or the clone/FVF with animated model issue?

david w
20
Years of Service
User Offline
Joined: 18th Dec 2005
Location: U.S.A. Michigan
Posted: 17th May 2012 02:42
The vertex color is what I was concerned about.
Fallout
23
Years of Service
User Offline
Joined: 1st Sep 2002
Location: Basingstoke, England
Posted: 17th May 2012 12:09
That problem was solved. One I changed the FVF to a correct value, I had a diffuse component and could access it in the shader.

Login to post a reply

Server time is: 2026-07-22 00:35:51
Your offset time is: 2026-07-22 00:35:51