Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Professional Discussion / Dark Shader 2 (or alternative) - Petition

Author
Message
Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 9th Apr 2013 23:02 Edited at: 14th Apr 2013 14:12
When working with a plan it can be quite annoying to receive requests that don't quite fit in; but here is a situation where I have to ask for something that might not be wanted by the community or the company, or who knows maybe lots of people want to buy a better shader tool. The best I can do is take a shot... If you disagree please state.

There is a lot going on at the moment with the next FPS Creator release and AGK; Dark Basic upgrades... hopefully. There is one product I would like to see improved or recreated is Dark Shader.

If not Dark Shader, then a third party tool could be created by one of the plugin stars; I'd be the first customer.

Dark Shader is really good:
Great at:
* Testing and tweaking shaders
* Testing lights
* Tweaking full screen effects
* Texturing and blending
* Compiling assembly code (at least sometimes it works in the engine)
* Crucial cube mapping and camera effect commands added to the engine

I'd say 90% of what is in Dark Shader at the moment is good, and shouldn't change; please keep all of the backdrop, camera, texture list and light tools intact. I am no fan of NVidia FX Composer; worst development tool I've used, and I've used them all!

What I would buy is Dark Shader 2 which would do some or all of the following

Function Text Editor
* Utilize a more complete text editor; for example something with a Select All (CTRL+A) feature and common right click menus with copy and paste
* The ability to change the size of the code font
* Intellisense, dropdown list of typical HLSL functions and semantics as you type
* An error report that does not get clipped by the preview window, or that provides scrollbars so you can see the whole error. Yes you can resize the window, but unless you have two monitors like me, the only way to see the whole error message is to undock the preview window.
* An element list (EG: a navigation list of functions and constants; click to navigate to the declaration)
* Ability to dock the text editor below or beside the project slider property panel; or to implement a floating property panel as an option.
* The ability to store snippets in the user interface
* The ability to insert texture samplers via a menu rather than having to search for needles in haystacks; list all texture sampler names in the menu.


Performance
Dark Shader is practically unusable when creating modern shaders that use bump mapping, per pixel lighting and reflection. The only way to get it stop crashing is to give the Preview.exe Above normal processing priority in the task manager. Even then it still crashes half the time. Making tweaks to code means having to restart Dark Shader sometimes.

* Optional donotgeneratedata parameter
It is true that even in the DarkBASIC engine, shaders can crash when a high level of complexity is used. The idea of using complex shaders is to stop having to create too many of them, each taking a great deal of time to compile during the loading call. Using the donotgeneratedata parameter in the engine helps prevent crashes 80% of the time on my machines; but this does not seem to be available in Dark Shader.

* Get rid of the 'preview window is not responding' message box. The inter-operation often requires the preview to spend 10 or 20 seconds to compile the shader or load a large texture; this message box is a bit irrelevant, in many cases the preview window is responding, it is busy loading.

* If there is anything else that can be done to improve shader performance in Dark Shader, or an alternative and the engine, please put this in the documentation. Yes dumbing down the HLSL code can improve the performance, using more vertex commands less pixel commands; but if there is anything else please let us know.

* The sliders take too long to load; I am sure the property sliders shouldn't take so long to get displayed. Is the program parsing the code in the shader file and adding each constant one by one, slider by slider? Perhaps a progress bar or some other indicator would be more tidy, at the moment I see sliders popping up one by one down the window and it looks like the program has crashed, when it hasn't.


Work flow

* Please get rid of, limit, or provide an option to get rid of the question 'would you like to pack the media with this project'; this appears every time you simply want to save the project.

* The not enough textures message doesn't need to be visible all the time; sometimes not applying all texture stages is required, reading the message all day can be frustrating and it obscures screenshots.

* Technique select; an option to pick the shader technique you want to use.

* Ability to drag and drop textures and shaders into the project or asset list from Windows; and if necessary provide a file explorer instead of having to go through a series of menus and folder browses to insert a texture from here and a shader from there. For that matter, any texture/shader you load should be added to the import list or some kind of recent asset list. (I guess due to the instability of complex HLSL in DarkBASIC in general, one would not have to keep reloading shaders all of the time if the program did not crash so much.)

* Limb textures, limb shaders. Please provide a means to apply shaders to limbs, not just objects; the same is true of textures. Having the ability to work with an object and not limbs is also desired. Having the ability to work with multiple objects does not seem necessary from my perspective, but someone may advise otherwise.

* Because not all objects are square or spherical in structure, a walk / fly navigation option is a must, you simply cannot navigate a scene at the moment. The arrow key and mouse drag movement is only good for simple objects.

Quote: "Green Gandalf

annoying issue: when I try to move the camera/object in the preview window the camera/object moves too fast so it's very hard to zoom in to an area of interest. That ought to be adjustable in some way. "
"


- Additionally, when an object is huge the camera moves to slow even when pressing the control key.
* Please provide option to adjust camera range; sometimes objects are small, and will get clipped out of sight when you try to zoom close to see the detail; such details just become hidden.

* In line with the above request, a way to organize the textures, stages, shaders and properties would need to be proposed. I would recommend either a node based flow chart style setup linking element to element via nodes; or treat everything in relation with some kind of a treeview control with textures, shaders and properties with a per limb or object root.

* Some kind of texture filter menu would be helpful, sometimes a texture needs to be brightened or blurred; going off into the Gimp is time consuming; having a texture filter menu with Brightness, contrast, blur, normal mapping, desaturation and saturation controls would be a life saver. If you wanted to increase the price of the product a bit more, some painting, cloning, copying and pasting texture tools would also be handy.

* Light-mapping; it would be really spectacular if light-mapping could be performed in Dark Shader; since shaders are being used to blend light maps. The feature should give an option to adjust the UV layer as well as all parameters available in Dark Lights. If this feature is implemented, I'd like to see multi-object shading implemented also, since maps contain multiple objects.

Support
* Dark Shader 2, or the alternative should document all of its commands; just like with every other plugin, it would be handy to know what all the functions do.

Importing and Exporting

* The export method shows a folder dialog when you use export All; this dialog never remembers the current folder, it sends you to the desktop; you always have to fish out the folder you want to export too which could be prevented by using the current working directory, or at least the project path. It would not surprise me if it is a Windows FolderDialog issue, but it is pretty bad.

* Please export the settings as-well as the shader. Even better; provide ability to save settings as presets that can be used later, accessed via a manu. State the preset name in a comment or string constant in the exported file

* Just like FX composer; provide XML project format so we can parse the shader settings and materials using modern file parsing procedures.

* Prove me wrong, but I think the full screen effect export code is a little inflexible and a bit misleading. It hides a single object, renders the effect camera, then shows the object and renders the scene. This would never get done in a real game, you wouldn't want to hide then show 1000s of objects 60 or so times per second just to render an effect; far better to use the Set Object Mask command (I think). Or, since the Advance2D plugin is fast at drawing images, forget using camera zero to render 3D, just output the effect camera to an image and paste the image before the full sync. No need to render two cameras and a quad plane. For that matter, set camera zero's backdrop to the effect camera image. Maybe this could be an option, but ultimately a hide and show of all objects per sync seems a little bit much.

Other issues posted by fellow users
TheComet:

Quote: "
This code works fine in any other HLSL compiler, but DarkShader won't compile it:

// note: maxIterations is a value passed into the shader by the host program
for( int i = 0; i != maxIterations; i++ )
{
// do stuff with "i"
}
"


Green Gandalf:
Quote: "DS's great advantage for me is its simplicity, it's just the bugs that are annoying.

My other pet hate, mentioned by Chris, is the inability to change technique. At the moment you have to comment out, or move, the earlier techniques to test a later technique. DS checks the syntax of all techniques - it's just the displayed technique that can't be changed easily. "


Summary

This is all I can think of, others may disagree or might add to the list; if someone could build a shader program for Dark Basic or update Dark Shader it would be good for the future because we should all be using shaders now, it will only make the engine look better if the whole process is made easier. I'd personally pay £40-£60 for such a tool. For now I will continue to dream and use Notepad++ to write my shaders and reload Dark Shader every 5 minutes for the next year.

Note
* Limiting the texture limit might be redundant due to Shader Model 3 supporting more than 8 textures; but this a wild guess at the moment.

mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 10th Apr 2013 00:21
Why don't you just write your own preview application and use one of the many free HLSL editors? Lazy?

«Just because you’re unique, doesn’t mean you’re useful»
Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 10th Apr 2013 01:58 Edited at: 10th Apr 2013 02:00
Come on dude. That's what I am doing; I've got no choice but to use an alternative editor and make my own preview window; how else am I supposed to tweak shaders that won't load in Dark Shader?

I just don't see much sense in reinventing the wheel when Dark Shader is there, in a position to become a more up-to-date shading and texturing application.

If I am the only person in the world who wants to pay money for a decent shader editor instead of using the free ones then I would have to build my own one; I'm just checking if someone else already has this in mind before I go ahead with doing my own thing.

If someone out there is thinking of creating a tool to make some money, here's at least one idea to add to your list. I'm simply stating what would be a good product. I work with shaders daily, have used NVidia FX Composer like I stated above, I don't like it, I prefer Dark Shader; which is a package I paid money for but don't work half the time; I'm willing to pay for something better than what is free; frankly nothing in life is free in the long run. In anycase what use are free HLSL editors that are not compatible with DarkBASIC in anycase?

mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 10th Apr 2013 11:51 Edited at: 10th Apr 2013 12:02
HLSL editor is just an editor, like DBP IDE, with syntax highlight, bells and whistles. Don't you say that you can't edit code outside of IDE? If you have very complex set of shaders, then you could use your actual applicaiton engine with a test level. That's what all developers do. You'd better ask an Ultimate Shader Pack 2.

P.S. I worked with Composer, Dark shader, Mental Mill... and I don't like them all. I'm using generic editor with basic functions like highlight.

«Just because you’re unique, doesn’t mean you’re useful»
Mobiius
Valued Member
23
Years of Service
User Offline
Joined: 27th Feb 2003
Location: The Cold North
Posted: 10th Apr 2013 13:59
I believe notepad++ has hlsl syntax highlighting.

I live for video games! (And beers, and football, and cars!)
See what I live for here: [url]http:\\www.TeamDefiant.co.uk[/url]
TheComet
18
Years of Service
User Offline
Joined: 18th Oct 2007
Location: I`m under ur bridge eating ur goatz.
Posted: 10th Apr 2013 16:52
I agree with most of the points you mentioned, Chris. What program do you use currently?

I use RenderMonkey to prototype my DBP shaders (when I'm writing them from scratch), and then I port them into DarkShader.

Can you add something to that list of yours? This code works fine in any other HLSL compiler, but DarkShader won't compile it:



TheComet

Taumatawhakatangihangakoauauotamateapokaiwhenuakitanatahu is a hill
Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch is a town
Chargoggagoggmanchauggagoggchaubunagungamaugg is a lake
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 10th Apr 2013 18:10 Edited at: 10th Apr 2013 18:12
I agree with several of Chris Tate's suggestions for Dark Shader - especially the ones about performance and the very basic editor which I find particularly annoying.

My pet hate is the "The preview window has stopped something or other do you wish to restart it?" message. Neither of the two options "Yes" or "No" seem to do anything useful and I usually have to close down Dark Shader and restart it. For that reason I have to remember to save my working code frequently. The problem seems to be particularly acute when I'm running DBPro at the same time and am switching between them. I hadn't noticed that bumpmapping was implicated in that - I thought it was quite general. I never encountered the problem on Windows XP 32bit, only on Vista 64bit (haven't tried it on W7 64bit yet).

Dark Shader is still my program of choice for developing shaders - programs such as FX Composer and RenderMonkey have both become far too complex and cumbersome for my taste. I would still use FX Composer 1.8 if it were compatible with modern operating systems - the "improved" FX Composer 2 is a nightmare to use by comparison. The feature I particularly miss from FXC1.8 is the ability to create cube and volume maps.

I mainly use DS for simple syntax checking, etc. I use a relevant DBPro program for checking the shader's functionality which is often awkward in DS. Not sure I'd want to change that - DS's great advantage for me is its simplicity, it's just the bugs that are annoying.

My other pet hate, mentioned by Chris, is the inability to change technique. At the moment you have to comment out, or move, the earlier techniques to test a later technique. DS checks the syntax of all techniques - it's just the displayed technique that can't be changed easily.
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 10th Apr 2013 19:33
Anybody tried Mental Mill?

«Just because you’re unique, doesn’t mean you’re useful»
Duke E
17
Years of Service
User Offline
Joined: 10th Mar 2009
Location:
Posted: 11th Apr 2013 10:58 Edited at: 11th Apr 2013 12:54
Pretty easy to write your own stub in DBPro to call in an external editor as a debugger.

This is a couple functions i used in a project.
Shows the use of some "undocumented" dark shader functions.



Edit: The function assumed you made a on error checklist prior to calling it, i removed that case.

Regards
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 11th Apr 2013 12:54
@Duke E

Thanks for that - I didn't know about those functions. How did you find out about them?

Do you know whether that method correctly handles warning messages from the shader compiler? I believe Dark Shader fails to compile shaders when there are warning messages but DBPro carries on regardless.
Duke E
17
Years of Service
User Offline
Joined: 10th Mar 2009
Location:
Posted: 11th Apr 2013 13:00 Edited at: 11th Apr 2013 13:04
@GG

I think i found a commandlist from some dark shader release (written by you i think) on the forum.
(Edit: No it was probably James H or Plystire that Reshacked the keywords and posted them)
Then i experimented alittle.

It was a while since i wrote that, don't know about the warnings =)

I remember it gives the same quirky error line numbers when you forget a semicolon or wrong enclosing parentheses.

Regards
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 11th Apr 2013 13:16
Yeah, wasn't me. I think I posted this:



Do you still have the full list of these other commands? I love experimenting.
Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 11th Apr 2013 13:21 Edited at: 11th Apr 2013 13:36
I'll add your points to the original post Comet and Green Gandalf.

Quote: "Don't you say that you can't edit code outside of IDE?"


Quote: "I believe notepad++ has hlsl syntax highlighting."


I think you missed my point there:

Quote: "I will continue to dream and use Notepad++ to write my shaders and reload Dark Shader every 5 minutes for the next year."


So that's what I am currently doing; I also use my engine to preview more than one object together.

So that answers the questions about my current work flow.

Quote: "If you have very complex set of shaders, then you could use your actual applicaiton engine with a test level."


I am, just to re-clarify:

Quote: "I've got no choice but to use an alternative editor and make my own preview window"


The preview window is my engine.

I am just not in the mood to put aside my project and spend months creating my own full featured editor; if I had the Dark Shader source code then woo hooh! Although it is probably in C++ which I do not have the time to get into yet.

I'll admit, this is the first time applying shaders to 100s of different limbs in a scene; far too tedious to be dealing with text files and jumping from one app to another. Yes loading about 10 or 15 shaders is maximum otherwise you'd have to wait minutes for them to compile; not all of them can be per-compiled by the dark shader assembly function and work; but using textures as parameters you can get unlimited individualized settings.

It is a bit like creating a 500 page Micrsoft Word document editing each page individually in a separate file in Notepad; we want to edit the whole document in one place with formatting applied so you can see the result.

I do have my own level editor, my engine can have shaders reloaded automatically without restarting the engine; but I wasn't planning on creating a shading editor, I was planning on creating a game, that's why I bought Dark Shader.

I know where you are coming from Handy, you're thinking like me a few months ago; but it has just hit me that every limb needs special attention side by side next to the other limbs, in the scene with light-mapping. Doing what you suggest is too tedious for this level of perfectionist level design; doing what you say would be like using MS paint to create textures; can be done, but too tedious.

This reminds me, Dark Shader does not allow you to preview multi-UV mapped objects; for example objects with light-mapping applied (unless you override the UV reference in the Vertex Shader)

Quote: "
Dark Shader is still my program of choice for developing shaders - programs such as FX Composer and RenderMonkey have both become far too complex and cumbersome for my taste."


I like Dark Shader's design for that reason.

Quote: "Pretty easy to write your own stub in DBPro to call in an external editor as a debugger.

This is a couple functions i used in a project."


Thanks for that, this is why I stated that the commands should be documented; however, I can load the shaders in my engine; it is Dark Shader that can't load the shaders.

Dark Shader needs what my engine has; the ability to complex load shaders and apply them various numerous limbs with a selectable technique. I really want to make a game, not a shader editor.

If not enough people feel like having a new Dark Shader tool and nobody feels like taking the project on board to create it and sell it; then I will create one in the engine, or as an open source project either in DBPRO or VB.NET. We'll see how it goes, maybe in a month or so someone will propose to make the update for Dark Shader, maybe not.

Duke E
17
Years of Service
User Offline
Joined: 10th Mar 2009
Location:
Posted: 11th Apr 2013 19:04
The commands are in the ShaderData.ini file, Plystire commented on it in the shaders thread 2009.

This one is pretty interesting:
a# = Get Effect Parameter Float Value("seconds")

I used it to compare the timing used in a shader with the DBPro clock, works great.
Suppose it could be used to extract complex and taxing calculation results from the shader to the DBPro code.

Regards
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 12th Apr 2013 02:27
@Chris Tate

Quote: "Yes loading about 10 or 15 shaders is maximum otherwise you'd have to wait minutes for them to compile"


Really? What's eating up all that time? [Actually I'm not very sure what you mean here. ]

@Duke E

Quote: "The commands are in the ShaderData.ini file, Plystire commented on it in the shaders thread 2009."


Thanks for clarifying that. I didn't think to look in that file.
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 12th Apr 2013 10:54
I think long load because shaders are not precompiled. Can DBP load compiled shader?

«Just because you’re unique, doesn’t mean you’re useful»
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 13th Apr 2013 01:29
Long load times are often caused by forgetting to turn off tangent and binormal calulations when they aren't needed by the shader.
EVOLVED
23
Years of Service
User Offline
Joined: 9th Feb 2003
Location: unknown
Posted: 13th Apr 2013 10:11 Edited at: 13th Apr 2013 10:12
Am I the only one who uses notepad only?

this thread has just hit a nail on the head for me when it comes to shaders and dbp. When having to load up the same shader over and over (like in my ALS each light needs a unique shader) each effect is compiled into assembly (as far as I know) and can end up dragging out loading times by quite a bit, I find that this is definitely a problem for me. I really wish dbp had a "clone effect" command that instantly copies an already existing compiled shader.

The alternative way is to maybe load an already compiled shader by using "get object effect" and "save effect" commands. But form what I can tell this just saves the compile HLSL functions as asm and needs some modding to add textures + shader constants etc.
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 13th Apr 2013 11:05
@Green Gandalf

How to do that?

@EVOLVED

Hi! How do you compile your shaders? DBP supports asm?

«Just because you’re unique, doesn’t mean you’re useful»
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 13th Apr 2013 13:09
Quote: "How to do that?"


Like this:

Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 13th Apr 2013 13:26
Quote: " "Yes loading about 10 or 15 shaders is maximum otherwise you'd have to wait minutes for them to compile"

Really? What's eating up all that time? [Actually I'm not very sure what you mean here. ]"


Literally, each of my shaders are taking 3-5 five seconds to load, I have to load about 3 versions of each shader to make use of unique settings; there is no clone effect command as Evolved stated; so my overall shader loading time is an average of 30 seconds and I am just getting started. Perhaps this would vary from machine to machine.

Loads of people complain about shader loading times all over the internet from engine to engine; so it is not just DBPRO; but I'm not sure why clone effect wasn't added.

I think compiling ASM with Dark shader speeds up the loading time; but not all of them work and Dark shader will not always load.

mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 13th Apr 2013 19:05
@Green Gandalf
Oh, this!

Also do you know how to compile shaders for DBP?

«Just because you’re unique, doesn’t mean you’re useful»
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 13th Apr 2013 19:20
Not specifically. I guess DBP just uses a call to the appropriate DirectX function which then probably uses the FXC.exe compiler. That is just a guess though.

This seems to be a relevant function (taken from the MS DX9 SDK docs):



That should make sense to C++ programmers.
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 13th Apr 2013 19:27
@Chris Tate

Quote: "Literally, each of my shaders are taking 3-5 five seconds to load"


Is that caused by loading the shader or by applying it to the objects concerned?
Zero G Scott
18
Years of Service
User Offline
Joined: 14th Dec 2007
Location: California, USA, Earth, Sol Sys
Posted: 13th Apr 2013 19:32
Lee is in the middle of his Reloaded project and has several blog hints stating that DBP will get some love in some form. I'm curious if anyone has brought these issues to his attention, this might be a good time in his development cycle to ask for increased shader support. In fact all of you should spam his e-mail till he caves!

This is very interesting stuff to me because I'm slowing getting up to speed with DBP and at some point in the near future I'll be diving into shaders myself, for now though I'm just too incompetent with shaders to know what to ask Lee for.
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 13th Apr 2013 21:40
Quote: "I never encountered the problem on Windows XP 32bit, only on Vista 64bit (haven't tried it on W7 64bit yet)."


Update.

Have now installed DS on my W7 laptop and have no obvious problems using it - yet. However, I get the familiar DBPro stopped working error message when I close it. Not exactly a showstopper but still annoying.

I installed the latest version from my order history. I then noticed something called the Dark Shader Runtime V1 and installed that as well. As far as I can tell that did nothing except waste my time. Anyone know what it is?

Haven't used it enough yet to see whether I get the same Preview Window stopped working problems - but it did remind me of another annoying issue: when I try to move the camera/object in the preview window the camera/object moves too fast so it's very hard to zoom in to an area of interest. That ought to be adjustable in some way. Anyone else any thoughts on this?
Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 14th Apr 2013 14:05 Edited at: 14th Apr 2013 14:13
Quote: "Is that caused by loading the shader or by applying it to the objects concerned? "


Loading process. I know this because my engine times each process separately. Applying doesn't seem to slow the program down and I don't think it should because once assembled into the GPU it is all plain sailing. The GPU is comfortable, but DBP is struggling; it just seems that something in the delivery from hard-drive via CPU to the GPU is causing the slow down; likely the compilation.

Sometimes DBPRO crashes when applying a particular shader of mine however; but this is rear and will be investigated later.

According to NVidia, based on the shader model version, the compile process is also optimizing dynamic flow controls and calculations; ever noticed a shader compile more quickly when the pixel return statement is placed before many calculations? I have

For this reason I am thinking the donotgeneratedata parameter stops what ever DBP is doing to get in the way of this optimization process. I am thinking DBP was built with shader model 1 in mind. It is also possible to prevent shader optimization by using flow control attributes or lower shader models (EG: Shader Models 2 and below)

Quote: "
annoying issue: when I try to move the camera/object in the preview window the camera/object moves too fast so it's very hard to zoom in to an area of interest. That ought to be adjustable in some way. Anyone else any thoughts on this? "


Will add that to the main post; it is along the lines of my complaint about navigation. I think pressing shift slows down camera movement, but this isn't always effective.

I've been researching shaders this weekend and am seeing that you are more likely to hit problems using model 2 or 3 as apposed to 1.1, due to the way DBP handles things. But for what I need, most of my work requirest version 2 or 3.

FX Composer has a shader debugger which I am learning to use now; hopefully this will provide more useful information.

Quote: " this might be a good time in his development cycle to ask for increased shader support. In fact all of you should spam his e-mail till he caves!"


The greater the demand for shader support and flexibility; the more feedback people post here, the more likely this will happen.

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 14th Apr 2013 16:01 Edited at: 14th Apr 2013 16:05
Quote: "For this reason I am thinking the donotgeneratedata parameter stops what ever DBP is doing to get in the way of this optimization process."


Interesting observations. In one application I'm aware of the simple expedient of setting that flag and calculating the tangents in the shader speeded loading times enormously (and had a negligible effect on in-game performance - but I thought the delay was at the set object effect stage rather than load effect. I'd have to go back to the original demo to be sure though.

Quote: "am seeing that you are more likely to hit problems using model 2 or 3 as apposed to 1.1"


Not surprising since 2 and 3 allow many more instructions and features. SM1 is very limited and ought to be forgotten.

Quote: "FX Composer has a shader debugger which I am learning to use now"


If you have the MS DX SDK (if not I highly recommend it - it's free after all) then you can use the shader compiler FXC.exe directly.

Quote: "ever noticed a shader compile more quickly when the pixel return statement is placed before many calculations? I have"


Well, of course it would - the compiler ignores everything after the return statement. Or have I misunderstood your point?

Quote: "I think pressing shift slows down camera movement, but this isn't always effective."


So it does! Thanks! [Note to self: perhaps it's time for me to read the DS Help files properly? After all, it's never too late to learn. ]

Edit Quick update on an earlier point:

Quote: "Haven't used it enough yet to see whether I get the same Preview Window stopped working problems"


Just had DS open with a moderately complex shader, done some browsing, compiled a DBPro shader demo plus a few other things and no problems with the Preview Window on this W7 machine so far. Perhaps there's something wrong with my Vista installation?
Mobiius
Valued Member
23
Years of Service
User Offline
Joined: 27th Feb 2003
Location: The Cold North
Posted: 14th Apr 2013 16:45
Quote: "Perhaps there's something wrong with my Vista installation?"

Perhaps it's just vista. lol

I live for video games! (And beers, and football, and cars!)
See what I live for here: [url]http:\\www.TeamDefiant.co.uk[/url]
Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 14th Apr 2013 17:00 Edited at: 14th Apr 2013 17:11
Quote: "Well, of course it would - the compiler ignores everything after the return statement. Or have I misunderstood your point?"


I'm mentioning the compilation in terms of loading the shader rather than actually running code after the return statement.

Other compilers including DBP won't do that; they compile code after the return/exitfunction/end statements; here a 10000 line program with an end statement at line one takes just as long as any 10000 line program; with shaders a return statement literal cuts off the compilation process of the pixel shader right there and then. Only syntax is checked. The thing with SM3 and below is I doubt you can return inside a branch statement.

What would clarify my point better is what happens when you do not reference a variable that is assigned a complex calculation; the compilation process speeds up; the GPU knows not to bother optimizing a variable that does not get returned. At least I've spotted that when changing the return variables around. See what happens during compilation when you take out a bump or cubemap variable out of the return calculation.

Furthermore the newer shader models also skip optimizing and compiling calculations in flow controls that can't be reached. Unless you override with attributes.

Quote: "If you have the MS DX SDK (if not I highly recommend it - it's free after all) then you can use the shader compiler FXC.exe directly."


Good, thanks; I'd rather not use FX composer for anything.

It has just crossed my mind that the Dark Shader crashes might not even have as much to do with the shaders as it seems, because on the preview window the shader is applied long before the editor responds to mouse clicks, right before it crashes. Might be something going on during shader parsing that is causing irritation.

I've seen Dark Shader crash using its own normal mapping shader, or loading a large 2048x2048 texture, so it is not always really complex shaders that cause problems.

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 14th Apr 2013 18:32
Quote: "I've seen Dark Shader crash using its own normal mapping shader, or loading a large 2048x2048 texture, so it is not always really complex shaders that cause problems"


I've only ever had this issue (so far) on Vista. Haven't seen this particular issue on XP or W7.

Quote: "the GPU knows not to bother optimizing a variable that does not get returned"


I doubt the GPU is involved in that at all.

Quote: "What would clarify my point better is what happens when you do not reference a variable that is assigned a complex calculation; the compilation process speeds up; the GPU knows not to bother optimizing a variable that does not get returned. At least I've spotted that when changing the return variables around. See what happens during compilation when you take out a bump or cubemap variable out of the return calculation"


The DX9 shader compiler has a number of switches and, if I recall correctly, one of them turns optimising on or off so it's possible DBPro handles that switch inconsistently. I also don't see how the DBPro DoNotGenerateExtraData flag should affect load times since it's nothing to do with shader compilation - it should only be relevant when it has a model with geometry to work on. Perhaps there are two different issues here and DBPro has confused them somehow?
Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 14th Apr 2013 19:52 Edited at: 14th Apr 2013 19:55
Quote: ""the GPU knows not to bother optimizing a variable that does not get returned"

I doubt the GPU is involved in that at all."


Sure, the compilation is done by the CPU, I guess what I meant by GPU is the shader loading process as a whole.

Quote: "
I also don't see how the DBPro DoNotGenerateExtraData flag should affect load times since it's nothing to do with shader compilation - it should only be relevant when it has a model with geometry to work on. Perhaps there are two different issues here and DBPro has confused them somehow? "


OK, so the DoNotGenerateExtraData parameter in the load effect command has no affect on loading times. So then that would mean there is no way to speed up the process from within DBPRO; improvement has to be made outside of DBPRO. The only way to speed it up is to pre-compile the assembly or dumb down the shader. And we need a clone shader command to make use of various techniques and settings without having to reload.

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 14th Apr 2013 20:48 Edited at: 14th Apr 2013 20:49
It would certainly be interesting to know what is causing the slowdown you're referring to. Perhaps I should revisit my earlier demos and see exactly where the time was spent in those? It's possible there are two quite different issues here, i.e. long load times for certain shaders and long setting times for certain shader/object combinations. I'll try to find time to set up a simple demo to look into this.

But either way, I agree it is tedious to have to load umpteen copies of the same shader just because you want each object to use different values for the shader constants. In some cases you can achieve this by doing things like storing the constants you need in separate textures and applying those to the objects. Slightly messy but works. Some sort of clone effect feature would be a good idea.

[Edit: corrected stupid typo. ]
Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 14th Apr 2013 22:20
I've not yet figured out what exactly is slowing down my shader loading times; I just know which shaders take longest to load; the ones with the most branch statements containing the most calculations; but that's all I know so far. I will be investigate this further to pinpoint the exact cause.

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 14th Apr 2013 22:42
Interesting. I rarely use much branching, perhaps that's why my experience has been somewhat different from yours? Still strikes me as surprising though since shaders don't usually contain much code. Perhaps there's quite an overhead involved in optimising?
Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 16th Apr 2013 02:32 Edited at: 16th Apr 2013 02:49
OK, so FXC.exe compiles my shaders properly using the legacy parameter in model 2.0 (which I think covers model 3).

Shaders that took 3 seconds to load, now take an average of 17 ms. And get applied in about 2 ms. So that was very helpful.

Now; that leaves one problem plus a new one I have noticed with certain shaders. First problem is not being able to load FXC compiled shaders into Dark Shader.

I'm hoping this is simply because I've used binary mode and there exists a text mode; I've yet to get that far looking into the parameter documentation.

Another problem now is a particular shader crashes my engine when applied to more than one object. So I will now spend some time finding out what's troubling DBP.

I've tried FX composers debugging tool which has a neat panel that allows it to compare the performance of your shader on various NVidia cards:

It takes ages to run the analysis on the shaders but it will hopefully be worth it in the long run. I also notice my uncompiled shaders load into FX composer with no delay compared to Dark Shader and DBP; however the FX composer cube-map handling is horrid; brings the program down almost to a halt.

Question

Why is there a shader error when you compile, let's say, the Dark Shader Scrolling Texture Shader; from within Dark Shader's compiler? It says error X2001 : LINE 29: shader version expected. I'm not sure why something compiled by the D3D function would throw up such an error. I've loaded the outputted ASM file using Dark Shader.

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 16th Apr 2013 03:13
Quote: "Shaders that took 3 seconds to load, now take an average of 17 ms."


How are you doing that in DBPro? Sounds useful.

Are you simply loading the asm file instead of the usual fx HLSL file? Or something else? Looks like an opportunity that I've missed.

Quote: "Why is there a shader error when you compile, let's say, the Dark Shader Scrolling Texture Shader; from within Dark Shader's compiler? It says error X2001 : LINE 29: shader version expected. I'm not sure why something compiled by the D3D function would throw up such an error. I've loaded the outputted ASM file using Dark Shader."


Good question. I've just tried that and get the same thing. It seems to be something to do with the preshader code. Perhaps DBPro/Dark Shader doesn't support that. Have you tried compiling the same shader using FXC.exe with the preshader option turned off (I think there's a switch for that)? You might be able to load that OK. I'll look at this in more detail sometime tomorrow as it's late now.
Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 16th Apr 2013 13:37
I'll look into your recommendation a little later.

Compiling my shader in FXC then loading the result into DBPRO reduces loading time. (So far) Only a few times it takes more than a second, but most of the time less than half a second for the most complex shaders.

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 16th Apr 2013 13:47
Quote: "Compiling my shader in FXC then loading the result into DBPRO reduces loading time"


What format is the file you are loading into DBPro and how are you doing it?
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 16th Apr 2013 14:53 Edited at: 16th Apr 2013 16:24
Quote: "I'll look into your recommendation a little later."


I've done some experimenting with some of my shaders and the problem does seem to be the preshader. One of my shaders calculated the reciprocal of one of the constants in either the VS or PS. That is obviously unnecessary - why not pass the reciprocal itself as the constant? Well the purpose of the preshader is to optimise that sort of thing. When I amended the code so that DBPro supplied the reciprocal then the preshader code was no longer needed and Dark Shader then loaded the asm code painlessly. Looks like Dark Shader is inconsistent in its handling of preshader code.

If the preshader is the problem in your case then you might try amending your code so that the preshader is unnecessary.

One thing baffles me though: how do you load the compiled asm code into DBPro? As far as I can see the texture samplers and shader constants are lost. Are you editing the asm version of the shader so it contains this information?

Edit Ignore above final paragraph. I've edited the DS asm output to include the necessary constant tables and samplers and everything is fine.

Thanks for raising this possibility, I wasn't aware that loading asm code was actually so easy and had such a performance impact (I haven't checked this point myself but I can see you're probably right about it). I'll do a few more tests and produce a simple demo for others to see the effects and what needs to be done. This is definitely a feature to include in any new version.
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 16th Apr 2013 16:29
Yay for demo, GG!

«Just because you’re unique, doesn’t mean you’re useful»
«If you contributed to the reason for locking, you may now find yourself on moderation, or in extreme cases in the grave»
Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 16th Apr 2013 17:20
Glad to hear the news.

I've still not gotten round to doing what you said today; busy.

For the record; here is how I compiled the shader using FXC.exe which is by no means a method I've fully tested yet; but so far it works.

The parameters I used in the June 2010 DirectX SDK FXC compiler are:

Quote: "fxc /T fx_2_0 /LD /Fo ShaderOutputFilename.fxo Shader.fx"


The /LD parameter switches to legacy compiler mode; which basically uses the Direct X9 compiler and bypasses all of the Direct X10 optimizations.

The output file is in binary format and works in DBP; with textures, techniques and constants and everything as normal.

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 16th Apr 2013 17:44 Edited at: 16th Apr 2013 17:45
Quote: "The output file is in binary format and works in DBP"


So is that one further compilation step, i.e. asm to obj or something else?


I've just tested a simple wood shader and tested each combination of asm vs HLSL and "extra data" versus "no extra data" and obtained the following results using 2000 copies of the shader applied to 2000 DBPro spheres:

HLSL + extra data : 6559 +391
asm + extra data : 5534 + 391
HLSL + no extra data : 6597 + 303
asm + no extra data : 5547 + 303

(times are ms for total load time plus total apply time respectively).

So it looks as if there are indeed two ways of reducing effect loading times - but neither is particularly dramatic in this instance (or on this machine?).

Quote: "Yay for demo, GG!"


Complete demo for above attached.
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 16th Apr 2013 17:48 Edited at: 16th Apr 2013 17:49
Quote: "The output file is in binary format and works in DBP; with textures, techniques and constants and everything as normal."


Addendum: Can we easily call FXC in DBPro? If so then we have the basis for a simple shader editor written in DBPro.
Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 16th Apr 2013 22:08
Yep

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 16th Apr 2013 22:11
Quote: "Yep"


Now who's going to do it?
Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 16th Apr 2013 22:24
I'm going to create an editor. It might not be called Dark Shader 2, but it will do much of what is in the list plus some other aspects of creativity. I'm going to release it with my own integrated suite so it it is unlikely to be free.

Still, there's quite an opportunity for an open source editor or for someone to create Dark Shader 2.

mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 16th Apr 2013 22:24
Quote: "The /LD parameter switches to legacy compiler mode; which basically uses the Direct X9 compiler and bypasses all of the Direct X10 optimizations."

If not use LD, what happens? Crash? Will it still work on DX9?

«Just because you’re unique, doesn’t mean you’re useful»
«If you contributed to the reason for locking, you may now find yourself on moderation, or in extreme cases in the grave»
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 16th Apr 2013 22:30
There's one way to find out ...
Mobiius
Valued Member
23
Years of Service
User Offline
Joined: 27th Feb 2003
Location: The Cold North
Posted: 17th Apr 2013 18:45 Edited at: 17th Apr 2013 18:45
Quote: "There's one way to find out ..."




Sorry, couldn't resist.

I live for video games! (And beers, and football, and cars!)
See what I live for here: [url]http:\\www.TeamDefiant.co.uk[/url]

Login to post a reply

Server time is: 2026-06-11 12:13:20
Your offset time is: 2026-06-11 12:13:20