Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Professional Discussion / Get Image - With Transparent Background Working Demo

Author
Message
Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 28th Apr 2011 03:24 Edited at: 14th May 2013 12:30
I have developed and posted here an in-game method for capturing 3D objects to an in-game image with a transparent background. That is to say, no backdrop, no background, perfect for use as a 2D overlay in-game.




What this is:
Turn an on screen object into an image. Use the image anywhere. Preserves all alpha information, not just simple on/off transparency.

What this is not:
It is not something you want to generate in real-time as animation. This process can take about 100ms (depends on size and complexity), where as a single frame at 60fps is only 16ms. So its too slow to do on each frame. You do this when loading or probably during live play a couple times wouldn't be noticeable.




Why do this at all:
If you have ever tried to mix 2D with 3D you'll notice all 2D is layered over 3D. Sometimes you need the 3D in front. The solution as been to make the 2D stuff 3D. This is a pain, and the minute the aspect ratio of the screen changes (different shape monitor), everything is out of place. This method makes the 3D object 2D, which is easier to work with, and gives more capability.




Other Methods:
At first glance, you could just screen grab an object on a black backdrop and use a colorkey, then paste it with the transparency flag on.



There's a big problem if the object is black, or requires subtle alpha levels.


Why not Set Camera To Image, and have the devs add a flag for transparent backdrop?



Awesome, but they'd still have to deal with the "Image Locked" error when moving that image to a memblock (even after the camera was deleted).


How does it work:
It is possible to set up a camera (off screen) that captures the object on both a black and white backdrop.



Then you can compare those images. Any Pixel that goes from pure black to pure white is transparent. Any Pixel that stays the same is opaque. You can record the pixels in a new image.



For 95% of the time this is all we need. When the object has transparency, this completely fails. We see bits of the backdrop, or other wierdness.



It is possible to identify partially transparent pixels since they differ from opaque pixels mentioned above, where they are merely slightly different in the two images. That's very easy to detect.



It turns out there's a bunch of wondrous math where you can compare a pixel from both images and determine what the correct RGBA values are. There's some loss of information when the object was drawn to screen, so this method is sometimes off by 1 or 2 values, not avoidable, and not noticeable.

Standalone Demo. Standard Version. v1.3 - No Files needed, New Optimizations


Image Kit Powered Version. v1.3 - Requires Image Kit v2, and Shader

Image Kit Shader - RenderTransparent.fx


[/b]Update: Latest Versions Added[/b]
Standard Version - Requires no files or plugins. Full functionality.
Image Kit Version - Requires Image Kit v2. More than 6x Faster!
http://forum.thegamecreators.com/?m=forum_view&t=176270&b=8

Any errors, oversights, or improvements are welcome.

WLGfx
11
Years of Service
User Offline
Joined: 1st Nov 2007
Location: NW United Kingdom
Posted: 28th Apr 2011 04:15
Wow... Excellent stuff. As soon as I get the chance I'll have a look at transferring this to GDK C++ and see what the difference is in speed.

Nice one...

Warning! May contain Nuts!
baxslash
Bronze Codemaster
11
Years of Service
User Offline
Joined: 26th Dec 2006
Location: Duffield
Posted: 28th Apr 2011 10:34
This looks really useful! I might use this for 'greenscreening' in my cut-scene editor. Thanks for sharing!!

Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 28th Apr 2011 13:04
Update: 16x Performance Boost on Semi-Transparent Pixels.

Oh and thanks!

Duffer
15
Years of Service
User Offline
Joined: 9th Feb 2003
Location: chair
Posted: 29th Apr 2011 06:54 Edited at: 29th Apr 2011 07:01
@ Mage,

Would there be any gains in terms of speed if you also use SvenB's Image Kit plugin?

[edit]

Have run the code btw - this excellent, v fast - would still be interested in speed comparison if utilising SvenB's Image Kit plugin....

a long time dabbler with DBC and DBPro with no actual talent but lots of enthusiasm...
Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 30th Apr 2011 00:38 Edited at: 9th May 2011 01:56
@ Duffer,

SvenB's Image Kit plugin is a massive speed loss. The implementation was simpler but the demo jumped 750% time taken to generate the image. He won't be happy if he reads this.

Also his plugin is not a work around for the Image System Locked error, that requires images generated by a camera to be copied to new images before they can be modified.


Standard Implementation - 60ms
Image Kit 2 Implementation - 450ms
Image Kit Version Here -


I didn't know about his plugin before you mentioned it, and had hoped it could help. He has some amazing features in that plugin though.


Edit: (Much Later) Removed failed Image Kit version. Sven B has proposed an ImageKit shader version instead.
Edit: (Much Later Still) Shader Version Implemented.

Duffer
15
Years of Service
User Offline
Joined: 9th Feb 2003
Location: chair
Posted: 30th Apr 2011 00:45 Edited at: 30th Apr 2011 00:45
@ Mage,

That is truly a shocker...

Basically, means community even more grateful for your code! Thank you.

a long time dabbler with DBC and DBPro with no actual talent but lots of enthusiasm...
Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 30th Apr 2011 00:50 Edited at: 30th Apr 2011 06:25
I modified the standard version posted at the top and the Image Kit version to display time taken. Test it out for yourself. I would love for someone to say I made a mistake and that Image Kit is really lightning fast. Doesn't look like it.

Sven B
13
Years of Service
User Offline
Joined: 5th Jan 2005
Location: Belgium
Posted: 30th Apr 2011 13:22 Edited at: 30th Apr 2011 13:25
Quote: "SvenB's Image Kit plugin is a massive speed loss. The implementation was simpler but the demo jumped 750% time taken to generate the image. He won't be happy if he reads this."


I can imagine that. IK dot and IK point() are faster if you only need (to change) one point of the image. It still needs to lock the image to retrieve the data (much like converting to a memblock). But it can limit the size it needs to lock in order to retrieve or change the pixel. In other words, by using the command in a for-next loop, you kind of have the equivalent of using make image from memblock (although a little faster) every loop.
If you need to apply a custom function to a large part of an image, then memblocks are always the way to go.
If you need to apply a function that Image Kit supports, then Image Kit will most likely be faster.

On a side note: Image Kit can be used to paste images on render targets (like images used with render to image) without getting the "locked image" error. If you need to apply a custom function, you can use a effect file and paste the two images together using a pixel shader.

Cheers!
Sven B

Sven B
13
Years of Service
User Offline
Joined: 5th Jan 2005
Location: Belgium
Posted: 30th Apr 2011 16:25
Hi Mage,

If you don't mind, I have applied your technique to Image Kit using render targets and a pixel shader.

.DBA:


RenderTransparent.fx


It can run in real time.

Cheers!
Sven B

Duffer
15
Years of Service
User Offline
Joined: 9th Feb 2003
Location: chair
Posted: 30th Apr 2011 17:02 Edited at: 30th Apr 2011 17:04
@ SvenB and Mage - between you you've created something v useful - thanks again.

I suppose it would make a difference (with version utilising shader) if the 3d object / 3d background was textured as well? what if that background was black as opposed to white? what if it was textured with various colours?

a long time dabbler with DBC and DBPro with no actual talent but lots of enthusiasm...
Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 1st May 2011 00:33 Edited at: 1st May 2011 01:01
Quote: "I can imagine that. IK dot and IK point() are faster if you only need (to change) one point of the image. It still needs to lock the image to retrieve the data (much like converting to a memblock)."


Yeah I figured this was happening.



Quote: "If you don't mind, I have applied your technique to Image Kit using render targets and a pixel shader."


Hahahah yes, I was hoping this would be the case. Now we're all the richer for it. Thanks.

I did run your code, and it crashed.
Program hands on this line: "ik load effect "RenderTransparent.fx", 1"

So then I updated to 7.6. Still crashed. Downloaded ImageKit v2 like 3 days ago. No meaningful error message, DB Pro just stops working it says. Hoping you know whats not working, so I can see this in action. Thanks.

Sven B
13
Years of Service
User Offline
Joined: 5th Jan 2005
Location: Belgium
Posted: 1st May 2011 01:33
Quote: "So then I updated to 7.6. Still crashed. Downloaded ImageKit v2 like 3 days ago. No meaningful error message, DB Pro just stops working it says. Hoping you know whats not working, so I can see this in action. Thanks."


Definitely needs some attention. It seems that it crashes when the .fx file couldn't be compiled as well. I've never had that problem before, so it kind of fell out of the sky today. I will look into it.
On a side note: have you actually created the actual RenderTransparent.fx file? The crash happens when the file doesn't exist or if the fx file couldn't be compiled. I've attached the file if you don't want to bother creating it .

Cheers!
Sven B

Attachments

Login to view attachments
Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 1st May 2011 02:02 Edited at: 1st May 2011 02:04
Awesome. Turns out windows lied and threw a hidden .txt extension onto the file. Fixed it.

Looks like it might be 3x as fast, on a single shot (20-25ms compared to figures mentioned above). Although when rigged for animation, the frames become much faster as the setup code can be skipped.

Thanks, very nice.

Sven B
13
Years of Service
User Offline
Joined: 5th Jan 2005
Location: Belgium
Posted: 1st May 2011 15:29
Quote: "Although when rigged for animation, the frames become much faster as the setup code can be skipped."


Glad you got it working. Thanks for pointing out that bug. I have updated the plugin and it should now be fixed.

Quote: "I suppose it would make a difference (with version utilising shader) if the 3d object / 3d background was textured as well? what if that background was black as opposed to white? what if it was textured with various colours?"


Hi Duffer,

The pixel shader can do exactly the same as Mage's code, but I just implemented it a little differently so you could see it was used in real time.
All you need is the black and white image. It doesn't matter how you get them, you just need to get them and pass them to the pixel shader .

Cheers!
Sven B

Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 2nd May 2011 20:10
@svenb
The demo you made has camera 0 as one if the inputs, is there any way to change this to another number so camera 0 isn't used except to display the resulting image?

Also can Image Kit crop an image like this that is System Locked? I don't mean resize since proportions need to be maintained. There's a few slow methods that eat at the performance gains. Making the camera simply be non standard sizes causes huge lag.

Thanks

Sven B
13
Years of Service
User Offline
Joined: 5th Jan 2005
Location: Belgium
Posted: 2nd May 2011 21:18
Hi Mage,

For the effect you basically need to have two images. The type (normal or a render target) of those images don't matter. The code that matters for the effect is:


Where CamBlackImg and CamWhiteImg can be any type of image.

Of course, you can use any variation of the ik paste image command. Check the help files for more information.

You can for example:
Use camera 1 to render the black image, then copy that image. Then you can use camera 1 again to render the white image, and paste that where you want using the effect. Don't forget to set the black texture using IK Set Effect Texture at the setup.

Quote: "Also can Image Kit crop an image like this that is System Locked? I don't mean resize since proportions need to be maintained. There's a few slow methods that eat at the performance gains. Making the camera simply be non standard sizes causes huge lag."


IK Copy Image can crop images, either normal images or render targets (the ones that produce the locked error). On a side note: changing the dimensions of a render target used with set camera to image will disable the rendering to that image.

Cheers!
Sven B

Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 4th May 2011 05:43 Edited at: 9th May 2011 01:48
@SvenB

Again thanks for taking time to lend a hand. I'm almost at a point where this can be wrapped up.

I do have a few issues still lingering:

1. The shader plugin approach really suits this type of problem. However I just cant seem to get that fancy shader output into an actual standard dbpro image. Pasting to screen appears easy. Managing 2 source images and a shader for every rendered image is kind of rough. I'm sure it's something trivial.

2. I am under the suspicion that the shader may only be 1bit alpha, instead of 8bit. Below I've posted an example of mine where the object is merely ghosted. I've also posted a modified version using Image Kit. In both images the outer regions are transparent, but with Image Kit the block appears solid. The original version without Image Kit has 8bit transparency showing a translucent cube.

(Also theres an issue with the image kit version also where previous images aren't clearing properly. But each image is still 1bit alpha.)

I read shaders at about a 1st grade level (probably less), but I'm not seeing a search algorithm that computes the Alpha level, and resulting RGB for partial transparencies. *** Rethought the math on that. It looks like the shader has the stuff, but theres a problem somewhere, maybe still in the shader maybe not.

Image Kit - Ghosted Object Example


Standard Version - Ghosted Object Example


Thanks again for any time and assistance you can provide.

Sven B
13
Years of Service
User Offline
Joined: 5th Jan 2005
Location: Belgium
Posted: 4th May 2011 22:53 Edited at: 4th May 2011 22:54
Hi Mage,

Quote: "1. The shader plugin approach really suits this type of problem. However I just cant seem to get that fancy shader output into an actual standard dbpro image. Pasting to screen appears easy. Managing 2 source images and a shader for every rendered image is kind of rough. I'm sure it's something trivial."


You can use IK Paste Image On Image to render to a render target image (normal images won't work, because pixel shaders can't handle rendering to normal images).

Quote: "2. I am under the suspicion that the shader may only be 1bit alpha, instead of 8bit. Below I've posted an example of mine where the object is merely ghosted. I've also posted a modified version using Image Kit. In both images the outer regions are transparent, but with Image Kit the block appears solid. The original version without Image Kit has 8bit transparency showing a translucent cube."


You're right, the pixel shader had some incorrect code. It should say this:


It didn't give the correct alpha value when the color was (semi-)transparent.
I used these formula's:

(C = Color ranging from 0.0 - 1.0, a = alpha)
Result of black image: C * a + 0 * (1 - a) = C*a
Result of white image: C * a + 1 * (1 - a) = C*a + 1 - a

We know the results, let's call them Cw (Color white) and Cb (Color black):
C*a = Cb
C*a + 1 - a = Cw
And we want to find C and a.

Subtracting both equations:
a - 1 = Cb - Cw => a = 1 + Cb - Cw (see line 28 of the pixel shader)
Now we have a, so from the first equation:
C*a = Cb => C = Cb / a

Of course we need to make sure that a != 0, if it is, then we choose a default value of (0.0, 0.0, 0.0, 0.0) which is black transparent.

So the problem was that the shader returned:
ColorBlack = (r, g, b, 1.0) -> Returned (r / a, g / a, b / a, 1.0 / a)
while it should have returned:
Returned (r / a, g / a, b / a, a)

My apologies for the mistake.

Cheers!
Sven B

Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 5th May 2011 07:01 Edited at: 5th May 2011 07:02
@Sven B

Thanks, I was guessing that it was something like that. Looks like the new shader is perfect.

IK Paste Image On Image If you check the example I posted just above with Image Kit, every instance of this operation seems to stack relentlessly. It's as if this operation remembers everything and cannot be cleared.

Deleting The Image, New Image Number, Reset Image Kit, Delete and Reload the Effect, nothing helps!

Thanks for the help.

Sven B
13
Years of Service
User Offline
Joined: 5th Jan 2005
Location: Belgium
Posted: 5th May 2011 18:15
Hi Mage,

The reason is probably because of the blend mode. It will paste to the image using transparency, but in your case you want to overwrite the previous image instead of a normal rendering.
Use IK Set Blend Mode to set the blend mode. Check the help files, and use BLEND_FORCE or BLEND_FORCENOALPHA.

Cheers!
Sven B

Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 9th May 2011 01:37 Edited at: 9th May 2011 01:45
Thanks, it's finished.
I've added the latest version at the top of the thread.


Included:

A standard version that doesn't use any plugins. I added new optimizations.

An Image Kit version, with the shader, that offers a 600% speed increase.

I wrapped everything up into a single function call so it can be copy + pasted.

baxslash
Bronze Codemaster
11
Years of Service
User Offline
Joined: 26th Dec 2006
Location: Duffield
Posted: 9th May 2011 15:32
Excellent work Mage, this will be very useful code!

RedFlames
11
Years of Service
User Offline
Joined: 25th Aug 2007
Location: Germania
Posted: 11th May 2011 16:16 Edited at: 11th May 2011 16:18
Hey guys, that is some very useful code there.

Actually I used some of the code and Sven B's shader to create a workaround to a bug that occurs when you try to use d3d_Funcs on RenderTargets. This has already be mentioned by someone on other threads like the d3d_Func's thread, so I thought it'd be worth sharing with the community... But I'm posting this here, before creating a new thread.

I'll attach all the needed files here for now. It works really good, and I am already using it for my GUI system (yeah one of those dreaded things ). So thanks Mage and Sven B
I mentioned you and everything inside the files.

Anyways, either we could direct people here, or someone (me?) creates a new thread for this, if that is okay with both of you. (After all you crated this, and I'm using it!)

Attachments

Login to view attachments
baxslash
Bronze Codemaster
11
Years of Service
User Offline
Joined: 26th Dec 2006
Location: Duffield
Posted: 11th May 2011 16:32
Quote: "Anyways, either we could direct people here, or someone (me?) creates a new thread for this, if that is okay with both of you. (After all you crated this, and I'm using it!) "

I would suggest that since you found a workaround to the d3d bug yourself you should start a new thread for it and credit these guys there. Just my opinion but I think it'll get more exposure that way.

Rais
15
Years of Service
User Offline
Joined: 7th Jul 2003
Location:
Posted: 11th May 2011 19:43
Hi Mage,
By loading the shader before the main loop and removing the IK resets etc from the snapshot function im getting a HUGE speed increase (6 or 7ms down to 1!).
Is there any reason why the shader was being loaded and deleted each time?
Duffer
15
Years of Service
User Offline
Joined: 9th Feb 2003
Location: chair
Posted: 11th May 2011 21:57
@ Redflames,

Agree with Baxslash - post it in Work in Progress as a project with all files etc - and don't forget to work in Rais' observation too!

Post the link here too!

a long time dabbler with DBC and DBPro with no actual talent but lots of enthusiasm...
Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 11th May 2011 23:56 Edited at: 11th May 2011 23:59
Quote: "By loading the shader before the main loop and removing the IK resets etc from the snapshot function im getting a HUGE speed increase (6 or 7ms down to 1!).
Is there any reason why the shader was being loaded and deleted each time? "


I completely Agree. I noticed this before also. I decided this was a tradeoff.

1. The reason I decided to have IK Reset in the function is because I didn't want any previous use of IK to interfere with the operation of the function. It's OK to remove it for this demo, but you might need it or something like it when you actually attempt to use this in a project, as a variety of settings may interfere.

2. I agree the shader can be loaded at the beginning of the program, instead of created+deleted on each function call. I decided including it in the function was the best way because it's the most compatible. I wanted everything wrapped up in one neat function call. It does it's job with as little tweaking needed as possible. I can't tell you how many times I've come across brilliant examples on these forums (especially shader demos) that completely and utterly fail to work for some seemingly impossible reason, because the example was in a 100 pieces, and there were 100 things that could go wrong when copying or using the example elsewhere.

My goal for this was to make this code as simple to use as possible. One function call. Just drop it into your program. I also wanted to remove any special case situations that I could, like requiring a bunch of stuff be added at the beginning or different sections of the program.

Also, SvenB posted an example of using his shader to enable animation, which requires the shader and other bits be added and managed throughout the entire program. I felt this was outside of the scope of the example. It also requires you to build your program a specific way, which was something I wanted to avoid.

So if you feel comfortable loading the shader outside of the function call then go for it. It's what I'll be doing. However, the most important goal for this demo is that it works for people. So simplicity got priority.

Login to post a reply

Server time is: 2018-12-18 19:12:16
Your offset time is: 2018-12-18 19:12:16