Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

AppGameKit Classic Chat / [BUG] PlayVideo significantly changing color and black levels of playing video

Author
Message
Westa
12
Years of Service
User Offline
Joined: 28th Oct 2011
Location:
Posted: 28th Aug 2017 08:52
Link
Been doing a number of tests on the PlayVideo and PlayVideoToImage capabilities and have discovered a major issue with appearance of the videos when played back in AGK.

1. The dynamic range of playing videos is being treated as RGB-Limited - meaning that videos with blacks at RGB(0,0,0) are effectively being displayed at level RGB(16,16,16) and videos with whites at RGB(255,255,255) are being shown at level RGB(235,235,235) - the net result - videos with no true blacks and crushed whites - making for flat image.

This issue in itself is something that can be hack to sort out for now with a "levels" sprite shader and PlayVideoToImage - but its not an ideal solution.

2. The more serious issue is that the color of the image is also being corrupted in other ways ... If you look at the attached images with waveform and vector scopes - you can see that there appears to be a HUE rotation of around -9degrees - which again could be hacked by a hue based sprite shader - though not ideal or efficient.

3. There also appears to be a major hue shift most noticeably in the cyan and yellow levels - at gamma lift of around 1.22 seems to assist this at least visually if not accurately via the scopes - but is not a workable solution - again a gamma sprite shader does help somewhat

However, none of the above are a working or acceptable solution - as the color space changes do not appear to be linear - i suspect its a REC601 to RBG mapping issue - which needs to be sorted in the core.

This bug represents a serious deal breaker for a project we are currently working on and needs to be resolved

Westa

Attachments

Login to view attachments
Back to top
Profile PM Email
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 30th Aug 2017 22:01
Link
There was a mistake in the YUV to RGB conversion that swapped two of the scaling values. Does the attached version produce a correct output? The 16 to 235 issue is something that I saw mentioned when researching YUV to RGB conversion but it wasn't clear if this should be done, most conversion routines don't use this reduced space.
This is the conversion we are using.

Attachments

Login to view attachments
Back to top
Profile PM Email
Westa
12
Years of Service
User Offline
Joined: 28th Oct 2011
Location:
Posted: 31st Aug 2017 14:20
Link
Hi Paul,

In theory the color space looks almost right but the black levels are still off - I suspect the Y scaling is still incorrect - currently black is still at (16,16,16) and white at (235,235,235).
which as a result also puts all the other colors out by a similar amount.

The colors across the top of the color bar on the reference image are actually mathematically related and represent 75% saturation of each color value.
blackpoint(255,255,255) - AppGameKit black showing (235,235,235)
greypoint (191,191,191) - AppGameKit greypoint showing (180,180,180)
yellow( 191,191,0) - AppGameKit yellow showing (180,180,13)
blue(0,0,191) - AppGameKit blue showing ( 16,16,183 ) etc etc

I can optically get it close by using LEVELS in photoshop - using 16 - 235 in the levels input section.
On everything I am reading the Y needs - 16 subtracted from "most" codecs - but it is a factor of the encoded format.
And just to mess things up even more - this conversation is ok for ( BT601 ) standard definition codecs - But REC 709 which is the high definition codec needs a slightly different matrix as well.

Westa

Attachments

Login to view attachments
Back to top
Profile PM Email
Westa
12
Years of Service
User Offline
Joined: 28th Oct 2011
Location:
Posted: 31st Aug 2017 14:49
Link
This is two of the more useful documentation links - are the references to the other conversation - is the signal YUV or YCbCr

https://en.wikipedia.org/wiki/YUV#Conversion_to.2Ffrom_RGB ->>

https://en.wikipedia.org/wiki/YCbCr
Back to top
Profile PM Email
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 31st Aug 2017 14:51
Link
Does the attached version work better?

Attachments

Login to view attachments
Back to top
Profile PM Email
Westa
12
Years of Service
User Offline
Joined: 28th Oct 2011
Location:
Posted: 31st Aug 2017 16:48
Link
Ummm - well - only kinda.

yes 0 Black is now 0 , 0 , 0
White is getting close at 254,254,254

But the rest of colors are still moving all over the place.

Yellow - 190 191 34 - should be 191,191 , 0
Blue - 0 0 156 - should be 0 0 191
etc etc etc - compare the two charts in the attached image - bottom is reference - top is what AppGameKit is generating currently.

There is a relationship between Y and Cr and Cb - that cannot be messed with - as a result you cant just change the Y starting point value without also fixing the matrix that corresponds to it.
See the links in the previous post from


Attachments

Login to view attachments
Back to top
Profile PM Email
Westa
12
Years of Service
User Offline
Joined: 28th Oct 2011
Location:
Posted: 31st Aug 2017 16:50
Link
Ok - this is where I got to eventually - this set of matrix values are industry best practice:

http://www.equasys.de/colorconversion.html

---------------------

The matrix you are using is:

"full-range YCbCr color into RGB" - which assumes Y is 0-255 - suitable for motion jpeg style codecs and some other color spaces.


BUT h.264 which is what mp4 is based on expects YCbCr 16-235-240 color data format (Y values in the range [16:235] and Cb/Cr values in the range [16:240]).


So for MP4 and others too I guess - what is likely needed is:

"RGB and YCbCr color for SD TV" - where ITU-R BT.601 is the color space

and

"RGB and YCbCr color for HD TV" - where ITU-R BT.709,is the color space"


Mathworks also has a similar take on things.

https://au.mathworks.com/matlabcentral/fileexchange/36417-yuv-files-reading-and-converting?focused=5229495&tab=function



---------------------



Back to top
Profile PM Email
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 31st Aug 2017 23:13
Link
The last attachment was using the BT.601 conversion, but I made a mistake in the conversion to integer math, the attached version should be correct, it probably won't be perfect due to using integer math but it should be closer. I'm not sure if we can detect when to use BT.709.

Attachments

Login to view attachments
Back to top
Profile PM Email
Westa
12
Years of Service
User Offline
Joined: 28th Oct 2011
Location:
Posted: 1st Sep 2017 06:59
Link
Hi Paul,

Yes its pretty close to the mark in most cases - which yes I can see do look like rounding errors - though most decoders do tend to always round down anyway - I guess the only question would be that WHITE (255,255,255) is not reporting quite true (254,254,254) which is likely a round down instead of a round up.

Now having said all this - I was doing a bit of reading regarding media foundation - and looking at their thumbnail sample - and it seems that media foundation should be able to do thisCbCr / YUV transform without you needing to use your own matrix transforms. In their example - they just tell MF that they only want want a "MFVideoFormat_RGB32" stream and MF works out what it needs to do to convert the incoming video to that format regardless of the input file format and codec.

Though ironically - MF may already be doing a codec conversion to a default YUV derivative known as NV12 - which may mitigate the need for worrying about BT601 / BT706 issues.

As I don't know what your code looks like - hard to understand how you are using MF - but this approach does seem to be more robust in terms of file input formats that having to create your own matrix transforms. When I run my color bars test video through their sample - the color transform results are exactly what I would have expected.

As a side note - it also has built in de-interlacing capabilities that can be turned on when needed - which would make for another nice ("hint") option

Cheers

Wes


Back to top
Profile PM Email
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 1st Sep 2017 13:56
Link
Initially I used MFVideoFormat_RGB32 since, as you say, it is the perfect output format for us, but on some videos it caused an exception in KernelBase.dll, particularly on videos encoded by Handbrake. I spent some time trying to work out why but never found an answer, luckily I discovered along the way that MFVideoFormat_YUY2 works so that's what we're currently using. I've attached the video that caused problems, if it works with the sample code you have can you send it to me so I can see if I can get it working? I've also attached a another version that might fix some rounding errors, let me know if it is any better.

Attachments

Login to view attachments
Back to top
Profile PM Email
Westa
12
Years of Service
User Offline
Joined: 28th Oct 2011
Location:
Posted: 1st Sep 2017 22:19
Link
HI Paul,

Ah ok - so yes then you already have an intermediate transform happening - its a pretty same bet it will likely always be BT601 I guess.

White levels and black levels look good - rounding on all other colours looks acceptable too.

Ok now I understand - ironically it was that video that started my questions - the colors of the sky looked way off and the blacks were all wrong

I've attached the Media Foundation sample for VideoThumbnail.zip that had no issues loading thumbnail frames at least from the video you attached. Its not streaming frame after frame of realtime playback - but maybe the way they setup the formmating may give a clue or two.

The sample comes from the Windows 7 Samples - Multimedia - Media Foundation library.

Cheers

Westa

Attachments

Login to view attachments
Back to top
Profile PM Email
Westa
12
Years of Service
User Offline
Joined: 28th Oct 2011
Location:
Posted: 1st Sep 2017 22:37
Link
Oh as an aside - in much of what I read - it was often mentioned that MFVideoFormat_ARGB32 (as opposed to MFVideoFormat_RGB32) was the baseline format for RGB transforms - did it also throw the kernel error ?

Westa
Back to top
Profile PM Email
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 5th Sep 2017 17:36
Link
Unfortunately MFVideoFormat_ARGB32 also throws an exception in KernelBase.dll. Thanks for the sample code, I'll have to go through it line by line to check for any differences, and since we have a reasonably working version now it's not a high priority.
Back to top
Profile PM Email
Westa
12
Years of Service
User Offline
Joined: 28th Oct 2011
Location:
Posted: 7th Sep 2017 00:44
Link
Hmmm frustrating hey - makes sense for now - just seems crazy to be having to do two time consuming matrix transforms every frame - once internally to YUV2 then again to RGB

Westa
Back to top
Profile PM Email
Back to top

Login to post a reply

Server time is: 2024-09-30 07:28:17
Your offset time is: 2024-09-30 07:28:17