Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

AppGameKit Classic Chat / Seamless / faster image loading

Author
Message
Santman
13
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 3rd Jun 2017 01:12 Edited at: 3rd Jun 2017 02:34
Link
Ok, so I find myself with a "need for speed" when it comes to loading large images and want to get as close to background streaming as possible, so time for some testing. In the interest of sharing (and maybe getting help) I decided to share the results.

Background: two images of dimensions 1920*1080 are being used, one a PNG (5.2mb) and the other a JPEG (1.9mb). The testing rig is my laptop which is an 8GB I3, with nvidia graphics and an SSD. So lets get to it.

Test 1: load the PNG 50 times :- time to complete, 7 seconds.

Test 2: load the JPEG 50 times :- time to complete, 7 seconds

WAAAAY to slow. So lets look at memblocks.....

Test 3: lets load the PNG as a memblock 50 times:- time to complete, 0.364 seconds.

Test 4: lets load the JPEG as a memblock 50 times:- time to complete, 0.076 seconds (0.000196 seconds PER IMAGE - perfect for a single frame).

Ok, now we're getting somewhere. So a 1920*1080 JPEG can be loaded as a memblock 50 times in less than a tenth of a second. So how do we make it usable? Let's load an image, make a memblock from it, then make 50 images from the memblock:

Test 5: load the PNG, convert it to a memblock, then make 50 images from the memblock:- time to complete, 0.369 seconds (0.000739 seconds PER IMAGE).

Test 6: load the JPEG, convert it to a memblock, then make 50 images from the memblock:- time to complete, 0.328 seconds (0.000653 seconds PER IMAGE).

So, what we see here is it's MANY MAGNITUDES quicker to load a memblock, then create an image from it than it is to load an image. But how do we load a memblock with IMAGE DATA than we can use? And do it in the background? The solution I am looking into is actually simple - dump the image as raw data (convert to a memblock, save it off as bytes), then load this back in "byte by byte" in the background then convert the memblock back to an image. Lets see how we get on - update to follow.

EIDT: I realized afterwards that the PNG was 4K, so I ran again with the same JPEG as a PNG.
Back to top
Profile PM
Santman
13
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 3rd Jun 2017 02:27
Link
Ok, so here's the first update.

Converting the JPG to a data file, took around 0.536 seconds, created an 8mb file. So it's big - let's see how long it takes to read it back in...
.
.
.
So reading the data back in took 0.566 seconds
Converting to an image took 0.0076 seconds

So how does this compare?

Loading a single JPG directly took 0.157 seconds - however during this load time the system is frozen. Converting the memblock to an image took 0.0076 seconds - so this is WHOPPING 20.6 times faster. So why would we use this? Maths.

If the image is 1920x1080 pixels, then it is 2,073,600 (+12) bytes. Loading as an image means we have to load this all in one go.....however loading from the data file means we can load it in "chunks" in between frames for as long as we want.

So in practice, my need is to load 9-10 images in the background of my game moving - though this could in theory be a constant load. So, it takes half a second, roughly, to load an image from data - so I will need 5 seconds to "stream" in the data in total.

One frame (at 60 fps) is equal to around 0.016 seconds...so what I know is that converting the memblocks to an image takes around 47% of a frame. My game renders (during a jump) at less than half a frame....so this render and conversion should be invisible to the user.

My jump sequence takes 8 seconds currently to transition between backgrounds (though I planned up to 10 to be acceptable while movement is on screen) so I will in theory have room to spare. But.....lets put that to a real world (ish) test....

In a demo (giving each "chunk" 0.008 seconds to load, or half a frame) it took 66 "half" frames to load, around 0.55 seconds.

On the face of it, seems to be a success, and (albeit very slowly) the code is "streaming" graphics in the back of rendering, and as long as the render takes less than half a frame, should run at 60fps. Of course, we could simply set it up to run at 60fps, or 30 or whatever, then simply give the loading code whatever is left of a frame to load...so if the frame took 25% of a second, we'd load images 33% faster.

Tomorrow I shall try this in the real world and see how Star Captain fares - and of course also when NOT loading from an SSD.

If it works as expected, I shall share the code once it's tidied up - in the meantime any idea's would be appreciated.
Back to top
Profile PM
PartTimeCoder
AGK Tool Maker
9
Years of Service
User Offline
Joined: 9th Mar 2015
Location: London UK
Posted: 3rd Jun 2017 08:33
Link
I have tried outsourcing loading large images to a plugin using a threaded function and although it works I get the odd spratical crash that I can not explain when calling the thread multiple times, I might have more luck with a mutex thread but its a windows only solution, a native tier 1 solution would be much better, I'm curious to see how you went about it.
Back to top
Profile PM
Santman
13
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 3rd Jun 2017 12:13
Link
I'm interested in your solution, it sounds like it has the potential to be much faster. This should suit my needs though, and comes with the handy bonus that my images will be in files that most people can't access. I spent some time thinking about It, and actually the streaming off the disk is the bottleneck for now, so on a decent system I wonder if some form of simple compression might make it quicker, if I can keep decompression fast.

Will play with it later, make a workable demo of the screen doing something while the images load in.
Back to top
Profile PM
Santman
13
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 3rd Jun 2017 15:14
Link
Ok, so I moved to my desktop which has a bit more oomf and run a few tests.

So first up loading the JPEG 50 times took 6.68 seconds (0.013365 seconds per image)

Next I loaded the same image from a memblock file 50 times, converted to an image - this took 0.709 seconds (0.001418 per image)

So that's 50 1920*1080 images loaded and ready to be used in 0.7 seconds, as opposed to loading 50 JPG images - faster by a factor of 9.4 times

The code for this is super easy, but comes in two stages:

Stage 1 - convert your images to memblock files:


Once these are saved, we can then load them using the following code:


So this should speed up general image loading times, at the expense of a larger size (for PC/MAC games this is more than a worthwhile trade off). But what about a mobile? I have an S8, but the player doesn;t seem to work (not ready for android 7??).

Anyone else care to try?
Back to top
Profile PM
Santman
13
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 3rd Jun 2017 16:13 Edited at: 3rd Jun 2017 16:19
Link
Ok, so I built a small tool that packs images, attached.

I tried this on a real world image from Star Captain (a PNG 4096*2048, 8mb), which packed into a file 32mb big.

The file size is a pain, however this is where the benefit is:
the PNG takes 0.262734 seconds to load one (50 takes 11.030 seconds)
the packed file takes 0.054911 seconds to load one image including coverting (50 takes 2.6596 seconds)

So that's a saving of 480% on loading a 4K image...which I think is sweet.

These pack files can also be "streamed" instead of loaded - that's the next test to come. Again, my desktop has an SSD so it would be cool to see results from a normal HDD or phone.

About the tool - this is very simple - open the project and dump any images you want into the "media\images" path and then run it, they will all convert. When it;s done, they will be in the write folder. To use the pack files, instead of using



you would now just do



EDIT:
I tried this on another drive, a regular 7200rpm drive. The results were as follows:

Loading the PNG 50 times took 10.4678 seconds (0.222224 per image)
Loading the pack file 50 times took 2.6076 seconds (0.056457 per image)

So the SSD makes little to no difference for me.

Attachments

Login to view attachments
Back to top
Profile PM
Santman
13
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 3rd Jun 2017 18:32
Link
Ok, so the idea works surprisingly well.

The video attached shows a demo of a simple bit of code. It loads a sprite and animates it at 60fps. When I click the left mouse button, the game continually loads two background images alternatively, over and over.

The first mode just uses the load image command, and freezes everything in the process of teh iamges loading - the code eventually levels out at 6fps.

The second mode shows the streaming - where both images load in the background and the frame rate stays almost perfectly at 60fps. This simple code gives the streaming code only what is left of the frame to maintain 60fps, however this is easily customised.

Seems to work quite well....

Back to top
Profile PM
Santman
13
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 3rd Jun 2017 18:56
Link
Ok, final update.

This video shows the same as above, only this time it has three stages:

Run 1 - this shows AppGameKit standard image loading looping through, hitting around 6fps.

Run 2 - this shows loading the same images but from memblocks, hitting around 30-40 fps.

Run 3 - this is the same images but streamed.



So, two new methods of loading have been explored here - one perfect for loading images quickly when the screen isn;t important, and another method that is slower but can be used to load images in the background while the screen is largely unaffected. Either method could quickly and easily be adapted to sound as well.

I would say this is a fairly successful exploration of image loading, hopefully someone will find it useful.

If anyone wants the demo of the streaming in action let me know.
Back to top
Profile PM
Santman
13
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 3rd Jun 2017 19:43
Link
Ok so one other thing has occurred to me.....it should actually be easy enough to stream out multiple memblocks as one single file. If we know the order and file sizes for images, we could read the file back in by setting the start position...so all images could be safely hidden in one massive file, unreadable to anyone else.

Just a thought - has anyone done this yet?
Back to top
Profile PM
nz0
AGK Developer
17
Years of Service
User Offline
Joined: 13th Jun 2007
Location: Cheshire,UK
Posted: 3rd Jun 2017 23:04
Link
I've made a loader which can load all media files from a single encrypted, compressed data file. I *really* need to tidy it up for release as it would be very useful.
It comes with a bunch of overloaded functions to do special loading (although not for speed) which includes subimage txt file handling, streaming media, shaders etc.
Of course, this is more centered on protecting the media than speedy loading (not speedy at all!)

A few other things to think about for speed:

PNG depletion (JPG as well, but tinyPNG is the best I found..)
Integer (vs byte) reading when moving data around (just modulo the last few bytes if not divisible by 4)
Someone (I forget who - sorry!) posted a nice idea about loading generated data statements directly into the compile. Not sure if this is totally relevant, but still related and interesting

Overall, a nice function there. I don't know if I could benefit from loading tons of on-the-fly images, as I tend to load everything up-front.
Maybe (if I ever get round to finishing it up) I could incorporate your loader into my media manager.

Unfortunately, I only develop for windows these days, but my loader will still work cross platform. If I was going to do multi-threaded loaders I'd do a plugin.

On Windows at least, I would think it possible with a dirty way to get mem-pointers to AppGameKit memory space and use an external plugin to push data into that memory space on multiple threads if you want to go nuts on this

My Robotron remake
My I,Robot remake WIP
My Defender remake
Back to top
Profile PM Email Website
Santman
13
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 3rd Jun 2017 23:55
Link
Sounds like an interesting plug in.

I said byte, but actually it reads everything as integers. Still wondering about some compression, though actually the fact it's raw image data isinthink what makes it so fast....so tier 1 compression would likely slow it down.

I wish I could do the plug ins, but that's way beyond me. I did get something similar with db pro before, using a plugin someone else wrote. It allowed you to run a program passing a variable, identify a free memory location, and had peek/poke and deek/doke commands, as well as memblocks creation direct from memory. I wrote a program that started a second passing a free memory location, that the two programs used to communicate. The second program could load over 1gb of data a minute in the background, passing the location back to the main program. Gave about a 5% performance hit if I recall.

Those were the days. Lol.
Back to top
Profile PM
nz0
AGK Developer
17
Years of Service
User Offline
Joined: 13th Jun 2007
Location: Cheshire,UK
Posted: 4th Jun 2017 00:15
Link
Well, if we can work something out that is viable,I can do the plugin.
I am curious about the need to get so much image data loaded on the fly though.
My Robotron remake
My I,Robot remake WIP
My Defender remake
Back to top
Profile PM Email Website
Santman
13
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 4th Jun 2017 08:49
Link
More than happy to give it a shot - What would you need from me?

I technically don't need to load it on the fly, I'm just being a perfectionist. In my game when the player jumps between star systems the new system needs to have textures for up to 16 planets loaded, and planets can potentially have up to 5 textures and these range from HD to 4k. At the moment it's seamless save for the pause as these load.

Though loading data while doing something could be used for all kinds of things.
Back to top
Profile PM
Xaby
FPSC Reloaded TGC Backer
17
Years of Service
User Offline
Joined: 17th Apr 2007
Location: Berlin
Posted: 4th Jun 2017 09:38
Link
@Santman

I don't realy understand, what you are trying to do. Or what your solution is.

If I have an image, which is about 4 MB as JPEG, but it is 4K x 4K, it would be about 64 MB in RAM (without texture compression like ASTC). If we save a "Bitmap" where do we do that?
Imagine you are on an Android System, and your APP is about 100 MB and the user has not much "disk space" free.

The next question is, when do you "convert" you JPEGs into Memblock-Raw-Pixel-Data? What if, your User opens your App the first time? How will the expirience be?

From what I understand, your method would need, the first time loading the App, also the time for JPEG decoding, and space on User side. It would be like an installation.
Your application on windows would be like 400 MB in Diskspace instead of 25 MB. (Diskspace is about 16 times the size of your well compressed JPEGs)

The next thing is, that Windows OS may cache Data from the Disk. So if you realy load 50 times the same File from Disk, it would be maybe the first time slow, but 49 times it is in Memory.
So for a propper test, it realy has to be 50 different files, not only copies of the same file, to avoid caching.

What about https://www.appgamekit.com/documentation/Reference/Image/LoadSubImage.htm LoadSubImage?

You could load only one JPEG with your FullHD-Pictures. 2048 x 1024 (with e.g. nearly FullHD) that would be 2 x 4 images on one 4K by 4K image. (8 images in one image)
Don't use PNG, because the decoding takes forever.


Also maybe some of your textures and planet images could build of "tiles" or combinations of images, without sacrificing to much quality. You could load on image maybe in "background" when showing Title-Screen, or Menu. Maybe you don't want your Player to wait 10 seconds in one, but maybe your Player could wait some seconds, while reading a Story on screen or listening to music. And maybe you have seperate parts, when your Player would only waiting 1 or 2 seconds in one.

So, how would your method improve the first launch of the game? The best way would be, we had nativly ASTC (Adaptive Scalable Texture Compression in AppGameKit 2 Tier 1. You than could push your files direct from Disk to Memory and would have small filesize and small usage or Memory.

Back to top
Profile PM Website
Santman
13
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 4th Jun 2017 11:43
Link
Hi Xaby.

You pose some good questions, but actually the demo I showed does show multiple images, so I don't think cahcing is the issue. In terms of purpose, this is very much a look from my own perspective - and I am developing only for windows so file size isn;t such an issue. Testing showed that JPEG and PNG essentially load much teh same time - but it;s the decompression that is the issue in terms of speed - the imahes saved off as memblocks are much larger but can load and be accessed much faster. Since the tool converts the images in advance (a seperate tool) then you are never loading images in your main project - only the data files. For me, sub iamges aren't an option - my game has the potential to need large numbers of very large images, and loads new data (from a potentially unlimited range of graphics) constantly.

I've just converted my current game to use this method - it certainly added value to myself by:
*The initial load time dropped from 15 seconds to 8 seconds
*The loading of new AI ships mid play no longer causes any noticable stutter (where before it did) - this is important for me as players can customise and add ships - there is a potentially unlimited number of graphics that could be needed here so pre-loading would require potentially sizable volumes of ram and would make tracking them much more of a pain
*Jumping systems loads new data much faster - and that's just loading the memblocks, once I put in streaming then the "jump" sequence will play on screen and the new graphics can load without interupting this

It has resulted in increased file sizes, but that's something I personally can live with - but I accept that will very much come down to the user.

An ideal solution would be threaded loading - if you are listening Paul and Rik! Lol.

Not saying my solution is the best, but there's no disputing that loading a memblock saved image seems massively faster than loading an image, for whatever reason.
Back to top
Profile PM
Golelorn
8
Years of Service
User Offline
Joined: 20th Nov 2016
Location:
Posted: 4th Jun 2017 13:15
Link
Santman,

Thanks! I'm very new to all this. I'm not sure what a memblock is but looks worthy of researching and further study on my part.
Back to top
Profile PM
Santman
13
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 4th Jun 2017 15:04
Link
More than welcome Golelorn- it's see an area that's more than worth looking into. Took me a while to get my head around It, but now I manipulate them a fair amount to do assorted damage effects.

Every day is a school day with coding....for all of us!
Back to top
Profile PM
Santman
13
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 8th Jun 2017 15:11
Link
So it took a couple of evenings but that's Star Captain fully implementing the streaming background graphics, shown in the video below...for this I left some text on screen to show it.


So the theory works in practice too....i use memblocks loading to remove the glitch that occurred as the game loaded in new graphics for AI ships, and the "streaming" option for loading new systems while the player jumps.

As pointed put, this isn't a solution for everyone but it suits my needs and what I want. As a side, memblocks for smaller images are actually not hugely larger than the images, depending on compression, so unless you are using HD graphics you could probably use a mix of the two techniques for mobiles too.

If anyone is interested shout and I'll post the streaming code, but it might be the easiest thing to follow my clumsy coding habits! Lol.
Back to top
Profile PM
JohnnyMeek
11
Years of Service
User Offline
Joined: 23rd Apr 2013
Location: Slovenia
Posted: 9th Jun 2017 06:27
Link
Does this work with sprite sheets?
Back to top
Profile PM
Santman
13
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 9th Jun 2017 09:33
Link
Regular sprite sheets, as in an image? Absolutely. It will work with any image saved as a memblock, any sound saved as a memblock, or any other data....its just reading the file integer by integer.
Back to top
Profile PM
smallg
Valued Member
18
Years of Service
User Offline
Joined: 8th Dec 2005
Location: steam
Posted: 10th Jun 2017 15:18
Link
curious if this would be useful to store all the images and sounds into memblocks and then saved to a file in advance using a separate project, you could use the memblock data to write files which hold the "images" and then copy across the files instead of the media you used?
just as a form of very basic media protection i mean but also if this method is faster anyway then you get both bonuses plus it saves on the space needed as the media is not included - just depends how long it takes to read from file rather than actually create the memblock to see if the initial conversion process is better or worse.
life\'s one big game
spec= 2.6ghz, 1gb ram, 512mb gpu, directx 9.0c, dbpro and classic
Back to top
Profile PM Email
Santman
13
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 10th Jun 2017 20:10
Link
Yeah, that should be fairly easy to do - planned something similar myself, but slightly different. As I load all the images as memblocks now I don't ever actually need to save them as images, I just retain the memblocks. For images, teh first three integers are width, height, depth - depth isn't used but if you store the width and height you can then just make these zero - that way the dimensions of the iamge are "hidden" - not useful for a memblock itself with just one image however...

When looking at memblocks we need to consider that the data is actually just sequentially stored. So an image of 8x4 pixels we want to understand as:
1,2,3,4,5,6,7,8
1,2,3,4,5,6,7,8
1,2,3,4,5,6,7,8
1,2,3,4,5,6,7,8

However in memblcok terms it's actually 1,2,3,4,5,6,7,8,1,2,3,4,5,6,7,8,1,2,3,4,5,6,7,8,1,2,3,4,5,6,7,8. So in fact, we can stream one image write into the next using the streaming option. So for example, in my video the Sol system has 22 images (currently)...but in the final build I aim to have just one large "PACK" file containing them all. To read individual images, I just need to set the offset of the memblock to load the image I choose.

You could actually extend this to every single image in your game, but loading speed might be something you want to look at as streaming integers is much slower than loading an image, and much much slower than loading memblocks.

For reference, the AppGameKit player worked from my laptop, and the results were almsot identical to the PC - loading memblocks was about 5 times faster, and streaming had an almost identical perforamnce, at least on an S8.
Back to top
Profile PM
Jerry McGuire
7
Years of Service
User Offline
Joined: 25th Mar 2017
Location:
Posted: 13th Jun 2017 17:11 Edited at: 13th Jun 2017 17:14
Link
+1 found this thread very useful for learning Nice video-demos too
iMac Book Pro, MacOS 10.12.4, Xcode 8.3.3;
iPhone 6, iOS 9.35; iPhone 5s, iOS 9.35; iPad (3rd gen), iOS 9.35;

Dell Precision T7400, Windows 7 Professional 64bit, Visual Studio Community 2015;
Back to top
Profile PM
Santman
13
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 16th Jun 2017 11:39
Link
Thanks Jerry, if you found any of it useful I'm glad. I'm surprised you decoded all my typos!

I've been busy fixing 3d lighting and adding blurring effects to the jumping in star captain, but tonight I'll try and post the full streaming code...hopefully someone else can jump in and point out any ways in which it could be optimised.
Back to top
Profile PM
Back to top

Login to post a reply

Server time is: 2024-11-24 14:24:45
Your offset time is: 2024-11-24 14:24:45