Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

AppGameKit Classic Chat / Multithreading: Async object/image loading possible in AGK ?

Author
Message
Sandwich
6
Years of Service
User Offline
Joined: 23rd Sep 2017
Location:
Posted: 10th Jul 2018 12:48
Hi all,

I was wondering if it is possible to load objects or images asynchronously behind the main game loop.
Necessary for large maps, where you don't want - or simply can't - load every single object/texture if you are only displaying 5% of them.
Synchronously loading larger objects/images would result in a very unsatisfying timing-behaviour while playing.

Somewhere I read that only one AppGameKit command can be performed at a time... is this true?

Thanks in advance,
Sandwich
Santman
12
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 10th Jul 2018 13:12
No, not in tier 1......or at least not quickly.

I put together some code that does it using memblocks, but the disadvantage there is that men blocks are larger.

That said, an image memblock loads 30-40 times quicker than a png or jpg.
Santman
12
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 10th Jul 2018 13:13
https://forum.thegamecreators.com/thread/219750
Markus
Valued Member
20
Years of Service
User Offline
Joined: 10th Apr 2004
Location: Germany
Posted: 10th Jul 2018 13:32
loading Async images is a good idea, should be a feature request.
so the sprites would be visible as soon the images are loaded, and the loading did not block the game.


AGK (Steam) V2017.12.12 : Windows 10 Pro 64 Bit : NVIDIA (390.65) GeForce GTX 1050 Ti : Mac mini OS High Sierra (10.13)
Santman
12
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 10th Jul 2018 14:11
Lol.....its been asked for a TON of times. Lol.

I imagine the download code could be altered to do it, except download from a disk instead.

Images that memblock work around handles, 3d objects not so well.... they take so long to set up in memory I've not found a difference between an object file and a memblock, but eventually I intend to looking at T2 to see if it can be done.
Alex_Peres
AGK Master
14
Years of Service
User Offline
Joined: 13th May 2009
Location: The Milky Way Galaxy
Posted: 10th Jul 2018 15:05
I used to load a video-file by the second thread in Tier2. But when I tried to load images - it was failed.

AGK is made for amateurs and it doesn't have any pro stuff.
Bengismo
6
Years of Service
User Offline
Joined: 20th Nov 2017
Location: Yorkshire, England
Posted: 10th Jul 2018 16:20 Edited at: 10th Jul 2018 16:42
Its a bit odd....I got the reverse results regarding loading png's vs memblock files

I found that on average a 1920x1080 png file loaded in 5.4mS and the 1920x1080 memblock file loaded in 21mS (the memblock file was created once by saving the memblock to disk using createFileFromMemblock() - its contents were the same as the png image as it was created from it) - both images look the same in AGK

I've tried loading 1x, 20x, 40x, and 60x times and averaged results and in every experiment...just loading the PNG is quicker in every case. I tried with various sizes of images too.

Quite Strange?? it might be worth posting up the test code for this and allow people to try and test the load times.


As far as the op is concerned
There are some good options if you have many files to load.

1) Load just one or two images per frame...split your loads over multiple sync() / game frames (Manage your loadimage calls) - have a function to manage your loads and spread them out
2) Load a memblock image but only load a part of the file at a time - Ie so on each game loop you have a function which loads up another 512kb and eventually creates the image in the background
3) A plugin which asynchronously loads the file in its own thread then it is polled to tell you when its done and you can then use your new image
4) Split large map images or data into multiple chunks and stream them in multiple load calls (ie - multiple images instead of a huge one)

Anyone of these can get round the loading time issues faced. Async loading sounds great but in reality you still have to wait for the image to load so you dont get it any quicker...you just dont have a massive stutter that occurs in the frame rate.
Santman
12
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 10th Jul 2018 18:25
Bengismo, number 2 is literally what my code does. The process is simple - when you need to load an image, drop it's details into an array, and up a counter. If the array contains any non loaded data, begin "streaming in" that memblock, a fixed number of inputs per frame, then when it's done use the last frame to create the image. Works quite well.

As for loading images, was that as a 32bit exe? Only thing I can think of - tested it on mobile as well and every single time loading the memblock is faster as there's no decompression. Will see if I can find the project.
Santman
12
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 10th Jul 2018 18:27
Ok, here's the full project.

Can't recall the details, but I think the loading can be tested by changing a flag somewhere.

Attachments

Login to view attachments
Bengismo
6
Years of Service
User Offline
Joined: 20th Nov 2017
Location: Yorkshire, England
Posted: 10th Jul 2018 19:12 Edited at: 10th Jul 2018 19:23
I will take a look at you streaming code but i quickly noticed your not deleting images so will end up with incorrect load times as your just filling memory with the same image over and over without any releasing. Eventually your measuring your pc having issues with memory allocation and having to page memory out and garbage collection which will interfere with results....id be more inclined to believe numbers from loading just 1 image so the files are not pre cached on the hard drive cache too which isnt realistic either.

Also in some tests your comparing just the conversion of the memblock to an image without any of the loading times? A lot of images arent being freed. I need to look through the code more carefully but am much more inclined to believe much more simple tests.

What i have found is that the load time is highly dependant on the compression in the png file.

As far as the op is concerned...
Wouldnt be hard to create a plugin that can genuinely create threads to do the loading and decompression in a lower priority process. That would be cool.
xCept
21
Years of Service
User Offline
Joined: 15th Dec 2002
Location:
Posted: 10th Jul 2018 19:50
I would also love to be able to use SaveImage() on a separate thread. I have an app that makes use of this command to save full-resolution images that can be up to 2732x2048 and it is a major bottleneck in the application (at which all activity freezes). I asked Paul about this since the PrintImage() command already saves to the photo library on a separate thread instantly and he said it may be possible to move file saving to a new thread. But it didn't seem like a high priority request.
Santman
12
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 10th Jul 2018 21:58
Bengismo, that's extremely messy code - it was a series of tests; started with one then moved to another so the code altered slightly, and so forth. I hear what your saying but completely don't accept it as it runs equally as fast on a mobile, and still faster than loading images - the thread has extensive results, and each new result is basically a new version of this play code.

In the end I built it into my 2d space game and it was working fine - not releasing images would liekly just be left over code in this test....obviously in real life you would release images, but from what I recall it's alternating the loaded images there just to show that it's not just the efect of loading then releasing it, and loading it again so quickly it;s still in memory. Try it on a phone, would be very interested.

Would also love it if someone could get a plugin working....but actually we could do it with AppGameKit now if they would just give us PEEK/POKE and run time variable passing. I did it with DBPro - main .exe ran a second .exe and passed in a memory location (Matrix utils), so that the second app could then write a flag to that location that the main .exe picked up. Hey presto, communication between the apps. I had a game rumming in 3D with a background program loading music files over and over and it could manage something ridiculous like 1GB a minute comfortably with minimal frame rate hit.

So what we need is:
The ability to get the memory location of a memblock
The ability to get a free memory location
The ability to read/write to a memory location
The ability to make a memblock from a memory lcoation

That's really it. Then main.exe starts the second, passes a memory location they can use to "talk" (similar to network code). Then the main .exe can pass the name of an image needing loaded, the second .exe then loads it and makes a memblock, passes the start back. Main exe then converts said memblock into an image and bobs your uncle.

Or to be more perfect.....allow two .exe files, or a second process, to share image memory so the image loaded via the second is available to the first. Main processreserves the memory, second fills it, first uses it.

I;ve asked for this in EVERY survey! Lol. I actually still have matrix utils .dll that does almost all of that if anyone wants a bash at the plugin....sure that would sell.
Bengismo
6
Years of Service
User Offline
Joined: 20th Nov 2017
Location: Yorkshire, England
Posted: 10th Jul 2018 22:12 Edited at: 10th Jul 2018 22:35
Id always release images as it just good practice and doesnt eat up memory resource. No problem, the code actually make a lot of sense....i shouldnt have criticised it. Its fine.

Plugin peek/poke works fine in the plugin as you can get a pointer....the command is something like getMemblockPtr() or Getimageptr() that are available in plugins so basically it should be possible to directly manipulate the memory contents in C/C++/C# or whatever!!

Not sure if your familiar with pointers (im guessing you will be) but thats the location of where the memblock is in memory so you can access it and read/write to it.


Ive never actually had a problem with load times myself so have never actually felt the need for any of this but it would be a nice option if anyone ever did genuinely have this issue.
Santman
12
Years of Service
User Offline
Joined: 15th Sep 2011
Location: Inverness
Posted: 10th Jul 2018 22:42
Pointers......nope, not familiar. My knowledge is essentially more basic language, with some sharers. That's why I didn't make the plugin.....I know how it would work, but never done C.

Once the world creator is up and running I aim to try T2 as AppGameKit is just too slow for basic loops.
puzzler2018
User Banned
Posted: 10th Jul 2018 22:54 Edited at: 10th Jul 2018 22:56
If use C# then NuGet UnmanagedExports

and use

using RGiesecke.DllExport;

and
[DllExport("PlayYouTube", CallingConvention = System.Runtime.InteropServices.CallingConvention.StdCall)]

like



C# is more freindlier than C++
Sandwich
6
Years of Service
User Offline
Joined: 23rd Sep 2017
Location:
Posted: 17th Jul 2018 08:42
Wow what a resonance Thank you all!

I forgot to say that I am using the C++ Version!

So the memblock-way (partially loading a memblock in each frame / background-thread) seems to be very interesting - except one little thing: Memblocks on disc are multiple times larger than a compressed JPEG.

So there are two ways: Have a large program folder (where everything is already stored as Memblock) or do some kind of installation process, where all images are converted into memblocks.

Because I am targeting mobile devices, I will do some kind between: When starting a level, I am going to install all necessary files (convert from JPG to Memblock) *before rendering* in a temporary folder if they don't exist. In game I want to load the memblocks step by step in the main loop, depending on the current loop time.

Is this the correct way or am I missing something?

Thanks again for the large amount of replays!

Login to post a reply

Server time is: 2024-04-19 03:12:17
Your offset time is: 2024-04-19 03:12:17