Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

AppGameKit Studio Chat / Memblock save game data (dynamic string length)

Author
Message
jd_zoo
4
Years of Service
User Offline
Joined: 12th May 2018
Location: Nova Scotia
Posted: 22nd May 2022 20:59
So I am growing frustrated with my save game CSV structure, and found a way to convert over to binary and use memblocks to directly load integers and floats without converting from strings. My save game data is only ~15% strings, and a save can be 1000 tokens in size.

This working example requires you to write and read back the data in the exact same order to take advantage of the built-in indexing. Integers and floats get 4 bytes. When inserting strings first an integer with the ByteLen() + 1 gets added followed by the string, this is all done in the background and keeps track of the offset as you read and write. The memory block size is maintained as a multiple of 4.

I have yet to implement into Automation, this was the first step as a proof of concept. I will post back with loading times but it won't be a direct comparison as I plan on re-doing the way the level gets generated after loading the data. It can't get any worse than reading back from strings, can it?

Anyway if this concept is of use to anyone out here it is a cleaned up copy pasted below.

JD

Check out Automation on Steam now
https://store.steampowered.com/app/1698690/Automation/
Kevin Picone
19
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 23rd May 2022 15:42 Edited at: 23rd May 2022 15:47
The concept is sound. You could just wrap the file IO functions and have the wrapper read/write to the current memory block. So the user calls some type of Output/write function that reset the current pointer and or recreate the block to flush out and legacy data. Ideally the library would handle the size and resizing on the users behalf, making sure the buffer is sized and only expands only when an overflow is likely, reducing the memory thrashing.

I know the code is proof of concept, wrote a quick replacement for Makemod4() function but it would seem AND isn't behaving in AGK.
.


This one seems to work.


PlayBASIC To HTML5/WEB - Convert PlayBASIC To Machine Code
jd_zoo
4
Years of Service
User Offline
Joined: 12th May 2018
Location: Nova Scotia
Posted: 23rd May 2022 22:16
Yep I agree this simplified version could get packaged or easily re-purposed, and yeah I have already modified the mod4 so that it is caught before creating the string's memblock. My full working demo has multiple memblocks for different data sets of the game, then creates a header with all their sizes before combining into a single memblock which gets written to disk. Another layer of complexion but possible to include as a full packaged set of save game tools.

One thing is that this results in a new file for each save game, as opposed to a new line in a single CSV file.
Check out Automation on Steam now
https://store.steampowered.com/app/1698690/Automation/
Kevin Picone
19
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 26th May 2022 03:07 Edited at: 29th May 2022 02:58
Hi jd_zoo;

Quote: "
My full working demo has multiple memblocks for different data sets of the game, then creates a header with all their sizes before combining into a single memblock which gets written to disk. Another layer of complexion but possible to include as a full packaged set of save game tools.
"


That's the way to go, only thing to be careful about would be the amount of number crunching.. That's not really a strength of the AGKS runtime; if there's a function to write a chunk of memblock data directly to disk; you could do the composite to the target rather than combine in memory-> then save it all.


Something I noticed the other day and I get the above code is mostly cut'n'paste from useage situation, but the AddInteger etc functions have a to create the buffer then copy old data across to the stream.



The issue with this is that if anyone ever needs to store a large block of Integers / floats or whatever; then for every push; there's an memory allocation combined with a buffer copy. So if the mem block was a megabyte say; They'd only have to push a 1000 values and it's thrashing a gigabyte of memory.

Easiest way to reduce or remove this would be to pre-size the output mem block up front. Then each time we push data; the function compares the current output address+size of data to be written against it's current size of the mem block. if the write will overflows the mem block; resize it.. otherwise; just drop the data to output address and bump the output address.


Example: ( Untested written in browser )


PlayBASIC To HTML5/WEB - Convert PlayBASIC To Machine Code
jd_zoo
4
Years of Service
User Offline
Joined: 12th May 2018
Location: Nova Scotia
Posted: 27th May 2022 01:44
Ahhhh ok yes thank you, my mind doesn't appreciate the back end workings! I was thinking one step at a time, but in the end the sync could be pretty crushing.

I really like the idea of incrementing it in preset sizes as you need more space... I have definitely learned a lot about this over last few weeks!
Check out Automation on Steam now
https://store.steampowered.com/app/1698690/Automation/
jd_zoo
4
Years of Service
User Offline
Joined: 12th May 2018
Location: Nova Scotia
Posted: 19th Jun 2022 15:40
Not a fair comparison, my old string looping through the CSV file was not an overly efficient approach in hindsight. But for my situation the flexibility with a binary setup is the first reason and speed is a bonus...

Check out Automation on Steam now
https://store.steampowered.com/app/1698690/Automation/

Login to post a reply

Server time is: 2022-08-09 09:39:42
Your offset time is: 2022-08-09 09:39:42