Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

AppGameKit Studio Chat / Is there a maximum file size to files we can load or #include into AGK Studio?

Author
Message
PHeMoX
6
Years of Service
User Offline
Joined: 9th Jan 2018
Location:
Posted: 20th Feb 2022 21:02
I'm trying to load basically a JSON / XML file that's very large, around 33 megabytes (yes, thousands of lines). AppGameKit Studio can't seem to deal with this at all. It gives an error and crashes AppGameKit Studio completely :

Uncaught exception
Cannot set image wrap mode to repeat on an image that is not power of two in size

In reality, the project has no images...

Any tips on how to proceed? Is this even remotely possible with AppGameKit Studio? I'm trying to use some open data, but it's one big huge file. Original file format is XML.
Virtual Nomad
Moderator
18
Years of Service
User Offline
Joined: 14th Dec 2005
Location: SF Bay Area, USA
Posted: 20th Feb 2022 23:56
haven't played with datafiles that large but if you want to send a link, i'll see what i can do with it?
PHeMoX
6
Years of Service
User Offline
Joined: 9th Jan 2018
Location:
Posted: 21st Feb 2022 11:31
Yes, it's basically Phaelax's XML tool code trying to load an xml file downloaded from https://projects.rvo.nl/opendata/iati.xml , but I've also converted this xml into a JSON using Notepad++ and tried to #include the file that way. It'll crash AppGameKit Studio.

Let me know if this is enough to work with. (I've also checked, but the XML tool code works fine with smaller xml files, it'll show the structure using Print. It may be that it'll go out of bounds when having so many lines and so many spaces. I'm guessing that's the problem. However, it wouldn't explain the crash when trying to #include basically an agc file with just the JSON in it.).
Virtual Nomad
Moderator
18
Years of Service
User Offline
Joined: 14th Dec 2005
Location: SF Bay Area, USA
Posted: 21st Feb 2022 17:07 Edited at: 21st Feb 2022 17:22
it's apparent that i don't know enough about .json in AppGameKit but phaelax's XML parser produced a (~95MB) .json file (after about 5-10 mins which had over 500k elements the last i looked - see ATTACHED which compressed down to ~4MB) which i then tried to load via MyArray.Load( "iati.json" ).

it did load the .json file (there is a few second pause before my test continued) but the array showed a length of -1 on completion.

i believe the array first needs to be defined as the proper type but i can't sort it out.

and, i don't see any info re: the xml file that might help define a Type to make it work?

if you're unfamiliar with .json in AppGameKit (as i am), see HERE but i believe this is the way to go.

otherwise, how did you try to #include it?

Attachments

Login to view attachments
adambiser
AGK Developer
8
Years of Service
User Offline
Joined: 16th Sep 2015
Location: US
Posted: 21st Feb 2022 17:42
@Virtual Nomad: The json in your zip appears to be invalid.

The ending looks like:

Looks like it reached the string limit of 100,000,000 bytes.

https://github.com/TheGameCreators/AGKTier2/blob/master/common/Source/uString.cpp#L912
https://github.com/TheGameCreators/AGKTier2/blob/master/common/Source/uString.cpp#L948
Memblocks also have this limit:

Can't link to common/Source/Wrapper.cpp on github because the file's too big.
Virtual Nomad
Moderator
18
Years of Service
User Offline
Joined: 14th Dec 2005
Location: SF Bay Area, USA
Posted: 21st Feb 2022 18:05 Edited at: 21st Feb 2022 18:21
i also now realize that i didn't explain the process i used above well enough in that, once phaelax parser was complete, i used dom.Save("iati.json") to export it.

and, i did try using his Types to reload:


...which is where the pause occurred and my belief that the file was loaded before it proceeded to my Do/Loop containing Print(dom.length) which yielded the -1 result.

(aside: the xml was parsed under MinimizeApp() which proved much faster and, once parsing was complete, RestoreApp() (in the snippet) was called to let me know it was finished - could this have some averse effect?).

as far as the limitation, PHeMoX's .JSON file should then be broken up into chunks and then loaded as smaller strings?
PHeMoX
6
Years of Service
User Offline
Joined: 9th Jan 2018
Location:
Posted: 21st Feb 2022 18:50
Thanks for looking into this guys! I very much appreciate it. I'm not sure if breaking up the file would work, as the main type starts on the first line and ends on the very last. I might have to at least manually change that then. There are plenty of chunks of data that are more of their own container so to speak, so maybe not impossible to divide in smaller chunks.

I do have a hard time believing it's breaking the 100,000,000 bytes limit though. Sounds like I need to write like a PowerShell code that cuts the file up into more manageable pieces or do this by hand (which doesn't sound like fun lol).
adambiser
AGK Developer
8
Years of Service
User Offline
Joined: 16th Sep 2015
Location: US
Posted: 22nd Feb 2022 01:54
Depending on how the file is being loaded, I have experienced and reported this bug, but it didn't give any errors about image wrap mode, just reported incorrect text.

Login to post a reply

Server time is: 2024-03-29 00:24:42
Your offset time is: 2024-03-29 00:24:42