Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

AppGameKit Studio Chat / Is there a limit to array sizes?

Author
Message
Jeku
Moderator
20
Years of Service
User Offline
Joined: 4th Jul 2003
Location: Vancouver, British Columbia, Canada
Posted: 14th Mar 2023 14:15
I'm introducing my father to AppGameKit and he wants to store characters in an array from a 3GB file. Basically he'd need 3 billion elements in a fixed array. Is there an issue with this or does he need to use smaller chunks? It would be great if he could store the entire array from the file in RAM as he has 16GB of RAM.

Thanks!
Raven
19
Years of Service
User Offline
Joined: 23rd Mar 2005
Location: Hertfordshire, England
Posted: 15th Mar 2023 01:07
Maximum Size for an Array is 4,294,967,296 Elements
AppGameKit does use Signed 32bit Integers., i.e. -2,147,483,647 to +2,147,483,648... there are edge cases when this can be a problem when working with values above 2.147 billion.

With this said., I wouldn't recommend using a Single Array... and not because it won't support what he wants but because of performance.
Array performance drops exponentially with size., for example cycling through 1 million elements is approx. 23x slower than 10 elements 100,000x ... it's the same number of memory / variable accesses but the larger array is slower.

Same is true for For...Next Vs. Repeat..Until


Note: I use 10million iterations as it's still quick to and showcases a clear difference (0.48 and 0.42 on avg. on my CPU)
For reference., increasing to 1billion iterations takes 50s and 43s respectively on my CPU

This does showcase the weird performance from various built-in logic, but more to the point that the increase in iterations doesn't linearly scale.
Again, another reason why he might want to use smaller datasets.

With this said... a question I have is why is he storing these characters?
If for example he is importing words BUT he still wants to store the characters individually, well he could do something like:



This isn't the most elegant solution as it's like 1am but will read an ascii / utf8 text document in... each Document element is a Word / Punctuation, while each Letter is stored in the .Word array
In this way if you want to know the Length of a given word., you just use Document[Word].Word.Length
And you can still access each individual letter as Document[Word].Word[Letter]

I'm sure if I were more awake I could come up with a better solution., but you get the idea.
Jeku
Moderator
20
Years of Service
User Offline
Joined: 4th Jul 2003
Location: Vancouver, British Columbia, Canada
Posted: 15th Mar 2023 18:13 Edited at: 15th Mar 2023 18:15
Hi Raven, nice to see you're still around here! It's been a long time eh

Thanks for your help. I would find it hard to believe TGC would use an unsigned integer internally for the max array size, so it SHOULD be signed (because you can't access a negative index in the array, or can you?) which theoretically should give around a 4.2 billion or so size.

Here is what my father wants to do--- there's a 3GB file with random letters that he wants to read in and display on the screen, each character being its own unique colour. The reason why we want to put it all into RAM is so we can do things with it, like change the x, y size of the screen resolution so we can shrink it horizontally and expand it vertically in "real time". I put real time into quotes because I understand moving around billions of pixels won't be fast. But if we want to dynamically adjust the resolution we don't want to have to re-read the text file each time.

If there IS a limit like you say, 2.1 billion or so elements, then we can make do and read in maybe 1 billion at a time. If we absolutely must read in the file each time we tweak the viewport, then we will have to do it that way.

Take care and thanks again for your detailed response!

- Jeku
Kevin Picone
21
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 17th Mar 2023 11:44

If you want manipulate bytes, then work in bytes. I'd have to check the AppGameKit documentation but I don't think there's a byte array so you'd have to nibble the data integer by integer which is performance suicide in a runtime.

What's the numeric range of the data is it just ASCII ? full 8 bit ?

One idea that comes to mind (if there's no null terms in the stream, would still work if there is) would be to build a custom font. So each character is just a unique palette colour (a colour tile), allow your to move through the stream from disk. For display text on the gpu side.

Another way would be move the problem the gpu side. So pull the data into a mem block, make a texture and write shader to render them.

The data is going to the sample down to fit on screen as s there's not enough pixels on a 4K screen to display it all in one view.


PlayBASIC To HTML5/WEB - Convert PlayBASIC To Machine Code
fubarpk
Retired Moderator
19
Years of Service
User Offline
Joined: 11th Jan 2005
Playing: AGK is my friend
Posted: 20th Mar 2023 21:36 Edited at: 21st Mar 2023 12:51
@Jeku
using an image for large amounts of text could be done similar to my method below that I threw together but for example
a 128* 128 pixel image could store if my math is correct 128^4 * 128^4. the below code will need work if you plan to go down that method
and you could always use the alpha channel for database indexing. (edited its now dumping an example etc correctly with bytes)
fubarpk
https://fubarpk.itch.io/

Login to post a reply

Server time is: 2024-04-23 09:20:19
Your offset time is: 2024-04-23 09:20:19