Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Professional Discussion / Can't load more than a few hundred images

Author
Message
Thebrianwing
15
Years of Service
User Offline
Joined: 15th Jun 2011
Location: United States
Posted: 23rd Dec 2012 06:15
Link
I'm loading images in using a for next loop and for some reason after about 490 (it varies between 495-499) it errors out with error code 506. It doesn't seem to matter the image numbers, as long as I load around 490 images it gives up. Is there an image number cap in DBPro?

Back to top
Profile PM Email
CumQuaT
AGK Master
16
Years of Service
User Offline
Joined: 28th Apr 2010
Location: Tasmania, Australia
Posted: 23rd Dec 2012 10:47
Link
You could be hitting an internal memory cap. How large is each image file on average? And how large are 490 of them put together?


Back to top
Profile PM Email
MrValentine
AGK Backer
15
Years of Service
User Offline
Joined: 5th Dec 2010
Playing: FFVII
Posted: 23rd Dec 2012 17:11
Link
I was aware of a 32 image cap... hmm... but yeah 500 images... must eat up at least 1GB...

Interested to see the result of this...

Back to top
Profile PM
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 23rd Dec 2012 17:39 Edited at: 23rd Dec 2012 17:42
Link
It's a good topic.

Is there any 100% way to know minimal system requirements for your game? I mean RAM and video memory sizes that are needed to run the application without memory error.

Other requirements are easy to be known: shader version, directx version, HDD free space.

*** Merry Chuckmas! ***
Back to top
Profile PM
Brendy boy
20
Years of Service
User Offline
Joined: 17th Jul 2005
Location: Croatia
Posted: 23rd Dec 2012 17:45
Link
Quote: "I was aware of a 32 image cap... hmm..."

there's no such cap for images, 32 might be the cap for cursors

BBB GUI Plugin
Landscape & Map Creator
Back to top
Profile PM Email
MrValentine
AGK Backer
15
Years of Service
User Offline
Joined: 5th Dec 2010
Playing: FFVII
Posted: 23rd Dec 2012 17:53
Link
Quote: "there's no such cap for images, 32 might be the cap for cursors"


Thanks Brendy!

Back to top
Profile PM
Andrew_Neale
16
Years of Service
User Offline
Joined: 3rd Nov 2009
Location: The Normandy SR-2
Posted: 23rd Dec 2012 18:36
Link
Quote: "Is there any 100% way to know minimal system requirements for your game? I mean RAM and video memory sizes that are needed to run the application without memory error."


As far as I remember, DBPro can't make use of more than 2Gb of RAM, so that is probably a safe bet, as well as being the least I would expect a Windows user (let alone gamer) to have these days anyway.


Previously TEH_CODERER.
Back to top
Profile PM Email Website
MrValentine
AGK Backer
15
Years of Service
User Offline
Joined: 5th Dec 2010
Playing: FFVII
Posted: 23rd Dec 2012 18:54
Link
Ah I think I got my confusion, 32 BITMAPS!

Quote: "Specify a bitmap number between 0 and 32. "


Back to top
Profile PM
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 23rd Dec 2012 20:37 Edited at: 23rd Dec 2012 20:43
Link
@Andrew_Neale
Quote: "As far as I remember, DBPro can't make use of more than 2Gb of RAM, so that is probably a safe bet"

How can you calculate how many "3d teapots" you can have in your game? Also you have two types of RAM: system and video.

*** Merry Chuckmas! ***
Back to top
Profile PM
Andrew_Neale
16
Years of Service
User Offline
Joined: 3rd Nov 2009
Location: The Normandy SR-2
Posted: 23rd Dec 2012 21:37
Link
Quote: "How can you calculate how many "3d teapots" you can have in your game? Also you have two types of RAM: system and video."


Indeed, but I don't think it is safe to count on the VRAM. As I understand it, the GPU will start making use of RAM (or page file depending on drivers) once it runs out of VRAM so, in terms of aiming to safely stay within memory limits, you're going to want to keep your total memory usage (meshes, textures, shaders, variables, arrays, sounds etc.) to the 2Gb RAM range when working with DBPro. The GPU can utilise the RAM when required but not the other way around, so it wouldn't be safe to aim for higher than the RAM limit.

If you know you have 2Gb maximum and you know the size of your assets/arrays/variables, both of which should be true, then it is relatively easy to work out how many of each thing you can afford to have. It may mean careful loading of assets to ensure there are no duplicates and only loading assets as you need them rather than everything up front, and even extending this to arrays where possible. This is what I've spent a lot of time in trying to optimise myself to keep memory usage to an absolute minimum at all times. It is surprising how easily 2Gb is used up!

I apologise if I'm wrong and am happy to be corrected for my own education if I have misunderstood something.


Previously TEH_CODERER.
Back to top
Profile PM Email Website
Sasuke
20
Years of Service
User Offline
Joined: 2nd Dec 2005
Location: Milton Keynes UK
Posted: 23rd Dec 2012 21:45
Link
One thing I've wanted to know is if you uvmap a mesh in an external program and load the mesh (object...) in DBP, is the texture loaded into memory when the mesh is loaded?

Since the mesh stores the ref data of the texture that's mapped to it, it must load it in some way right?
Back to top
Profile PM Email
Andrew_Neale
16
Years of Service
User Offline
Joined: 3rd Nov 2009
Location: The Normandy SR-2
Posted: 23rd Dec 2012 22:33
Link
Yes, and I remember reading that DBPro automatically manages that in terms of trying to load each texture only once (i.e. it recognises the same file being referenced and doesn't load it again). Though I don't know if that means only within the textures from objects loaded like that or if it also checks against any textures you may have manually loaded.


Previously TEH_CODERER.
Back to top
Profile PM Email Website
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 23rd Dec 2012 23:09
Link
@Sasuke
In certain programs you can remove texture names when exporting model.

@Andrew_Neale
Quote: "and you know the size of your assets/arrays/variables"

As far as I understand you, you just simply open task manager and watch memory usage there?

*** Merry Chuckmas! ***
Back to top
Profile PM
Andrew_Neale
16
Years of Service
User Offline
Joined: 3rd Nov 2009
Location: The Normandy SR-2
Posted: 24th Dec 2012 09:17
Link
Quote: "As far as I understand you, you just simply open task manager and watch memory usage there?"


That would work. The other option is that because you know the size of DBPro's base data types, and from that you can work out the size of any UDTs you have made, you can use them directly or multiply them by array sizes to workout how much memory each variable or array is using respectively. Dynamic arrays are obviously a little trickier, so you'd probably have to have a think as to how many elements it could have in a worst case scenario and use that as an estimate.


Previously TEH_CODERER.
Back to top
Profile PM Email Website
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 24th Dec 2012 13:36
Link
Okay, and how do you know when your video card's memory is full?

*** Merry Chuckmas! ***
Back to top
Profile PM
Latch
19
Years of Service
User Offline
Joined: 23rd Jul 2006
Location:
Posted: 24th Dec 2012 17:29
Link
Quote: "Okay, and how do you know when your video card's memory is full?"


Don't these commands help?



Quote: "It doesn't seem to matter the image numbers, as long as I load around 490 images it gives up. Is there an image number cap in DBPro?"


Quote: "You could be hitting an internal memory cap. How large is each image file on average? And how large are 490 of them put together?"

Most likely.

It's not the number of images (65335 I think is the cap), it's the amount of memory being used. Plan your images carefully. Use only the image size you need. You don't need 1024 X 1024 sizes and above for all your images. Sometimes 64x64 is just right for whatever you may need. In one 1024 x 1024 image you can fit 16 256x256 images, so keep that in mind. Also, there is a way of loading in your images with compression, but I can't think of the flag to add to Load Image. This will save on memory as well but might hit performance a bit.

Also, try and keep your images to powers of 2. That means 32x32, 64x64, 128x128, 256x256, 512x512 etc. Even though you may be able to load in other sizes, your graphics card will almost always do an internal conversion to a power of 2 size if you are using the images as textures for 3d objects and eat up some resources in the process (your image may also get skewed when finally rendered).

Enjoy your day.
Back to top
Profile PM
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 24th Dec 2012 22:24
Link
Quote: "graphics card will almost always do an internal conversion to a power of 2 size"

Then why set camera to image works perfect with any image size, like 100x100? I'd say that this size issue relates to OLD graphics cards.

system tmem available() <- always shows 2048 (win XP 32)
system dmem available() <- i always got crash on this
system smem available() <- hmm... this thing shows 2000 at start and decreases any time i load something... maybe that's what I need!

*** Merry Chuckmas! ***
Back to top
Profile PM
Morcilla
23
Years of Service
User Offline
Joined: 1st Dec 2002
Location: Spain
Posted: 25th Dec 2012 12:08
Link
Here is a little snippet that will tell how many images can be loaded into a particular PC:


(Change the 512x512 as needed to test other image sizes...)

Here it loads 1408 512x512 images.

Quote: "... the GPU will start making use of RAM (or page file depending on drivers) once it runs out of VRAM ..."


It happened in the past that some versions of DBPro were not behaving this way:

Bug reports - U5.8 mipmapped textures only use VRAM

Back to top
Profile PM Email Website
Latch
19
Years of Service
User Offline
Joined: 23rd Jul 2006
Location:
Posted: 25th Dec 2012 19:35 Edited at: 25th Dec 2012 19:48
Link
Quote: "Then why set camera to image works perfect with any image size, like 100x100? I'd say that this size issue relates to OLD graphics cards.
"

It's not that it doesn't work or it's a problem, it's just the way the texture memory is handled and moved around. It's likely that a 100x100 image is taking up the same space or indexing as a 128x128 image for example, but at the lower resolution of 100x100 so you end up using more resources for lower quality.

When mipmapping or texture resizing is needed, power of two sizes allow for easy and quick translation while avoiding float conversions. A mipmap of 128x128 might look like:

128x128
64x64
32x32
16x16
8x8
4x4
2x2
1x1

While at 100x100 without a power of 2 conversion:
100x100
50x50
25x25

The demonstration here is the detail you can achieve with power of 2, and the memory difference. You would expect much less memory usage with the 100x100 image than the 128x128, but if the 100x100 is upconverted, then your memory management predictions go out the window while you are left with a lower resolution.

Most hardware will accept whatever size image you give it, then convert it to power of 2 for use later. You can avoid this initial step and be in more control of your memory usage if you start with textures that are already power of 2. I don't know if directx iteslf does the conversion of non-power of two images before passing them to video memory to maintain compatibility even with much older hardware that only accepted power of two image from the get-go. If it's a software conversion, then it might be slower than the hardware conversion.

Also, using bit operations and math on power of 2 blocks is much faster and less processor intensive than non power of 2 especially once floats get involved.

Enjoy your day.
Back to top
Profile PM
Thebrianwing
15
Years of Service
User Offline
Joined: 15th Jun 2011
Location: United States
Posted: 26th Dec 2012 20:03
Link
I don't think it's an issue of RAM space because the images I'm loading in are 1024 x 768 and are mostly white space because they are test images and only have a disk space of 8 KB. When I did the math 500 of them are only about 2 or 3 MBs. That and when I watch my task manager when loading the images in I don't get anywhere near 0 free memory. This theory as to why I'm having this issue makes sense but doesn't seem to be the problem. I will try the code Morcilla posted and see if that tells me anything.
Back to top
Profile PM Email
Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 26th Dec 2012 21:04
Link
So, does anybody know how internal mipmapping affects the RAM size of a texture?

Back to top
Profile PM Email Website
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 26th Dec 2012 21:50
Link
If mipmap then image size is doubled.

*** Merry Chuckmas! ***
Back to top
Profile PM
Brendy boy
20
Years of Service
User Offline
Joined: 17th Jul 2005
Location: Croatia
Posted: 26th Dec 2012 21:54 Edited at: 26th Dec 2012 21:55
Link
Quote: "because the images I'm loading in are 1024 x 768 and are mostly white space"

white space doesn't matter. What format are they in? They all get converted to internal dx format after loading where image with size 1024x768 with mipmaps can take up to 4MB

Quote: "So, does anybody know how internal mipmapping affects the RAM size of a texture?"

it adds 100% of the image size to it

BBB GUI Plugin
Landscape & Map Creator
Back to top
Profile PM Email
Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 26th Dec 2012 22:03 Edited at: 27th Dec 2012 00:15
Link
Thanks

Edit:

Might be a business opportunity for a plugin developer to create a texture resource manager; something to unload textures not used by objects far away.

Back to top
Profile PM Email Website
Thebrianwing
15
Years of Service
User Offline
Joined: 15th Jun 2011
Location: United States
Posted: 27th Dec 2012 00:02
Link
I'm not mipmaping them and they are .png format. I ran the memory test and can load well over 11,000 .png images without mipmapping, though the speed at which they are loaded into the test program increases around 500 which is where I seem to be having my problems. Any other thoughs?
Back to top
Profile PM Email
Brendy boy
20
Years of Service
User Offline
Joined: 17th Jul 2005
Location: Croatia
Posted: 27th Dec 2012 06:30
Link
You said you're getting error 506. What is error 506?
You said you're gonna check the memory consumption of your program whem loading images, so how many MB of memory is used when the error happens?

BBB GUI Plugin
Landscape & Map Creator
Back to top
Profile PM Email
Thebrianwing
15
Years of Service
User Offline
Joined: 15th Jun 2011
Location: United States
Posted: 27th Dec 2012 06:46
Link
Error 506 is a "cannot load image at line *" error. I ran the memory test as stated above. It says that I have 4 GB of system memory available which I'm guessing it is reading from some Windows number since Windows also seems to have issues recognizing that I have 8 GB installed. The amount of available memory the test shows (using SYSTEM SMEM AVAILABLE) only begins to decrease somewhere around 450-500 images loaded. Curious everything begins to change around the 500 mark. Anyone else with any helpful info?
Back to top
Profile PM Email
Brendy boy
20
Years of Service
User Offline
Joined: 17th Jul 2005
Location: Croatia
Posted: 27th Dec 2012 08:00
Link
Quote: "It says that I have 4 GB of system memory available which I'm guessing it is reading from some Windows number since Windows also seems to have issues recognizing that I have 8 GB installed."

you probably have 32bit version of windows where 4gb is maximum memory size

Quote: "I ran the memory test as stated above. It says that I have 4 GB of system memory"

i asked how much memory your program consumes, not the total system memory. Each 32 bit program can consume maximum 2GB of memory. Dbpro programs are 32 bit.

Start task manager and then run your program in windowed mode so you can find your program in task manager's list and tell me how much memory your program consumes at the time of the crash.

BBB GUI Plugin
Landscape & Map Creator
Back to top
Profile PM Email
Morcilla
23
Years of Service
User Offline
Joined: 1st Dec 2002
Location: Spain
Posted: 27th Dec 2012 10:40 Edited at: 27th Dec 2012 10:41
Link
Quote: "So, does anybody know how internal mipmapping affects the RAM size of a texture?"

You can do a test with the code above, just add ,1 at the end of the load image sentence.

Quote: "I ran the memory test and can load well over 11,000 .png images without mipmapping"

The same happens here, but the execution doesn't ends like it should.
It shouldn't throw that 506 error. Instead, it should open a dialog (exit prompt sentence) telling how many images were loaded. This works this way (fine) when using mipmapped textures.
This reveals that in the long run there is a problem with non-mipmapped textures.

Quote: " The amount of available memory the test shows (using SYSTEM SMEM AVAILABLE) only begins to decrease somewhere around 450-500 images loaded."

Are you running a laptop or a system with shared video memory?
Here (desktop PC) memory starts to consume at the very start.

Initially it shows the free RAM, and it goes down as the images load. When consumed (around 800 images loaded), it starts to 'rebound' around 6-7Mb, showing that the memory paging has started, then the amount of images loaded goes up to those 1400 images loaded.

So, our systems are not behaving the same. It seems that yours load into VRAM directly, while here it begins to load directly into RAM.
I recall someone said that DBPro is CPU oriented, maybe I shouldn't get the textures directly loaded into RAM. After all, my card has 1Gb dedicated (GTX 295) but it doesn't seem to do any difference with my previous (512Mb) card.

Of course it is obvious that texture loading must be treated programatically (don't try to load 500 textures at once).

Nevertheless, textures should be always loaded 1st into VRAM, this behaviour may be a bug

Back to top
Profile PM Email Website
Latch
19
Years of Service
User Offline
Joined: 23rd Jul 2006
Location:
Posted: 27th Dec 2012 19:40
Link
There was a flag to load images directly into VRAM - I don't remember what it is. Also, pngs are expanded to the bitdepth X width X height adjusted for the pitch of the direct draw surface, so the memory size on disk is the compressed size.

Laptops aren't the best platforms to reliably relate performance or graphics behavior. Remember, laptops are stripped down to be as light as possible so their components are always sub-par. Metals replaced with plastics, less robust chip sets, tiny plastic moving parts in HDs and such, thinner wiring with higher resistance (more heat) etc.

Enjoy your day.
Back to top
Profile PM
Thebrianwing
15
Years of Service
User Offline
Joined: 15th Jun 2011
Location: United States
Posted: 27th Dec 2012 21:05
Link
I know there are idiots out there that frequently wind up in situations like having 32 bit Windows while having more than 4 GB RAM installed. The only reason I am posting here is because I am not one and have exhausted all of the common reasons for an error like this to occur. I am looking for someone who is acquainted with this particular white rabbit. So far, I haven't gotten any really helpful information or ideas except for the memory test which did not error out when using non-mipmapped images on a couple occasions, Morcilla; but finished with the exit prompt.

This being said, I ran a few tests I thought up last night and it does appear that I am reaching the 2 GB RAM limit when loading in all of my map images. I have found a way to work around this RAM cap for the time being but am hoping someone knows of a way to increase the amount of RAM that DBPro has to use. I will have to just load in a few hundred of the maps at a time for the finished product to allow for lower system requirements but I would like to avoid loading screens at least while I am still working and testing the alpha. Anyone know of a way?
Back to top
Profile PM Email
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 27th Dec 2012 22:42 Edited at: 27th Dec 2012 22:43
Link

Honestly, I think you are doing something wrong.

*** Merry Chuckmas! ***
Back to top
Profile PM
MrValentine
AGK Backer
15
Years of Service
User Offline
Joined: 5th Dec 2010
Playing: FFVII
Posted: 28th Dec 2012 00:22
Link
Dynamic Media/Dynamic Loading?

Back to top
Profile PM
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 28th Dec 2012 18:23
Link
@MrValentine
Yeah, DBP can load even 4096 dds on the fly with ease.

*** Merry Chuckmas! ***
Back to top
Profile PM
Morcilla
23
Years of Service
User Offline
Joined: 1st Dec 2002
Location: Spain
Posted: 29th Dec 2012 10:36
Link
I suggest to use .dds instead of .png
The test above uses .bmp, but the behaviour depends on the file format.

I had texture problems in the past and .dds seem to 'fit' much better into memory.

And forget about the error 506, it should always pop up, ajem.

For production, I use 'Divide Texture Size' parameter (for Load Image), only when it is detected that SYSTEM SMEM AVAILABLE is below a certain range.
This way, if a PC doesn't has enough memory, the program will start up, but with lower resolution images on the fly.

Back to top
Profile PM Email Website
Back to top

Login to post a reply

Server time is: 2026-07-07 10:59:10
Your offset time is: 2026-07-07 10:59:10