Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Professional Discussion / Optimizing loading

Author
Message
Cescano
11
Years of Service
User Offline
Joined: 4th Sep 2014
Location: Italy
Posted: 2nd Jan 2015 19:41
The start loading could take some time if you need to load a lot of stuff, why don't make a post writing everything that could help the loading time? I start by writing the few things i know, Others may continue:

- Use .dbo files for objects or .x files for bones animated objects

- Make instances or clones for objects instead of loading them multiple times

- Use .wav files instead of compressed audio files such as .mp3 for sound effects

- For music files such as .mp3 convert it to a lower quality if possible (not too much) (maybe let the player chose from the settings)

- Don't use huge images with more than 4096 pixels on 1 side (2048 on older computers will slow down too)
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 2nd Jan 2015 20:49
Use DDS images instead of PNG, BMP, JPG, etc.



Powered by Free Banners
Cescano
11
Years of Service
User Offline
Joined: 4th Sep 2014
Location: Italy
Posted: 2nd Jan 2015 21:07
Do DDS keep transparency such as png and bmp?
Also there is a difference in video memory taken by the images using one or the other?
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 2nd Jan 2015 22:46
Quote: "Do DDS keep transparency such as png and bmp?"


Yes - and there are many different pixel formats supported by DDS files.

Quote: "Also there is a difference in video memory taken by the images using one or the other?"


Yes and no - all non DDS images get converted to the same format, RGB or ARGB. I believe that DDS files retain their format because they use native DirectX formats so compressed DDS formats remain compressed in memory, i.e. no conversion to a non-compressed RGB format takes place. That's my understanding of how DBPro works anyway.



Powered by Free Banners
Rudolpho
20
Years of Service
User Offline
Joined: 28th Dec 2005
Location: Sweden
Posted: 2nd Jan 2015 22:49
Quote: "Do DDS keep transparency such as png and bmp?"

Yes, DDS pretty much corresponds to the image format used internally by DirectX (the acronym is for DirectDrawSurface). As such it is faster to load because it won't have to be converted to the DDS format behind the scenes as happens when you load another image type (this is the same reason as to why it is faster to load a DBO object than import some other 3D file format).

Quote: "Also there is a difference in video memory taken by the images using one or the other?"

As said they will be maintained internally as DDS images anyway once loaded. Using a source DDS file lets you control certain settings that DBPro will otherwise make for you during the conversion however, such as how many mip levels to use, colour formats etc.

MrValentine
AGK Backer
15
Years of Service
User Offline
Joined: 5th Dec 2010
Playing: FFVII
Posted: 3rd Jan 2015 02:33
If you do not have any software to create .DDS files, Using DBPro simply load a texture image and save it out with the .dds extension in the file name...

- Try to reduce polygon data on loaded objects
- Try to use objects with a single texture image for the entire object

Cescano
11
Years of Service
User Offline
Joined: 4th Sep 2014
Location: Italy
Posted: 3rd Jan 2015 02:46 Edited at: 3rd Jan 2015 02:57
I am using the plugin in Photoshop to save them as .DDS, the only problem is that these texture are a pain in the ass, i mean, when i try to save them, i can only save them if the width and height of the texture is the same number, and also it only accept multipy of 2 (so for example 64x64, 128x128, 256x256 are ok, but not for example 256x512 or 640x640 or 600x600).

Edit 1: Also i do not know exactly which settings to use when exporting from Photoshop, i am using (DXT1 rgb 4bpp no Alpha) for normal textures and (DXT1 Argb 4bpp 1 bit Alpha) for the transparency ones.
For all of them i select "volume texture" and "generate mipmaps"

Edit 2: I noticed also that i have problems when trying to convert .bmp textures to .dds textures (transparent ones) when the .bmp texture is using 24 bit of depth, no problem for the ones with 32 bit of depth. I used paint to manage some textures so the max depth for .bmp was 24 bit.
When i try to convert a 24 bit .bmp to a dds (i used these .bmp to make static shadows), the shadow is just a normal square plain, instead for example having all the shadow details for that object, the weird thing is that if I open these textures they seems to be fine, and the transparency part is not lost (rgb is still 0,0,0).
By the way there is a difference with the preview of these textures looking for them opening the "texture folder" in Windows, the 32 bit ones (which works fine) have the transparency part White on the preview, while on the 24 bit ones (not working fine) have that transparency part black
MrValentine
AGK Backer
15
Years of Service
User Offline
Joined: 5th Dec 2010
Playing: FFVII
Posted: 3rd Jan 2015 03:23
Quote: "If you do not have any software to create .DDS files, Using DBPro simply load a texture image and save it out with the .dds extension in the file name..."


No need for 1x1 scale...

Cescano
11
Years of Service
User Offline
Joined: 4th Sep 2014
Location: Italy
Posted: 3rd Jan 2015 04:48 Edited at: 3rd Jan 2015 06:45
Changing all the textures i have from jpg, png, bmp to dds drastically lowered the loading time, exactly from around 32-33 seconds down to 4.5 seconds, which is great considering all the data I am loading (considering also i have 7-8 mp3 loaded which is not a fast loading format, for a total of around 53 MB of music).

The only problem is that i used some textures with weird sizes such as 128x512, or something like that, so that when i converted them to .dds i had to change the texture size to 512x512 which stretched the texture making it a bit bad looking, there isn't a way to use a .dds texture with 1 side longer than the other? Why there is this limit? the multiply of 2 is already an annoying limit, plus this one is a bit Tricky deal with this format

Edit: The only way i managed to use .dds textures with different width and height is by converting them inside dbpro, the only problem i see is that even if the size is smaller (keeping 128x512 for example instead of converting it to 512x512), the final image is WAY bigger than the one stretched to 512x512, something like 680 kb instead of 170 kb, why this happens? I ve even tried the 5 different compressions methods in the save image command, but i still get 681 KB of image regardless the number i chose
Rudolpho
20
Years of Service
User Offline
Joined: 28th Dec 2005
Location: Sweden
Posted: 3rd Jan 2015 11:52
Quote: "there isn't a way to use a .dds texture with 1 side longer than the other? Why there is this limit? the multiply of 2 is already an annoying limit, plus this one is a bit Tricky deal with this format"

That should be possible, you probably have some legacy setting turned on in your Photoshop plugin, look around for it.
The reason is that older graphics cards could only work with textures whose dimensions were an even power of two; even newer cards tend to handle such textures more efficiently (though on newer cards still this probably doesn't matter at all anymore). Still so, you should be fine as long as both sides are a power of 2, they shouldn't need to be the same as each other.

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 3rd Jan 2015 13:12
Quote: "For all of them i select "volume texture" and "generate mipmaps""


That doesn't sound right. What are the choices other than "volume texture"?

Quote: "The only way i managed to use .dds textures with different width and height is by converting them inside dbpro, the only problem i see is that even if the size is smaller (keeping 128x512 for example instead of converting it to 512x512), the final image is WAY bigger than the one stretched to 512x512, something like 680 kb instead of 170 kb, why this happens?"


Because you did something wrong perhaps? I've just tested this with a sample PNG 128x512 image (attached) and the resulting DDS file is 33KB compared to the original 177KB. Here's the code I used:



Just read the Help file for those two commands.



Powered by Free Banners
Cescano
11
Years of Service
User Offline
Joined: 4th Sep 2014
Location: Italy
Posted: 3rd Jan 2015 14:06 Edited at: 3rd Jan 2015 14:18
I was using the same commands without the 1 at the end of both (i tried to add to the save image also but not on the load image)

By the way the other 2 choices other than "volume texture" are "2d texture" and "cubemap"

Edit:

I have tried again to do save image and load image with the same code you use, and i still have a 681 KB file, seems also that i am not able to see the width and height or depth looking on the file properties, while i am able to see it looking for a .dds file saved with the Photoshop plugin, also the ones saved with dbpro does not have a preview looking on the texture folder, while the ones saved with Photoshop have the preview (the icon)

Edit 2:

Using "2d texture" let me have different sizes for the textures (finally!!!), the only limit is now have width and height as a multiple of 4, which could be acceptable
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 3rd Jan 2015 16:14
Yes, option "2d texture" would be the one - as you've discovered.

Quote: "I have tried again to do save image and load image with the same code you use, and i still have a 681 KB file"


Can you create a simple project with just those 2 lines and a typical sample 128x512 image, zip it up and upload it here? I just don't see why you're having this problem.



Powered by Free Banners
Ortu
DBPro Master
18
Years of Service
User Offline
Joined: 21st Nov 2007
Location: Austin, TX
Posted: 4th Jan 2015 00:22
The directx SDK has an excellent tool for converting images to dds.

Instead of just scaling an image to ^2 you can enlarge the canvas with unused space. May need to adjust UV, but generally it's better to just work on a square ^2 image space to start with

Cescano
11
Years of Service
User Offline
Joined: 4th Sep 2014
Location: Italy
Posted: 4th Jan 2015 00:39 Edited at: 4th Jan 2015 00:52
Gandalf, i tried with an image downloaded from the web 128x512 and the size of the dds was normal, almost the same as the png or the jpg downloaded, but when i try to one of my images a 74 KB jpg image will be converted into a 682 KB dds file, which is huge compared to the source one, I attach here the 2 images, jpg and dds.

For the code in dbpro i have used exactly the load image and save image as you wrote previously.

For Ortu, from now i will surely start working with a same width and height texture if possible, I was worried about the resizing of the images because of some stretching and even more worried about the UV and UVW maps i had on some models, which have taken hours and hours of work.

By the way the problem seems now solved as setting on the plugin the "2d texture" instead of "volume texture" fixed what I needed to do, so now I can use 128x512 textures for example.

We could continue the post talking about Optimizing the loading so, for the moment I am very happy for what I achieved by converting the textures into the .dds format, I really didn't know this, I started from 33 seconds of loading and now my world loads in just 4.5 seconds, obviously I still need to add several things but i still have some Optimizing to do yet, 4.5 seconds of loading for a 14k lines project is great considering all the stuff I load, some textures are even big as 2048x2048, 1 texture I use for the sky is 4096 pixel on 1 side, so when I will set the option to decide to load textures in low, mid and high quality, the upload time could be even better for lower resolutions.
I want to say also that I am working on a notebook, so definitely not the best for fast loading as I don't have a SSD
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 4th Jan 2015 00:54
That image is 758x230 which is much larger than 128x512. That also means that it can't be converted to a compressed DXT1 format which is probably why DBPro converted it to the default RGB format.

The dimensions must be multiples of 4 for DXT1 compression. Stick to multiples of 4 and you should be OK.

I rescaled your image to 756x228 pixels and then converted to DDS. See attached files. Hope this helps.



Powered by Free Banners
Cescano
11
Years of Service
User Offline
Joined: 4th Sep 2014
Location: Italy
Posted: 4th Jan 2015 01:07
I understand now, by the way now i prefer using the Photoshop plugin to convert the images instead of using dbpro, so I can adjust it without the risk of having a huge dds file
Cescano
11
Years of Service
User Offline
Joined: 4th Sep 2014
Location: Italy
Posted: 4th Jan 2015 05:27
Only 1 thing is not clear, on the Photoshop plugin, even if I chose dxt1 Alpha (argb) or dxt1 no Alpha (rgb) the size of the image is the same, which differences will they have then when loaded in memory? I mean, i know the Alpha is for the transparency, but if the image is the same size I can just keep the Alpha as a standard setup even for normal textures? So that I don't risk to forget to turn the Alpha argb when i need transparency? (the plugin Always keep the last save)
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 4th Jan 2015 13:41 Edited at: 4th Jan 2015 13:42
Quote: "which differences will they have then when loaded in memory?"


Good question.

Yes, I would use the version with alpha for the reason you give. Understanding the full details of the DXT1 and related formats is indeed an arcane art. Here's a relevant extract from the Microsoft documentation:

Quote: "Opaque and 1-Bit Alpha Textures (Direct3D 9)
Texture format DXT1 is for textures that are opaque or have a single transparent color.

For each opaque or 1-bit alpha block, two 16-bit values (RGB 5:6:5 format) and a 4x4 bitmap with 2 bits per pixel are stored. This totals 64 bits for 16 texels, or four bits per texel. In the block bitmap, there are 2 bits per texel to select between the four colors, two of which are stored in the encoded data. The other two colors are derived from these stored colors by linear interpolation. This layout is shown in the following diagram."




Quote: "The 1-bit alpha format is distinguished from the opaque format by comparing the two 16-bit color values stored in the block. They are treated as unsigned integers. If the first color is greater than the second, it implies that only opaque texels are defined. This means four colors are used to represent the texels. In four-color encoding, there are two derived colors and all four colors are equally distributed in RGB color space. This format is analogous to RGB 5:6:5 format. Otherwise, for 1-bit alpha transparency, three colors are used and the fourth is reserved to represent transparent texels.

In three-color encoding, there is one derived color and the fourth 2-bit code is reserved to indicate a transparent texel (alpha information). This format is analogous to RGBA 5:5:5:1, where the final bit is used for encoding the alpha mask."


Hope that clears things up for you.



Powered by Free Banners
Cescano
11
Years of Service
User Offline
Joined: 4th Sep 2014
Location: Italy
Posted: 4th Jan 2015 18:10
Yes I already looked at that, but did not understand well, in which case is usefull save the .dds files with the DXT1 no Alpha if even saving it using DXT1 Alpha takes the same amount of memory?
From what I can read i understand that in the Alpha mode the 4th pixel is used for transparency, so it takes the same memory as no Alpha BUT the image has a lower quality because use 3 pixels out of 4 to save the image data instead of 4 out of 4? I don't know if I get it right...
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 4th Jan 2015 18:20
I guess that's right - but I wouldn't worry about it unless you're having problems with quality.



Powered by Free Banners
Jeff Miller
21
Years of Service
User Offline
Joined: 22nd Mar 2005
Location: New Jersey, USA
Posted: 4th Jan 2015 18:37
Back to your original question on conserving loading time, I recall that a previously active forum member (Cash Curtis) was working on ways to reduce time spent in loading textures and reported the following. He had several objects that used the same texture, and as each object was loaded DBP would load the very same texture again and again. I think his solution was to save out the objects without the reference to the texture, load them in bare, load the texture image ONCE, and apply the texture to the various objects within DBP.
MrValentine
AGK Backer
15
Years of Service
User Offline
Joined: 5th Dec 2010
Playing: FFVII
Posted: 4th Jan 2015 18:41 Edited at: 4th Jan 2015 18:42
Jeff, you need to watch my video

https://www.deepredonline.com/vabblog/videoon.html

EDIT

Jump to 4 minutes and play...

Cescano
11
Years of Service
User Offline
Joined: 4th Sep 2014
Location: Italy
Posted: 4th Jan 2015 18:50
Yes Jeff I think that is usefull too, I ve never used dbo files with textures by the way, I ve Always used to load them within dbpro and then texture the objects, also because I make models with 3ds max saving them in .x and then converting to .dbo, so the textures are lost already when i save them in .x
Cescano
11
Years of Service
User Offline
Joined: 4th Sep 2014
Location: Italy
Posted: 8th Jan 2015 00:06
I want to correct my 1st post, when talking about this:

- For music files such as .mp3 convert it to a lower quality if possible (not too much) (maybe let the player chose from the settings)

After making some tests, loading a 8 MB mp3 320 kbps or converting it to a 3 MB mp3 128 kbps, takes aprox the same time to load, I don't know why, but maybe someonelse more expert than me could explain it.

Loading it as .wav will takes a bit more than 1/3 to load, but for music files it's an unduable thing as will be too big (it was 36 MB in my test, and I didn't chose the best quality)

On my computer, loading 100 times a 3:36 mp3 song takes around 3 seconds, so i think keeping music files as mp3 is still a better choice, even if you have 30 musics to load on your game 1 second of loading won't hurt the loading that much, while the huge size of the songs in .wav will hurt a lot the game size (36 MB x 30 songs that's around 1GB of music instead of 240 MB for the 320 kbps or 90 MB for the 128 kbps)
Rudolpho
20
Years of Service
User Offline
Joined: 28th Dec 2005
Location: Sweden
Posted: 8th Jan 2015 00:56
Quote: "On my computer, loading 100 times a 3:36 mp3 song takes around 3 seconds, so i think keeping music files as mp3 is still a better choice, even if you have 30 musics to load on your game 1 second of loading won't hurt the loading that much"

You probably don't need to preload your music; load that as needed instead (and unload the old music files at the same time to save on memory usage).
For example, you are going to your title screen - load the title theme and unload anything else; next you are going to some intro cutscene - you unload the title theme and load your background tune for that one, and so on.

Cescano
11
Years of Service
User Offline
Joined: 4th Sep 2014
Location: Italy
Posted: 8th Jan 2015 01:19
Ok maybe I will do that then, I don't know why I didn't do that, probably when i started to load it i thought it would take too much time loading it in the meanwhile to have a smooth gaming, but after doing the test 0.03 seconds to load a music will be unnoticeable, better save RAM as you suggested
Mage
Valued Member
19
Years of Service
User Offline
Joined: 3rd Feb 2007
Location:
Posted: 23rd Jan 2015 22:22 Edited at: 23rd Jan 2015 23:36
Hi everyone! I haven't posted in a while...

1. Don't tie your loading to framerate.

If you make an animated loading screen, don't sync after each and every item is loaded. Otherwise loading is capped at frames per second.

Keep track of the Timer() and use an If statement to skip SYNC if too little time has passed.

This can be a massive load time improvement.



I normally load about 1000 objects into a level. Many of these objects are supremely quick to load and many times the object list has empty sections in it too. I was using SYNC after each. I made this tiny little change and cut 95% of the load time, minutes to mere seconds.

2. Proper Asset Management

The art of skipping what you don't need to load, and doing it automatically.

Quote: "Jeff Miller: Back to your original question on conserving loading time, I recall that a previously active forum member (Cash Curtis) was working on ways to reduce time spent in loading textures and reported the following. He had several objects that used the same texture, and as each object was loaded DBP would load the very same texture again and again. I think his solution was to save out the objects without the reference to the texture, load them in bare, load the texture image ONCE, and apply the texture to the various objects within DBP. "


Yes this is where proper planning and large scale programming structure come into play.

I always load objects and texture separately. I understand that some people encode texture paths into objects and get the textures to load automatically when loading the object. I never do this. Less control. Less versatility. Harder to read.

Dark Basic Professional doesn't really give you good management tools for detecting whether an image has been loaded, let alone what the image number would be.

So you really need to make a rudimentary game engine for yourself -so to speak-. Write wrapper functions for the Load Image, Delete Image commands (and possibly for object commands too) and create a management system yourself. Generalized functionality to enhance DBP that could be dropped into any program.

Store an automated list of loaded assets, their path names, their indexes, and a counter for usage tracking. I mean create functions that are wrappers for Load Object and Load Image. You call these functions instead of directly using the commands. These functions are then able to essentially intercept your load request. They can return an object/image number that they find already loaded from a master list or they can load your image/object and store the details like path name and index so the loaded asset is "on record". Then increment that assets counter to track how many active uses there are.

Of course then a wrapper for delete object/image would delete the image/object and remove it from the master record of loaded assets. I mentioned storing a master list of assets, with a usage counter. This is because if you are deleting an image and the counter reaches 0, you know nothing else is using the image and that it can actually be deleted from memory.

You can create specialty functions for ensuring you load an asset that isn't shared if need be. It's your management system.

You also don't decide what an image/object number will be. Let your management system find and select unused ones on it's own and handle that business for you. When you call the wrapper function to load an image it returns an image number for you to use. All the complicated bits are handled by the automated system.

A system like this also allows built-in error protection. Your management system can check for problems. If you supply a bad file path for example, loading an image can then return a 0 index. You can then handle the error rather than the entire program crashing to desktop.

All in all I have found that this kind of asset management is most effective with images. Objects aren't quite as able to be shared around like images and often need to be separately loaded.

I could go into more detail, there's some more advanced stuff you can do with this and describing all of this with so few words is tough. Suffice to say I was able to reduce my memory footprint by about 75% which is a lot when you are approaching 2GB.

3. Proper Asset Management Leads To Screen Loss Recovery Capability.

Because you are now completely managing your loaded assets, when the screen device is lost, and all your assets are flushed from memory, you can easily load them back into memory in an automated process.

This is essential for a large scale program where hundreds of objects may be active in the game.

You will need to also look into how to detect and recover from screen loss issues to do this.

Screen Loss Recovery Examples Here:
http://forum.thegamecreators.com/?m=forum_view&t=202959&b=1

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 24th Jan 2015 01:31
Good suggestion.



Powered by Free Banners
Mage
Valued Member
19
Years of Service
User Offline
Joined: 3rd Feb 2007
Location:
Posted: 24th Jan 2015 02:39
Well I don't post often, gotta make it count.

Mage
Valued Member
19
Years of Service
User Offline
Joined: 3rd Feb 2007
Location:
Posted: 24th Jan 2015 08:53
I just realized some of you are talking about using DDS files since they load faster than BMP and slightly faster than PNG.

Don't use DXT1. It's lossy compression. Use DXT4 for textures as it will maintain lossless texture quality and 8bit alpha. Use DXT5 for normal maps since it's designed to make normal maps look less blocky with smoother gradients. It also supports 8bit alpha which some games use as a specular channel instead of a separate image.

You can include mip maps into your textures since they will be better than the ones DB Pro generates, and loading them is faster than DB Pro generating them at the expense of adding 1/3 more file size.

I suspect some of you are using DXT 1 not knowing you are degrading your textures.

Cescano
11
Years of Service
User Offline
Joined: 4th Sep 2014
Location: Italy
Posted: 24th Jan 2015 13:22
I do use DXT 1 for most of my textures, I did not notice any different actually comparing them to the pmg,jpg,bmp i was using before, so as long as they use less memory than dxt4 will be the best choice for the game performance

Login to post a reply

Server time is: 2026-07-05 10:49:32
Your offset time is: 2026-07-05 10:49:32