Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Professional Discussion / Tutorial to read in DBO format using DBPro and exporting to a format

Author
Message
Bored of the Rings
User Banned
Posted: 19th Nov 2022 16:11 Edited at: 25th Nov 2022 07:48
I'll be doing a tutorial here on how it's possible to read in DBO and convert to a format. Will be concentrating on obj, x and some other formats.
In additon, I will show how to write out to DBO.

This is a complex task, so you will need to.be able to use pointers and structures and even arrays and pointer to pointers as well as use the Win Api.
Furthers posts will be placed here.

All tutorial Part links below:

Part I - Setting up some initial variables and loading the DBO file into memory (pointer).
https://forum.thegamecreators.com/thread/228776#msg2672403

Part II - Adding the sObject structures and related structures, DX and other constants and reading in DWORDs/Embedded DBO codes + other functions required to find specific structure item pointers.:
https://forum.thegamecreators.com/thread/228776#msg2672426

Part III - Initial Construction of an Object and Object Frame
https://forum.thegamecreators.com/thread/228776#msg2672464
Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
LBFN
16
Years of Service
User Offline
Joined: 7th Apr 2007
Location: USA
Posted: 19th Nov 2022 21:08
Sounds like an interesting tutorial. I wonder, is it possible to convert .dbo to .ms3d format and still retain the animation / joint info?



So many games to code.....so little time.
Bored of the Rings
User Banned
Posted: 19th Nov 2022 21:25
yes definitely
Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
User Banned
Posted: 20th Nov 2022 08:53 Edited at: 17th Jan 2023 14:10
I originally wrote a program DBOFastReader which reads in DBO and exports static/animated objects out to DirectX and OBJ formats using DBPro commands and IanM Matrix1Utils.
This program / tutorial is more nearer to a VS C version of DBOLoader but with DBPro workarounds due to restrictions/limitations DBPro has.

DBO Format documentation can be found in your DBPro install folder under "DBO". In there you should be able to see a file DBOFormat.doc. This spec document, gives you a basic overview of how the structure works and how it can be loaded in. Under the same folder, you have the original DBO VS source code.

For the tutorials you will need Matrix1Utils written by IanM which allows us to use ALLOC and various Array Pointer commands, plus loads more.
NOTE: I have not used #include as when compiling with DBPRO and hitting errors does not show true line number , so once the program is complete / tested and works like a charm, then we can start adding include files to organise or our code better.

Part I - Setting up some initial variables and loading the DBO file into memory (pointer).

Initial Code:

As I get a bit more time on my hands, more and more breakdown info on what the code is doing will be added below the code snippet.

[code removed temporarily]

How does DBO work?
DBO works by embedding special codes that are represented in DWORD (LONG) format i.e. 4 bytes e.g. 1,0,0,0 = 1, 101,0,0,0 = 101, anything over 255 will carry over into the next byte.

How does the code work?
This 1st part of the code works by setting up some initial global variables, i.e. pointers to the DBO data and embedded codes and size. This is because DBPro does not understand "&" symbol used in VS C.
In DBPro we can allocate memory using memblocks, banks and ALLOC ( the last 2 you need IanM Matrix1 utils plugin).

Constant variables are initialised including a complete list of embedded DBO code 1 - 406 and some WIN32 API constants needed for the CreateFileA function. This gives us more control on the way a file is read in and is very fast. Originally , I used MAKE BANK FROM FILE and then obtained pointer from there.

To use the "CreateFileA" and "ReadFile" functions, we have to load in the "kernel32.dll" dll. Then obtain a handle to the CreateFileA function. Pointer variables are parsed into the ReadFile function.
dppBlock is used to allocate memory with size of the file obtained from *dpdwSize (i.e. return the size value contained in pointer dpdwSize). Print dpdwSize and Print *dpdwSize are 2 different things. The 1st returns the memory address and the 2nd returns the actual value stored in that address.

ppBlock is used as a "floating" address pointer to obtain the next DBO address, but is not set up as a pointer. It's purely a copy of the base pointer pDBOBlock which is static meaning the value will never change.

Anything that was setup as allocated memory is deallocated using FREE command at the end of the program (CleanUp function). I've added a message statement (for those that have Robert Knights BLUEGUI plugin) just to ensure that the cleanup works correctly without prematurely "bombing" out the program. For the ppBlock variable, it simply gets "NULLED" or set to zero.

In the future tutorial sections, there will be references to LPSTR, LPVOID and other LPxxx etc type vars , these are simply windows structures that really can be translated as DWORDs as they are all pointers to a particular windows data type.
Don't forget that all strings in DBPro are essentially DWORDs containing the address that points to the actual text of a string.

So if you have a fear of pointers and pointer to pointer, turn away now
Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
LBFN
16
Years of Service
User Offline
Joined: 7th Apr 2007
Location: USA
Posted: 21st Nov 2022 13:14
Had couple of errors. DBP didn't like the use of NULL as a constant and had trouble with the EXTRACT FILENAME$ and EXTRACT FILEEXT$ commands in the FileShortName function. I changed NULL to NOLL and remmed out the other two and it ran fine.



So many games to code.....so little time.

Attachments

Login to view attachments
Bored of the Rings
User Banned
Posted: 21st Nov 2022 13:32
@LBFN - thanks for taking the time to try the code. You need IanM Matrix1 plugin to use the EXTRACT FILENAME$ and EXTRACT FILEEXT$ commands, or I have my own version functions for those if you need them. It's weird that #constant NULL 0 doesn't work for you. Not sure why. Of course, change as you need to , as long as it works for you.
Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
LBFN
16
Years of Service
User Offline
Joined: 7th Apr 2007
Location: USA
Posted: 21st Nov 2022 15:00
Quote: "You need IanM Matrix1 plugin to use the EXTRACT FILENAME$ and EXTRACT FILEEXT$ commands"

I have the Matrix1 plugin. I have it in the plugins-user folder. I see 33 different .dll's with it. I tried moving it to the plugins-licensed folder, but it still doesn't like the extract filename / fileext commands.



So many games to code.....so little time.
Bored of the Rings
User Banned
Posted: 21st Nov 2022 15:12 Edited at: 21st Nov 2022 15:16
@LBFN - the Matrix1util plugin dlls should all go in plugins-user. Also, make sure you have c++ restribs for VS2010 and that dotnet 3.5 is installed under windows features

Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others

Attachments

Login to view attachments
Bored of the Rings
User Banned
Posted: 22nd Nov 2022 09:44 Edited at: 17th Jan 2023 14:11
Part II - Adding the sObject structures and related structures, DX and other constants and reading in DWORDs/Embedded DBO codes + other functions required to find specific structure item pointers. This is complex.

[code removed temporarily]

What's Been Added?

1. Initialisation of Windows related Constants
2. DirectX related Structures
3. Added DBO related structs (only sObject structure for now)
4. Added Functions: ConstructObject, ReadString, ReadDWORD, ReadCODE, Stricmp (compare string), memcpy, GetFieldType$, GetStructItemCount, GetStructPtrs(ptr, FindStructItem

What do the functions do?

ConstructObject
parses in the ppObject pointer and current ppBlock pointer (incrementing or floating pointer to the next address or code if you like). Reads in header info (the first 24 bytes) and then sets up an array of sObject type that will hold our object data. without that you can't save out a dbo structure.

ReadDWORD
As it suggests , reads in a DWORD value at ppBlock address and stores it in ppDest which points back to dwVersion etc which are set up as ALLOC'd memory pointers of size 4 i.e. a DWORD to hold an address. It then increments the ppBlock pointer by 4 bytes

ReadCODE
As it suggests , reads in an embedded DBO code and the Code Size. It then increments the ppBlock pointer by 4 bytes for both dwCode and dwCodeSize. For all possible embedded codes see the DBO related #contants 1-406

Stricmp
Compares 2 strings and returns 0 if they match. A sort of quick version of the C function stricmp.

memcpy
Copies a number of bytes (dwLength) into a pointer (which has previously been ALLOC'd).

GetFieldType$
Returns an array item structure as text e.g. DWORD = D, String = S etc. Called within my GetStructPtrs function. See code comments for full list

GetStructItemCount
Returns the number of items (or fields) within an array

GetStructPtrs - removed in Tutorial III as it was found not necessary and link array / unlink array quite slow
Reads all array items (fields) within an array UDT by parsing in an array pointer which is linked to a temporary array of same structure type and details about the item are stored in a pointer list e.g. ptrlist() as ptrlist_t

FindStructItem
Returns an address of an array item by parsing in the address pointer variable and number of the item. I've added the item number in the structures using []. This is because we have no way of using ppObject->pFrame type methods.
Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
User Banned
Posted: 22nd Nov 2022 09:50
all tutorial Part links will be posted to 1st post to keep it all together in one place for convenience and so you don't get "lost down a rabbit hole"
Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Mage
17
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 24th Nov 2022 23:21
Great work so far, eagerly awaiting any info on writing to DBO.

Mage's Modular Code - Get awesome UI controls and powerful Bitmap Fonts!
Screen Device Recovery - Stop your apps from crashing when minimized/tabbed/screen locked.
CPU Friendly Frame Limiter - Make your app generate less computer heat + noise, and use less battery life!
Bored of the Rings
User Banned
Posted: 25th Nov 2022 05:17 Edited at: 25th Nov 2022 07:49
@Mage - many thanks. - I will come onto saving to DBO , the main functions will be:

SaveDBO ->
->DBOConvertObjectToBlock ->
-> ScanObject
->DBOSaveBlockFile

The good thing is we have the size of the DBO already and so we would create a block of memory of size pdwBlockSize , I would be using ALLOC although you can use what you like in your own programs whether it be memblocks or banks.

We need a few WriteXX functions e.g. WriteCR, WriteSpeechMark, WriteComma. These are pretty simple.

Top of my memory, we would write the following:

1. Write header info - 24 bytes
2. Write the root frame and size of the root block
3. Write each embedded code along with size and then the data i.e. dwCode, dwCodeSize, pData - pData could be Frame Name or Translation Matrix etc
4. Write the Animationset Root and size of the block ,and then the data ( if Animation exists)
5. Write any custom data dwCode = 406 and size, then the data in bytes (if custom data exists)

So there are 3 main blocks, Root, Animation and Custom Data.

I will get onto all this as soon as I can.
Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
User Banned
Posted: 25th Nov 2022 05:40 Edited at: 17th Jan 2023 14:12
Part III - Initial Construction of an Object and Object Frame

Updated Code (fixes/updates)

[code removed temporarily]

What's going on
1. Added 2 constant vars: sizeof_D3DXMATRIX 64 and sizeof_D3DXVECTOR3
2. Added structure 'charofbytes' to copy string content
3. Added sFrame structure and related structures
4. removed ptrlist_t structure
5. added 4 global vars to capture embedded code / sizes
6. added array ppFrame of sFrame structure type
7. removed GetStructPtrs function and sObjectPtrList array
8. added Case .. Select conditions for Frame name, transform, mesh, child and sibling frames, offset, scale, rotation
9. added ReadMatrix / ReadVector functions
10. updated ReadCode function
11. updated FindStructItem function
12. added FindStructItemType function
13. updated CleanUp function
Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
User Banned
Posted: 27th Nov 2022 07:42
Tutorial 3 needs a little update. For now i have just commented out the call to ReadString within the ConstructFrame function. Tons to come, for those that want to write to DBO i will start to make a separate tutorial so it doesn't gets mixed up with this one, will see how it all works out. It's just basically getting the info stored in Object structures and converting to DBO , poking lots of bytes etc.
Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
User Banned
Posted: 30th Nov 2022 06:10 Edited at: 30th Nov 2022 06:10
More updates to come for tutorial 3 as we need to be able to expand the frame pointer list with each "array insert ...." and return/access each structure item value i.e. make the array dynamic. So the object is to e.g. find a DBO frame, create a frame object, copy the frame contents/data to the frame array structure. This will allow us to access the content when exporting to a required format or save back out in DBO format if we wanted to.
Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
User Banned
Posted: 4th Dec 2022 11:40
updates to Tutorial 3 to come again, which will include all frame related data i.e. Frame Name (strings are a little tricky as DBPro doesn't support char [max_size] type structures, Frame Transform Matrix, Frame Mesh Frame Child; Sibling; Offset; Rotation and Scale. Determining which way to go with strings. In my DBOFastReader, code is a lot different to C code, but if we want to save to DBO correctly, the format has to be saved correctly based on what's been loaded at what pointer location/sizes/data etc.

Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
User Banned
Posted: 4th Dec 2022 15:51
strings issue sorted, decided which direction to go in. Not the way I wanted to go as such but fits perfectly and nearest to C code, so will be using a type 'charasbytes' type which defines all "MAX_STRING" ->256 byte vars, but you don't have to 'poke' each one manually it's done with memcpy function and using a temporary ALLOC memory pointer variable. Tutorial 3 to be updated hopefully tonight.
Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
User Banned
Posted: 4th Dec 2022 22:50
Part III code now updated. Breakdown of what's going on, what's been added/removed etc will be added also.
Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
User Banned
Posted: 5th Dec 2022 06:57
Tutorial iv will be adding the sMesh structure, setting up an array to hold the mesh data and adding in the mesh construction object function.
Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
User Banned
Posted: 16th Dec 2022 12:09
A delay with the Tutorials for this side of things as DBPro is having issue with some structure defs, looking into. May have to update structures , will see what the underlying issue is first.
Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
User Banned
Posted: 17th Dec 2022 05:56
this issue was down to :
pVertexDeclaration as t_charfivetwozerobytes // [17-(536)] D3DVERTEXELEMENT9 []for custom FVFs [ MAX_FVF_DECL_SIZE ]

now changed to :
pVertexDeclaration as D3DVERTEXELEMENT9 // [17-22]

this is always set to a dwCodeSize of 4 and pData is always 0 from what I've seen in my experience with DBO , so luckily it's not going to affect anything, when loading/saving out your own formats

D3DVERTEXELEMENT9 type is defined as:

type D3DVERTEXELEMENT9
Stream as WORD
Offset as WORD
Typ as BYTE
Method as BYTE
Usage as BYTE
UsageIndex as BYTE
endtype

and the limit is 65 * D3DVERTEXELEMENT9

It looks like a limitation of DBPro and types as I can compile fine in another project and use the same type as before i.e. pVertexDeclaration as t_charfivetwozerobytes. This compiles fine, but not when used with a larger list of types within types.

In my DBOFastReader program, this CODE i.e. 120 (DBOBLOCK_MESH_VERTEXDEC), is not used, but my guess is if you want to load/save to DBO in your fav model program, we need all codes for completion as the model program will probably look for it (maybe). So I added this DBO code 120 (DBOBLOCK_MESH_VERTEXDEC) into this program.

Still lots to do on both loading/saving DBO tuts and programming. it's working so far until I hit this issue which has now been worked around.




Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
User Banned
Posted: 19th Dec 2022 18:17
To add from last post, the pVertexDeclaration is only used when FVF = 0 i.e. DBOs such as GameGuru Max where the FVF size is larger than your standard e.g. FPSC classic / Dark Matter type DBOs. I'm looking into updating my DBOFastReader tool to now include this, although I handle GG Max DBO's in a different way.
Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
User Banned
Posted: 20th Dec 2022 21:58 Edited at: 20th Dec 2022 22:17
another little error crept in:

"bBoundProduceResult boolean" should be changed to "bBoundProduceResult as boolean".
This shifts the array item positions, which I will post updates soon(ish), hopefully. Reworking the TYPEs.

[update-ouch, another error : "bUniverseVisibleas boolean" should be changed to "bUniverseVisibleas as boolean"]
Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
User Banned
Posted: 22nd Dec 2022 17:21
will try and post more updates tonight/tomorrow.
Pro Programmer / Data Scientist, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Mage
17
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 26th Dec 2022 07:08
Interesting read. Haven't been about to follow too deeply due to time constraints as of yet.

Mage's Modular Code - Get awesome UI controls and powerful Bitmap Fonts!
Screen Device Recovery - Stop your apps from crashing when minimized/tabbed/screen locked.
CPU Friendly Frame Limiter - Make your app generate less computer heat + noise, and use less battery life!

Login to post a reply

Server time is: 2024-03-28 18:14:05
Your offset time is: 2024-03-28 18:14:05