Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Professional Discussion / Converting to/from DBO format - Progress

Author
Message
Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 3rd Oct 2018 08:45 Edited at: 3rd Oct 2018 08:49
I have been working for a few months very sporadically (95% of it the last 5 days) on converting DBO between different formats byte by byte. My end goal is I want models made in my model editor of choice to be converted to DBO with no jank or missing functionality (like animations or vertex weights). These are longstanding pains people have complained about on these forums.

The first step was to decompile the editor's file format. That was fairly easy. The second step was to decompile DBO file format. That was like 100 times harder. I managed to get this done tonight.

A few things I noticed about DBO format:

Documentation is vague. Thank goodness the source code for DBP is available, and I was able to use that to fill in the knowledge gaps when documentation left information out. This was needed a lot.

DBO files have a few unexplained sections which the documentation promises can be skipped. (Supposedly things like terrain data)

One of the sections of DBO files is called "Arbitrary Value" and the value is an arbitrary number.

The file format stores everything in a sort of optimized manner making it difficult to sort out. Most of the information is stored nested in frames, linked lists and data sets. Contrast that to my editor's file format which lays everything out like dumping an array, much simpler.

Now the work turns to doing actual conversions of the data. Does anyone know anything about Matrices or Frame Matrix or Animation Matrix? There's some university level math stuff I am I afraid might bog me down if I need to start recalculating stuff. Documentation says nothing here.

I'd ask more questions but I haven't run into the really nasty stuff yet.

Bored of the Rings
13
Years of Service
User Offline
Joined: 25th Feb 2005
Location: Middle Earth
Posted: 3rd Oct 2018 09:38 Edited at: 3rd Oct 2018 13:52
I've been reworking my DBO2X (and OBJ) converter for skinned and non skinned meshes (which also reads in GameGuru EBE and PBR enhanced DBO's which programs like Fragmotion can't read in due to the FVF. Yes, the documentation is dreadful, but got it all working (took almost 1.5 years to get right). Here is a small code snippet of the enhanced code (draft currently) which now uses as close to C code as possible using pointers / memory (IanM's Matrix Utills ALLOC or bank). Alloc is slower and banks a lot faster. The code doesn't show all the embedded DBO codes at the moment as it's a reWIP and I'm still doing more tests / debugging.

I haven't used any special mathematics (and I'm a stats programmer), just obtained the matrix / animation data (if any) and formatted to X specific format. Although, I need to figure out bone weight recalculations for GameGuru so they don't come out disfigured in some cases.

Code has also been added to the Code Snippets thread.



The attached debug output was produced using the "hummer_dynamic" DBO found in the Desert Storm FPSC pack (pack 75 I believe).
This DBO contains multi-materials and textures and you actually have to write the code to generate the X format to handle multi mats / textures correctly.
My DBO2X (v1.4) can be found here if you want to try it --> https://forum.game-guru.com/thread/219806#msg2602285 OR here for direct download from my GMAIL account https://drive.google.com/open?id=1QAwOkqgdSmQJqMYR76XlmgOCN-4cfpAU
Professional Programmer, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others

Attachments

Login to view attachments
Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 3rd Oct 2018 20:31 Edited at: 3rd Oct 2018 20:36
Thanks this and the 2005 spec sheet were a big help.

Well you certainly know this mountain well. Part of the problem with the documentation is it doesn't describe big picture how the model is arranged. It mentions child and sibling frames without explaining the difference between them. So when you see a bunch of empty frames there's a question of why that's there. Maybe important or maybe the existing converter I have been using tossed in a bunch of junk data. FVF is simple enough to load in, but the data isn't explained well. Docs mention for example FVF_XYZ which seems to be vertex x/y/z but then there's also FVF_XYZRHW with a description "float X, Y, Z transformed" which is not much to go on. Right handed? Or this one from the FVF section "FVF_PSIZE float point size for sprites".

One of the things saving me a little though is that I am constrained by the editor's format which doesn't use all of these things. I am a little doubtful about multi-material mesh support. What is that really? How is that different than using the 8 or so textures the FVF section can store? Knowing this would give me a better indication of whether or not I can even do anything with the feature.

Another thing is this style of using buffers/pointers/and blocks of data. It is quite impressive and as far as DBP is concerned you need the engine running as fast as possible. However for my purposes taking 30 seconds instead of 5 isn't worth much. I have focused more on reading the byte and storing it in an appropriate variable. Now when I start to do conversions this might bite me in the ass or it might make certain things easier. For example if I need to return multiple variables from a function, then a fancy packed data block is easier to return. So we will see how that plays out and if I have problems.

Additionally I had an idea that if your converter has a command line interface, I could put it in the same folder as a tool I made and interface them, and the reverse also. That would let someone bridge conversion capabilities of both tools in one action rather than needing to manually launch both. Not sure if you're on the same page with something like that. Seems like a good idea.
Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 6th Oct 2018 04:42 Edited at: 6th Oct 2018 04:43
I managed to convert the entire 3D mesh of a character from DBO to my editor.
It was hard because documents give variables names and tell you how to read them from a file but not how to use them. So a number of questions had to be investigated and also there were bugs in the loading code that allowed the model to be loaded slightly incorrectly but not enough to initially reveal a problem until conversions took place. (Like do vertex indexes start at 0 or 1).

- I was surprised that so far the coordinates and u/v data didn't need any serious math, just complicated repacking.

- DBO format seems to store U/V coordinates for each vertex, while my editor stores U/V data with each polygon (3 sets for 3 vertices). This seems to cause models to become unwelded when converted to DBO. More investigation is needed. Imagine if you had a sphere and your texture was the front and back side by side, those edge vertices can only have one u/v coordinate so the only solution would be to duplicate the vertex. Seems like this might be at play here.

My next step is to export the bone data which then leads to exporting the animation data. I am seeing some serious convoluted mess with the bone structure in the DBO. I'll need to really take a hard look. I have noticed that the whole DBO is broken into sections called frames, followed by animation data section. The frames are named after the bones and are in a hierarchy. The first frame contains all of the mesh data. The mesh data contains data for most but not all of the bones. Why? I don't know yet. So there's no obvious way to correctly extract the bone data. There's no simple list of bones and things they are attached to.

Forward momentum.
Bored of the Rings
13
Years of Service
User Offline
Joined: 25th Feb 2005
Location: Middle Earth
Posted: 8th Oct 2018 09:54 Edited at: 8th Oct 2018 19:27
for indexes, these start at 0, see output snippet below and are WORDs:

will try and provide more info after work. In the example below, this was taken from the Elhumongo.DBO which has 1 mesh frame that contains the mesh data. There are 2 types of frame, mesh frames and bone frames for the animation data, and there are DBO's that don't have any Frames and are purely mesh (apart from a possible root frame).
there is a hierarchy for the bone frames so when outputting to say X format, have to be output in the right order.

pBuffer = 164674059 dwCode = 116 dwCodeSize = 65280 pData = 164674059 pBlock = 131072
ptr to dwCode = 164674051 pBuffer = 164674059 dwCode = 116 dwCodeSize = 65280 pData = 164674059 pBlock = 131072
pBuffer = 164674059 pData = 164674059 WORD = 0,
pBuffer = 164674059 pData = 164674061 WORD = 2,
pBuffer = 164674059 pData = 164674063 WORD = 1,
pBuffer = 164674059 pData = 164674065 WORD = 2,
pBuffer = 164674059 pData = 164674067 WORD = 0,
pBuffer = 164674059 pData = 164674069 WORD = 3,
pBuffer = 164674059 pData = 164674071 WORD = 2,
pBuffer = 164674059 pData = 164674073 WORD = 5,
pBuffer = 164674059 pData = 164674075 WORD = 4,
pBuffer = 164674059 pData = 164674077 WORD = 5,
pBuffer = 164674059 pData = 164674079 WORD = 2,
pBuffer = 164674059 pData = 164674081 WORD = 3,
pBuffer = 164674059 pData = 164674083 WORD = 6,
pBuffer = 164674059 pData = 164674085 WORD = 5,
pBuffer = 164674059 pData = 164674087 WORD = 7,
pBuffer = 164674059 pData = 164674089 WORD = 5,
pBuffer = 164674059 pData = 164674091 WORD = 6,
pBuffer = 164674059 pData = 164674093 WORD = 4,
pBuffer = 164674059 pData = 164674095 WORD = 2,
pBuffer = 164674059 pData = 164674097 WORD = 8,
pBuffer = 164674059 pData = 164674099 WORD = 1,
Professional Programmer, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 8th Oct 2018 18:06
Well I have only tested DBO files that were converted from X format so far, so this is interesting as you suggest other DBO types with significantly different structure.

Quote: "will try and provide more info after work. In the example below, this was taken from the Elhumongo.DBO which has 1 mesh frame that contains the mesh data."

I have noticed so far that only one bone ever contains significant mesh data (vertices/bones/polygons). Are you saying that the proper mesh can instead be split up across multiple bones? (So far every model I have seen has a 'body' bone attached to the root bone with all significant mesh data)

Quote: "There are 2 types of frame, mesh frames and bone frames for the animation data, and there are DBO's that don't have any Frames and are purely mesh."

I have noticed that the bone frames are only the ones that have a mesh code section, their mesh section is typically empty except for the one frame with all of the mesh data packed into it. These frames are named after each of the model's bones and have a proper hierarchy.

On large models I see there are also these seemingly empty useless frames with no names and no mesh code sections. Always a sibling frame. I am not sure what these are for or if they are just junk data from the original 'conventional' conversion process to DBO. Small trivial models do not seem to have these.

Quote: "... and there are DBO's that don't have any Frames and are purely mesh."

That's good to know. Right now I have it looking for a root frame and in the frame processing code it looks for Mesh codes and other things. Maybe add a mesh code check to that top level loop along side the frame code check.

Quote: "there is a hierarchy for the bone frames so when outputting to say X format, have to be output in the right order."

I have that mostly worked out. The real mystery are these frame matrix things. They are a 4x4 grid of float numbers. The bottom row seems to be X,Y,X,? for the bone position. However there's like 3 other rows and no indicators what those numbers are for. Probably Scale and rotation, but which?, and yet there are still way more numbers than what that would need. I have seen this grid in X model files too. Perhaps looking at that specification would provide an explanation.

Bored of the Rings
13
Years of Service
User Offline
Joined: 25th Feb 2005
Location: Middle Earth
Posted: 8th Oct 2018 20:08 Edited at: 8th Oct 2018 20:22
Quote: "I have noticed so far that only one bone ever contains significant mesh data (vertices/bones/polygons). Are you saying that the proper mesh can instead be split up across multiple bones? (So far every model I have seen has a 'body' bone attached to the root bone with all significant mesh data)"

So, yes, in most cases, the root frame will have a child frame that contains the main "body" mesh. As is the case with the example Elhumongo. That is, the child frame to the root frame is named "El_humongo" but isn't a bone i.e. there isn't an embedded DBO code 301 bone name named "El_humongo" only a frame name that contains the mesh. However, there is an animation frame (embedded code 211) named El_humongo .
For some FPSC classic models, there can be more than 1 mesh in 2 different named frames e.g. the main body mesh and a separate mesh for say, a sword or weapon of some sort.
In a lot of cases, where there are "no name" bones, I tend to remove these, but for some Weapon HUD .X/.DBO files these are required asthey most likely have assigned vertices that affect animations.
In general, I do a lot of validation checking to make sure things tally up. So if there are 60 bone influences, I do a check to make sure that there are 60 bone weights present as well as various other cross checks.


If I think of anything else, I'll add here, it's quite involved.
Professional Programmer, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 8th Oct 2018 22:06
Bored of the Rings wrote: "In a lot of cases, where there are "no name" bones, I tend to remove these, but for some Weapon HUD .X/.DBO files these are required asthey most likely have assigned vertices that affect animations. "

That is troubling and I have reason to doubt this because it seems the only real link vertices have to a bone is by matching the name. In the Mesh Data Block all the vertices get listed. Then the polys which each list 3 vertex id numbers. Then the bones get listed without their hierarchy but each bone has a list of vertices connected. These bones can only be matched to the hierarchy by name, unless there is some clever sorting pattern I have missed that one can then deduce the correct bone to frame matching.

However each frame has a Frame Matrix (even those empty ghost frames) which is from what I can tell meant to determine the orientation of that frame/bone relative to the parent/bone. So if a bone was the child of a "no name" bone then I could see the "no name" bone being important. Though I have never seen this as yet. These no name bones sometimes seem to appear on the ends of the hierarchy with no clear rule governing when they appear. What I do is export from the editor to X format and then resave the object as DBO then I am reading the DBO so this process may be adding some junk into the object.

Bored of the Rings wrote: "For some FPSC classic models, there can be more than 1 mesh in 2 different named frames e.g. the main body mesh and a separate mesh for say, a sword or weapon of some sort."

Well that's a mess to handle. It's not enough to just know that. You'd have to know whether the vertex list started at 0 (which would be sensible) or some other number, because the polygons and bones reference the vertex id's. So if all of those are being pooled into a master list for conversion to another format they would need to be adjusted to higher numbers. So the devil is in the details.

I find this while format is one big convoluted tangle of nesting loops, a house of cards. Very clever and probably quite speed efficient. Just not straight forward at all.
Latch
12
Years of Service
User Offline
Joined: 23rd Jul 2006
Location:
Posted: 10th Oct 2018 20:00 Edited at: 10th Oct 2018 20:06
Hello,

I don't know if it will help, but there is an old DBPro export plugin for Deled 3d. What I remember about Deled 3d format (the non XML format) was that it was very straight forward and easy to parse. My point is that if you can look at the plugin for DBPro and see how the simple Deled format was exported to DBPro, it might give you some insight into what is what.

here is the link though I can't remember if the source is included.

http://www.delgine.com/plugins/viewPlugin.php?catid=40&catdesc=Exporters&contentid=29
Enjoy your day.
Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 11th Oct 2018 19:57
Thanks for the suggestion.

At this point the only obstacle to being able to fully convert formats is converting the angles in each animation frame. The values require some clever university math, I haven't had the opportunity to exhaust all options with it just yet. I have noticed that this might not be a conversion needed when converting between DBO and X format. It's an issue caused by the editor's format. The angles stored in the X model and DBO model are the same values.

Here is an example of what I am working with:
Rotation in the DBO and X formats: X:0, Y:0.707107007504, Z: 0, 4th rotation value: 0.707107007504
Rotation in the editor format: X: 0, Y: -1.57079613209, Z: 0

So you can see there is not much here to work with. Some how 3 XYZ rotation values turned into 4 values. Both appear to be in radians. They don't seem to correlate at all. The model appears to have been rotated 180 degrees between the two formats part of the original conversion process when I was creating the DBO from a model made in the editor. I need to account for that and maybe the values will look simpler to sort out when that happens. Like how can an angle turn into a 0? The only reason I can think of is if there is a rotation 90/180/360 degree rotation of some sort. So a clue there.

I have looked at the source code for DBP and the editors X format exporter, but these are difficult to use in this case. DBP source code tends to read a whole block of data, store it as a block and in the labyrinth somewhere the block is peeled apart and processed. The editors X exporter will say something like get rotation matrix or transform vector and proceed to use internal API's that I can't use or examine. So there's no A = B + C moments.

What I am planning to do is to simply create a model with 2 polys and 2 bones. Then I will create very specific animation rotations and examine how those values get converted. Then hopefully I can learn the conversion process. If not then I will need to learn some serious matrix math.



Mage's Modular Code - Get awesome UI controls and powerful Bitmap Fonts!
Screen Device Recovery - Stop your apps from crashing when minimized/tabbed/screen locked.
CPU Friendly Frame Limiter - Make your app generate less computer heat + noise, and use less battery life!
Bored of the Rings
13
Years of Service
User Offline
Joined: 25th Feb 2005
Location: Middle Earth
Posted: 11th Oct 2018 20:34 Edited at: 11th Oct 2018 20:40
ah yes, for ROTKEYS/ROTDATA animation data (not MATRIX), I had to swap some data around i.e. for X format, W,X,Y,Z but in DBP is X,Y,Z and W:



code below, also ensured object didn't move/rotate to wrong position when the animation was selected in Fragmotion.

Professional Programmer, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 11th Oct 2018 21:00
I don't suppose you know that W refers to in this case? It would explain why the rotations are 4 values in DBP and only 3 in the editor.

Bored of the Rings wrote: "code below, also ensured object didn't move/rotate to wrong position when the animation was selected in Fragmotion. "

This tells me what I was suspecting. That all these different codes and things don't necessarily come in the same order for all models. So Frag Motion must assume some order that screws up the animation a little if your DBO isn't arranged that way. Because (for people reading this) when reading the DBO file there's a lot of [ID CODE][DATA][ID CODE][DATA]. Maybe the Bone name comes before the coordinates, maybe before, etc.


Mage's Modular Code - Get awesome UI controls and powerful Bitmap Fonts!
Screen Device Recovery - Stop your apps from crashing when minimized/tabbed/screen locked.
CPU Friendly Frame Limiter - Make your app generate less computer heat + noise, and use less battery life!
Bored of the Rings
13
Years of Service
User Offline
Joined: 25th Feb 2005
Location: Middle Earth
Posted: 11th Oct 2018 21:15 Edited at: 11th Oct 2018 21:25
I believe W means Weight.

Quote: ""code below, also ensured object didn't move/rotate to wrong position when the animation was selected in Fragmotion. ""

It's the same problem whatever modelling program I use and also when loading into GameGuru.

For codes/codesize/data, there is a specific ordering. Here's a complete output of all codes and ordering on the example I used previously i.e. Elhumongo.DBO. NOTE: this is a reWIP and so some of the output / counts for the embedded DBO codes need fine tuning.

It's a good idea to split the main data i.e. Root data from the animation data. I've yet to do that in the reworking project, and will do so shortly. So you might see some odd codes in wit hthe animation data at the moment, so can ignore those ones but in anim data you should /will eventually see only codes from 200 onwards.

Just to re-iterate, if anyone else is reading this, my program reads in a DBO (or binary X file) and produces an ASCII direct X format or OBJ. The reason I dis the program was so that I could read the formats in human readable form and change texturefilename in the X format mainly for Gameguru tools I wrote (mainly to import objects/segments and levels from FPSC to begin with).
The program works with skinned / animated meshes / objects and static objects.

What I also did in the program was to add ID's of frames so I could link any meshes etc to linked frames as well as texture / materials / bones etc.
In some cases , pBLOCK / pData does not exist for e.g. boolean flags e.g. USE MULTIMAT and similar codes.
Professional Programmer, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others

Attachments

Login to view attachments
Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 11th Oct 2018 22:04
Very strange. I would have expected that value to be 1.0 if it was a weight as the exporter I used does not support weights. Also 0.707107007504 is repeated with other values which are not weights.
Thanks for the info.

Mage's Modular Code - Get awesome UI controls and powerful Bitmap Fonts!
Screen Device Recovery - Stop your apps from crashing when minimized/tabbed/screen locked.
CPU Friendly Frame Limiter - Make your app generate less computer heat + noise, and use less battery life!
Bored of the Rings
13
Years of Service
User Offline
Joined: 25th Feb 2005
Location: Middle Earth
Posted: 11th Oct 2018 22:55 Edited at: 11th Oct 2018 22:56
correction, sorry it's been ages since I worked with this. axis is made up up XYZ and W is rotation around that axis. So, I believe it's WXYZ . I guess it could named R, but I just used W as read in some reference docs somewhere and can't recall where, maybe microsoft docs or EZRotate docs.
Professional Programmer, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Ortu
DBPro Master
10
Years of Service
User Offline
Joined: 21st Nov 2007
Location: Austin, TX
Posted: 13th Oct 2018 19:31
4 values in an angle is usually a quaternion

https://en.m.wikipedia.org/wiki/Quaternions_and_spatial_rotation
http://games.joshkirklin.com/sulium

A single player RPG featuring a branching, player driven storyline of meaningful choices and multiple endings alongside challenging active combat and intelligent AI.
Bored of the Rings
13
Years of Service
User Offline
Joined: 25th Feb 2005
Location: Middle Earth
Posted: 13th Oct 2018 22:12
Correct. I guess I should have just said that ha . Oh well it's been a long week .
Professional Programmer, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Mage
11
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 14th Oct 2018 19:31 Edited at: 14th Oct 2018 19:44
Thanks for the tips, little details like this are very helpful. I have seen the term Quaternion mentioned in the documentation. So I will need to familiarize myself with this concept. I have to go through the added step of taking apart the math here not just unpacking and repacking existing angle values. Before I began all of this I knew that it would eventually end up in the lowest dungeon on the highest peak with this demon to slay. Now I am standing face to face with it. The hardest part of the process.



When I have enough free time I will be able to go deep with the math and hopefully sort this out.

Mage's Modular Code - Get awesome UI controls and powerful Bitmap Fonts!
Screen Device Recovery - Stop your apps from crashing when minimized/tabbed/screen locked.
CPU Friendly Frame Limiter - Make your app generate less computer heat + noise, and use less battery life!

Login to post a reply

Server time is: 2018-10-16 20:51:08
Your offset time is: 2018-10-16 20:51:08