You may need to setup the Newton World in order for it to work.
I've used the code below for my specific purposes (creating collisions on a .X mesh) and it makes an Unoptimized collision. I haven't tried optimized collisions in a long long time due to the dodgy behavior in the 1.2+ version of Newton... so I don't know if that may be your problem.
setupNewton()
input "Filename (no extension) to convert:", file$
if file exist(file$+".x") = 0
print "File does not exist!"
wait key
end
else
load object file$+".x", 1
col = NDB_NewtonCreateTreeCollisionUnOptimized(1)
NDB_NewtonTreeCollisionSerialize col, file$+".col"
wait key
endif
NDB_NewtonDestroy
end
if memblock exist(1) then delete memblock 1
if mesh exist(1) then delete mesh 1
function setupNewton()
` This initializes the Newton world. This must be called before
` any other calls can be made.
NDB_NewtonCreate
NDB_SetVector 1, -4000.0, -500.0, -4000.0
NDB_SetVector 2, 4000.0, 500.0, 4000.0
NDB_NewtonSetWorldSize
endfunction
Also if your model is really big you may want to use the "limb by limb" flag,
A 3D marble platformer using Newton physics.