The purpose and usage of dbPickObject is explained in the help. This function tells you which object is under the mouse when you click and you can even limit the search to a range of object ID numbers, so other objects are disregarded during the raycasting. Now, if you limit the search to only your terrain object, then this can help you a lot, because afterwards from the pick vector data you can get the coordinates where the terrain was clicked. For example, if you have a terrain object and a box object, then this code will move a box to where you clicked on the terrain:
if (dbMouseClick()) {
dbPickObject(dbMouseX(), dbMouseY(), IDTERRAIN, IDTERRAIN);
x = dbCameraPositionX() + dbGetPickVectorX();
y = dbCameraPositionY() + dbGetPickVectorY();
z = dbCameraPositionZ() + dbGetPickVectorZ();
dbPositionObject(IDBOX, x, y, z);
}
I tested this with the terrain demo application of Dark GDK. Note that dbPickObject does not work with a matrix, only with terrains and objects, and in the thread which I quote below, it is also said that this command is "too slow for RTS". You need to test if it's fast enough for you or not.
As far as I see the problem, you can go two ways in solving it. The first way is to somehow convert the mouse-click (2D) coordinates into game world (3D) coordinates and compare them with the game world coordinates of your objects, to see which of them fall inside your rectangle. That's where dbPickObject can help, or you can use dbPickScreen as well but the usage of that is very tricky. You have to guess the distance, normalize the vectors, etc. There is a good explanation about that in this thread:
http://forum.thegamecreators.com/?m=forum_view&t=129031&b=10
The other way would be to use only screen coordinates. You can use the dbObjectScreenX and dbObjectScreenY functions to discover the screen coordinates of your objects and you can compare them to the mouse rectangle coordinates. This may be a little easier. You can also use the dbObjectInScreen function to determine if a unit is within the visible screen or not and compare only those which are visible.
There is also an RTS tutorial for Dark Basic Professional here:
http://zimnox.com/dbcc/?page=tutorials
Although it's a different programming language but the concepts should be easy to undestand and convert.
I've never tried to write an RTS so the only help I can give is to point to these threads which may be useful to you.