Think about this slightly differently and you can use the normal mouse in an RTS game.
You are trying to find the mouses position in 3D space when there is no inbuilt command to do so. However there are commands that convert 3D space into 2D, so why not use those
findRange=9999
findTarget=0
for unit=1 to 1000
if object in screen(unit)
rngX=abs(object screen X(unit)-mouseX())
rngY=abs(object screen Y(unit)-mouseY())
if rngX+rngY<findRange
findRange=rngX+rngY
findTarget=unit
endIf
endIf
next unit
I hope that helps
Pneumatic Dryll, Outrageous epic cleric of EQ/Xev
God made the world in 7 days, but we're still waiting for the patch.