I don't think you need to worry about knowing the button size.
I would do this by using an object for the cursor and just detecting collision with the object against the planes used for the buttons. A bit like this:
`3D button press demo
`move the sphere over the rectangle and press the left mouse button
sync rate 65
sync on
hide mouse
`make object for buttons
for i = 1 to 6
make object plain i, 100,50
next i
`make a sphere to use as the cursor
make object sphere 100, 10
color object 100, rgb(255,0,0)
`position the buttons
position object 1, 100,100,0
position object 2, 100,0,0
position object 3, 100,-100,0
position object 4, -100,100,0
position object 5, -100,0,0
position object 6, -100,-100,0
`initial position of the cursor
xcursor# = 0
ycursor# = 0
do
`position the cursor
inc xcursor#, mousemovex()
inc ycursor#, mousemovey()*-1
position object 100, xcursor#, ycursor#, 0
`detect collision of he object against the planes
ob = object collision(100,0)
`if an object has been hit
if ob <> 0
if mouseclick() = 1
`left click the mouse to scale the button
scale object ob, 50,50,50
else
`release or don't click to scale the object to full size
scale object ob, 100,100,100
endif
endif
position camera 0,0,-500
point camera 0,0,0
sync
loop
Hope this helps.