Quote: "I want to do this without DBP commands for the calculations (just using mathematics)"
I don't understand why you don't want to use the DBPro solution when it is right there for you.
You could calculate if a bounding box for the farther object is visually within the bounding box of the closer object, adjusting for the angle from the camera each object is positioned
Kind of like simulating the cameras FOV and object culling
Compute the view frustum in world coordinates and then determine if the bounding volume intersects it or not.
Here is a link that might help
http://www.gamedev.net/community/forums/topic.asp?topic_id=589338