You want to make sure two points are on opposite sides of a triangle right? I think you can just do this with dot product and cross product.
You have P1 P2 and P3 defining the triangle, and you have a point A. Consider P1 the origin, so now we have a triangle defined by (0,0,0), P2-P1 and P3-P1, with point A-P1.
The cross product will get a vector perpendicular to the surface of the triangle, that is: (P2-P1)x(P3-P1). To find where A-P1 lies along this line, take the dot product:
((P2-P1)x(P3-P1)).(A-P1)
To find out where another point B lies along this line, calculate the dot product again:
((P2-P1)x(P3-P1)).(B-P1)
If these two values have opposite signs, they lie on different sides of the triangle.
For two vectors, the dot product is defined by:
(Ax,Ay,Az).(Bx,By,Bz)=Ax*Bx+Ay*By+Az*Bz (a scalar value)
and cross product:
(Ax,Ay,Az)x(Bx,By,Bz)=(AyBz-ByAz,AzBx-BzAx,AxBy-BxAy) (a vector value)