Sorry your browser is not supported!

So essentially, you have several statements:
A is orthogonal to C
B is orthogonal to C
There exists a rotation matrix R such that:
(rVec and fVec are shorthand for right vector and forward vector)


    [1]
R*A=[0]=rVec
    [0]

    [0]
R*C=[0]=fVec
    [1]


R*B=?

So,
R*A=rVec
A=R^(-1)*rVec

or in other words,
       [A.x  R01  R02]
R^(-1)=[A.y  R11  R12]
       [A.z  R21  R22]

We can also see from the equation R*C=fVec that

       [A.x  R01  C.x]
R^(-1)=[A.y  R11  C.y]
       [A.z  R21  C.z]


If we can find r01, r11, and r21, then we know r^(-1) and therefore we know R.


Now... I'm not sure how to solve for those values using the facts above... however, we do know that R is an orthogonal matrix, and each row vector has a length of one. We also know that the cross product between two vectors gives a perpendicular vector, so in other words, either A x C = Rk1 (where Rk1 is the vector at column one of matrix R^(-1)) or C x A = Rk1. I'm not sure which, so we'll have to experiment to find out. In other words:


R^(-1) =  [  A  AxC   C  ] or  [  A   CxA   C   ]
(remember, this is a square matrix because A, AxC, and C are all column vectors)


Since AxC=-CxA, one of the above matrices is a rotation matrix (the one we want), and one is junk (without the two properties R^(T)=R^(-1) and det(R)=1. I believe it'd be a reflection.)

So we could calculate both matrices and their determinants, but I think we can solve this by trying to solve a simpler problem.


Say the following is a rotation matrix:
[1 0 0]
[0 1 0]
[0 0 1]

either (1,0,0)x(0,0,1)=(0,1,0) 
or     (0,0,1)x(1,0,0)=(0,1,0) 

if it's the first one, then we plug in AxC to solve our problem, and if it's the second one we plug in CxA.

By definition, two vectors F and G
FxG=(F.y*G.z-F.z*G.y, F.z*G.x-F.x*G.z, F.x*G.y-F.y*G.x)


After calculating it out, we see that:
(1,0,0)x(0,0,1)=(0,-1,0) 
(0,0,1)x(1,0,0)=(0,1,0) 

Therefore, R^(-1)=[ A  CxA  C]

calculating CxA...

CxA=(C.y*A.z-C.z*A.y, C.z*A.x-C.x*A.z, C.x*A.y-C.y*A.x)


So...

       [A.x  C.y*A.z-C.z*A.y  C.x]
R^(-1)=[A.y  C.z*A.x-C.x*A.z  C.y]
       [A.z  C.x*A.y-C.y*A.x  C.z]

and since the inverse of a rotation matrix is equal to its transpose...

  [A.x                    A.y                    A.z]
R=[C.y*A.z-C.z*A.y  C.z*A.x-C.x*A.z  C.x*A.y-C.y*A.x]
  [C.x                    C.y                    C.z]

So, B would become R*B. So you want to calculate:


[A.x                    A.y                    A.z] [B.x]
[C.y*A.z-C.z*A.y  C.z*A.x-C.x*A.z  C.x*A.y-C.y A.x]*[B.y]
[C.x                    C.y                    C.z] [B.z]

and you have your answer! I think... if I didn't go wrong somewhere...