@Diggsey
Yes thanks, I am on that site frequently these days.
Well this is turning out to be an interesting project. For whatever reason, I have kept mining for a solution using Euler angles and trigonometry. I have made a lot of progress but am stuck at a new point. I have the camera rolling properly, but only through half the range.
Basically it comes down to taking the the dot product of two vectors, one is the "camera up" and the other is a projection of the "ship's up" onto the camera's/screen plane. From this I can get the angle between them, thus the angle to roll the camera, by taking the inverse cos (arccos).
dbSetVector3(UP, up.x, up.y, up.z);
dbSetVector3(CAMERA_UP, cameraUp.x, cameraUp.y, cameraUp.z);
dbSetVector3(CAMERA_FORWARD, cameraForward.x, cameraForward.y, cameraForward.z);
dbCrossProductVector3(INTERMEDIATE, CAMERA_FORWARD, UP);
dbNormalizeVector3(INTERMEDIATE, INTERMEDIATE);
dbCrossProductVector3(PROJECTION, INTERMEDIATE, CAMERA_FORWARD);
dbNormalizeVector3(PROJECTION, PROJECTION);
float dotProduct = dbDotProductVector3(PROJECTION, CAMERA_UP);
if (dotProduct > 1) dotProduct = 1;
if (dotProduct < -1) dotProduct = -1;
float const PI = 4*atan(1.0);
float roll = 180/PI * acos(dotProduct);
Now my only problem is that acos() returns only between 0-180 degrees, but of course the roll is needed from anywhere between -180 to 180 (or 0-360) degrees. In real life we use the right-hand rule or visually inspect if the rotation is positive or negative, but in code and in math... how can I determine this here?
When I was tackling a similar problem before with arctan I was able to devise the quadrant from the x and y components and apply the CAST rule. But here the coordinate system is no longer neat, it is an arbitrary plane so looking at x,y,z components isn't the same... Help?