Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

AppGameKit Classic Chat / [BUG] Sensor Delay on S7 - Acceleration + rotation sensors are broken

Author
Message
Jack
19
Years of Service
User Offline
Joined: 4th Oct 2004
Location: [Germany]
Posted: 20th Mar 2016 16:27 Edited at: 20th Mar 2016 16:34
I have discovered a huge bug inside the RAW sensor commands, on my S7.
First, the S7 acceleration sensor is working as it should, but not in AGK.

The gyro and magnetic sensors seem to work, but the acceleration sensor doesn't return the actual acceleration.
In fact it does, but to a very slow rate. This may be a result in some change of Sensor_Delay on newer models.
The same problems occur, if I try to use the fused rotation.
I think the problem with the acceleration delay has also broken the fused rotation commands.

The problem does not appear on my old S3.

One interesting thing is, that if I use the StartGPSTracking command, the acceleration delay is gone! But the fused rotation
is still broken. I need the fused rotation for a VR app.

(Can we get a command to set the Sensor_Delay rate manually? pls)

Edit: If you need videos, or screenshots to demonstrate that problem, drop me a line. I would like to contribute to improve AGK. Cheers!

[/url]
Digital Awakening
AGK Developer
22
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Sweden
Posted: 20th Mar 2016 21:59
Here's just a guess. But could it have anything to do with it being Gear VR compatible?
Jack
19
Years of Service
User Offline
Joined: 4th Oct 2004
Location: [Germany]
Posted: 21st Mar 2016 00:29 Edited at: 21st Mar 2016 06:57
Hello Digital Awakening,

I don't know. Other VR Apps are working, but I am shure it has to do something with the sensor_delay,
because of the diffrent behavior of the acceleration sensor, when using the StartGPSTracking() command

[/url]
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 21st Mar 2016 15:00
I had a look at the code and the accelerometer delay was set at 10fps, it must have been that way since we first supported Android. Most devices seem to ignore it, as it is only a hint, but I've set it to 60fps for the next version.
Mobiius
Valued Member
21
Years of Service
User Offline
Joined: 27th Feb 2003
Location: The Cold North
Posted: 21st Mar 2016 20:37
Cool, can't wait to try it!
Jack
19
Years of Service
User Offline
Joined: 4th Oct 2004
Location: [Germany]
Posted: 22nd Mar 2016 11:38
Nice! Thank you

[/url]
Jack
19
Years of Service
User Offline
Joined: 4th Oct 2004
Location: [Germany]
Posted: 18th Jun 2016 20:44 Edited at: 18th Jun 2016 23:28
As I already mentioned in diffrent threads, the fused rotation are still not working properly on the Samsung S7.
It's like a delay if I try to rotate the phone, it will get to the right position, but only after 1-2 seconds.
Paul said, it should be the magnetic sensor that will cause the problem, but he has also no idea how to get the fused rotation working again.

It looks like some type of delay still exists. I checked the gyro with a simple code:
Quote: "
RotateCameraLocalX(1,-GetRawGyroVelocityX())
RotateCameraLocalY(1,-GetRawGyroVelocityY())
RotateCameraLocalZ(1,GetRawGyroVelocityZ())
"


and the delay was not there. But as we now, the gyro has a natural drift and has to be used in some sort of fusion to work properly. My code is also not the problem, because it works on the Samsung S3 without any delay.
I tried to influence the fused rotation command with a small magnet, that I moved near my phone, and the rotation went crazy.
Am I the only one with trouble on the S7? Other VR apps are working correctly, so the sensor should be fine. I tried to calibrate the magnetometer, disabled the most packages on the phone and even installed a custom OS "Superman rom" on the S7, but without any new results.

Can this get fixed?
And is there an alternative, like a custom fused rotation code, based on accelerometer and gyro and maybe the magnetometer only for the initial rotation?

GearVR would work with AppGameKit, if this could be fixed.
It's a huge market for AGK.

I will make some videos with the S3 and S7 to show you the issue.

[/url]
CJB
Valued Member
20
Years of Service
User Offline
Joined: 10th Feb 2004
Location: Essex, UK
Posted: 19th Jun 2016 09:06
To get really solid head tracking, you need to fuse accelerometer, gyro and magnetometer (not just for initial rotation, but to also prevent drift).
V2 T1 (Mostly)
Phone Tap!
Uzmadesign
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 19th Jun 2016 19:12
Can you try the attached version of the AppGameKit player and see if it works any better? It uses an alternative fused sensor called the game rotation vector that doesn't use the magnetometer, but is only available since Android API 20

Attachments

Login to view attachments
Jack
19
Years of Service
User Offline
Joined: 4th Oct 2004
Location: [Germany]
Posted: 19th Jun 2016 22:45 Edited at: 19th Jun 2016 22:47
Hello Paul, thank you for the new player!

It works a lot better on the S7 - the rotation delay is gone.

The only difference to the S3 with the old player is a wobble if I move the phone linear. (Even when I rotate the phone not exactly on the middle point of the phone)
It's like if I rotate the phone in front of me to the left and right (real world Y-Axis), I get a smooth camera rotation of about +-20 degrees on the Z-Axis, once I stop moving, the camera smoothly stops and everything goes back to normal

It looks like linear acceleration is getting into the rotation calculation. Is this possible?

[/url]
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 20th Jun 2016 02:13
I've attached another version, this one grabs the sensor values from Java and passes them to AGK. Unfortunately this isn't a workable solution as it takes 2-3 milliseconds to pass the data, but it will reveal if the NDK is at fault.

It also prints the raw values to the screen, the X,Y,Z,W values should vary between -1 and 1 as you rotate the device, and the length should always be 1, or very close to it. Some devices don't return the W value and keep it at 0, but AppGameKit should account for this and calculate it internally. So the rotation should still work, but it would be interesting to see if this is one of those devices.

Attachments

Login to view attachments
Jack
19
Years of Service
User Offline
Joined: 4th Oct 2004
Location: [Germany]
Posted: 20th Jun 2016 03:07 Edited at: 20th Jun 2016 03:28
The wobble still exists in this version

the X,Y,Z values are between -1 and 1, like you said
the W value is between 0 and 1. I can't get the W value negative.
The length is between 0.999998 and 1.0.

EDIT: Am I wrong or is the performance of the new player way better than before?
.DAE objects are also working without a flaw. +1

[/url]
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 20th Jun 2016 12:55 Edited at: 20th Jun 2016 12:58
That all looks fine, the W value doesn't have to be negative, since there are two quaternions for every possible rotation, a negative W one and a positive W one. There isn't much more I can do, I assume other VR apps are doing their own sensor fusion, or using another sensor SDK. Since the S7 is quite new maybe it will get an update to fix it?

Quote: "Am I wrong or is the performance of the new player way better than before?"

We've overhauled the image loading routine to make it much faster!
Jack
19
Years of Service
User Offline
Joined: 4th Oct 2004
Location: [Germany]
Posted: 20th Jun 2016 19:23
Is there something like linear acceleration sensor in the SDK, that is used to clear the accelererometer from gravity and linear acceleration?

Accelerometer RAW Data = rotation acceleration + linear acceleration + gravity.

The linear acceleration is a virtual sensor based on some calculations, that can be recalculated by hand

For the kalman filter only the rotation acceleration is required, I really think the native linear acceleration sensor stays at 0.0 and won't be removed from the "total" accelerometer data so this problem occurs at all fused sensor code.
If it's possible, I would like to check the sensor fusion code for this. It would also be nice, if you could give me a player, that prints the native android linear acceleration data (if they are available) on screen. If this is the problem, I could remove the linear acceleration by hand and give the code to you

Quote: " We've overhauled the image loading routine to make it much faster!"

Yes, the loading times also improved significantly, but I mean the performance while running with 3d and textures. It got a lot better, as far as I see.

[/url]
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 20th Jun 2016 20:05
You can see a list of sensors here https://developer.android.com/reference/android/hardware/SensorEvent.html#values, we were using TYPE_ROTATION_VECTOR and in the new player I changed it to use TYPE_GAME_ROTATION_VECTOR. Both are fused sensors but the code that calculates this is buried in the Android OS, so I don't know what it does.
CJB
Valued Member
20
Years of Service
User Offline
Joined: 10th Feb 2004
Location: Essex, UK
Posted: 21st Jun 2016 08:56
Found a good tutorial on sensor fusion here. (3 pages). You need to fuse all 3 sets of sensors for smooth head tracking. So to get some really basic, but useable head-tracking, we'd either need something like this built into AppGameKit or port the code into an AppGameKit version ourselves.

The alternative would be to implement support for a VR SDK (Google VR SDK?)

V2 T1 (Mostly)
Phone Tap!
Uzmadesign
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 21st Jun 2016 14:49
The comments section of that tutorial suggest it was only considering Android 2.3.3, and that since Android 4.0 it now has sensor fusion built in with TYPE_ROTATION_VECTOR, which is what we are using. I would prefer to use a built in function that will be kept up to date by Google if possible. That said I had a look at the VR SDK to see if they are using some other kind of sensor fusion, but their sdk-controllerclient example also uses the TYPE_ROTATION_VECTOR sensor.
Jack
19
Years of Service
User Offline
Joined: 4th Oct 2004
Location: [Germany]
Posted: 21st Jun 2016 17:34 Edited at: 21st Jun 2016 17:45
Hello Paul, I found a practical solution for this problem using only the standard android sensors.
If you give me the gravity sensor and the linear acceleration type values from android I will create one of the smoothest VR controls right now - even better than any single fused command set and with an enormous tolerance to wrong data.
I would code it in AppGameKit T1 and you could easily port the logic and implement it in AGK. Can we get this done?

EDIT: If you want full GearVR support (I do^^), we just have to be able to add a signature file from Occulus over the IDE before compiling the APK and AppGameKit would be able to create GearVR apps for the Occulus store.

[/url]
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 22nd Jun 2016 15:20
The gravity and linear are also fused sensors, so in theory they too can be calculated from the other sensors. I'm too close to a release to add them in to the next version, but if you want to try using them anyway I would recommend modifying the tier 2 project to use them and see if you can get anything out of it. To make the modifications browse to "Tier 2\apps\interpreter_android_google\AGKPlayer2\src\main\jni" open main.c, and at the very bottom you will find the sensor init commands. Sensor 11 is the TYPE_ROTATION_VECTOR sensor.
Jack
19
Years of Service
User Offline
Joined: 4th Oct 2004
Location: [Germany]
Posted: 30th Jun 2016 14:31 Edited at: 30th Jun 2016 14:34
Quote: "but it will reveal if the NDK is at fault."


Recently I started to install AppGameKit and NDK in Android Studio and came across the architecture of the S7 called "snapdragon" it`s kind of unique. This architecture can have a custom NDK compile and maybe this is causing the error?

AGK Tier2 Android guide:
Quote: "It will compile three libraries, one for armeabi, one for armeabi-v7a, and one for x86, these cover the three most popular architectures that run Android. The libraries will be placed in the appropriate folders for Android Studio to pick up next time it builds an APK of your project."


Am I wronger than wrong?


I am also stuck at the NDK compile process:
Quote: "D:/AndroidDev/android-ndk-r12/build//../build/core/prebuilt-library.mk:45: *** Android NDK: Aborting . Stop."

but this actually gave me the idea about the snapdragon thing.

[/url]
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 30th Jun 2016 14:52
SnapDragon is the name of Qualcomm's system-on-chip, which includes an ARM CPU, among other things, so it would be covered by the armeabi-v7a architecture.

As for the error, we recently discovered a bug in the 32-bit version of the r12 NDK, the 64-bit version of r12 seems to work. Otherwise I'd need to see the rest of the log file to see what happened before it stopped.
Jack
19
Years of Service
User Offline
Joined: 4th Oct 2004
Location: [Germany]
Posted: 30th Jun 2016 14:58 Edited at: 30th Jun 2016 15:16
Quote: "so it would be covered by the armeabi-v7a architecture."

Good to hear that! (But somehow I hoped to find the issue with that )

Are there still some differences between the NDK and LLVM compiler? Like mentioned here:
https://developer.qualcomm.com/software/snapdragon-llvm-compiler-android
I have to ask, because there should be a bug somewhere that will kill the sensor fusion on the S7.


Quote: "I'd need to see the rest of the log"

This one line is actually the full log. I am also using the 64bit version of the NDK, maybe.

Why maybe? That's why:

(I got the x86_64 download for windows... but I don't know...)

(Taken from the NDK website)


EDIT: Just tested the 2.020 update on the old S3, it broke the sensor fusions there
I am so sorry for this never ending story...

[/url]

Attachments

Login to view attachments
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 30th Jun 2016 15:41
Well that's confusing, it wasn't like that last time I checked it! I would say the x86_64 version is the 64-bit version, but I'm still using r11b 64-bit since I know that works.

Quote: "Just tested the 2.020 update on the old S3, it broke the sensor fusions there"

Completely? Or does it work but incorrectly?
Jack
19
Years of Service
User Offline
Joined: 4th Oct 2004
Location: [Germany]
Posted: 30th Jun 2016 16:03 Edited at: 30th Jun 2016 16:24
Quote: "I'm still using r11b 64-bit since I know that works."

I will try to find the r11b version.

Quote: " Completely? Or does it work but incorrectly?"


Well, when I rotate my phone 90 degrees to the left, the camera moves with it, but when the moment is finished, the camera turns back to 0 degrees on the Y axis, even though the phone stays at 90 degrees rotation.
X and Z axis seem to work.
The linear acceleration issue is now also in there, even though it's 10-20 times less hard than on the S7, but wasn't there before.

[/url]
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 30th Jun 2016 16:53
If there is nothing important on the S3 can you try factory resetting it? That would rule out any software changes.
Jack
19
Years of Service
User Offline
Joined: 4th Oct 2004
Location: [Germany]
Posted: 4th Jul 2016 23:25 Edited at: 5th Jul 2016 09:27
Quote: "If there is nothing important on the S3"


Unfortunately I can't reset the S3

But I made some progress with the NDK (After 1 week of programming depression). The AppGameKit android guide is correct and helped me a lot. The problems came from the diffrent versions of AS, NDK and JDK that I used. After I downloaded the exact versions mentioned in the guide (I got them using a Internet wayback machine), everything worked perfectly. (The new version of Android Studio does not have Android 5.0.1 in the SDK Manager, only Android 5.1) Now I can compile AppGameKit in Android Studio! I don't know what will happen, if I update Android Studio, the NDK or JDK, I am still glad I could make it work.

Is there any guide how I could reach all the sensors provided by Android Studio? Maybe also how to set the sensor delay? As far as I can tell, I need them inside the template.cpp passed from the Android Studio Java code, or am I wrong? But as it seems, there have to be deeper changes than only in the template.cpp/template.h and this is quite new for me. Any hints on that one?

My current state is, that I managed to build a simple skybox in T2 with a simple level, and I added the CameraRotationQuad command, that does behave like in T1, including the glitch. (You can see it in the attachments, I've created a video with Android Studio. 0-20s I do rotate the phone only on "my Y-Axis", to the left and to the right. At about the 20s I move it away from me and back to my head, you can clearly see the wrong motion/ wrong tilt and I have tried not to rotate the phone on the X-Axis or the Y-Axis, as good as I can)

These four floats are holding the Rotation Quad:

cam_w = agk::GetRawRotationVectorW2();
cam_x = agk::GetRawRotationVectorX2();
cam_y = agk::GetRawRotationVectorY2();
cam_z = agk::GetRawRotationVectorZ2();


I would like to know how I could create additional floats for example for the TYPE_LINEAR_ACCELERATION or TYPE_MAGNETIC_FIELD_UNCALIBRATED....



Once I got that, I could finally start to create the improved sensor fusion code.

Greeting, Jack

EDIT:
I have not found a board to comment the new "report post button", but for me the red color is way to offensive
It's still good visible with less saturation:



Sorry for going off- topic again.

[/url]

Attachments

Login to view attachments
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 5th Jul 2016 14:43
If you open "template_android_google\AGK2Template\src\main\jni\main.c" and scroll all the way to to bottom (and then up a bit) you will see the sensor setup code. The sensors are retrieved using ASensorManager_getDefaultSensor() and the rates are set in engine_handle_cmd() which handles the app going to the background and foreground.

The ASENSOR_TYPE_GAME_ROTATION_VECTOR and ASENSOR_TYPE_ROTATION_VECTOR constants are not defined in this version of the NDK, so they are referenced by their numbers 15 and 11 instead.
Jack
19
Years of Service
User Offline
Joined: 4th Oct 2004
Location: [Germany]
Posted: 5th Jul 2016 21:05 Edited at: 5th Jul 2016 21:14
Thanks! I found everything I need! Thank you for your support Paul, you're awesome!

As far as I can see, the sensor data is packed in a seperate thread. Very good work! Maybe I could reach the 1000Hz with the VR gear.

I've started the development in Java/ Android Studio and the data looks promising! (But there is still much to do) During my hacking, I found out, that it should be possible to build a precise but less stable fusion or a stable but less precise fusion. I think both should be done to get the best results, maybe with a float from 0.0 to 1.0 to get a seamless translation, because all the devices are very unique. Once this is done, I will add the optional magnetometer to the fusion in order to make it even more stable (if the magnetometer doesn't trick me again). The next days I'll take a deep look at the oculus SDK, maybe there are new hits how to do this better.


Now I just have to know, how to call Java functions ("methods") in NDK, as far as I inspected the main.c, this shouldn't be that difficult (?),
but this working enviroment is still new for me and currently I can't get how to do this.

[/url]
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 6th Jul 2016 13:40
Quote: "As far as I can see, the sensor data is packed in a seperate thread."

The thread is used to run all the NDK code and runs about 60fps, whilst the Java parts are run on the main thread.

Quote: "Now I just have to know, how to call Java functions ("methods") in NDK, as far as I inspected the main.c, this shouldn't be that difficult (?), "

I would stay in the NDK as much as possible as jumping out to Java is quite expensive, nearly a millisecond per call. But if you need to do it, all the Java calls are in Core.cpp, and are mostly boiler plate code
Jack
19
Years of Service
User Offline
Joined: 4th Oct 2004
Location: [Germany]
Posted: 13th Jul 2016 11:38 Edited at: 13th Jul 2016 12:18
Alright!
There is actually some progress:



the red camera is the original sensor fusion from android passed to AppGameKit as "GetRawRotationVectorW2, X2, Y2, Z2", the green camera represents the custom rotation.
As you can see, the new method will eliminate the whole wobble and can even help to reduce gyro drift, if the phone is shaking. There are still some small bugs in the quaternion calculation code it will result in a failed frustrum culling so objects that are actually on screen get culled out, the precision of the quat calculation has to be perfect in order to get this done. Maybe floating point errors, this should be easy to fix.

The next thing on the task list would be to get the magnetometer working again, so the camera will always point in the same direction, once initialized. I have to create a new concept for this using the magnetometer raw data and the custom game_rotation_vector with a custom kalman filter. Maybe I should try it with the old rotation vector first, this one has already the magnetometer build in, but due to the rotation delay, it's quite possible that I have to code a custom implementation for this one too.
I bet I could even improve the accelerometer data into this custom fusion, but it will be hard work though.

I have also taken a look inside the oculus SDK. They are actually using different sensors build in the gear vr headset. This sensors are much better, running on 1000Hz. There is actually a chance to get them in AppGameKit, but I have still no clue how to combine the oculus SDK with AGK. There are also good news: both of them are using the same Android SDK and can be compiled using the same NDK.
I don`t have the skills right now to fuse the oculus SDK into AppGameKit, I have still to learn a lot on this segment.


Paul, do you have data about the speed difference of code written in AppGameKit or NDK? And what is about the timer() command in agk? The docs mentioned
"Calling this command will internally make a system call to get the amount of time passed, so avoid calling this command unnecessarily. " - how does it affect the performance, and are there faster methods to get the frame time?

Are all OpenGL calls in a seperate thread and commands like sin(), abs() in another? Like the music that doesn't get slower, even if the FPS is dropping?

I need this information to improve the performance of the working code

[/url]
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 14th Jul 2016 16:08
Quote: "do you have data about the speed difference of code written in AppGameKit or NDK?"

I've not done any tests, but AppGameKit Tier 1 code will be slower than NDK code.

Quote: "And what is about the timer() command in agk?"

Sorry, the help is badly worded, I'll change it for the next version. Timer() makes a system call so the value will change every time it is called, so if you were calling Timer() continuously expecting it to stay the same inside a single frame then you should call it once and save that value in a variable. I wouldn't expect the call itself to take much time at all.

Quote: "Are all OpenGL calls in a seperate thread and commands like sin(), abs() in another?"

OpenGL runs in the same thread as the other code, including AppGameKit Tier 1 code. The only bits that are separate are the OGG music thread, and the Java calls to things like In App Purchase, Facebook, and Adverts
CJB
Valued Member
20
Years of Service
User Offline
Joined: 10th Feb 2004
Location: Essex, UK
Posted: 18th Jul 2016 11:15
Quote: "The only bits that are separate are the OGG music thread, and the Java calls to things like In App Purchase, Facebook, and Adverts"
... and all the 2D physics stuff right?
V2 T1 (Mostly)
Phone Tap!
Uzmadesign
Paul Johnston
TGC Developer
21
Years of Service
User Offline
Joined: 16th Nov 2002
Location: United Kingdom
Posted: 18th Jul 2016 14:00 Edited at: 18th Jul 2016 14:01
Box2D uses threads, but they are inline, i.e. the code gets to Box2D code, splits into multiple threads, then joins back to a single thread before continuing.

I also forgot that most of the networking is on a separate thread.

Login to post a reply

Server time is: 2024-09-29 17:27:11
Your offset time is: 2024-09-29 17:27:11