SOLVED
Since this actually isn't an error in AppGameKit I thought I should try to fix it myself before annoying support.
When I started the MSI-logging as per Hubdules advice I found that it boiled down to:
DEBUG: Error 2896: Executing action WiseNextDlg failed
The general advice is to(re)register vbscript.dll
Since the installer is a 32-bit application I think the 32-bit one is all you need to register. Run this in an administrator command prompt: regsvr32 c:\Windows\System32\vbscript.dll
and try to install again.
Registering vbscript didn't help me, but it was the source of my problem. I found
this article that basically tells you to rename HKEY_CURRENT_USER\Software\Classes\Wow6432Node\CLSID\{B54F3741-5B07-11CF-A4B0-00AA004A55E8} in the registry.
The vbscript class seems to have an erroneous key in there.
I hope this helps some one else encountering this error.