Hi Inflictive Studios.
It does seem back to front that you need to start the physics before you can test for hardware, but hardware needs to be enabled on start! The PhysX SDK forces this, I presume you start->test->stop->start with hardware?
That aside. Hardware acceleration only works on cloth, soft bodies and particles. Each of these need the hardware flag to be set on creation, the scene also needs hardware to be set as well as the SDK as a whole(which is set with start(true)).
I hope this helps, I can't test it as I don't have NVIDIA hardware on any of my computers at the moment. Let me know how you get on.
EDIT: You may also need to set a hardware flag when creating the soft body/cloth mesh also.