Quote: "what's the best way to handle 2000/3000 objects at once?"
The trick is not to have to handle thousands of objects at once. Split the entire world into a grid (or series of grids), and only handle/update things around the player. It is also good spread out the updating over multiple loops.
Quote: "I'm thinking physX object, objects with shaders, lighting and shadows etc, whats going to handle all of this?"
In the project that I have been working on, I have separate "systems" which handle each of these. I use a master list (an array) to store all of my IDs, then the functions which need the list have their own arrays which store the index of the item in the master list.
The entire world is split into a series of grids which is then used to optimize everything (Object LOD/Collision/Recycling, Sounds, particles, etc...).
I have a Load_Texture(Filename$) function which only loads an image once, and returns that ID each time the image is attempted to be loaded.
I have also written Find_Free_Object(), and Set_Object_Free(objID) functions which do exactly what you describe by simply using an array, and a global variable which stores the last returned ID.
Each Type of static object uses a defintion file, which stores all of its properties (LOD,Textures,shaders, etc..). Then I keep an object "Instance" array which stores the positions of all of the static objects, and is then used to update the LOD and collision.
I then use an array to store the indexes of the instances which are in each sector. It looks like: SectorObjs(SecQTx,SecQTz,NumOfObjTypes,MaxSecInstanceQT)
For fast lookups I just calculate which sector the player is in, and use a for-next loop to check the surrounding sectors.
For physics I setup the static rigid bodies for the entire world, then delete the objects and replace them with simple triangle objects (using the same IDs) which are then excluded. This same logic can be used to apply physics to DBpro Instanced objects.
I am using Evolved's deferred shading (now called "Advanced Lighting") for the lighting as it supports multiple shadow casting lights (point, spot, cube, and directional), Normal/parallax mapping, post processing, etc... but it requires a pretty nice graphics card to even think about using.
