Why would you do it that way?
Even with something as simple as randomly created cubes the load time is increased, and rendering time is increased. At those numbers, excluding objects dynamically starts to have an impact on your game performance, and so do the visibility checks themselves. If your objects have any complexity at all, they will not only put pressure on your system memory, but also on your graphics memory and at some point will force your graphics card to retrieve object vertex and texture data from system memory during rendering, again slowing the rendering further.
One way to deal with your scenario is to link your ground clutter to blocks of terrain, and link their visibility to that of the terrain. When that terrain is visible, you create your ground clutter using instancing. When it isn't, you delete the ground clutter objects, or reuse them for another terrain block that has come into view.
Using basic simple techniques like that and others, you simply don't need large numbers of objects.
