I've been wondering for some time if I could stretch a UV map over numerous objects, which would help in this case, ie pieces of a wall.
Most other games that have destructable objects just cheat,using an image or object for showing a percentage of damage (maybe 5 images, 100% health down to nothing in stages). Probably easier than taking the pieces apart, and also probably quicker than having lots of smaller objects or images that make up the 'wall' or object.
For a large scale scenario as you're mentioning, then set stages of destruction would be a better option.
Warning! May contain Nuts!