Check your "setup.ini" and look for the lines...
lightmaptexsize=x
systemmemorycapoff=0
If your "lightmaptexsize" is set to 1024, try lowering the value to either 512 or 256. I exceed the VM cap if I set mine to 1024, but no issues when set to 512, and I use a 38x38 grid, 5 layers up/down.
The "systemmemorycapoff=0/1" works as follows...
systemmemorycapoff=set to 1 to disable the forced memory cap, increasing the chances your build may crash
Any changes you make to the "setup.ini" must be made with FPSC closed first.
There's no problem that can't be solved without applying a little scripting.