List[ ] in PowerShell works IDENTICAL to how it does in C#., essentially this is identical to std::vector< > in C/C++
In essence what it does is hides the Complexity of creating a New Array then filling it with all the existing data and updating the pointer references for you... but you're still essentially using a Static Array.
As such Adding / Removing Elements does come at a (noticeable) performance hit.
Now the beauty of Contiguous Arrays as opposed to a Linked List Array (which is an Array consisting of Structure that points to the Previous and Next Element) is that they're exceptionally fast to perform Sequential Operations on., such-as Search and Sort., but this comes at the cost that there is overhead when you Expand or Contract them.
What I'd recommend doing is setting the Values to "Null", that you no longer need... Sort the Array; then when you have a certain number of Null Objects (say 10) you can then resize the Array to just remove them.
Remember that you can use Operators (+ - += -=) on Arrays rather than the Elements to perform operations.
So $myArray -= 10 would remove the last 10 Elements and create said new Array for you in a single operation (rather than individual elements); and this works out faster.