Hey Bolle, that slow down totally happens for me too. The more media that is added to my game, the time it takes to initially load is exponentially increased. This is not a problem after loading when I am using library media to build scenes, but slows down tools such as the sprite sheet slicer, which require reloading of the media each time they are used.
Because of this, I perform any actions such as sheet slicing in a smaller project file whose only immediate media files are those being tested/configured. I then switch my now-visual-editor-friendly media over to the project I intend to use it on, so that once I wait through the tedious load I can get right to building scenes. The load also becomes less tedious with a cup of coffee and a side activity
My guess is that the algorithm used to generate the metadata related to the editor’s media library is brute force in nature. In addition, if the metadata related to editing each project is not held in a persistent state after each session, that brute force algorithm would have to run each time the project is opened. I believe the visual editor is, in a way, “reinventing the wheel” each time you start it. I may be wrong though!
There are some other quirks I run into using the visual editor. Like if I copy an assembled animation, my editor crashes upon attempting a paste operation. But if you ultimately use the editor as a rudimentary tool that you can supplement with your own coded utilities, which in your case may be a script to help manage media, then I find it can be a handy piece of software!