I guess I have come a long way from the beginning, but I have been working on this for about a month, now. I am getting really frustrated, so I came here to see if anyone has some suggestions to perfect this. Plus, its always nice to vent and get your thoughts out.
It was really bad at the beginning. My paths were taking from 10seconds to about 20MS being the fastest. Trial and error got me down to about .5MS-2MS.
I use a Navmesh based A* pathfinding. To make the Navmesh I use Blender's Navmesh function - quite handy. I do not use a heap, I use AGK's sorting. My testing showed it was faster than using a heap. I tried out a navmesh with less nodes, and for some reason I got slightly worse performance not just in terms of the pathfinding, but also my agents would not surround my player as well. Its possible my algorithm was including too many neighbors, thus slowing it down. Testing and adding neighbors is a huge bottleneck for A*, if you are unfamiliar.
Currently, to surround the player I have 8 nodes that surround the player that can be occupied, once an agent chooses that node the node is no longer available(or if the node is inside a wall). The agents move to those nodes. I currently have a bug that doesn't make the agents land perfectly on those nodes. I have been unable to track down the errant code/algorithm.
As far as trying to make the agents avoid each other:
I want to preface this by saying this might need more trial and error on my part:
I have tried Raycasting - it usually misses any obstacles. I wanted my agents to "bounce" off each other if they got too close. Sadly, the raycast is very unreliable. It probably needs further refining(ie testing each individual object). I currently send out 5 rays. Straight, MidRight,Right,MidLeft,Left. It still failed to detect an obstacle 99% of the time. I am sure I am doing something wrong, I just haven't figured it out. I got it to work perfectly using it on the player to avoid obstacles.
SphereCast provides too random of a point to use.
I have tried ObjectSphereSlide - this has potential but the agent is likely to get stuck on the obstacle. It would be really nice for the developers to show this in use. I don't know how to use this properly.
Note: The player does not avoid obstacles, because I wanted to test the agents ability to find the player.