Hey guys, we've been working on several server improvements concerning multithreading! This post will get a little technical again, so brace yourselves!
The reason why the update is taking so long, even though we promised otherwise (we know, sorry for that!) is that multithreading is a very complex issue that can easily introduce bugs when not done correctly.
Multithreading
We've improved the multithreading handling of the server. With the upcoming update, the server will use all worker threads for a single update of a sector.
The Avorion server uses a thread pool for updating sectors, and then gives work packets to these threads.
Before this update, the server created one work packet per sector update, meaning sectors were updated in parallel, but each sector was only updated by a single thread. Due to technical reasons a server tick/frame can't be finished until all sectors have finished updating, and it often happened that the server had to wait for one thread that took very long to finish, leaving the remaining threads (and thus CPU cores) idle (doing nothing).
For example, take a server with 8 worker threads, 10 sectors in memory and there's a large battle going on in one of them, sector 10. Sectors with battles take longer to update, so let's say that the sector with the large battle takes 70ms on average to update (which is a very realistic number that we've seen plenty of times) and the others each takes 2ms. With 8 workers, sectors 1 to 9 would be finished after 4ms, and then these threads would be idling (sitting around doing nothing) since they didn't have any more work to do, while the remaining thread, that's updating sector 10, would still have to run for 66 more milliseconds. The server can't finish the frame and has to wait for the thread to finish the update, while the other 7 worker threads do nothing, which is an obvious waste of CPU power.
With this update, we've changed the way that this system works. Instead of creating a single work packet per sector, each sector is subdivided into many small work packets that can then all be updated in parallel. This way we can update all sectors in parallel AND have all worker threads work on a single sector.
This is a graph of what sector updates look like now (you should enlarge it or look at it in a separate tab):
This is a profiling image of a server with 7 worker threads and with 2 HUGE faction battles (30 ships vs 30 ships per sector, one near the barrier, both including carriers and about 150 fighters each) going on in 2 different sectors. The X axis is time, while each row represents a worker thread. The colored bars are the work packets that the threads have to work through. An update frame goes from the purple bars [1] to the grey boxes [3], which represent the end of the sector update step. I've included multiple frame updates on top of each other so you get a better idea of how different they can look, even though it's the same 2 sectors each time. Yay multithreading!
Same color means the same kind of work. A few examples: The pale red ones [5] ship AI updates, purple [4] are turrets aiming and turning and blue [6] in the 4th row are fighter AI updates. The green [7] at the end of the frame is update sending and the golden and black ones [8] are collision handling and detection. When there's multiple work packets with the same color above each other, that means that multiple threads are working on the same logical update, but distributed over the different threads.
When there are no bars at some time (the white gaps) then that means that the worker is idle, either because there is no work to do or because a thread from another program has been scheduled by the operating system to do some other work. This is solely influenced by the operating system and we have no control over this. But basically, the denser the colors are packed, the better. These gaps will get more the more different other programs you run on the same machine as the Avorion server.
So what does all of this mean? To make it short: Avorion will run a LOT better on machines with multiple cores! The updates you're seeing here are only about 5ms long - meaning that 2 huge battles in 2 sectors can be updated within, on average, 5 milliseconds! (CPU: Intel i7-6700K with 4 cores @ 4.00GHz & Hyperthreading)
These are the two battles that were going on at the time of the profiling:
Of course there are still outliers (for example when a ship is spawned or destroyed), but even those have been reduced to maybe 3 - 5 times the average update time (5ms means we get outliers from 15 ms to 45 ms every 15 frames or so).
Multithreading Part 2: Async Scripting
We've also introduced a new way of executing scripts (actually this is not true, it's been in since the Alliances update, but we haven't used it yet): Async calls. These are mostly used to asynchronously generate ships that spawn during play time.
While profiling we realized that generating the ship plans take up 80% - 98% of the time required to spawn ships, so in order to reduce server lag while spawning huge armadas ship plan generation is now done asynchronously and the server won't lag as much.
RCON Interface
With the new update we'll also introduce an RCON Interface to the server, with which you can control the server with typical RCON tools. There are plenty of tools at your disposal, so simply pick the one that suits you most.
When is it coming?
Soon! We're done with all the features we want and we'll do several more bugfixes, but then it's ready. We'll also post detailed patch notes once the update goes live.
Have fun!