Hey everyone,
As part of our investigations into server performance, we have identified another potential cause for some of the recent server crashes being experienced and have now been able to verify a further fix.
While this fix may not resolve all instances of server crashes - as there are many different factors that could be causing crashes for different servers which we are continuing to investigate - we believe it is the likely cause for some of the recent crashes due to the way in which the server handles new RCON connections.
Previously, the Hell Let Loose server would create some data directly in response to receiving an RCON connection request. Since changing servers to run multi-threaded, as part of a previous improvement to rubber-banding in vehicles, that request could come in on any thread.
This meant that it could happen while Unreal Engine is doing what is known as a garbage collection. It is not valid to create objects while garbage collection is happening, and so if this happened, it would cause the server to crash. With increased amounts of RCON connections to a server, the probability of having this timing overlap with the garbage collection would increase, which meant server crashes could become relatively frequent.
We have now ensured all object creation is deferred to the next server tick, on the main thread to make sure that this cannot happen. This has been tested with a tool that creates a huge number of connection requests to the server, continuously, to battle-test its robustness. Previously, this tool could crash a server in a matter of minutes, now the team has been able to play for hours at a time while the server remains stable.
This fix requires a client-side update, but does not have any changelog items, and will now be going live as a hotfix.