The search for the needle in a haystack hopefully have found an end. After several unsuccessful approaches to get the server performance under full control it seems that we finally could track down the root cause.
It turned out that the server CPUs of some of our game servers are of an earlier generation (ZEN2) which seems not to have the required memory management features for Linux. This lead to an unfortunate way of memory usage, causing 100% load at peak times and, thus, the server performance went down and playing users experienced severe connection problems and lags.
To solve this problem we are going to replace the currently used CPUs with CPUs of the next generation (ZEN3). We expect the hardware exchange to happen next week but are dependent on the server hosting company and, thus, at this point of time we cannot provide an exact date when this will happen.
In the meantime, we are currently running a service which restarts PHP and clears allocated kernel memory every hour. This seems to help but is not the desired solution.
We thank you for your patience and, again, are truly sorry for the poor server performance of the last weeks which kept you as well as us stressed out. Let’s hope that we can close this chapter once and for all.