At first, we would like to apologize for the recent performance issues and problems you may be experiencing on new backend servers. Please be assured that we are working on solutions to improve performance and gameplay experience with high priority.
With this producer’s note we would like to provide some more detailed information to let you know which challenges we are currently facing and which measures already have been carried out, are in our minds and are planned to be deployed as soon as possible.
At first, we need to clarify that we have to distinguish between three main issues we are facing right now.
New Backend Server Outage (15 June 2022)
The recent server outage on Tuesday, 15 June 2022, was completely disconnected from the other issues related to the new backend.
Long story short, we got a false error notification from our monitoring which stated that we had a problem with our key value storage but it turned out that in reality some database proxies had some trouble.
When we found out that the error notification was false, we already deployed measures against the reported issues which, unfortunately, caused a high latency. Ironically, exactly the opposite of what we wanted to achieve. Anyhow, this was rolled back and the real problem with the database proxies could be solved and, eventually, the server performance went back to normal.
Nobody is perfect, mistakes can happen. We can only truly apologize for the unfortunate circumstances caused, thank you for your patience, and ask for your understanding.
Performance of New Backend Servers
Starting to migrate servers went quite smooth in the beginning. The migration of small game worlds caused smaller issues which could be solved rather quickly. The more we advanced, the more confident we got and decided to start migrating bigger and more active servers.
Unfortunately, we underestimated the impact of the data amount which needs to be loaded simultaneously and its impact on the server load. As a result, the new backend servers’ performance started to decrease.
As a counter measure we upgraded the server power by adding additional game servers. Furthermore, the code base is being reviewed and refactored to identify and optimize parts of it that are expensive in terms of performance.
To see if our actions indeed have a positive impact on the server performance, we integrated specific methods to measure performance to be able to track if an optimization leads to an efficient improvement. The optimization is being done in iterative steps which enables us to monitor the possible improvements more closely.
What are the measures in detail? Redundant data is being reduced, old and no longer be needed data will be cleared on regular basis via scripts. The program code is being optimized to shorten response times and we are going to change the way how temporary session data is being saved. The same data is being required for many parts of the game. With an intelligent memory caching we expect to achieve an extremely better performance.
In some areas we are facing some limitations, though. Currently, we are dealing with two different code bases which need to be compatible. This hinders us to execute some improvements which can only be done once all servers have been migrated.
Bugs Occurring on New Backend Servers
The last topic, we would like to discuss are the bugs which are currently occurring on new backend servers.
While we are focusing on improving the server performance first, the bugs you have reported are not being kept without any attention. We assume that the majority of the occurring issues will no longer occur once the server performance is on a stable level.
Nonetheless, some of our developers look intensively into identifying the sources of the reported bugs and are working on solutions where we could see that the problems are not only connected to performance issues. Again, the code base is being revised and refactored where we think it could help to fix a bug.
We would have wished that the server migration would have been done trouble-free. Sadly, the reality shows a different scenario. It is as annoying to us as to you but we have to face the current situation and deal with the challenges with high priority to make your and our lives less stressful.
Again, we appreciate your patience and thank you for your understanding and support but can also understand your disappointment that you are currently facing some uncomfortable situations while playing the game. We are on it!