Check out our Discord, Facebook, Twitter, and Twitch to get the latest development news and updates.
Hello everyone!
Thank you for the patience over the last 24 hours while we have worked to make a more stable environment for the game.
- Yesterday, we were able to rapidly scale up game servers along with demand which worked for a bit until the next layer of issues were revealed.
- We found that game servers could get in a state where they appear to not have players on them and would be cleaned up (closed) incorrectly, causing players on those servers to get dropped; that is now corrected
- We hit a cap in how certain databases were able to scale, which would cause slow response time to character & quest data loading across the board. We were able to upgrade those databases to much beefier versions during the maintenance and are no longer experiencing slow database responses.
- During the maintenance we also cleaned up cross-service messages that are not actually required and were eating up capacity; our available server capacity was not supporting nearly the number of players it was intended to before this change
- As the night wore on and greater volumes of sustained players have been online a new class of issues has been uncovered. Race conditions can cause newly spun up servers to be sent all the travel requests for everyone online, rather than just the amount of requests the server was designed to handle. We are implementing changes to the load balancer now that will stop this from happening.
- Those code changes take time to get right, and test to ensure we're not making things worse. In the meantime we have been band-aiding the issue by adding egregiously more capacity than is called for given our current player volumes - which only helps a little given the nature of the current bug.
We are working to resolve server issues that cause players to disconnect frequently or disconnect when they are zoning/traveling throughout the game. This includes players who are attempting to zone from the character select screen to any place in-game.
The team has tirelessly been working to resolve these issues through the night and we are continuing to work on them throughout the day. For the latest information, check out our Discord where we are continuing to update our community about our progress as often as we can.
- Echtra Bean