We are currently experiencing some server issues resulting in crashes and redeploys. We’ll do our best to keep you informed of what is going on and the progress of fixes!
For the most up-to-date information on server status, please follow the H&G Server Status Twitter feed - https://twitter.com/hgserverstatus.
Update: Wednedsay October 25th, 2017 – Noon
What do we know, and what are we doing?
We have rolled out an optimization and we believe to have found the cause of the issue. However, we will need to monitor the servers over the coming weekend before we feel safe announcing the servers fixed.
-----------
Update: Monday October 16th, 2017 – Noon
What do we know, and what are we doing?
We are happy that there were no server crashes this weekend. However, this is not the same as having identified a solution yet. We are adding a few more tools to analyze the servers alowing us to hunt down the cause of the issue.
Next update will be Monday 23OCT17 around Noon.
-----------
Update: Friday October 13th, 2017 – Noon
What do we know, and what are we doing?
We have updated a server framework to improve logging further. We are on the right track to identify a solution, but as it is a complex subject, there are a lot of different places in the software we need to examine in detail and update if necessary.
How do I know if this affects me?
We have not seen any crashes since the last update, but continue to monitor the situation while building a permanent fix.
When can we expect a permanent fix?
We are working on a permanent fix. We know the issue only shows up under heavy load, and we’re standing by to restart any servers that might experience issues.
What are you going to do to make up for this?
All active Veteran Memberships are paused when servers are offline. All active Ribbon Boosters are completely reset to be used again at your convenience. When we are past this episode and the servers are stable again we promise to make it up to you guys!
-----------
Update: Wednesday October 11th, 2017 – Noon
What do we know, and what are we doing?
Unfortunately our temp fix did not prevent a crash last night, we’ve added more monitoring in today’s hotfix and are keeping a close eye on it. We are still working on a long-term fix for the servers.
Next update will be Friday 13OCT17 around Noon.
-----------
Update: Tuesday October 10th, 2017 – Noon
What do we know, and what are we doing?
One crash last night, identified as an update gone wrong that unfortunately pulled down a server causing a restart but not related to other issues. The previous issue has not re-appeared, the temp fix is holding and we are working on a long-term fix for the servers. See update from yesterday for more info on what caused the issues.
Next update will be Friday 13OCT17 around Noon.
-----------
Update: Monday October 9th, 2017 – Noon
What do we know, and what are we doing?
No crashes over the weekend. The updates introduced Wednesday and Thursday last week works as designed and keeps the servers from running out of memory. We have identified, what we believe is, the main cause of the server issues and we are working on a proper fix. For the technically inclined the issue stems from the amounts of locks concurrentdictionary can create when used to ensure our object-index-table is thread-safe, this can sometimes spike from the normal < 300 to more than 60 mio. Which kills the memory on the server.
Next update will be Tuesday 10OCT17 around Noon.
-----------
Update: Friday October 6th, 2017 – Noon
What do we know, and what are we doing?
We see that the fixes implemented late Wednesday and Thursday reduced the risk of crashes. We are constantly monitoring the situation and keeping an eye on things. Our tech. partners are also ready to help with any issues that might show up over the weekend.
Next update will be Monday 09OCT17 around Noon.
-----------
Update: Thursday 05OCT17 - Noon(-ish)
What do we know, and what are we doing?
We are still working with outside experts to help us go through data and eliminate causes for errors. This is slow work, but this additional manpower helps speed things up a bit. We have implemented a few workarounds that reduce the risk of crashing, but have not yet identified the specific causes of the crash.
Next update will be Friday 06OCT17 around Noon.
-----------
Update: Wednesday 04OCT17 - Noon
What do we know, and what are we doing?
We have eliminated a number of possible causes and are going through memory dumps. Unfortunately this is very slow work with these 128gb+ memory dumps. We are working with experts in the field to try and both speed debugging up, and also help ID and prevent these issues from happening.
Next update will be Thursday 05OCT17 around Noon.
How do I know if this affects me?
If you were online and playing during the crashes you may have experienced disconnects, battles ending prematurely if the server crashes and seeing the “Server Down for Maintenance” screen.
When can we expect a fix?
We hope to get more useful information out of the crash data, and hope that can help us pinpoint the issue. We know the issue only shows up under heavy load, and we’re standing by to restart any servers that might experience issues.
What are you going to do to make up for this?
All active Veteran Memberships are paused when servers are offline. All active Ribbon Boosters are completely reset to be used again at your convenience. When we are past this episode and the servers are stable again we promise to make it up to you guys!
-----------
Update: Tuesday 03OCT17 - Noon
What do we know?
The debugging info we added during last week made it possible for us to get a lot more data out of the crash and we are analysing data as fast as we can.
What are we doing to fix the issue?
We have caught the crashes that happened and received the data from them, including memory dumps and stack traces etc. and are digging through them.
Whats is new since the last update?
We saw a few crashes and are comparing the info from the debugging tools.
How do I know if this affects me?
If you were online and playing during the crashes you may have experienced disconnects, battles ending prematurely if the server crashes and seeing the “Server Down for Maintenance” screen.
When can we expect a fix?
We hope to get more useful information out of the crash data, and hope that can help us pinpoint the issue. We know the issue only shows up under heavy load, and we’re standing by to restart any servers that might experience issues. Next update will be Wednesday 04OCT17 around Noon.
What are you going to do to make up for this?
All active Veteran Memberships are paused when servers are offline. All active Ribbon Boosters are completely reset to be used again at your convenience. When we are past this episode and the servers are stable again we promise to make it up to you guys!
-----------
Update: Monday 02OCT17 - Noon
What do we know?
We saw a crash on Friday and one Sunday night. The debugging info we added during last week made it possible for us to get a lot more data out of the crash. One issue has been identified and a hotfix is under way. But we know that this was not the only reason for the crashes.
What are we doing to fix the issue?
We have caught a crash and are analyzing the data we got from it. Hopefully this can give the programmers better insight into what is misbehaving.
Whats is new since the last update?
We saw two crashes over the weekend, and the debugging info added last week gave us much more information to work with.
How do I know if this affects me?
If you were online and playing during the two crashes you may have experienced disconnects, battles ending prematurely if the server crashes and seeing the “Server Down for Maintenance” screen.
When can we expect a fix?
We hope to get useful information out of the crash data, and see if that can help us pinpoint the issue. We know the issue only shows up under heavy load, and we’re standing by to restart any servers that might experience issues. Next update will be Tuesday 03OCT17 around Noon.
What are you going to do to make up for this?
All active Veteran Memberships are paused when servers are offline. All active Ribbon Boosters are completely reset to be used again at your convenience. When we are past this episode and the servers are stable again we promise to make it up to you guys!
-----------
Update: Friday 29SEP17 – Noon
What do we know?
We continue to monitor the servers to get more info. No errors has been present the last 48 hours, but we have yet to pinpoint why the errors occurred.
What are we doing to fix the issue?
We have applied a number of extra debug tools to our setup to enable us to catch the issues should they reappear giving us more data to analyze.
Whats is new since the last update?
We no longer alternate between two server setups but are ready to do so again should the issue reappear.
How do I know if this affects me?
We have not seen any real crashes for days so currently you should not see any difference.
When can we expect a fix?
We still have to be careful and be sure the servers are working correctly before we declare them ‘fixed’ – for now we consider them stable but not fixed. Next update will be Monday 02OCT17 around Noon.
What are you going to do to make up for this?
All active Veteran Memberships are paused when servers are offline. All active Ribbon Boosters are completely reset to be used again at your convenience. When we are past this episode and the servers are stable again we promise to make it up to you guys!
-----------
Update: Thursday 28SEP17 – Noon
What do we know?
We continue to monitor the servers to get more info – the more we look for errors the more they apparently hide. No errors has been present the last 24 hours.
What are we doing to fix the issue?
We’ve analyzed the memory dump we took, and can conclude, at least from that specific dump, that we have no apparent memory leaks or fragmentation issues that could be the cause of the issues we’ve been experiencing. We’re currently gathering data on thread usage and thread allocations to see if there’s anything to be found in that area.
Whats is new since the last update?
We continue to alternate between two identical server setups that is however not something you as a player will notice. Errors has not re-appeared.
How do I know if this affects me?
Last couple of night we have not seen crashes. So currently you should not see any difference.
When can we expect a fix?
We have to be careful and be sure the servers are working correctly before we declare them ‘fixed’ – for now, next update will be Friday 29SEP17 around Noon.
What are you going to do to make up for this?
All active Veteran Memberships are paused when servers are offline. All active Ribbon Boosters are completely reset to be used again at your convenience. When we are past this episode and the servers are stable again we promise to make it up to you guys!
-----------
Update: Wednesday 27SEP17 – Noon
What do we know?
We continue to monitor the servers to get more info – there do not seem to have been any changes i status since yesterday.
What are we doing to fix the issue?
We are still at work analysing the data we got from the memory dumps to pinpoint the problem. It does seem that memory usage alone is not to blame. Investigation continues.
Whats is new since the last update?
Currently we alternate between two identical server setups that is however not something you as a player will notice.
How do I know if this affects me?
Last couple of night we have not seen crashes. So currently you should not see any difference.
When can we expect a fix?
We have to be careful and be sure the servers are working correctly before we declare them ‘fixed’ – for now, next update will be Thursday 28SEP17 at Noon.
What are you going to do to make up for this?
All active Veteran Memberships are paused when servers are offline. All active Ribbon Boosters are completely reset to be used again at your convenience. When we are past this episode and the servers are stable again we promise to make it up to you guys!
-----------
UPDATE: SEPTEMBER 26TH, 2017 – 12:00 CEST
What do we know?
We are monitoring the servers closely to get more info. After moving to a new server, we have not seen any crashes overnight, but this is not a secure indication of the issues being fixed.
What are we doing to fix the issue?
We are analysing the data we got from the memory dumps to see if we can pinpoint what went wrong and if it is a hardware issue. Then see if we can identify this type of errors earlier so we can fix it faster if other hardware develops same issues. We have added resources to the servers to ensure that this was not caused by hardware limits
Whats is new since the last update?
Currently we’ll continue to run on the secondary server (it has the same specs as the main one).
How do I know if this affects me?
Last night we have not seen any crashes. So currently you should not see any difference.
When can we expect a fix?
We have to be careful so we know the servers are working correctly before we declare them ‘fixed’ – so for now, next update will be today Wednesday 27SEP17 at 12:00 CEST.
What are you going to do to make up for this?
All active Veteran Memberships are paused when servers are offline. All active Ribbon Boosters are completely reset to be used again at your convenience. When we are past this episode and the servers are stable again we promise to make it up to you guys! We have seen some great suggestions in the forum and we hear you.
Sorry for the inconvenience.
----
Update: September 25th, 2017 – 16:00 CEST
What do we know?
Over the last week we have seen a number of server crashes and general instability in the servers. Some of these issues has been fixed in hotfixes, but not all. Defective RAM blocks has been ruled out.
What are we doing to fix the issue?
We are still eliminating areas that could be the culprit. We have added more debugging tools and we are taking down the main war server (without it crashing) to get a complete memory dump that we can analyze. Meanwhile the game will continue to run on the secondary identical war server with the same version of the game.
Whats is new since the last update?
We have eliminated some sources of error and are now focusing on getting at server memory dump on the operating table for detailed analysis.
How do I know if this affects me?
You may have experienced disconnects, battles ending prematurely if the server crashes and seeing way too much of the “Server Down for Maintenance” screen.
When can we expect a fix?
We are working as fast as we can on gathering more information while simultaneously standing by to restart any servers that might crash.
What are you going to do to make up for this?
All active Veteran Memberships are paused when servers are offline. All active Ribbon Boosters are completely reset to be used again at your convenience. When we are past this episode and the servers are stable again we promise to make it up to you guys!
Sorry for the inconvenience.
-----------
Update: September 25th, 2017 – 12:00 CEST
What do we know?
Over the last week we have seen a number of server crashes and general instability in the servers.
What are we doing to fix the issue?
We are eliminating areas that could be the culprit. And adding more debugging tools to help pinpoint the issues.
Whats is new since the last update?
We have changed the physical ram blocks on servers to be certain that the reason wasn’t hardware specific.
How do I know if this affects me?
You may have experienced disconnects, battles ending prematurely if the server crashes and seeing way too much of the “Server Down for Maintenance” screen.
When can we expect a fix?
We are working as fast as we can on gathering more information while simultaneously standing by to restart any servers that might run out of memory or crash.
What are you going to do to make up for this?
All active Veteran Memberships are paused when servers are offline. All active Ribbon Boosters are completely reset to be used again at your convenience. When we are past this episode and the servers are stable again we promise to make it up to you guys!
For the most up-to-date information on server status, please follow the H&G Server Status Twitter feed – https://twitter.com/hgserverstatus.
Sorry for the inconvenience.