Today is the day!
We have finally fragged the lag issue it seems when our servers will be patched one last time at 15:00 alongside some other fixes.
We don’t want to jinx it, there may still be gremlins lurking around, but we seem to be back to normal. Today, with our hopefully last lag related server patch, you will find your well-deserved apology gifts on your inventory:
- 3 Flamethrowers (so you can flame enemies and think of us)
- 1 additional character slot
- 2 new Bioware augmentations (Zero Essence, usable for awakened characters as well)
Many people have given us good advice on how to handle the issues and told us what they suspected the culprits were. As it turns out, we had three main issues and tragically two of them have very little to do with our team’s ability to code and code well. It is both more tragic and more comical tale (that is if it hadn’t affected our rating s and sales so massively. Maybe one day, looking back, we can laugh without bitterness about it).
The first issue was not enough optimization in certain areas – as described previously, the game had been working so brilliantly from a technical and performance standpoint, that we did not dedicate too much resources to it. We knew we needed to do more optimization, but it did not seem to be a big concern. If it hadn’t been for the other two things, we might have been right.
Issue two was one of our middleware licenses actually putting the break on our servers once a certain number of concurrent users were active (which we were not aware of). Above that number it would slow down throughput, so all our other services (which kept listening to certain commands or pushing things to the client) slowed down with it. As some of them were serialized, meaning they were acting like a chain, they were actually really waiting for something to happen from the previous link of the chain, multiplying the effect of the license slowing down the speed at which commands were coming and going and putting other services like load balancing and thread management into overdrive, crashing CPU. Our servers overheated with waiting and managing that slowness, absurdly enough. This was fixed easily enough once discovered but it took us a few days to identify this weird behavior and the culprit.
The third and most tragic point has to do with the letter D. As in “enableD=garbage collector”. While we optimized the garbage collector (which cleans up “leftovers” on the server, so they don’t clog memory) and following some of the suggestions we got from our code-slinging community members, it did not seem to do the trick. We dug deeper and deeper into garbage collector settings but to no avail – each time the game would just use up the memory available and despite further optimization efforts this did not change considerably. The reason was finally found to be a missing “D” in the config file, which read “enable=garbage collector” instead of enabled. Of course we had looked at every line of this configuration file what must have been a hundred times, but you know how the brain tends to just create the word it expects to see, especially when you did not have more than two hours of sleep for days fighting a lag that was killing your game? This was one of the cases (and enable and enabled are not that far apart). Once we found it, we hit ourselves over the head repeatedly with blunt objects.
All three issues combined killed us in a most unfortunate way. Now with those resolved we actually have 10.000 concurrent players in our mass test and the servers don’t even get warmed up about that. We are cautiously optimistic we are ready to declare the lag fragged for good!
Now we can get back to actually improving gameplay and adding end-game missions and rewards! Please spread the good news and let your friends and Steam or Metacritic ratings know, so we can try and get the game off the ground again.
Thank you all for your patience and support in this really rough patch of our life as a developer.