Hoi chummers,
Many of you have been first hand witness to the frantic post-launch struggle of Shadowrun Chronicles as we patched time and time again only to still end up with a lag when playing. Now to the players this looks like we are constantly running against the same wall until our heads bleed and have only managed to make small dents in said wall. From the inside however, it is a completely different story and we wanted to give you an inside perspective on what is going on.
In the last 6 days we have applied an increasingly large number of fixes and changes, from optimizing garbage collection, sequential threading to deep-copy of data to wrenching open our middleware, which actually contributed to our issue in slightly invisible ways (think of middleware as an engine under a hood, so you don’t always see what is going on exactly but you can hear the engine stuttering)
Each of the changes and fixes was profiled and supposed to bring a good amount of improvements – and it did – we are winning the battle against lags millisecond by millisecond, but it is a war of attrition. This is exacerbated by the amount of time we spend in lag and new players getting bad impressions while our Steam review ratings plummet. Now we have a simple but cruel choice: Do it properly and test a solution for days while our game rots away in lag-land or pushing a fix online with a minimal period of testing, which however results in a bit of profiling and a sanity check if the game explodes immediately or shortly after starting the servers - tempered by a judgement call by our coders of “how dangerous it is versus how likely to fix our problem”. So we send out an annoyingly high amount of patches and updates with annoyingly small effect and are basically really field testing things on the live servers, which we all know we should not do, but then we are between a rock and a hard place. This is why online games always launch as beta…
All this time, our brave coders fight invisible foes such as a maximum user cap in a license that then puts the brakes on certain parts of the performance and carry the burden of working above and beyond the call of duty while being rained with derision by angry customers. The rest of the team watched in agony and frustration as we could not do anything to help and all other things took a back seat to solving the problem while our sales and our users patience dwindles away. Imagine the morale in the team! And still we worked at night, with the community team taking the brunt of the damage while taking turns to be there 24/7. I want to personally thank them and the remaining team for enduring and persevering. You guys and gals are the finest team I have had the privilege to work with. Even though we fragged this up you have been exemplary in every aspect.
So what now? Well, yesterday’s update resolved a lot of issues (reducing frequency and of lags), but foreseeably opened others exploding our servers after around 3 hours of operations. Today we rolled out two server updates – the second one of them dealing with the infamous garbage collector and settings. As this pushed CPU load to a baseline of around 30% and lag spikes appearing again, we reverted the changes…only to see lag creep in 2 hours into operations. So now we relaunch servers every two hours…because this is the only option we have.
Tonight we have finally accepted that “hoping this will be the fix that really does it” is simply wishful thinking. We have run out of silver bullets and we will focus on making the frequent server restarts automated and manageable, creating schedules and having automated in game announcements preferably everyone can see, so everyone can live more comfortably in the emergency state. This should create some air for us to breathe and look at our data and find new options to improve performance.
As a personal note, many of you – even those who have been critical of the game – have been incredibly supportive and understanding in the past days. Helping us, explaining things to others and cheering us on, honoring the work and dedication we have put in. This literally is what has got us through this week (other than coffee and sheer stubbornness) and we are deeply grateful for that.
So let’s see what the next days may bring and keep our fingers crossed.
Oh and Legend Run Event No.3 with none other but Jordan Wiseman himself happens on Friday 3 pm PST or Midnight CEST! Lag or no lag!
See you in the shadows
Primetide