Hi all!
It's been a bit of a content drought for you all as our devs have been mostly focused on optimization and framerate issues. We've got some betas out now that people are playing with, and the results on the performance front look really encouraging. We're still tracking down a few crash issues caused by the massive code refactor, so we're not ready to release quite yet, but you should be able to play around with the new tech soon!
But that's all coding. As the devs have been neck deep in the engine, I've been hard at work plotting out the design portion of the next several contents worth of NIMRODS updates so that once the refactors are done, and our devs can dedicate nearly all of their time to content again, they'll have plenty of design docs to chew through! Expect new biomes, new enemies, new augments, new chassis, and a new optional boss with mechanics that I'm sure will take most of you all by surprise!
But planning for the future can only go so far. There comes a point at which you realize that you're planning so far ahead that landscape of NIMRODS might change drastically between now and then. I hit that point a few weeks back. So instead of planning out another update into the future, I pled for the precious time of one of our devs, the mysterious force known only as zzzzzz, and he charitably donated a few days to one of our most ambitious design projects yet. But before we unveil that, let's talk a little bit about the way NIMRODS was balanced.
The Old Way:
Previously, whenever I came up with a new augment, I would try to estimate the damage boost that augment would give using a very complicated formula, then tweak the numbers so that the estimated DPS would come out to a certain target amount. Our goal was for a tier one augment to multiply your damage by ×1.4, a tier two or three augment to multiply your damage by ×1.3, and a tier four augment to multiply your damage by ×1.45. In some cases, estimating DPS was easy. For instance, the Cryo Magazine multiplies your Bullet Damage by ×1.4. This pretty strongly suggests that if you take it, your DPS will go up by about... well... 1.4. Seems reasonable, right?
Others are a little harder. What does multiplying your Fire Rate by ×1.4 do? Well, while you're firing, it might multiply your damage by ×1.4, but you're not always firing. Sometimes you're reloading. Therefore, the actual DPS boost is much lower. The DPS bonus from Fire Rate actually depends on your Reload Speed, which is different for each chassis. Short-reload weapons like the Assault Rifle benefit more from Fire Rate than slow-reloading weapons like the Hydra Minimissile Swarmer. In cases like that, we tried to balance the augments for an average-case scenario, which intentionally made fire rate better on some weapons than others.
Others stats are even harder than that. Like, how about Penetrations? You'd think that if you double your Penetrations, you'd double your DPS, because your bullets are hitting twice as many things. But those of you who have been with us for a while and have a lot of runs under their belts will know that that's not the case. For instance, what happens if you go from 10 penetrations to 20? Is it really reasonable to suggest that every time a player fires, they can line up 20 enemies in a row?
Previously, I'd been estimating the DPS of augments assuming what I call an "arbitrarily target-rich environment," which is fancy-speak meaning that the player is constantly surrounded by infinitely thick enemies. In this case, the player never misses, and their shots can meaningfully pierce any number of enemies. Why did I assume that? Because we just didn't have any good data to show what we should use as an "average case" scenario for the player.
But this got me into huge problems when I did the Piercing Ammo! The Piercing Ammo takes a penalty to its Reload Rate in favor of a massively increased Penetration count. Sounds fine on paper, but everybody in testing said that the Piercing Ammo felt incredibly weak and not fun to play with because the Penetrations weren't compensating for the Reload Speed drawback. Why? Because I'd assumed an arbitrarily target rich environment, and the player's actual experience—particularly in the early game, when you're likely to take piercing ammo—didn't reflect that. Eventually I threw my formulas to the side and just arbitrarily cut the reload penalty to less than half of what it was. That's the version you're playing with right now. It felt bad to depart from my DPS calculations and just guess what the right answer was, but my calculations clearly weren’t cutting it because I had garbage assumptions. It was at this point that I realized we needed better data.
My first thought was to try to ask zzzzzz for some analytics we could use that would keep track of how many enemies, on average, a player was hitting with any given number of penetrations… but the more I thought about that approach, the more I realized what a rabbit hole that was. Maybe we could have gotten that data through analytics, but then what about weapon inaccuracy? What about split bullets? What about the Effect Size stat? Or any other number of calculations that assumed an arbitrarily target-rich environment? Should I really curse our devs, busy enough already, with the task of developing a complicated analytic system for every stat that I didn't have good numbers on?
We needed a better general case solution. Luckily, we found one...
Enter Analytics:
Instead of trying to get better data to more accurately predict what we thought that the DPS would be, I reasoned, what if we just actually measured the player's DPS with analytics? If we could figure out which augments were too powerful and which were too weak, then we could rebalance the augments to buff the strong ones and nerf the weak ones without needing to calculate their hypothetical DPS at all! I know it sounds obvious when you read it like that, but we hadn’t done it yet because it’s more complicated in practice than it sounds. There are currently 125 billion combinations of chassis and augments in NIMRODS... and that doesn't include the effects of genes, trophies, or tunings! To figure out which augments were too powerful and which were weak, we'd need to take a comparatively small amount of data - a paltry few million samples or so - and use it to estimate what effect each augment would have on any given build.
After a little research and a trip to visit my old university stats professor, I felt like I had a good approach: Log-Transformed Multiple Linear Least Squares Regression. It might sound complicated, but it's actually a really newbie statistical method. There might be a better one out there, but sadly, Fiveamp is an order of magnitude or two too small to have a dedicated data scientist, so we’ve all got to make due with our respective undergrad stats courses.
But despite it's simplicity, we were hoping that feeding this algorithm the telemetry data we’ve been collecting since last patch would give us good enough numbers to help us make some meaningful balance improvements. Specifically, the data we collect includes, amongst other things, the player’s damage: For every run, we take many snapshots of the player's current build and how much damage per second they're doing with that build.
This gave us the raw data we needed to do some serious number crunching. Initial tests with a few hundred samples seemed grim. But increasing the number of samples to 30,000 led to some highly encouraging results! So we tried it on all 1,000,000+ samples... and it completely fell apart. (For you data nerds out there, what we mean by "fell apart" is that our R-Squared went from 0.170 to 0.006.) For a while, we despaired, until we realized what was going on...
Anomalies
Take a look at this graph, which represents how much damage players are doing compared to how much we expect them to do with that combination of augments:
See that one point at the top, there? That player was doing a hundred duodecillion times as much damage per second as they "should" have been doing given our best estimates of how much each augment should have contributed to damage.
This is a big difference, needless to say. We looked into it, trying to figure out where this number came from, and their build didn’t seem particularly broken. Eventually we came to the conclusion that this fine fellow wasn’t necessarily playing NIMRODS, but rather a creative reinterpretation of NIMRODS. In other words, the code they was playing with may not necessarily have been the code as we wrote it.
This is fine. You’ve all purchased the game, so you should have the right to play with it the way you see fit. But it does wreak havoc with our analytics. Eventually we decided to just throw out any runs where at any point somebody did more than a 20,000 times the damage they "should" have been dealing. This restored balance and harmony to the world of our linear regression (R-squared went all the way up to 0.92, which is wonderful) and our numbers made sense once again.
I'm still nervous that if we do this sort of outlier culling, then maybe we'll one day miss an actual broken build (such as when players on our discord server figured out how to make an infinite loop between increasing hp and bullet damage) so we’ll have to find a better way to identify outliers caused by people playing our game in unintended ways like this.
Results:
In the end, we got a fancy spreadsheet giving a "Damage multiplier" for each of our augments. In other words, if you take a given augment, it will multiply your damage by roughly this much:
You can see a full spreadsheet HERE
There are some clear winners and losers, mostly centered around the "arbitrarily target-rich environment" fallacy. Homing turned out to be massively OP. Why? Because if you're already hitting with every shot, homing doesn't help much, so homing was given other modifiers to increase its damage further. In contrast, stuff involving penetrations is underperforming, for all the reasons explained earlier.
There are some cool finds, like the Sniper Barrel dealing much less damage than I'd anticipated. (Maybe when people need to move fast to avoid monsters, they stop shooting, lowering their DPS?) There are also some false positives; for instance, anything that grants XP drop rate or XP collection rate bonuses will look underpowered because it contributes to your character's ability to level up in a less orthodox way than "Damage ➡ Dead monsters ➡ XP". But even with a few caveats, this information is a gold mine. Based on our findings, we've made changes to dozens of augments in the coming patch, mostly buffing underperforming augments but also nerfing a very few overperforming offenders. We’re keeping our changes small this time around as we "break in" our new statistical methods, but as we refine and gain more confidence in our new tool, we'll be a little more daring with our balance adjustments while we keep iterating until we get the balance we're looking for.
Thanks for helping us out on our journey, and we can't wait to release this next patch for you all!