Hello Riftbreakers!
We have often emphasized that adding a multiplayer module to The Riftbreaker is a very difficult task. The game and its engine have been designed as single-player from the ground up. For the past couple of years, we have been redesigning most game systems and refactoring the game’s code to make online coop possible. Introducing such major changes to The Schmetterling Engine does not come easy. Not only do we have to solve brand-new issues, but we also need to manage the game’s performance. We can’t allow anything we add to negatively impact how the game runs on your end. On the contrary, we want to use this opportunity to improve our performance. Today, we would like to show you what improvements and optimizations we found while working on the co-op mode.
Allowing players to build gigantic bases and fight against a sea of creatures both come at a large performance cost. That cost increases even more in multiplayer - which you can see even on the gif above.
Modifying The Riftbreaker to support the multiplayer game mode introduces many performance overheads. We have used quite aggressive optimizations when the game supported only a single player. For example, we packed or entirely deleted some effects that were taking place beyond the visible playing area. For multiplayer purposes, that is no longer possible. The server needs to calculate everything happening around all players in the world at all times to relay the complete gameplay information to all players. Because of this, many of the engine’s components take significantly longer to compute. It is unfortunate, but sometimes there is no way to avoid this additional cost. However, it’s still not hopeless, and there are other areas where we can look for improvements.
Our goal is to allow you to do all the fun things you could do in singleplayer, but together with friends, and at a good performance level.
We decided to make up the losses in performance by doing a massive optimization pass across all the game’s systems. We set an ambitious goal for ourselves - we wanted the performance in multiplayer to match that of singleplayer. This way, we could guarantee a smooth online play experience while significantly improving the game’s performance when running solo. We created a set of new benchmarks with massive bases and enormous attack waves. Then, we got to work.
BENCHMARK ENDGAME IDLE
During this test, the mech stands in the center of an enourmous base for a couple of minutes. During that time, we check the behavior of the game at times of relative peace. We run all benchmarks at minimum graphics settings, so that the GPU is never a performance bottleneck.
Our first assumption was that our players enjoy building large, complex bases with thousands of buildings. We can infer that when a couple of players meet up to play together, they will build even more monstrous fortresses. We have to be ready for that. A map filled with various buildings, factories, towers, and a massive power grid is a great benchmarking scenario for us since it significantly strains all the game systems. Therefore, we created a huge end-game scenario base and started running tests to see what we could optimize.
This graph shows the changes in performance over the course of our work on the multiplayer mode. Sometimes it gets better, sometimes it gets worse - we always keep tabs on our performance markers.
Game logic simulation is running at a constant rate of 30 updates per second so, our target was to reduce the calculation time for all gameplay systems to 33 milliseconds in total. One thing to note here is that game logic simulation is running independently from rendering. For example, a beefy GPU can render the game at fluid 120FPS while the CPU might be struggling to achieve 30 game updates per second. In case the CPU can’t keep up with game logic simulation the game slows down. Another example - if the CPU can simulate only 10 logic updates per second, then a second of in-game time will take 3 seconds in real-time - the game will run three times slower than it normally should. The Riftbreaker has been out for several years now. By this point, we know very well how the game systems relate to one another. We also have a couple more years of experience and knowledge under our belt. That made us confident we could reach our performance target.
Lately, our optimization efforts have resulted in a nice performance improvement - the game runs much better in heavy load scenatios.
As we mentioned in our previous ‘Co-Op When’ articles, separating our game features' logic and visual parts was one of the biggest hurdles. To summarize, we started moving the virtual representation of many effects and events to the clients so that the server doesn’t have to perform irrelevant operations. That gives the server more time to work on the game logic. When we reworked our systems to work this way in multiplayer, we quickly realized that it could also benefit singleplayer. The new versions of our systems could now operate on smaller data sets and stopped colliding with each other so much. Thanks to this, many more operations in the game can now be carried out simultaneously, parallel to each other. When we saw the first results, we started working on removing as many dependencies between our game systems as possible. When game systems can operate independently, we can better saturate the CPU cores and achieve significant performance gains compared to the old architecture.
BENCHMARK ENDGAME ULTRA WAVE
It is rare for a base in The Riftbreaker to stand idle for a long time. The game is really put to the test when a large number of enemies attack you. To test this scenario, we created another benchmark called ‘Endgame Ultra Wave.’ In this scenario, we launch the final attack wave from the Brutal difficulty at a large base. To spice things up even further, we added a couple of thousands of extra enemy units into the mix, spawning at various points during the test. The base is constructed in such a way that it is guaranteed that the creatures will reach the player’s structures and destroy at least some of them. This allows us to test the game’s performance in hardcore combat scenarios. In addition, we created a separate version of this benchmark for each biome, as those significantly affect the game's performance.
In case of the jungle biome, we use the same base as during the 'idle' benchmark, but this time the base is being attacked by a horde of angry mobs. Each biome has a separate test scenario.
The Swamp base atttack scenario features tons of pipes and dozens of Canceroths.
We chose such a high-stress scenario because it is much easier to optimize, contrary to what you might initially think. When we work on performance, we make extensive use of different profiler tools. A profiler is a piece of software that you can attach to the game process and record the activity of the individual systems of the game. Thanks to the profiler, you can clearly see how long the calculations for each system take and which CPU core they utilize. The profiler will also mark those frames where logic calculations took over 33 milliseconds to complete. We can analyze those frames and choose targets for optimizations. After we’re done, we run the process again and again. The tools that we use are:
Tracy: for a general overview of how long system simulation takes, and how various processes are scheduled on CPU cores. This piece of software has been developed by a good friend of ours. You can download it right here: https://github.com/wolfpld/tracy
When we have an overall idea of what takes long to compute, we switch to one of the following depending on the platform:
Visual Studio Diagnostic Tools (PC - shown on the screenshot), Sony Razor CPU profiler (PS5), Microsoft PIX profiler (Xbox Series) - this is where most of the work is done.
Since the Fungal Swamp biome is brand-new, it came with a whole host of performance issues. Luckily, the new benchmarks allowed us to catch them early.
We targeted the most performance-hungry systems and adapted them to become more multiplayer-friendly. The nature of multiplayer gameplay and the need for perfect synchronization of client and server states forced us to detect changes in entities much more efficiently. A combat scenario, where tons of buildings get destroyed, and hundreds of creatures perish, is a perfect scenario to make such optimizations because those changes happen all the time. For example, the ResourceGraph system produced a performance spike for several seconds whenever a factory was destroyed. We have made great improvements in that regard since then. Our systems work with smaller data sets now, they are much faster and free up the resources for other systems that might require computational power. The benefit of that can be seen both in single and multiplayer game modes.
This is what the performance graph should NOT look like. During our tests it turned out that complex pipe systems produced gigantic lag spikes under high stress scenarios. When you're fighting for milliseconds, a six second stall is not what you want to see.
It is also important to note that we prepared a separate benchmarking scenario for each biome in the game. This matters for us since various biomes require players to utilize different building techniques and advanced technologies that might not be viable on other maps. Each of those tech chains might come with its own individual issues. Here’s a fresh example of such an issue. We have recently constructed benchmarks for two Fungal Swamp biome scenarios: a forest map with relatively low liquid density and a swamp map with liquids making up most of the play area. Naturally, the swamp map requires the player to use pipes and buildings associated with liquids extensively. The benchmark for that map ran terribly slow, with the game logic update reaching up to 250 milliseconds instead of the budgeted 33. We were surprised to find out that the main culprit was the PipeSystem, which took 180 milliseconds for each update. Additionally, if any pipes got destroyed, the game hung for between one and seven seconds. It became clear that our pipe code is inefficient, and we should look for optimizations there.
We have already started working on improving the PipeSystem. The performance in this specific scenario is still far from the 33 millisecond target, but it's just the beginning of optimization.
CONCLUSION
Years of working on the multiplayer mode for The Riftbreaker have been a tremendous challenge. We have learned and improved a lot in this time. We strongly believe that the effort we put into this will benefit both the players who like to play solo and those eager to get their hands on the multiplayer version. We’re doing our best to allow you that - we will start a closed beta test as soon as we are ready to take your feedback. We still have a couple of gameplay aspects to figure out, but rest assured - we’re getting there. Join us on www.twitch.tv/exorstudios for previews of our survival runs. If you have any more question, feel free to ask us directly on www.discord.gg/exorstudios and in the comments below.
See you next time!
EXOR Studios