Hello Riftbreakers!
Once again, it’s time to share some news about the development of The Riftbreaker co-op mode. When we released the last article, we told you about the major breakthrough that allowed us to test this game mode with more than two players simultaneously. We told you our next optimization targets and how we aimed to reduce the data transferred between the game’s server and its client PCs. This time we will tell you what we have been doing over the past few months, what improvements we made, and what comes next.
Our previous articles are available here:
https://store.steampowered.com/news/app/780310/view/3381659291157676103?l=english
https://store.steampowered.com/news/app/780310/view/3701435238673426124?l=english
EntityComponentSystem Optimizations
Much of our work has gone into optimizing the EntityComponentSystem, or ECS, for short. It is responsible for storing data about all in-game objects. The properties of almost everything you see in The Riftbreaker are stored within the ECS structures - the amount of HP enemies have, the light emitted by particles, the destruction system of buildings, and many more. It is a genuinely massive part of the game, so it is no surprise that we can find many places to improve within its code. Our current form of ECS in the game has been created with single-player gameplay in mind. The data for all entities are stored at all times. When an entity is visible or otherwise relevant to the gameplay, it is held in memory for immediate access. If an object is temporarily not used for any calculations, its data is packed and kept for later reference. The Riftbreaker is a huge game, so the amount of data the ECS has to handle is quite substantial. As we told you last time, large data structures are not desirable for multiplayer, where every byte of transfer and every millisecond count.
Thus we began the great work of rewriting the EntityComponentSystem, one component at a time, to optimize it for co-op scenarios. We have been working tirelessly, deciding which components have to stay on the server and be synchronized with the client PCs and which elements can remain solely at the client PC’s discretion. As we told you last time, we aim to keep things that are irrelevant to the gameplay state on clients only. However, with as many as 291 components to check, verify, and potentially rewrite, it still is a massive undertaking. However, we have made a lot of progress already, and we are sure that we will be able to enjoy the benefits of it very soon.
Minimap Rework
One of the main problems we encountered while playtesting the Riftbreaker in co-op mode was something that you could see during our dev streams (yeah, we stream co-op gameplay fairly regularly! Tuesday and Thursdays, 3 PM CEST at twitch.tv/exorstudios. Co-op streams don’t happen every time, but we try to show off our progress as often as possible.) was the broken minimap. The map would only show the player what they discovered themselves. Whatever your co-op partners found or built would not show up until you went there by yourself. Moreover, some objects outright refused to show up entirely - for example, resource deposits. This seems like not a big deal, but in reality, it made terrain orientation really tricky. It was also almost impossible to tell where attacks would come from, as their markers did not show up either. The only means of communication and marking places of interest was placing Rift Portals. While placing Portals next to valuable spots is generally okay, more was needed for players to enjoy the game.
With that in mind, we decided to rework the minimap from the ground up. The new minimap will collect data individually gathered from all players, combine all information into one on the server and then distribute a copy for each player. This way, we can ensure that every party member has access to the same knowledge at all times, making coordination and planning much more manageable. Reworking the minimap system also allows us to introduce performance optimizations. The minimap might not be the most breathtaking feature, but it’s resource-intensive. Every object marked on the map is a separate draw call for the GPU. With thousands of units, hundreds of bullets, and dozens of buildings appearing simultaneously, the cost of rendering the minimap grows quite drastically. Luckily, now is when we can make significant changes and reduce the strain on your PC’s resources.
The Sound of Silence
When it came to sounds in our early tests of The Riftbreaker’s multiplayer, the experience was hit-and-miss. During one session, you would hear all sounds playing back fine, and during others, nothing. Sometimes, you would hear sounds only if another player was hanging around in the vicinity of your screen. The problems resulted from the way we handled spatial sounds. The implementation was fine - but again, it was made with single-player gameplay in mind. Here’s a short breakdown of how the sound system in The Riftbreaker works and what we did in order to fix the problems we encountered.
We used so-called ' ears ' to simulate the effect of various sound effects coming from different parts of the screen (or entirely off-screen). Think of those ‘ears’ as a set of microphones attached to the game’s camera, each recording the sounds from the direction it is facing. When a sound is being played, we check whether it can reach one of the ears. Then, the sound’s distance from the ear would determine its volume and placement in the dimensional mix of the entire soundscape of the game. It is also worth noting that the mixing algorithm depends on a user’s sound system - it’s different for stereo headphones and for a 7.1 surround sound setup. In a multiplayer setting, the information coming from the ears of all the client machines mixed and resulted in various errors. The solution that we used in single-player was clearly insufficient.
To combat this issue and reduce unnecessary data transfers, we decided to handle sound information on the server side and send complete sound mix information to the client. Based on the position of each client’s ears, the server prepares information about all the sounds that should be audible for that player and sends it as a complete set of instructions. The only thing that’s left for the client to do is to play back the sounds from the disk. This means we won’t have to play with no sounds during our Twitch streams soon!
TurretSystem
While looking for potential candidates for further optimizations, the TurretSystem caught our attention. This system handles all the operations of defensive towers you place in your base during your gameplay. It takes care of aiming, shooting, and checking if enough ammo or energy is available for the tower to take a shot. We discovered that the calculation costs related to this system rose unexpectedly high when the player had many towers in their base. We expected the system to take up more resources for huge bases with hundreds of towers, but the numbers we saw did not match up with our calculations. We dug deeper.
Upon further investigation, we found out that the TurretSystem was optimized well, and we did not spot any immediate errors or places where we could make improvements. Clearly, it was not where the problem lay. Then, we decide to check all the systems that interact with turrets. This is finally when we got to the root of the problem. It turns out that the ResourceManager, the system that was responsible for checking if the towers had enough ammunition to operate, would take up as much as 10 milliseconds of calculation time per frame due to a bug. This problem was introduced when we refactored the resource system to handle resources appropriately for multiple players (e.g. each player’s ammunition is handled separately). When we fixed that bug, the time dropped from the dreadful 10ms to a much more reasonable 0.4ms. This is a great example of how expanding the game engine to support multiple players can drastically decrease performance and what we have to do to combat this type of problem. While working on adding multiplayer support, one of the most difficult, most time-consuming, and always ongoing tasks is to optimize the code in such a way that it runs at least as fast as single-player code.
Saves Work Too!
One more thing that we have been working on is the save system. After a couple of hard weeks of work, we implemented our first version of this system into the co-op version of The Riftbreaker. In its current form, it is quite simple and far from what it will be in the final version - please bear that in mind. Currently, the server saves only the state of the world and the ongoing missions. It does not store any information about the players, meaning that when we load a saved game on the server and connect to it with one of the clients, we are treated as a completely new player. Saving player progression and establishing an interface for reconnecting players is something that is still left to be done..
One of the main problems with testing unfinished products is the lack of stability. The server can crash anytime for any reason, ending our playtest on the spot. While some crashes happen 100% of the time, and there is nothing we can do about them apart from fixing them, others occur randomly. When you test software, you want to avoid being held up waiting for a fix to an issue that does not occur 100% of the time. While we can’t load player progression at the moment, even the current state of save/load implementation is of great help to us during development. If a crash occurs, we can simply come back to the latest saved game state and try again. If we crash again, we have a great way to reproduce a bug and make it easier for a programmer to track it down. If the game doesn’t crash, we can continue looking for more potential issues and speeding up our work.
There’s still a lot of work remaining to get the save/load system to where it needs to be, but we can already claim a major milestone as the core of the mechanism is already functional.
Conclusion
Over the course of the past few months, we managed to complete quite a few tasks and reach milestones we set for ourselves on our road to a functionally complete multiplayer build. Our programmers have laid the groundwork for the introduction of new technologies and optimizations that we have been planning for The Riftbreaker.
We have already mentioned the first of them in this article - the general-purpose optimized octree finder algorithm. An octree is a data structure where each node of the tree has exactly eight subnodes. Those subnodes can be divided further and further into more nodes. Octrees are an extremely useful tool when dealing with large, rather unorganized data sets. Dividing the data into smaller subsets significantly speeds up queries. We are currently making improvements to our current general octree search implementation that will allow us to see significant gains in various areas - both data transfers and performance. The new optimized octree organizes entity data into linear structures based on their spatial positioning. Most spatial entity queries look for entities that are close to each other, so having them close to each other within memory improves read speed and reduces CPU cache misses. Furthermore, the new algorithm is capable of returning entity component data along with the entities that are the result of the original query. Our current implementation has to divide that into separate operations, which is quite expensive. The new algorithm is very promising. However, we have to propagate its usage along all of our in-game systems to reap the benefits. It is one of the major tasks that are ongoing at the moment, and we hope to be able to report the results to you soon.
Another ‘big thing’ our programmers are working on is an entirely new dependency system within the EntityComponentSystem. Right now, when two entities make use of the same data, each of them has to create a copy of that data for ‘personal’ use, essentially wasting memory. While we are talking about sub-kilobyte numbers here, this number quickly adds up in a game like The Riftbreaker, where we often deal with hundreds of thousands of entities on one map. With this new system in place, entities will be able to share data with each other without the need to make a copy beforehand. This will reduce the amount of RAM needed by the game, make data reading quicker, and will positively impact the network transfer volumes.
We know that most of you would only like to hear the answer to a single question - when is coop going to be ready? The honest answer is that we don’t know yet. Looking back at the amount of work that we’ve poured into the online co-op, we can easily say that it already equals making a small game, and there’s already quite a bit of work to be done and a lot of unknown factors at play.
Our “immediate” goal is to achieve a stable multiplayer build with most of the game’s single-player functionality working in multiplayer through a local network. The biggest tasks that are required to achieve this milestone is - improving multiplayer CPU performance, reducing data transfer, and achieving full stability and feature parity.
As you can see, we have a clear vision and plans on how to make the co-op mode in The Riftbreaker a reality. The only thing we are asking in return is patience - we are doing our best to let you play the game with your friends as soon as possible. If you want to monitor our progress closer, visit our Discord at www.discord.gg/exorstudios and talk to us! We’re here to answer all your questions. You can also check out our streams every Tuesday and Thursday at 3 PM CEST over at www.twitch.tv/exorstudios - we try to stream our co-op tests as often as possible.
EXOR Studios