Hello Riftbreakers!
Last time we discussed the Multiplayer version of The Riftbreaker, we told you about our happy little accident that led to the creation of the deathmatch mode. It turned out to be one of the most helpful research and development tools we could have hoped for. In today’s update on our Co-Op development progress, we will tell you what issues we have solved already, what our testing routine looks like, and what milestones we plan to hit in the immediate future.
If you missed any of our previous Co-Op updates, you can catch up on them here:
https://store.steampowered.com/news/app/780310/view/3381659291157676103
https://store.steampowered.com/news/app/780310/view/3701435238673426124?l=english
https://store.steampowered.com/news/app/780310/view/3657536564724315072?l=english
https://store.steampowered.com/news/app/780310/view/3657539010840327508
CONNECTION THROUGH SEPARATION
Opening bioanomalies is easier with two flamethrowers at the ready!
After the deathmatch mod came into existence, we decided to package it into a separate product and publish it as a hidden item on Steam for internal use only. This special build is stripped of all the single-player functionality and currently only supports the DM mod. We have been using it for the past few months to test connectivity. We decided to prepare a specialized build for two main reasons:
- A separate, much smaller build allows us to test, improve, and iterate on the new game versions much faster. It also ensures that all testers are on the same game version (mismatched versions happen more often than you think!)
- Removing the unnecessary elements lets us focus on networking and connectivity issues without the unnecessary overhead cost from the game’s content.
By removing the majority of the game’s logic flow, redundant Lua scripts, and tons of other data, we can focus on what is truly important for us at this moment - improving the online playing experience, and that comes by optimizing transfer rates.
When we first started running multiplayer tests, we brute-forced the game to transfer all the gameplay state data from the server to the client. As a result, our data transfer rates reached upwards of 25 megabytes per second. Many people don’t even have access to an internet connection quick enough to transfer that much data consistently. Yet, our first tests demanded the game to send, receive, unpack, and apply all changes to the client’s game world multiple times a second. It barely worked over the local area network. We had a long journey ahead of us.
Most of the work we described in our previous articles about co-op development aimed to reduce the amount of data transferred between the client and the server. With each iteration, we were sending less and less data through the network. The latency was still high, but less data caused the game to run at a tolerable level of performance for much longer, allowing us to collect more information about the problems we were facing. At this point, all our tests were conducted by running a Survival Mode mission with two or more players connected to the same server. Making the server run both the game logic and handle the networking simultaneously was a tall order, given how much happens in each second of The Riftbreaker’s gameplay.
That last blade bounce was totally calculated.
Many of the problems we faced while testing this way resulted from the clients desynchronizing from the server. At some point, the performance overhead for the server was too high to handle all the tasks, and the clients expecting a server update did not receive it in time, resulting in lags, disconnections, and crashes. If only there were a way of stripping the game to the bare minimum, allowing us to focus on fixing the connectivity issues first without worrying about all the overhead costs generated by the Survival Mode mission running in the background…
Enter Deathmatch. A small and simple map, a couple of basic weapons, friendly fire, and no enemy units. Running tests in this environment gave us more insight in a couple of weeks than trying to complete Survival missions in the months before. We could find problems and reach conclusions previously hidden from us underneath all the unnecessary data. Thus, we started conducting daily playtests in a variety of configurations. Sometimes, we ran a dedicated server on a PC with no other tasks. Other times, we used the listen server option to maximize the load on the host PC. Profiling all our game sessions gave our team much insight into what was happening.
Sometimes it is complete, non-stop mayhem.
Using the data gathered during our playtests allowed the programmers to introduce the necessary changes. We started a lengthy back-and-forth process, where they would push a new build to the rest of the team and ask for a playtest while keeping watch on the network stats and ongoing threads. We kept on carving off chunks of data that were not necessary to be transferred. Bit by bit, month by month, hard work started showing its effects. Looking back at the start of the multiplayer refactoring process, we managed to reduce the data transfer rate from 25 megabytes per second to 200 kilobytes per second per client. This is still far from our target (10 to 20 kilobytes would be best), but it is a significant victory overall. The DM build helped us a lot in speeding up our iteration time. Best of all, the work we've done using the PVP build is transferable to the main project.
After we published our article about the PVP mode, many feared we would neglect the Co-Op mode. We want to assure you that this is not the case. Thanks to separating the PVP-only build from the main branch of the game, we built a perfect network testing sandbox for the most basic feature set on online gameplay. It doesn’t include any bells and whistles, just the essential game elements running unobstructed - precisely what we need to focus on the basic feature set.
NECESSARY ADDITIONS
The significant optimization of the data transfer we described in the previous chapter allowed us to start moving towards playing The Riftbreaker PvP on a PC outside the office network for the first time. To make that dream a reality without a VPN, we needed to add some features first. The list was as follows:
Host game screen with several options:
- The option to run a dedicated server (in headless mode - the host PC does not render the game)
- The option to run the game in listen server mode (the server PC also renders the game and allows its owner to play)
- Map selector list
- Basic game options, such as frag and time limit for any given map
Join game screen:
- Public server browser, listing the server’s name, latency, map name, and player count
- Refresh button, forcing the game to retrieve the public server list ahead of a scheduled update
- The option to join any public server.
Looking at this list, we realized that The Riftbreaker’s main menu will not be able to accommodate all these options in its current form. Therefore, we reworked the main menu layout to separate the single and multiplayer options from each other in their own submenus (it was already quite dated and cramped, so the facelift was much needed anyway). The layout we’re currently testing looks like this (please bear in mind that it’s not final and will likely change before the public release):
The options included in the current version of the main menu are still quite limited, but given our experience with bloat from earlier, we decided to focus on the bare essentials for now.
The server browser uses the Steam Data Relay Network. Despite the ‘Steam’ name prefix, this library is not limited to the Valve platform only. It can be used to establish connections between players on different platforms without any issues. However, to make things easier for us and take it one step at a time, we limit ourselves to playing with Steam for the time being. Any team member can set up a server with just a few clicks, which will become visible to everyone on the same game version. Then, it’s just a matter of clicking ‘join’ and we’re in the game.
The bots are quite bad, to be honest, but they still get the upper hand on us sometimes!
Not all of us are always available to test how the game behaves during online play. Our jobs generally need us to be focused on things for long periods. Sometimes, it’s challenging to get into that rhythm. Drawing someone away from their current task for a test can significantly disrupt their workflow. As a partial solution to this issue, we programmed very simple bots. VERY simple bots. They know the player’s positioning at all times and simply run forward, shooting as often as possible, as long as they have the ammo. They are useless as opponents but give us the ability to run basic tests with no involvement of more people around the office. Their development took only a few days, which we will quickly get back at a profit, considering our team’s time. With the help of our bot army, we can quickly populate our servers and run automated tests of game rules. As a bonus, whenever a real player wants to join the game, one of the bots is kicked out to make room for that player, so it is still possible to call someone in for help in case you need them.
It’s worth noting that none of this work will be discarded later when we move to more advanced testing. The main menu rework, the server list, game hosting, and joining servers - all these elements are essential, and we would have to make these happen either way. Completing these tasks also allowed us to make our tests more structured and semi-automated.
COMPUTER, RUN DIAGNOSTICS
We've taken these stats from a session hosted by a friend hundreds of kilometers away. Worked pretty well and consistently!
With the new functionality in place, we could finally keep tabs on the multiplayer build. We came up with a set of tests to help us catch bugs and measure the performance of the game. We’d love to tell you that every build that leaves the hands of programmers goes through rigorous testing, but we never lie to you - this is just a set of guidelines. However, we have a short list of basic multiplayer functionality that we try to ensure is operational. The list is as follows (in no particular order):
- Is it possible to name your server using the game’s menu?
- Is it possible to name your server using the configuration files?
- Does the server name show up correctly on other PCs?
- Does setting the warmup time limit work as intended?
- Does setting the frag limit work as intended?
- Does setting the time limit work as intended?
- Does setting the player limit work as intended? Can it be set to zero?
- Does the max bot count setting work properly?
- What happens if you set all the game rule value sliders to zero?
- Does the server keep the settings after loading another map?
- Do all the settings work both in the listen server and dedicated server mode?
- Can you use the Steam friends list to join the game? Does it break if the friend you’re joining is in the menu, loading the game, their avatar is dead, etc?
- Does the server browser display server names or IP addresses?
- Does the server browser display the actual number of players on the server?
- Does the server browser work with both dedicated and listen servers?
- Does the refresh button work correctly?
- Does the server browser refresh automatically?
- Does the game work as intended when a player joins a game session filled with bots?
- Can players still join the server when the maximum player limit is met?
- Can you set up a listen server on an external network and connect to it from the office?
- Can you set up a dedicated server on an external network and connect to it from the office?
- Can you set up a dedicated server on our internal network and connect to it from the outside?
- Can you set up a listen server on our internal network and connect to it from the outside?
- Can you see all running servers in the server browser menu in all the cases above?
- Can you use the ‘Join Friend’ feature in all the cases above?
This list is non-exhaustive. Our testing is often focused on the areas under active development. For example, we’ve recently been trying to figure out how to fix the problem that causes the player to lose weapons permanently on death. It is made worse by the fact that the problem persists when switching maps (on the other hand, it means you won’t lose your inventory during planetary travel in co-op, yay!) Naturally, we will pay more attention to the details surrounding our current area of interest, but keeping track of the basic functionality is paramount. Testing all these things will likely continue way beyond the final release.
WHERE DO WE GO FROM HERE
Clone wars, Riftbreaker style. Note that the game wasn't running too well during this run.
We still have a lot of work regarding optimizing the data transfer. As mentioned before, we ideally want to get the amount of incoming data down to about 10-20 kilobytes per second per player. We have already greatly reduced the amount of data transferred between the PCs during gameplay. However, we still generate more information than necessary. That is because we have not yet reworked all the in-game systems for multiplayer context. On the other hand, it gives us a couple of clear targets where we are sure to find performance gains.
Special effects are one of the significant areas of the game that still require a rework. More specifically - separating the audiovisual representation of the effects taking place in-game from the logic underneath them. We mentioned it in more detail in the first Co-Op Update article if you want to know more. Here’s the TL;DR: Currently, game logic runs exclusively on the server. For example, if a player shoots a gun, all effects happen on the server first and are then “copied” to clients. Instead of doing the same work twice, the server can command the client to spawn the effects on their end, reducing the overhead for the server. It sounds simple, but it is a monumental task, and its development is an ongoing process, as we need to rework all systems in the game to accommodate that.
In a similar vein, we are reworking the game logic so that not all of it has to be carried out on the server first. Some more minor game logic elements can be handled by the clients and later only verified by the server. However, introducing changes to how game logic unfolds carries huge implications for saving files. We still maintain backward compatibility with all (unmodded) game versions, going back to version 1.0 (PC players did not even have the chance to download it! Version 1.0 only exists in the physical disc console versions of the game. To run it, you need to have no internet connection for the entire gameplay duration. Still - it is officially supported). Even though the number of users that play such old game versions is minimal, we maintain save file compatibility as a rule.
With such a huge rework, touching almost all aspects of the game, you will find a lot of issues, even in the most unexpected places. Music is the best example - what could be so difficult about playing an .ogg file? The problem is that The Riftbreaker features an adaptive music system. When a player is fighting, the music changes to a ‘combat’ playlist. When you’re building the base, we use the ‘building’ playlist. We want to keep that feature for each player individually. However, there are certain moments in the game when the game’s logic forces a certain type of music to be played - during the final attack, for example. We want the server to be able to overrule the clients in this case, which requires reprogramming the system to work in a more complex client/server architecture. This is only one of dozens of problems we encounter, each needing a different treatment.
We discovered many issues when we started running our connection testing gauntlet. The high number of possible ways to connect to a game session and other variables, like the type of network your PC is connected to, cause numerous problems. Sometimes, they become apparent immediately, while others remain hidden for weeks. We carefully examine the causes of these issues on a case-by-case basis. We can’t possibly catch all mistakes on our own - which is also why we want to test the multiplayer portion with you.
Ragequit is one of our moderators and modders of The Riftbreaker. He immediately started modding the multiplayer build. We are not surprised at this point.
Starting this week, we have been allowing access to the multiplayer build of The Riftbreaker to people outside EXOR. Those players are handed keys to the hidden Steam product and are welcome to test the game with us and host their own sessions. This is a major milestone we are very proud of, and we hope that you will join us! The invitations will be handed out via Discord, so if you have not joined www.discord.gg/exorstudios - now is the best time to do so. The deathmatch test will eventually evolve into the real deal - a Co-Op beta test, which makes it even more exciting. We can’t wait!
CONCLUSION
The basic deathmatch gameplay loop, coupled with the essential interface features for hosting and joining the game, gives us an excellent tool for testing network performance. It has already yielded great results, and we expect it to improve. That said, we don’t want to develop this game mode any further. Any updates we introduce will only be aimed at making the game run better. We plan to invite selected Discord members to test the game with us, slowly opening the Deathmatch demo to more and more people. However, Co-Op remains our primary focus.
We never promised a release date since we knew the development process would be very long. Nobody could predict how long. Yet, you’re here with us, reading the articles, following the latest developments, and sharing your feedback. We would like to express our enormous gratitude for your support and patience. You will be rewarded with the best multiplayer experience we can offer - that’s the only thing we can guarantee.
In the meantime, we’re going back to work. The programmers - back to unspaghettifying The Schmetterling Engine’s code. The designers - to creating a new part of the adventure in the coming World Expansion III. We hope you’ll come on both of these adventures together with us. Join our Discord at www.discord.gg/exorstudios and watch us stream twice a week at www.twitch.tv/exorstudios.
Thank you once again!
EXOR Studios