Howdy folks!
We have something really special here today. We've mentioned previously that we'd like to share more information about how our multiplayer systems work, and here we have it.
Here's a history and retrospective of the Multiplayer systems in Armello, written by our lead developer, Karl Burdack.
The History of Multiplayer in Armello
How did it take 3 years for Armello to transition from ‘Peer to Peer’, to ‘Cloud Hosted Dedicated Servers’?
The short answer is we were always busy, fixing issues, adding features for monthly updates, or porting to new platforms. We were also a very small team, of four permanent coders.
The rest of this article is the long answer!
It’s been a winding path for Armello Multiplayer. As a studio, we have learnt much in three years about how to deliver updates to a live game. However we recognise there are still many improvements to be made.
NMA’s
The biggest issue for Armello Multiplayer has been deadlocks in the gameplay event queue. The event queue is a list of actions, or rules, the game takes to move the game forward. They are executed one by one.
If an event does not complete, the next event cannot begin. These are seen as freezes in gameplay, that we refer to as ‘NMA’ which is short for ‘No More Actions’. These freezes mostly affect multiplayer, since they occur when one peer’s version of events go out of sync with another, or a bug in our code fails to complete an event.
An analogy, would be if you played a physical board game. Where one of the four players, on their turn, could draw cards then take 3 actions. If that player sat at the table and refused to finish drawing their cards (NMA), all the other players would wait forever for their turn, and get very annoyed!
September 2015: Version 1.0: Photon Matchmaking
As a beginning, let’s start with the first full public release of Armello, Version 1.0.
The first Armello multiplayer architecture operated by relying on Photon Relay servers. The Relay Server receives messages from players in a match, and then sends those messages in a consistent order to all players. We also had an optimization for when the local player was having a turn, that they would immediately execute their actions, eliminating latency.
Remaining Issues:
- Players would timeout and be disconnected after only approx 10 seconds of lost connection. This was a limitation where Photon could not be configured to behave differently.
- It did not support future improvements to Matchmaking we had planned, such as ranked matchmaking, in a way compatible with how we wanted to implement it.
- This system had no way to block cheaters, issue bans, or prevent pirate copies of the game from playing.
January 2016: Version 1.2: Steam Matchmaking and Peer to Peer
We evaluated the Steam Matchmaking and Peer to Peer (P2P) Networking, and decided it was more flexible and suited our future needs.
After implementation we immediately saw improvements in latency and connection timeout tolerance. Steam matchmaking also automatically blocked banned, or pirated copies joining multiplayer.
Changing to Steam P2P networking, transitioned Armello to a pure Peer to Peer network architecture.
In order to support reliably ordered messages across all the peers, we had to continue to use a Relay for messages. So instead of using Photon Relay Servers, we now hosted the relay on one of the peers. This peer would be picked at the start of the match, and would remain the Relay host for the remainder of the match
Remaining Issues
The biggest issue with this change, was that dealing with disconnects became much harder. If the current Relay host (Amber) were to disconnect, a new peer needs to be consistently chosen, and all messages that were outgoing from the previous host at the time of disconnection, need to be reconciled in case any were lost to some players. It’s likely that Amber disconnecting while Thane takes an action, might let Thane execute his own action (red arrow) but for the message to never arrive at Sana or Mercurio (black arrows).
This lead to difficult issues for us down the road.
June 2016: Version 1.4: Optimized networking
For 6 months, part of the Armello team were focused on porting to the Xbox One. Unfortunately, at that time Photon was not available for Xbox One. So in order to support that platform, we had to port the multiplayer to support a third solution: Unity Networking.
Unity Networking on Xbox One at the time, had much stricter message size limits. So we had to get serious about optimizing Armello’s sloppy network message sizes. It’s something we had on our list of things to do, and now it had become a priority.
In many cases, these optimizations reduced the amount of data sent and received by a factor of 100.
What? How could a simple Move action in a turn based game, consume 2500 bytes. Easily:
These optimizations were easy to bring over to the Steam release of 1.4. It would also be an important optimization down the road for Mobile platforms. These optimizations have benefited players on connections with poor bandwidth, as well as reducing CPU overhead on below minimum spec machines.
Unknown to us at the time, would be that the initial Unity Networking port to the Xbox One, would also be useful when starting out implementing dedicated servers!
Jan 2017: Version 1.6: Multiplayer Stability Update
Poor disconnect handling had now been plaguing our multiplayer community for 6 months, and we began work on an improved model for the Peer to Peer Relay Host
An example of how disconnect could cause stability issues:
In this situation, the system must be designed to recover the message Thane had sent, and re-send it to Sana. If the message is not recovered correctly, it can lead to an NMA situation.
To address these inconsistencies with disconnections, we implemented improvements to how disconnections were handled. We implemented a better message tagging and reliable message recovery protocol. A simplified example of how lost messages are recovered is shown below:
Remaining Issues:
While this was promoted as a Multiplayer Stability Update. Many players complained that it made their experience worse. We reflected on two areas of weakness where we must improve:
- Measurement: We must be able to better measure build stability during development (testing), and in live builds (analytics)
- Automated testing: We must automate the testing process of multiplayer, so that an enormous volume of games can be played, testing all possible combinations of actions.
- ‘Double disconnect’: There were many cases where if two peers disconnected rapidly, our implementation would not handle it well. We would need to add automated tests for this case.
- Undetected by us at the time, was that the optimization of immediately executing the local players action (indicated with red arrow in previous images), was also the cause of much of the disconnect instability, due to ordering issues when switching hosts.
May 2017: Version 1.7: Automated Testing
After the disappointing results of 1.6, we worked for around 6 months to implement an automated testing system for Armello. We added support for multiplayer Bots which would play fully through games many times a day.
Immediately we found an enormous volume of NMA bugs. We were now able to more easily address these, with full logging from all our test agents. During this period, we released several patches, usually once a week, as we found and fixed the issues.
Automated test PC's. These nifty HP micro PC's have a low power usage, but enough CPU power to run several tests of Armello in parallel, usually without graphics.
Remaining Issues:
- After this round of patching, disconnections remained the major cause of instability, and we determined to look at this next.
- Analytics were still lacking enough detail for us to track live build stability.
August 2017: Version 1.9: Improved Disconnect Stability
We hooked up our automated tests to live analytics, so that we could analyze thousands of test games per day. We increased our test coverage to include disconnection. We also began to receive our first live Analytics for live build stability, and were able to track live issues, and our progress at reducing instability.
What we discovered was that the most common stability issues arose when players disconnected during the end of their turn. In many cases, the issue was related to the ordering of remote messages, compared to the locally executed message (indicated in red in previous images)
The fix was for players to take turns being the relay Host. The player who was having their turn would have the active Relay host, which would dispatch messages to all other hosts. When players end their turn, the active host moves to the next player.
This way, the active player, would continue to always immediately execute actions on their turn, such as movement.
Problems:
- Multiple players disconnecting at the same time were still causing stability issues, which were difficult to address easily with our existing peer to peer architecture and certain game events.
November 2017: Internal Prototype of Dedicated Servers.
We were faced with a choice, continue to improve the peer to peer model, while fixing issues in very complex Armello game code. Or look at all the issues as a whole, and decide on the best outcome going forward.
We decided to prototype Armello running with a Client/Server networking architecture. We would begin to look into the costs of hosting dedicated servers.
Fortunately, Armello was already well suited to Client/Server architecture, because it already broke up actions into discrete events which execute one after the other in a deterministic way on all clients. Deterministic means: The results of the action executing is exactly the same on all clients.
Initial prototyping was promising, but some additional problems would need to be overcome:
- A hosting service for the servers would need to be found
- Server hosting costs would need to be evaluated. As every hour of a dedicated server costs money. We didn’t want to dramatically burden our indie game with an unsustainable cost for running multiplayer.
- The server version of the game would need to be heavily optimized for CPU usage, Memory Usage, and build size. So that it could be more cheaply hosted.
- Latency for the active player would be unacceptable for some players. Additional work would need to be done to hide the latency by allowing the player to move, and revert that action if the server decided differently.
February 2018: Version 1.10, iOS Port
When porting to iOS, we needed a new, non-steam inventory and online account management service. We decided to integrate a service named ‘PlayFab’.
This work turned out to have a fortuitous benefit; because PlayFab would later become our cloud hosting service for Dedicated Servers on the PC platform. We now had the final piece of technology required to make this happen.
Planning for Version 2.0
For our planning for 2.0, we surveyed our community, what changes they would suggest for Armello 2.0. When we looked at the top 5 multiplayer related requests:
The number one requested feature, was Multiplayer Reconnect. This is a feature that would allow players to rejoin a game that had already begun, after they were disconnected for some reason. Also note resolving NMA’s as 4th on the list.
We also keep an eye on our monthly negative reviews, here is an example from April 2018:
This month is fairly typical for reviews. 25% of negative reviews are attributable to NMAs. Also, 8% of players gave us a negative review for lack of a Reconnect feature.
Many of the complaints are down to taste, such as Randomness (RNG). However NMAs and Reconnection are something we can improve, without fundamentally changing the game of Armello.
- Reconnect: This is by far simpler with a Client/Server architecture. Where the Server has a complete copy of the game, which can be replicated to clients attempting to reconnect. If we are ever to support this feature, we would first need working dedicated servers.
- Resolve NMAs: We knew that the ongoing difficulty with testing and preventing NMA’s would be much simpler in a client/server architecture. In particular, having a dedicated server does away with the problem of a ‘host’ disconnecting completely.
With solid information backing our instincts, we knew changing to Dedicated Servers might be the most important change to Armello we could make.
July 2018: Version 1.12: Dedicated Server Beta.
For the first 6 months of 2018, we worked on fully implementing Client/Server in Armello. This included:
- Setting up the backend technology to let us easily upload new builds of the server into the cloud. (PlayFab)
- Developing optimizing builds of Armello to be hosted in the cloud as a server.
- Improving disconnect to take advantage of the stable dedicated server.
- Repurposing our automated testing, to be testing hundreds of cloud hosted games a day, and to pressure test the cloud hosting service.
- Improving client side movement actions to hide latency
- Fixing a whole bunch more NMA’s along the way.
August 2018: Version 1.12.1, Patch 4: Dedicated Servers switched on permanently
After about 4 weeks of beta testing the dedicated servers. We switched them on for all multiplayer games on 20th August.
The following chart tracks server load time over 30 days, which is a good measure of performance. In this chart, the lower the line, the faster servers are launching into games. In the first few weeks, we had to work hard to eliminate issues where servers would crash, or freeze. We also had to tune the load balancing of how many game servers were allocated per cloud machine.
By the first week of september, in Patch 7, we had resolved all known crash issues, and had load tested the server into a stable configuration. From Patch 7 onward, server load times, and availability have been rock solid.
One of our reasons to switch to dedicated servers, was increased match stability, by reducing NMA rates. Now that dedicated servers are fully operating, we can compare public multiplayer games completed against those that failed to complete because of an NMA. The following chart shows 30 days of games, with the transition from Peer to Peer to Dedicated Servers marked in red. Note the significant decrease in NMA’s as a percentage of games (in light blue):
We are very happy with these results. And we hope our players are too! More games are being completed than ever before, with stability issues such as NMA’s and disconnects at all time lows.
The future
While we aren’t promising anything (when do we ever do that?), having dedicated servers does open up some interesting future possibilities for Armello beyond adding reconnect capability. We will be announcing some of these in future. In the meantime, keep the suggestions coming!