An introduction to Pax Dei’s shards, servers, and zones
As you may know, the World of Pax Dei consists of a few regions (Gallia, Anatolia, Gothia…) split into several provinces (Ancien, Merrie, Kerys…). A province is a very large geographical area with several valleys interspersed by big mountain ranges. In specific provinces called heartland provinces, we have multiple valleys we call home valleys, and those are where you can build your home. And, of course, you also have dark dungeons and caves that mainly exist below the surface of these provinces.
But how do these geographical features map down to actual physical servers, and how do they affect how you play the game?
Worlds/Shards
The first choice you will face when creating a new character in Pax Dei will be the selection of a specific instance of a World, also commonly referred to as a Shard. Even though the world of Pax Dei is geographically vast, it can only accommodate a certain number of players in total. This is why we run multiple copies of the world, each being identical but completely independent of the others in terms of actual inhabitants. Each such copy we call a shard.
We estimate we’ll start with a maximum population in a single shard of around 7,000 players. A single shard is run wholly within one physical cloud hosting center. We will run shards in different availability regions worldwide, e.g., North America and Europe.
The factors that will mainly influence your choice of shard will thus be the following:
- Latency to your main game computer
- Seat availability, as some shards might be full
- Shards where you have many friends already playing
Zone Servers
From a computational point of view, a shard consists of a large collection of various servers and cloud services, each of them fulfilling a specific function. The most numerous ones are what we call zone servers. This is because the simulation of a whole world is much too computationally heavy to be handled by a single hardware instance and is thus divided into smaller computational units that we call zones. A single zone server is a Dedicated Unreal Server that handles the simulation of a specific part of the world, e.g., one home valley.
At any given time, you, as a player, are automatically connected to a zone server. Whenever you move across a zone boundary, your connection automatically moves to the adjacent zone server. This is what we call a zone transition. In most cases, these transitions should be seamless, except for a few important exceptions:
- Transitions between different provinces or into dungeons will typically incur some loading time as the client loads an entirely new map
- There is currently no visibility of other players or NPCs across zone boundaries
Zone Instances
As mentioned above, a zone server is a dedicated Unreal Server handling the simulation of a small subsection of the world. Eventually, a zone server can also become overwhelmed with too many players.
Currently, this limit is around 150 players, but it is expected to change with better hardware and optimizations. To avoid having to close off zones when the maximum number of players is hit, e.g., during peak hours, we introduce the concept of zone instances instead. The mechanism is such that when we surpass the maximum number of players in a zone, we distribute the whole set of players within a given zone boundary between two or more separate instances of the zone. Each zone instance is actually running a separate zone server for the same zone, but players within a given zone instance can not interact with players in another instance.
This separation only applies to direct player interactions, but the persistence of the buildings, amongst other things, works across instances, meaning the buildings are visible to all. Once the zone population levels go down, instances merge together. For regular operation, after a shard has reached maturity, instancing should only happen in exceptional cases, but it might be common at the beginning.
Persistence Backend
Apart from zone servers, the rest of the backend handles coordination between zone servers, transactional game logic, communication, and persistence. There are situations where the clients connect directly to this backend through secure APIs, e.g., for authentication. Still, in most cases, the zone servers are the ones communicating with the backend to specific services such as inventory service, resource management, etc.
All of this infrastructure is then deployed, coordinated, and scaled through a Kubernetes architecture running on top of a cloud infrastructure.
Questions
Can you explain what a backend engineer does exactly?
A backend engineer is someone who works on any of the services that reside in the backend but also in the interaction part between the zone servers and the backend. Typically, backend engineers define and implement the APIs used in communication between Unreal and backend services. They will also determine the data structures and persistence logic for all persisted data in the master database. Finally, they will also work on deploying and monitoring all the underlying infrastructure.
How is the cooperation between the backend and the front end working?
Mostly, this interaction will go through some well-documented APIs. The API defines all the operations that are allowed for a given service. The backend will also validate and enforce that the API client is authenticated correctly and has permission to call the given function. The API response is then typically data or validation that clients use to display to players through a customized UI.
What is the biggest challenge when working on an MMO like Pax Dei?
The biggest challenge is how complicated the tech stack is. The whole tech stack comprises various components, some being 3rd party components (e.g., the Unreal Engine or PostgresDB) and others fully developed in-house. Most of these are not specifically designed to be used in an MMO.
The functions of these components range from physics simulation, graphics rendering, AI behavior to transactional integrity and high scalability of server transactions. They cover multiple programming languages and must be built and deployed in perfect harmony to function all together as intended.
Finally, testing an MMO without actual players is difficult, hence the importance of early user testing.
Will there be language-specific shards, or will all players be able to play together?
It is something we want to monitor. We will start with no special language restrictions, but some shards with clear language bias will probably emerge. We will make our best effort to surface this kind of metadata to players to help them make an informed decision on which shard to join.
We also don’t want to force you to play in a specific region. We mean that if you are a European player and want to play with friends in the US, you should be allowed to do so (while accepting more latency).
Will we have some shards dedicated to PvE without any PvP interaction?
No, the plan is currently that all shards will be identical.
How many players will be per shard?
We will start with a target of about 7,000 players per shard (in 3 to 4 heartland provinces). As we bring more heartland provinces online, the maximum population per shard will grow accordingly.
With the expected high demand at launch, how do you plan to avoid queues?
We can spin up additional shards as needed to accommodate more and more players, but the zone-instancing mechanism will also help mitigate the initial load per shard.
What are your plans to minimize latency for the players?
Shards will be available in different geographical regions, so new players should be able to find a shard that is geographically close to them. Furthermore, the combat system and typical MMO interactions are such that they are more tolerant of lag as compared to first-person shooters.
Will you have weekly maintenance? Will it be different per region to respect the time zones?
We design our system in a way that they shouldn’t need maintenance except for major client and server updates, but, at the moment, we still need to do some downtimes and sometimes complete wipes of the Worlds. The selected time frame will certainly attempt to minimize disruption for players.
How will you manage the population of the Worlds? Do you plan to allow character transfers?
Yes, definitely. It is necessary for players' quality of life and population management.
But, let’s split this question into different cases: moving a character from one World to another is pretty trivial, so that is something we will undoubtedly allow (as an extra service or for free if the goal is to desaturate the population of a given world).
Moving a plot and the building(s) on it is a far more complex beast. We can’t simply transfer the buildings as the space could not be available in the new World, and we can’t easily move the buildings to a ‘similar space’ because the buildings' structure depends on the terrain they’re on. Ideally, we will have a system that lets you save your building structure as a blueprint or at least put all their components in a magical bag (don’t ask about the lore behind this, please).
Finally, when it comes to Clan transfer, we are currently considering it, but because of multiple reasons (priorities, workforce, unfinished features…), we can’t say it will happen before too long. We will provide more details on transfers and their availability later.
If I want to play with my friends, how can we ensure we'll be in the same World? Can I join them even if their World is full?
When a World becomes overcrowded, it will be locked to safeguard the quality of service.
So, our objective is to strike a balance between preventing overcrowding and ensuring friends can easily play together.
The key here is to be smart about the indicators we will use and how we guide people towards specific Worlds.
In critical situations, we would like to be able to offer character transfers to a populated World for free, as it’s one of the best ways to ease overcrowding. We will provide more information at a later time.
What is your stance on third-party programs and add-ons?
We're open to third-party programs and add-ons (as long as they are not about cheating), as we recognize that these contributions can often enhance user experiences and provide valuable functionalities. However, at the moment, we regrettably lack the necessary resources to offer proper support for them.
We genuinely hope that in the foreseeable future, we will be in a position to provide the support you deserve. Rest assured, we will keep you informed and share more information as it becomes available.