Losing a machine now costs you capacity instead of your customers' demand, and the survivors carry the extra. Traffic has priorities, big datacentres run about three times faster, and a customer's p99 clears within minutes of recovering.
Changed
Failure and recovery
- Losing a machine costs you capacity, not your customers' demand. The workloads that survive pick up the requests the dead one was serving, so they run hotter and answer slower until you replace it.
- Survivors under extra load can fail in turn. If they cannot absorb what they inherit, the overload spreads to the machines and links they depend on.
- The load goes back when the machine returns. You do not need to move anything to recover it.
- A customer's card shows how many workloads are answering. It also counts the degraded ones, so a fleet one failure away from an outage no longer reads as healthy.
Traffic priority
- Live requests now come before bulk copying. A database catching up a read replica, an object store rebalancing and a CDN filling a shard all yield to traffic a customer is waiting on.
- Health checks always get through. A load balancer's probes have the highest priority, so a busy link cannot make it decide a working server is dead.
- A database's standby copy keeps its priority. The customer's write is not finished until the standby has it, so it counts as live traffic rather than catch-up.
Speed
- Big datacentres run about three times faster. A save with nearly a thousand servers and a thousand customers now takes a third of the time per step, so it keeps up at higher game speeds.
Serverless
- The serverless pool shows requests turned away instead of a queue. Requests no longer wait in line. The panel counts the ones the pool had no room for and shows what a cold start cost.
Fixed
Game clock
- Time no longer stops after you switch windows. The game pauses only while the pause menu or the end-of-run summary is on screen, and it resumes when you close them.
- The setting that paused the game when it lost focus is gone. The game keeps running while another window has focus.
Network
- Two servers on wide bonds no longer crash the game. Traffic between two machines that were each bonded across several links could stop the simulation. That traffic now spreads across the paths it has.
- A brief spike no longer opens a link congestion incident. A link has to stay congested for several checks in a row first, so a card no longer appears and clears within a second.
- Pulling the last lead from a bond removes the bond. An empty bond used to stay on the switch and report itself failed, raising incidents you could never clear, with no broken cable to find.
- Bond incidents you cannot explain clear when you load. Loading clears any of those stuck reports on your save, along with any that a disbanded bond left behind.
- A flood spreads across every line your gateway shares traffic on. One line used to take all of it while another sat idle, and a Failover backup carries none until it takes over.
- Random floods at a one-zone site no longer exceed its lines. They now stop at 80% of what those lines can receive, while a site with a second zone online still takes full-size attacks.
- At a one-zone site, a random flood leaves room on your Failover backup. The game sizes the flood to your smallest line, so the backup has headroom when it takes over from a failed primary.
- Server-to-server traffic uses every path between two machines. Each conversation between two workloads picks its own route, so a customer running many workloads no longer sends all of it down one.
Customers
- A recovered customer's p99 no longer sits at 2 seconds for hours. The SLA figure now covers the last five minutes and the live figure the last minute, so both return to normal within minutes.
- A broken spread promise shows the numbers that broke it. The finding now shows the service over its limit, measured the way the contract asks, instead of totals that made a broken promise look kept.
Racks
- A device console draws over the rack panel behind it. The rack's side panel used to sit in front of a console you had just opened and take every click in that strip.
Notes
- Your saves load, but new ones do not go back. Saves from v0.1.67 and earlier open normally. The save version is now 58, so a game saved here will not open in an older build.
- Load once to pick up the new congestion timing. Existing saves kept the old, too-short timing, and loading replaces it. Nothing moves and nothing goes down.
- Loading clears each customer's latency history. Old saves carried hours of it. The figures refill within five minutes of play, and a latency incident that was open stays open until the customer recovers.
- The two p99 fields on a customer record changed meaning. `p99_latency_ms` covers the last five minutes and `p99_recent_ms` the last minute. Both now rank every request, counting a failed request at the timeout.
- Client and server must match. Both need to be on v0.1.68.
- A customer's detail record gains two fields. They give the number of workloads answering and how many of those are degraded.