Icarus Week 176 Update | 4 new Brick Signs to place and personalize

This week, we’re adding a brand new set of Brick Signs, with four variants for you to craft, adorn and place around your base.

We’ve also got some insight for you into our development process after we had a perforce outage last week that impacted our development timelines.

Finally, we’ve got some details on an ongoing feature development - a solo pause option.

This Week: Brick Sign Set

This week, we’re adding a new set of Brick Signs, with four different variants for you to craft.

These are unlocked at Tier 3 and crafted on the Advanced Masonry Bench. Ten clay bricks, two concrete mixes, and four iron nails are required for each sign.

The four variants come in floor and wall options, with tall and angled variants for use in different terrains and settings. Just like all other signs, these can be edited to display your own text or icons.

You may have noticed in last week's patch notes that we also improved how signs appear in-game by removing the artificial glow and making them look less digital. The full note is below:

Update Sign widget material to use Decal domain which allows it to take on normals of sign mesh and handles lighting better, removing emissive glow, making the signs look less digital and cohesive within the scene

This Week: Perforce

Last week two Perforce crashes happened on two days in succession. Functionally, this turned our 5 day work week into only 3 days for our normal weekly patch.

Perforce is the source control system we use for Icarus, keeping the developers working in alignment on the code and assets required to make all the different versions of the game (at minimum each week is a new version). Historically, our Perforce instance has been running (not up-time) since June 2016, with our first Icarus code commit seen as 2019.

Since those first commits, the source control has grown to 6TB of project data. We use a Google Drive to keep all the full sized Icarus art assets, so the true scale of the project files is much more. The 6TB is just the raw foot print of the source code and binary assets to build the game for distribution, with the latter accounting for the majority of the 6TB. And then to ‘serve’ the data effectively to developers and build servers bring along additional requirements.

The reason any of this is part of a blog post is because one of our servers experienced a “Blue Screen of Death” recently and caused corruption in our Perforce database files. Bringing Perforce back up required multiple database reads, rebuilds, and verifications. Last week two crashes happened on two days in succession, significantly effecting operations. As developers, we mitigated this by using the time to playtesting on steam and in a few cases juggled feature work in “offline” mode.

Hosting the 6TB of critical data means a lot more than a PC with a few SSDs. Huge failover batteries protect our whole office site, multiple high IO drives for RAID, spares, and backups, and we need NVMe drives to push the architectural limit to serve this data fast over the local network to multiple people. Many syncs early in the morning can saturate our 10Gb uplink.

Since we’re covering our live data footprint, we should also mention that for backuping up the data we provision 20TB of HDD storage so we can have daily backups, snapshots over time, and space left over to take a full fresh backup on demand. But you can’t just have one back up, you actually need a duplicate of your backup, so there’s another 20TB. Mirror that all again, as we must save it all in another form off site in the the cloud, so we provision another 20TB of cold archival storage in a datacenter that is in geographical region outside of New Zealand. You can see how just the 6TB critical data for Icarus has quickly escalated to requiring 100TB of provisioned storage for its stable existence.

Recently, rapid system halts and “blue screen of death” has caused us data corruption in our Perforce database, to the point Perforce is turned off for repair and the developers have to immediately adjust their work schedule. What does repairing Perforce involve? Perforce by design works and manages our files, the versions, and meta history for those, through several dozen paired databases. When the system halts and those databases are open or are being written, they break and they break in ways that are very bad for productivity. Luckily, there are specific Perforce tools and commands that are run as a check on the database file pairs and we keep documented in our internal wiki the process.

Fixing the corrupted perforce instance has a few steps. After the first attempted read of the database set, all the corrupt files are identified. A second pass is done to write out new replacement databases identified from the last successful checkpoints. These new replacement databases are replaced in their directory and another different integrity check is run on all the databases. This step is very slow and can take a several hours. If everything goes well, there are no more corrupt database files found. But if we find another corruption in this check, we have to redo the whole process until all corruptions are fixed.

Ultimately, the Perforce service is returned to working condition, with developers and the build servers now all syncing at the same time. Now the developers are happy again and continue to work unimpeded in Unreal and the build servers can go back to assembling builds for you.

Coming Soon: Solo Pause

We’ve been working on a new feature for Solo play. It allows players to pause their session rather than the game continuing to run in the background.

This is quite a large change and requires layers of experimentation and testing internally to iron out any bugs or issues that arise. For example, multiple mission chains have to be tested as a pausing feature may impact (or impact, vice versa) their logic.

At this point, we’re putting out early iterations of this change into our experimental branch, which gives us a much larger testing pool to catch bugs or oversights that slip through the cracks. We’ll be ready to release this onto the main branch once we are comfortable it is at a standard we are proud of.

A quick note: this will not work on multiplayer or dedicated servers. This is due to the presence of other players and their own sessions, as well as the logic required to run a dedicated server.

Next Week: Chef Backpack

Next week, we’re adding a new ‘Chef’ Backpack for culinary arts specialists.

This will allow prospectors to craft better foods with longer-lasting buffs. Alongside a decrease in the food consumption requirement, there is also a buff to water consumption, so you’ll be able to last longer without two of three pillars of your prospector's well-being.

Your support makes these updates possible.

https://store.steampowered.com/bundle/40692/Icarus_Pets_Bundle/

https://store.steampowered.com/bundle/35727/Icarus_Complete_the_Set/

Changelog v2.2.51.135901

New Content

Fixed

Future Content