A Letter From Our Producer - 10th June 2020

Hello Squaddies!

Your Producer Arnaud here, along with designer Tom ‘Fuzzhead’ Fancsy. It’s been a while since the last producer letter and with Beta 19 recently out, it is the perfect time to huddle up for a sitrep about all the developments happening with Squad under the hood, as well a taste for our future plans.

First - we want to thank you for your reception to our Black Lives Matter post. The positive messages we received from players exemplified the type of welcoming, inclusive and supportive community we want to continue to expand within the game.

The current State of Squad

The B19 Update

As our community is acutely aware, our latest release took longer than we had intended. We are grateful to our players for their patience! This update included a full Engine Upgrade to Unreal 4.23 - a challenge in itself but we aimed to literally rip out the skeleton of the game and reinforce it to one lined with Adamantium. As we found out, this wasn't the most seamless process and while we were able to make significant changes we were also met with other issues such as our VOIP system breaking, dozens of client and server crashes, game modes not working correctly, file size directory doubling in size, LOD Issues, to name a few.

We also appreciate your patience with the several releases we had to provide in order to support the issues that arose from the b.19 release.

Despite these obstacles, this Engine Upgrade brought a lot to our internal development capabilities. You can read the previously released detailed patch notes for B19 here, but we wanted to touch a bit more on the specific improvements this Engine Upgrade provided:

1. Powerful, new diagnostic tools will enable our Devs & QA to more quickly find and tackle issues that affect performance and stability. This advancement allows the team to go full steam ahead on optimization.
2. 4.23 has been a real boost for Environment Art -  several advancements have been made, some of these are:

• Auto-Instancing Static Meshes - this is the single biggest improvement to Environments for 4.23. It required extra work from our end to get working correctly with Squad, but this has vastly increased our performance budget in many situations. Before this, when there were multiple instances of the same static, level designers would manually have to go through and set them to "Instance" based on their proximity to each other to save memory cost. This is now automatically done in 4.23, which in most cases means more optimization, and no extra development time - Sweet!
• Fog Improvements - 4.23 has improved the fog system, so now our environments can have 2 layers of height fog at once. For example, a map can have low fog that's down in a valley, and can also have a thin fog that affects the entire map. We know fog has been a hotly debated topic, especially regarding the reduction of view distances when helicopters were introduced. We will continue to tweak the fog levels to find that balance between epic long-distance views and vehicle on vehicle long-range battles and the need to optimize and improve performance - the more view distance, the more strain on a player's computer performance, typically.
• Lighting Improvements - 4.23 has improved the accuracy of shadows in some cases, but this needs to be tweaked manually. We know that lower shadows settings have been an issue in Squad for a long time, and are working towards a solution that is both performant, and looking great! We will use every tool available to us to continue to bolster our lighting efforts, and 4.23 has introduced some more tools. We would like to hear what you think of some of the new lighting improvements on our map layers, as well as your thoughts on shadow settings.

B19 Features

While B19 was primarily focused on the 4.23 Engine Upgrade, we couldn’t resist adding a few features that were prototyped but never implemented.

Body Dragging

The first new feature to B19  is something that has been requested for many years, and something that the Squad dev team always wanted to implement but was not able to fit into the schedule. That feature is Dragging - the ability to grab your teammates and literally drag him/her to cover to provide First Aid securely.

This prototype has its roots in our former ragdoll simulation overhaul back in A15.4. We decided to fix the many issues with our ragdolls not being networked properly or looking correct. One issue was one player seeing the ragdoll body in a different location than another player. All of these issues had to be resolved one by one, with a lot of trial and error. Without those issues getting resolved, dragging would never have been possible. We then got a rough working prototype of the dragging system. Several months later, we were able to take that prototype, work out most of the bugs with it, and get it implemented!

We believe players will enjoy the added immersion, as well as the utility of being able to pull your squadmates to safety. More tweaks and fixes for the dragging feature will likely occur down the line. We would love to hear your feedback on how your experience with the dragging system has been!

Helicopter Tail Rotor Destruction

Another new feature we added for B19 is the ability to destroy tail rotors with projectiles. Previously in B17/B18, the Helicopter tail rotors were only destroyable by colliding with terrain, buildings or other vehicles. Now players can target these tail rotors directly, with small arms, AT weapons, and most effectively with larger calibre rounds like 12.7mm, 14.5mm and 25/30mms. We hope you have now had a chance to witness the spectacle of knocking out a helicopter’s tail rotor and revel in watching the bird spin out of control into a crash landing!

Nvidia Highlights

The final major new feature for B19 is adding Nvidia Highlights support for Squad. This new feature allows players that use GeForce experience to have moments in the game automatically save clips based on specific triggers in-game. This feature can be turned on and off and maybe a fun way to relive those moments if you are like me and often forget to hit the "save last 5 minutes" button after something epic has happened. We have only added a couple of initial trigger events for B19, they include: destroying a helicopter with an AT Launcher, Triggering an IED and knifing an enemy. More trigger events will be added in future releases. After using the Nvidia Highlights system, we'd love to hear from you about what you think would be good trigger events for this new feature!

B19 Challenges

B19 full development was underway early in December 2019, roughly 3 weeks before the release of B18. We were aware that Unreal Engine upgrades had caused a lot of headaches and unplanned issues in the past, so we prepared a plan of action that would sequester the update into its own file branch. This allowed the rest of the development to happen unaffected by the 4.23 upgrade until the significant issues of 4.23 could be worked out and safely merged into our main file branch.

Early on, while 4.23 upgrade was being done in an isolated branch, there were many issues, some of which included:

• Several major client and server crashes, some related to PhysX DLLs and NetGUID.
• Many FX particles appeared to play at 1.5x the speed they should play at. LINK
• Dithering appeared to look much more severe on 4.23 than previously, it turned out that it was not related to dithering, but rather due to a new way that 4.23 handles world position and pixel depth offset.
• UMG Optimisations vanished - icons and vehicle scopes went to 4k because UI textures no longer generate mipmaps automatically in 4.23.
• Squad directory took double the size of disk space on hard drive due to 4.22 issue.
• Vehicle wheel effects were spawning waaaay too many particles, causing tons of overdraw and crushing framerates. 
• Environment meshes looked like they were culling/disappearing when in fact they were LODing down to a specific mesh after LOD0 after the 4.23 engine upgrade.
• Gridlines on the map were flickering while zooming the map in or out after 4.23 upgrade.
• Helicopters were impossible to fly due to constant warping after the 4.23 upgrade.
• RAAS and AAS Capture Points did not unlock after the 4.23 upgrade. 

Once all the many major issues were resolved, we finally had the go-ahead to merge 4.23 into our main file branch, this occurred in late January 2020. After this, the real testing could commence with some internal testings with up to 20 or so developers. Those tests went reasonably well, though some additional issues came up that needed to be fixed:

• DOF (depth of field) settings changed a lot in 4.23, which caused all sorts of issues with blurry ADS scopes, blurry Helicopter instrument panels and other things. This issue took a significant amount of time to figure out and nail down, and it was exacerbated by a discrepancy between how things looked in the editor vs in game.
• Capture points became neutral then captured when the owning team player left and then re-entered the cap zone.
• The Low-Quality Audio option in the audio menu caused several in-game issues after 4.23. This option was removed as we were under the impression that in 4.23 this setting would not provide only a negligible performance improvement, and it still continued to create many issues. We later realized despite our best intentions, this was incorrect. We’ll dig into that in more detail later on in this post.

After this round of internal playtesting and fixes was over, we felt confident enough to hold our first closed public test session of B19 with the 4.23 upgrade in place. This test had roughly 70 players and was successful (no major crashes), but several more issues came up that took considerable time to deal with.

• Proximity VOIP transmission was delayed by several seconds, it was only first observed in public playtesting and did not affect the Squad/SL comms.
• Special unicode text characters that used to work in Squad B18 were no longer working in Squad B19 due to 4.23.
• Vehicle Turret & Emplacements were lagging / rubber banding/ warping when moving the camera around.
• Grenade projectiles did not respond to Physmats nor Material Effects anymore.
• When placing Map markers, there was a long delay between placement and them actually appearing on the map.
• FOBs were able to capture objectives.

These issues were fairly significant and challenging to track/test locally, which slowed our development down. Eventually, they were fixed. B19 went to open public testing on the Squad Public Testing app in late March / early April 2020. This was when we discovered the following new issues:

• Capture Points could not be taken when a FOB or Rallypoint was nearby.
• Placement of FOBs and Mines would break at seemingly random times.

Those final issues that were found in the open public testing were addressed, and we were confident that we had finally ironed out the remaining priority bugs that would have caused any degradation from the previous B18 experience. B19 was released May 5th 2020.

Initially, things since the release seemed to be going well, all the previous reported issues had been resolved, and we thought we had a smooth launch on our hands. Initial reports from server owners were that there were much higher than average “Ticks Per Second” on the servers, which meant server health was improved upon for B18. Unfortunately, B19 issues were not over. The real problems of B19 started rolling in:

On May 7th 2020, reports began trickling in about clients crashing in the server browser. As we began to investigate, soon after huge waves of players were crashing in the server browser, and it had dropped the player population of Squad to very low numbers. With all hands on deck, we had assembled a task force team to combat the threat, which turned out to be a concentrated external attack on the Squad community. Through fast action, the threat was mostly neutralized by quick action by our task force team, b19.1 and b19.2 addressed the issue and by Saturday May 9th, things had stabilized.

• Players were reporting much worse client performance in B19 compared with B18, which was supposed to be the opposite. There were several culprits here: b19.3 addressed map performance on Tallil, Mutaha, Al Basrah, and other maps, that were using an outdated “HISM” tool, that now caused massive performance issues with 4.23, this was not picked up on our previous tests in those maps. More client performance problems were addressed in b19.4.1 that were caused by Bleeding Icons on the HUD, the way these were drawn was no longer efficient in 4.23 and caused massive hitches on players machines. Another client performance issue that appeared on several maps was fixed in B19.4, this was caused by an old Telephone Cable Component tool. This tool was used on several maps to connect telephone cables to the poles, and it became bugged with 4.23: normally the cables are placed in the editor, and upon runtime CPU process generates them on the map only for a second, and then they go to “sleep”. 4.23 caused this “sleep” function to break, so the cables never slept, which caused an extreme strain on client CPUs, constantly trying to calculate how the cables should sit.
• Certain Players were not able to launch Squad at all. Through investigation, it was discovered that a certain new Plugin in 4.23 had “AVX” enabled, which caused some CPUs to be incompatible with Squad. Disabling this plugin by default restored compatibility for these Players.
• 4.23 had changed a lot under the hood regarding Visual FX. Extensive work had been done to optimize most VFX in Squad for B19. Unfortunately, more work was needed to get back to B18 levels of optimization on that front, so B19.4 saw an extensive amount of effects changed and optimized to meet the standard. This was all mostly done “under the hood”, so from the players perspective little should have changed visually, other than getting back FPS when lots of effects like smoke grenades, explosions, etc are happening all at once.
• The Low-Quality Audio option is a setting in Squad that is in place specifically to cater to Players with AMD FX CPU’s. The AMD FX CPU does not process the UE4 audio engine correctly, so a special mode was created to help reduce the performance burden that UE4 imposes on this particular CPU. This Low-Quality audio mode reduces audio complexity, the number of sounds that can play at one time, etc. The 4.23 engine upgrade broke several sounds in this Low-Quality Audio mode (missing sounds for M249 SAW and other weapons, various distance sounds not working correctly, etc). The information available at the time of the engine upgrade suggested that only a very small portion of AMD FX users required this sound quality mode in 4.23, so the decision was made to fore-go putting effort into delaying B19 further (which was already months delayed), in order to fix the compatibility issues for the Low-Quality audio mode. We came to realize after the full release of B19, that this was not the case, in fact, ALL AMD FX users suffered massively with the B19 update, and the Low-Quality audio mode was their lifeline to getting Squad to perform well enough in order to have an acceptable experience. Although AMD FX CPU’s fall significantly below our minimum system requirements for Squad, our team wanted to do our best to support these players where we could. As a result, we also dedicated time to fix all the 4.23 compatibility issues with the Low-Quality audio. These fixes were implemented in b19.4.1 and b19.4.2, released May 26th and June 1st. Based on all the other struggles with B19 and getting it ready for launch, It was not an easy initial decision to drop the support for Low-Quality audio mode, and unfortunately, a lot of  Players running AMD FX had to suffer for 3-4 weeks before getting relief to play Squad again, we apologize for this.
• One positive aspect to B19 during this time was the implementation from our dev ops team of our new “Flexible releases feature”. This allows us going forwards, to release “micro” versions, which don't require both the Server and Client (player) to update. So for the b19.4.1 patch, the “.1” indicates its a micro release, and was only necessary for the players to download from steam - Servers did NOT have to update to b19.4.1 because the changes only affected player’s local machines. This saves server owners a lot of hassle having to update and restart servers, and vice versa, when there is a server-side only patch, only the servers need to update and Players do not have to. This will be an incredibly useful feature in the upcoming months ahead, as we will be able to roll out hotfixes in faster succession, as well as trialling fixes for specific issues on public servers, for issues where it is not as easy to reproduce in a test environment.
• Player’s have reported major rubberbanding/jitter/desync issues compared to B19 with Helicopters, Ground vehicles, and Soldiers. We are still working to address these issues - we are working diligently to solve these issues and bring the experience back to B18 levels, and then improve further. B19.5.1 was released to Servers on June 9th, which was a fix pulled in from UE 4.24, which should alleviate a lot of the rubber banding issues with helicopters. The issues with ground vehicles rubberbanding/desyncing is a top priority and are currently being worked on. Soldier desyncing issues are also being investigated and prioritized.

The Modding SDK and B19

The Mod SDK for B19 was released shortly before B19. Several issues needed to be addressed to get our mod SDK compatible with Unreal Engine 4.23. Shortly after the B19 release, several crashes and other issues had been reported regarding cooking mods with B19. There is ongoing work to resolve all the issues our modding SDK is experiencing due to the 4.23 engine upgrade. Our main focus currently is the shadercore crash when loading custom mod maps. Our team is talking to select modders to get more light on the issue. We are making it a priority to get this resolved and understand the frustration our modding community is currently feeling.

Dev Team Process - A Look-Back, and Movement Forwards

In light of all these recent obstacles, it became clear that we needed to change our approach toward Releases. In the past, we had let you know our commitment to getting out more frequent, monthly releases.

We recognize that the engine upgrade process has been a very lengthy one. So going forward, we will be taking measures to keep our releases within a more satisfactory timeframe. We currently have no plans to do another engine upgrade this year. However, we may decide to add additional new features from Unreal Engine 4.24 / 4.25 if it is low risk and beneficial to Squad, especially for optimization.

Our plan moving forward includes:

• Following a new, updated workflow internally.
• Shifting how we “package up” releases, allowing us to release in smaller chunks more frequently
• Ultimately this should allow us to release updates within shorter periods.
• Continued commitment to transparency in our development process, along with more timely transference of this information.

What future Squad Content Can Players Expect?

Although Squad is closing in on having all its core features, you can bet your ass our dev team will still be delivering premium content for the foreseeable future!

We’ll be releasing a new roadmap with further details soon, so stay tuned for that. We want you to know about our upcoming plans, and we want to fill you in on more of the details as content gets closer to completion. There is unannounced new content to come, with our development plans for Squad already extending into 2021!

A few things you can expect to hear more about and see on our roadmap very soon:

• We have a new map coming! We know some of you have heard rumours about Fallujah! Let us tell you, this map is a superb work of art!
• We have a new faction coming! 

We have already started in on more than one new faction, and while these are much further out in terms of development, we look forward to sharing with the player base soon! For these new factions, we want to provide deeper purpose and gameplay features that fit their personality.

We have plans for even more additional content beyond this that we’ve already started work on!

Our roadmap will be dropping over the next few months, where you can expect to learn a bit more about this.

What to Expect in B20 and Beyond

• After the long haul of B19, the Programming team has started a deep dive into optimization. The focus is on tackling as many networks focused optimization issues and features as possible.
• Increased Player Pop Server Tests - increased player count is indeed on our minds. We know this is an essential aspect of bringing the full Squad experience together. If server performance allows, we will unlock the increase of players to whatever upper limit still retains a stable server performance. We also know it's going to be very challenging to arrive at 100 players, and it may just not be possible with our current engine. But we will do our absolute best to push this engine to its limits, while still delivering a quality experience.
• Option for Squad VOIP / SL VOIP L/R Ear Selection - This is a long-requested feature by many SL’s and CO’s, and it is currently under development.

If you’ve made it this far, we really appreciate your dedication! It took a lot of time and work to finalize B.19, as well as hotfix some of the new issues it brought. We are working toward a more manageable release schedule. We look forward to pushing ahead with this new update strategy!

Until next time Squaddies, stay frosty!

Arnaud ‘Arno’ Lebesnerais - Producer
Tom ‘Fuzzhead’ Fancsy - Game Designer