Hey everyone,
We hope you're enjoying Update 7 and battling through the streets of Carentan.
The team have been hard at work combing through all of your feedback, from bug reports to your thoughts on the new features and changes that came with U7.
Thanks to your bug reports the team were quickly drawn to FPS issues and game crashes reported in Carentan and in some cases, other maps.
We're pleased to announce that Update 7 Hotfix v1 will go live tomorrow (Friday) morning BST.
The main focus on this initial Hotfix will be:
- Optimisations for Carentan
- Optimisations for new assets
- Player model optimisation
- High certainty fix for Crash 01
- Speculative fix for Crash 02
To tell you more about Hotfix v1, as well as the follow up Hotfix v2 that we're already working on, Lead Developer Max took a moment to pen a Hotfix and Development Status update for you all:
An update from Lead Developer Max - Hotfix v1
Hi everyone,
Thanks for jumping into Update 7 and for the great feedback.
We're keen to give you the full status update of the immediate next steps.
As many people have reported, several significant issues presented themselves immediately once a large number of players joined the game for this new update. While we QA each release for weeks, the largest issues normally only present themselves as soon as you start throwing tens of thousands of players at the game. This is always hugely disheartening for us as a team, as we've spent the better part of 6 weeks QAing the game and fixing more than 900 issues to prep the Update for launch. As you can imagine, having the work overshadowed by frustrating issues for you all is unideal at best! Fortunately, we've been provided with huge amounts of very useful feedback by you - the community - as well as our brilliant QA team who have been able to live-test in-game all week. We're also fortunate that our crash uploader has been working perfectly, enabling us to quickly assess the frequency distribution and triage the most common occurring to the least occurring.
The whole team has been working overtime on this as we know how fundamental many of these issues are to a great experience.
Current client crash breakdown:
- Client Crash 01
Makes up approximately 28% of crashes experienced.
- Client Crash 02
Makes up approximately 40% of crashes experienced.
- Client Crash 03
Makes up approximately 30% of crashes experienced.
FPS Dip:
We are actively investigating a severe FPS drop of approximately 20-120 seconds that can happen mid-game on Carentan (we are looking to see if this is happening elsewhere). We are confident that we will be able to find and address the issue, but it will require some time and QA testing to make sure we understand the problem to its fullest, and make sure we don’t introduce new bugs and issues upon trying to resolve.
Carentan Optimisation:
While we had been continually testing and optimising Carentan on a huge variety of machines prior to the Update, there is something unusual occurring specifically with Carentan that emerges at 100 players, and varies wildly from machine to machine - often a mixture of different hardware causing very mixed outcomes. Players with traditional middle-spec machines have experienced higher FPS in Carentan than players with high-end machines. In addition, there have been other complicating issues caused by what we are terming the “FPS Dip”.
What we do know is that optimisations were needed on the CPU side. Ironically, the map itself culls objects too effectively, and those objects (and the act of culling them) are run through the CPU - often locking the CPU up and causing all sorts of issues as a result.
To this end, we have implemented several fixes that should immediately see a performance increase. With the performance increase, we will start to get a better idea of what are standalone issues, and what are issues that are a result of that poor performance (CPU bottlenecking as a result of poor performance may be contributing to or causing the FPS dips experienced).
We also want to let you know that Carentan - like all maps we release during Early Access - will be subject to continual fixes and optimisation. We will be working continually to improve all aspects of all 9 maps we have currently in the game to meet a series of benchmarks for minimum acceptable FPS. Now that the animation systems update has been completed, we’re in a great position to dedicate our time to this and the polish and fixing of bugs that have arisen.
Immediate Plan:
Hotfix V1: Friday Morning BST
We have been testing a Hotfix to address a majority of the issues experienced over the last two days. As soon as we started to receive crash log uploads and could profile more broadly in Carentan we began immediate work on troubleshooting them.
The Hotfix will address the most common crashes, as well as include optimisation for Carentan that will improve frames across all machines.
- Optimisations for Carentan and all new assets present.
- Optimisations for player models (will list frames across all maps - especially in high population servers).
- High certainty fix for Client Crash 01
- Speculative fix for Client Crash 02
- Ongoing investigation Client Crash 03 (this may be tied to or solved by the fix for CC02)
Hotfix V2: Release TBD
While Hotfix V1 aims to address the majority of issues the player-base is experiencing, Hotfix V2 will address any issues that Hotfix V1 failed to solve, or any that crop up after the deployment of HF V1. As a team, we always want to build contingency plans out to make sure we have the best ability to solve any recurring issues quickly - as opposed to leaving these issues live in the game until the next Update.
We will be deciding on the timing of HF V2 immediately after assessing the effectiveness of HF V1.
Free Weekend:
As many of you may already know, we have a Free Weekend organised for this coming weekend. Fortunately, one of the reasons we brought forward our Update day to Tuesday was to give us several days of buffer in order to apply a Hotfix before the Free Weekend got under way.
During Early Access, it’s always a tricky balance to combine marketing events with significant systems overhauls. We really appreciate your patience with us as we work to get this right.
The New Meta, Loadouts, Content, Bugs and Everything Else
While our focus has been on addressing the immediate critical issues affecting players, we’re collating all feedback from our channels and building out a document to capture all perspectives and feelings about the new meta, loadouts and general game functionality.
In addition to this, some small but humorous animation bugs have popped up - most around leaving a vault or climb. We also have recorded various bugs and issues regarding black marks on the map and Recon marks capturing your own team, as well as the turnback mechanic on Omaha Beach not functioning properly.
While technical issues and bugs will always be triaged and fixed, we’re making sure we play the game continuously in order to get a sense of the new meta. We’re reading all feedback and are playing alongside the community in order to watch where it develops.
Generally, we like to give each meta shift several weeks of playtime to watch as the community evolves with it.
After this time, we’ll be looking at what changes should be made, balanced with coming features and changes to specific roles.
Finally, we’re also looking at the option to disable or scale all camera movements. We can see that some players are experiencing motion sickness due to new camera animations.
Roadmap
Due to the hotfix and patching that will be undertaken, we’ll be revising the next roadmap to address any residual issues, as well as to make sure release timing is correct.
As soon as we have revised timings, we’ll be releasing the roadmap.
It’ll feature a huge list of everything you’d expect, but with a particular focus on optimisation and bug fixing.
Conclusion
While Hell Let Loose is an Early Access game, we are very keen to not use that as an excuse for poor performance, crashes or any other issues you may experience.
Moving forward, we’re now free to spend significant time optimising all the content that has been added in the last year. Fundamentally, we’re incredibly relieved not to have any entire systems overhauls necessary to complete.
Update 7 was a huge and very risky step forward for Hell Let Loose, but it was a necessary step in order to make sure we were set up correctly for the future. As with all fundamental overhauls during Early Access, it’s always a fine line to navigate between marketing the game and managing the risk associated with the changes.
We’re very excited as a team to work from our new benchmark to continue to improve all aspects of the game, as well as implement new features.
As a team, we’ll be playing a lot this weekend and next week. If you see us in-game, please come and say hi!
Lastly, thank you for all your patience with us during the course of this Update and the ones before. Updating such fundamental aspects of the game has come with huge challenges and we appreciate that it’s been a turbulent experience. Please know that moving forward the technical risk we’ll need to undertake will be much less significant, and that the work done so far puts us in a great position for the future.
New & Updated Tactical Maps!
Following on from Max's update, lets take a look at both brand new and updated tactical maps for your strategic pleasure!
First up we've got the newest addition to the game, Carentan!
Next up we have the freshly overhauled Foy!
As well as the updated Omaha Beach map!
Last but not least we have the classic HLL map, Hurtgen Forest!
Not only are these maps a fantastic way to plan your battle strategy before the bullets start flying, but you can even line them up with real-world maps to see exactly where you're fighting in-game.
Steam Free Weekend - Live Now!
Those looking to join the frontline or get their friends to join their unit, we've got good news!
Starting at 6pm BST / 7pm CEST today we're hosting our next Steam Free Weekend.
The free weekend will run until 6pm BST / 7pm CEST on Monday 20th July and includes a 25% discount on the game!
https://store.steampowered.com/app/686810/Hell_Let_Loose/
This is an incredibly exciting time for all of us as we open the landing craft doors to both new players and those who want to see how the game has progressed though Early Access alike!
We know that as a community you'll welcome these new recruits with open arms, bringing them into your units, showing them the ropes and maybe even giving them a spin in your shiny new Shermans and Panthers!
Genuinely, we can't say thank you enough to you all for that - those actions really show off the best of the Hell Let Loose community and keeps it growing.
If you happen to be one of the new free weekend players, we've got various global community hubs (Including here on Steam) where you can chat HLL, bask in glorious memes and find yourself some other players or a clan to play with on the regular!
Before we wrap up we'd just like to say thank you to every once again for your support this week.
From the huge game changing Update 7 deployment, to your patience and feedback so we could create a Hotfix for FPS and crash issues you have all been fantastic - thank you.
We've always believed that you, the community, are the heart of Hell Let Loose and this week, once again, you showed us why.
See you on the frontline.