When you work in an indie studio, you often find yourself needing to do a couple of things at once. We sometimes have to become testers/accountants/electricians/cleaners for a day. That’s what makes this job so fun - you never know what the next day is going to be like. That being said, in order not to become buried in repetitive tasks we automate some processes. One such process is benchmarking and that’s what we are going to tell you about today.
Visit r/programmerhumor subreddit for more quality content.
Benchmarking is the process of measuring the performance of a game or app. Every night, once the clock strikes midnight, one of our buildmachines wakes from its deep slumber, builds the game package (fresh .exe + all the assets) and runs a series of automated tests. We built tools into The Riftbreaker that output all the data in a form that is easily readable for regular human beings. We also accumulate all the results on our build management server in the form of graphs.
We already showed you this sad corner of tech and dust - it runs our benchmarks every night. The pink pony makes sure it does.
Our benchmarks are scripted scenarios that are intended to run in the same, or at least predictably similar way every time. The first one simply opens the app and shuts it down, measuring the time necessary to perform these actions. Then, we have two battle scenarios - one with a reasonable amount of enemies and one with an endless horde attacking a huge base. These tests show us the game’s performance in low-intensity and high-intensity scenarios.
We test the performance in high-intensity scenarios...
There are several reasons why we conduct these tests with such regularity and pay close attention to the results. First of all, the successful completion of each of the benchmarks lets us know that the game is working properly. With more than 10 people committing changes to the game every day it is not always possible to make sure that nothing went wrong. Sometimes a change that worked locally for one of the developers breaks the game for everybody else. Since we run tests every night (we can also run them “on-demand” during the day) we have fresh results every morning. If any of the tests fail we can immediately start fixing the issues, ensuring that we have a new, operational version of the game every day.
...as well as low-intensity ones.
Running tests every day lets us know that all the game elements work the way we intended them to. If any of our changes cause game systems to act incorrectly, the results of benchmarks change drastically. For example, one time when our towers stopped working, the number of enemies left alive at the end of our high-intensity battle scenario increased dramatically. We immediately knew that something was wrong and started looking for solutions.
What is this "GameplayState"? Why does it take so much time? Do we need it? Can we get rid of it?
Apart from making sure that the game works on a macro scale, we also gather individual data for much smaller markers. The cost of every system, measured in milliseconds, is being recorded and plotted on a graph. It allows the programmers to see how their work affects the overall performance of the game. They can see how much time each process takes to calculate everything that is necessary to render the next frame. This method allows us to determine which processes affect performance the most and determine the candidates for further optimization. We also measure the overall loading time of the game and take measures once it becomes way too long for comfort.
The loading time of the game since the very beginning. More content makes the game take longer to load, but we will find a way to improve it.
At the moment we run benchmarks on one PC configuration only. Later in the development cycle, we are going to run benchmarks on more machines, both high and low-end, as well as consoles to ensure that the game runs properly on as many hardware configurations as possible.
There is one idea we are toying with and we would like to ask for your opinion here. A lot of games have a built-in benchmark that lets you quickly check how well your PC will handle the selected settings. That’s useful, but there is nothing more you can do with that information. Would you like us to introduce ‘benchmark leaderboards’ and let you compete with other users for the highest score? Let us know in the comments and on our Discord! www.discord.gg/exorstudios
That’s everything for today. Take care and see you next time!
Other social media:
www.facebook.com/exorstudios
www.twitter.com/exorstudios
www.mixer.com/exor_studios
www.twitch.tv/exorstudios
www.youtube.com/exorstudios