A quick preamble before the post. I'll mostly be discussing machine learning, so I want to preface with how development is going. I made steady progress in April and I'm finally feeling the benefits of my rework that began in November. But as I move into May, the weight of the festival is really starting to dawn on me. I'm worried about completing the demo in time, and I'm struggling to decide which features can wait and which ones need to make it in. I'll continue to work hard in this coming month, but I can't guarantee I'll complete the 100m dash in time. It may just take a few more iterations.
Outset of Machine Learning
I wrote a while ago about refactoring my entire code base. The goal was to automate some processes and abstract functionality to create a more streamlined development for interconnected game systems. As I talked about it then, I had yet to work on "modular actions", which was a way to categorize any action for AI parsing.
In April, I set up the basis for modular actions and began working on AI. My intention with the AI has always been to create an intelligent foe for the player. Part of the reason for my refactor was to make the game world a more comprehensive whole for AI. I knew that the modular actions would greatly help that goal, but I didn't realize how easily that would translate to machine learning.
It's funny because machine learning was something I looked into many months ago. I wrote it off as something too ambitious for the project's scope (at the time the code's foundation was not as robust as it is now). But as a byproduct of creating this new code base, it became much easier to implement machine learning. I kind of stumbled into it though. I was making the AI and thinking "they should be able to remember actions, so that they can reference which reactions went well, and which ones didn't." And with modular actions, the AI's 'reactions' were in an array.
Basics of Machine Learning
If you know anything about machine learning, you'll know that-- at the most basic level-- machine learning is the process of an AI selecting from a numbered list of choices. It then catalogues those selections and assigns them with a score based on some predefined scoring system.
Machine learning at first glance may seem like some path toward sentient AI, but it really just represents an alternate perspective into crafting artificial intelligence. The traditional approach is to lay out every possible scenario an AI might come across and hard-code a response to that scenario. "If the player slashes at me with a sword, then I will block". In a sense, you are writing the AI's knowledge, but leaving it with no wisdom to make its own choices.
Machine learning, in the most laissez-faire approach, is the exact opposite. Rather than considering what scenarios an AI might go against, you consider what outcome you want from any possible engagement. The easiest example would be a simulation of a straight 100m dash. As a programmer, you would create a scoring system which rewards closing in on the finish line and discourages moving away from the finish line. "If I move closer to the goal, +1 point. If I move further from the goal, -1 point." You then give the AI a numbered list of 5 actions: stand still or move forward, backward, left, or right. The AI randomly selects one of the 5, then calculates a score based on that choice and the outlined rules. Over a thousand iterations, the AI realizes that option [1](moving forward) is the only proper choice. Rather than giving an AI knowledge, you give the AI the wisdom to make its own choices.
Here's how machine learning is beneficial: using the last example of a 100m dash, let's say some game designer places an obstacle midway through the track that blocks the AI from moving forward. You, as the programmer, had no idea the designer was going to do this. With the traditional approach to crafting AI, you'd need to go back into the code and hardcode a case for coming across an obstacle. But with machine learning, all you need to do is run another thousand iterations for the AI. It will come across the obstacle and get stuck for a few hundred iterations, but eventually-- through pure dumb luck-- it will figure out that it can move right/left to bypass the obstacle and continue towards the goal.
Crafting Machine Learning
With all of that explained, let's jump back to my own project: I've just finished making this modular action system, which just so happens to create an array of actions for the AI. This array can easily be abstracted into a numbered list of choices for the AI. I need some way for the AI to learn from the player's actions. And that's when the epiphany struck me. My mind whipped back to that research on machine learning, and I started working on crafting a memory for the AI.
It was surprisingly easy to set up, and I gave the AI one simple rule : "If I have more health after this encounter, +1 point. If I have less health after this encounter, -1 point." There's a lot more nuance to be coded (I'm not going to rely on random selection), but the foundation is in place. I've been testing around with the machine learning, and it's already proving to be a better methodology for achieving my goal of crafting an intelligent enemy for the player.
Machine Learning Is Not A Shortcut
Machine learning may seem like some magic way to avoid coding, but there's a lot of work into defining the rule set for the AI. Again, this isn't some path towards sentience, it's just a different methodology. In my game currently, the AI's run into a case where the player makes a melee attack. The AI chooses to dodge (based on a calculation of how fast it can move and how soon the player will strike), and moves on the quickest path away from the strike's origin (which is another calculation as well, for the shortest distance out). However, while this works in most cases, in one instance the AI moved backwards into an explosive barrel. The scoring was processed, and the encounter was tagged with a score of '-5', since the AI lost 5 health from the explosion.
Unlike with the previous example of an obstacle in the middle of a track field, the explosive barrel can be moved, and the player can attack the AI in a multitude of locations from a multitude of different angles. We can't simply run a thousand iterations to have the AI avoid the explosive barrel because every iteration will be different than the last one. More importantly, the AI doesn't know that the negative score was due to the explosive barrel.
So with the case of trying to avoid moving into third-party hazards during an encounter, simple machine learning won't solve the problem. I need to go in and design a method of detection for the AI to sense if they are moving into a safe position as they attempt to dodge an attack. This is just one example of machine learning not being a shortcut to great AI, but hopefully it sheds some light on the reality of machine learning: it's just alternative method of programming. Rather than crafting cases, we craft rules. For my game particularly, I think the latter approach will result in the sort of gameplay I'm aiming for.