This dev log entry is not really about a particular bug. Although this very bug is a good example of what fixing bugs may look like.
I changed the death effect for a few reasons. One was the performance (which will be covered in a separate log entry) and another was that I didn't really like it that much. The effect was added very early into the development just as something to hide the disappearance of the character.
I can't do ragdoll, I can't turn every robot into many separate pieces (it would make the robot generation longer and the game already is sometimes a bit behind). The existing death effect combined two things: it looked not so great and worked not so great. I needed something that will look good and will also not hog GPU. Maybe they could turn into dust or something? They could glow up and turn into numerous bits. I realised that this is just like the death effect in Another World. I didn't want to have the skeleton (GPU, generation time) although maybe I could have some random parts inside in the future.
I decided to have a unique shader for that. That wasn't even using the normal rendering pipeline to make it as light as possible. Plus I needed something to do billboard rendering (billboards are quads perpendicular to the camera, or in non-vr games, to the camera plane). After a few iterations and tweaking, the death effect looked good enough. It only required some sound to be added (a mix of thunder, sand, tile breaking, steam and electric short). And the performance was much better! Job well done, right?
No. Because I noticed that many times the effect appears somewhat closer than it should be. It was especially weird in VR because you could see that it should be behind objects but it was rendered in front of them. And sometimes does not appear at all.
The first step of fixing a but is to find out when it happens.
My initial idea was that there was something not right with the material setup. Depth mask? Checked that, no.
I thought that maybe it is unique to Quest. It wasn't as I noticed that while testing PC builds.
I added some code to trigger the death effect on demand without killing any character.
After taking a glance at the shader where I couldn't see anything wrong, I decided to take a look into what was going on GPU. Enter RenderDoc, an amazing tool that lets you capture a frame, inspect shaders, meshes, GPU state.
One of the features it has is to have a look at what goes past the vertex shader. And let me quickly explain how rendering works, first.
The game provides information about stuff to render, mesh (what do we render), parameters (like colour, light setup, bones to see the character animated) and shaders (that tell how everything is processed from the input data to actually render it on the display). This goes to GPU. GPU then runs vertex shader to calculate the location of all vertices (to apply bones placement, then model, camera/view and finally project it to coordinates useful for further processing). Then the data goes to fragment shader that does the calculation of what every pixel actually looks like. This is very simplified as some steps might be moved from fragment shader to vertex shader, can be combined together, etc.
I thought, that the fragment shader couldn't be responsible for why the dust appears closer than it should. I focused on the vertex shader then. One of the features that RenderDoc provides is to have a look at the vertex data already processed by the vertex shader. You can have a look at it exactly how it looks in the game or... you can turn the whole thing around. And that's when I noticed that the dust is rendered in front of the character. And that's completely unrelated to the portals.
The white lines tell what the camera sees, with the camera/player's head being on the right here. The lines in the middle should be rendered at where the robot is.
I switched to an empty test level to have only things I need to be rendered there (and to make the iteration much quicker).
But I still couldn't figure out what was going on.
I decided to divide the shader code into smaller blocks that I could disable/change to see what works. I quickly realised that the problem was with orientating the dust to be rendered perpendicularly to the camera/eye. While doing that, I realised that I was mixing coordinate systems (mine is x-right, y-front/away from camera, z-up, opengl is x-right, y-up, z-back/towards camera). I fixed that but as expected, it didn't help much.
I started to check if the orientating is bugged. The thing about orientating billboards is to place the centre properly and then offset the corners using a transform that is based on the relative location of the centre of the billboard and the camera. The centre of the billboards seemed to be fine but when I was adding even non reorientated vertices, it was closer. But when I was adding only two or three components, it was fine. I had something like that in my code:
position += reorientateTransform * cornerOffset;
Which looked fine, it was adding orientated corner offset to the position, right? Even taking away reorientateTransform still was giving bad results. I realised then that I am working on vec4, four-component vectors. And the position is described with three components (xyz) and the fourth one set to 1. That fourth component is important as when transforming it with a matrix with setting it to 1 it creates position/location, using also position/location/translation part of the matrix but if that would be set to 0, it would ignore that info. And although reorientateTransform did not have any translation info, I decided to switch the 4th component of cornerOffset to 0.
And that worked. I had no idea what was wrong but that worked and I decided to call it a day.
Only when I got up from my desk, while I was talking to my wife, I had realised what was actually wrong.
With 4th component set to 1, the result of multiplication also had that component set to 1. And the position in the end, as a result of addition, had the 4th component set to 2. Which in the end resulted in the vertices being halfway between the camera and their actual location.
The bug was fixed.
The billboards are rendered at where the robot is.
And the bug happened because my mind was working on a higher level. And also because outside shaders, I use functions and methods that are descriptive (like location_to_world, vector_to_local) instead of using operators. But when I went to shaders, I just assumed that the result is the same, completely ignoring the position being a four-component vector.
It was a very basic error to make. Funny thing is that more complex bugs are sometimes easier to find and fix. This one took me a few hours.
Fixing bugs is a somewhat fun process. First, you have to confirm a bug, then, if it's possible, to separate it and check if it can be done in a controlled environment (so nothing else interferes), then you can take a look into the bug itself and if you have right tools, use them. Finally, if there's nothing obvious, divide it into smaller parts and try to find the cause. The worst comes, experiment by doing seemingly random stuff and have a look at the results.