DevBlog #4 - First small feature, todo list, sensitive data, server data

TL;DR

• Started to create new small features
• Making on a TODO list
• Removing sensitive info from the GitHub online source save
• Downloading data from the server to verify it

First small new feature: music slider

For a first new feature, I'm starting really small. As I worked recently on the Options, I added a small thing I always wanted for the game: a real time music volume slider. So we can hear in real time the volume of the music when changing it in the options.

TODO list

I am making a TODO list for everything to do for the game. I want it to stay really simple, and to be manageable line by line, so I can easily put higher the things I want to do first.

I selected a few action words:

• add: a new feature
• remove: something to remove from the project (including cleaning)
• change: modify an existing aspect
• fix: correct a bug

For writing a TODO element, I am starting with the action verb, then I write the part of the project concerned, from the , then the expected result. Everything is separated by commas.

Example :
add, Options, music, slider, real time volume change

For working on the TODO list, I am using Notepad++, with CTRL+SHIFT+ARROW (UP or DOWN), so I can easily move a line or a group of liens up or down.

I intend to publish the TODO list (or a simplified version) later, with a percentage estimate of the completion of the rest of the game.

GitHub: removing a file from history

When cleaning the folders of the project, I realize that there is a file with information I absolutely do not want uploaded anywhere on the internet. Sadly, all the project files are saved in GitHub, and a copy of them will stay online even if I delete them. Because this is the very purpose of a git history, to always save the whole history of all files.

So I am forced to rewrite the whole git history, to remove all mentions of this file and its content.

I use BFG Repo Cleaner and I follow the procedure.

git clone --mirror https://github.com/marckruzik/giac
java -jar bfg.jar --delete-files "myfile.txt" giac.git
cd giac
git reflog expire --expire=now --all && git gc --prune=now --aggressive
git push
The file deletion command cannot target a precise file path, only the file name. Hopefully, the file I have to delete has a unique name.

And after all of this, the history rewrite works!

On the bad size, rewriting the git history implies to reupload the whole git history. Even compressed, that's still 500 Mb of data. Here, the big textures are once again a problem.

Besides, the upload speed is slow, around 0.8 Mbps (Mega bits per second). Please note the lower "b", it's "bits", not "bytes", so it is 0.8 MB per second. To transfer 500 MB, it takes around 1 hour 20 minutes, and needs to avoid all internet connection problems (and if it happens, do the whole upload again). Commits usually take only a few seconds, as there is usually only a few Kb of code file changed. So I hope this very slow commit process won't create too much problems in the future.

Server Data

Data on the server seems to be fine. It looks like the game continued to record data over the years.

One moment I thought there was a bug with incorrect data. But it happened because I forget that when developing the game, there is a special "developer" settings, which connects the game to a developer version of the database. And this version of the database is out of date. So I set up a copy of the most recent data, and now everything is fine in my development version.

I use Postman to simulate the game's requests to the server. It simplifies the process a lot.

But I still need to confirm that the game's data on the server is ok.

Database download

To download the database, years ago, I was using the PhpMyAdmin online interface. Or Php scripts. I wanted to just download the last automatic save from the cloud (I am using OVH). Sadly, the database contains blob data (binary data), as all level solutions are saved in binary to take as few place as possible in the database.

It works very well to take less size, but it makes everything else more complicated (like export and import).

So, back to PhpMyAdmin, I do an export of the whole database with the necessary option "hexadecimal notation":

• UTF-8 (by default)
• compression: zipped (the download size goes from 50 Mb to 7 Mb)
• Dump binary columns in hexadecimal notation (by default)

All those binary data are now in a more manageable hexadecimal format.

The database dump is in MySQL format. But the game database uses the SQLite format, and it needs a conversion. Years ago, I made a simple Ruby script to convert MySQL to SQLite, by removing non working stuff, and replacing a few words and characters. Now I want to do it correctly by using a good tool, and I see that this is a complex process, usually involving bash scripts (with the .sh extension), running on CygWin or WSL (Windows Subsystem for Linux). CygWin and WSL are complete environment taking easily from 800 Mb to several Gb of data, just to run the most simple script.

This is the kind of situation where I realize the amount of shortcuts I took to develop God is a Cube. No, making my own database conversion script, and correcting problems as they come, this is not the way to go. I should have taken the time to actually document myself seriously on all this stuff to do it correctly, rather than just trying to "make it work as fast as possible". I was feeling a great pride to be able to do this kind of stuff so fast, but I did not learn any real knowledge this way.

I found a simpler solution, MySQL2SQLiteWin, which just needs an Awk executable (a programming language specialized in dealing with data formatting in text files), a SQLite executable, and a simple bash script. Sadly, running on my database dump throws a lot of errors "hex literal too big", as, I suppose, the hexadecimal are too big for the script. I looked rapidly, some of my hexadecimal data can be more than 6,400 characters. But I also see that short hexadecimal values (like 47 characters) can throw the same error. After some research, I see that the script think my hexadecimal data is a number, but it is in fact a text.

I realize this hexadecimal situation is managed by my old "fast and stupid" script. So I convert the hexadecimal numbers into hexadecimal text (in Notepad++, Find 0x(01[0-9][0-9a-fA-F]+) and Replace with X'$1'). Now the errors "hex literal too big" are gone!

I think that my use of binary blobs for data storing is an edge case, and that using a script for that will fail. A custom solution, fast and as simple as possible will give far better results. So it looks like the old way "fast and stupid" script is once again the way to go. This time, I'm writing the script in C#.
using System.Text.RegularExpressions;

string content = File.ReadAllText("godisacube_db.mysql");

// Remove engine
content = Regex.Replace(content, @"ENGINE=MyISAM DEFAULT CHARSET=utf8", "");

// Remove comments
content = Regex.Replace(content, @"^/*!\d+ .**/;", "", RegexOptions.Multiline);

// Remove other MySQL commands
content = Regex.Replace(content, @"^SET .", "", RegexOptions.Multiline);
content = Regex.Replace(content, @"^START .", "", RegexOptions.Multiline);
content = Regex.Replace(content, @"^USE .", "", RegexOptions.Multiline);
content = Regex.Replace(content, @"^CREATE DATABASE .", "", RegexOptions.Multiline);
content = Regex.Replace(content, @"^COMMIT;", "", RegexOptions.Multiline);

// Remove several-lines MySQL commands
content = Regex.Replace(content, @"(?:\r?\n|^)\sALTER TABLE .?;", "", RegexOptions.Singleline);

// Remove unsigned
content = Regex.Replace(content, @") UNSIGNED", ")");

// Change hexadecimal value into text
content = Regex.Replace(content, @"0x(01[0-9][0-9a-fA-F]+)", "X'$1'");

// Replace single quote bt two single quotes for escaping
content = Regex.Replace(content, @"\'", "''");

File.WriteAllText("mod.sql", content);


To execute this script, I am using csi.exe. I found it easily on my computer thanks to Everything, but I think a Windows search can locate it as well.

"C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\Bin\Roslyn\csi.exe" script.csx

Then I launch the SQLite import to create a SQLite database.
sqlite3 output.sqlite ".read mod.sql"

If the SQLite commands were more complex, I could create a SQLite script.
.read "mod.sql"
.exit

And launch this SQLite script with a Windows command.
sqlite3 output.sqlite < script.txt


Next time

Next time, I will start to work on the data exploration, and create charts to better visualize data from various info of the game.