Watch this great presentation by Microsoft Azure covering (among other things) RetroArch's AI Translation capabilities
AI Service
Welcome to the future! Sometime ago, a #RetroArch bounty got posted proposing OCR (Optical Character Recognition) and Text To Speech services being added to RetroArch.
Some months later, and here we are – a bounty hunter valiantly took on the challenge and there is now a fully fledged AI Service up and running that works seamlessly with #RetroArch!
You use the AI Service like this – you enable the AI Service (should be enabled by default), you then setup the server URL (could be a local network address if you have the server up and running in your own network, or a public IP/URL in case you’re going through a service). After that, you only need to bind a button or key to the so-called “AI Service” action. You can bind this key by going to Settings – Input – Hotkeys.
In this video, you can see each of the two modes that the AI Service currently is capable of doing –
Speech Mode – Upon pressing the AI Service button, a quick scan is done of the text, and the recognized text is then translated to speech. You can press the AI Service button at any time and it will try to process the current snapshot of the screen it made. This mode is non-interruptable, meaning the game will continue running when you hit this button, and the output speech will take as long as it takes for the server to respond to your query and pipe the sound to RetroArch.
Image Mode – In image mode, it tries to replace the text onscreen with the output text. For instance, in the video you see above, the game is played in Japanese, so when we hit the AI Service button, it tries to replace the Japanese text with English translated text. This mode is interruptable – this means that when you hit the AI Service button, it pauses the game and shows you an image with the replacement text UNTIL you hit either the AI Service hotkey or the Pause hotkey again, then it will continue playing.
We encourage everybody that wants to submit feedback to us on this amazing revolutionary feature to go to our Discord channel and in specific the #retroarch-ai channel. We’d love to hear your feedback and we’d like to develop this feature further, so your input and feedback is not only appreciated but necessary!