On June 5, 2023, Apple has presented the Apple Vision Pro and with it a new "spatial computing" interface based on (among other things) eye tracking and hand gestures. While the keynote presentation had its controversial moments, the reports by journalists who were able to experience early devices were quite enthusiastic.
So, this made me think: how could I integrate some of the features of this spatial computing interface into the Tilt Five version of icosi-do? To answer this question, let's step back a bit: At its core, eye tracking provides a direction in 3D space, which is similar to the tracked direction of Tilt Five glasses. The hand gestures potentially provide much more information, but the most basic information is just a single Bit: users are or aren't pinching with their hands, which is similar to pressing or not pressing a single button.
How to build a user interface based on a tracked direction and one button? It turns out that this is not only similar to using Google Cardboard but also to using icosi-do with the Tilt Five glasses but without the Tilt Five wand. And even better: I've already dealt with this situation and implemented a kind of reticle that can be controlled by moving the Tilt Five glasses! This covers the tracked direction.
What about the button? Instead of the trigger and buttons of the Tilt Five wand, the buttons of a gamepad controller, mouse, or keyboard may be used. To create an interface without such controllers, icosi-do could use an infrared camera of the Tilt Five glasses and detect a specific hand gesture. If the hand gesture is detected, the game could behave as if a button was pressed. Easy, right?
But which hand gesture is suited best? "Pinching" is not ideal because some users might "pinch" with their thumb and index finger forming a small angle (almost parallel fingers) as if they try to hold a large "pinch" of salt between thumb and index finger. This would be difficult to detect because of fingers overlapping in the camera image. A better hand gesture is the "OK sign", i.e., forming an "O" or "ring" with the thumb and index finger, and spreading the other fingers to form a "K". An alternative is the "ring gesture" (or "ring sign"), which doesn't require spreading the other fingers.
How to detect an OK/ring sign/gesture? I did a few experiments with the available infrared camera of the Tilt Five glasses and concluded that the gesture has to be performed in front of the retroreflective part of the gameboard, otherwise there probably isn't enough contrast between fingers and background to reliably detect hand gestures. Also, this isn't a big limitation because the gesture has to be visible to the camera anyway. The easiest way to detect an OK/ring sign/gesture in front of the retroreflective gameboard is probably to classify the pixels into "foreground" (high intensity of retroreflected infrared light) and "background" (low intensity of infrared light diffusely reflected by skin), run a connected-component analysis and search for a connected component ("blob") that represents the inside of the closed ring formed by thumb and index finger. My assumption is that this blob should be larger than the blobs corresponding to the dots on the boundary of the gameboard and smaller than the blobs corresponding to the rest of the visible retroreflective parts of the gameboard.
The interesting questions is then: how to run an efficient connected-component analysis on a relatively large image at more than 10 frames per second? My first attempt was to create a GPU-based pyramid algorithm for this. It worked in some cases, but wasn't very reliable. Also: I concluded that I had to analyse a low-resolution version of the camera image on the CPU in any case, which inspired another idea: a GPU-based downsampling algorithm that preserves the connected components of the image, followed by a connected-component analysis on the CPU.
So, that's what I'm working on right now. To be clear: this won't include any eye tracking nor will it allow to detect more than a single hand gesture. But at least this might be a step towards a controller-free interface when using Tilt Five glasses, which might be a lot of fun.