Thanks to everyone who has provided feedback about the first test of hand control for Tilt Five users in icosi-do! (If you wonder: these discussions happened on the Tilt Five discord.)
Here is a summary of my conclusions:
1) Performance matters: the approach of downsampling the camera image on the GPU and continuing the CPU computations with a 32 x 32 pixels image appears to provide a sufficient refresh rate for somewhat smooth interactivity.
2) The connected component analysis works quite well to detect whether thumb and another finger touch each other and thereby complete the OK/ring sign/gesture. The passive haptic feedback of this moment is a nice touch (pun intended). Of course there are limitations due to the single, infrared camera, but within these technical limitations, it appears to work very well.
3) The "head cursor", i.e. controlling a 3D reticle with head movements, doesn't feel very good. This becomes particularly obvious when comparing the feel of rotating the puzzle with hand control and the feel of selecting rods - the former feels a lot better. One part of the problem might be the required control loop between head movements and movements of the 3D reticle, which feels quite unnatural. Furthermore, the required head movements make the next point worse:
4) The limited field of view of the tangible camera is a real problem. The visualization of the camera image helps, but ideally the interaction design should encourage users to keep their hand in the center of the field of view such that they don't lose tracking. The field of view of the tracking camera would improve the situation, but I don't think that it would completely solve it. Also, I'm not sure whether or when images of the tracking camera are going to be available.
So, plan for future (i.e. current) work:
1) I'll focus on rod selection. Since the selection is confirmed by closing the ring formed by thumb and another finger, my current idea is to aim at (and highlight while "hovering over") a rod with an "open clamp" hand pose where the tips of thumb and another finger are somewhat close to each other and the center point between them is in line of sight of the selected rod. (The lack of depth information limits the feasible designs.)
2) To implement this design, I'm trying to find the positions of all fingertips in the image by computing the shortest-path distance of dark pixels from the image boundary and picking the pixels with locally maximal distance. I hope that I can easily identify the position of thumbs based on their small y-coordinate. The second fingertip might then just be the closest fingertip in the image. I suspect that positions on a 32 x 32 grid will not be accurate enough for a pleasant interaction, but that's a problem for another day (and I have an idea how to address it).