Slicing fruit with a glance

The rules fit in one breath. Fruit is tossed up from the bottom of the screen, hangs for a moment at the top of its arc and falls back. You cut it before it drops. Every whole fruit that gets away costs you, and every so often something that is not fruit comes up with the rest, so you cannot just slash at everything.

After weeks of teaching my laptop to read my hands for Precog, this was the obvious next question. If the camera can follow a fingertip, it can be a blade.

The title screen. Everything after it runs in the browser, camera included.

Precog was written in Python, and the Python version of the hand tracker kept crashing on my Mac's graphics drivers. So PULP lives in the browser. The hand model runs as WebAssembly next to the game, a canvas draws at sixty frames a second, and the sound is synthesised on the fly. There is no server: open the page, allow the camera, play.

Your index fingertip leaves a trail. Only fast movements cut, so you can move your hand into position without slicing anything by accident, then flick.

Hands were the warm-up. The real question was the eyes.

Then a harder question: what if you only had your eyes?

Gaze is much noisier than a fingertip. So each session starts with a calibration: thirteen dots appear one at a time and you look at each. Seven more dots check the result, so the game can tell you honestly how well it is following you before you start.

Behind it is a small ridge regression from the eye features of a face mesh to screen positions. Leave-one-dot-out cross-validation picks between a straight-line and a curved fit, and how strongly to smooth.

In eye mode a cut is either a quick flick of the eyes across a fruit or a short dwell on it. Blinks are filtered out, otherwise every blink would be a slash.

Hand mode is fun. Eye mode is still a beta I keep tuning: the speed a swipe needs, how long a dwell lasts and how sensitive the blink filter is all want a real person in front of a real webcam, and that is the part I am working on now. What I already learned is that a calibration you can check matters more than a cleverer model.

Slicing fruit with a glance

Comments

In the same orbit