● A hand-gesture interface for macOS · October 2026

Precog

Wave at your Mac. Swipe between desktops in mid-air, pinch to click, grab a window and throw it to the other screen.

or just scroll

Python · OpenCV · MediaPipe · PyObjC. Runs locally; no video leaves the machine. The hand on the right is one of my real recordings.

recording · swipe left · take 1 of 6

Prologue

The interface from the film, on a laptop

In Minority Report, detectives sort evidence by sweeping their hands through the air. The film needed gloves and a wall of glass. Today a laptop camera and an open hand-tracking model are enough to try the same idea on an ordinary desk.

Precog turns each camera frame into 21 points per hand, compares them with gestures I recorded, and acts on macOS directly.

21landmarks per hand

~30frames per second

2hands at once

0extra hardware

01Swipe desktops with three fingers, the way a trackpad does it.
02Point and pinch to click, right-click and drag windows.
03Switch apps from a strip of icons that follows your hand.
04Snap windows to halves and quarters by pulling your hands apart.
05Pause everything by showing ten spread fingers.
06Teach your own poses and movements, bound to any action.

Act I

Seeing

From camera pixels to a hand you can measure.

Hand tracking

Twenty-one points per hand

Every frame goes through Google's MediaPipe hand landmarker. It returns the wrist, four joints on the thumb and four on each finger, plus a guess at left or right.

Precog never looks at pixels after this step. Poses, swipes, pinches and the standby check are all geometry on these points.

Hover a point for its name. Amber: the hand-size ruler, wrist to middle knuckle. Circle: the palm centre.

recording · pointer posepoint 8 · index tip

Making hands comparable

Near or far, the same hand

A hand close to the camera looks twice as big as one at arm's length. So every hand is redrawn relative to its own wrist and divided by its own size: the distance from the wrist to the middle-finger knuckle.

Movement is measured the same way. The palm travels in hand-widths, not pixels, so a swipe is the same swipe whether you sit close or lean back.

1 · as the camera sees them

Act II

Recognising

Comparing what the camera sees with what you taught it.

Three kinds of gesture

Poses, movements, and both hands

Three kinds of gesture, all recorded by example, about six takes each. Below are my own recordings, replayed exactly as the camera saw them.

pose · app switch

PoseA hand shape held still. Fires after 0.45 s with the palm nearly still, and not again until you let go.

movement · swipe right

MovementA shape plus a path. Precog keeps the last seconds of each hand and keeps asking if the recent stretch looks like a recording.

two hands · expand

Two handsAdds the gap between the hands and the path of their midpoint. "Expand" is mostly a growing gap.

Dynamic time warping

A slow swipe and a fast swipe are the same swipe

People never repeat a movement at the same speed. Precog resamples every movement to 16 steps, then lines them up with dynamic time warping: each step is paired with the step it resembles most, as long as the order is kept.

The cost of a pair is the difference in hand shape plus 0.35 × the difference in path. Averaged along the cheapest route through the grid, that gives one distance.

two of my swipe-left takes

Teaching it

Six takes set the threshold

For each take, Precog measures how far it is from its nearest other take. A take far from the rest (a misread hand, a much slower attempt) is set aside. The threshold comes from the spread of what's left, with a floor and a ceiling.

distance to nearest other take

Act III

Acting

Desktops, windows, apps and clicks, driven by your hands.

Five modes, five colours

Precog always shows which mode it is in

The colour of the hand skeleton in the corner of the screen says what your hands will do right now. Most of the time you are in gesture mode; the others open for one task and close again.

GestureRecorded gestures and the built-in three-finger swipe are live.default mode

ManipulateYour knuckles steer the cursor. Pinch to click, pinch and move to drag.enter: "pointer" pose

App switcherA strip of recent apps follows your hand. Pinch to open one.enter: "telephone" pose

LayoutA 3 × 3 grid of window layouts with a live preview.enter: expand, two hands

StandbyIgnores everything except the gesture that wakes it.enter and leave: ten fingers

Under the trackpad

Sliding desktops the way a trackpad does

The obvious approach, pressing Ctrl+→ in software, does nothing: macOS ignores synthetic key presses for switching desktops. A real three-finger swipe sends a private "dock swipe" event with a progress value instead.

Precog sends that same sequence: Began, a stream of Changed as your hand moves, then Ended to commit or Cancelled to spring back. Past a third of the way, or with a flick, it commits.

Drag the desktop on the right to try the rule.

progress follows the hand · 4 hand-widths = one desktopprogress 0.00

Clicking in mid-air

A pinch that behaves like a mouse button

The cursor follows your knuckles, which barely move when you pinch. As thumb and index close in, the cursor slows down; once they touch, it freezes, so the click lands where you aimed.

Press and release use different distances (0.32 and 0.48 hand sizes), so a trembling pinch can't click twice. It only becomes a drag after 45 px of movement.

Illustrative signal; the thresholds are Precog's real settings.

thumb-to-index distance ÷ hand size

Off switch

Ten spread fingers put it to sleep

A gesture interface needs a way to say "not now". Both hands, all fingers out and spread wider than a normal open hand, held still for 0.8 s: Precog stops listening until you do it again.

Wider than normal is measured, not guessed. ./precog-run calibrate records your relaxed open hand; my own sits at a finger gap of 0.45 at the 99th percentile, so the switch needs 0.46.

standby check

Live

The recogniser, running in this page

This is Precog's matcher ported to JavaScript, loaded with my real recordings. Each bar is the best distance to one gesture; the black tick is its threshold. Cross it and the gesture fires.

Without a camera, it is watching a stream of my recordings. Turn the camera on and it watches you: swipe left or right to jump between sections, expand both hands for the contents, ten fingers for standby.

Nothing is recorded or sent.

demo stream · recorded takes

Epilogue

What it took

Built over a long weekend. Most of the work was not the model; it was making a Mac accept input it was never designed for.

Code on GitHub ↗ PULP: the same hands, as a game ↗

Synthetic keys are ignoredDesktop switching only listens to real trackpad events, so Precog posts dock-swipe events with the hardware payload, ported from the open-source tool iss.

Pinned to an older model runtimeMediaPipe 1.x crashes on macOS 27 (Metal service unavailable), so Precog stays on 0.10.21.

Pick the camera that is actually liveAn iPhone Continuity Camera can take the default slot while the built-in one stays black. Precog tries each and keeps the first with a real image.

Tested by replaying handsRecorded gestures replay through the real loop with a fake tracker, so changes to matching are checked without waving at the screen.