Seeing the Invisible

The signal

Half a grey level

Blood flowing into the skin changes its colour, mostly in the green channel, by tiny amounts. In MIT's reference face clip, the average green of the skin moves by about half a grey level out of 255. On a normal scale it is a flat line; zoom in two hundred times and the heartbeat is there, buried in noise.

amplify ×60

Real data: mean green of the forehead and cheeks in MIT's face clip, 10 s at 30 fps. Rows: as recorded on the 0–255 scale; the same zoomed in; band-passed to the pulse frequencies; added back, multiplied.

The trick

Watch each pixel through time

Wu and colleagues at MIT had a simple idea in 2012. Don't track objects. Treat every pixel as a time series, keep only the frequencies you care about, make them bigger and add them back. Because it looks at fixed places rather than moving things, they called it Eulerian, after the way fluid dynamics describes flow.

Blur in spaceA Gaussian or Laplacian pyramid. Heavy blur averages away noise; a pulse covers whole patches of skin anyway.

Filter in timeA band-pass for each pixel: 0.8 to 2 Hz for a pulse, 0.4 to 3 Hz for breathing.

AmplifyMultiply what is left by α: about 50 to 80 for colour, 10 to 20 for motion.

Add backPut the amplified change on top of the original frame.

CollapseRebuild the image from the pyramid. The invisible is now visible.

Everything runs locally on the CPU with NumPy, SciPy and OpenCV. At 640 pixels wide it takes about 10 ms a frame for colour and 13 ms for motion, so a live 30 fps camera keeps up.

Reading the heartbeat

From colour to beats per minute

To read the rate, the forehead and cheeks are averaged every frame and turned into a pulse wave with the POS method (Wang et al., 2017), which is robust to lighting and small movements. The strongest frequency between 42 and 210 bpm is the heart rate. On the MIT clip it reads 52.7 bpm; the paper reports 50 to 60.

Real data from the same clip. Top: the POS pulse wave. Bottom: its spectrum between 42 and 210 bpm; the dot marks the peak.

Why the textbook version fails live

Coherent colour

Plain EVM amplifies everything in the pulse band. On a live webcam, a small head movement at about 1 Hz is in that band and is far stronger than the pulse, so it explodes into blotches of colour. My first live test looked exactly like that.

The fix uses the pulse itself as a reference. The face's pulse wave is computed live, and every pixel keeps a running regression against it over about three seconds. Only the part of each pixel that follows the heartbeat is amplified. Movement and noise don't follow it, so they drop out, and so do background and hair. A sudden jump pauses the learning for about a second.

20 → 5grey levels of change caused by motion on a shaky test clip, live (42 → 5 offline)

~70%of the pulse signal kept at the same gain

±0.12soft limit on the added colour, so nothing saturates

Motion

Move the edges, not the brightness

For motion, the original method amplifies intensity changes. It works for tiny movements, but as α grows the edges grow halos and the noise grows with them. The phase-based method (Wadhwa et al., 2014) measures where each edge is, using a Riesz pyramid, and moves the edge itself. Try it: the bar below really moves 0.25 px.

α ×20

Simulation of one row of pixels. Top: as filmed. Middle: linear magnification, brightness change × α, clipped to the screen range. Bottom: phase magnification, the edge moved × α.

On MIT's sleeping-baby clip, the zipper of the onesie sways 0.4 px. I measured how far it moves after magnification and how much a still patch of blanket flickers:

Sway of the zipper line (px)

Noise in a still area (grey levels)

originallinearphase

Three space-time slices of the baby clip: almost flat in the original, small ripples with linear magnification, clear breathing peaks with phase magnification

Space-time slices: one column of pixels across the chest, stacked over time from left to right. Each bump is a breath.

In practice

A live instrument

The same code runs on clips, on the webcam, or on any stream (an RTSP camera, say), and can serve the result to a browser. With Tailscale, I can watch it on my phone. Keys switch between colour and motion, the method, the gain and the view: side by side, magnified, original, or a heat map where every pixel shows how much it pulses in step with the face. Skin lights up; moving edges, strong but out of step, stay dark.

What I learned

Running averages need bias correction. Without it, the first seconds come out about ten times too strong.
Go deep in the pyramid. With levels down to 16 px, most of the motion stayed in the unmagnified residual and the gain collapsed. Going to 4 px fixed it.
Measure the right thing. To compare motion methods, track the position of an edge. Counting changed pixels rewards the halos of the worse method.

References: Wu, Rubinstein, Shih, Guttag, Durand & Freeman, Eulerian Video Magnification for Revealing Subtle Changes in the World, SIGGRAPH 2012 · Wadhwa, Rubinstein, Durand & Freeman, Riesz Pyramids for Fast Phase-Based Video Magnification, ICCP 2014 · Wang, den Brinker, Stuijk & de Haan, Algorithmic Principles of Remote PPG (POS), IEEE TBME 2017. Test clips: MIT's reference videos.

Seeing the invisible