Dual Camera Infrared Chroma Key

Anyone who has shot with a green screen can tell you that chroma key is not a perfect system. You’re shooting a subject in front of a green wall and then telling a computer to discard any pixels that are greener than a certain amount. Problems arise when green light bounces off the back wall and onto your subject – this is called “green spill”. Additional problems arise if your subject happens to be wearing green or has green eyes. Semi translucent areas are trouble as well, such as frizzy hair, the thin parts of the ear, etc.

While working in Paul Debevec’s graphics lab at the University of Southern California’s Institute for Creative Technologies, it occurred to me that there ought to be a keying system that uses an invisible wavelength of light. I described my idea to Paul and he encouraged me to to build an infrared “green screen” and capture system.

Step one was building a video camera that records red, green, blue and IR light channels. I prototyped this by mounting two consumer grade video cameras on a plank of wood, each positioned to look through an 80/20 beam splitter. One camera was set to capture a regular image of the subject, the other camera had a filter over the lens that blocked all visible light and only permitted IR light to pass through.

Step two was illuminating a reflective screen behind the subject with infrared light. Turns out you can buy IR light emitters from most security camera shops. They’re popular for lighting garages with so security cameras can shoot in the dark.

Step three was shooting a subject in front of the screen with the two cameras.

Step four is where the magic happened. Load the two shots into After Effects, align them and set the IR channel as a traveling matte for the RGB. Add a background and voila. A perfect matte!

The infrared keying system is described the SIGGRAPH 2002 Light Stage paper:

Paul E. Debevec, Andreas Wenger, Chris Tchou, Andrew Gardner, Jamie Waese, Tim Hawkins (2002). A lighting reproduction approach to live-action compositing. Proceeding SIGGRAPH '02 Proceedings of the 29th annual conference on Computer graphics and interactive techniques. Pages 547-556


Category Research

Date July 2002

This clip is raw footage of myself in the light stage, lit with 156 programmable LED lights set to reproduce a spinning light probe sample taken in a cathedral in Italy.

This clip shows raw footage from a second camera mounted to the first with a 90/10 beam splitter. This camera had a visible light filter on it so only infrared light would pass through.

This clip shows a composite of the raw footage with the spinning light probe doubling as a background. The infrared footage was used as a traveling matte.