Computational Optics

What if every phone camera were a spectrometer?

A phone camera is built to make pictures that look right to people. Its sensor sits behind a mosaic of red, green and blue filters, and everything downstream of the sensor is tuned to make three color channels that please the eye. But the silicon underneath sees more than that. The filters overlap, they leak, and the sensor responds to light outside the band its color channels nominally cover. That extra information is usually thrown away.

Our computational optics work is about getting it back. We've developed a method for estimating hyperspectral image data, where each pixel carries a full spectrum rather than three numbers, from the commodity image sensors already inside phones, tablets, headsets and laptops. We've filed a provisional patent application on it.

Why it matters

Hyperspectral cameras can tell materials apart that look identical to the eye: healthy and stressed leaves, one pigment and another, a real fabric and a dye match. They measure what something is made of, not just what color it appears. They're also expensive, bulky, power-hungry and slow, because they rely on dispersive optics, tunable filters or scanning to sample many narrow bands.

There are billions of phone cameras. If even a useful approximation of spectral data can be pulled out of them, without changing the hardware, spectral imaging stops being a lab instrument and becomes something that can be deployed at scale: in a field, in a clinic, in a museum, on a film set.

The idea in brief

The method has four stages:

1. Characterize the sensor. In the lab, we light a calibrated target with one narrow band of light at a time and record how each color channel of the phone responds, including its leakage outside the nominal passbands.
2. Capture paired data. The phone and a reference spectral camera photograph the same scenes at the same moment, so every phone pixel can be paired with a measured spectrum.
3. Learn the mapping. A model, analytical, machine-learned or a hybrid of the two, learns to turn RAW sensor values into spectra.
4. Deploy it. The mapping runs on the device, in the cloud or split between them, and turns a RAW frame into a spectral cube, a set of bands or application-specific measurements.

How It Works walks through each stage for a technical reader who isn't an optics specialist. Characterizing Sensors covers the lab procedure in detail, and Learning the Spectrum covers the data and the model.

The lab behind it

This is measurement work, and it needs a place where measurements repeat. We built the ISO-5 Optical Cleanroom for it: positive-pressure, particle-controlled, with an optical bench, a tungsten-halogen source, monochromators covering the ultraviolet through the near-infrared, long-pass filters, fiber optics and diffuse reflectance standards. The Lab describes the room and every instrument in it.

Where it goes

The same pipeline that serves a botanist or a conservator also serves a cinematographer. Spectral data can explain why two costumes match on the monitor and clash under LED light, give a renderer real material spectra, and make color in mixed reality behave like color in the room. Applications covers imaging, science and media production, and Research Directions covers the open questions we're pursuing.

The work connects to our other imaging and spatial-computing projects: the CameraSmartPlate for tracked capture, HoloFlect and the xR Ecosystem for mixed reality, and the iOS LiDAR Scanner for geometry from the same class of phone.