Big Blue Ceiling

Volumetric Streaming over RTC

Realtime 3D capture of a person, merged from two sensors and streamed over the internet.

Eddie OffermannHardware selection and rig build, software architecture, engineering, demonstration

Role
Hardware selection and rig build, software architecture, engineering, demonstration
Areas
Telepresence, Hardware, Computer Vision
Two Intel RealSense depth cameras on black 3D-printed plates, mounted at either end of an aluminum rail on a small stand.

This exploration focused on realtime volumetric capture and streaming over the internet.

We selected the sensors, then designed, built and calibrated the test rig: two Intel RealSense D435 cameras on 3D-printed mounts and an 80/20 frame, positioned so each fills in what the other can't see. We built two spacings. The narrow rig sits on a monitor and fills occlusions tightly for laptop-sized screens. The wide rig spreads the sensors along a longer rail and fills much more, but only suits television-sized displays.

We wrote the software that merges the two volumes, correcting for alignment and color, and encoded the result in a format that can be streamed between remote devices and rebuilt on the far end over current network infrastructure.

It became the foundation for the rest of Enhanced Presence.

  • The first realtime stream: two sensors merged into one volume.
  • The same stream from another angle.
  • The dual-sensor rail mounted on a light stand in front of a wall, one camera at each end.
    The wide rig on a stand, with the sensors far apart to fill more occlusions.
  • The aluminum rail on its stand above a workbench, with two cameras on printed plates and screws laid out beside them.
    Assembling the rig.
  • Two silver RealSense cameras next to black 3D-printed mounting plates and loose screws.
    The cameras and their printed mounting plates.
  • A RealSense camera fixed to a black printed plate at the end of an aluminum 80/20 rail, on a blue silicone mat.
    Each sensor bolts to the 80/20 rail through a printed plate.
  • A silver depth camera seated on a black printed plate on a blue silicone mat.
    A printed plate, sized to the sensor.
  • A narrow version of the rig, with two RealSense cameras on a light-colored printed bracket, clamped on top of a monitor.
    The narrow rig, sized for laptop and desktop screens.

  • Enhanced Presence

    A run of prototypes chasing the feeling of being in the room with someone far away.

    • Telepresence
    • XR
    • Hardware
    • Computer Vision
    • Research

  • Glasses-Free 3D Telepresence

    Volumetric calls on a 45-inch autostereoscopic display, no glasses required.

    • Telepresence
    • Hardware
    • Computer Vision
    • Research

  • North Star Stereoscopic Telepresence

    A preview of a fully 3D, ultra-HD video call with realtime perspective shift.

    • Telepresence
    • Hardware
    • Computer Vision
    • Research

  • Projection AR Tabletop Gaming

    Augmented reality projected onto an ordinary table, with touch.

    • XR
    • Hardware
    • Computer Vision
    • Telepresence
    • Research