Fraunhofer – Light Field Moves to Video
April 2015 – At NAB 2015, we met with Fredrick Zilly of Fraunhofer of Germany on the directions and advanced technology in the imaging area. As a followup to the demonstrations of NAB 2014 that were centered on light field imaging for still photos, this year the technology shown was advanced to video image processing. The systems once again used computational imaging and either a multi-camera array or a single camera with a multiple mirror rig.
This research is driving the next generation of movie production which allows for image capture and then depth of focus, object ID and lighting determination from the raw data content in post-production. In 2014, they were showing stop motion action with single frame processing. Now the system supports live action and filming in a software stitched 10,000×2000 resolution image. The rig that was shown in the booth was a Sony F3 in combination with a 360 degree multiple mirror rig with a 10 camera array.
Fraunhofer lightfield camera mirror rig and Depthmap Sony F3 camera
The camera array captures the image and the Sony F3 is used to capture and build the depth map. The system that is in place is resolution independent; it is just more processing for higher resolution. The data being captured is currently 32 bit for the camera color information and 16 bits on the depth map. This information is used with the object ID and the lighting targets to product the CGI result. At this time, they are using NUKE with Vray for a relighting tool on the images.
One of the challenges with video is the need to move the camera. The system was designed with a multi-camera rig – mirror & camera array plus the single camera that can be mounted and moved on a dolly for coverage. These camera systems were tested at the 2014 world cup for live streaming content delivery.
Continued research is on extending the technology to HDR, HFR, Multi Exposure data and images. A core function is in the development of a standard for data representation for light field video and depth maps. These are being done in cooperation with the OpenVR consortium.
