Project Lighthouse
March 25, 2014, GPU Technology Conference, San Jose, CA—Mike Romey from ZOIC Studios described their implementation of a virtual production pipeline. The ZEUS (Zoic environmental universal system) has many functions such as tracking monitors for the rigged cameras and other equipment, real-time render for actors on green screens, previsualization, markup, etc. for work on virtual sets.
Lately, the number of visual effect shots has started to increase. The traditional sequence started with shots which had numerous versions and added notes. Now television shows are getting many more sequences that require special effects shots. For example, one episode of “Arrow” had 29 sequences with the hundred and 99 shots and 1786 versions. “Once upon a time” had 20 sequences and between 300 and 850 shots resulting in 2 to 6000 versions, all processed within a 2 to 3 week deadline.
The advent of virtual production allows for more shots, versions, notes in less time at the cost of much more storage. At the same time, the render requirements have gone up exponentially which causes severe constraints in production. Rendering is not scalable, and has a variable footprint over time. As a result it’s extremely difficult to define vendor time versus the schedule. As a result, the holy grail is to move towards real or near real-time final frame render.
Lighthouse is a joint development with Nvidia, the Chaos group’s V- Ray, and Zoic. The project is an attempt to balance costs, performance, and functionality. The first hardware configuration cost about $7000, had 5GB of memory, two K20 Tesla boards, and one Quadro board. This version contains 7488 graphics cores and was fine until assets grew beyond 6GB.
The second version uses K40 Tesla boards with 12GB of RAM and 8456 cores. It’s capable of processing full HDTV signals and is also designed to be fairly modular so the company can upgrade to the latest GPUs. This version is now in full production use and will probably remain in place for those jobs that don’t require the latest hardware.
The software handles images of hard surfaces, hair, fluids, and other physical characteristics. The current render pattern originates with camera data, gets highlights and mood from the lighting department which is then merged with data from the layout cameras. All these data are rendered on the first pass, through seven render buffers and include functions like compositing, ambient occlusion, and utility layers for XYZ displacements, noise, etc.
Some of the challenges with V-Ray 3.0 include having multilayer extraction going to a single output, multilayer shadows were set as existing assets in a CPU, not a GPU. Porting functions from the CPU to the GPU was difficult because of the hard baked user interface that depended upon CPU functions. The another challenge was that the software is designed for a single computer for all the buffers. Including noise reduction and ambient occlusion in the flow increased the system complexity.
Some of the workflow changes included a separate beauty pass and ambient occlusion to reduce processing time. They pre-calculate lighting, taken from camera data to calculate the light, and output a single cache file for all the lighting. As a result of these changes, the virtual production and now track live lighting changes in near real-time.
In the future, they hope to offload light bake to the GPU and commit a computer per asset. Their developing NiCad app that can display the baked textures in GPU procedural shaders. These textures include legacy, noise, etc. since they don’t have the money to regenerate assets all the time, so they need to reuse existing assets. They hope to add mesh displacement and mass light of support. These features are important due to the hundreds of lights that are used in some scenes.
AMD and occlusion render pass is very clean, so there is no noise in the image. Predating the lighting into the cache file allows all rendered elements to be processed in one compute cycle. This process integrates all components at once, so lighting, specular, Z. etc. are processed in one pass.
The GPU accelerators reduce processing time from 16 hours a CPU to 1 1/2 hours. Although the CPU only cost $5000 versus $18,000 for the GPU accelerators, they’re still getting a good return on investment. The CPUs take 27,000 hours to process a full season compared to 4.4 thousand hours with GPU. They hope to take advantage of improving hardware performance over time, because the studios are continuing to change and add more effects.
They are spending some time on general functions and adding special functions to their systems. Some of these new capabilities are not in full production because are still evaluating and vetting functions. Eventually they will do more work in the cloud that the current configuration prevents migration of functions to the cloud. There are questions about costs for work in the cloud versus renting a supplementary infrastructure for short durations of most productions.
