Data Management In Animation
April 4, 2011, Storage Network World, Santa Clara, CA—Mike Cutler from Dreamworks Animation described the high-performance data management system they used for animation production. Although most people don’t consider cartoons as high technology, their system requirements constantly push the boundaries for state-of-the-art data processing centers.
DreamWorks started in the 90s and the artists were also programmers at that time. Most of the tools they used and still use were developed in-house. Now DreamWorks has grown to be over 2500 people with 2000 and located in Glendale California and the balance in Redwood City. All these people work in their production processes as a single virtual company.
A film takes about three years from start to finish, with first 18 months in story development, storyboarding, and creating models and sets, then 400 to 600 people take the next 12 months to finish the movie. They are working on 10 to 25 films at any one time.
An animated film has high compute requirements, as evidenced by the current compute and storage farms. Currently they have 34,000 Linux cores running 64-bit Red Hat enterprise level V tied together in a series of one and 10 Gb backbones into their network. A typical film creates over 1 billion files, ranging from spatial data which are small files, to plot, camera, and lighting tools. Creating textures with highlights and shadows takes lots of mathematics.
The render farm, where these calculations take place, take on jobs from 10 to 100,000 files per job, and can take from ten minutes to ten hours for processing. A typical frame will consume 50 MB of data and the render farm handles 300 to 700,000 jobs per day. Very little of the data is streaming, which results in very low network I/O. The virtual studio is physically located in Redwood City, Glendale and Las Vegas, so 50 percent of the rendering is processed off site, partially due to disaster recovery and power concerns.
This geographic dispersal causes lots of data churn as files are not always local. To mitigate latency issues, the three sites are connected through 10 Gb pipes. The render farm works on shared objects which need to copy links to files for work. Many of the files are erased and regenerated before saving and the data resides somewhere within the system. Currently there are 10 billion files in the system.
In terms of workflows, the artists log into a film, but not a specific site. Every movie goes through many iterations and a daily schedule for production would include rough layouts going to camera, talent, lighting, effects, and texture processing. The director and producer check the dailies, critique the content and then the individual departments work on fixing those images.
As a part of the workflow, the system has to determine which files the render farm needs and tries to assign appropriate files on the one job per machine basis. Jobs are submitted on a random, constant basis. One challenge is maintaining dependency graphs which helped map geometry into textures. A job queue may have 500 frames with lots of common features in adjacent frames, so some of the small texture files may be needed for multiple jobs. As a result, they’ve developed specialized caches to ensure higher locality when processing the jobs.
A typical job might use 90 to 120 twelve core hosts feeding one file server. They can virtualize the file servers and the attendant caches, but latency mitigation becomes a challenge due to the 8 to 9 ms latency across the 10 Gb links. They’ve developed a way to cache across wide areas by making the data local. They also look at the types of data that can be replicated as commonly used data types across the network. These practices help with the latency, much of which greatly affects the artists productivity and morale. A new satellite campus in Bangalore is exacerbating this latency problem. They manage the cache with cache farms and replicate data as needed to keep as much software local as reasonably possible.
Moving to an offshore campus exposes globalization issues which are also cultural issues. When they moved to other locations they replicate existing facilities, seed the site with their own people, teach the locals the requirements and flows for production, and develop ways to maintain their standards and brand identity. In this sense, Bangalore was an experiment in moving to other sites, and may not be a scalable process. One of the next films they release will be a joint Bangalore and US development with separate scenes developed at each site.
One challenge to have to face is the software learning curve. Most productions require multiple movies at one time. That’s one reason why they have so many films in production simultaneously, because it lets them keep their highly skilled people constantly working in their specialty areas. Otherwise these people would have gaps in their schedules and the resulting downtime cannot be productively filled. Eventually they want the capability of making any function in any site.
They use NFS as their main file system but this raises many challenges. They must have global data representations but also must distinguish local, replicated, and cache copies of those data. Metadata is one key to help collaboration between the studios. Data cohesion is labor-intensive and their transport model doesn’t scale. They cannot just copy their data to all the sites because they currently have over 4 petabytes on line.
Today’s solutions to the resource management problems are to increase the number of render cores and their attendant caches. However, caching solutions may not scale cleanly across multiple sites. NFS provides a global name space and LDAP allows automounting. Data migration drives collaboration, because the artist work on data sets across locations with shared libraries. The rewrite files in the global namespace use resource affinity, dependencies and graphs steer the process jobs with data outputs.
Much of that is transient, from 80 to 120 TB are needed for a movie plus from 40 to 60 TB of expandable space for reuse every couple of hours. There is a middle layer for the length of dependency graph or through the life of the project and also there are permanent data. Data cohesion invokes temporal or geographic file ownership with multiple masters. This causes the data to become stateful and makes it difficult to make metadata conform to their allied data blocks.
An example of this issue would be that location any is the owner of some data and location be uses that data for a rendering job. Now, see wants to change the data but must inform any of the changes so all of the users can reuse those files. The good thing is that metadata is much smaller than the data blocks, so storing and tracking metadata results in a 90 percent reduction in storage and transport compared to the full data blocks.
In the future, they hope to develop object based metadata and develop some type of fuzzy acknowledgment mechanism to allow multiple mastering, sharing, and reuse. They are looking into a mechanism for cloud data representations using some type of web object transport model. In addition, they’re looking for some types of social engineering to complement technical challenges such as showing the progression bar for date of availability to the artists. There are working towards a flexible compute resource model, some type of cloud environment and some type of object based project grouping of data sets.
