Users talk about digital storage

 August 3, 2010, Culver City, CA Creative Storage Conference hosted a panel on the use of digital storage in entertainment and media. Marty Shindler, CEO of Ocilus3D moderated the panel, which moved quickly into questions and answers. Panel members included Paul Chapman, senior vice president of technology at Fotokem, Katie Fellion, founder, vp business development, and post supervisor at Light Iron Digital, Jason Goodman, CEO of 21st Century 3D, Sean Vilbert, director of digital archives at Paramount, and Tony Lopez, director of information technologies from Legend Films.

The advent of 3D into the major studios has raised a number of issues and necessitates a lot of planning meetings prior to the shoot. Chapman stated the archive is not usually included in planning meetings and no budget is allocated for storage. Fellion agreed that people get involved in the issues of where to go, how to get there, but don’t address the challenges of storage. Not only is the hardware important, but the various formats and protocols hinder users’ understanding of the paths through post. Goodman added that on-set systems have specific media formats. To make matters worse, people have the idea that digital is free and unlimited. Because media is unique to each camera, the normal process for file transfer from cameras to first-level storage is to unload the card, transfer the contents, then shoot with the same card. Users need to get a director of information technology (DIT) on the set. This person is critical because someone has to manage the data, backup, and visual check processes.

The challenges of temporary functions on the data is a possible issue because the user cannot reformat if the data are corrupted. Vilbert suggested that filmmakers start using new procedures and follow them through post, schedule, and sequences. They should validate all transferred data frame by frame to ensure that the data are valid, before releasing the media back into production. The DIT must also perform these functions to validate duplicated data.

The next question related to film and costs. For digital, the only expense is storage. Goodman noted that storage is not free, and is not just used for storing dailies, etc. Files have mixed formats and the storage media for cameras are specific to that machine. Fellion added that the existence of mixed formats in the first-level storage and the delivery formats of the various tools involve lots of standards leading to incompatible files formats.

A question on the 2-D to 3-D transition drew this from Chapman. The acquisition of images changes the functions within the production crews. The move towards 3-D made director of photography rely on the DIT for a lot of functions. To not lose critical content, it’s important to use people to keep track of the files. The new production changes require custom workflows to extract left and right images and there are no existing solutions for this. Filmmakers using file-based cameras need to develop new and robust workflows. The number of new shows are all inventing workflows.

Goodman added that the field is rapidly changing, and the need for transcoding as an intermediate step is one of the issues. The implication is that this will require additional storage and CPU time. A fully baked plan is necessary, and a 3-D cinema storage package will require more than two times the storage of a 2-D production. Post houses are inventing workflows and everything is new. Everyone needs to find something else that doesn’t impact storage and data. The biggest problem is that cash limited functions impact everything else in the production flow.

Fellion suggest that a typical shoot might use four cameras in four different locations. This type of production amounts to 12 rigs, during 95 days on location. Directors and producers want to edit all of this in one place. The camera footage plus the mastering data prevent any shortcuts whatever because this is 600 terabytes of data.

Lopez continued with the comments that on set functions are not new but everything is not possible now that should be. The stereographer’s vision must be represented. The half petabyte of storage is an organizational nightmare. The solution is to use digital asset management tools that work within the digital pipeline, that understand workflows and check-in and checkout processes. The artists have no idea of data flow and management. Vilbert noted that the focus need to be on the documentation to stay on the delivery schedule.

Storage on set was the topic for the next question. Fellion stated that 32 TB per camera just for raw data makes HDD arrays difficult to manage.

Weather impacts on production seems to be an issue. Fellion observed that usually editing equipment is in a hotel room so there’s plenty of time to transfer content from the cameras. The challenge is to make sure that rigs small enough to get through the doors. All onsite storage needs to have soft shutdown capabilities to limit damage on ower failures or interruptions. Goodman agreed that weather is always an issue. Other issues that on-site productions have to address are travel, physically moving storage media to the editing site, and other alternatives for data management.

A question on budget issues brought forth a number of issues. Vilbert suggested budgets set limits to data protection. Just like the days of film, people associated with media transfer must document the carrier, the person delivering the media to the carrier, the person receiving the media, etc. Even with minimal budgets, users need to identify policies for storage and media transfers.

Lopez postulated there is no correlation of budget with quality. Low budgets imply unsupported functions, but integrated asset and storage management packages help. Chapman added that experience is much more important than budget. The fact is no budget operations only start from scratch rather than institute good operating practices.

Question “is half petabyte enough?” brought a response from Chapman. Lots of vendors are producing storage arrays in the half of a petabyte range.

Question on standards brought agreement from everyone, that good standards are needed. Chapman observed that the industry needs working practices for archives. Organizations can develop forward looking standards with minimal specifics. If the standards are too open and general they are useless. Goodman noted technology is rapidly changing and represents a moving target for standards.

Lopez chimed in that storage and studios are a driver for applications. Studio systems administrators exist at all the major studios but have to from scratch every time. It’s best to start with some form of best practices. Everything is application dependent and no standards exist. There’s too much variability across software creators.

Vilbert suggested that the standards for archival storage like LTO identify some important parameters like block size. Other standards like TAR per file and per tape help to manage the archive processes. Migration is a challenge if the old applications no longer exist. Protection across analog and digital elements is challenging. Tracking variances and defining versions is a big deal. The set proliferation breeds more data management requirements.

Question on audio raised a brouhaha. Goldman suggested that there is really has no effect on audio. Chapman disagree and noted that 3-D or multi-dimensional sound can enhance 3-D visuals. Vilbert stated audio is simple because the files are much smaller.

Question on 3-D conversion drew limited response. Lopez said that as in every new venture, there are some bad operators. There are few competitors in the field, because transforming the files for conversion requires a lot of expertise. The biggest issue is that data structures prevent any standard workflow. It’s important to identify all the deliverables, formats, etc. before starting work to ensure completion. Colorization is a simpler process. They had 40 people in the US plus 80 more in India to do the work. Doing the same in 3-D is a complete change. A 3-D conversion they just did required more than 300 people working on 7 Tbytes of data. It’s very demanding.