Storage for Video Editing at Creative Storage

 August 3, 2010, Creative Storage Conference, Culver City, CA—The new digital workflows are forcing change in the structure of a production company. The average video runs between 200-800 GB per project. The average editor processes about 8 TB of data, mostly held on Firewire (IEEE 1394) RAID arrays. The smaller producers have no IT staff and a small budget for hardware. Technology changes are altering the production processes resulting in over 400 formats and a wide range of data rates for the workflow. As the high end benchmark, the workflows for Avatar generated over 4 petabytes of data.

One panel looked at storage requirements for creative generation. Larry Jordan was the moderator and the panel was comprised of the following people.
Steve Kantor, freelance editor, Rob Caldeira, marketing director at Hitachi GST,
Mark Fuccio, senior director of products and marketing at Data Robotics, Dean Neumann, senior marketing manager at NetApp, and Bill Moran, senior marketing manager at Avid

Kantor described the working environment and requirements for a small facility. Production storage requires three main components; speed, size, and security. Typical requirements would be 1200-1500 Mbits per second data rate, 3 times the equivalent capacity that would be used for tape, and a system that is reliable and redundant. The good thing is that the costs of storage have dropped over time so a TB of storage only costs about $400.

The main tools for editing in the smaller operations are the Avid video products and Apple’s Final Cut. The Avid video product has a number of formats that depend on the output frame size and data rate. It generates a master and a backup file. Final Cut generally runs with storage on a network. To help manage the storage requirements, the workflows should employ video codecs for most of the editing processes. These codecs could be compound, full, or digitally lossless. Employing transcoding can reduce rendering time and costs while also reducing storage requirements.

Digital production needs lots of capacity and bandwidth. The optimal mix would be to get as much of both as possible. HD storage requirements are in the range of 558 GB/hr
Shooting ratios for a 30 minute feature are about 2-3 TB, a documentary might run 60-200TB/hr, reality TV might be as high as 4-20 TB/hr due to the requirement to record 24 hours at a time accounting for about 23-35 TB per episode. Other types of content have the following characteristics; episodic TV tends to run about 15:1 shooting ratio (15 hrs input for 1 hr broadcast). The production might have three editors working on 45 – 57 hours of material resulting in 2-4 TB for broadcast. Often, companies will be operating on 3-5 episodes in pipeline at a time.

Rendering operations require speed. A storage system for rendering will have a RAID 5 or 6 array with 2 methods of backup. For close to real-time results, users want to stream their data with minimal downtime for expansion and maintenance. In many cases, the removable drives store off-line files and materials for archiving rather than on-line storage, due to the need for the high bandwidths editing requires. Because smaller users don’t have the budget for the high-end equipment or the IT expertise needed for the enterprise-class storage, they encounter equipment problems such as hang-ups or crashes when the low-end drives have hardware or software conflicts.

Caldeira talked about the “deep opportunity” trends in storage. As of March 2010, YouTube was uploading 24 hours of video per minute. This data volume is greater than the sum of all the networks for the past 10 years. iTunes has 10 billion song titles, and Flicker has over 4 billion photos online. In ’09 the market for storage grew and continues to grow today. The market for PCs sold 300 million units, and the average number of storage units in the hard drive industry is a ratio of 6:1 for hard drives to PCs. The total number of hard disk drives was over 600 M units. The solid state drives (SSD) accounted for 11 million units.

Projected growth for the industry is to 1.5 billion HDD by 2015. One large driver is video, which requires drives optimized for throughput as well as overall capacity. The need for speed and capacity is driving system architectures from direct attached storage towards network-based storage. To improve performance, non-volatile memories are being integrated into hard disks. Pricing depends upon aerial density and total parts count. As both change in the right direction over time, costs drop over time.

The state of the art for storage is 600-700 GB/square inch. This results in small form-factor drives with 1-2 TB drives of capacity. By ’20 the storage density is expected to be in the range of 10k GB/square inch which results in a 90 TB drive in the same 3.5 inch form factor. To get these higher densities, drive manufacturers are working on bit patterned media, thermal assisted recording, and shingled magnetic recording.

The biggest challenge for flash-based SSDs is bit density. The maximum size of a SSD will always be lower than that possible in HDD. The ability to scale the lithography within ICs is the major holdback for greater density, with the result that costs for a SSD will always be higher than those for HHDs. Other non-volatile technologies under consideration are ferro-magnetic, and holographic media. The technologies will continue growing thru ’40.

Fuccio commented that the video industry needs fast, affordable, and expandable storage. More that anything, the content creation industry needs speed and capacity. Over time, the data volume, bandwidth, and interfaces for HDDs have changed to meet their requirements.

The technologies for RAID storage have been static since ’84. RAID is a complex environment better suited for the corporate space. It has management issues and is fairly inflexible and difficult to reconfigure. For example, a fibrechannel interface costs $1,000 and requires a lot of set-up effort. In comparison, an eSATA interface that delivers about the same performance is about $200 per port. Therefore, a storage setup based upon iSCSI on GigEthernet is a good, standard package that delivers robust and flexible storage for low cost at 125 MB/s .

To meet the growing and changing demands for storage, users need access to incremental increases in the number and types drives for greater ease of use. By virtualizing the storage arrays, the management software can create a system with one drive data protection overhead, ease of provisioning, and simple expansion in any increment.

The hardware and software interfaces like iSCSI, FireWire, or USB are an issue in system architectures. Because the hardware ands software interfaces are well known, the choice of drives depends more on the user’s applications and workflow.

Moran offered an example of collaborative storage. In one network news center, editors were working on new incoming material for a story that was being broadcast at the time. In this big system environment, many editors and feeds are all getting processed simultaneously. To perform these functions in real time requires an inherent quality of service perspective that allows isochronous enter and play out capabilities. The editors need equal access to common content in the system, and system must operate at the scale of the operations.

The architecture must address issues of capacity, redundancy, networking, and costs. Underlying all of this is some form of latency management that can address I/O deadlines and priorities, network schedules, and avoid hot spot file structures. All of this requires distributed intelligence to achieve proper functionality. Intelligent workflows can solve many problems associated with storage and other integrated components in the system. Existing tools understand the basic file flow and can archive completed files while communicating intelligently with the rest of the storage system.

Neumann proposed a cloud-type storage system to perform the functionality required in production, postproduction, and package delivery. In a distributed work environment, metadata is bumped around on heterogeneous platforms and media, through multiple applications from multiple vendors. The various tools run on different operating systems and protocols. In many cases, the content is linked to other functions and versions need some type of protection.

HD and other high density media modes have broken the existing flows. Now the delivery vehicles push, store, and replicate in one pass leading to a situation where everything exists online. To manage all of this requires metadata driven data management structured as an object oriented flow. Ideally, local and remote virtual storage are integrated.

Users, such as studios and broadcasters, can use a cloud –type storage system and the systems do not have to be Internet-based. Globalization and automation are the key determinants for storage structure and architectures. In the high-end systems, the users will be working over leased lines with high bandwidth on demand. The use of a T1 line is inadequate for anything but archive functions due to the limited bandwidth. External providers can customize a cloud system for any specific workflow and the cloud can be internal. If the system is Web based, it can be designed to be content aware and automatically generate the appropriate metadata for the functions.

In response to a question on file formats and operating systems, the panel responded that small shops tend to be back-end centric for their storage needs. The best approach to mix the formats to virtualize. Interoperability is possible through a storage attached network, which can have multiple file partitions to handle the different file structures. A bigger issue than that storage is local reliability and the avoidance of human errors.