Content archiving and asset management

 August 3, 2010, Culver City, CA-the Creative Storage Conference looked at storage for content delivery in a panel with disruptive innovator Joe Wojdacz, moderating the panel. The panel members were: Tom Goldberg, vp product development at Cache-A, Nathan Adams, Media Distributors, Chris Duffy, product management at Quantum, Richard Hennessy, client IT architect at IBM, Jeff Hartman, ceo at ProMax Systems, and Sean McKee, IVC/Point 360.

Adams mentioned the changes in audio from analog to digital formats are now moving to the video realm. Now asset management and workflows are keys as production changes from film to a file-based structure. Because of these changes, storage requirements and complexity have increased exponentially. The on-going challenge is to store a large volume of data, keep it safe, and, if necessary, recover it.

A good rule to follow is to have three separate locations for the current data at all times. Therefore, provisioning the storage system actually needs three times the amount of storage that you planned for. To optimize this amount of storage, first you must visualize the whole process to prepare a plan.

Storage has three temporal components; short, medium, and long term. Short-term functions include daily cycle inputs and outputs and can use a number of different platforms including FireWire, BluRay, SD cards, and LTO. Medium-term storage can be on RAID arrays in network attached storage (NAS). This level of data storage residing on a network is highly suited for disaster recovery functions. Long-term storage would be archives on something like LTO version 4 or 5 which have projected a 25 year backup lifetime but have fairly slow read and write times.

The biggest issue with short-term storage is there’s only one copy of the content at the camera. As soon as it’s dismounted from camera, it has to be transferred via Shotput or some other program into the file system. A cautionary note, drag and drop does not check for data integrity upon file transfer completion. The biggest problem with this transfer point is human error, and the only solution is to get experienced talent to perform the function.

Midterm challenges are mostly related to costs. Moving to a NAS array allows for the implementation of large storage volumes at a relatively low cost. Changing the flow by using codecs or other mezzanine formats reduces data volume and on-set color correction eliminates that step in post, which helps control the costs. It should not be surprising that poor or incomplete planning causes costs to increase fairly dramatically. The minimum provisioning for this level is two times the capacity at the ingest stage.

Disaster recovery is another function of themed-level storage. It’s very important to back up all files as soon as possible. Without an instantaneous and automatic backup system, the ability to restore damaged or lost file segments becomes impossible. This backup system must be at least as fast as the working storage, or a lot of data can get lost during data transitions and interruptions. LTO is too slow for this immediate-need process.

Long-term LTO is a bottleneck in the system but is an essential part of an archival flow. To manage all of these storage systems requires extensive asset management capabilities. There are many software products that address asset and storage management, so it’s best to se which one(s) work well with your existing tools and flows.

Goldberg noted that data management is now in many non-information technology groups. Workflow are evolving to include archive, data and platform exchange, and protection. Available platforms for linger-term storage include DVD, which is slow and BluRay, which is more expensive than DVD and still slow. In the future, there may be holographic storage media that is higher capacity and faster for archiving. On-line memory is expensive, but is the only mechanism that can handle the volume and bandwidth for production functions.

The greatest challenge in developing a robust storage system is that hard drives are unreliable and online hard drives burn a large amount of energy. LTO has access latency issues and interchange problems. On the other hand, LTO cost per gigabyte is very low and it has a 30-year projected storage lifetime. All of the IT-based solutions are very expensive and require highly trained staff. If you build your own systems it reduces the costs for equipment, but then requires intensive and constant maintenance to keep it functioning.

Ideally, an archive appliance for inactive content has to be something like a digital toaster, so it can be used for field masters, scope, post, and content interchange without needing a network administrator to run it. A basic platform would include Gigabit Ethernet for the network transport, acquisition media and other inputs connecting to a box running Linux, and a virtual tape environment centered on an LTO-based storage system. The user interface for such a system is like a webpage and only requires a lightweight asset management tool to keep track of the various files and metadata. Commercial products that have these characteristics have both online and remote access support so the system can stay online without much user intervention.

Duffy stated that workflows affect and determine SAN provisioning. The silo nature of content and workflow processes encourages the users to work in a virtual storage pool to share content. Adding in LAN access, driven by a policy-based data management system, enables the application to see the files as local to the user’s machine. In addition, the system can handle remote feeds and processes.

By establishing realistic storage tiers, the user can have online archiving, mobile content distribution and disaster recovery. Additional considerations for server performance and number of users can affect the storage platforms and network architectures as increases in the number of users affects both latency and bandwidth. A shared storage pool and asset management software enables the best use of the critical storage systems.

Hennessy noted that the older workflows and backup systems are inadequate for the job. For example, in older tape systems, the tape index was not on the tape, because the tape was considered one continuous file. Therefore, to be user friendly, a system needs a low IT-ness, with standard system file access mechanisms such as LTFS on LTO 5. The tape file index should be in the index partition, with a copy in the data partition. The user can get much better asset management by using the extended attributes for the files.

Today’s users need to have real-time access to many of the files in the systems, so a mixture of HDD and LTO in the storage systems will be better than just one type of storage. Careful planning and budgeting for the amount and type of storage will reduce cycle times and cost, especially when compared to the total times for tape. A hybrid system not only reduces costs and cycle times but also increases portability.

Hartman looked at the best practices in the industry. He noted that media duplication across enterprise accounts for about half a terabyte of data in a normal work environment, depending on the number of process steps. These extra files can have a negative effect on the inputs, outputs, total capacity, and storage bandwidth in the storage systems. In most cases, the storage devices are selected for speed and capacity, with little regard for latency and throughput.

Most users don’t include backup to low volatility hardware as a part of their storage systems, so they always have to be aware of the need for data recovery. In addition, since HDDs suffer from file corruption and total drive failure, the users have to be aware of the overall robustness and reliability of their systems. One important rule is be to back up what you are unwilling to re-create.

After the various production processes are completed, the files are put into archive. The nature of this operation is partially due to habits and intentions. It’s still important to do this step but it’s also a large consumer of storage capacity. In theory, an archived file is static, it shouldn’t change. The reality is that peole often need access to historical footage for a current project.

Users must understand the value and ramifications of backup and archive. In a good working environment, backup and archive require separate strategies. Integrating (automatic)backup into the workflow reduces the time and effort per step, while protecting the modified data sets. A comprehensive data storage plan must address the question “where is data critical?” and support the needs of the users at that level.

A recommendation is to have an awareness meeting early in the production flow to develop workflow diagrams and storage plans. Then, the backup equipment and plans can be built, based on the original storage plan. The storage plan must identify parameters such as storage frequency and location. Automation requires an existing plan or you cannot tell what you are automatically saving. The awareness meeting should identify and assign responsibilities for all aspects of storage as well as key points in the production flow.

McKee stated that film archiving required three separate black and white color separations, which enabled a hundred year backup lifetime. Issues in restoring digital files are much more complex. The file formats and media of today may not exist in the future. One alternative is to use Bayer encoding on the files to generate a single black-and-white frame embedded with calibration and alignment patterns as well as color information.

In response to a question on long-term storage and integrated asset management, the panel responded that the metadata must be used to manage content through virtualized storage and integrated filesystems. Many of the existing file format standards come with an API which can provide some data protection and recovery functions. There are various strategies for data replication to keep content intact.

Versioning and duplication require planning and clear intent. Users need to look at various tools for asset management to see what they do and how they identify duplicate files. In most cases, these tools automatically identify repeated files and permit the user to choose appropriate actions. Automatic de-duplication technologies still are not robust enough for production environments because very small image changes may prevent easy identification of duplicate files

Archive and back implementation seems to be very difficult. The software exists to enable these functions and automate the storage tasks. Some new tools are emerging that merge asset management and storage management, which are not the same thing. The merging of both functions into a single file view makes both jobs easier.

Users should consider watermarks to automatically identify old files, and combine watermarks, extended attributes, and metadata to make data recovery much easier. Users must consider that over time all hard disk drives will fail, so must plan accordingly. The relative scarcity of talented IT people and minimal budgets for storage hardware exacerbates these problems. Therefore, multiple copies of all files must exist within the system.