A DUMMIE'S LOOK AT COMPUTER VIDEO THROUGHPUT

White Paper

October 15, 2001

by Ron Lindeboom
ron@creativecow.net
Creative Cow, Cambria, California USA

A DUMMIE'S LOOK AT COMPUTER VIDEO THROUGHPUT
(or 'A Dummy Looks at Computer Video Throughput' -- take your pick!)

About this article:
In a very basic look at issues like "What Is an Array", "What is RAID", and "What Is Striping" -- Creative Cow's Ron Lindeboom explores the issues that make digital video possible on personal computers. He examines the fundamentals and gives newer users the kinds of basics that they need to know to successfully configure their video drive solution.

In the world of digital video, great amounts of data must travel in a "sustained" stream in order to avoid losing frames, audio, etc., during both the capture and playback process. The amount of data moving between the computer and the hard drive(s) must maintain a high enough level that it never "dips" to a capacity that forces the system to "drop frames" or audio as it races to keep pace with the data being processed through the video capture card, etc.

A good way to think of this would be to picture your video capture card as throwing baseballs and the harddrive as the catcher that must catch them. If the video capture board in your computer starts throwing more balls than your many-armed catcher can catch -- then balls are going to be dropped. That's what's happenning as data is pumped into your system and your drive(s) must have enough "hands and catcher's mitts" to avoid dropping frames and data.

In the early days of digital video on the desktop, this phenomenon was addressed in what were called "AV Rated" or "Sustained Rate" drives. These drives were capable of supporting these high bandwidths needed in the process of desktop video. Today, you don't see those terms much anymore. By the time you finish this article, you'll know why and also the changes that have made drive prices drop, capacities sky-rocket and overall performance curves to swing strongly in favor of the user.

Over the years, on all of the lists and forums I frequent, I have heard one new video editor after another ask why their video is dropping frames. After all, they bought a huge drive and "Man, it's 20gigs! So why isn't my system working? I get a couple of seconds of good video and then it starts dropping frames!"

They rarely go far enough in their diagnostics to narrow it down to the drive throughput and usually they blame their software: "Premiere sucks!" Not to pick on Premiere here, it's simply the largest selling video editing software out there and so it catches the most blame. But most capture, playback and drop-frame problems are solved by resolving "throughput" issues, not software problems. And throughput requires balancing a computer's hard drive(s) and the controller that negotiates between the computer and the drives.

NOTE: Since we are dealing with basics, please be advised that using your base "C" drive (or the "system" drive for Mac users) for video storage and playback is asking for trouble. You need a second drive (or more) for video. This drive is dedicated to video and audio files. Playback will be much more reliable if you are not trying to use your C drive for this purpose.

DV: In the world of DV, with its data-rate of just 3.6mbps to maintain, even a single IDE drive will usually handle the job quite handily. For more on EIDE drives in DV, read Kathlyn Lindeboom's article at the Cow.

M-JPEG: Throughput problems are much easier to resolve in the world of DV than in the analog world of Motion-JPEG (M-JPEG). In this world, all the rules change. And playing by the rules of M-JPEG has traditionally meant high-priced SCSI drives along with expensive SCSI controller cards like those from ATTO and others.

But back in 1997, all that began to change when drive systems started appearing in the market that were built around low-cost off-the-shelf IDE drives -- usually in an enclosure that used SCSI firmware to exchange the data between the computer and the drive. By creating a drive system that used multiple IDE drives in an array, users could get SCSI-level performance from far-cheaper IDE drives. Some of these systems, like Medéa's VideoRaid systems, use a built-in hardware controller card that automatically recognizes this array of multiple IDE drives working together as a single drive. No special software needed! The race to change the marketplace was on.

Today, most of the SCSI drive manufacturers are now little remembered footnotes in computer history -- casualties of how well the idea of using IDE arrays has changed the market forever.

In the early days of NLE systems back in 1994-5, users had to pay approaching $4,000 or more for 9gigs of AV-rated SCSI storage. That has dropped (at the time of this writing) to the point where 36gigs of UW SCSI sells for about $500. But a 40gig Ultra ATA/66 IDE drive sells for around $125 or so. Not only is there a substantial cost advantage to using IDE drives but SCSI drives operate at heat levels that exceed their IDE counterparts -- especially important when considering a system that uses internal drives. With issues like cost and cooler performance to consider, these differences are fueling a move to IDE arrays.

HOW RAID WORKS...
RAID ARRAYS: What Are They?

In the world of personal computers, hard drives are recognized by an individual ID such as the "C" drive, "D" drive etc. The individual "behaviours" of these drives are based on their speed, drive type (SCSI, IDE, 1394, etc.), etc., and you cannot break the limits of their individual "throughput ceilings" without the system "bottle-necking" and giving you errors like dropped frames, etc. Since the drive can only handle as much data at any one time as it was designed to handle, then your drive must "grab what it can and lose the rest" if you exceed that ceiling, This is what is happening when you "drop frames."

In a RAID system -- which stands for: Redundant Array of Independent Disks -- you can combine multiple single drives into a shared system wherein each drive carries only a part of the overall load. This allows a user to combine multiple lower throughput drives (like low-cost IDE drives) into a system that looks like a single drive to your system and has a total throughput high enough to handle video, etc. This is known as an "array" of drives. Using this arrayed scenario, your computer now sees the "array" of drives as a single drive and will treat it that way when digitizing, etc., from your digital video system.

For example's sake: Let's say that you have a hard drive with a sustained rate throughput of 9mbps. This would be high enough to handle DV footage which is less than 3.6mbps -- but you could not work with many of the M-JPEG analog formats that require much higher throughputs.

But by using an array of four of these drives -- each drive pulling 9mbps -- they would now yield an aggregate throughput of 36mbps sustained (4x9=36). This would allow many more M-JPEG formats to be used. Uncompressed video takes over 20mbps sustained for a single stream. And as stated earlier, if you are using DV as your primary format, even a single drive that can maintain speeds in excess of 3.6mbps is sufficient for the 5:1 compressed nature of DV files.

In an IDE RAID system you can use up to four drives in the array. In SCSI, you can have up to 15 drives in the array.

NOTE: While the advantages are obvious when you consider the positive aspects of using arrayed drives -- there is a serious downside to this technology and it is this: If you lose one of the drives in the arrary, you lose your project. You cannot usually recover your project from just the remaining good drives. Companies like Medéa have released new RAID3 systems that combine video array performnace (known as RAID-0) with a redundant "mirroring" protocol (RAID-3) that protects valuable data.

In computer "geekspeak," another factor to be aware of is the Mean Time Between Failure (MTBF). With an array of drives, the MTBF will equal the MTBF of the lowest-rated individual drive, divided by the number of drives in the array. If the MTBF is too low for your liking or confidence, you can always look at the "mirrored" RAID protocols like RAID-3 and RAID-0/1.

Here is a basic look at RAID types applicable to video and their purpose:

RAID-0: This is what you'd use for digital audio/video use. It basically takes multiple individual drives amd ties them into a single drive "array." It offers no verification or "fault tolerance."

RAID-1: Appears to your computer as a single drive but is actually two drives, one an exact duplicate of the other. This realtime "mirroring" allows for data protection in mission critical operations in fault-tolerant uses like banking, government, business, medicine, education, etc.

RAID-0/1: This protocol allows for the speeds needed for digital video in the RAID-0 portion of the system and includes what is usually a near realtime RAID-1 system for redundancy and data protection. Requires a minimum of four drives to acheive: two as the primary data RAID and two as the mirror drives to create the exact duplicate of the primary drives.

RAID-3: This RAID level allows for the data drives to be striped and includes a back-up disk for data protection. Requires at least three disks to implement. Controller design is a complex procedure and the resource intensive nature of the process makes it very difficult to accomplish with software-only solutions.

SOFTWARE STRIPING OR FIRMWARE STRIPING:

Usually, RAID systems require some form of drive "striping" software to create an array from multiple drives. The most popular over the years have been Charismac's Anubis and RAID and REMUS for Mac -- and today, Windows NT and Windows 2000 have striping as a function of the OS's formatting options and W2K tools like RaidToolBox take things even further in options and functionality. Other varieties of Windows will require some form of software utility or a hardware card that automatically creates the solution in firmware. Software drive formatting utilities like those just mentioned, do things invisibly for users that used to have to be done with arcane and cryptic codes. Today, tools like Anubis not only stripe the drives together but automatically set the drives for maximum performance for digital video.

The basics of "striping" are understood by the following points:

Multiple drives are combined to form a single more powerful array;
These drives are then "interleaved" by setting the sectoring value;
Sectors can be as small as 512k and as large as several gigs;
Smaller sectors mean the read/write heads will access "stripes" across drives more frequently;
Larger stripes mean less frequent interleave sectors and a less efficient "striping" interleave;
Digital video and audio should therefore use smaller sectors not larger ones.

New systems like the Medéa drives (as mentioned earlier), need no special software as they use a firmware system inside the drive enclosure. New cards such as the Promise RAID controllers also do the same and control multiple IDE drives from a single PCI board.

You need at least two drives in order for a RAID system to work and "No, you can't use partitions on the same drive." (In case you were wondering!) A card like a Promise RAID controller goes into your computer's PCI slot and once there, your computer will recognize it as a SCSI device. (Windows users note: This is as long as your BIOS settings are as outlined in the support documentation.) These cards come in two and four drive configurations and as such, double and quadruple your drive performance.

Lastly, let's say that you put two 5,400rpm 20gig drives together with two 5,400rpm 6gig drives. Do they then become a 52gig array??? No. They will all format at the level of the lower capacity drive. So, if your lowest drive is a 6gig, all your drives will format as 6gig drives even your 20gig drives. So make sure that you plan correctly or you may find yourself wondering why with 52gigs in the capacity (as in the previous example), you are only seeing 24gigs.

Feedback? Write Ron Lindeboom at ron@creativecow.net

©2001 by Ron Lindeboom and creativecow.net. All rights are reserved.
[Top]
This article first appeared on creativecow.net and is reprinted here with permission.
All screen captures and textual references are the property and trademark of their creators/owners/publishers.

RAID-0:	This is what you'd use for digital audio/video use. It basically takes multiple individual drives amd ties them into a single drive "array." It offers no verification or "fault tolerance."
RAID-1:	Appears to your computer as a single drive but is actually two drives, one an exact duplicate of the other. This realtime "mirroring" allows for data protection in mission critical operations in fault-tolerant uses like banking, government, business, medicine, education, etc.
RAID-0/1:	This protocol allows for the speeds needed for digital video in the RAID-0 portion of the system and includes what is usually a near realtime RAID-1 system for redundancy and data protection. Requires a minimum of four drives to acheive: two as the primary data RAID and two as the mirror drives to create the exact duplicate of the primary drives.
RAID-3:	This RAID level allows for the data drives to be striped and includes a back-up disk for data protection. Requires at least three disks to implement. Controller design is a complex procedure and the resource intensive nature of the process makes it very difficult to accomplish with software-only solutions.