Will RAID improve 4K editing performance?

[kat.hayes]

I want to get a G Tech thunderbolt enclosure that will house two drives to connect to my 5K iMac. I want to use the enclosure for editing video shot in both HD and 4K from various cameras, including everything from phones to ones that are shot at higher bitrates like some Sony models. I want the drives to perform well, should I configure them in a raid format to speed things up, or just leave the drives as JBOD format?

Thanks.

 

[macduke]

Yes but you need a backup. RAID 0 stripes data between drives and if one goes you lose everything. SSD is the best but most expensive.

 

[kat.hayes]

The enclosure that i am looking at only holds two drives. Am I able to configure it for the RAID format that is fast and also redundant?

 

[joema2]

The enclosure that i am looking at only holds two drives. Am I able to configure it for the RAID format that is fast and also redundant?

No, two drives can only do RAID-0. I have several G-Tech RAID enclosures, plus various other brands. G-Tech is pretty good.

In general, editing H264 4k is CPU limited, not I/O limited. If you transcode to a lower-compression codec like ProRes, then it can become I/O limited since the data rate increases by about 6x or 8x due the less dense encoding.

On both Premiere and FCPX you often must transcode to proxy to get best performance when editing H264 4k. Proxy is lower compression and only 1/2 or less the size of the originals, so that not only reduces CPU load but also I/O load.

However it makes sense to use two-drive RAID-0 when doing 4k editing because eventually you need to move around big files and it's faster for that. You always need a good backup -- even if you're using a redundant format like RAID-5. With RAID-0 you need this even more.

 

[kat.hayes]

1. Why do you use several enclosures? Just curious about your workflow.

2. I use to transcode to ProRes back in the FCP 6 days because it was necessary for ingesting AVCHD media. Since switching to Premiere, I haven't done this. In general, is there a reason to do this these days with either Premiere or FCPX?

3. Ive never heard of transcode to proxy. You mentioned that is lower compression and only 1/2 the size of the original. Wouldn't that mean more compression in order to get 1/2 the size of the original?

 

[joema2]

..Why do you use several enclosures? Just curious about your workflow.

Because I have about 200 terabytes total storage and it all won't fit on one array.

...I use to transcode to ProRes back in the FCP 6 days because it was necessary for ingesting AVCHD media. Since switching to Premiere, I haven't done this. In general, is there a reason to do this these days with either Premiere or FCPX?

In general no -- both FCPX and Premiere can usually edit H264 and AVCHD natively without transcoding. However there are some exceptions: (1) 4k often requires transcoding to proxy to yield smooth editing performance (2) If the video files are removed from the AVCHD folder tree before import, for FCPX this should be transcoded to provide best performance.

....Ive never heard of transcode to proxy. You mentioned that is lower compression and only 1/2 the size of the original. Wouldn't that mean more compression in order to get 1/2 the size of the original?

Proxy has long been an integral feature of FCPX and Adobe finally added this to Premiere recently. With FCPX proxy is fixed at 1/2 the linear resolution of the original file, which is 1/4 the total areal resolution. E.g, if the original UHD 4k is 3840 x 2160, proxy will be 1920 x 1080. The proxy codec is ProRes, so due to the lower density the proxy file is about 1/2 to 2/3 the original size instead of the 1/4 size a straight resolution change would imply.

Premiere allows selecting various proxy resolutions.

 

[kat.hayes]

How many enclosures do you keep connected to your Mac at a time? I also have lots of bare bones drives that I keep in cases on a shelf and I put them into a docking station as needed. It would be nice to have easier access to them as needed.

Any idea how many drives can be connected simultaneously via Thunderbolt before it starts causing a slow down with the Thunderbolt port?

Thanks.

 

[joema2]

How many enclosures do you keep connected to your Mac at a time?...Any idea how many drives can be connected simultaneously via Thunderbolt before it starts causing a slow down with the Thunderbolt port?...

I usually have four RAID enclosures concurrently connected (about 64 TB) out of about 10 total. My 2015 iMac 27 has Thunderbolt 2 and most of my RAID enclosures are Thunderbolt 2, so I've never seen a bus-related performance problem if everything is TB2. However if I put my TB2 RAID-0 SSD array on the same daisy chain as a Thunderbolt 1 array, that will slow it down some.

However I/O can be limited by more than the maximum sequential transfer rate. In the case of small random I/O requests, it's limited by I/O per second rate, which equates to a fairly low megabytes/sec number.

 

[cynics]

@joema2

Since it seems you are very well versed in this subject with real world experience what enclosure(s) would you recommend considering the OP's needs? With and without a budget in mind? Thanks.

While no where near your expertise on this topic the biggest mistake I made (and I'm sure plenty of others have to) and my suggestion when it comes to enclosures of any type is to plan ahead.

I wish I would have consider more bays with my 2 bay NAS. Now my only options are to upgrade the drives for more storage and lose the use of the current drives or (more then likely) get a bigger NAS and just not utilize the additional bays until I need them. PLUS this opens the opportunity for more/better RAID levels.

 

[JamesPDX]

If you'd like to go with SSDs and prefer not to have the noise of fans polluting your edit bay's noise floor, then look here: https://www.blackmagicdesign.com/products/blackmagicmultidock

 

[joema2]

@joema2

Since it seems you are very well versed in this subject with real world experience what enclosure(s) would you recommend considering the OP's needs? With and without a budget in mind?....plan ahead...

There is no simple answer. The OP mentioned a two-drive G-Tech, that's probably the G-RAID, available in various capacities but commonly 8TB. It is pretty good, uses software RAID -- either built-in AppleRAID or you can use SoftRAID: https://www.softraid.com/

In general I'd recommend Thunderbolt over USB 3 because it seems a bit more reliable, plus they are easy to configure due to daisy-chaining. However USB 3 drives or even arrays are OK in some cases. I have about 20 of the Seagate Backup Plus Fast 4TB drives and they are very handy and relatively fast for a bus-powered drive: https://www.amazon.com/dp/B00HXAV0X6/ref=cm_sw_r_cp_ep_dp_Lekjzb2DJ6RT3

However for serious work you normally want an AC-powered drive using external power, not a bus-powered drive.

For RAID-5 a good choice is the OWC Thunderbay 4 series. They are available in various capacities from 4TB to 40TB: https://eshop.macsales.com/shop/Thunderbolt/External-Drive/OWC/ThunderBay-4. You use SoftRAID with those, which works very well and avoids tying you to a proprietary RAID format. With hardware RAID if the chassis fails you can't put those drives in any other brand of enclosure; with SoftRAID you can. SoftRAID is as fast as hardware RAID, or faster in some cases.

For highest performance SSD RAID-0 is good. One of the least expensive ways to achieve that is the OWC Thunderbay 4 Mini chassis plus adding your own SSD drives such as the Samsung 850 EVO. It's under $3000 for 8TB if you build it that way.

https://eshop.macsales.com/shop/Thunderbolt/External-Drive/OWC/ThunderBay-4-mini
https://www.amazon.com/dp/B010QD6W9I/ref=cm_sw_r_cp_ep_dp_Q8jjzbEDYEDZF

If cost was no object, I'd have all SSD storage. 40TB is available from OWC pre-configured for $28,000: https://eshop.macsales.com/shop/Thunderbolt/External-Drive/OWC/ThunderBay-4-mini-RAID5

If I needed shared or NAS storage for FCPX video editing, the best products are from LumaForge: https://lumaforge.com/jellyfish/

Direct attached storage is usually faster and less expensive than NAS but NAS has some benefits. My 8TB SSD RAID-0 array does about 1500 MB/sec write, 1860 MB/sec read, which would be difficult to achieve using a NAS. However LumaForge NAS products are specifically optimized for FCPX and can achieve extremely high performance due special I/O and caching methods.

Spinning RAID arrays can be very fast for sequential transfers, and that covers a lot of video editing. However there is a significant subset of video editing that involves small random I/Os such as when FCPX is making database calls. That's where SSD really is beneficial.

You're right it's important to (1) Plan ahead and (2) Always back up, even if using redundant RAID.

 

[cynics]

There is no simple answer. The OP mentioned a two-drive G-Tech, that's probably the G-RAID, available in various capacities but commonly 8TB. It is pretty good, uses software RAID -- either built-in AppleRAID or you can use SoftRAID: https://www.softraid.com/

In general I'd recommend Thunderbolt over USB 3 because it seems a bit more reliable, plus they are easy to configure due to daisy-chaining. However USB 3 drives or even arrays are OK in some cases. I have about 20 of the Seagate Backup Plus Fast 4TB drives and they are very handy and relatively fast for a bus-powered drive: https://www.amazon.com/dp/B00HXAV0X6/ref=cm_sw_r_cp_ep_dp_Lekjzb2DJ6RT3

However for serious work you normally want an AC-powered drive using external power, not a bus-powered drive.

For RAID-5 a good choice is the OWC Thunderbay 4 series. They are available in various capacities from 4TB to 40TB: https://eshop.macsales.com/shop/Thunderbolt/External-Drive/OWC/ThunderBay-4. You use SoftRAID with those, which works very well and avoids tying you to a proprietary RAID format. With hardware RAID if the chassis fails you can't put those drives in any other brand of enclosure; with SoftRAID you can. SoftRAID is as fast as hardware RAID, or faster in some cases.

For highest performance SSD RAID-0 is good. One of the least expensive ways to achieve that is the OWC Thunderbay 4 Mini chassis plus adding your own SSD drives such as the Samsung 850 EVO. It's under $3000 for 8TB if you build it that way.

https://eshop.macsales.com/shop/Thunderbolt/External-Drive/OWC/ThunderBay-4-mini
https://www.amazon.com/dp/B010QD6W9I/ref=cm_sw_r_cp_ep_dp_Q8jjzbEDYEDZF

If cost was no object, I'd have all SSD storage. 40TB is available from OWC pre-configured for $28,000: https://eshop.macsales.com/shop/Thunderbolt/External-Drive/OWC/ThunderBay-4-mini-RAID5

If I needed shared or NAS storage for FCPX video editing, the best products are from LumaForge: https://lumaforge.com/jellyfish/

Direct attached storage is usually faster and less expensive than NAS but NAS has some benefits. My 8TB SSD RAID-0 array does about 1500 MB/sec write, 1860 MB/sec read, which would be difficult to achieve using a NAS. However LumaForge NAS products are specifically optimized for FCPX and can achieve extremely high performance due special I/O and caching methods.

Spinning RAID arrays can be very fast for sequential transfers, and that covers a lot of video editing. However there is a significant subset of video editing that involves small random I/Os such as when FCPX is making database calls. That's where SSD really is beneficial.

You're right it's important to (1) Plan ahead and (2) Always back up, even if using redundant RAID.

Thanks for taking the time with a thorough answer. Its nice to see what people in the know recommend so I have reference points if/when I need something in the future.

Very good point on the proprietary RAID formats too. Once the enclosure dies you're stuck getting an identical (or near identical replacement) depending on the hardware manufacturer.