Video Editing hardware workflow Query

DragonToe said:

Hi all,

I am currently upgrading my work station. I have been researching cost-effective ways to expand my current setup. The main thing I want to do is to split the workflow up correctly. Namely, have a separate external hard drive for each stage of the workflow: 1) capturing and storing video files; 2) generating previews and media cahches; 3) exporting final video. I want a separate external hard drive for each job, as this seems to be the standard most effective way to run an efficient workflow without bottlenecks.

The trouble I am having is choosing which external hard drives to use. In particular, which connection to go with. I am tempted to daisy-chain 3 firewire hard drives.

BUT I am using a 21.5 inch late 2009 iMac (soon to have 16GB RAM - currently only using 8GB). Graphics ATI Radeon HD 4670 256 MB, 3.06 GHz Intel Core 2 Duo. . I have heard some folks say that daisy-chaining firewire hard drives from a single firewire port (which my iMac has) is bad, and causes bottlenecks (whilst others say it;s fine). My iMac port is firewire 800.

Other options are gigabit ethernet hard drives? Should I get solid state hard drives? Do they come with firewire connection? Should I get 1 SS and 1 normal hard drive?

Or should I get 1 external firewire hard drive and just partition it to save daisy-chaining from a single port? Or doesn't it matter?

I am aso interested in NAS workflow, but assume that I won;t have the money to implement that yet as it means setting up a server and stuff.

Just looking for some general advice on the nest way to go.

I currently run 3 separate hard drives from my iMac (which has the software installed). 1 seagate USB 2 hard drives, which are used for archive, video footage and project files, the other is a Lacie firewire 400 connected via firewire which can go to 800, this drive is used for exporting/rendering, preview files and media cache. With this setup it's starting to clog and is really quite slow.

Any advise is welcome and appreciated.

Click to expand...

I would NOT invest in any more Firewire equipment. Apple is dropping it. It is pretty much a dead-end technology. It was once the best. If you have FW gear keep it but I would not buy new.

There is little point in having a separate hard drive for each job. If you need lots of space and very good performance get RAID. Your plan means that 2/3 of your disk are sitting idle and not contributing to over all system performance or I/O bandwidth. Beter to buy three disks (or five) and place them in a RAID box

Gigabyte Ethernet is reasonably fast. It's close to FW800. The trick is to buy a RAID box with enough performance to take advantage of the 1GB speed. Many low-end boxes will have the 1GB network connection but lack the ability to "flood" the network.

The RAIDed disks will all contribute to bandwidth because the RAID hardware should split the load over the disks and you gain some reliability because the box can handle a failed disk drive.

But RAID is NOT backup. Still keep multiple copies of your work in multiple locations.

The principal reason for splitting workflow to different HDD is to minimize repositioning of the heads; I imagine this improves responsiveness as well as longevity of the HDD (speculation on my part).

Since you do not have really fast input / output available (ie no eSata, Thunderbolt or USB 3.0), my suggestion would be to forget about splitting workflow and instead attach the largest SSD you can afford via Firewire. It goes without saying that a robust backup plan will be needed and you will probably need to keep current project only on the SSD and archive finished works on another (maybe a couple of clones) HDD.

Food for thought?

Maybe someone with way more experience in these matters can chime in!

Cheers

Your bottle-neck is your computer.
You only have one bus option (firewire) on that iMac I do believe.
At this point just make sure you have decent space available to work with.
I don't even think setting up two Firewire 800 drives as RAID 0 will help.
You need to have them on two separate bus ports.
Is this your iMac model?
If you really want to keep this setup and have money to spend, see if you can replace the internal HD or better yet the CD and place an extra drive in that system. A second HD (SSD best) will be way faster than using the system/external FW800 drive.

To add, just replace the FW400 with an actual FW800.
I suggest GRAID.
Ive edited on a GTECH mini with a 2008 Uni MBP for years.
Went from HD to RED 4K with no major issues.
Sure 4K was slow but heck it was slow back when I was editing SD footage

DragonToe said:

Thanks for the feedback people.

Doesn't the separate HDD for each part of the workflow speeds things up? Even if they are relatively slow drives and connections? The computer has to read raw video footage from one drive, whilst generating previews, writing, and media caches, onto another, and then exporting a final output to a third. This is the advice given by many video editing professionals, and Adobe. Are you folks saying that this is not the case?

Can you RAID an SSD drive?

Do you think it's worth doing these upgrades on my current workstation? Or just start saving for a whole new setup?

Thanks for your responses, I'm very grateful.

----------

Oh, and yes, Mbox, that is my computer model.

Click to expand...

There may still be some advantage to splitting the files between different SSD (again I am by no means an expert), but if you are not comfortable with opening your iMac to make the changes suggested by mBox and are looking for best external solution, a single large SSD in a firewire enclosure is probably it.

Remember, you only have the one firewire port and speaking relatively, USB 2.0 sucks.

You can RAID SSD but the only advantages in this case would be (1) larger volume in the case of a RAID 0 or spanning, or (2) redundancy (not a backup) in the case of RAID 1. This is because any single SSD will saturate the firewire connection, so a RAID cannot increase the speed at which data is being read or written!

What you can do with an SSD is improve performance by eliminating the time it takes to reposition the HDD heads. The following is a simple explanation:

Recall that in a HDD, the data is stored on disks that are spinning and the heads must be repositioned each time data is to be read or written; this takes a small amount of time. In the case of video editing, the disk is constantly being read to and written on. It is the constant going from one file to another that makes it advantageous to split files when using HDD.

When using an SSD, there are no moving parts so even though the transfer speed can be the same (a RAID of HDD can provide impressive transfer rates after all), there is practically no delay whatsoever between reading or writing one file and then reading or writing the next!

Like mBox suggested, there are other options available if you want to open your iMac but even then I don't think there would be anything wrong with using a single SSD for everything; this would give best possible results although somehow you are going to have to include large HDDs in your workflow for archives.

Cheers

DragonToe said:

Just read this from another thread:

When asked if a user should get an external SSD via firewire 800 the response was:

'Wouldn't be worth it at all via firewire. SSD would be throttled down to 7200rpm HDD speed.

Click to expand...

It is all about context!

It is very true that over firewire an SSD will not significantly outperform a 7200 rpm HDD in terms of transfer speed (ie the rate at which data is transferred) if you ignore the fact that it is only the outside tracks of the HDD platters are able to transfer data at the highest rate.

However, as previously explained, the benefit of going to a single SSD is preventing the slight lag, and perhaps wear & tear also, that is associated with positioning of the HDD heads for each read and write operation.

A single SSD eliminates the need to use multiple HDD and allows you to have all external data on your fastest input / output port (ie firewire).

If you were to replace your internal HDD or optical drive as suggested by mBox, then you would be connecting via SATA and there would be a significant difference in transfer speed. Again, if you were to install an SDD, there would be little advantage splitting workflow between internal SDD and any other drive provided it has adequate capacity.

Cheers

It's often better to begin with a good enclosure and then choose the drive (spinning or SSD) because you can swap the drive out later.

The Oyen Digital SSD drive you linked to is in this enclosure:

http://www.amazon.com/MiniPro-eSATA...404242210&sr=8-1&keywords=oyen+2.5"+enclosure

I have the model just prior to this one (with a HGST Travelstar 1 TB 7200 rpm disk in it), and it's been a good performer. Your next Mac will have USB3, and so does this enclosure. So it can last you a long time.

Over FW800 I get just above 82 MB/s, according to BlackMagic disk speed test.

DragonToe said:

It still seems a bit risky. I dunno. Hmmmm. But the benefits...It seems like it would be roughly 20 times faster than my current workflow, right? In which case it is immensely temping!

Click to expand...

I can't imagine that installing an SSD will result in 20 times faster workflow.

Replacing your HDD with an SSD can only make a significant difference if the performance of your existing HDD is the limiting factor in your workflow and other components are underutilized. To get a better idea of what upgrades you need, open activity monitor to see where you stand with respect to CPU and RAM utilization (it would be interesting to get information regarding GPU utilization as well but you will have to resort to a third party app for that I believe).

If you replace your HDD with an SDD but CPU was already performing at 90% then obviously there is little room for improvement.

I see from your other thread that you are doing some pretty heavy lifting with this iMac; I've got to wonder if these are paying jobs and if so, why you don't invest in a more powerful computer (more CPU cores, faster interfaces such as USB 3.0 and Thunderbolt, better graphics card, etc) since you would surely be able to take on more work by turning around existing jobs faster!

My son has maxed out late 2012 21.5 inch iMac except for the 1 TB HDD; the logic at the time was that faster storage can be added later via USB 3.0 or Thunderbolt.

He is extremely happy with his computer but I will probably have to custom build a PC so he can render a project in Blender (he is making a short animated film for submission to a film festival that I think will take about 6 months to render since he seems limited to CPU only). I figure I can build a computer with dual GTX 750 ti for less than $750 that will render his project in a fraction of the time. Perhaps a custom rig just for rendering projects is something for you to consider, depending on software licences and OS compatibility (Blender is free and works on OS X, Windows and Linux).

It is after all about having the right tools for the job, although most of us do have a budget to stick to.

For myself, installing an SDD in an iMac would be outside my comfort zone unless the 2009 iMacs are substantially easier to work with than the 2012 models (I checked the videos for my son's computer before he purchased it so that I understood what upgrades could be done after purchase). As for using the SSD in a Windows computer, will not be a problem except that you may have to reformat the disk (not my area of expertise) since the connection will be same the (SATA).

Cheers

First thing's first: best wishes on your upcoming engagement!

Now about your iMac: I think that the best thing you can do is replace the existing HDD with an SSD or install an SSD in the optical bay per mBox's suggestions. This should give you pretty darn close to the max 300 MB/sec limit of the SATA2 interface. I remain convinced that with an SSD available, there will be no need to divide your workflow between drives. The SSD needs to be large enough for your current projects plus you will probably want to put OSX and applications on it as well so that your iMac is as snappy as possible. In general larger SSD are faster than smaller ones but you are limited by the SATA2 interface anyways so no big worry.

If you install the SSD in the optical bay, you can experiment with splitting files between the internal HDD and the SSD; I'd like to hear back if there is any benefit. I think your external drives should be used for backups and archives of old projects only.

Depending on how much you spend on that SSD and what you are comfortable spending, a RAM upgrade probably wouldn't hurt but you could probably put that off until after the SSD installation. This will give you a chance to gauge whether you need even better performance and check the memory pressure assuming the HDD is no longer a bottleneck. You might be able to put your savings toward that ring or next rig.

Best of luck.

Cheers

DragonToe said:

Thanks for the feedback people.

Doesn't the separate HDD for each part of the workflow speeds things up? Even if they are relatively slow drives and connections? ....

Click to expand...

It is only is faster if you can get all the operations running at the same time. That is, overlap the reads and writes. I doubt you can do that. The software will go "read-process-write-read-process-write with not a lot of overlap.

The RAID controller I wrote about WILL do that even for just reading one long file. Three of more disks will always be moving data in parallel. What RAID does is evenly distribute data over all the available disks. So even reading a single file will involve many drives. Of course you pay for this performance, the better systems are close to the cost of a low-end computer because this is exactly what they are.

Even in the best case with two drives (one input, one output) you will never have two drives reading at the same time.

If yo really want performance go with SSD. Keep the project you are wiring on all on one SSD and then move data off when you are done. This is by far better but yo need an SSD big enough.

You r biggest problem is the I/O port. Neither FW400 nor Ethernet are really fast. The RAID held but to go faster you need a faster port, Thunderbolt.

A quick note.. I had a small 120gb SSD laying around (that was slated to go in my PC) and I had an older lacie rugged enclosure that was pretty much shot (hard drive was bunky..)

this is the one with FW400,FW800 and mini usb.

I cracked it opened and luckily it was a SATA interface.. so I took the old drive out, replaced it with my SSD.. hooked it to my mac and bam.. its fast!

Yes yes.. FW800 throttles back SSD speeds, but i moved an edit project from an external G-Raid that was running via FW800 to this SSD combo.. WORLD of difference.

The project is mainly photos and h.264 footage from phones (home video project) and before (using premiere CC, with the below iMac) load times were slow.. and scrubbing the timeline was sluggish (due to the h.264 files).

But with the project on the SSD, I can scrub the timeline like I do here at work (with APR 4444 on a loaded mac pro)

SO.. I can say a 256GB or higher SSD in a FW800 enclosure is very helpful.

DragonToe said:

Hi guys,

I'm shopping for an internal SATA SSD now, BUT i've com across this:

http://www.amazon.co.uk/Seagate-ST2...1404998908&sr=8-8&keywords=ssd+internal+drive

what's the deal with these hybrid drives? It says 600Mb per second. Is this really faster? It seems like it is not a real SSD, but claims to be as fast? Any hints?

Click to expand...

From what I understand those hybrids are good for boot drives.. as over time they learn what apps you use the most and cache the needed files on the SSD part, which yields faster boots and opening programs faster..

I dont know how well they would be for video drives.. they may have decent bursts for read/write.. but I'm not sure how well they fair with sustained reads or writes (rendering, reading large video files)

DragonToe said:

..
what's the deal with these hybrid drives? It says 600Mb per second. Is this really faster? It seems like it is not a real SSD, but claims to be as fast? Any hints?

Click to expand...

It's just a normal hard drive with a bunch of Flash used as a cache. It's best use is as a system drive

For video you'll be jumping around to much to take advantage of any cached data. What you need is the larger SSD you can afford. It needs to be big enough to hold all the files you are working with.

Don't believe that 600Mb per second spec. That is the speed of the cache not the rotating plater. For video you care mostly about the "sustained" IO speed not the burst speed.

Best reward vs risk option for you sounds like an SSD in a Firewire 800 enclosure, just as Chad3eleven commented on. Get largest SSD you are comfortable with from a $ point of view and go for it!

Do you have an existing FW800 enclosure you can purge, and drop in an SSD?

There are a handful of external drives that are SSDs, and have firewire 800.

http://www.amazon.com/MiniPro-240GB-External-FireWire-Portable/dp/B004GIIZMM

But, I will say for years prior to thunderbolt and ssd, normal 7200 RPM/firewire 400 drives were fine.. granted FW800 is much faster.

As for adding an internal drive on your iMac.. if its a risk that you are not willing to take, then dont do it! I will be doing this in the coming months on my iMac once the warranty is up as well..

And keep this in mind, having multiple drives daisy chained, and keeping media on one and caches on another while your apps on your main drive isnt always faster. You will be putting more stress on the CPU and internal hard drive controllers because they have to navigate all of the multiple drives you have, especially with an older iMac.

FireWire800 + SSD > FireWire800 + 7200 drives.. but it comes down to cost vs. performance.

and yes.. my SSD/FireWire 800 experiment is great, it rips!