Video Editing hardware workflow Query

Discussion in 'Digital Video' started by DragonToe, Jun 29, 2014.

  1. DragonToe macrumors newbie

    Joined:
    Jun 29, 2014
    #1
    Hi all,

    I am currently upgrading my work station. I have been researching cost-effective ways to expand my current setup. The main thing I want to do is to split the workflow up correctly. Namely, have a separate external hard drive for each stage of the workflow: 1) capturing and storing video files; 2) generating previews and media cahches; 3) exporting final video. I want a separate external hard drive for each job, as this seems to be the standard most effective way to run an efficient workflow without bottlenecks.

    The trouble I am having is choosing which external hard drives to use. In particular, which connection to go with. I am tempted to daisy-chain 3 firewire hard drives.

    BUT I am using a 21.5 inch late 2009 iMac (soon to have 16GB RAM - currently only using 8GB). Graphics ATI Radeon HD 4670 256 MB, 3.06 GHz Intel Core 2 Duo. . I have heard some folks say that daisy-chaining firewire hard drives from a single firewire port (which my iMac has) is bad, and causes bottlenecks (whilst others say it;s fine). My iMac port is firewire 800.

    Other options are gigabit ethernet hard drives? Should I get solid state hard drives? Do they come with firewire connection? Should I get 1 SS and 1 normal hard drive?

    Or should I get 1 external firewire hard drive and just partition it to save daisy-chaining from a single port? Or doesn't it matter?

    I am aso interested in NAS workflow, but assume that I won;t have the money to implement that yet as it means setting up a server and stuff.

    Just looking for some general advice on the nest way to go.

    I currently run 3 separate hard drives from my iMac (which has the software installed). 1 seagate USB 2 hard drives, which are used for archive, video footage and project files, the other is a Lacie firewire 400 connected via firewire which can go to 800, this drive is used for exporting/rendering, preview files and media cache. With this setup it's starting to clog and is really quite slow.

    Any advise is welcome and appreciated.
     
  2. ChrisA macrumors G4

    Joined:
    Jan 5, 2006
    Location:
    Redondo Beach, California
    #2
    I would NOT invest in any more Firewire equipment. Apple is dropping it. It is pretty much a dead-end technology. It was once the best. If you have FW gear keep it but I would not buy new.

    There is little point in having a separate hard drive for each job. If you need lots of space and very good performance get RAID. Your plan means that 2/3 of your disk are sitting idle and not contributing to over all system performance or I/O bandwidth. Beter to buy three disks (or five) and place them in a RAID box

    Gigabyte Ethernet is reasonably fast. It's close to FW800. The trick is to buy a RAID box with enough performance to take advantage of the 1GB speed. Many low-end boxes will have the 1GB network connection but lack the ability to "flood" the network.

    The RAIDed disks will all contribute to bandwidth because the RAID hardware should split the load over the disks and you gain some reliability because the box can handle a failed disk drive.

    But RAID is NOT backup. Still keep multiple copies of your work in multiple locations.
     
  3. RCAFBrat macrumors 6502

    Joined:
    Jul 10, 2013
    Location:
    Montreal, QC
    #3
    The principal reason for splitting workflow to different HDD is to minimize repositioning of the heads; I imagine this improves responsiveness as well as longevity of the HDD (speculation on my part).

    Since you do not have really fast input / output available (ie no eSata, Thunderbolt or USB 3.0), my suggestion would be to forget about splitting workflow and instead attach the largest SSD you can afford via Firewire. It goes without saying that a robust backup plan will be needed and you will probably need to keep current project only on the SSD and archive finished works on another (maybe a couple of clones) HDD.

    Food for thought?

    Maybe someone with way more experience in these matters can chime in!

    Cheers
     
  4. mBox, Jun 29, 2014
    Last edited: Jun 29, 2014

    mBox macrumors 68020

    Joined:
    Jun 26, 2002
    #4
    Your bottle-neck is your computer.
    You only have one bus option (firewire) on that iMac I do believe.
    At this point just make sure you have decent space available to work with.
    I don't even think setting up two Firewire 800 drives as RAID 0 will help.
    You need to have them on two separate bus ports.
    Is this your iMac model?
    If you really want to keep this setup and have money to spend, see if you can replace the internal HD or better yet the CD and place an extra drive in that system. A second HD (SSD best) will be way faster than using the system/external FW800 drive.

    To add, just replace the FW400 with an actual FW800.
    I suggest GRAID.
    Ive edited on a GTECH mini with a 2008 Uni MBP for years.
    Went from HD to RED 4K with no major issues.
    Sure 4K was slow but heck it was slow back when I was editing SD footage :p
     
  5. DragonToe thread starter macrumors newbie

    Joined:
    Jun 29, 2014
    #5
    Thanks for the feedback people.

    Doesn't the separate HDD for each part of the workflow speeds things up? Even if they are relatively slow drives and connections? The computer has to read raw video footage from one drive, whilst generating previews, writing, and media caches, onto another, and then exporting a final output to a third. This is the advice given by many video editing professionals, and Adobe. Are you folks saying that this is not the case?

    Can you RAID an SSD drive?

    Do you think it's worth doing these upgrades on my current workstation? Or just start saving for a whole new setup?

    Thanks for your responses, I'm very grateful.

    ----------

    Oh, and yes, Mbox, that is my computer model.
     
  6. DragonToe thread starter macrumors newbie

    Joined:
    Jun 29, 2014
    #6
    Is it risky to replace the iMac CD drive with a HDD? Have you done it? It seems quite expert and sounds like it could go wrong, seen as how Macs are resistant to too much upgrading?

    Also, which area of the video editing workflow do I need to dedicate my speed to? I assumed it would be the preview file writing & reading/media cache area (and not the reading of raw video files, or exporting). Am I correct in this assumption?

    Thanks for your time.
     
  7. RCAFBrat macrumors 6502

    Joined:
    Jul 10, 2013
    Location:
    Montreal, QC
    #7
    There may still be some advantage to splitting the files between different SSD (again I am by no means an expert), but if you are not comfortable with opening your iMac to make the changes suggested by mBox and are looking for best external solution, a single large SSD in a firewire enclosure is probably it.

    Remember, you only have the one firewire port and speaking relatively, USB 2.0 sucks.

    You can RAID SSD but the only advantages in this case would be (1) larger volume in the case of a RAID 0 or spanning, or (2) redundancy (not a backup) in the case of RAID 1. This is because any single SSD will saturate the firewire connection, so a RAID cannot increase the speed at which data is being read or written!

    What you can do with an SSD is improve performance by eliminating the time it takes to reposition the HDD heads. The following is a simple explanation:

    Recall that in a HDD, the data is stored on disks that are spinning and the heads must be repositioned each time data is to be read or written; this takes a small amount of time. In the case of video editing, the disk is constantly being read to and written on. It is the constant going from one file to another that makes it advantageous to split files when using HDD.

    When using an SSD, there are no moving parts so even though the transfer speed can be the same (a RAID of HDD can provide impressive transfer rates after all), there is practically no delay whatsoever between reading or writing one file and then reading or writing the next!

    Like mBox suggested, there are other options available if you want to open your iMac but even then I don't think there would be anything wrong with using a single SSD for everything; this would give best possible results although somehow you are going to have to include large HDDs in your workflow for archives.

    Cheers
     
  8. DragonToe thread starter macrumors newbie

    Joined:
    Jun 29, 2014
    #8
  9. DragonToe thread starter macrumors newbie

    Joined:
    Jun 29, 2014
    #9
    Just read this from another thread:

    When asked if a user should get an external SSD via firewire 800 the response was:

    'Wouldn't be worth it at all via firewire. SSD would be throttled down to 7200rpm HDD speed.
    You would need a thunderbolt enclosure. However as a normal external you wouldn't benefit from the speed even if it were in a TB enclosure as data can't be read fast enough from your internal HDD so that its written to the SSD at the SSD max speed. When reading data from the SSD you wouldn't notice a difference to a HDD. Only if you use it as an external boot drive it would be worth it.'

    So should I just get an external HDD with firewire 800 connection? will this even make a big difference from firewire 400? I mean, a noticeable difference?
     
  10. RCAFBrat macrumors 6502

    Joined:
    Jul 10, 2013
    Location:
    Montreal, QC
    #10
    It is all about context!

    It is very true that over firewire an SSD will not significantly outperform a 7200 rpm HDD in terms of transfer speed (ie the rate at which data is transferred) if you ignore the fact that it is only the outside tracks of the HDD platters are able to transfer data at the highest rate.

    However, as previously explained, the benefit of going to a single SSD is preventing the slight lag, and perhaps wear & tear also, that is associated with positioning of the HDD heads for each read and write operation.

    A single SSD eliminates the need to use multiple HDD and allows you to have all external data on your fastest input / output port (ie firewire).

    If you were to replace your internal HDD or optical drive as suggested by mBox, then you would be connecting via SATA and there would be a significant difference in transfer speed. Again, if you were to install an SDD, there would be little advantage splitting workflow between internal SDD and any other drive provided it has adequate capacity.

    Cheers
     
  11. monokakata macrumors 68000

    monokakata

    Joined:
    May 8, 2008
    Location:
    Hilo, Hawai'i
    #11
    It's often better to begin with a good enclosure and then choose the drive (spinning or SSD) because you can swap the drive out later.

    The Oyen Digital SSD drive you linked to is in this enclosure:

    http://www.amazon.com/MiniPro-eSATA...404242210&sr=8-1&keywords=oyen+2.5"+enclosure

    I have the model just prior to this one (with a HGST Travelstar 1 TB 7200 rpm disk in it), and it's been a good performer. Your next Mac will have USB3, and so does this enclosure. So it can last you a long time.

    Over FW800 I get just above 82 MB/s, according to BlackMagic disk speed test.
     
  12. DragonToe thread starter macrumors newbie

    Joined:
    Jun 29, 2014
    #12
    Hi all.

    Ok, so i've researched the replacing the DVD drive with an SSD option. It seems doable. Have you guys done it? Would you recommend it? I have opened a G4 Mac before, and installed RAM in my iMac, but never the front of an iMac. I have watched YouTube guides on how to do it, and it seems reasonably straight forward, but would you guys suggest that I try it having not done it before? Or would you suggest an Apple retailer? Do they even perform such upgrades?

    If I install it and it goes wrong for some reason, is it easy to fit the old DVD player back in? Are there any typical problem areas to watch out for? Which is the best internal SSD to get? I reckon I could make do with a 150GB, but would much prefer around 500GB.

    It still seems a bit risky. I dunno. Hmmmm. But the benefits...It seems like it would be roughly 20 times faster than my current workflow, right? In which case it is immensely temping!

    ----------

    Plus I can always take the SSD out when I finally build a better computer and use it in, for instance, a Windows custom build, right?
     
  13. RCAFBrat macrumors 6502

    Joined:
    Jul 10, 2013
    Location:
    Montreal, QC
    #13
    I can't imagine that installing an SSD will result in 20 times faster workflow.

    Replacing your HDD with an SSD can only make a significant difference if the performance of your existing HDD is the limiting factor in your workflow and other components are underutilized. To get a better idea of what upgrades you need, open activity monitor to see where you stand with respect to CPU and RAM utilization (it would be interesting to get information regarding GPU utilization as well but you will have to resort to a third party app for that I believe).

    If you replace your HDD with an SDD but CPU was already performing at 90% then obviously there is little room for improvement.

    I see from your other thread that you are doing some pretty heavy lifting with this iMac; I've got to wonder if these are paying jobs and if so, why you don't invest in a more powerful computer (more CPU cores, faster interfaces such as USB 3.0 and Thunderbolt, better graphics card, etc) since you would surely be able to take on more work by turning around existing jobs faster!

    My son has maxed out late 2012 21.5 inch iMac except for the 1 TB HDD; the logic at the time was that faster storage can be added later via USB 3.0 or Thunderbolt.

    He is extremely happy with his computer but I will probably have to custom build a PC so he can render a project in Blender (he is making a short animated film for submission to a film festival that I think will take about 6 months to render since he seems limited to CPU only). I figure I can build a computer with dual GTX 750 ti for less than $750 that will render his project in a fraction of the time. Perhaps a custom rig just for rendering projects is something for you to consider, depending on software licences and OS compatibility (Blender is free and works on OS X, Windows and Linux).

    It is after all about having the right tools for the job, although most of us do have a budget to stick to.

    For myself, installing an SDD in an iMac would be outside my comfort zone unless the 2009 iMacs are substantially easier to work with than the 2012 models (I checked the videos for my son's computer before he purchased it so that I understood what upgrades could be done after purchase). As for using the SSD in a Windows computer, will not be a problem except that you may have to reformat the disk (not my area of expertise) since the connection will be same the (SATA).

    Cheers
     
  14. DragonToe thread starter macrumors newbie

    Joined:
    Jun 29, 2014
    #14
    Hi.

    Thanks for your reply.

    Yes, I am going to upgrade when I can afford it to a better custom built computer, but at the mo I just need to juice up my current computer.

    The reason I think it would be much faster with an internal SSD is because video editing in AE, and other programs, requires a 3 pronged workflow. This means using 3 locations, which should be different drives according to Adobe, simultaneously reading & writing. With USB2 and firewire 400 there is obviously a big lag, because they can only transfer data at a slow rate. With the SSD it seems that I can perform this 3 pronged task much faster because it can read and write to various locations simultaneously. I am upping the RAM of my computer as well, but I don't think that is as important as the drives because whilst rendering it usually uses about 20% of my RAM (at the bottom of the AE render queue).

    Plus, if i just upgrade to an external HDD firewire 800 (instead of my 400) that will only increase 1 of the workflow areas, because the other 2 data locations will be still running USB2, so things won't speed up. I think.

    This seems to make sense to me because let's say I am previewing a sequence in AE, it has to generate previews (from one drive), whilst reading the raw footage (form another drive), and then export a new file (to another drive). It follows logically that these tasks will be slowed down as there are 3 relatively slow buses to get to these locations. Times that sluggishness by 3 and the whole task is slowed significantly. With an internal SSD it will be up to the cores and RAM etc. to push as hard as they can without the data access being the slow point.

    If I've got this wrong please feel free to explain where I am going wrong. But a new computer is not an option at the moment (I am saving for an engagement ring, and don't earn enough money as it is).

    p.s. good luck to your son for his project.
     
  15. RCAFBrat macrumors 6502

    Joined:
    Jul 10, 2013
    Location:
    Montreal, QC
    #15
    First thing's first: best wishes on your upcoming engagement!

    Now about your iMac: I think that the best thing you can do is replace the existing HDD with an SSD or install an SSD in the optical bay per mBox's suggestions. This should give you pretty darn close to the max 300 MB/sec limit of the SATA2 interface. I remain convinced that with an SSD available, there will be no need to divide your workflow between drives. The SSD needs to be large enough for your current projects plus you will probably want to put OSX and applications on it as well so that your iMac is as snappy as possible. In general larger SSD are faster than smaller ones but you are limited by the SATA2 interface anyways so no big worry.

    If you install the SSD in the optical bay, you can experiment with splitting files between the internal HDD and the SSD; I'd like to hear back if there is any benefit. I think your external drives should be used for backups and archives of old projects only.

    Depending on how much you spend on that SSD and what you are comfortable spending, a RAM upgrade probably wouldn't hurt but you could probably put that off until after the SSD installation. This will give you a chance to gauge whether you need even better performance and check the memory pressure assuming the HDD is no longer a bottleneck. You might be able to put your savings toward that ring or next rig.

    Best of luck.

    Cheers
     
  16. ChrisA macrumors G4

    Joined:
    Jan 5, 2006
    Location:
    Redondo Beach, California
    #16
    It is only is faster if you can get all the operations running at the same time. That is, overlap the reads and writes. I doubt you can do that. The software will go "read-process-write-read-process-write with not a lot of overlap.

    The RAID controller I wrote about WILL do that even for just reading one long file. Three of more disks will always be moving data in parallel. What RAID does is evenly distribute data over all the available disks. So even reading a single file will involve many drives. Of course you pay for this performance, the better systems are close to the cost of a low-end computer because this is exactly what they are.

    Even in the best case with two drives (one input, one output) you will never have two drives reading at the same time.

    If yo really want performance go with SSD. Keep the project you are wiring on all on one SSD and then move data off when you are done. This is by far better but yo need an SSD big enough.

    You r biggest problem is the I/O port. Neither FW400 nor Ethernet are really fast. The RAID held but to go faster you need a faster port, Thunderbolt.
     
  17. DragonToe thread starter macrumors newbie

    Joined:
    Jun 29, 2014
    #17
  18. Chad3eleven, Jul 10, 2014
    Last edited: Jul 10, 2014

    Chad3eleven macrumors regular

    Joined:
    Dec 11, 2012
    #18
    A quick note.. I had a small 120gb SSD laying around (that was slated to go in my PC) and I had an older lacie rugged enclosure that was pretty much shot (hard drive was bunky..)

    this is the one with FW400,FW800 and mini usb.

    I cracked it opened and luckily it was a SATA interface.. so I took the old drive out, replaced it with my SSD.. hooked it to my mac and bam.. its fast!

    Yes yes.. FW800 throttles back SSD speeds, but i moved an edit project from an external G-Raid that was running via FW800 to this SSD combo.. WORLD of difference.

    The project is mainly photos and h.264 footage from phones (home video project) and before (using premiere CC, with the below iMac) load times were slow.. and scrubbing the timeline was sluggish (due to the h.264 files).

    But with the project on the SSD, I can scrub the timeline like I do here at work (with APR 4444 on a loaded mac pro)

    SO.. I can say a 256GB or higher SSD in a FW800 enclosure is very helpful.
     
  19. Chad3eleven macrumors regular

    Joined:
    Dec 11, 2012
    #19
    From what I understand those hybrids are good for boot drives.. as over time they learn what apps you use the most and cache the needed files on the SSD part, which yields faster boots and opening programs faster..

    I dont know how well they would be for video drives.. they may have decent bursts for read/write.. but I'm not sure how well they fair with sustained reads or writes (rendering, reading large video files)
     
  20. ChrisA macrumors G4

    Joined:
    Jan 5, 2006
    Location:
    Redondo Beach, California
    #20
    It's just a normal hard drive with a bunch of Flash used as a cache. It's best use is as a system drive

    For video you'll be jumping around to much to take advantage of any cached data. What you need is the larger SSD you can afford. It needs to be big enough to hold all the files you are working with.

    Don't believe that 600Mb per second spec. That is the speed of the cache not the rotating plater. For video you care mostly about the "sustained" IO speed not the burst speed.
     
  21. DragonToe, Jul 14, 2014
    Last edited: Jul 14, 2014

    DragonToe thread starter macrumors newbie

    Joined:
    Jun 29, 2014
    #21
    Hey guys.

    I know this thread is ongoing, but it's a big deal for me to potentially go for such a big upgrade.

    I've heard some nasty stories about this SSD replacing DVD drive in the iMac. Some people lose picture and so on. I have installed RAM, and taken my PC apart and built it up from components, but I've never taken an iMac apart. Is it relatively easy, so long as one is careful? Or are there a few things that can go wrong? Also, the link to the whole SSD and trinkets (casing, suction pads, etc.) is form America. Is there a UK equivalent? A whole package? Or will I have to buy all the bits separately? A lot of people that performed this upgrade use the SSD package mentioned above (OWC SSD kit: http://eshop.macsales.com/item/OWC/DDIMSSD240/).

    Also, do you guys really think it's worth it? Over just buying an external 7200rpm HDD firewire 800 external? Or even two of these daisy-chained? I'm really conflicted about this. I want to do it, but the notion of breaking my iMac is quite scary. And I don't have the spare cash to then go through troubleshooting fixing it with repair parts or taking it to a specialist.

    What do you think?
     
  22. RCAFBrat macrumors 6502

    Joined:
    Jul 10, 2013
    Location:
    Montreal, QC
    #22
    Best reward vs risk option for you sounds like an SSD in a Firewire 800 enclosure, just as Chad3eleven commented on. Get largest SSD you are comfortable with from a $ point of view and go for it!
     
  23. DragonToe thread starter macrumors newbie

    Joined:
    Jun 29, 2014
    #23
    Hi Guys,

    Ok. That sounds like a happy medium ground. But do I have to buy a casing and SSD separately? Or can I go with an SSD external with firewire 800 already built in? i.e. what is the benefit of buying the two items separately and combining?
     
  24. Chad3eleven macrumors regular

    Joined:
    Dec 11, 2012
    #24
    Do you have an existing FW800 enclosure you can purge, and drop in an SSD?

    There are a handful of external drives that are SSDs, and have firewire 800.

    http://www.amazon.com/MiniPro-240GB-External-FireWire-Portable/dp/B004GIIZMM

    But, I will say for years prior to thunderbolt and ssd, normal 7200 RPM/firewire 400 drives were fine.. granted FW800 is much faster.

    As for adding an internal drive on your iMac.. if its a risk that you are not willing to take, then dont do it! I will be doing this in the coming months on my iMac once the warranty is up as well..

    And keep this in mind, having multiple drives daisy chained, and keeping media on one and caches on another while your apps on your main drive isnt always faster. You will be putting more stress on the CPU and internal hard drive controllers because they have to navigate all of the multiple drives you have, especially with an older iMac.

    FireWire800 + SSD > FireWire800 + 7200 drives.. but it comes down to cost vs. performance.

    and yes.. my SSD/FireWire 800 experiment is great, it rips!
     
  25. DragonToe thread starter macrumors newbie

    Joined:
    Jun 29, 2014
    #25
    Hi all.

    Been mulling over this quandry and still haven't decided which way to go.
    It seems a waste using an external SSD via firewire 800 bus as it will only transfer 800 Mbits/s, which equates to 100 MB/s. This leaves much of the SSD's potential speed untapped (up to 500MB/s). And so it seems like a waste of money. I know I can have multiple tasks running, which is ideal, but I still feel uncomfortable about that loss of speed with the firewire 800 choking it up.

    I guess the wise decision is to go with the internal SSD in the optical DVD drive, as users have suggested above. But it's just REALLY SCARY.
    I reckon I could do it - but it's just the thought of something going wrong, or, even worse in a way, things not being QUITE RIGHT when I boot up, and not being able to diagnose the problem.

    I could go a lot cheaper and buy 2 or 3 7200 rpm HDDs and diasy-chain them via the single firewire port? But I've heard that daisy-chaining reduces the speed even further. This would still be faster than my current setup though I'm guessing (1 500rpm HDD firewire 400 for previews, cacheing and exporting; 1 500rpm HDD USB 2 for raw video files; software ran from internal iMac local drive).

    I'm really stuck here. Don't know which way to go. If anyone has successfully performed the SSD in the DVD drive upgrade in a late 2009 iMac please reply and let me know how difficult it was.

    Thanks guys.

    My iMac spec (upgraded to 16 MB RAM).
     

Share This Page