DIY Video Editing Powerhouse Help

Discussion in 'Mac Pro' started by screenwriting99, May 8, 2009.

  1. screenwriting99 macrumors newbie

    screenwriting99

    Joined:
    May 7, 2009
    Location:
    Los Angeles
    #1
    Hey guys. I'm a new Mac User and Seasoned Video Editor who is switching from the PC world. Please be gentle on me. I just pulled the trigger on the new 2.66 Octo mac pro this morning. I need to figure out a couple of things that have been causing me major confusion.

    1) How fast can I get the throughput on 4 1TB Hard Drives using OSX Leopard software Raid 0? Fast enough for 2 or more Streams of 35mbs a second HD XDCAM Footage at 60i? Link to camera here http://pro.jvc.com/prof/attributes/features.jsp?model_id=MDL101851&feature_id=01

    (The reason for going OSX softraid is everyone keeps telling me internal hardware raid cards are now incompatible because of the new drive tray design. The 4 Drives I am going to be using are the WD RE3 1TB Drives)

    2) Would 12gbs (2gb x 6 via OWC) of Ram be enough for Editing this kind of Footage in Final Cut Pro or Avid 3.5 in near real time?

    Any advice would be greatly appreciated as I have been living on the boards for the last two weeks trying to figure this out. Trying to get the most bang for the buck without taking out a loan. Unless necessary. I am Editing a Long form Documentary for a Non Profit group. Thanks.
     
  2. giffut macrumors 6502

    Joined:
    Apr 28, 2003
    Location:
    Germany
    #2
    You ...

    ... won´t need much RAM for Final Cut at the moment, as it still is a 32Bit application and therefore limited to a maximum of 4GB (1,5 to the application itself, 2,5 to the user). Nevertheless the more RAM you have, the more performant your machine will stay while using other applications at the same time, let´s say Photohop or Motion. It is also recommended to stack the machine´s triple ram slots for some advantage in memory speeds (not much, but sometimes handy; you already had that in mind, I guess). If you need maximum RAM, you still can fill in the fourth slot.

    The SoftRAID will be enough, for sure. Depending on size needs, I would recommend to partition two of those drives: First one 128GB RAID partition and second 800GB temporary storage on each of those drives. Then use Apples SoftRAID to create a RAID 0 of the two 128GB partitions. Those will give you quite some decent speed, more than enough for your needs. I would use the two other partitions for temporary data/video storage of files you won´t need at the moment: Copy anything you want to wortk on to the RAID 0 first. Use the two other harddrives to create backup strategies for the RAID 0 and the data partitions. For those two drives I would recommend 1,5 or 2TB sizes.

    Get, if you haven´t already, the ATI 4870 videocard, as Apple offers quite substantial hardware acceleration for their Pro Apps - NVIDIA still lacks some performance here, due to insufficient driver optimization.
     
  3. screenwriting99 thread starter macrumors newbie

    screenwriting99

    Joined:
    May 7, 2009
    Location:
    Los Angeles
    #3
    4870 versus Geforce or Quadro?

    Thank you for your Advice on the softraid being enough. I was actualy told at the Apple Store not to get the 4870, but instead wait for the new Quadro Card. What do you guys think? Will the Stock Geforce 120 crd be ok for Video Editing in the XDCAm format?
     
  4. giffut macrumors 6502

    Joined:
    Apr 28, 2003
    Location:
    Germany
    #4
    Yes, ...

    ... for the beginning, it is ok. Just start working now and let your workload/techniques straighten out for a couple of months. If you then experience serious bottlenecks regarding the videocard, e.g, you still can upgrade. Same goes for proving monitors, possible capture/hardware accalaration cards, control devices, additional video gear etc. pp.

    The Nvidia Quadro is much more needed if you do heavy 3D related work - which you don´t. It is overkill for your needs. You are probably much better suited with a acceleration/capture card, like from Aja (Kona 3), Blackmagic (Decklink HD Extreme) or Matrox (MXO2, native XDCAM support) - for later. Your main interest now should be file/data managment and safety, as your video projects are now solely file based; then how to use the software in best fashion (speed, quality wise).

    Remember: "You upgrade in need for, not to something" (Wladimir Poppovitch)
     
  5. screenwriting99 thread starter macrumors newbie

    screenwriting99

    Joined:
    May 7, 2009
    Location:
    Los Angeles
    #5
    Software Raid versus Software speed cont...

    What would the throughput be in raid 0 using four 1TB drives in a set using OSX? Would it be sufficient for XDCAM 444 10 bit video at 35mbs?
     
  6. giffut macrumors 6502

    Joined:
    Apr 28, 2003
    Location:
    Germany
    #6
    You ...

    ... wouldn´t need to RAID 0 all four drives, don´t do it. One drive approximately gives you around 70MByte/s, RAID 0 with two drives around 140Mbyte/s. Doing it with first partitions of drives will give you another boost from around 20 to 30%.

    XDCAM is no uncompressed format, therefore its strains on disc I/O is neglectible. Its actual throughput is around 3.5Mbyte/s - comparable to DV PAL/NTSCS. Fast calculation: One single harddrive will give you all the throughput for more than a dozen XDCAM streams. You wouldn´t even need a RAID 0 configuration to begin with - but its handy for general I/O workflow optimization (like putting Final Cut Procache files onto it, same for Photoshop etc.).

    Much more, though, you need a fast processor to cope with rendering XDCAM - you got that one for now, for sure!

    I could think of a setup like this:

    Internal drives:

    1x640GB drive, 2x1TB drives, 1x2TB drive

    First drive (640GB): Boot drive, all software, all your personal data, any data which is not needed for video work

    Second drive (1TB): two partitions; 1x128GB (first of the two), 1x800GB

    Third drive (1TB): two partitions: 1x128GB (first of the two), 1x800GB

    RAID 0 the two 128GB partitions and use it for caching files (Final Cut Pro, Photoshop, Color, Motion etc.), use the two 800GB partitions for temporal data storage (rough cuts, work in progress etc.pp.)

    Fourth drive (2TB): video file storage

    External drives:

    First external drive: 2TB, Backup of video files
    Second external drive: 640GB, Backup of cloned System HD (weekly scheduled)
    Third external drive: 1TB, Time machine Backup of System drive (daily)
    Fourth external drive: 1TB, Backup of temporary storage, maybe scratch RAID 0

    For external storage I would recommend getting the Highpoint RocketRaid 2314 eSata controller: http://www.amug.org/amug-web/html/amug/reviews/articles/highpoint/2314ms/

    It offers 4 native external eSata ports, which can run port multiplier cases. Those are cases for 4 or 5 harddiscs all in one, which then can run via one single eSata port: http://www.amug.org/amug-web/html/amug/reviews/articles/firmtek/5pm/

    So with a 4 Port eSata card you can get up to 20 HDs connected, which comes handy for your storage needs. Stay with individual HDs for the moment, and think about redundant RAID levels (like RAID 6) later on, when your storage needs exceed single drive capacities.

    Choose your backup strategy according to your risk attitude - but don´t save big money on it too much: Do at least get reliable backup protection for your boot/data drive and for the native video file storage coming from your JVC. You can get protection for your work in progress later on, when budget flexibility allows it. This would suit well, I think, for the time being.
     
  7. screenwriting99 thread starter macrumors newbie

    screenwriting99

    Joined:
    May 7, 2009
    Location:
    Los Angeles
    #7
    Thanks

    Thanks for the Raid Breakdown. I am going to use your Formula for sure. It makes sense. What would I do with the 2TB? Just leave it off the Raid 0 regular HFS formatting, correct? Since it won't be part of the raid 0 set.
     
  8. giffut macrumors 6502

    Joined:
    Apr 28, 2003
    Location:
    Germany
    #8
    Use ...

    ... Apples "Disk Utility" to format all drives with "HFS extended journaled". The same applies to partitions. When you create any RAID, it will have one single file structure - with OSX always use "HFS extended (journaled)".

    OSX itself uses virtual memory- as much as Windows does - and the boot drive is the location for those hidden swap files. Let´s say you extensively use Photoshop and it would use the boot drive, too, for caching files; same goes for Final Cut or any high I/O demand. By just using the boot drive you now have the harddrive jumping around to read/write all those cached files. Caching is always done, when physical memory no longer is enough or files are too huge.

    By separating it, ergo letting the system have one spot to put them on, and applications another one - you even might let each application have one on its own, e.g.: For photoshop a fast, but small Sata SSD (flash) could be used. Same goes for Final Cut Pro. If I/O demand is high, you start using schemes like RAID 0 to even further prolong this for more throughput (does not significantly speed up access times, but read/write throughput; for fast access times you need fast discs; the best option for this is SSD - but expensive).

    You can create partitions as much as you like, though, but I would recommend to keep the structure clean: Clear names for harddrives and one physical harddrive is one named partition; create partitions only where necessary (separate boot volume, cache volume), for sorting and organizing use folders on the harddrive, NOT partitions. See it this way: Soon you will find lots of harddrives on your desktop. Creating lots of additional partitions makes it stuffed with disc icons. Keep it tidy.

    Don´t use the 2TB option for any RAID, just use it as plain data disc - no stressful I/O racings. Maybe you could for the moment leave the RAID 0 off your harddrives altogether. Come back to it, when you really need it. Just use separate drives, do start working, then look out for I/O problems. I doubt you will have them for what you plan to do. For this, of course you should be more specific about your exact workload.

    I, e.g. have one 200GB boot drive, another two 160GB cache drives, each for my virtual machine images and my Logic/Photoshop projects, then several data drives up to 1TB. My budget is limited, therefore old technology is still in use, but never mind: Even by just choosing different drives for caching you can speed up your machine quite significantly. The I/O is the real bootleneck today, and harddiscs are becoming slow compared to SSD for those tasks (different RAID schemes are the history of getting rid of those bottlenecks, but with enormous costs of controllers and connecting infrastructure, high performance drives etc.pp.). Nowadays it seems to be almost commoditiy, but don´t get fooled: RAID always is only the best possible compromise tailored for specific tasks. Only use it, when in need for it.
     

    Attached Files:

  9. Tesselator macrumors 601

    Tesselator

    Joined:
    Jan 9, 2008
    Location:
    Japan
    #9
    4 SATA 1GB Black drives should give you in the neighborhood or 380 MB/s to 420MB/s.

    Is that enough for two of the new 35 MB/s 1080i60 streams? Yes.
    Can FC 6.x do that without shuttering? No!

    Motion Can tho. :D So can QuickTime, iTunes, and a whole host of other editors.
     
  10. gertruded macrumors regular

    Joined:
    Jul 5, 2007
    Location:
    Northwestern Illinois
  11. screenwriting99 thread starter macrumors newbie

    screenwriting99

    Joined:
    May 7, 2009
    Location:
    Los Angeles
    #11
    Yes. Thank You Giffut - DDR3 Stacks

    You know a lot and I appreciate you and everyone else sharing your wisdom w me. I will take the advice on the drives. As far as the Ram goes, were you saying that filling in fours would be better....aka....2gb x 8 for 16gbs ram rather than getting 2gbs x 6 for 12gb Ram? I thought the memry was supposed to be faster in tripples not fours since it is DD3?
     
  12. Fomaphone macrumors regular

    Joined:
    Jan 11, 2009
    #12
    i routinely edit more than one stream of 4:2:2 1080p footage from an HVX, with FCP, and either motion or after effects and photoshop open simultaneously. no issues so far. my scratch drive is a software raid 0 of two 1tb disks. my system drive is a 1tb from apple.

    compressor can be a pain in the ass. but that's not news.

    i also output video to an external preview monitor. that's only slow in after effects. i'm getting 6x2gb from OWC as soon as i have some cash.
     
  13. screenwriting99 thread starter macrumors newbie

    screenwriting99

    Joined:
    May 7, 2009
    Location:
    Los Angeles
    #13
    Fomaphone...What Codec are you using?

    I will be using the new Final Cut Codec on the GY-HM700. It is apparently 35mbs a second with a quicktime wrapper. Just want to make sure were comparing Apples to Apples. Also ou said you stripped two drives in Raid 0. What brand did you use? Everyone keeps suggesting the Black Drives but I was considering a triple set of WD RE3 drives in Raid 0 3 TB total. Also did you just use osx?
     
  14. Fomaphone macrumors regular

    Joined:
    Jan 11, 2009
    #14
    software raid 0, WD digital caviar black.

    i edit in DVCPRO 1080p 24, prores 422, and uncompressed SD. i also mess around with 32-bit lighting in after effects and that's where i feel like i'd really see a big improvement with more RAM.
     
  15. LaDirection macrumors 6502

    Joined:
    Jul 14, 2006
    #15
    I don't understand why you would go to the trouble of getting all that pro gear just to edit what is basically a high end consumer camcorder that records at 35mbps. You could have bought 1 iMac with an additional FW800 external hard drive and it would have made a difference.
     
  16. Tesselator macrumors 601

    Tesselator

    Joined:
    Jan 9, 2008
    Location:
    Japan
    #16
    Define issues.

    It never stutters or drops frames with output to monitor?
     
  17. UltraNEO* macrumors 601

    UltraNEO*

    Joined:
    Jun 16, 2007
    Location:
    近畿日本
    #17

    Tesselator, m8. Why can't FCP do it?

    Personally I'm not a fan.. much prefer Premiere, just easier to use and more flexible IMO.
     
  18. Tesselator macrumors 601

    Tesselator

    Joined:
    Jan 9, 2008
    Location:
    Japan
    #18

    I haven't profiled anything to actually know where the bottleneck is. But 1080 i or p barely plays back smoothly with just one stream. With two streams it can play about 5 frames before it drops a few. Manipulating 2 streams smoothly? LOL! I guess not huh? In FCs2.2 playback of 1080 video maxes out a single CPU core so this tells me it's the application's fault and that to use such a poorly written app one would need a very fast processor - probably around 5 Ghz or more. Premiere on the other hand runs all 8 cores at about 60% or so. Much better but still pretty heavy.

    This is an 8-core 2.66 (X5355) with 12 GB RAM and a 3-drive RAID 0 that gets 250 ~ 400 MB/s. The video card is a 7300 so if that's the bottleneck I guess it's just me - tho I kinda doubt the video card has much to do with playback in FC 6.x

    FC/FCP/FCS is a dog. Premiere is better but still a little on the slow side when dealing with 1080p. I think to play with 1080 i/p at 30/60 (professionally) you need an OC'd processor and faster memory. For the popular Mac apps anyway. I know for example SpeedEdit on the PC can do everything FCP can do (and more) and I've edited 4 streams of 1080p @30 in real time very smoothly on a dual xeon 3.2 from 2001 or 2002 with 4Gig RAM and 4-drive RAID5. I guess it can do more than 4 too tho I haven't had the need yet.

    BTW, FC has nothing special so you're missing anything. Motion is good tho!
     
  19. screenwriting99 thread starter macrumors newbie

    screenwriting99

    Joined:
    May 7, 2009
    Location:
    Los Angeles
    #19
    No Realtime playback in anything?

    If this is true that I cannot get multiple 1080 streams to play back in realtime or play out in realtime on the Macpro Nehalem 2.66 Octo then why did I buy it? If FCP Cannot, or Premiere Cannot should I have bought a PC? Is there any card such as a Decklink (Any Version) that would give me the acceleration to make everything work in realtime multi stream HD? Also regrding why im using the aka Prosumer Camera? Because I am doing a quick and dirty documentary and that is all the budget allows. I think that because it records in Final Cut wrapped files it makes the most sense for speed of ingest to edit etc...
     
  20. giffut macrumors 6502

    Joined:
    Apr 28, 2003
    Location:
    Germany
    #20
    Ram

    First the terminology: RAM is called in "GigaBytes" (GB). Your camera has a throughput of 35 MegaBit per second (Mb/s). You notice the lower "b": because it means "bits" and not "bytes" (same goes for your cable/DSL/network connection). Now, one "byte" consists of 8 "bits". It is mostly agreed upon to use the capital "B" for byte, the lower one for "bit". Take your cameras specification of 35Mbit/s: divide the value with 8, you´ll get roughly 4.3MByte/s.

    The RAM technology behind the new Mac Pro (Intelks "Nehalem" processors") is somewhat different. It consists of a triple channel interface, let´s call it the three superhighways. Each highway consists of two lanes for each diraction, which go directly into your processor. Each of your two Nehalem processors got its own RAM banks/slots you see on the motherboard when you slide it out.

    Now, you have eight RAM slots, though. This is somewhat of a puzzle Apple implemented, because 8 is not a multiple of 3. Better would have been to use 9, e.g.; almost all other manufactures went with fitting their motherboerds with RAM slots in multiple of three. The triple channel interface gives you around 30% more memory speed than the so called dual channel interface. When you fill all of your RAM slots, you won´t be using triple channel, but dual channel, therefore you get lower throughput. Now don´t be scared: Using the dual channel mode you will get the exact same speed as all former Mac Pros had in terms of memory. Also you won´t feel too much of a difference with 99% of most applications, including the pro ones. If you use Motion or Adobe After Effects, though, this makes quite a difference.

    Here is a fine test sequence on the difference this makes: http://barefeats.com/nehal04.html

    Basically it comes to terms of your usage: Do you need 16GB of RAM, go for it. But I would suggest you start up slowly. You mentionned 12GB RAM you ordered together with the machine. And that´s enough for now. Because of the multiple of 3, you will be able to utilize the triple channel RAM mode and get superior RAM speeds. Like I mentionned before: Check and balance your needs and demands in comparison to your workload. I frankly don´t see any need for you at the moment.

    Stay cool on maximizing your equipment right now. You already have a killer system, which is way overboard for most tasks you are going to throw at it. I think your main interests needs to focus on training the system and a backup plan.

    You definitely will be able to replay several streams of XDCAM with your machine. XDCAM is rather new, and Final Cut Pro is scheduled to be completely revamped. This is a software issue, not a hardware one.

    Check Apples own recommendations regarding XDCAM: http://images.apple.com/finalcutstudio/resources/white_papers/Sony_XDCAMHD_whitepaper.pdf

    The files coming from your camera are NOT good for native Final Cut editing. You need to get the "XDCAM Transfer Software" from Sony, too, so the native files coming form the camera are rewrapped for XDCAM EX format Final Cut uses natively: http://www.sonybiz.net/biz/view/ShowContent.action?site=biz_en_EU&contentId=1166605189229 Another option would be to use the software "MPEG streamclip" (higly recommended: http://www.squared5.com/svideo/mpeg-streamclip-mac.html), open the clips and use the "Export to Quicktime" setting with the codec option "XDCAM EX 1080p" or accordingly. After the clips are rewrapped, they will play nicely with Final Cut.

    XDCAM is a highly compressed, long GOP format. It needs heavy transcoding to work with. This means, especially the software must be optimized for it, so all your available processing power can be utilized.
     
  21. Tesselator macrumors 601

    Tesselator

    Joined:
    Jan 9, 2008
    Location:
    Japan
    #21
    I tend to agree with this. After all I can have the quick-time player playing 4 streams without a hick-up. Lightwave3D or Motion can mix 4 or 5 streams and spit it out or play it back faster than realtime. Remember, my sentences were all based on if you're going to do it with the current version of FC. I have no idea if they're going to "revamp" it or not nor how successful that revamping will be.

    Currently if you want to start right off editing smoothly I think you need a Kona card. I have not used one but I've watched demos of it and it looks like a can-do solution. For video editing at such frame sizes, I use the Video Toaster on a PC. I'm mostly however, in applications like D'Fusion, Nuke, or Shake - or in the 3D apps like XSI, Lightwave, Houdini, and Maya.

    First, don't let anyone give you crap about your camcorder. That unit ROCKS! and is better than my $60K camera from just 8 ears ago. So, in their ear with a can of beer! :p

    PC or Mac.. I dunno. Whichever works I guess. I have everything I need already covered so I'm not looking to rebuild any pipelines - meaning I dunno what software works well on my Mac for 1080... I just know the newest Final Cut can't cut it. Neither can iMovie and neither can AfterFX or Premiere. Tho i have to admit I haven't tried AfterXF on Mac yet. On the PC Premiere is faster then AfterFX so I'm assuming the same for Mac. Anyway giffut seems to know what he's talking about on Mac regarding video which is more than I can say for myself (beyond my very limited tests) so I guess it would serve you well to consider his messages. I just wanted to answer your two questions in a very direct way so you wouldn't be too disappointed when you actually began. ;)
     
  22. MacUserPeggy macrumors newbie

    Joined:
    Mar 17, 2009
    #22
    If you wish use redundant RAID levels(like RAID 6) .I think this external hardware raid card is suit you.

    Is HighPoint RocketRAID 3522 OR RocketRAID 4322 Link to here

    http://www.highpoint-tech.com/USA/rr4322.htm

    http://www.highpoint-tech.com/USA/rr3522.htm
     

Share This Page