2.6GB/s (2600MB/s) RAID array transfer rate via Thunderbolt 2

Discussion in 'Mac Pro' started by FireWire2, Jun 18, 2014.

  1. FireWire2, Jun 18, 2014
    Last edited: Sep 2, 2014

    FireWire2 macrumors 6502

    FireWire2

    Joined:
    Oct 12, 2008
    #1
    One of mine client, who wants to editing 4K video with AVID.
    - less than 15ms latency
    - support 4K @ 50fps
    - Red camera raw footage
    Base on his calculation, he needs RAID5/6 thunderbolt solution that can transfer around 2600MB+ per second to his new MAC Pro for 4K editing.
    Note: There are 2x TB monitors connect to this nMP too

    Any Idea how to get this done?

    TIA

    UPDATE:
    Mission accomplished! Fastest Thunderbolt RAID (for now, lol)

    Since i do not know NLE process much, therefore I wont go there...:)

    But my customer - Burbank in CA, near Hollywood, he wants 2600MB/s (big B) and as much as storage that I can get for him.
    After weeks of struggle to come up with solution, last Friday we're able to get 2600MB/s consistently with RAID50 - (not RAID60) to new a MAC Pro using 2x Thunderbolt ports connect to two 12x drives thunderbolt raid boxes from DATOptic

    We tried Areca, Promise Tech thunderbolt 2 enclosures with the same setup, same HDD, where ARECA has a little bit of fan noise, but not Pegasus2 R8

    With 3x boxes of AREAC or Pegasus2 R8 as RAID50, we're able to get the goal transfer rate. But our goal is only use TWO TB2 ports

    Here come the rescue.
    I called DATOptic and told them the issue. They offer me 100% money back guarantee in 30 days, but I have to do all the tests, which is fine by me.

    As you can see the previous posted, the only combo that get us 1300MB/s plus is T12-S6.TB2 (including 2m TB cable) with 12x Seagate 4TB 5900 rpm HDD

    I guess 1TB plate does make the different.

    After debate and read through this post we decide to try RAID50/60.

    It appears RAID50 lot more evenly in both Read and Write. This make sense because RAID60 requires lots more I/O Processor resource

    The RAID5 is handle by the T12-S6.TB2
    We create 2x RAID5 form DATOptic boxes via WebGUI.

    The Web GUI exactly the same as Areca, turn out DATOptic use Areca RAID engine.

    With two volumes, we stripe these arrays in DU to create a RAID50
    [​IMG]

    One mistake we made at first, that is using Thunderbolt port 1 and 3.
    We can not get over 1400MB/s, turned out we connected to the
    same TB2 bus.

    Here is ref for Tbolt2 bus in nMP
    http://www.tekrevue.com/tip/mac-pro-thunderbolt-performance/
    And testing the hot plug of the unit, we simply move the cable from port #3 port to port #2 port. It takes about 30~45sec the volume appears back to the system

    No reboot required, even with 24x HDD and 8x fans are running, there is hardly fan noise, very quiet.

    Here are the test results of RAID50 with 88TB usable capacity:

    AJA

    [​IMG]

    Quick Bench:

    [​IMG]

    Avid Test

    [​IMG]

    Here are the test of RAID60 with 80TB usable capacity:

    Quick Bench:

    [​IMG]

    Avid Test

    [​IMG]
     
  2. Raddock macrumors member

    Raddock

    Joined:
    Jul 7, 2008
    Location:
    Houston
    #2
    Apparently you can daisy chain Lacie's drives to achieve 3000 MB/s transfer rates. https://www.lacie.com/us/products/product.htm?id=10623

    Scroll down to the info box where it shows them daisy chained.

    Looks like you might also be able to do the same with their SSD Raid system. https://www.lacie.com/us/products/product.htm?id=10621
     
  3. AidenShaw, Jun 18, 2014
    Last edited by a moderator: Jun 20, 2014

    AidenShaw macrumors P6

    AidenShaw

    Joined:
    Feb 8, 2003
    Location:
    The Peninsula
    #3
    Your client is hallucinating.

    High end enterprise systems on 24 Gbps SAS can hit about 2.1 * 10^9 bytes per second. RAID-0 striping across at least two 24 Gbps SAS links (and dozens of spinning drives on each link) can hit your client's perceived demand. Oops - no SAS on the new Mac Pro. Fail.

    A RAID-0 across a pair of top-end PCIe SSDs could hit it too - but the new Mac Pro doesn't have PCIe slots. Fail again.

    First step - have your client work with Red and Avid to come up with realistic requirements.

    Second step - find a dual socket workstation with 80 PCIe lanes for your client.

    That 3GB/sec is aggregate to multiple RAID arrays, not the speed to a single array. If you could do software RAID-0 in the host across the arrays you could get the aggregate to a single filesystem, but at the expense of reliability.
     
  4. beaker7 macrumors 6502a

    Joined:
    Mar 16, 2009
    #4
    50fps at 4k in .r3d, even at 6:1 is only 97 MB/s.
     
  5. Pressure macrumors 68040

    Pressure

    Joined:
    May 30, 2006
    Location:
    Denmark
    #5
    Honestly, I'm scratching my head as well.

    Why would he ever need 2.6GB/s transfer rates?

    REDCODE offers the following compression rates (5K RED Epic).
    18:1 = 1.48 GB/Minute
    15:1 = 1.77 GB/Minute
    10:1 = 2.65 GB/Minute
    8:1 = 3.34 GB/Minute (Default setting)
    5:1 = 5.26 GB/Minute
    3:1 = 8.70 GB/Minute

    Even at 3:1 (23.98fps) you only need around 150MB/s sustained per clip.
     
  6. FireWire2 thread starter macrumors 6502

    FireWire2

    Joined:
    Oct 12, 2008
    #6
    Thank you all for the inputs. Appreciated very much

    Pressure
    I'm an IT guy.. don't know what he wants to do.. but as far as clients concern, he is paying me to make it...I will try my best to earn that $$$ :)

    Raddock
    Lacie solution as you suggested, does not work, Thunderbolt 2 transfer rate is cap @ 1400MB/s

    AidenShaw
    Thank you for your suggestion, I will try a RAID60 and RAID50. As of now I can get 1300MB/s to 1400MB on RAID6/5 respectively
     
  7. FireWire2 thread starter macrumors 6502

    FireWire2

    Joined:
    Oct 12, 2008
    #7
    1400MB Read/Write Thunderbolt

    Update:

    I'm able to get:
    1300MB/s RAID6 and 1400MB/s on RAID5 with new MAC Pro

    It's not where my client wants
     
  8. beaker7 macrumors 6502a

    Joined:
    Mar 16, 2009
    #8
    Did your client ask you to build a machine with storage speeds 25 times faster than they need?
     
  9. FireWire2 thread starter macrumors 6502

    FireWire2

    Joined:
    Oct 12, 2008
    #9
    Yes, he wants an array @ 2600MB/s to his new MACPro. He has money to burn :)
    We all know about "WANT and NEED", right?
     
  10. beaker7 macrumors 6502a

    Joined:
    Mar 16, 2009
    #10
    Sure, but you said that's what he calculated he NEEDS in order to edit that footage, which is demonstrably incorrect.
     
  11. blackwoodfx macrumors member

    blackwoodfx

    Joined:
    Nov 15, 2010
    Location:
    Las Vegas, NV
    #11
    RED user and camera shop here. We have multiple RED Dragon (6K) cameras and I am very familiar with this problem. Your client is correct to want 2600 MB/s RAID performance. Editing 4K and working in RAW (6K) requires monstrous bandwidth. 1000 MB/s will not provide real-time continuous playback and edit performance. TB2 and the nMP is hopeless overall. TB3 will finally get us to what we need. Tell your client to either down step to a full-size MP with SAS to RAID 50 (recommend JMR Silverstor) and then wait for TB3 in x-years.
     
  12. FrankHahn, Jun 23, 2014
    Last edited: Jun 23, 2014

    FrankHahn macrumors 6502a

    Joined:
    May 17, 2011
    #12
    The transfer limit of Thunderbolt 2 is 20 Gbps/8 = 2500 MB/s. Thus, a single thunderbolt 2 connection can't achieve 2600 MB/s.

    One suggestion: Get two Promise Pegasus2 R4's. Put four Crucial M550 (or another brand) 1T SSDs into each Promise Pegasus2 R4 (total 8 SSDs). Do not daisy-chain the two Promise Pegasus2 R4's. Connect them separately to two different TB ports on two different controllers on the nMP. Use the JBOD mode on each Promise Pegasus2 R4. If you can make a RAID0 out of these 8 SSDs, you can then achieve 2500 MB/s or above.

    You may also consider using two LaCie LBD Thunderbot2's. Do not daisy-chain them. Connect them separately to two different TB ports on two different controllers on the nMP.

    I plan to make a RAID0 or a RAID5 using four 512GB SSDs and a Promise Pegasus2 R4. I have got 4 SSDs (Toshiba Q Series Pro 512GB). But I am still waiting for the diskless Promise Pegasus2 R4 from the Apple Online Store.
     
  13. CASLondon macrumors 6502a

    Joined:
    Apr 18, 2011
    #13

    What he says......

    although, I've been intrigued by the various upgrade options these guys have put together for the classic mac pro..if correct, that gets you near 2g rates

    http://www.maxupgrades.com/istore/index.cfm?fuseaction=product.display&product_ID=354&ParentCat=414
     
  14. macs4nw macrumors 601

    macs4nw

    #14
    Not to be nit-picking here, especially when it comes to theoretical maximum transfer speeds, but aren't you forgetting a zero there…..?
     
  15. Pressure macrumors 68040

    Pressure

    Joined:
    May 30, 2006
    Location:
    Denmark
    #15
    He is indeed.
     
  16. FrankHahn macrumors 6502a

    Joined:
    May 17, 2011
    #16
    Sorry, I did loose a zero! I have made corrections! Thanks!

    ----------

    Corrected!
     
  17. chfilm macrumors 68000

    chfilm

    Joined:
    Nov 15, 2012
    Location:
    Germany
    #17

    It should work, otherwise Lacie wouldn't quote it on their website.
    I also saw a video of multiple Lacie Little Big disk2 s software raided together and it was faster than 1400 mb/s.

    ----------

    Hey, quick question: How is it that we need such a HUGE bandwith for dragon footage? Is the compression that much different from old EPIC footage?
    I was always wondering, if the data rate of red/epic footage is so low on paper as it has been posted in this thread, where lies the bottleneck in my station? Is it the hard drives or really just processing power?

    I've got a nMP with 8 cores and dual d700 connected to a Pegasus1 R6 with 12TB and am not able to playback epic footage at 1/2 without caching for a few seconds in redcine. Would I need a red rocket in order to play it back smoothley? Or is the pegasus 1 too slow with it's 680 mb/s read?
     
  18. TzunamiOSX macrumors 6502

    Joined:
    Oct 4, 2009
    Location:
    Germany
    #18
    Yes, on 3 different TB2 connectors you can get 3x1350 MB/s

    A real test demonstrate, that one TB2-Bus is limited to 1350 MB/s

    http://barefeats.com/hard186.html
     
  19. chfilm macrumors 68000

    chfilm

    Joined:
    Nov 15, 2012
    Location:
    Germany
    #19
    I see! Thx
     
  20. deconstruct60 macrumors 604

    Joined:
    Mar 10, 2009
    #20
    Are you sure your client knows the difference between 2600MB/s and 2600Mb/s ? That the capital letter makes a difference. Folks with more money then sense may not know the difference. If they don't know that, part of being a professional is to tell them.... not to drain their bank account on ignorance.
     
  21. deconstruct60 macrumors 604

    Joined:
    Mar 10, 2009
    #21
    Again this teeters between "want" and "need" but 2 Thunderbolt monitors why? If client wants to maximize available Thunderbolt bandwidth then dropping those will help. If "require" that the monitors are dropped on a very long cable away from the nMP perhaps. "needs" the extra USB/FW/Ethernet ports ... perhaps ...

    But if 'sunk costs' in TB monitors already .... if they have money to burn that shouldn't be a problem.


    Likewise everything bound up in a single RAID 6 imagine.... requirements or simplification?


    As a storage bus the TB v2 controllers are pragmatically about 1500MB/s conduits ( ~ 3x PCI v2 *** ) . Still have to trunk down to x4 PCIe v2 implementations to get on/off the Thunderbolt network. The upside is that there are three of busses in Mac Pro 2013. 3 * 1500MB/s ( 4,500 MB/s) means it is possible. It just isn't going to be some fixed, turn-key solution.

    What looking for a RAID 0 solution layered on top of 2 (or 3 ) RAID-6 subsystems. Layering RAID 0 on top of virtual disks that are extremely unlikely to fail is a relatively very small additional risk.

    If "require" a "one wire/cable" solution then no; there isn't one.


    There is more than a little amount of self-inflicted pain here in editing directly in RAW 4-6K files here. There is gobs of redundant data being dragged across the storage subsystem probably for not alot of real necessity.




    *** While the Thunderbolt v2 data bandwidth is 20Gb/s the bandwidth of PCIe specific data is not the same thing. Thunderbolt data packets carry PCIe data but they are not the same thing or bandwidth. To any one node on a Thunderbolt network the transmission speed to the host PCIe network is limited by the PCIe switching limits present in all TB controllers. All the data on a TB v2 doesn't have to all be bound for just one node on the network.
     
  22. deconstruct60 macrumors 604

    Joined:
    Mar 10, 2009
    #22
    http://www.areca.us/products/thunderbolt2.htm


    There is a Fail here ..... just not on the Mac Pro side. There is huge difference between there not being a being able to order directly from Apple's store inventory and there being no SAS solutions.

    Will need more than one, but since this thread is an "unrestricted budget" exercise... that isn't an issue.
     
  23. FireWire2, Jun 23, 2014
    Last edited: Jun 23, 2014

    FireWire2 thread starter macrumors 6502

    FireWire2

    Joined:
    Oct 12, 2008
    #23
    Fastest Thunderbolt RAID

    Just an update:

    We'd test ARECA, Promise tech Tech, and DATOptic thunderbolt RAID enclosure

    So far the only ONE enclosure that can delivery consistently over 1300MB/s

    We are going to create a RAID50/RAID60 out of 2x identical RAID volume

    I think we can get more 2600MB/s
    We are waiting for the 2nd unit coming this week

    Can not wait
     
  24. TheHerdForever macrumors member

    TheHerdForever

    Joined:
    May 11, 2012
    #24
    My thoughts exactly. Personally, I thought his client was doing drugs.
     
  25. FireWire2, Jun 23, 2014
    Last edited: Jun 23, 2014

    FireWire2 thread starter macrumors 6502

    FireWire2

    Joined:
    Oct 12, 2008
    #25
    That was my original though, but after he ran some latency test from Sony and AVID...
    @ 1300MB/s it supported 2K only... I guess he does not use compress codec

    I guess he is right
     

Share This Page