Want storage solution 10-15TB

Discussion in 'Mac Pro' started by Varmann, Mar 23, 2010.

  1. Varmann macrumors member

    Joined:
    Jan 3, 2010
    #1
    I am interested in getting a better storage solution than I have today,
    but I am a bit confused about viable alternatives.
    I am also a raid newbie (I have so far regarded raid as
    "yet another layer that can go wrong").

    I want:
    - Storage solution for that can handle >10TB data
    - Possibility to expand up to 15TB
    - Security is a priority, with a possibility for off-site backup.
    - No real need for high bandwidth
    - Budget $3000-4000 (excluding discs)

    What I have today:
    MacPro 2x2.8 '08, 16GB ram
    Internal: 1 Intel SSD boot, 5 x 2TB WD (green non-raid) in single disk setup
    Sonnet Tempo E4P Esata card
    External: 2 Esata cabinets (Sonnet+ Firmtech) each with 5x2 TB disks JBOD.

    Some of my thoughts:
    Using 2, or even better 2x2 , external sata cabinets for backup.
    Some sort of raid solution as main storage on my Macpro, with a capacity
    equal to 6x2TB. Using an external cabinet seems necessary(?).

    I would be very happy for suggestions and advice!
     
  2. norbiegee macrumors newbie

    Joined:
    Mar 23, 2010
    #2
    Cheap Storage

    Check out Data Robotics.
    Drobo.com

    My company just bought a second unit. You can use your SATA drives in any combination. Works well and its is really, really cheap.

    Total unit with 16 TB storage (Drobo Elite with 8 2TB Drives) was about 4200.00

    2 iSCSI ports.

    If you want Firewire 800 go to a DroboPro and that is a bit cheaper.
     
  3. sgunes macrumors member

    Joined:
    May 25, 2009
    #3
    Look also into building a custom storage/media server.
    The best bang for your buck case is the Norco 4220 with 20 hotswap 3 1/2 bays. It costs about $350 +S&H at newegg and $300 + S&H at mwave.
    I built a media server with it using left over motherboards and CPU. You can use any operating system from Ubuntu server to UnRAID or even Windows :)eek:).
    I filled mine up with 20+3 2TB drives (WD GP and Seagate about 10 each).

    More info about these media servers at avsforum.com here and here.
     
  4. nanofrog macrumors G4

    Joined:
    May 6, 2008
    #4
    Use a RAID card for the primary storage, and eSATA + PM enclosures for backup.

    For the RAID (capable of containing a sufficient level of storage), I'd go with:
    ARC-1680ix-16
    External 8 bay enclosures (MiniSAS connections, and they come in silver too)
    Cables (need 1x per port used on the card, so 4x if you use 2x enclosures)

    Use enterprise HDD's, as it's a SAS card. To find those that will work, check the HDD Compatibility List. Personally, I've had good luck with WD's enterprise drives, and they work with this controller.

    By going with this route, you can implement RAID (more than a stripe set), as it sounds like you need redundancy (you won't lose all the data if a drive dies). There's multiple levels, and as a bare minumum, you'd want to go with a RAID 5 (can tolerate the loss of a single disk), though a RAID 6 may be more suitable (allows the failure of 2x disks and still retain data).

    For a RAID 5, you'd need to run a minimum of capacity requirements + 1 additional disk (used for parity data). For RAID 6, it's 2x additional disks.

    So to start out with 12TB (using 2TB drives):
    RAID 5 = 7 drives required
    RAID 6 = 8 drives required

    BTW, the hardware listed (not including disks), is under your budget.
     
  5. psingh01 macrumors 65816

    Joined:
    Apr 19, 2004
    #5
    HP Media Smart server is a cheapish NAS device. 4 SATA HD bays and I think you can connect external storage sources to it too. It uses Microsoft Windows Home Server as it's OS.
     
  6. belvdr macrumors 603

    Joined:
    Aug 15, 2005
    #6
    I've used Nexsan without issue. They've been great. Of course, these are rack mount units, but I tell you the SATABeast is an awesome device!
     
  7. codymac macrumors 6502

    Joined:
    Jun 12, 2009
    #7
    Any comments as to how loud that case is? I've been wanting to build up a block-level solution but noise has consistently been a stumbling block with large storage.
     
  8. Varmann thread starter macrumors member

    Joined:
    Jan 3, 2010
    #8
    Thank a lot everyone!
    Lots of nice ideas!

    That was one of my first thoughts, but I have not yet decided what route to go. I got some more alternatives to investigate.

    Is it possible with the areca card to combine the disks in that external cabinet (8) with the internal disks (4) to get a 12 disk raid 6 setup?
    (I have understood from other threads that you can connect the internal disks to a sas raid card)

    What is the best way to arrange the cables from the internal sas-connecters to the external cabinet. As long as you have one expansion slot free I see a possibility, but if that one gets occupied, do you have to drill extra holes?
     
  9. sgunes macrumors member

    Joined:
    May 25, 2009
    #9
    Everyone replaces the built-in screaming Delta fans with quiet fans. I ordered 7 or 8 Rosewill 80mm fans with the best cfm/dB ratio from Newegg for about $2-3 each. I have the server case in my study (without any enclosure) and it is not any louder than the other computers in the room. Somebody also made a custom shroud to replace 4 80mm fans with 3 120mm fans.
     
  10. hyram macrumors regular

    Joined:
    Jun 15, 2009
    #10
    nanofrog has the right solution for RAIDs the size you're looking at. But I'll add a few comments:D

    While the Areca cards are good, there are others that work as well. The Highpoint 4322 has served me well for about 18 months. Down side is that their customer service is lacking. Alto makes good cards as well, but $$$. I like the idea of CalDigit but they seem to be twitchy.

    As for SAS drives, get ready for sticker shock. Good SATA drives can be just as reliable with good thruput without the need sell a kidney.

    There's a question you need to answer before you pull out the checkbook: Do you really need that much storage? The current technology trends kind of dictate that things will drasticly inprove within about 5 years. If you plan for more storage than you need in that time frame, you're wasting money. 10TB is a lot...

    Good luck!
     
  11. Repo macrumors 6502a

    Repo

    Joined:
    Feb 11, 2009
    #11
    Someone already mentioned Drobo (Data Robotics) which is a fantastic option. If you're looking for something "Mac themed," then check out what G-Technology has to offer.
     
  12. tomllama macrumors regular

    Joined:
    Jan 7, 2007
    Location:
    CA
    #12
    If speed is not a great requirement, you can save a lot by not doing a RAID.

    I have a FirmTek/SeriTek eSATA card and 5PM enclosure now. 5 drives @2 TB each allows your 10 TB minimum and you could add 2 5PM enclosures. It is 'cheap', allows software RAID, and is easy but does not work with Windows (via boot camp - it gets used just fine with Parallels). It can be shared by other Macs on the network too.

    I added an HP Media Smart Server (EX495) yesterday to make it easier to share across the network and between OSX and Windows. You can load 4 internal drives into it and hang a PM box off it via eSATA to get 18 TB pretty easily (and additional USB external drives are possible too if you need/want them). With a gigabit connection it's not bad speed-wise (but it's far from RAID speed). The EX495 cost less than a good RAID card.
     
  13. deconstruct60 macrumors 604

    Joined:
    Mar 10, 2009
    #13
    What is the RAID 5/6 rebuild time going to be if using 2TB "blocks" ?
    Using largest disks possible as RAID 5/6 building blocks lends to the longest rebuild times (assuming the disk bandwidth is fixed). That will be the time will the RAID is at reduced protection capacity. It is roughly going to be about the time to zero and initialize a 2TB disk. That takes a while.


    The other issue with RAID 5/6 is that you do not want to expectantly loose power. They both require writing a logical data block out out multiple disks to complete as a transaction to have a coherent representation of the data.


    If don't need continuous 10-12 TB of disk space ( one logical volume) then multiple external enclosures should work. Not sure it would be wise to use the internal drives to be members in a logical volume with the external ones. The other issue is that sending all of the your data through one PCI-e slot. You don't care about speed (some kind of online digital archive perhaps), but also have a single point of failure.

    Again if speed isn't an issue can use one of the two Etherports as a dedicated iSCSI connector ( a modest 4 or 8 way Gigabyte switch to 4 to 8 storage units and wouldn't consume a PCI-e slot ). As long as only need Gb/s amounts of data all of it on that single pipe will work out. Additionally, backing up 10TB is going to be a pain (and again easier if not one huge logical volume. )
     
  14. nanofrog macrumors G4

    Joined:
    May 6, 2008
    #14
    Yes, this is possible.

    But to use the HDD bays, you'd need an adapter sold at one place, MaxUpgrades (here). Tad pricey, but it works.

    Run them out of a PCI bracket if at all possible. If however you do have all the slots filled, then you would have to modify the case (cut or drill holes to fit the SFF-8087 connectors through).

    Areca is the ODM for Highpoint's RR43xx series. Unfortunately however, Highpoint doesn't have the best support out there (ask those that have one how difficult it's been to obtain the EFI boot portion of the firmware). :( Nor do they include internal cables (SFF-8087 - 4i*SATA). This can up the cost in the end (typically go for ~$30USD each, so for a 12 port card, there's ~$90USD worth of internal cables in the box).

    I also like the fact you can upgrade the cache via the DIMM socket (watch the ranking though, as it needs to be 8 or 16 ranked; 4 won't work).

    Fortunately, you don't have to use SAS disks. :) SATA will work (much cheaper, and come in larger capacities), but you do have to watch the HDD Compatibility List, as SAS cards are picky, and the cable length has to be kept to 1.0 meter (so it can be a PITA for placement of the enclosure).

    Typically only enterprise versions will work, though with WD brand, you can "cheat" and use the TLER Utility on the consumer lines to change the timings for RAID. Great for backup arrays and/or tight budgets (but I wouldn't recommend this for a primary array).

    It would make sense if the usage is for video/graphics work (they burn through capacity like mad as I understand it).

    Otherwise, it is definitely in the OP's interest to carefully examine the needed capacity requirements.

    ~1hr per TB with RAID 5 (128k stripe size). RAID 6 is going to be a bit longer (1.25hr / TB or so IIRC). With recent drives anyway (i.e. WD RE3 & Seagate ES.2's, ~1.5?yr old models now).

    Definitely not.

    A UPS is a necessity with parity based arrays rather than an option. The card battery is a good idea too, but it has limitations (i.e. cache can't contain all of the data), were a UPS can do better, assuming it has adequate run time. Ideally, users should run both.


    It works, and you can still recover. Areca's even have a hidden recovery function that allows arrays that would otherwise be toast to be rebuilt. There are limits of course, but it's better than other makes out there in my experience.

    At this point, you're getting in to a second card and identical array. Much more fault tolerant, but tends to be on the really expensive side (I typically only see this on servers, not workstations). ;)

    So this is getting into SAN territory (i.e. commonly found using FC based gear).

    Possible, as is ATA over Ethernet (if network access isn't necessary). Either way, you're dealing with a storage server independent of the system. Speed can even be assisted by teaming the Ethernet ports.

    Something the OP could think about though. :)
     
  15. frimple macrumors 6502

    Joined:
    Nov 18, 2008
    Location:
    Denver, CO
    #15
    Let's define what you're looking for a little bit better shall we? Do you want some kind of direct attached storage or are you looking for a SAN/NAS solution?? If you are (and at this capacity) have you considered the networking implications of this kind of setup?

    Assuming you want a SAN setup, it's really hard.... REALLLY hard to beat the price of a home built storage solution. Using your budget of $3-4k (without drives) I was able to build a server with these specs for $3,200 w/o drives:

    (I'll give you a newegg wishlist link if you want)
    16 drive chassis (3.5" bays) though it is a rack mount chassis...
    Tyan mobo (with dual gig nic's of course!)
    2 quad core 2.0 AMD processors
    16 GB memory
    Areca 1261 controller
    2x Intel X-25E SSD's for OS

    So if you spec'd this with 16 2TB drives (WD Blacks @ $279 each) you'd have about 22TB of usable space, and the total cost would be ~$7,800.... ****! :eek:

    That's damn cheap in the world of storage!!! But of course the problem would become how would you connect to it. There are readily available SAN linux images out there on the web, and with the Areca handling the heavy lifting for you, you could use the server hardware for something practical (VM Server??).


    Anyways, at the price point you're looking at you really need to define how big you want to go. Keep in mind you're still going to have to back all this data up.. nevermind the maintenance cost (failing drives/hardware) and YOUR time when something goes wrong and you're trouble shooting it. I often see pro-amatures fall into a gap where they can afford the initial cost of hardware but fall short on the back end with maintenance and upkeep. The last thing you want is to pay the contract rate of someone like me to tell you that you need to replace X, Y and Z.

    What are you going to be using this storage for? What do you really need out of it?
     
  16. nanofrog macrumors G4

    Joined:
    May 6, 2008
    #16
    Somehow, I've the impression the OP is after a DAS system. Nice setup though, and as you say, it's a great system for the $$$.

    Very good point. Such a system can become a nightmare if the administrator doesn't know what they're doing.
     
  17. frimple macrumors 6502

    Joined:
    Nov 18, 2008
    Location:
    Denver, CO
    #17
    Yes, absolutely. But when you start talking about storage above 4TB I think NAS/SAN should be considered, even preferred to DAS. All that data for just one workstation? Such a waste ;)


    Yes the upkeep and administrative costs of a system like this are by far the most overlooked costs. God forbid you get a bad lot of drives and end up replacing 50% of them in the first year. Sure they won't cost anything to replace (except shipping maybe...) but just think of the time in array rebuilds!
     
  18. nanofrog macrumors G4

    Joined:
    May 6, 2008
    #18
    :eek: Never! :D :p

    Seriously though, it really does depend on whether or not the OP is an idependent or in an environment that it can be shared with other users. In the latter case, SAN is definitely the way to go, as it's cheaper.

    Imagine DAS of this magnitude for say a 50 systems (single site). :eek: The cost is prohibitive. SAN certainly makes sense in such situations. But without further information, I'm presuming it's a single system at this point, so SAN or NAS isn't the best solution IMO (use the horsepower already owned with a little help from Areca ;)).

    Especially with consulting rates ~$180USD avg. or so.

    Bad lots and sequential serial no's are a massive No-No. Rebuilds from hell, and a serious chance the entire array will be toast (or sets that contain too many members). Gets seriously expensive, and FAST.
     
  19. VirtualRain macrumors 603

    VirtualRain

    Joined:
    Aug 1, 2008
    Location:
    Vancouver, BC
    #19
  20. codymac macrumors 6502

    Joined:
    Jun 12, 2009
    #20
    Thanks! My environment would be similar so I appreciate the info.

    Yup. It's also really hard to beat because there's a product/feature/price gap that scratchbuilding slots right into.
     
  21. Varmann thread starter macrumors member

    Joined:
    Jan 3, 2010
    #21
    Ok, to further explain my needs:

    Today, I have only 500GB left on a 5 x 2 TB disk set. So I certainly need more space.

    I am a dedicated film lover, with a collection of 1200 dvd's, which I have now ripped (yes, I want the original files on disk as well!). I also have transferred my 1500 cd's and scanned 5000 colour slides, but those are minor items, spacewise. I am creating a "media heritage" of a 30-year collection, which I will display in a "poor-mans-Kaleidescape"-setup.

    Back to the thread - I am very tempted by Nanofrogs raid solution - having that raw speed at my fingertips would really be something!

    But, do I really need it?
    To be honest - No!

    I will have a single user setup.
    I guess I need:
    - 1 main storage with some fault tolerance.
    - 2 switchable on/off-site backups (I guess JBOD in Esata PMs might be enough here).

    Looking at suggestions here, the 8-disk DroboPro seems like a suitable main storage, with the advantage that it may be put in another room to reduce noise. Minimum setup trouble is always nice. However, I am bit bothered about how "wellproven" this system is.

    Some more questions:
    - For a Drobo setup what is most suitable, enterprise or consumer drives (I will buy the WD brand)?

    - Should I use/wait for 4k sector disks, to get some more space?

    - I am lucky to live in an area with very stable electricity (one power failure every two years) so I have so far only used spike protection. My thought was that a UPS with 10 min power, would not help much if something happened during a big disk copy during night. Should I rethink this?


    Thanks everyone, I am very impressed by the knowledge contained in this forum!
     
  22. sgunes macrumors member

    Joined:
    May 25, 2009
    #22
    It still looks like you are looking for a media server. I did the same from DVDs to BluRays/mkv's, from pictures to home videos.
    Read my post above.
    Go to the avsforum.

    Rule 1: You can never have too much storage.
    Rule 2: You will outgrow your storage much sooner than expected.

    The most recommended cases are the Supermicro with 24 bay hot-swap drives and (cheaper) the Norco with the 20 bays.
    Go big now and you won't regret it later.
     
  23. nanofrog macrumors G4

    Joined:
    May 6, 2008
    #23
    In your case, the speed would have been a nice side effect, as you desire some degree of fault tolerance.

    Given the capacity needs, 10 isn't the best solution IMO (too many drives needed, as you can only use 50% of the total drive capacity). That leaves parity RAID (5 or 6, depending on the tolerance level you wish).

    But this means a proper RAID card, and a UPS at a bare minimum before drives in order to deal with the write hole issue (even though most of your usage will apparently be reads, you don't want the data to be corrupted during the write process).

    BTW, there are 24 port cards to go with 24 port enclosures (it does seem like you've a lot of potential to burn through capacity, and rather quickly). Areca's ARC-1680ix-24 is the same card I linked previously, but with additional ports. ;)

    The link you gave is a Media Server. But as you're only using a single system, you don't actually need it (not serving mulitple locations, so you don't need NAS).

    However, if you plan on doing this in the near enough future, it would be best to go ahead and go with NAS to reduce your costs later.

    They're software based, and aren't recommended for parity arrays. Even with a UPS, I still don't trust them (heard of some bad experiences over this).

    I prefer to build a separate system and use a hardware controller for parity based NAS, as frimple suggested. Cheaper, and customizable compared to a commercially available product in the same budget.

    But again, you'd have to clarify if there's a need for NAS in the future (reasonable), or if it will remain a single system.

    Drobo's are typically used with consumer models (lowers costs), but it won't hurt to use enterprise drives (built to be more reliable.

    Even a DIY NAS (purely software) typically uses consumer models to lower costs as well.

    No.

    Yes. You may not be getting power outs, but you are likely getting brown-outs (low voltage, but not a full power outage; i.e. flickering on incandecant bulbs is one indicator). Brown-outs are actually more dangerous, as they can damage electronics (cumulative as well, so multiples tend to be the cause). And no power grid is immune to this phenomena.

    A surge suppressor does nothing for this type of condition, but a UPS will (maintains a minumum voltage). There's different types, and it's reflected in the cost (there are articles that help explain them, so it's worth a search; this link will hopefully get you started)
     
  24. vvrinne macrumors member

    Joined:
    Feb 28, 2010
    Location:
    Helsinki, Finland
    #24
    I would look at QNap for a fast stable NAS solution. Been running one of their 5 disk boxes for a couple of years now and I can't really find any fault with it. The 8 disk models would probably suit you, you could get up to 14GB fault tolerant with 8 x 2GB.
     
  25. Varmann thread starter macrumors member

    Joined:
    Jan 3, 2010
    #25
    Thanks nanofrog, for the info about UPS, I surely will get one now.
    Living in an area that has a major thunderstorm every five years make you feel falsely secure.

    I do not need a NAS, and I can't see any demand in the future.
    I have also actively tried to reduce the numbers of computers in my apartment in recent years from five (mac/win/linux) to one (mac). In addition, I am now moving to a smaller apartment, so space is much more an issue than "budget". Otherwise, I certainly see the advantages with a separate mediaserver!

    I am moving towards this solution:
    - UPS
    - Areca sas-card
    - 12-disk raid6 (4 internal + 8 external), with possibilities to expand.
    - separate bootdisk (understood thats is possible by freeing up 4 internal sata-ports in the '08, when using a sas card)
    - 2 sets of PM esata-enclosures for backup, one off-site.

    That will give me 20 TB expandable raid6 storage, with 2 backups.
     

Share This Page