PDA

View Full Version : Building Remotely Accessible 30TB Server




mackpro
Apr 9, 2011, 06:39 PM
Hello kind folks of MacRumors.com-

I am wishing for help and advice deciding what type of hardware to purchase for a 30TB server. It must meet the following requirements:

• 30TB RAID (Redundancy important for backup purposes)
• Remote Accessibility in that I would like to be able to pull files from it anywhere that I have an internet connection.
• Macintosh friendly– it does not need to run OS X Server, it just needs to connect without hassle to Macs.

This will be used for at my office for archiving video. We may want to use Final Cut Server to sort the files in the future. I have been tasked with building this setup and am hoping for suggestions from one of the brightest communities on the web. :D

Your time in responding is much appreciated.

:apple:



assembled
Apr 10, 2011, 02:31 PM
You might find it easier to split this into two projects, one for the server and RAID storage, and a separate one for remote access. I would advise against trying to get a single device to do both.

The use profile and availability requirements for the server and storage (online use, nearline storage or archive storage) will determine what is required, as a for instance, you might go for 16 2TB disks in a single controller RAID5 array connected to a single server running SMB or NFS for archive, whereas for online, you might go for a clustered server solution with 34 1TB disks in dual controller RAID6 array with 2 hotspares.

For remote access, how will remote users be authenticated ? what bandwidth will they have available at their end and at the server end ? how many concurrent users will there be ? what availability do you need for remote access

There are two main types of remote access to files. The traditional method is network level access over either PPTP, L2TP, IPSec or SSL. The other method is to access files through a HTTPS server.

For network level access I would probably use a Cisco ASA device, which could be clustered with a second one in an active/passive failover cluster if you have high availability requirements. I would use SSL rather than IPSec for remote VPN users as IPSec can have issues with some NAT firewalls (home routers) whereas SSL works over just about everything, also as long as you have an LDAP authentication back end (such as AD or, OD) having different groups of users having different access rights can be much simpler to maintain.

For web browser access I would probably go for the Barracuda SSL VPN device.

paduck
Apr 10, 2011, 02:39 PM
The easy part is going to be to connect it to the Internet. Basically, you just need a router that will port forward or let you login to a Virtual Private Network (VPN). Then it will be just like being on your LAN when you login remotely. Given that you are talking about hosting video, you might want to look at what the Internet connection's upload speed is and plan accordingly.

On the server side there are a lot of options. It always gets said, so I will say it first - RAID is not about backup. It is about availability. You need to have a backup strategy and RAID isn't it. That is going to add to your costs since you are talking about a lot of data.

In terms of the array, The only way to get to 30TB realistically is some sort of RAID. You probably are going to want something like RAID5 or RAID6 which will allow for one or more drive failures without losing the whole array. Something to consider - to get to 30TB under those circumstances you are going to need upwards of 40TB of hard drives. So your hardware is going to have to support that. For reliability, you might want to consider buying enterprise class drives.

You could get some sort of rack-mounted array that can support 48+ drives (assuming a bunch of 1TB drives in the array, although you could go with 2TB drives as well). You can link several smaller cases together and connect them to a Mac Pro with additional hardware that supports RAID. That assumes you want to go the Mac route for the software side. With Lion merging the serve OS that could make things a lot easier.

How many users are you planning on supporting? Are you planning on doing more than just hosting video files? Email? Calendar? Web services? (what sort of Internet connection you have plays in here as well). If it's just a couple users, your server probably doesn't need to be a high end Mac Pro. Still, look at what you are buying. This could easily be a $25,000 plus project. And don't forget the whole backup strategy which can be expensive in terms of equipment and more expensive in terms of disciplined time investment.

I've mentioned the Internet connection a couple of times as well. Don't forget the LAN configuration either. You are going to want a Gigabit switch that can support multiple users and lots of data. A second Ethernet port on the comput for both more speed and redundancy is probably a good idea as well.

All of this may mean that it is worth some investment with a professional to design the system and provide you with a maintenance contract and support for the hardware and software. In a business environment, that might be worth the investment.

Apple doesn't have Xserve anymore, but they may have something else coming down the pike (don't hold your breath). That might be a good option. Otherwise you can always go with Linux or Windows - but if all your other equipment is Apple, I'd try and stick with the all-Mac configuration to keep things simple.

Good luck, sounds like an interesting project. I think you'll learn a lot.

Nuc
Apr 10, 2011, 07:11 PM
I've been looking at the same setup but start off with maybe 8Gbs and expand from there. The following website is what I've been looking at: http://lime-technology.com/

I'm looking at the server that has 12 bays. I don't know how all this stuff works yet...but I'm learning. There is also a forum that has a lot of good info.

Keep us posted on what you decide.

monokakata
Apr 10, 2011, 08:26 PM
I'd suggest contacting a large post-production house and seeing if they'll show you how they do it.

My son works at a big place in NYC and I've seen their machine room (mostly Apple). His outfit does what you're talking about, 24/7, with access all over N America and Europe, both for the editors and their clients.

I would suppose that any good-sized house does the same, and since you're specifically doing video -- well, that's what the post houses do for a living.

Sometimes imitation is an excellent solution.

Matty-p
Apr 11, 2011, 03:45 AM
Hello kind folks of MacRumors.com-

I am wishing for help and advice deciding what type of hardware to purchase for a 30TB server. It must meet the following requirements:

• 30TB RAID (Redundancy important for backup purposes)
• Remote Accessibility in that I would like to be able to pull files from it anywhere that I have an internet connection.
• Macintosh friendly– it does not need to run OS X Server, it just needs to connect without hassle to Macs.

This will be used for at my office for archiving video. We may want to use Final Cut Server to sort the files in the future. I have been tasked with building this setup and am hoping for suggestions from one of the brightest communities on the web. :D

Your time in responding is much appreciated.

:apple:
http://www.activestorage.com/activeraid.php#tab5
fill it with 2tb drives and get a mac mini server or a mac pro server or eqv for remote (with a fat pipe if you don't have one allready)

or use this (32tb option) for the storage either one of these or the active raid will do fine
http://www.g-technology.com/products/g-speed-es-pro-xl.cfm
http://www.g-technology.com/products/g-speed-fc-xl.cfm

mackpro
Apr 11, 2011, 10:56 AM
Thank you all for your help so far! I'm getting closer to some conclusions thanks to you!
The speed of my internet connection has been brought up a couple times, I have a T1 connection and plenty of bandwidth.

I've quoted a few replies and have questions for some of you who have helped a lot already.
Please bear with me as I ask some questions:




For remote access, how will remote users be authenticated ? what bandwidth will they have available at their end and at the server end ? how many concurrent users will there be ? what availability do you need for remote access

For web browser access I would probably go for the Barracuda SSL VPN device.

1. I was thinking that I could set up FTP access to the data because the max concurrent users would be around 10.
2. If this is not possible what advantages does the Barracuda device provide?

You could get some sort of rack-mounted array that can support 48+ drives (assuming a bunch of 1TB drives in the array, although you could go with 2TB drives as well). You can link several smaller cases together and connect them to a Mac Pro with additional hardware that supports RAID. That assumes you want to go the Mac route for the software side. With Lion merging the serve OS that could make things a lot easier.

How many users are you planning on supporting? Are you planning on doing more than just hosting video files? Email? Calendar? Web services? (what sort of Internet connection you have plays in here as well). If it's just a couple users, your server probably doesn't need to be a high end Mac Pro. Still, look at what you are buying. This could easily be a $25,000 plus project. And don't forget the whole backup strategy which can be expensive in terms of equipment and more expensive in terms of disciplined time investment.

1. I'm wondering what type of rack-mounted array you recommend assuming I get a Mac Pro.
2. I'm looking at about ten users max, usually only a few. How much RAM do you recommend equipping the Mac Pro with assuming I get at least a quad-core.


http://www.activestorage.com/activeraid.php#tab5
fill it with 2tb drives and get a mac mini server or a mac pro server or eqv for remote (with a fat pipe if you don't have one already)

or use this (32tb option) for the storage either one of these or the active raid will do fine
http://www.g-technology.com/products...-es-pro-xl.cfm
http://www.g-technology.com/products/g-speed-fc-xl.cfm

• These look great, but I'm thinking I might be able to build something to meet my needs for less money. I have the skills to put a server together and set it up, this is just my first time creating a rig from the ground up. Thoughts?

Matty-p
Apr 11, 2011, 11:14 AM
Thank you all for your help so far! I'm getting closer to some conclusions thanks to you!
The speed of my internet connection has been brought up a couple times, I have a T1 connection and plenty of bandwidth.
• These look great, but I'm thinking I might be able to build something to meet my needs for less money. I have the skills to put a server together and set it up, this is just my first time creating a rig from the ground up. Thoughts?

(whats a t1 im guessing your not from here in europe? here t1 means a tier 1 upstream but t1's don't do leased lines wich im guessing is what americans call a t1?) how much out and in bw exactly and what size files are you talking about?

in that case id always grab a 4u supermicro chasis/mobo/psu this ones good
http://www.supermicro.com/products/chassis/4U/846/SC846E2-R900.cfm
takes 24 3.5" drives so filled with 2tb's thats 48tb with out raid,
raiud wise i'd suggest 3 volumes (or raid arrays) all 8 drives in raid 6 (if two out of the 8 disks fail you still loose no data) so 36 tb with raid fill it with one or two cheaper i5/7 quad cores and 12+gb ram

paduck
Apr 11, 2011, 02:59 PM
Thank you all for your help so far! I'm getting closer to some conclusions thanks to you!
The speed of my internet connection has been brought up a couple times, I have a T1 connection and plenty of bandwidth.

[...]

• These look great, but I'm thinking I might be able to build something to meet my needs for less money. I have the skills to put a server together and set it up, this is just my first time creating a rig from the ground up. Thoughts?

In answer to your earlier question about memory, if are just pushing files, maybe email and calendars to ten users (usually 2 or three), you probably don't "need" more than 8GB. You could go with 16GB. Don't get the memory from Apple as you will pay a premium. That is an easy cost saver there, but it only costs $400 to take it to 16GB, so on a $25k project, that might not be too much.

I'm not sure what you mean by "T1" for network connection. If it is a traditional T1, that is only about 1.5Mbps - which isn't really that fast. I have 20Mbps down/4 Mbps up from my cable company (but that is for residential use). Up speed is going to be important if you are serving files.

If I were building a rig for a business, I would try to get as much stuff that is turnkey as possible rather than building major parts myself. Time is money and you will spend a lot of time building a rig - and then having to maintain it. You are better off investing in one of the (expensive) arrays listed about (or similar). You just plug it into a rack and your MacPro and you will be good to go. Plus, at least a couple of those listed had redundant power supplies, etc. That is going to give you a lot of confidence in system availability (downtime, which is also money).

jtara
Apr 11, 2011, 09:55 PM
The speed of my internet connection has been brought up a couple times, I have a T1 connection and plenty of bandwidth.

If you have a T1 connection, you do NOT have plenty of bandwidth. In fact, in today's world you have approximately none.

T1 is 1.5mbit/sec.

On my home Internet connection (Cox cable) I have 30mbit/sec down and 5mbit/sec up.

T1 will probably support one user at a time, and that user will find it slow.

You need to make sure you have symmetrical bandwidth, since you are running a server. If you have a T1, you probably do, but, like I said, it's a snail.

If you can't get more bandwidth at a reasonable price in your location, consider co-locating your server at a data center. It will almost certainly be cheaper than bringing the bandwidth into your building.

Others have suggested that perhaps you meant "tier 1":

http://en.wikipedia.org/wiki/Tier_1_network

which is a somewhat-vague term referring to a small handful of backbone providers that supply super-fast connectivity at speeds up to OC-48c (2.5gbit/sec). Usually this is only available in data centers (where you run an Ethernet cable to a "meet me room") or in specially-wired office parks (where basically every tenant is an ISP or has special data needs), but you will usually only get a full choice among multiple tier-1 providers in a data center.

jtara
Apr 11, 2011, 10:06 PM
Hello kind folks of MacRumors.com-

I am wishing for help and advice deciding what type of hardware to purchase for a 30TB server. It must meet the following requirements:

• 30TB RAID (Redundancy important for backup purposes)
• Remote Accessibility in that I would like to be able to pull files from it anywhere that I have an internet connection.
• Macintosh friendly– it does not need to run OS X Server, it just needs to connect without hassle to Macs.

This will be used for at my office for archiving video. We may want to use Final Cut Server to sort the files in the future. I have been tasked with building this setup and am hoping for suggestions from one of the brightest communities on the web. :D

Your time in responding is much appreciated.

:apple:

If you insist on doing this in a single box, consider running Linux. There are quite a number of good vendors for reliable, high-density Linux boxes and storage systems.

You can run the netatalk package on the Linux server, which would allow network connections use the AFP (Apple) protocol. You can even use the Linux server as a Time Machine server, with the right settings. Of course, you can also run Samba to support Windows users if needed. I wouldn't use Samba or NFS for Mac users when AFP is a perfect match and is available and pretty well-supported on Linux.

However, given the amount of storage you need, it probably would make sense - as others have at least hinted - to get a dedicated SAN box for the storage, and a separate box for remote access. This might be, for example, a Cisco VPN product, or a Linux box.

Not sure if your 10 users are just the external users, or both internal and external? If it's 10 users total, a single Linux box might be reasonable for the whole thing.

robzr
Apr 12, 2011, 12:15 AM
I'm suprised nobody has mentioned 3 TB drives, you can get 7200 RPM 3TB drives for $179 or 5400 RPM 3TB drives for $149 these days, why not reduce the number of bays you need by ~50%.

It doesn't sound like you need incredibly high performance given that it's for archiving video and your internet uplink is only 1.5 MB.

You didn't mention your budget, if it's on the low side, I'd go with a Linux NAS, you could build the whole thing for under $4k.

$335 20-bay SATA chassis http://www.newegg.com/Product/Product.aspx?Item=N82E16811219033
$300 for 2x Intel SASUC8I 8 port SATA controllers
$80 for 4x SFF-8087 cables
$500 for a Mobo, RAM, CPU
$2100 for 14x 3TB 5400 RPM drives @ $149 each
$150 ~800watt quality power supply
$150 misc cooling fans & wiring, dvd drive

Run it RAID-6 with 2x hot spare on Linux. If afpd meets your performance expectations, you're good to go. I have a 10x 5200rpm 2TB RAID-6 linux NAS media server on a 3 or 4 year old low end athlon x2, I get 250 MBps on reads with software RAID-6. afpd is single threaded and will likely be a frustrating bottleneck for writes, it will hang during writes while it syncs to disk periodically. The above config would have no problem saturating gig ethernet during a read and come close to it during a write. Using it as a SAN via iSCSI may allow for higher performance, although you'd need to bounce incoming connections off another server.

So thats your low end, look into redundant X-Serves with DAS arrays for your high end (or EMC if you have a few hundred grand to blow).

Keep in mind RAID is not a replacement for backup, if you are not going to backup to external media at least consider a filesystem with extra space and snapshots, and maybe a second identical server you can replicate to. A higher quality chassis, redundant power supplies, hardware RAID, an X-Serve w/ DAS would be nice steps up.

Rob



Hello kind folks of MacRumors.com-

I am wishing for help and advice deciding what type of hardware to purchase for a 30TB server. It must meet the following requirements:

• 30TB RAID (Redundancy important for backup purposes)
• Remote Accessibility in that I would like to be able to pull files from it anywhere that I have an internet connection.
• Macintosh friendly– it does not need to run OS X Server, it just needs to connect without hassle to Macs.

This will be used for at my office for archiving video. We may want to use Final Cut Server to sort the files in the future. I have been tasked with building this setup and am hoping for suggestions from one of the brightest communities on the web. :D

Your time in responding is much appreciated.

:apple:

Matty-p
Apr 12, 2011, 03:02 AM
I'm suprised nobody has mentioned 3 TB drives, you can get 7200 RPM 3TB drives for $179 or 5400 RPM 3TB drives for $149 these days, why not reduce the number of bays you need by ~50%.

It doesn't sound like you need incredibly high performance given that it's for archiving video and your internet uplink is only 1.5 MB.

You didn't mention your budget, if it's on the low side, I'd go with a Linux NAS, you could build the whole thing for under $4k.

$335 20-bay SATA chassis http://www.newegg.com/Product/Product.aspx?Item=N82E16811219033t

$300 for 2x Intel SASUC8I 8 port SATA controllers
$80 for 4x SFF-8087 cables
$500 for a Mobo, RAM, CPU
$2100 for 14x 3TB 5400 RPM drives @ $149 each
$150 ~800watt quality power supply
$150 misc cooling fans & wiring, dvd drive

Run it RAID-6 with 2x hot spare on Linux. If afpd meets your performance expectations, you're good to go. I have a 10x 5200rpm 2TB RAID-6 linux NAS media server on a 3 or 4 year old low end athlon x2, I get 250 MBps on reads with software RAID-6. afpd is single threaded and will likely be a frustrating bottleneck for writes, it will hang during writes while it syncs to disk periodically. The above config would have no problem saturating gig ethernet during a read and come close to it during a write. Using it as a SAN via iSCSI may allow for higher performance, although you'd need to bounce incoming connections off another server.

So thats your low end, look into redundant X-Serves with DAS arrays for your high end (or EMC if you have a few hundred grand to blow).

Keep in mind RAID is not a replacement for backup, if you are not going to backup to external media at least consider a filesystem with extra space and snapshots, and maybe a second identical server you can replicate to. A higher quality chassis, redundant power supplies, hardware RAID, an X-Serve w/ DAS would be nice steps up.

Rob

Yea I was suggesting Linux or Unix for the supermicro build,
3tb are a bad idea for a pruduction environment cos there less reliable slower have far longer rebuild time for raid are more expensive the list goes on

nOw2
Apr 12, 2011, 07:06 AM
I cannot add much to that already said, but:
• 30TB RAID (Redundancy important for backup purposes)
No - redundancy provides availability, not backup. If it is important data, budget for a 30TB backup solution.

mackpro
Apr 12, 2011, 05:48 PM
You didn't mention your budget, if it's on the low side, I'd go with a Linux NAS, you could build the whole thing for under $4k.



Budget is about 10,000 USD, I figured maybe if I provide this information you all could suggest some more specific pieces of equipment.

Also, my internet connection is a T1 and I have seen it go up to 5mb down 3mb up.

jtara
Apr 12, 2011, 06:18 PM
Also, my internet connection is a T1 and I have seen it go up to 5mb down 3mb up.

Then it isn't a T1.

http://en.wikipedia.org/wiki/Digital_Signal_1

3mbit/sec is still pretty slow for this. That's only 375K bytes/sec. A 600MB media file (the size of a CD, and apx. the size of an H.264-compressed full-length SD video) would take about a half-hour to transfer.

Now, with my 30mbit/sec Internet service, I expect to be able to transfer, from the biggest sites anyway, at maybe 20mbit/sec. At certain times of the day, I get the full 30. At 20, though, that 30 minutes shrinks to 4 minutes.

Depending on what kind of connections your users have, you could frustrate the heck out of them with this setup.

MOST businesses have no legitimate need for running online servers in-house. Certainly, for the typical web hosting, it almost never makes sense to do it in-house. Because of the large amount of storage needed in this case, though, what makes sense is probably co-location. You do need to minimize the physical size, because one component of colo charges is how many "U" (=1.75 inch) units the equipment occupies.

Colo facilities typically have "hands-on" support available. It can be fairly expensive (like $50/hour) but how long does it take to swap a hard drive? And with a RAID array, you could probably get by with no attention for a little while. If the place is in-town, you can go there yourself to replace a drive if needed. Otherwise, you give them a spare or two, and if something happens, they charge you, say, 15 minutes to swap out the drive (which will take them 5 minutes).

Matty-p
Apr 13, 2011, 07:35 AM
Then it isn't a T1.

http://en.wikipedia.org/wiki/Digital_Signal_1

3mbit/sec is still pretty slow for this. That's only 375K bytes/sec. A 600MB media file (the size of a CD, and apx. the size of an H.264-compressed full-length SD video) would take about a half-hour to transfer.

Now, with my 30mbit/sec Internet service, I expect to be able to transfer, from the biggest sites anyway, at maybe 20mbit/sec. At certain times of the day, I get the full 30. At 20, though, that 30 minutes shrinks to 4 minutes.

Depending on what kind of connections your users have, you could frustrate the heck out of them with this setup.

MOST businesses have no legitimate need for running online servers in-house. Certainly, for the typical web hosting, it almost never makes sense to do it in-house. Because of the large amount of storage needed in this case, though, what makes sense is probably co-location. You do need to minimize the physical size, because one component of colo charges is how many "U" (=1.75 inch) units the equipment occupies.

Colo facilities typically have "hands-on" support available. It can be fairly expensive (like $50/hour) but how long does it take to swap a hard drive? And with a RAID array, you could probably get by with no attention for a little while. If the place is in-town, you can go there yourself to replace a drive if needed. Otherwise, you give them a spare or two, and if something happens, they charge you, say, 15 minutes to swap out the drive (which will take them 5 minutes).
so op your from ameria right? here were i live in the uk bisiness have generally leased lines that is dedicated symetric bandwith for them only and its garenteed. here the general costs are for dedicated 10 meg down 10 meg up about £400 a month over copper pairs, or depending where you are about £600 for 100mb down 100mb up dedicated over fiber. id figure out the difference betwwen a eqiv connection in america and your cost of your current curcuit and try budget that in too. 10 meg up 10 meg down will be plenty.

mackpro
Apr 13, 2011, 09:55 AM
Then it isn't a T1.

http://en.wikipedia.org/wiki/Digital_Signal_1

3mbit/sec is still pretty slow for this. That's only 375K bytes/sec. A 600MB media file (the size of a CD, and apx. the size of an H.264-compressed full-length SD video) would take about a half-hour to transfer.

Now, with my 30mbit/sec Internet service, I expect to be able to transfer, from the biggest sites anyway, at maybe 20mbit/sec. At certain times of the day, I get the full 30. At 20, though, that 30 minutes shrinks to 4 minutes.


In my post I was referring to megabytes not megabits. I get a solid 50megabits down and 10megabits up. I think this should be sufficient to get the system up and running with a possible upgrade in the future.

Having said that, I am usually working with files less than your 600MB benchmark because they are all webpage elements.

Sorry for the confusion, thank you all for your advice. I've been carefully reading each post and appreciate your continued support.

Ap0ks
Apr 13, 2011, 11:31 AM
so op your from ameria right? here were i live in the uk bisiness have generally leased lines that is dedicated symetric bandwith for them only and its garenteed. here the general costs are for dedicated 10 meg down 10 meg up about £400 a month over copper pairs, or depending where you are about £600 for 100mb down 100mb up dedicated over fiber. id figure out the difference betwwen a eqiv connection in america and your cost of your current curcuit and try budget that in too. 10 meg up 10 meg down will be plenty.Who is your provider please? That's a lot cheaper than I've been quoted :D

Edit: To keep my post on topic, OP you may want to have a look at this (http://www.avsforum.com/avs-vb/showthread.php?t=1045086) (although it seems a bit OTT spec-wise), this (http://www.simplynas.com/data-robotics-drobopro-48tb-business-bundle---3-x-drobo-pros-and-24-x-2tb-hitachi-seagate-sata-ii-drives.aspx) (how cheap NAS storage can be) and this (http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/) (just because ;))

Attach a server to serve the storage and terminate VPN connections and you should have a working solution for a reasonable cost.

Matty-p
Apr 13, 2011, 11:53 AM
Who is your provider please? That's a lot cheaper than I've been quoted :D

Edit: To keep my post on topic, OP you may want to have a look at this (http://www.avsforum.com/avs-vb/showthread.php?t=1045086) (although it seems a bit OTT spec-wise), this (http://www.simplynas.com/data-robotics-drobopro-48tb-business-bundle---3-x-drobo-pros-and-24-x-2tb-hitachi-seagate-sata-ii-drives.aspx) (how cheap NAS storage can be) and this (http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/) (just because ;))

Attach a server to serve the storage and terminate VPN connections and you should have a working solution for a reasonable cost.
have sent you a pm

Matty-p
Apr 13, 2011, 12:05 PM
In my post I was referring to megabytes not megabits. I get a solid 50megabits down and 10megabits up. I think this should be sufficient to get the system up and running with a possible upgrade in the future.

Having said that, I am usually working with files less than your 600MB benchmark because they are all webpage elements.

Sorry for the confusion, thank you all for your advice. I've been carefully reading each post and appreciate your continued support.
ah right sounds better, what are your actually type/size of files if you could do a average?

jtara
Apr 13, 2011, 02:30 PM
In my post I was referring to megabytes not megabits. I get a solid 50megabits down and 10megabits up. I think this should be sufficient to get the system up and running with a possible upgrade in the future.

Sure, that sounds sufficient for your needs.

I assume this is "business" service. If not, best to check your contract to make sure your proposed usage is consistent with the contract terms. Asymmetric bandwidth often comes with a "no servers" policy.

jtara
Apr 13, 2011, 02:47 PM
so op your from ameria right? here were i live in the uk bisiness have generally leased lines that is dedicated symetric bandwith for them only and its garenteed. here the general costs are for dedicated 10 meg down 10 meg up about £400 a month over copper pairs, or depending where you are about £600 for 100mb down 100mb up dedicated over fiber. id figure out the difference betwwen a eqiv connection in america and your cost of your current curcuit and try budget that in too. 10 meg up 10 meg down will be plenty.

Yes, S. California.

Colo is still cheaper, at least here.

For example, Hurricane Electric offers a FULL CABINET (42U) in San Francisco with 100mbps uplink speed for $600 month.

Ubiquity offers 4U at $125/month, 1/4 cabinet at $475/month both with 100mbit/sec uplink in Los Angeles. 1/2 cabinet at $850/month (about the same as the amount quoted above for a leased line alone) but with 1000 mbit/sec uplink). I don't imagine the OP would need more than 1/4 cabinet (10U), and could probably fit this into 4U.

Note "bandwidth" quota is much lower (1mbit/sec for the 4U, 5mbit/sec for the 1/4 cabinet) than the uplink speed. You can burst to 100 or 1000, but, of course, if you sustained that rate constantly, you'd be nicked for the extra bandwidth. (It's actually rather more complicated than that.) In the case of the OP, it doesn't seem likely he'd exceed the base charge. Look up "95'th percentile billing", which is standard, at least here in the U.S.

For the OP, the existing connection seems to make sense, since he already has the leased line, presumably to support in-house user's Internet access.

However, for others that might have a server need that would not be easily met by their existing leased line, (or might violate terms of service) colo might make much more sense.

The reason it is so much cheaper is because it takes the leased line completely out of the equation. There's just an Ethernet cable to another room. The expensive "last mile" is eliminated.

I've quoted some U.S. prices, but I'd assume the economies are similar in Europe. Delivering bandwidth to your business costs a lot more than delivering bandwidth to a data center.

robzr
Apr 13, 2011, 09:23 PM
Yea I was suggesting Linux or Unix for the supermicro build,
3tb are a bad idea for a pruduction environment cos there less reliable slower have far longer rebuild time for raid are more expensive the list goes on

I disagree, if you followed that logic then why stop at 1 TB or 2 TB, why not build on 72 GB SCSI. "Production Environment" is a little bit of a arbitrary distinction, this is a small business with relatively low parallel usage on a tight budget, 3 TB drives would be perfectly suitable. Real high performance "production environments" would be using 15k FCAL, SCSI or SAS in enterprise chassis, but we obviously don't have the budget for that.

RAID rebuild time is going to be more affected by RAID-6 instead of RAID-5, by software RAID instead of hardware RAID than by 3 TB vs 2/1 TB. The double parity overhead of RAID-6 makes it very slow, but for a budget 30 TB filesystem I can't think of a better tradeoff.

Advantages of 3 TB drives - chassis needs fewer bays, future expandability with the same bays is better, administration is easier (fewer drives, fewer potential points of failure), power draw is significantly less - if you've shopped for colo's lately you'll find that power is pretty expensive these days, moreso than rackspace or arguably bandwidth.

RAID-5 would be ridiculous especially if you were to go with 31 1TB drives, that would be so risky it's ludicrous. Two drives go out and you just lost all your data. Thats 3% of your drives can fail and you're SOL.

RAID-6 with 12x 3TB drives you can lose 2 drives before you lose data, or 17% of your drives. Much safer. Thats 2 out of 12 instead of 1 out of 16 or 31. I think the advantages of 3TB are clear.

Rob

MattInOz
Apr 14, 2011, 02:26 AM
build your own (http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/)

Might be worth a read

mackpro
Apr 14, 2011, 11:19 AM
build your own (http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/)

Might be worth a read

This looks like a very real possibility. I've got three main questions.

1. How on earth can I build a case like that, or what options exist for building a case like this?
2. What type of software is available to run something like this, assuming I have a working knowledge of Linux (GUI Versions)?
3. What are the pros/cons of a setup like this compared to just getting a storage array and connecting it to a Mac Pro considering the fact that I want to be able to pull data from this remotely.

...And doesn't 4GB of ram seem like too little?

jtara
Apr 14, 2011, 12:06 PM
This looks like a very real possibility. I've got three main questions.

1. How on earth can I build a case like that, or what options exist for building a case like this?

In a machine shop. They provide the 3D drawings. If they're open-licensing (or even closed-licensing) this, then there might be somebody building and selling the case, as well.

Might also want to look at the recently open-sourced Facebook plans. They've open-licensed and open-sourced their entire datacenter, including the servers they build themselves.

But I suspect there are plenty of products available like this off the shelf. It's just some sheet metal and off-the-shelf circuit boards.

2. What type of software is available to run something like this, assuming I have a working knowledge of Linux (GUI Versions)?

Any Linux distribution. All it is is a high-density PC with a whole BUNCH of SATA drives plugged-in. If you want a GUI, plug in a video card.

3. What are the pros/cons of a setup like this compared to just getting a storage array and connecting it to a Mac Pro considering the fact that I want to be able to pull data from this remotely.

It IS a storage array. Implemented as a Linux server.

However, any RAID is just going to be software RAID. It's just a BOD. (= "bunch of drives")

...And doesn't 4GB of ram seem like too little?

They're just using it as a file server. It isn't doing anything else. I'm assuming they determined that 4GB is sufficient for filesystem cache, given the possible throughput of the drives and network interface.

BTW, hot-plugging SATA drives works just fine in current Linux kernels. I do it all the time. I have a couple of carrier-less SATA docks on the front of my Linux system. You open a little door, pop in the bare SATA drive, close the door and in a few seconds the system recognizes the drive. I'm not using RAID - I just do this for making backups. SATA is designed for hot-plugging, assuming the backplane is designed for that. Of course, you should flush and dismount any filesystems on the drive before removing. I'm assuming in a software RAID configuration, you'd be able to hotplug the drives and that would be recognized and the array would be re-built.

If you do this, make sure you use very, very, very reliable fans! The fans fail, and this system is toast. Literally!

Nice thing is, this all fits in 4U, so it's cheap colo.

robzr
Apr 14, 2011, 01:30 PM
If you do this, make sure you use very, very, very reliable fans! The fans fail, and this system is toast. Literally!

Nice thing is, this all fits in 4U, so it's cheap colo.

Thats pretty cool, but you'd probably be better off with an off-the-shelf case for this project. Fans, that is an area where you will see a difference in the quality of case design. If you look at mid-high range rackmount equipment you'll see they often have rows of fans two deep for redundancy, if one fan fails there is another one right behind it; and the ducting is designed to provide sufficient and even airflow over all critical components. Like jtara says if a fan goes and you don't catch it, components can heat up and excessive heat is a killer.

If you go the Linux route, you'd be wise to run smartd to monitor hard drive temperatures and email/sms you if the temp hits an upper limit.

You should also choose a chassis with good airflow design (ie: ducting such that the air blows evenly over the drives, and also covers the other components appropriately). Too low of temp is just as bad as too high of temp for drives, a poor chassis design will result in some drives being too warm and some being too cool. Check out Googles study for details on this, it's a must read for anyone interested in DIY RAID.

You can also look into fan speed controllers, to keep airflow from being excessive, some are thermostat regulated. This is the type of thing that is built into a high quality rackmount system, one of the reasons they are so expensive.

Rob

jtara
Apr 14, 2011, 02:13 PM
If you look at mid-high range rackmount equipment you'll see they often have rows of fans two deep for redundancy, if one fan fails there is another one right behind it; and the ducting is designed to provide sufficient and even airflow over all critical components.
If you go the Linux route, you'd be wise to run smartd to monitor hard drive temperatures and email/sms you if the temp hits an upper limit.


This is one of the reasons I'm a fan of IBM servers. They do the double row of fans, they have a dual power supply option, and both the fans and power supplies are hot-pluggable. If you do the cabling right (and they supply a bracket for this) you can just slide the box out of the rack on the rollers, take the top off (flip lever), pull a fan (flip a little latch, it comes right out), and plop a new one in, and never skip a beat.

Of course, the system will notify you if something requires service. There is a separate network connection for the onboard management processor, and you can do remote reboots (watching the BIOS messages), even update the BIOS remotely.

Similar set-up for power supplies.

Their on-site service is also great. I had a couple of servers installed in a Wall St. location and required service. (OK, Broad St. to be exact.) When IBM says two hours, they mean two hours. (You pay more for better response time.)

The thermal design is of course excellent. Beautiful clear plastic ducts for the CPUs. Not the typical box where they just blow a bunch of fans over the components and pray.

Consultant
Apr 14, 2011, 03:45 PM
This looks like a very real possibility. I've got three main questions.

1. How on earth can I build a case like that, or what options exist for building a case like this?
2. What type of software is available to run something like this, assuming I have a working knowledge of Linux (GUI Versions)?
3. What are the pros/cons of a setup like this compared to just getting a storage array and connecting it to a Mac Pro considering the fact that I want to be able to pull data from this remotely.

...And doesn't 4GB of ram seem like too little?

Does not include labor and support costs.

Custom built case will be more expensive for 1 copy. Diagram is included on that page.

mackpro
Apr 18, 2011, 09:46 AM
Is it possible to hook a Mac Pro up to this, and if so will OS X recognize this much storage?

http://www.bhphotovideo.com/c/product/709613-REG/Dulce_Systems_978_0000_0_PRO_RXqp_Chassis_.html

Transporteur
Apr 18, 2011, 09:51 AM
Is it possible to hook a Mac Pro up to this, and if so will OS X recognize this much storage?

http://www.bhphotovideo.com/c/product/709613-REG/Dulce_Systems_978_0000_0_PRO_RXqp_Chassis_.html

I don't see any Mac drivers, so most likely not possible with OS X.
If you want Mac support, check out Promise.

If you want to save some money, build your own ZFS box based on OpenSolaris or FreeBSD.

mackpro
Apr 18, 2011, 10:01 AM
I don't see any Mac drivers, so most likely not possible with OS X.
If you want Mac support, check out Promise.

If you want to save some money, build your own ZFS box based on OpenSolaris or FreeBSD.

It says that it supports Mac right here
http://img571.imageshack.us/img571/7702/screenshot20110418at110.png (http://img571.imageshack.us/i/screenshot20110418at110.png/)

Uploaded with ImageShack.us (http://imageshack.us)

My question is: will a Mac recognize all 32TB of storage?

Transporteur
Apr 18, 2011, 10:14 AM
Whoops, missed that.

Well, why wouldn't a Mac recognise 32TB? Of course it will, however, you only get 32TB of usable storage if you configure that device as JBOD or RAID0 (which is bonkers).

But to be honest, the probability that everyone here has got a Mac with that very unit, so that he can tell you specific details is almost zero.
So your best bet is to call the Dulce and ask them anything you want to know.

For $5000 (does it actually include drives? If not, add another $3000) I definitely wouldn't buy that thing without knowing every little detail about it.

mackpro
Apr 18, 2011, 10:19 AM
Whoops, missed that.

Well, why wouldn't a Mac recognise 32TB? Of course it will, however, you only get 32TB of usable storage if you configure that device as JBOD or RAID0 (which is bonkers).

But to be honest, the probability that everyone here has got a Mac with that very unit, so that he can tell you specific details is almost zero.
So your best bet is to call the Dulce and ask them anything you want to know.

For $5000 (does it actually include drives? If not, add another $3000) I definitely wouldn't buy that thing without knowing every little detail about it.

In this particular case I was planning on using RAID 5, but the Mac would still need to be able to handle managing 32TB of data.

I am in the process of researching this device and several other mass-storage solutions. This very thread is actually a part of my research because crowd sourcing on a board like this is great for solving a complex IT problem.

Transporteur
Apr 18, 2011, 10:26 AM
In this particular case I was planning on using RAID 5, but the Mac would still need to be able to handle managing 32TB of data.

I am in the process of researching this device and several other mass-storage solutions. This very thread is actually a part of my research because crowd sourcing on a board like this is great for solving a complex IT problem.

Why exactly do you want to use the Quad Pack Rack? It features exchangeable quad drive boxes, which aren't useful unless you need to transfer them from one rack to another across different locations.
Check out the other Dulce (http://www.dulcesystems.com/html/products.html) systems instead.

If you configure a 16 drive unit as a RAID 5 (which I personally wouldn't to as you'd have only a single parity drive) the Mac would have to handle 30TB in case that you use 2TB drives.

Anyhow, practically there is no limitation to volume sizes in OS X. Well, in fact there is, about 8EB, but you most likely won't be able to go over this limit in the next 20 years. ;)

Edit: And now the most important recommendation:
If you want really good recommendations and background information, drop the user nanofrog a message. He's the storage expert in this forum.

nanofrog
Apr 18, 2011, 04:05 PM
If you insist on doing this in a single box, consider running Linux. There are quite a number of good vendors for reliable, high-density Linux boxes and storage systems.
Linux is definitely the way to go. Unfortunately, using a ready-made solution will definitely exceed the budget.

DIY would be better cost wise, but it comes with the compromise of self-support. And it's likely to exceed the budget as well. :(

I'm surprised nobody has mentioned 3 TB drives, you can get 7200 RPM 3TB drives for $179 or 5400 RPM 3TB drives for $149 these days, why not reduce the number of bays you need by ~50%.
At this level, enterprise disks would be required with a hardware RAID card for stability reasons (RAID6 given the member count), and recommended if done via a software implementation (i.e. multiple arrays in Z-RAID2 + 2 hot spares per array due to both the member count, and it's to be used for remote access <I treating it as a remote system too = no person available 24/7 to fix a fault>).

A Hitachi 3TB Ultrastar goes for $350 each from newegg (here (http://www.newegg.com/Product/Product.aspx?Item=N82E16822145477&cm_re=3tb_internal_hard_drive-_-22-145-477-_-Product)).

2 arrays @ 7 members each + 2 hot spares per (RAID6 or Z-RAID2), gives a usable capacity of 30TB.

Cost = $6300.

That doesn't leave much, and it certainly won't cover both the server and a backup solution (assuming a JBOD configuration, there's another $1800 in 3TB consumer grade Hitachi Deskstars <not my favorite brand either for consumer disks due to failures>).

Total disks = $8100 (no boards, enclosures, or cables). :eek: :(

Keep in mind RAID is not a replacement for backup...
Absolutely.

Given the amount of data involved, I'd skip trying to use anything but a proper backup (there are inexpensive ways of doing it for the requirements, but eSATA may not be an option; i.e. single eSATA port would take ~2 days @ 200MB/s). Adding more disks (consumer units would work here due to the lower duty cycle) on a second set of non-RAID controllers would probably be the cheapest way to go (may need another disk enclosure; say 3TB disks and a 24 bay Norco enclosure (http://www.newegg.com/Product/Product.aspx?Item=N82E16816133046)).

There's also a bit of a concern with SATA due to the cable length limitations (1.0 meters), as the SFF-8088 to fanout adapters used tend to introduce too much contact resistance (results in unstable arrays - seen this before). In such cases, shorter cables are needed, which means a custom order (say 0.5 to 0.75 meters).

RAID rebuild time is going to be more affected by RAID-6 instead of RAID-5, by software RAID instead of hardware RAID than by 3 TB vs 2/1 TB. The double parity overhead of RAID-6 makes it very slow, but for a budget 30 TB file system I can't think of a better trade-off.
RAID 6 or Z-RAID2 would be the way to go in terms of redundancy vs. performance trade-off for the given capacity requirement IMO as well.

Advantages of 3 TB drives - chassis needs fewer bays, future expandability with the same bays is better, administration is easier (fewer drives, fewer potential points of failure), power draw is significantly less - if you've shopped for colo's lately you'll find that power is pretty expensive these days, moreso than rackspace or arguably bandwidth.
There are cases where using smaller capacity drives has an advantage (no need to keep buying enclosures), and additional performance due to the additional parallelism from additional members in a set.

But in this case, the budget may not work out (need to go back and see if an additional enclosure + 2TB disks is cheaper, but I don't think it will be).

RAID-5 would be ridiculous especially if you were to go with 31 1TB drives, that would be so risky it's ludicrous. Two drives go out and you just lost all your data. Thats 3% of your drives can fail and you're SOL.
Exactly. Even if someone was there 24/7 to catch a failed disk (i.e. no hot spare), there's still the risk of a second failure during the rebuild process, which is becoming more critical with increased platter densities.

If you want to save some money, build your own ZFS box based on OpenSolaris or FreeBSD.
Given the budget, I don't see away around this approach.

Edit: And now the most important recommendation:
If you want really good recommendations and background information, drop the user nanofrog a message. He's the storage expert in this forum.
Gee. Thanks for the mess you got me into... :eek: :D :p

mrichmon
Apr 18, 2011, 10:51 PM
I am wishing for help and advice deciding what type of hardware to purchase for a 30TB server. It must meet the following requirements:

• 30TB RAID (Redundancy important for backup purposes)


One thing you should consider is that using a RAID array is not a backup strategy. A RAID array will help guard against a failed hard drive though at the expense of increased overall risk.

Individual hard drives have a probability of failure that is expressed as a "Mean Time Before Failure" (MTBF). Typically the MTBF for a hard drive is in the thousands of hours (eg MTBF = 500000 operation hours). With a single volume stored on one hard disk then the volume is dependent on a single hard drive so the MTBF for that volume is equal to the MTBF for the hard disk. With a volume stored on a RAID array the volume is dependent on some number of hard drives. With every additional drive in the RAID array the probability of a hard drive failure in the array increases.

With the right RAID levels you can guard against these hard drive failures and provide recoverability of the data. But being able to recover the data by rebuilding an array is not a backup strategy. RAID will also provide much faster average data rates than the bare storage media will provide. (This was the original motivation for developing RAID approaches.)

If you have 30TB of data that you care about then you might want to consider a backup strategy that will protect against:

catastrophic hardware failure (fire/over-voltage/etc) that would eliminate more drives than your RAID level can rebuild from,
building damage/theft of hardware.


Not trying to come down on your plans. I know from experience designing multi-TB and small-PB storage systems that TB-scale storage without a backup strategy is a very bad day waiting to happen.

If this 30TB storage system is a scratch store then maybe you don't need a backup strategy. I've rarely seen a store of 10+TB where the users could afford to loose the contents. Sure, you can configure a large data-store and have it run successfully for a long time. But hardware failure does happen, it is only a matter of time.

kram96
Apr 18, 2011, 11:14 PM
Have you considered a Synology NAS?
It can do everything that you've asked for and more but for way less.

http://www.synology.com/us/products/RS2211RP+/

Capable of 30TB and scalable to 66TB.
Link aggregation with throughput of 198MB/sec

$3k for the server
$140 3TB Drives x 10 = $1400

VPN could be a good solution for a tunnel you can install open VPN on the box or get a hardware VPN router

With your budget, buy a second and use as a backup. The units can sync automatically and encrypted on the same network or to a remote location.

This is assuming that you have a switch that will handle two sets of aggregated links.

nanofrog
Apr 19, 2011, 08:56 AM
$140 3TB Drives x 10 = $1400.
Even though it's only 10x disks (which would only allow for either a JBOD or RAID 0), consumer grade disks aren't the way to go for this much capacity used for primary storage. As the backup pool, consumer grade disks are fine due to the lower duty cycle (say 8 hrs per day on average).

It's cheap, but it's not going to be reliable enough for a primary pool. Enterprise grade 3TB disks aren't cheap ($350 per, which is more than double the 5400rpm consumer versions, not quite that much for the 7200rpm consumer units; all 3TB units).

Some might consider this splitting hairs, but I see it as a critical difference for the uptime requirement.

robzr
Apr 19, 2011, 03:52 PM
Even though it's only 10x disks (which would only allow for either a JBOD or RAID 0), consumer grade disks aren't the way to go for this much capacity used for primary storage. As the backup pool, consumer grade disks are fine due to the lower duty cycle (say 8 hrs per day on average).

It's cheap, but it's not going to be reliable enough for a primary pool. Enterprise grade 3TB disks aren't cheap ($350 per, which is more than double the 5400rpm consumer versions, not quite that much for the 7200rpm consumer units; all 3TB units).

Some might consider this splitting hairs, but I see it as a critical difference for the uptime requirement.

http://storagemojo.com/2007/02/20/everything-you-know-about-disks-is-wrong/

From what I've seen, there isn't much of a benefit to "enterprise" hard drives within the same family, that is to say a Consumer SATA 3 TB 7200 rpm vs a Enterprise SATA 3 TB 7200 rpm. Google's study shows they use "consumer" grade, and numerous studies have shown that there is no significant difference in failure rate.

I say save the money, spend it on a few more hot spares and you'll be better off; cheaper and safe. If moneys no object, or you know what firmware tweaks the Enterprise class drives have and whether or not they'll affect you, then this isn't your thread ;)

Also the common trend seems to indicate that MTBF/MTTF is pretty meaningless in the real world...

The OP indicated that he has a Mac Pro he can use, if thats the case then go DAS by all means, Linux will be a compromise...

Rob

Matty-p
Apr 19, 2011, 05:18 PM
Wirelessly posted (Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8A293 Safari/6531.22.7)

Op
You have thought about a 1 k Mac mini / eBay xserve with a 5k 16bay promise filled with 16*2tb drived at 1 k that's still only 7k and will fit in a 4u Colo
http://www.google.com/m/products/detail?gl=uk&client=ms-android-google&source=mog&q=promise+16+bay&cid=12414427445159617004

nanofrog
Apr 19, 2011, 05:49 PM
From what I've seen, there isn't much of a benefit to "enterprise" hard drives within the same family, that is to say a Consumer SATA 3 TB 7200 rpm vs a Enterprise SATA 3 TB 7200 rpm. Google's study shows they use "consumer" grade, and numerous studies have shown that there is no significant difference in failure rate.
I realize where you're coming from.

Disks nowadays are definitely made from the same primary components (enterprise models used to provide different mechanicals, such as spindle motors and servos). But there are some differences (whether they matter or not is debatable, such as "cherry picked platters" and additional sensors), but as you mention, the firmware is significant in some cases (definitely with hardware RAID cards).

If running a Linux box and using ZFS, consumer disks could be used. But I've seen too many failures lately, particularly of consumer grade disks (Seagate and Hitachi in particular), so I'm hesitant to trust them for high duty cycle requirements (definitely agree that stated specifications aren't what happens in the real world, such as ~31% failure rates with Seagate 7200.12's just off of newegg's user reviews <qualified them by actual failure rates I could determine from the reviews, just presume the 1 or 2 egg ratings = failures, as some got 5 stars due to newegg's return policy>). I even take the time to surface scan disks when possible out of the bag, and find that the enterprise models don't exhibit as many bad sectors in general (can lend one to think the "cherry picked" platters do have a positive influence).

I'm not convinced of the previous heat arguments (sudden death; but in general with electronics, heat over time tends to have a negative effect on overall lifespan). As per vibration, I'm a bit torn (multiple arrays in the same enclosure and no rubber dampening, I'm concerned that the vibration could be a problem on the newer disk types as the platter densities are high and the heads ride closer to the surface than past disk designs - I did note that the articles have some age to them, and don't reflect disks made in the last few years; article is 4 years old).

The OP indicated that he has a Mac Pro he can use, if thats the case then go DAS by all means, Linux will be a compromise...
Then you're talking about a RAID card, enclosures, and enterprise disks (need them for the recovery time settings), where the disks alone would eat the majority of the budget ($8100 of it). That would leave $1900 for card/s and enclosures (SAS expander for backup disks and 1:1 port/disk ratio for the primary pool would likely be more cost effective).

Linux is a compromise, but based on the stated budget, it seems to be the way to go.

robzr
Apr 19, 2011, 07:31 PM
I realize where you're coming from.

Disks nowadays are definitely made from the same primary components (enterprise models used to provide different mechanicals, such as spindle motors and servos). But there are some differences (whether they matter or not is debatable, such as "cherry picked platters" and additional sensors), but as you mention, the firmware is significant in some cases (definitely with hardware RAID cards).


I'll believe that when I see a study that shows that, but with 200,000 disks in the Google & CMU studies showing, and Netapp's rep stating that theres not a reliability advantage, I think I'll save the extra cash unless I know I need to outlay it.




I'm not convinced of the previous heat arguments (sudden death; but in general with electronics, heat over time tends to have a negative effect on overall lifespan). As per vibration, I'm a bit torn (multiple arrays in the same enclosure and no rubber dampening, I'm concerned that the vibration could be a problem on the newer disk types as the platter densities are high and the heads ride closer to the surface than past disk designs - I did note that the articles have some age to them, and don't reflect disks made in the last few years; article is 4 years old).


If you check out the Google study (http://labs.google.com/papers/disk_failures.pdf) theres a pretty clear correlation of heat, both too high and too low, to drive failure trends.



Then you're talking about a RAID card, enclosures, and enterprise disks (need them for the recovery time settings), where the disks alone would eat the majority of the budget ($8100 of it). That would leave $1900 for card/s and enclosures (SAS expander for backup disks and 1:1 port/disk ratio for the primary pool would likely be more cost effective).

Linux is a compromise, but based on the stated budget, it seems to be the way to go.

Sans Digital AS316X2HA, 16 disk RAID 6 DAS, $3295 @ Amazon including a RAID card and all cables, redundant power supplies. 14x Hitachi 0S03230's gives you 30 TB RAID-6 with 2 hot spares, and you still have 2 empty drive bays for expansion. Grand total $5255, and you can get a second identical setup for backup and you're only $510 over the $10k budget.

Run Mac native afpdserver and it will humiliate the Linux box performance-wise while being easier to maintain and integrate. Netatalk is great, but afpd has some serious performance issues, and avahi has some stability issues in my experience.

Rob

nanofrog
Apr 19, 2011, 10:17 PM
I'll believe that when I see a study that shows that, but with 200,000 disks in the Google & CMU studies showing, and Netapp's rep stating that theres not a reliability advantage, I think I'll save the extra cash unless I know I need to outlay it.

If you check out the Google study (http://labs.google.com/papers/disk_failures.pdf) theres a pretty clear correlation of heat, both too high and too low, to drive failure trends.
There are instances where this logic makes sense (definitely for backup or some software based storage implementations <no parity based arrays in order to avoid the write hole issue associated with 5/6/...>).

But of the links you've provided, they're 4 - 5 years old, and don't take a couple of things into consideration.

Higher platter densities and smaller distances between the platters and heads.
Cost cutting (i.e. Hitachi shifting their primary manufacturing from Malaysia to China). IIRC, they spun the Shenzen facilities into high gear ~2007, and it's become the sole source of their consumer disks (haven't seen a Malaysian labeled consumer disk in years now). QC's never been the same since...


Sans Digital AS316X2HA, 16 disk RAID 6 DAS, $3295 @ Amazon including a RAID card and all cables, redundant power supplies. 14x Hitachi 0S03230's gives you 30 TB RAID-6 with 2 hot spares, and you still have 2 empty drive bays for expansion. Grand total $5255, and you can get a second identical setup for backup and you're only $510 over the $10k budget.
Have you verified that those disks will work with the controller that comes in that package?

Given it's a RAID card, I wouldn't just presume they would (dealt with too many consumer disks causing stability issues on RAID cards). BTW, the photo of that card resembles Areca (they do offer ODM services, and have provided products to Sans Digital and Highpoint), and I've seen first hand how picky Areca's can be over disks.

If they won't work, they're moot on that controller (or any other they don't work with, and RAID cards tend to be difficult with consumer models due to the recovery timings coded into the firmware). The days of running WDTLER.exe on consumer grade WD disks is over.

BTW, not sure if you've heard, but WD is attempting to acquire HitachiGST (article (http://www.computerworld.com/s/article/9213650/With_Hitachi_buy_Western_Digital_moves_further_ahead_of_rivals?source=rss_storage&utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+computerworld%2Fs%2Ffeed%2Ftopic%2F19+%28Computerworld+Storage+News%29)).

rcorbin
Jul 11, 2011, 12:44 AM
I'm suprised nobody has mentioned 3 TB drives, you can get 7200 RPM 3TB drives for $179 or 5400 RPM 3TB drives for $149 these days, why not reduce the number of bays you need by ~50%.

It doesn't sound like you need incredibly high performance given that it's for archiving video and your internet uplink is only 1.5 MB.

You didn't mention your budget, if it's on the low side, I'd go with a Linux NAS, you could build the whole thing for under $4k.

$335 20-bay SATA chassis http://www.newegg.com/Product/Product.aspx?Item=N82E16811219033
$300 for 2x Intel SASUC8I 8 port SATA controllers
$80 for 4x SFF-8087 cables
$500 for a Mobo, RAM, CPU
$2100 for 14x 3TB 5400 RPM drives @ $149 each
$150 ~800watt quality power supply
$150 misc cooling fans & wiring, dvd drive

Run it RAID-6 with 2x hot spare on Linux. If afpd meets your performance expectations, you're good to go. I have a 10x 5200rpm 2TB RAID-6 linux NAS media server on a 3 or 4 year old low end athlon x2, I get 250 MBps on reads with software RAID-6. afpd is single threaded and will likely be a frustrating bottleneck for writes, it will hang during writes while it syncs to disk periodically. The above config would have no problem saturating gig ethernet during a read and come close to it during a write. Using it as a SAN via iSCSI may allow for higher performance, although you'd need to bounce incoming connections off another server.



Hey Rob,

Was looking into my next project (home media storage) and I came across this thread and your post. I put together a quick wishlist for what you described above

http://secure.newegg.com/WishList/PublicWishDetail.aspx?WishListNumber=20749087

If you have time can you take a quick glance? Trying to see if I am missing anything like cables/fans or there are any gotchas like incompatibilities in hardware. I only want 30TB in RAID6 but I wanted two extra drives at home in case of failures, but want 2 drives for OS in RAID1..so 14 drives total plugged in.

Thanks!

-r

snakebitesmac
Jul 11, 2011, 09:10 PM
I would recommend something similar to my setup.

My server runs on OpenIndiana(Solaris). I have about 10tb of ZFS raid storage.

I'm usually a little paranoid about data integrity.

This might work for you.
2x 150gb 2.5 sata drives for mirrored system disks
14x 3tb hds(2x 7 drive arrays in dual parity )

You can expose shares to mac users via Netatalk.