Building Remotely Accessible 30TB Server

You might find it easier to split this into two projects, one for the server and RAID storage, and a separate one for remote access. I would advise against trying to get a single device to do both.

The use profile and availability requirements for the server and storage (online use, nearline storage or archive storage) will determine what is required, as a for instance, you might go for 16 2TB disks in a single controller RAID5 array connected to a single server running SMB or NFS for archive, whereas for online, you might go for a clustered server solution with 34 1TB disks in dual controller RAID6 array with 2 hotspares.

For remote access, how will remote users be authenticated ? what bandwidth will they have available at their end and at the server end ? how many concurrent users will there be ? what availability do you need for remote access

There are two main types of remote access to files. The traditional method is network level access over either PPTP, L2TP, IPSec or SSL. The other method is to access files through a HTTPS server.

For network level access I would probably use a Cisco ASA device, which could be clustered with a second one in an active/passive failover cluster if you have high availability requirements. I would use SSL rather than IPSec for remote VPN users as IPSec can have issues with some NAT firewalls (home routers) whereas SSL works over just about everything, also as long as you have an LDAP authentication back end (such as AD or, OD) having different groups of users having different access rights can be much simpler to maintain.

For web browser access I would probably go for the Barracuda SSL VPN device.

The easy part is going to be to connect it to the Internet. Basically, you just need a router that will port forward or let you login to a Virtual Private Network (VPN). Then it will be just like being on your LAN when you login remotely. Given that you are talking about hosting video, you might want to look at what the Internet connection's upload speed is and plan accordingly.

On the server side there are a lot of options. It always gets said, so I will say it first - RAID is not about backup. It is about availability. You need to have a backup strategy and RAID isn't it. That is going to add to your costs since you are talking about a lot of data.

In terms of the array, The only way to get to 30TB realistically is some sort of RAID. You probably are going to want something like RAID5 or RAID6 which will allow for one or more drive failures without losing the whole array. Something to consider - to get to 30TB under those circumstances you are going to need upwards of 40TB of hard drives. So your hardware is going to have to support that. For reliability, you might want to consider buying enterprise class drives.

You could get some sort of rack-mounted array that can support 48+ drives (assuming a bunch of 1TB drives in the array, although you could go with 2TB drives as well). You can link several smaller cases together and connect them to a Mac Pro with additional hardware that supports RAID. That assumes you want to go the Mac route for the software side. With Lion merging the serve OS that could make things a lot easier.

How many users are you planning on supporting? Are you planning on doing more than just hosting video files? Email? Calendar? Web services? (what sort of Internet connection you have plays in here as well). If it's just a couple users, your server probably doesn't need to be a high end Mac Pro. Still, look at what you are buying. This could easily be a $25,000 plus project. And don't forget the whole backup strategy which can be expensive in terms of equipment and more expensive in terms of disciplined time investment.

I've mentioned the Internet connection a couple of times as well. Don't forget the LAN configuration either. You are going to want a Gigabit switch that can support multiple users and lots of data. A second Ethernet port on the comput for both more speed and redundancy is probably a good idea as well.

All of this may mean that it is worth some investment with a professional to design the system and provide you with a maintenance contract and support for the hardware and software. In a business environment, that might be worth the investment.

Apple doesn't have Xserve anymore, but they may have something else coming down the pike (don't hold your breath). That might be a good option. Otherwise you can always go with Linux or Windows - but if all your other equipment is Apple, I'd try and stick with the all-Mac configuration to keep things simple.

Good luck, sounds like an interesting project. I think you'll learn a lot.

I've been looking at the same setup but start off with maybe 8Gbs and expand from there. The following website is what I've been looking at: http://lime-technology.com/

I'm looking at the server that has 12 bays. I don't know how all this stuff works yet...but I'm learning. There is also a forum that has a lot of good info.

Keep us posted on what you decide.

I'd suggest contacting a large post-production house and seeing if they'll show you how they do it.

My son works at a big place in NYC and I've seen their machine room (mostly Apple). His outfit does what you're talking about, 24/7, with access all over N America and Europe, both for the editors and their clients.

I would suppose that any good-sized house does the same, and since you're specifically doing video -- well, that's what the post houses do for a living.

Sometimes imitation is an excellent solution.

mackpro said:

Hello kind folks of MacRumors.com-

I am wishing for help and advice deciding what type of hardware to purchase for a 30TB server. It must meet the following requirements:

• 30TB RAID (Redundancy important for backup purposes)
• Remote Accessibility in that I would like to be able to pull files from it anywhere that I have an internet connection.
• Macintosh friendly– it does not need to run OS X Server, it just needs to connect without hassle to Macs.

This will be used for at my office for archiving video. We may want to use Final Cut Server to sort the files in the future. I have been tasked with building this setup and am hoping for suggestions from one of the brightest communities on the web.

Your time in responding is much appreciated.

Click to expand...

http://www.activestorage.com/activeraid.php#tab5
fill it with 2tb drives and get a mac mini server or a mac pro server or eqv for remote (with a fat pipe if you don't have one allready)

or use this (32tb option) for the storage either one of these or the active raid will do fine
http://www.g-technology.com/products/g-speed-es-pro-xl.cfm
http://www.g-technology.com/products/g-speed-fc-xl.cfm

mackpro said:

Thank you all for your help so far! I'm getting closer to some conclusions thanks to you!
The speed of my internet connection has been brought up a couple times, I have a T1 connection and plenty of bandwidth.
• These look great, but I'm thinking I might be able to build something to meet my needs for less money. I have the skills to put a server together and set it up, this is just my first time creating a rig from the ground up. Thoughts?

Click to expand...

(whats a t1 im guessing your not from here in europe? here t1 means a tier 1 upstream but t1's don't do leased lines wich im guessing is what americans call a t1?) how much out and in bw exactly and what size files are you talking about?

in that case id always grab a 4u supermicro chasis/mobo/psu this ones good
http://www.supermicro.com/products/chassis/4U/846/SC846E2-R900.cfm
takes 24 3.5" drives so filled with 2tb's thats 48tb with out raid,
raiud wise i'd suggest 3 volumes (or raid arrays) all 8 drives in raid 6 (if two out of the 8 disks fail you still loose no data) so 36 tb with raid fill it with one or two cheaper i5/7 quad cores and 12+gb ram

mackpro said:

Thank you all for your help so far! I'm getting closer to some conclusions thanks to you!
The speed of my internet connection has been brought up a couple times, I have a T1 connection and plenty of bandwidth.

[...]

• These look great, but I'm thinking I might be able to build something to meet my needs for less money. I have the skills to put a server together and set it up, this is just my first time creating a rig from the ground up. Thoughts?

Click to expand...

In answer to your earlier question about memory, if are just pushing files, maybe email and calendars to ten users (usually 2 or three), you probably don't "need" more than 8GB. You could go with 16GB. Don't get the memory from Apple as you will pay a premium. That is an easy cost saver there, but it only costs $400 to take it to 16GB, so on a $25k project, that might not be too much.

I'm not sure what you mean by "T1" for network connection. If it is a traditional T1, that is only about 1.5Mbps - which isn't really that fast. I have 20Mbps down/4 Mbps up from my cable company (but that is for residential use). Up speed is going to be important if you are serving files.

If I were building a rig for a business, I would try to get as much stuff that is turnkey as possible rather than building major parts myself. Time is money and you will spend a lot of time building a rig - and then having to maintain it. You are better off investing in one of the (expensive) arrays listed about (or similar). You just plug it into a rack and your MacPro and you will be good to go. Plus, at least a couple of those listed had redundant power supplies, etc. That is going to give you a lot of confidence in system availability (downtime, which is also money).

mackpro said:

The speed of my internet connection has been brought up a couple times, I have a T1 connection and plenty of bandwidth.

Click to expand...

If you have a T1 connection, you do NOT have plenty of bandwidth. In fact, in today's world you have approximately none.

T1 is 1.5mbit/sec.

On my home Internet connection (Cox cable) I have 30mbit/sec down and 5mbit/sec up.

T1 will probably support one user at a time, and that user will find it slow.

You need to make sure you have symmetrical bandwidth, since you are running a server. If you have a T1, you probably do, but, like I said, it's a snail.

If you can't get more bandwidth at a reasonable price in your location, consider co-locating your server at a data center. It will almost certainly be cheaper than bringing the bandwidth into your building.

Others have suggested that perhaps you meant "tier 1":

http://en.wikipedia.org/wiki/Tier_1_network

which is a somewhat-vague term referring to a small handful of backbone providers that supply super-fast connectivity at speeds up to OC-48c (2.5gbit/sec). Usually this is only available in data centers (where you run an Ethernet cable to a "meet me room") or in specially-wired office parks (where basically every tenant is an ISP or has special data needs), but you will usually only get a full choice among multiple tier-1 providers in a data center.

mackpro said:

Hello kind folks of MacRumors.com-

I am wishing for help and advice deciding what type of hardware to purchase for a 30TB server. It must meet the following requirements:

• 30TB RAID (Redundancy important for backup purposes)
• Remote Accessibility in that I would like to be able to pull files from it anywhere that I have an internet connection.
• Macintosh friendly– it does not need to run OS X Server, it just needs to connect without hassle to Macs.

This will be used for at my office for archiving video. We may want to use Final Cut Server to sort the files in the future. I have been tasked with building this setup and am hoping for suggestions from one of the brightest communities on the web.

Your time in responding is much appreciated.

Click to expand...

If you insist on doing this in a single box, consider running Linux. There are quite a number of good vendors for reliable, high-density Linux boxes and storage systems.

You can run the netatalk package on the Linux server, which would allow network connections use the AFP (Apple) protocol. You can even use the Linux server as a Time Machine server, with the right settings. Of course, you can also run Samba to support Windows users if needed. I wouldn't use Samba or NFS for Mac users when AFP is a perfect match and is available and pretty well-supported on Linux.

However, given the amount of storage you need, it probably would make sense - as others have at least hinted - to get a dedicated SAN box for the storage, and a separate box for remote access. This might be, for example, a Cisco VPN product, or a Linux box.

Not sure if your 10 users are just the external users, or both internal and external? If it's 10 users total, a single Linux box might be reasonable for the whole thing.

I'm suprised nobody has mentioned 3 TB drives, you can get 7200 RPM 3TB drives for $179 or 5400 RPM 3TB drives for $149 these days, why not reduce the number of bays you need by ~50%.

It doesn't sound like you need incredibly high performance given that it's for archiving video and your internet uplink is only 1.5 MB.

You didn't mention your budget, if it's on the low side, I'd go with a Linux NAS, you could build the whole thing for under $4k.

$335 20-bay SATA chassis http://www.newegg.com/Product/Product.aspx?Item=N82E16811219033
$300 for 2x Intel SASUC8I 8 port SATA controllers
$80 for 4x SFF-8087 cables
$500 for a Mobo, RAM, CPU
$2100 for 14x 3TB 5400 RPM drives @ $149 each
$150 ~800watt quality power supply
$150 misc cooling fans & wiring, dvd drive

Run it RAID-6 with 2x hot spare on Linux. If afpd meets your performance expectations, you're good to go. I have a 10x 5200rpm 2TB RAID-6 linux NAS media server on a 3 or 4 year old low end athlon x2, I get 250 MBps on reads with software RAID-6. afpd is single threaded and will likely be a frustrating bottleneck for writes, it will hang during writes while it syncs to disk periodically. The above config would have no problem saturating gig ethernet during a read and come close to it during a write. Using it as a SAN via iSCSI may allow for higher performance, although you'd need to bounce incoming connections off another server.

So thats your low end, look into redundant X-Serves with DAS arrays for your high end (or EMC if you have a few hundred grand to blow).

Keep in mind RAID is not a replacement for backup, if you are not going to backup to external media at least consider a filesystem with extra space and snapshots, and maybe a second identical server you can replicate to. A higher quality chassis, redundant power supplies, hardware RAID, an X-Serve w/ DAS would be nice steps up.

Rob

mackpro said:

Hello kind folks of MacRumors.com-

I am wishing for help and advice deciding what type of hardware to purchase for a 30TB server. It must meet the following requirements:

• 30TB RAID (Redundancy important for backup purposes)
• Remote Accessibility in that I would like to be able to pull files from it anywhere that I have an internet connection.
• Macintosh friendly– it does not need to run OS X Server, it just needs to connect without hassle to Macs.

This will be used for at my office for archiving video. We may want to use Final Cut Server to sort the files in the future. I have been tasked with building this setup and am hoping for suggestions from one of the brightest communities on the web.

Your time in responding is much appreciated.

Click to expand...

robzr said:

I'm suprised nobody has mentioned 3 TB drives, you can get 7200 RPM 3TB drives for $179 or 5400 RPM 3TB drives for $149 these days, why not reduce the number of bays you need by ~50%.

It doesn't sound like you need incredibly high performance given that it's for archiving video and your internet uplink is only 1.5 MB.

You didn't mention your budget, if it's on the low side, I'd go with a Linux NAS, you could build the whole thing for under $4k.

$335 20-bay SATA chassis http://www.newegg.com/Product/Product.aspx?Item=N82E16811219033t

$300 for 2x Intel SASUC8I 8 port SATA controllers
$80 for 4x SFF-8087 cables
$500 for a Mobo, RAM, CPU
$2100 for 14x 3TB 5400 RPM drives @ $149 each
$150 ~800watt quality power supply
$150 misc cooling fans & wiring, dvd drive

Run it RAID-6 with 2x hot spare on Linux. If afpd meets your performance expectations, you're good to go. I have a 10x 5200rpm 2TB RAID-6 linux NAS media server on a 3 or 4 year old low end athlon x2, I get 250 MBps on reads with software RAID-6. afpd is single threaded and will likely be a frustrating bottleneck for writes, it will hang during writes while it syncs to disk periodically. The above config would have no problem saturating gig ethernet during a read and come close to it during a write. Using it as a SAN via iSCSI may allow for higher performance, although you'd need to bounce incoming connections off another server.

So thats your low end, look into redundant X-Serves with DAS arrays for your high end (or EMC if you have a few hundred grand to blow).

Keep in mind RAID is not a replacement for backup, if you are not going to backup to external media at least consider a filesystem with extra space and snapshots, and maybe a second identical server you can replicate to. A higher quality chassis, redundant power supplies, hardware RAID, an X-Serve w/ DAS would be nice steps up.

Rob

Click to expand...

Yea I was suggesting Linux or Unix for the supermicro build,
3tb are a bad idea for a pruduction environment cos there less reliable slower have far longer rebuild time for raid are more expensive the list goes on

I cannot add much to that already said, but:

mackpro said:

• 30TB RAID (Redundancy important for backup purposes)

Click to expand...

No - redundancy provides availability, not backup. If it is important data, budget for a 30TB backup solution.

mackpro said:

Also, my internet connection is a T1 and I have seen it go up to 5mb down 3mb up.

Click to expand...

Then it isn't a T1.

http://en.wikipedia.org/wiki/Digital_Signal_1

3mbit/sec is still pretty slow for this. That's only 375K bytes/sec. A 600MB media file (the size of a CD, and apx. the size of an H.264-compressed full-length SD video) would take about a half-hour to transfer.

Now, with my 30mbit/sec Internet service, I expect to be able to transfer, from the biggest sites anyway, at maybe 20mbit/sec. At certain times of the day, I get the full 30. At 20, though, that 30 minutes shrinks to 4 minutes.

Depending on what kind of connections your users have, you could frustrate the heck out of them with this setup.

MOST businesses have no legitimate need for running online servers in-house. Certainly, for the typical web hosting, it almost never makes sense to do it in-house. Because of the large amount of storage needed in this case, though, what makes sense is probably co-location. You do need to minimize the physical size, because one component of colo charges is how many "U" (=1.75 inch) units the equipment occupies.

Colo facilities typically have "hands-on" support available. It can be fairly expensive (like $50/hour) but how long does it take to swap a hard drive? And with a RAID array, you could probably get by with no attention for a little while. If the place is in-town, you can go there yourself to replace a drive if needed. Otherwise, you give them a spare or two, and if something happens, they charge you, say, 15 minutes to swap out the drive (which will take them 5 minutes).

jtara said:

Then it isn't a T1.

http://en.wikipedia.org/wiki/Digital_Signal_1

3mbit/sec is still pretty slow for this. That's only 375K bytes/sec. A 600MB media file (the size of a CD, and apx. the size of an H.264-compressed full-length SD video) would take about a half-hour to transfer.

Now, with my 30mbit/sec Internet service, I expect to be able to transfer, from the biggest sites anyway, at maybe 20mbit/sec. At certain times of the day, I get the full 30. At 20, though, that 30 minutes shrinks to 4 minutes.

Depending on what kind of connections your users have, you could frustrate the heck out of them with this setup.

MOST businesses have no legitimate need for running online servers in-house. Certainly, for the typical web hosting, it almost never makes sense to do it in-house. Because of the large amount of storage needed in this case, though, what makes sense is probably co-location. You do need to minimize the physical size, because one component of colo charges is how many "U" (=1.75 inch) units the equipment occupies.

Colo facilities typically have "hands-on" support available. It can be fairly expensive (like $50/hour) but how long does it take to swap a hard drive? And with a RAID array, you could probably get by with no attention for a little while. If the place is in-town, you can go there yourself to replace a drive if needed. Otherwise, you give them a spare or two, and if something happens, they charge you, say, 15 minutes to swap out the drive (which will take them 5 minutes).

Click to expand...

so op your from ameria right? here were i live in the uk bisiness have generally leased lines that is dedicated symetric bandwith for them only and its garenteed. here the general costs are for dedicated 10 meg down 10 meg up about £400 a month over copper pairs, or depending where you are about £600 for 100mb down 100mb up dedicated over fiber. id figure out the difference betwwen a eqiv connection in america and your cost of your current curcuit and try budget that in too. 10 meg up 10 meg down will be plenty.

Matty-p said:

so op your from ameria right? here were i live in the uk bisiness have generally leased lines that is dedicated symetric bandwith for them only and its garenteed. here the general costs are for dedicated 10 meg down 10 meg up about £400 a month over copper pairs, or depending where you are about £600 for 100mb down 100mb up dedicated over fiber. id figure out the difference betwwen a eqiv connection in america and your cost of your current curcuit and try budget that in too. 10 meg up 10 meg down will be plenty.

Click to expand...

Who is your provider please? That's a lot cheaper than I've been quoted

Edit: To keep my post on topic, OP you may want to have a look at this (although it seems a bit OTT spec-wise), this (how cheap NAS storage can be) and this (just because )

Attach a server to serve the storage and terminate VPN connections and you should have a working solution for a reasonable cost.

Ap0ks said:

Who is your provider please? That's a lot cheaper than I've been quoted

Edit: To keep my post on topic, OP you may want to have a look at this (although it seems a bit OTT spec-wise), this (how cheap NAS storage can be) and this (just because )

Attach a server to serve the storage and terminate VPN connections and you should have a working solution for a reasonable cost.

Click to expand...

have sent you a pm

mackpro said:

In my post I was referring to megabytes not megabits. I get a solid 50megabits down and 10megabits up. I think this should be sufficient to get the system up and running with a possible upgrade in the future.

Having said that, I am usually working with files less than your 600MB benchmark because they are all webpage elements.

Sorry for the confusion, thank you all for your advice. I've been carefully reading each post and appreciate your continued support.

Click to expand...

ah right sounds better, what are your actually type/size of files if you could do a average?

mackpro said:

In my post I was referring to megabytes not megabits. I get a solid 50megabits down and 10megabits up. I think this should be sufficient to get the system up and running with a possible upgrade in the future.

Click to expand...

Sure, that sounds sufficient for your needs.

I assume this is "business" service. If not, best to check your contract to make sure your proposed usage is consistent with the contract terms. Asymmetric bandwidth often comes with a "no servers" policy.

Matty-p said:

so op your from ameria right? here were i live in the uk bisiness have generally leased lines that is dedicated symetric bandwith for them only and its garenteed. here the general costs are for dedicated 10 meg down 10 meg up about £400 a month over copper pairs, or depending where you are about £600 for 100mb down 100mb up dedicated over fiber. id figure out the difference betwwen a eqiv connection in america and your cost of your current curcuit and try budget that in too. 10 meg up 10 meg down will be plenty.

Click to expand...

Yes, S. California.

Colo is still cheaper, at least here.

For example, Hurricane Electric offers a FULL CABINET (42U) in San Francisco with 100mbps uplink speed for $600 month.

Ubiquity offers 4U at $125/month, 1/4 cabinet at $475/month both with 100mbit/sec uplink in Los Angeles. 1/2 cabinet at $850/month (about the same as the amount quoted above for a leased line alone) but with 1000 mbit/sec uplink). I don't imagine the OP would need more than 1/4 cabinet (10U), and could probably fit this into 4U.

Note "bandwidth" quota is much lower (1mbit/sec for the 4U, 5mbit/sec for the 1/4 cabinet) than the uplink speed. You can burst to 100 or 1000, but, of course, if you sustained that rate constantly, you'd be nicked for the extra bandwidth. (It's actually rather more complicated than that.) In the case of the OP, it doesn't seem likely he'd exceed the base charge. Look up "95'th percentile billing", which is standard, at least here in the U.S.

For the OP, the existing connection seems to make sense, since he already has the leased line, presumably to support in-house user's Internet access.

However, for others that might have a server need that would not be easily met by their existing leased line, (or might violate terms of service) colo might make much more sense.

The reason it is so much cheaper is because it takes the leased line completely out of the equation. There's just an Ethernet cable to another room. The expensive "last mile" is eliminated.

I've quoted some U.S. prices, but I'd assume the economies are similar in Europe. Delivering bandwidth to your business costs a lot more than delivering bandwidth to a data center.

Matty-p said:

Yea I was suggesting Linux or Unix for the supermicro build,
3tb are a bad idea for a pruduction environment cos there less reliable slower have far longer rebuild time for raid are more expensive the list goes on

Click to expand...

I disagree, if you followed that logic then why stop at 1 TB or 2 TB, why not build on 72 GB SCSI. "Production Environment" is a little bit of a arbitrary distinction, this is a small business with relatively low parallel usage on a tight budget, 3 TB drives would be perfectly suitable. Real high performance "production environments" would be using 15k FCAL, SCSI or SAS in enterprise chassis, but we obviously don't have the budget for that.

RAID rebuild time is going to be more affected by RAID-6 instead of RAID-5, by software RAID instead of hardware RAID than by 3 TB vs 2/1 TB. The double parity overhead of RAID-6 makes it very slow, but for a budget 30 TB filesystem I can't think of a better tradeoff.

Advantages of 3 TB drives - chassis needs fewer bays, future expandability with the same bays is better, administration is easier (fewer drives, fewer potential points of failure), power draw is significantly less - if you've shopped for colo's lately you'll find that power is pretty expensive these days, moreso than rackspace or arguably bandwidth.

RAID-5 would be ridiculous especially if you were to go with 31 1TB drives, that would be so risky it's ludicrous. Two drives go out and you just lost all your data. Thats 3% of your drives can fail and you're SOL.

RAID-6 with 12x 3TB drives you can lose 2 drives before you lose data, or 17% of your drives. Much safer. Thats 2 out of 12 instead of 1 out of 16 or 31. I think the advantages of 3TB are clear.

Rob

build your own

Might be worth a read