PDA

View Full Version : I need some help. I have 20 TBs worth of storage; and I need more. Need help!!




WiiDSmoker
Sep 21, 2010, 08:28 AM
Currently I have 20 Terabytes worth of storage. 10 2 Terabyte drives. They are all completely full. I'm estimating that I need about 35-40 more Terabytes to complete my collection. Unfortunately I haven't even been able to begin to back things up; which I'm worried about, but not currently at this time.

I've contemplated running a bunch of 5 Bay NAS's; but I'm thinking I'd like to try something else.

Is there anyway I can get a long box that can work like a NAS; but hold 10-20 SATA drives. Or something!

I need some ideas. Please help :(



OllyW
Sep 21, 2010, 08:33 AM
Something like this (http://www.amazon.com/EliteNAS-EN316L-BXE-Hardware-Rackmount/dp/B00342UTAC)?

http://ecx.images-amazon.com/images/I/41WMuE4MXQL._SS400_.jpg

reebzor
Sep 21, 2010, 08:38 AM
I required more storage so I built an 8Bay freenas box, specs are in my sig and it cost me around $700. Your cheapest option is going to be finding a PC case (most likely going to be a rack mount server case) that can hold more than 10 drives, build a PC inside of it and throw freenas on it.

Not only will you need a case that can support the drives, but you'll need a raid card as well. I bought a Highpoint card, which I love, so I'll recommend it to you as well. I know they have a bunch of cards that support more than 10 drives, so check them out.

Keep in mind I'm assuming that someone that requires 50TB of storage is also somewhat technically minded. Picking out parts and building a custom NAS server can be pretty tricky so if you have no idea what I'm talking about make sure you do a lot of research.

Seydlitz
Sep 21, 2010, 09:41 AM
Currently I have 20 Terabytes worth of storage. 10 2 Terabyte drives. They are all completely full. I'm estimating that I need about 35-40 more Terabytes to complete my collection. Unfortunately I haven't even been able to begin to back things up; which I'm worried about, but not currently at this time.

I've contemplated running a bunch of 5 Bay NAS's; but I'm thinking I'd like to try something else.

Is there anyway I can get a long box that can work like a NAS; but hold 10-20 SATA drives. Or something!

I need some ideas. Please help :(

You will need cash... even the freenas way is expensive. I would say it's almost impossible to do better than just buying off the shelve. The reasons why it will be hard/expensive.:

- You will need a special controller PCI card with the extra connectors. These are expensive.
- You will need to buy special server HDD's. It can be done with consumer grade HDD's, but it's playing with fire (literally , it gets HOT)
- You will need a very big, well cooled case.
- You need a server grade PSU (actually you want two PSU's, it's a redundancy factor)
- 40 TB means that you will need approx 60TB just to create a Raid array... and that's still not a backup. I would definitely recommend at least raid 5, preferably raid 6. The reason is that even if you backup this array you want some redundancy (with these amounts of data you probably won't have a completely real-time backup, the reason is in the following post).
- You want two arrays... and preferably the second one off-site (a data-center? Family? Friends?). Loosing so much data is just inconceivable, so you want to create a backup, hence the second array. Off-site is the safest way, just do an incremental sync at night (this is why you might not have a completely synced backup during the day). If it's in your house you can just do a 1-1 sync... but you are not disaster proof. Pick your poison.
- You need to have a powersupply that can cater that many SATA power connectors. Once again you want to have a server grade powersupply.
- You need to setup freenas so you can sync with ISCSI or off-site. This might be challenging, though there will be people who have done this.

To be honest: unless you are really a techie who loves to tinker... not a very practical solution. The realistic option would be to buy a two 8U 19" racks and host one of it at home and one of it in a data-center. But be ready to fork out 10K for each array......

darkgroove
Sep 21, 2010, 10:20 AM
This may not answer your question at all, but what exactly is filling up all that space? I have a feeling it will be full DVDs and Blu-Rays.

In cases like this, it almost makes more sense to just rip the stuff out of those files that you don't need. The savings can be huge (although the time you will spend will also be huge).

Most big budget DVDs have several other languages and special features you will never watch. Save the full DVDs for movies you don't want to lose anything for, but just save the movie for other films.

But yeah, that amount of storage is currently in the realm of, "you will need to figure it out yourself" and no easy solution exists. Sorry kid.

belvdr
Sep 21, 2010, 10:48 AM
We used a few of these at work. About $50k USD:

http://www.nexsan.com/satabeast.php

iowamensan
Sep 21, 2010, 10:50 AM
Sounds like you need help in the same way the people on Hoarders need help. I have 2 TB on my iTunes server, and that is holding 670+ movies and about 600 TV shows. If you are needing like 60TB, you are talking about thousands upon thousands of movies. 10,000+. If you are buying these legally, you are looking at well over $100,000 worth of movies, so you would apparently have enough money to buy whatever kind of storage you think you need. If you aren't filling these TB's legally, (which I'm pretty certain is the case) I'm going to keep my eye on the news waiting for the largest piracy case in history.

darkgroove
Sep 21, 2010, 11:14 AM
Sounds like you need help in the same way the people on Hoarders need help. I have 2 TB on my iTunes server, and that is holding 670+ movies and about 600 TV shows. If you are needing like 60TB, you are talking about thousands upon thousands of movies. 10,000+. If you are buying these legally, you are looking at well over $100,000 worth of movies, so you would apparently have enough money to buy whatever kind of storage you think you need. If you aren't filling these TB's legally, (which I'm pretty certain is the case) I'm going to keep my eye on the news waiting for the largest piracy case in history.

I would agree that, regardless of how you got the movies, you are keeping 90% of what you will probably never watch again. Every now and again I go through my 5TB collection of movies and shows (which is pretty epic in my opinion) and prune out the shows that I thought I wanted to keep but really don't.

emvath
Sep 21, 2010, 11:33 AM
I was all set to come on here and stick up for the OP by saying that you shouldn't make assumptions about what is filling up his harddrives. It could well be that he is into digital video creation which can fill up terabytes fast, or be a photographer with tons of raw photos. Whatever, the point is that it is possible that these aren't pirated movies.....

But then I saw that his username was WiiDSmoker :rolleyes:

ChrisA
Sep 21, 2010, 12:15 PM
...
Is there anyway I can get a long box that can work like a NAS; but hold 10-20 SATA drives. Or something!

I need some ideas. Please help :(

Assuming that you really do need all this data and that your need for storage will continue to grow over the coming years. You are just going to have to make the jump to a professional level storage system. I'm very partial to Solaris based servers because of their ZFS file system. But if you'd rather not go that way because of the learning curve assoicated with Solaris you can use Linux. Either way you end of with a NAS system that has the performance to steam many real time HD video streams oover gigabyte Eithernet cable(s)

You will need a high performance server and a robust stagage system and a large UPS and a rack tomount it all in. You might need to think about cooling but it will not really be that big of an issue onless you put the rack on a closet that is sealed.

We use EMC/Clarion here. They are cost effective and reliable. But do configure RAID and multiple hot spare drives. The system will fail over to the spares and you can replace the failed drives whenever you get around to it



http://www.emc.com/products/launch/clariion-ax4/index.htm
Prices are very roughly on order of $1K per TB of configured starage. Yes, higher then the price of bare disk drives.

WiiDSmoker
Sep 21, 2010, 12:39 PM
Something like this (http://www.amazon.com/EliteNAS-EN316L-BXE-Hardware-Rackmount/dp/B00342UTAC)?

http://ecx.images-amazon.com/images/I/41WMuE4MXQL._SS400_.jpg

I'm actually looking into the Norco 4220 (http://www.newegg.com/Product/Product.aspx?Item=N82E16811219033&Tpk=norco%204220). It's only $299 and it can hold 20 Hot-Swappable SATA/SAS Drives.

Sounds like you need help in the same way the people on Hoarders need help. I have 2 TB on my iTunes server, and that is holding 670+ movies and about 600 TV shows. If you are needing like 60TB, you are talking about thousands upon thousands of movies. 10,000+. If you are buying these legally, you are looking at well over $100,000 worth of movies, so you would apparently have enough money to buy whatever kind of storage you think you need. If you aren't filling these TB's legally, (which I'm pretty certain is the case) I'm going to keep my eye on the news waiting for the largest piracy case in history.

All my DVDs and Blu-Rays are 100% legit; store bought. The problem I have, is that I want to keep all movies in their ISO format as well as encoded into mkv/m4v.

I was all set to come on here and stick up for the OP by saying that you shouldn't make assumptions about what is filling up his harddrives. It could well be that he is into digital video creation which can fill up terabytes fast, or be a photographer with tons of raw photos. Whatever, the point is that it is possible that these aren't pirated movies.....

But then I saw that his username was WiiDSmoker :rolleyes:

Nope. Just 100% movie rips. WiiDSmoker is a play on words. Nintendo Wii. Nintendo DS. And of course weed and smoker :p


I'm looking into getting an Atom based Motherboard with a 560 watt 45-50 amp Powersupply. And then I would need to get PCI Express SAS/SATA Raid cards.

HobeSoundDarryl
Sep 21, 2010, 01:31 PM
A couple of unraids (perhaps using that Norco box(es))? That can be fairly cheap compared to other solutions listed here, and is pretty good for media storage. Expand it as you need it without rebuilds.

Also, if the OP owns all those discs, why keep an ISO copy too? Why not dramatically reduce his storage requirements with the MP4/M4V versions, and just go get the disc when he wants the rest?

Lastly, if he does own that many discs, money is not really an object. If that's the case, maybe unraid (the poor man's big server) is not the way to go. Just spend big and buy big and best.

reebzor
Sep 21, 2010, 02:56 PM
I'm actually looking into the Norco 4220 (http://www.newegg.com/Product/Product.aspx?Item=N82E16811219033&Tpk=norco%204220). It's only $299 and it can hold 20 Hot-Swappable SATA/SAS Drives.

I'm looking into getting an Atom based Motherboard with a 560 watt 45-50 amp Powersupply. And then I would need to get PCI Express SAS/SATA Raid cards.

Just for some comparison, I have a norco 2208, a 500w psu, and an i3 2.93ghz. Although I havent really come close to maxing out the i3, I would let an Atom run that big of a storage array. I have the Highpoint 2720 card which supports up to 8 6GB/s SAS/SATA drives but you'd need to either get a few of them, or just one card that supports 20 drives. Check out the 2760 (http://highpoint-tech.com/USA_new/series_rr2700.htm)

VoR
Sep 21, 2010, 03:06 PM
Lot's of case options available to you if you search around - certainly doesn't need to cost silly money.

I think solutions like freenas/nexentastor/unraid work just as well (better) as the expensive commercial solutions - they just come with less (no) support and aren't bought by an accountant through an enterprise contract - way to generalize :)

Personally, I would use (and do) go for a bsd/solaris based machine and use zfs - which also avoids the huge expense of a raid card.

danpass
Sep 21, 2010, 03:16 PM
I use an offsite server ............................. I call it Netflix :D

WiiDSmoker
Sep 21, 2010, 03:31 PM
Just for some comparison, I have a norco 2208, a 500w psu, and an i3 2.93ghz. Although I havent really come close to maxing out the i3, I would let an Atom run that big of a storage array. I have the Highpoint 2720 card which supports up to 8 6GB/s SAS/SATA drives but you'd need to either get a few of them, or just one card that supports 20 drives. Check out the 2760 (http://highpoint-tech.com/USA_new/series_rr2700.htm)

Thanks for the info man. That card looks awesome. May I ask what motherboard you are using for that i3? Is it onboard video?

BertyBoy
Sep 21, 2010, 03:48 PM
You're into the realms of needing a tape management system and tape robot. You allocate so much space for permanent storage of files and another large area for stuff to be restored from tape. Either with a strict retention policy, ie files returned from tape are only kept on disk for 24 hours / 7 days, or when a file is restored to disk, it overwrites the oldest files on the disk.
A tape robot and tape management system should be readily available on the 2nd user market, if you can get them before they go to India / China.
The big firms just lease them from the big suppliers.

edk99
Sep 21, 2010, 07:15 PM
If you are thinking about the Norco here is a good site to look at if you haven't seen it already. http://hardforum.com/showthread.php?t=1498268

DoFoT9
Sep 21, 2010, 07:30 PM
All my DVDs and Blu-Rays are 100% legit; store bought. The problem I have, is that I want to keep all movies in their ISO format as well as encoded into mkv/m4v.

problematic. what mediums do you play your movies on? computer connected to TV? etc.... maybe there is a work around.

assuming you have the money, you would setup your main RAID system - preferably a parity based RAID, RAID5 if you want speed, RAID6 if you want redundancy.

had you considered online backups? Amazon or similar? they charge $0.15 per GB. 50TB would cost $7500! holy smokes haha!

reebzor
Sep 21, 2010, 10:38 PM
Thanks for the info man. That card looks awesome. May I ask what motherboard you are using for that i3? Is it onboard video?

Just a cheap ECS H55H-CM board. I do use the onboard video but 99.9% of the time I access the server from the webconsole. If FreeNAS intrests you check out their forum or you can PM me if you have more specific questions.

ChrisA
Sep 21, 2010, 11:47 PM
I'm actually looking into the Norco 4220 (http://www.newegg.com/Product/Product.aspx?Item=N82E16811219033&Tpk=norco%204220). It's only $299 and it can hold 20 Hot-Swappable SATA/SAS Drives.



All my DVDs and Blu-Rays are 100% legit; store bought. The problem I have, is that I want to keep all movies in their ISO format as well as encoded into mkv/m4v.



Nope. Just 100% movie rips. WiiDSmoker is a play on words. Nintendo Wii. Nintendo DS. And of course weed and smoker :p


I'm looking into getting an Atom based Motherboard with a 560 watt 45-50 amp Powersupply. And then I would need to get PCI Express SAS/SATA Raid cards.

Atom based server??? Most low cost RAID cards are actually software RAID. The CPU will need to be able to handle RAID calculations and stream the data out the network and possible re-build a faild drive all at the same time. Most people who build these systems are using at least dusl CPU systems. YOu really don't want t a low end CPU. Think about backups it will take hours or even days to back this think up and alll that time the CPU is "cooking" and I assume you'd still like to be able to access you data while the backup goes on.

Next think about software. What file system will you use, volume managment. You will nedd a battery backup system. All those drives have 16 or 32 MB of RAM cache and it is lost when power fails with 20 drives you can have 1/3 GB is RAM at any one time.

It should be obvious we are talking about a 64 bit CPU and 64-bit OS

Flash SWT
Sep 22, 2010, 01:37 AM
Sounds like your first step is to set a budget and then decide if you want a do-it-yourself option like FreeNAS or you want a commercial solution. Either way it sounds like a fun project and I hope you'll keep us updated as you go!

If you want to go the NAS route you can definitely do better then a bunch of 5 disk units. Here is one example of a 12 disk NAS:
http://www.readynas.com/?cat=73

.

mstrze
Sep 22, 2010, 12:22 PM
Devil's advocate:

You are keeping the ISO versions of your discs as a backup I am guessing, but surely the DVD and BluRay medium you ripped them from is a MUCH more stable medium than hard disc?

Either go the tape route someone mentioned and just put those ISO files on them and store them off your local drives or just be like most other sane people and get rid of the duplicate ISO files. 40TB is an insane amount of storage space, especially when 80% or more of it is something that you really are not even going to be using again.

I'm sure in 15 years or so computers will have 40TB of storage standard, but right now it's an overwhelming and unnecessary amount given your situation.

If you have the money and patience for it, knock your socks off I suppose, but you really should rethink your methods...especially since you haven't even been able to back up your drives up to this point. I assume you are keeping the ISO files in case something happens to the originals. Heck, you could lose all of that in an instant and STILL have to go back to your original source discs if the drives fail.

codymac
Sep 22, 2010, 12:34 PM
Most people who build these systems are using at least dusl CPU systems.

Personally, I've come to the conclusion that something like a discounted low-end HP DL180 from a VAR starts to quickly make more sense than a scratchbuild.

Hellhammer
Sep 22, 2010, 12:43 PM
had you considered online backups? Amazon or similar? they charge $0.15 per GB. 50TB would cost $7500! holy smokes haha!

Transferring 50TB at 10Mb/s would take roughly 463 days :rolleyes:

PeterTk
Sep 22, 2010, 01:25 PM
You need to build one of these:

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

emaja
Sep 22, 2010, 02:31 PM
You need to build one of these:

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

I think I just had a "man moment" in my pants - LOL!

DoFoT9
Sep 22, 2010, 04:12 PM
Transferring 50TB at 10Mb/s would take roughly 463 days :rolleyes:

its a suggestion, maybe the OP has a fast fibre connection? brings it down to.. ohh.. a month! :D

You need to build one of these:

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/
in this case, thats a VERY good suggestion! nice.

JasperJanssen
Sep 22, 2010, 05:20 PM
At the low end in premades, expect to pay double the raw drive cost for your system.

Atom is seriously not worth your while.

At the low end, you will want to run your RAID in software.

If you get one of very slightly tricked-out motherboards with a dual-core intel, you should have at least 6 and maybe up to 10 SATA ports in the motherboard, and it should be possible reasonably cheaply to get a board supporting up to 4x pcie 4x-16x slots.

Supermicro AOC-SAT2-MV8 is a PCI-X 64/66 card (will work in a PCI 32/33 slot) with 8 SATA ports. ~ 10-15 per port.

Supermicro AOC-SASLP-MV8 is a PCIe 4x (will work in PCIe 16x) card with 2 4x SAS ports. Including the breakout cables to 4x SATA, 8 ports will be slightly more expensive than the SAT2, but the increased controller bandwidth will make up for that. ~15-20 per port.

Sharkoon Rebel12 Value. With very minor modding (bending some steel out of the way at the top) this case will hold 12 5.25 bays, cheaply.

Coolermaster 4in3 -- using 3 5.25 bays, this holds 4 drives, with a 12 cm fan for airflow. Better airflow/cooling/noise/price performance than the option below. ~ 4-5 per HD.

Cheap 5-in-3 hotswap SATA bay -- in the same space, put 5 drives, and they're hotswappable. ~ 20-25 per HD.


Putting that all together, you get 16 or 20 drives in each Sharkoon case, and you can bolt them together with a single motherboard plus one or two Supermicro cards.

You can also get two Sharkoons, bolt them side by side, and route all the drives to a single motherboard. That would be up to 40 drives or at current capacities 80 to 120 tera, requiring 4 supermicro cards plus the motherboard.

Pricing it out (very roughly, high end estimates):

Small option, 32 TB in 16 drives:

PC parts including an 8x SATA motherboard: 400
SSD boot drive: 100
Power Supply, comma, hefty (but one will do): 100
Sharkoon case: 70
4x Coolermaster 4in3: 60
Supermicro card plus cables: 150
16 drives: 2000

32 TB NAS for 2780 + minor effort.

Bigger option, 80 TB in 40 hotswappable drives:
PC Parts including an 8x SATA, 4x pcie16 motherboard: 500
SSD Boot drives for mirroring: 200
Power supply, comma, hefty, comma, two of them: 200
2x Sharkoon case: 140
8x hotswap 5x SATA cage: 1000
4x Supermicro card plus cables: 600
40 drives: 5000

80 TB NAS for 7640 plus a bit more effort.

Mind you, these are raw capacities, I would recommend you make raid6 sets of 8 drives, losing about a quarter raw capacity, and maybe raid0 stripe those together, so 24/60 TB formatted capacity.

HobeSoundDarryl
Sep 22, 2010, 05:56 PM
For media storage where speed should not be that important, why not unraid: http://lime-technology.com/wiki/index.php?title=Overview and http://lime-technology.com/

Macworld says this: http://www.macworld.com/article/146120/2010/02/unraid_server.html

He could use that 20-bay Norco for the box. Maybe 2 of them?

I keep seeing all these recommendations for various forms of RAID, but why would he want to link multiple drives if speed wouldn't matter that much (after the initial loading of all that data onto them)?

I also argue that if he actually owns the DVDs/BDs for all that media, why not let them be his master copies for backup and not put the ISOs on there? That would dramatically downgrade the storage needs while still holding the digital copy he would use most of the time on a much smaller server.

There's also the 67TB option for less than $8K: http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ Macworld says this: http://www.macworld.com/article/142653/2009/09/backblazestoragepod.html I wonder if you used 2TB drives instead of 1.5 like they show, could this thing yield 90TB?

dlegend
Sep 22, 2010, 06:29 PM
Just out of curiosity, how many dvds and blu rays do you have?

ChrisA
Sep 22, 2010, 11:41 PM
I keep seeing all these recommendations for various forms of RAID, but why would he want to link multiple drives if speed wouldn't matter that much (after the initial loading of all that data onto them)?..


Everyone who has never done this thinks it's easy. It is at first and the managment of the storage starts to take hours out of your week.

The Reason for RAID is reliability is easy management. Lets say a SATA disk has an average life of four years. This means that if you own four disks, on average one will fail about every 12 months. If you own 40 one wil fail every 8 weeks. This guy, with his primary and backup system will have 40 drives. Every 6 or 8 weeks on average he
will have to deal with loosing a tarabyte of data. (assuming 1TB drives)

Maybe I should say that again to let it sink it. The drives will fail routinly at last one a month and each will take with it 1TB of data. The more drives the higher the number of failed drives per unit of time.

Why RAID? Because RAID can recover from failure automatically without the admin having to be there and without need to stop the system and repair it. You simply load it up with hot spare drives that are empty and let the software do the re-build and when the number of hot spares is low you swap bad for good drives. When I was testing storage system for a dot com I worked for I'd simulate a disk failure by opening a chassis removing the power connector from a drive. I'd first load up the system with a big symulated work load from a database and web server and then break stuff. The system could continue just fine with multiple disk drives being killed. I even removed one CPU chip from a running server. The crashed a server but not the web site. When you have 100 of anythig you are going to have a few of those broken at any one time and if you design the system crrectly a broken disk or broken server is not an emergency.

Te article was wrong about the CPU requirements. Rebuilding a RAID is lots of work and takes hours and you'd like to continue streaming data just like normal while the repair continues.

So why do most people gladly pay $10K for a storage system? Because their time is valuable AND because their data is worth 20 or 1000 times the $10K

All that said does anyone over the age of 8 watch the same movie twice? Who has time to watch that much TV? This project seem to be something you'd want if you were a cable company or y a library

JasperJanssen
Sep 23, 2010, 06:15 AM
For media storage where speed should not be that important, why not unraid: http://lime-technology.com/wiki/index.php?title=Overview and http://lime-technology.com/

Macworld says this: http://www.macworld.com/article/146120/2010/02/unraid_server.html

He could use that 20-bay Norco for the box. Maybe 2 of them?

I keep seeing all these recommendations for various forms of RAID, but why would he want to link multiple drives if speed wouldn't matter that much (after the initial loading of all that data onto them)?

I also argue that if he actually owns the DVDs/BDs for all that media, why not let them be his master copies for backup and not put the ISOs on there? That would dramatically downgrade the storage needs while still holding the digital copy he would use most of the time on a much smaller server.


For one thing, unraid is ludicrously expensive -- $150 for up to 16 drives is the biggest package mentioned, which means that this particular guy would have to negotiate his special needs for an even more expensive package.

There are several reasons for raid. One is speed, but frankly, this is irrelevant in this case. The most important one is consolidation of multiple drives into a single filesystem. You do *not* want to manually distribute data over 40 drives. If the storage is pure archival, I suppose you *could* reasonably easily stuff the drives manually and hit something like 80-95 percent capacity, but that would be at the complete expense of any type of organisation whatsoever. A single filesystem is *much* easier to deal with.

The other reason is fault tolerance. It's not *entirely* the case that failures are distributed evenly over time. Consumer-grade HDDs, when they have specs for this, tend to be specced at between .8 and 1.2 million hours MTBF. That would mean that with 40 drives, you should expect the first fault at 20.000 hours, give or take, which is 2.3 years. However, with 400 drives, you would expect a drive failure every 2000 hours, ie every 2-3 months.

Once the drives hit 3-5 years you will start seeing actual lifetime related failure rather than the type of failure measured by MTBF.

Now, that's pure math -- with 40 drives, which is big enough for statistics to take effect, I would expect at least a 10 percent mortality rate in the first year. So you *will* be dealing with drive failures regularly. If your data is unimportant enough to be lost without recovery at that rate, you might as well just chuck it now. If you're willing to re-rip 2 tera worth of data every 3 months.. Well, you might as well just chuck the ISOs now and re-rip when you need them instead.

The question isn't "Why RAID?", it's "what kind of RAID?"

Something to get out of the way first: JBOD/RAID0, just say no. Either one would allow you to consolidate the disks into one file system, but any drive failure would trash the entire system, which Would Be Bad.

RAID1: Well, it's secure and easy, but losing fully half your capacity smarts. Even if you do do it, you will probably want to run two twenty-disk raid5/6 +hotspare arrays and mirror those, rather than mirroring individual drives and then striping or concatenating them, or mirroring two giant stripes/concats.

Which leaves raid5/6 and its variants.

My gut feeling is that 8-drive raid6 arrays still sounds decent, for the traditional raid setup, but out of forty drives you should probably count on at least one if not two hot spares simply so the rebuild will start immediately and there is less (but not remotely zero) chance of multiple drive failures taking out the array.

The backblaze approach, with 45 drives off 9 5-port port multipliers, is a pretty darn good one for this type of system. I've looked at PMs in the past, but have never managed to find them available anywhere for consumers to purchase, backblaze seems to have solved that problem for the world.

FreeNAS with ZFS is probably the best easy, non-customized way to manage a system like this. If you have 45 devices, I'd say 3 hotspares and 3 pools each with 14 devices in z2 or 3 sounds right, put together into one tank. That leaves 36 or 33 devices worth of effective storage.

You're still going to have to do backups, though, because if something unforeseen happens that causes some of the wrong disks to go out simultaneously, your data is toast.

HobeSoundDarryl
Sep 23, 2010, 07:11 AM
I appreciate both responses. Thanks. I'm not sure though that the pro-raid arguments reflect an understanding of the unraid solution.

For example, with unraid you use user shares to consolidate physical drives into single (virtual) volumes. It presents itself as if they were set up as raid0 when, in reality, there are 2+ drives each set up independently of each other (the independence is invisible, as 2+ drives consolidated into a user share look like 1 bigger drive). When a drive fails, the other volume(s) are still "whole", meaning they can be accessed because their contents are not spread over multiple discs (like raid) such that if one drive goes down, some of their content in each & every file goes down with it.

Those raid options with built-in backup (like raid 5) are great because you can swap in a new drive and the raid can rebuild itself (resurrecting the lost drive onto the new drive). Unraid uses a dedicated parity drive to do the same. Still, the difference is that in Unraid you store one of the movies or ISO on one physical volume, not spreading it over multiple volumes. In a catastrophic event where more than one drive is lost before you notice, RAID will lose EVERYTHING, while unraid will lose only the movies stored on the drives that failed- all the other drives will still be accessible (and the content they hold "whole).

In this case, the guy already has a dedicated physical backup. He claims to own all these movies on disc. Assuming that's true, I would take that to mean that raid-like backup is not a major priority. He can always use the physical discs to resurrect the system in a catastrophic loss. However, it doesn't really matter. Both raid and unraid have backup options to restore a dead hard drive in the array. Raid however loses everything if 2 drives manage to go down. Unraid will still have access to all but what was stored on those 2 drives.

Unraid volumes don't sacrifice some space on every drive to support the backup. 2TB is 2TB in unraid. One disc gets dedicated as a Parity drive (a backup for all discs in the array), but otherwise, all hard drives have all of their space fully available for storage. Raid has to give up a chunk of space on each drive to make it's backup model work. This may be a "6 of 1, half dozen..." argument, but it might matter- more so in this case, where backup might be covered by physical movie discs on hand.

The unraid tradeoff is speed. Unraid works with single volumes at a time, so read/write speeds are limited to individual hard drive read/write speeds, with writes slowed a bit more for the parity drive to adjust it's setting to backup new files that are added. But this particular application is great for that, as speed of read/writes is not crucial; most of the time, he would be reading- not writing- movie files. Once the parity drive is set up and all his movies are in place, writes would be limited to when he adds a new movie. Most of the time, he should be reading from the array.

The implication that RAID can recover via swapping in a replacement drive and unraid can't is wrong. Unraid will rebuild a lost drive too when you swap in a new one.

The implication that unraid would show every drive as a separate volume is wrong. It lets you decide how many drives to bundle together into a single "virtual" volume (called a user share). In this case, the OP could bundle all the drives into a single volume, though it might make more sense to bundle them as M4V and ISO volumes, or maybe even categorize them in other ways (movies, TV shows, photos, etc or maybe by genre or popularity).

Another big advantage to unraid is that you can mix and match drive sizes. If the OP starts with 2TB drives today but then wants to add some >2TB drives (when they are released) tomorrow, no problem. If he has some <2TB drives laying around with nothing to do, no problem. If he has some old IDE drives gathering dust, no problem. unraid will take, mix and match sizes (and drive speeds) and add that to the array.

The implication that unraid is expensive compared to Raid solutions is most mysterious of all. I have been looking for a big storage solution myself for a very long time. One of unraids best points is the very low cost of it. I can't hardly find big storage raid-based solutions that don't seem to always start at fairly big bucks for just the driveless hardware to start. Since the cost of the drives should cancel themselves out in either solution, I'd love to see- say- a 30 drive raid box beat 2 15-drive unraid boxes on driveless box cost. While you can do better than the pricing on the unraid website for 15-drive boxes- especially if you build your own- 2 of their turnkey, ready-to-load (with up to 30 drives) boxes can cost as low as $1199 each. Could someone point me to a good 30+bay Raid 5/6 driveless box available for less than $2500? Note, this is not asking for just a case... I'm talking about something more apples-to-apples, as in plug in some drives and you have a storage server (not plug in some drives, and cards, and memory, and power supplies, etc).

And there would be no custom cost negotiations, the OP could just use a couple of 15-bay boxes to meet his needs. Trying to do this in one bigger box (all the bays he needs) via raid is going to be a lot more expensive. But if anyone can show me that that is not true, I'd appreciate finally seeing a raid solution big enough for this kind of need that will come in at lower cost than a couple of Unraids. I've been looking for that a very long time myself.

Lastly, the new version of unraid apparently will work with 20 drives- a perfect match for the OPs desire for something like that Norco box. 2 of those could yield 80TB of gross storage. Allocate 2 parity drives (one for each box) and the net will be very near 80TB of gross storage too, in just 42 drives. How many drives would it take to accomplish the same via one of the raid with backup options?

No doubt about many advantages of raid- especially speed- vs. unraid. But this guys application is more of a write once: read many times thing, where he apparently has hard copy backups for all content on hand anyway. Other than that first transfer of data onto the server, write speed should not be important to him. And serving movies is probably a 1-movie-at-a-time (or maybe a few movies at a time) proposition, so read speed is no big deal either. Since unraid read speeds are pretty much limited to hard drive read speeds, the experience should be as fast as storing a movie on a Mac hard drive now and streaming it (aka plenty fast, for this application).

WiiDSmoker
Sep 23, 2010, 08:48 AM
At the low end in premades, expect to pay double the raw drive cost for your system.

Atom is seriously not worth your while.

At the low end, you will want to run your RAID in software.

If you get one of very slightly tricked-out motherboards with a dual-core intel, you should have at least 6 and maybe up to 10 SATA ports in the motherboard, and it should be possible reasonably cheaply to get a board supporting up to 4x pcie 4x-16x slots.

Supermicro AOC-SAT2-MV8 is a PCI-X 64/66 card (will work in a PCI 32/33 slot) with 8 SATA ports. ~ 10-15 per port.

Supermicro AOC-SASLP-MV8 is a PCIe 4x (will work in PCIe 16x) card with 2 4x SAS ports. Including the breakout cables to 4x SATA, 8 ports will be slightly more expensive than the SAT2, but the increased controller bandwidth will make up for that. ~15-20 per port.

Sharkoon Rebel12 Value. With very minor modding (bending some steel out of the way at the top) this case will hold 12 5.25 bays, cheaply.

Coolermaster 4in3 -- using 3 5.25 bays, this holds 4 drives, with a 12 cm fan for airflow. Better airflow/cooling/noise/price performance than the option below. ~ 4-5 per HD.

Cheap 5-in-3 hotswap SATA bay -- in the same space, put 5 drives, and they're hotswappable. ~ 20-25 per HD.


Putting that all together, you get 16 or 20 drives in each Sharkoon case, and you can bolt them together with a single motherboard plus one or two Supermicro cards.

You can also get two Sharkoons, bolt them side by side, and route all the drives to a single motherboard. That would be up to 40 drives or at current capacities 80 to 120 tera, requiring 4 supermicro cards plus the motherboard.

Pricing it out (very roughly, high end estimates):

Small option, 32 TB in 16 drives:

PC parts including an 8x SATA motherboard: 400
SSD boot drive: 100
Power Supply, comma, hefty (but one will do): 100
Sharkoon case: 70
4x Coolermaster 4in3: 60
Supermicro card plus cables: 150
16 drives: 2000

32 TB NAS for 2780 + minor effort.

Bigger option, 80 TB in 40 hotswappable drives:
PC Parts including an 8x SATA, 4x pcie16 motherboard: 500
SSD Boot drives for mirroring: 200
Power supply, comma, hefty, comma, two of them: 200
2x Sharkoon case: 140
8x hotswap 5x SATA cage: 1000
4x Supermicro card plus cables: 600
40 drives: 5000

80 TB NAS for 7640 plus a bit more effort.

Mind you, these are raw capacities, I would recommend you make raid6 sets of 8 drives, losing about a quarter raw capacity, and maybe raid0 stripe those together, so 24/60 TB formatted capacity.

Wow. Thank you for all this information! You're certainly a rare gem :)

Hellhammer
Sep 23, 2010, 08:55 AM
its a suggestion, maybe the OP has a fast fibre connection? brings it down to.. ohh.. a month! :D

You need a damn fast fibre then. As you know, we got some speedy fibers here and the fastest upload speed you can get is 10Mb/s while fastest download is 200Mb/s. Even if you had EXTREMELY fast upload, e.g. 100Mb/s, it would still take 46 days. IIRC uploads cost more for the ISP thus upload speed is usually lower

AAPLaday
Sep 23, 2010, 09:01 AM
When would you get the time to watch all of those films? I watch maybe 2 a week at most. Many of them aren't worth re-watching :eek:

WiiDSmoker
Sep 23, 2010, 10:13 AM
When would you get the time to watch all of those films? I watch maybe 2 a week at most. Many of them aren't worth re-watching :eek:

I'm a hoarder :(

dlegend
Sep 23, 2010, 10:39 AM
I'm a hoarder :(

I think I am to with some things. How many Dvd's and Blu Rays do you have? I'm really curious.

I hope you're able to figure something out. What are you are currently doing now for the 10 drives that you own? There are 3 TB drives out now, that might keep the number down that you need.

HobeSoundDarryl
Sep 23, 2010, 10:46 AM
Could you send a link pointing to a 3TB drive? Do you mean 1 drive, as in a single 3TB 3.5" internal SATA or similar? I'm not aware that anything over 2TB in one drive has been released.

Cliff3
Sep 23, 2010, 10:53 AM
Could you send a link pointing to a 3TB drive? Do you mean 1 drive, as in a 3TB 3.5" internal SATA or similar? I'm not aware that anything over 2TB in one drive has been released.

Seagate has released a 3TB external drive in their Go-Flex line. It has not yet been released as a bare drive. Anandtech reviewed it recently. Bare drives will be available soon, I am sure.

HobeSoundDarryl
Sep 23, 2010, 11:14 AM
Great, then pairing this impending advance (3TB drives) with my long rant up above about unraid vs. raid... and considering that the OP says he is full at 20TB and needs more, just one unraid box with 15 bays at $1199 or so seems like it would give him up to 45TB of storage.

He could put in maybe 8 of these new 3TB drives, adding 4 more TBs of space to his current situation, then add additional drives as needed. Again, since unraid can use various sizes of drives, if 3TB drives are a stepping stone to 4TB, maybe by the time he was close to filling the 24TBs, he could add 4 additional TBs via drive #9, and still have 6 bays left for future storage growth.

I'm still looking for any of the pro-raid guys to come back with any near-turnkey (all the parts are there- just insert the hard drives) raid box capable of the same, and available for around that $1199 range. I'm genuinely interested is such a thing actually exists.

The OP could also go ahead and build his own with that 20 bay NORCO box, use the 8 or so bays now with unraid "as is", then upgrade to the new version of unraid when it is released so that he is ready to fill in all 20 bays when he gets to that point. Via all 3TB drives, that would triple his current storage in a single box, though again, if he buys what he needs now, the >3TB drives to come could actually grow that same solution into something even more than a 60TB, one-box solution.

From my perspective, this is looking more and more like the way to go for this kind of need. But I (too) would welcome any good alternative via any raid with backup box.

iPhysicist
Sep 23, 2010, 12:18 PM
Currently I have 20 Terabytes worth of storage. 10 2 Terabyte drives. They are all completely full. I'm estimating that I need about 35-40 more Terabytes to complete my collection. Unfortunately I haven't even been able to begin to back things up; which I'm worried about, but not currently at this time.

I've contemplated running a bunch of 5 Bay NAS's; but I'm thinking I'd like to try something else.

Is there anyway I can get a long box that can work like a NAS; but hold 10-20 SATA drives. Or something!

I need some ideas. Please help :(

How about BUYING the media as DV or Bluray Disc? 2000 Blurays would still fit into a nice shelf.

jeremy.king
Sep 23, 2010, 12:34 PM
Get http://www.drobo.com/products/droboelite.php x 6

JasperJanssen
Sep 23, 2010, 01:00 PM
Unraid is presumably organised as a filesystem-level form of concatenation/jbod internally.

I didn't see that it has the option of a parity drive -- that does narrow the gap quite a bit.

Unraid, like FreeNAS or Ubuntu w/software RAID, run on exactly the same hardware -- a PC with a bunch of hard drives attached to it. The difference is that UnRaid apparently costs $150 for up to 16 drives, and will presumably cost more for more drives. FreeNAS (So named because it's FreeBSD based, not for free-as-in-beer) is free-as-in-beer.

For a 40 drive system, unraid probably costs $300+, and there is no cost on the FreeNAS side to offset that, as the hardware is the same.

A raid-based system *can* run in N+1 mode, that's basic RAID5. Does UnRaid support hotspares? It doesn't seem to be a bad solution, I'm just not sure it scales well to the sort of volumes we're talking about. It might work reasonably well for the 16 drive setup, if you're willing to pay the cost.

It's actually a pretty good idea, as far as I'm concerned, but I'm not sure their implementation is worth what they charge for it. In this particular use case, it definitely has its good points.

The current 3 TB drive is a 5-platter, 600 GB/platter monster that slurps power, emits relatively high quantities of heat and noise, and potentially may have reliability issues (due to the above). Using it as an internal drive will also void the warranty. The 2 TB drives today are (AFAIK) all 4-platter, 500 GB/platter devices, available in 5400/5900 rpm "Green" versions, that are probably a better fit. Do take note of TLER though for raid purposes. Most of them don't have it, outside Enterprise/Raid Edition products, which can cause them to drop out of an array at the drop of a hat. Some of them have more problems than others.

What's coming next that's really interesting is 666 GB/platter, bringing 3-platter 2 TB and 4-platter 2.5 TB capacity points. That'll be next year, probably, though. For pretty close to all the other capacity points than 3 TB, 600 GB offers no real advantage over 500 GB platters -- it offers 600/1200/1800/2400/3000. *Maybe* we'll see 4x625=2500 in the intermediate, but it doesn't seem terribly likely. Even 2 TB is still coping with "what the hell do I need that much storage for", so I don't think there'll be many hard drive manufacturers chomping at the bit to put out minor capacity increments.

HobeSoundDarryl
Sep 23, 2010, 01:10 PM
Unraid is presumably organised as a filesystem-level form of concatenation/jbod internally.

I didn't see that it has the option of a parity drive -- that does narrow the gap quite a bit.

Yes, here's good info about that: http://lime-technology.com/technology

Conceptually, I'm pretty impressed with the unraid proposition for this kind of application. It is JBOD-like, but that seems good for this kind of need... especially against future expansion needs as those larger storage capacities come online.

In any event, whether we use current 2TB drives, coming 2.5TB or 3TB drives, it is still looking like a low-cost way to go for this when the hardware box and all the parts within it are included. Everytime I go big raid box hunting (for anything much more than 8 bays), it seems the "good ones" are all starting around $4999 and going up. Just about every time I take a good look around, I seem to keep finding the (potential) "one" best suited for my own needs priced at around $7999.

On the other hand, this unraid thing looks very good in comparison, as long as the application is much like the OPs... especially where write speeds are not very important and read speeds are "good enough" at single hard drive read speeds.

mstrze
Sep 23, 2010, 01:31 PM
How about BUYING the media as DV or Bluray Disc? 2000 Blurays would still fit into a nice shelf.

He owns all of them already, he just feels the need to keep all of the original, full-sized ISO rips on his drives for some reason. :confused:

iPhysicist
Sep 23, 2010, 02:22 PM
He owns all of them already, he just feels the need to keep all of the original, full-sized ISO rips on his drives for some reason. :confused:

OK, but then he already has the best backup - the physical medium itself.

OP if you want to keep it simple, convert them to a quality .mkv and stop this nonsense. Keep all your Originals in a dark and well air conditioned room and they will last you for at least 2 Decades and think over your storage problem again then!

HobeSoundDarryl
Sep 23, 2010, 02:35 PM
OK, but then he already has the best backup - the physical medium itself.

OP if you want to keep it simple, convert them to a quality .mkv and stop this nonsense. Keep all your Originals in a dark and well air conditioned room and they will last you for at least 2 Decades and think over your storage problem again then!

By then it will all fit in his inner-ear embedded iPhone 25- a magical, whimsical, device with smellivision and portable hollodeck technology. ;)

WiiDSmoker
Sep 23, 2010, 02:47 PM
OK, but then he already has the best backup - the physical medium itself.

OP if you want to keep it simple, convert them to a quality .mkv and stop this nonsense. Keep all your Originals in a dark and well air conditioned room and they will last you for at least 2 Decades and think over your storage problem again then!


I know you are right 100% about this, but for my Blu-Rays I at least want to keep them in .ISO for the Boxee Box. As for my DVDs, I am encoding them all to M4V and then deleting the ISOs for them.

In the end, I will still need a significant piece of hardware for my Blu-Ray ISOs.

AAPLaday
Sep 23, 2010, 03:02 PM
Good luck with all that. I wouldn't have the patience :D

Transporteur
Sep 24, 2010, 09:50 AM
Although there are some people around here that undoubtedly have decent knowledge about storage systems, you'd better post your thread in a dedicated storage systems forum like the HardForum (http://hardforum.com/forumdisplay.php?f=29).

This forum simply isn't the right contact point for your requirement.

What I recommend though is ZFS.

roidy
Sep 24, 2010, 10:33 AM
I know you are right 100% about this, but for my Blu-Rays I at least want to keep them in .ISO for the Boxee Box. As for my DVDs, I am encoding them all to M4V and then deleting the ISOs for them.

In the end, I will still need a significant piece of hardware for my Blu-Ray ISOs.

Boxee only has a limited support for Blu-Ray ISO's, no menu support:(

WiiDSmoker
Sep 24, 2010, 11:39 AM
Boxee only has a limited support for Blu-Ray ISO's, no menu support:(

Yeah. Disappointing, but I'll take it.

ortuno2k
Sep 24, 2010, 03:46 PM
Wow! Very interesting thread, with lots of comments from knowledgeable people!
I am too, wondering, how is it possible to store all that much data? And, how would backups work?
I currently have about 3.5 TBs of data, including programs, movies, music, photos, etc... and I'm having a hard time deciding on how to back this up.
Whatever you do, keep us informed!

This certainly sounds like an interesting (and expensive) project.

Good Luck!

THX1139
Sep 24, 2010, 05:42 PM
I have a solution that would cost less than a $100 bucks to implement, and cheaper over the long run because it uses bare bones drives.

First off, I don't understand why anyone would need constant access to 40TB of files unless you were a business serving content on demand. For the average home user, why do they need to have 24/7 access to an enormous movie/tv show collection? Waste of money and electricity! Why not just keep everything stored on OEM drives on a shelf? Then put your collection on a database that tells you which drive the content is on. Do a search for the show you want to watch, then just pop the drive in and your good to go. Your drives won't wear down as fast because they will be sitting idle until you pop them in. And, you can add to the collection simply by buying a new drive.

The cost of 40 terabyte drives is relatively cheap if you buy them as needed. Then you buy a Voyager Q dock (http://www.newertech.com/products/voyagerq.php) and hook that to your computer via eSata or Firewire 800.

Anyone who needs constant access to 40TB of files needs to get out in the fresh air more often. However, 40TB sitting on a shelf waiting for the rare time you will watch it is way more efficient.

HobeSoundDarryl
Sep 24, 2010, 06:42 PM
I have a solution that would cost less than a $100 bucks to implement, and cheaper over the long run because it uses bare bones drives.

First off, I don't understand why anyone would need constant access to 40TB of files unless you were a business serving content on demand. For the average home user, why do they need to have 24/7 access to an enormous movie/tv show collection? Waste of money and electricity! Why not just keep everything stored on OEM drives on a shelf? Then put your collection on a database that tells you which drive the content is on. Do a search for the show you want to watch, then just pop the drive in and your good to go. Your drives won't wear down as fast because they will be sitting idle until you pop them in. And, you can add to the collection simply by buying a new drive.

The cost of 40 terabyte drives is relatively cheap if you buy them as needed. Then you buy a Voyager Q dock (http://www.newertech.com/products/voyagerq.php) and hook that to your computer via eSata or Firewire 800.

Anyone who needs constant access to 40TB of files needs to get out in the fresh air more often. However, 40TB sitting on a shelf waiting for the rare time you will watch it is way more efficient.

He says he owns all those DVDs and BDs. If he's going to go somewhere to look up the right drive, then pop in the one he needs, he might as well skip the lookup and just select the DVD or BD. Then, he doesn’t' even have to pay for the 40 drives.

Personally, I dig having all my movies on demand myself, but they all fit on just one small 2-drive RAID right now. That's much preferable to locating the right disc, etc.

However, I also appreciate the desire to have the BD's stored so that he can watch full 1080p video, something my :apple:TV solution can't possibly do, even though I can store 1080p video in iTunes, and it will play there just fine.

I appreciate your feedback about energy usage- and that's certainly true. So I vote again for unraid because green drives in unraid can spin down, then spin up on demand when you want a movie stored on a particular drive. That's almost as green as your idea and certainly a lot greener than spinning a bunch of striped drives because a little bit of the move is stored across all of them.