Let's Talk Storage (Home & Workplace)

Sophia. · Mar 18, 2013

Hi Everyone,

I co-own a small company, we do graphics and video work for companies and websites. We are a new company and so far we have four staff including myself.We have a large a mount of data and if I'm honest it's all over the place and not everything is back-up securely.

I would be grateful for any advice you could give on how to sort things out.

At home I have a Mac Pro, Core i7 PC, Mac Mini a Drobo 5N (15 TB) and several external hard drives all Western Digital.

Total storage = 33 TB.

At the office we have 1 Mac Pro, 1 PC workstation and 1 MacBook Pro each.

Total storage = approximately 200 + TB

I'm looking to consolidate and back up my data storage both at home and at work. With some sort of portable storage for taking work home or to clients etc.

I'm also looking for a new "Time Machine".

Home Budget - £2000

Work Budget - £5000

deconstruct60 · Mar 18, 2013

Sophia. said:
Hi Everyone,

I co-own a small company, we do graphics and video work for companies and websites. We are a new company and so far we have four staff including myself....

At the office we have 1 Mac Pro, 1 PC workstation and 1 MacBook Pro each.
...

"each" means that is the setup for each workbay and there are 4 workbays? Or the four folks timeshare on the three computers (one of each type)?

Sophia. · Mar 18, 2013

deconstruct60 said:
"each" means that is the setup for each workbay and there are 4 workbays? Or the four folks timeshare on the three computers (one of each type)?

Sorry for not being clear we each have our own work area with our own Mac Pro, PC and MacBook Pro a total of 12 machines.

deconstruct60 · Mar 18, 2013

Sophia. said:
Sorry for not being clear we each have our own work area with our own Mac Pro, PC and MacBook Pro a total of 12 machines.

It is OK. I thought it probably was 12. Just making sure someone doesn't propose a NAS solution that scales to 4 when you really need 12 (or maybe just 9-10 on average. Laptops on but no all that active usually.).

wallysb01 · Mar 18, 2013

Sophia. said:
At home I have a Mac Pro, Core i7 PC, Mac Mini a Drobo 5N (15 TB) and several external hard drives all Western Digital.

Total storage = 33 TB.

At the office we have 1 Mac Pro, 1 PC workstation and 1 MacBook Pro each.

Total storage = approximately 200 + TB

I'm looking to consolidate and back up my data storage both at home and at work. With some sort of portable storage for taking work home or to clients etc.

Home Budget - £2000

Work Budget - £5000

Wait, 200TBs of data? Is that right? That would 50x4TB drives and at $200 a peace you're looking at $10K just in drives.

Macman45 · Mar 18, 2013

wallysb01 said:
Wait, 200TBs of data? Is that right? That would 50x4TB drives and at $200 a peace you're looking at $10K just in drives.

With that amount of data, whatever you choose is going to cost, but I would talk to Promise...They will advise you on a good RAID solution which is what I use, but nowhere near that amount of data. They are friendly folks, and have static as well as portable storage...Thunderbolt for the most part though...You are going to have to consolidate it all and go from there, but a call to the folks here should help:

http://www.promise.com/contact_us/contact_us.aspx?region=en-US&m=598

I currently have an R4, and will shortly be adding in more...solid reliable real-time storage and backup, but be prepared to spend..

Giuly · Mar 18, 2013

That will be tough. 220TB means that you'll need 55 4TB hard drives, which at roughly £100/pcs will run you £5500 already, and on top of that you'll basically need a 19" rack solution.

As far as your home goes, the 12-bay Synology DS2413+ (here for UK people), which is future proof, filled with 3TB Western Digital Red drives. This will fit your budget.

The Synology works pretty much the same as your current Drobo and is as easy to use, but the nice thing is that you can further expand it with a 12-bay expansion unit for a total of 24 drives should the need ever arise.

Sophia. · Mar 18, 2013

wallysb01 said:
Wait, 200TBs of data? Is that right? That would 50x4TB drives and at $200 a peace you're looking at $10K just in drives.

It's made up of lots of externals some of which are over 10 years old. That we have kept hold of over the years, 500GB drives etc. I worked it out today and it's roughly 200 TB.

We're talking lots and lots of little drives. Most of our tech is personal items that we've reused for work.

It's incredibly ineffective.

PhilMcCracken · Mar 18, 2013

It really sounds like you need to get a server installed and have all that stuff transferred.

I good server rack (networked, and the ability to VPN in from home would be a plus) could solve your issues, but it could be costly with the amount of storage you're looking at. Not to mention the work of transferring the data you have onto the server and getting it organized.

wallysb01 · Mar 18, 2013

Sophia. said:
It's made up of lots of externals some of which are over 10 years old. That we have kept hold of over the years, 500GB drives etc. I worked it out today and it's roughly 200 TB.

We're talking lots and lots of little drives. Most of our tech is personal items that we've reused for work.

It's incredibly ineffective.

Well, if you really have that much data, want to centralize it all and your budget is 7K USD, I just don't think you have a solution. At least not one I'm aware of. This is the type thing you're going to want, but its way out of your budget once you figure in that you need two of them and all the drives:

http://www.siliconmechanics.com/i38875/4u-storage-server.php

I supose to save some costs you could take the drives out of your old externals where you can and plug them in, but generally those external drives are pretty craptastic.

It sounds to me that you have so much data, that you need to go through it all and really figure out what you want on your new server and what you want to archive and shove in some safety deposit box (which still won't be cheap if you want to put 10-100's of TBs on tape for archiving).

You are making my "big data" problems look not so big...

Sophia. · Mar 18, 2013

Thanks everyone for taking the time to comment.

wallysb01 said:
Well, if you really have that much data, want to centralize it all and your budget is 7K USD, I just don't think you have a solution. At least not one I'm aware of. This is the type thing you're going to want, but its way out of your budget once you figure in that you need two of them and all the drives:

http://www.siliconmechanics.com/i38875/4u-storage-server.php

I supose to save some costs you could take the drives out of your old externals where you can and plug them in, but generally those external drives are pretty craptastic.

It sounds to me that you have so much data, that you need to go through it all and really figure out what you want on your new server and what you want to archive and shove in some safety deposit box (which still won't be cheap if you want to put 10-100's of TBs on tape for archiving).

You are making my "big data" problems look not so big...

We store all our work in it's raw format be it photographs, video, audio. I think it's best to do it that way but I could be wrong.

the £5000 is our current profit once overheads and salaries have been paid. I was hoping that I wouldn't have to add to it to get the storage we needed but I could. The price of the Storform iServ isn't that bad if it takes care of work once and for all.

It's not ideal for home storage though...

Giuly · Mar 18, 2013

Sophia. said:
It's not ideal for home storage though...

The Synology I posted above is pretty much as good as it gets.

Sophia. · Mar 18, 2013

Giuly said:
The Synology I posted above is pretty much as good as it gets.

Thank you, I had not heard of Synology before now. After awhile of reading other threads it seems they are top notch.

deconstruct60 · Mar 18, 2013

Sophia. said:
It's made up of lots of externals some of which are over 10 years old. That we have kept hold of over the years, 500GB drives etc. I worked it out today and it's roughly 200 TB.

Depending upon how you did project version control a huge chuck of that could be just duplicated data. Large media projects replicated by 2-3 users that are essentially now the same thing minus some few small project deltas.

I think total is really the whole archive+backup+active. Not all of that may to be to be hosted "live" on spinning disks. Those kinds of sizes are where tape, e.g., LTO-5, has traction. Especially for archiving stuff that don't need to access.

De-duplicating those projects may take work. There are some fancy storage systems that do automatic de-dupe but not on this kind of budget.
That may be a project have to do manually over time in "spare time". If there are special disks marked as being the "master copy" the overhead on the manual process should be manageable.

What need to consider though is some kind of a system which allows the 4 work bays to share master copies without massive duplication. That would reduce the total TB footprint and make the budget better able to cover a solution.

If there aren't tons of duplicates here, then ouch.

Sophia. · Mar 18, 2013

deconstruct60 said:
Depending upon how you did project version control a huge chuck of that could be just duplicated data. Large media projects replicated by 2-3 users that are essentially now the same thing minus some few small project deltas.

I think total is really the whole archive+backup+active. Not all of that may to be to be hosted "live" on spinning disks. Those kinds of sizes are where tape, e.g., LTO-5, has traction. Especially for archiving stuff that don't need to access.

De-duplicating those projects may take work. There are some fancy storage systems that do automatic de-dupe but not on this kind of budget.
That may be a project have to do manually over time in "spare time". If there are special disks marked as being the "master copy" the overhead on the manual process should be manageable.

What need to consider though is some kind of a system which allows the 4 work bays to share master copies without massive duplication. That would reduce the total TB footprint and make the budget better able to cover a solution.

If there aren't tons of duplicates here, then ouch.

We actually had this conversation today. There is one master and one backup of everything we've done as well as the untouched data for the past 6 to 8 months we've been a working company. Everything is cataloged by client, type, date and project.

What is on which drive and the location of that drive. So it shouldn't be that hard, we are lucky in that we have work until at least September. It just means I'll have to stay behind after hours fun.

Thank you for your input it's been most helpful.

sarcosis · Mar 18, 2013

I don't think I saw this before in the thread, but you should also consider having a set of backups offsite somewhere just in case something crazy happens. I know it's adding yet another layer on top of what you are already looking at, but protecting that data should be very important and you won't be fully covered if you don't have the 3 things: 1.) Onsite Master, 2.) On Site Backup, and 3.) Off Site Backup.

deconstruct60 · Mar 18, 2013

Sophia. said:
We actually had this conversation today. There is one master and one backup of everything we've done as well as the untouched data for the past 6 to 8 months we've been a working company. Everything is cataloged by client, type, date and project.

Tape drives have relatively huge (order of magnitude ) upfront cost but the tapes are about 1/2 the price of a HDD. So for example (going to use dollars not pounds.... and some even numbers )

$2000 drive+connectivity.
$40 per 1TB tape
vs.
$80 1TB HDD

Once you start making master and duplicate then can make both for $80 on tape but $160 on HDDs. That is a gap of $80. 2000/80 => 25 pairs until hit breakeven on the tape unit and after that start getting savings.

You may consider inventory you collection of drives from oldest (or most operating hours) to youngest and start to consolidate and migrate a copy of one of those pairs off to tape.

Once you have cleared one of the pairs you can consider recycling that drive to your "sneakernet" use for client/home travel. You could pull a couple from the "newer" end of that list if these old ones are "too slow" for day to day "sneakernet" usage. Once you have enough for sneakernet and a good set of offline "stand by" disks then can start to retire based on "old age".

Long term something similar would need to be done with the tape. If start with a LTO-5 drive then when finally get to a LTO-6 (or LTO-7 if gap large enough) drive part of the exercise is to move oldest tapes to newer media. Nothing lasts forever, but with tape you should have long enough lifetime to take several years to incrementally get to recycling the oldest media to newer stuff. (eventually newer drives won't read older tapes well. but the density goes up so should be able to consolidate over time.).

If it was only 10-15 drives a similar process can be done by incrementally moving to larger HDDs. Going from 1 to 4 TB drives. Problem is that HDDs prices aren't cratering over time anymore. HDDs need to grow in capacity to survive against SSDs but that is going to cost R&D money and the survivors can "buy market share" by reducing margins lower. They reduce themselves out of business that way.

With your budget limitations I'd only assign a portion this year to starting to unwind this collection off to tape. Perhaps go through 6-10 drive pairs and then use the rest to break-even to cover duplicating for offsite new growth storage. With lower costs after break even, in year 2 (and maybe 3 if see high new growth) incrementally to do some more chunks until have split and consolidated this original large set. Recycling some of the drive duplicates back into new clients could keep some new clients on disk if think will jump back into those files quickly. Also periodically would want to do test tape restores to make sure backups really do work.

P.S. I'd resist the notion of piling your current collection into one "super giant" bucket than then have to deal with how to back that up. Unless you need all of your stuff "online and available all the time", that will just create bigger problems. You work "online" store should be whatever number of months client files need to be active. "number of clients in that window" * "conservative project size" ==> about size of online device looking for.

phrehdd · Mar 18, 2013

Seems like you have a lot going on and not only do you need storage, backup but security.

The Synology is a decent unit as are several other larger NAS units. For more on NAS go to SmallNetBuilder site. They give reviews on various NAS. I like Synology but I also like QNAP and Thecus. Each has their perks.

Tapes - absolutely a "business" way to handle storage backups. The downfall of tape however remains backups (depending on what you get) can be slow and also if you need to restore, that too will take time. However, they are tried and true and given the volume of data you have, it might be not only a cost effective way to do things but meets some criteria of disaster recovery via rotation and stored off site.

3rd party storage services via WAN - (back up to a 3rd party site via Internet) has become more and more popular and for good reason as not only do they provide the hardware, but usually it is far enough away to cover facets of disaster recovery.

Security - you really need to give better consideration to security. Beyond yourself, it is not wise having others having access to all files all the time or to be able to take them home at will with no way to track. There are multiple solutions for this so I'll just say give people access to exactly what they need to have access to (for immediate work) and via permissions, lock the rest away. Security is complex but not complicated once you get some understanding of what you need to protect.

External drives - If you plan to let people use external drives, I would suggest you investigate encrypted drives. If a drive was to get lost, it would make it far more a challenge to access the data.

Though you might find that much data is only worth something to you or your client, it is in a sense your clients' product and thus needs to be protected.

In short - examine NAS choices, consider tape, consider 3rd party via Net back ups and lock down files.

Giuly · Mar 18, 2013

deconstruct60 said:
$2000 drive+connectivity.
$40 per 1TB tape
vs.
$80 1TB HDD

It's more like $30 per 1.5/3TB tape.

deconstruct60 · Mar 18, 2013

Sophia. said:
Sorry for not being clear we each have our own work area with our own Mac Pro, PC and MacBook Pro a total of 12 machines.

Another clarifying question is about what sort of network is in place between the 12 machines. If a low speed network partially drove the use of directly attached external drives then networking would be an addition hurdle in addition to storage.

The upside of working off centralized storage is that don't have to make copies. The downside is that bandwidth can be lower if too many folks are all coming in through too narrow "pipes".

For example if pulling ProRes it can be 200Mbps per stream. If that is a local disk and a fast interface that isn't a big hurdle. If that is a remote disk and a 1GbE connection then it is. So if everyone is wired up with 1GbE then video projects might not work so well. For speed folks may end up making local copies and then working from there. That's likely going to land you pretty close to where your at now with external drives.

Bonding/aggregating two 1GbE ports together can get over the 200Mbps barrier but need switches and client ports on the workstations that deal with that.

Giuly said:
Synology DS2413+ ......, but the nice thing is that you can further expand it with a ...

The expansion boxes costing about as much as the NAS head node is slippery slope. If out to create one super giant volume that makes some sense. But if will be composing multiple target volumes to store things then not so much. Another NAS node with another set of Ethernet ports increases storage and network bandwidth. The more data you have for these kinds of projects the more bandwidth likely going to need.

More data behind the same two ethernet ports is nice for low access archives ( massive media libraries with 1-2 users ) more than very active storage to RAW media files. For example a single ProRes video stream will saturate that boxes network bandwidth ( if aggregated. single port would throttle the bandwidth. )

Giuly · Mar 19, 2013

I'm assuming that she's working alone at home, so it's one user and the DS2413+ will be perfectly fine for augmenting her Drobo 5N.

Sophia. · Mar 19, 2013

Thank you all very much for your input. There is a lot of reading to be done and I'm at work so I will read again before replying fully later on. I really appreciate your time and effort.

sarcosis said:
I don't think I saw this before in the thread, but you should also consider having a set of backups offsite somewhere just in case something crazy happens. I know it's adding yet another layer on top of what you are already looking at, but protecting that data should be very important and you won't be fully covered if you don't have the 3 things: 1.) Onsite Master, 2.) On Site Backup, and 3.) Off Site Backup.

We had considered using a online storage company I know for our off site backup. They also do hosting so we'd likely setup our website with them and get a fair discount which is always nice.

deconstruct60 said:
Another clarifying question is about what sort of network is in place between the 12 machines. If a low speed network partially drove the use of directly attached external drives then networking would be an addition hurdle in addition to storage.

Our internal network in our office was setup as a favor by a friend of my partners. I believe it uses some sort of fiber optics, I can't go into more detail as I don't know the proper terminology. I'm not very tech savvy I'm sure you'll have picked up on that.

I can tell you that it's extremely fast and can handle simultaneous workloads from the four of us with ease. However I'm not sure what kind of spare capacity it has if we we're to expand and take on more people.

Thinking of setting up a sever with my Mac Mini:

2.6GHz Quad-Core Intel Core i7

16GB 1600MHz DDR3 SDRAM

2x3TB Serial ATA Drive @ 7200 RPM (replaced original drives)

Giuly said:
I'm assuming that she's working alone at home, so it's one user and the DS2413+ will be perfectly fine for augmenting her Drobo 5N.

You are correct I work alone and home or with my Boyfriend who has his own business but also helps out when he has spare time.

I've actually sold the Drobo since starting this thread (I had another copy of the data stored on the Drobo). I found that it's system for storing data was to restrictive. If the Drobo it's self had failed we'd have had to buy another Drobo before I could access the data.

I managed to sell it for a profit and have purchased a G-Tech G-SPEED Q

with the money. Should be here today at some point.

Thanks again all I will reply in greater length at some point tonight.

Giuly · Mar 19, 2013

Sophia. said:
You are correct I work alone and home or with my Boyfriend who has his own business but also helps out when he has spare time.

I've actually sold the Drobo since starting this thread (I had another copy of the data stored on the Drobo). I found that it's system for storing data was to restrictive. If the Drobo it's self had failed we'd have had to buy another Drobo before I could access the data.

I managed to sell it for a profit and have purchased a G-Tech G-SPEED Q with the money. Should be here today at some point.

Thanks again all I will reply in greater length at some point tonight.

If your G-Tech G-Speed Q fails, you'll have to buy another one of those as well, also it tops out at 16TB. Also you'll be limited to SATA 3GBit/s, which limits the throughput to 300MB/s and is not quite "high-speed" to me.
And as far as DAS go, the same money would've bought you a 6-bay Thunderbolt attached Promise R6 plus hard drives, and you'd have six 1TB hard drives that come with the R6 left to repurpose.

Sophia. · Mar 19, 2013

Giuly said:
If your G-Tech G-Speed Q fails, you'll have to buy another one of those as well, also it tops out at 16TB. Also you'll be limited to SATA 3GBit/s, which limits the throughput to 300MB/s and is not quite "high-speed" to me.
And as far as DAS go, the same money would've bought you a 6-bay Thunderbolt attached Promise R6 plus hard drives, and you'd have six 1TB hard drives that come with the R6 left to repurpose.

The disks can be installed in other machines while we get a replacement though unlike the Drobo. Plus I got it on sale it was only £459 + 2 year warranty. It's only for time machine anyway and I still have money left over from the sale of the Drobo.

simsaladimbamba · Mar 19, 2013

Sophia. said:
Thinking of setting up a sever with my Mac Mini:

2.6GHz Quad-Core Intel Core i7

16GB 1600MHz DDR3 SDRAM

2x3TB Serial ATA Drive @ 7200 RPM (replaced original drives)

No need for a 2.6 GHz quad core server, a 2.3 GHz quad core Mac mini, even the dual core i5 Mac mini, will more than suffice.
But also know, that the Mac mini only takes notebook drives, which are 2.5", and there currently are no 3 TB HDDs in that form factor. The 3 TB HDDs you looked at are 3.5" desktop HDDs and will not fit inside the Mac mini.

And if you get the Mac mini as server, there is no need for that much capacity inside of it if you connect the HDDs externally to it, but as NAS features its own server software anyway.

I may not have come upon this reading the entire thread, thus the following question: Is it 200 TB of data you have or 200 TB of storage capacity?

Do you ever plan to take the week or two to properly sort it and consolidate the data?

Let's Talk Storage (Home & Workplace)

macrumors member

macrumors G5

macrumors member

macrumors G5

macrumors 68000

macrumors G5

macrumors 68040

macrumors member

macrumors member

macrumors 68000

macrumors member

macrumors 68040

macrumors member

macrumors G5

macrumors member

macrumors 6502a

macrumors G5

macrumors 601

macrumors 68040

macrumors G5

macrumors 68040

macrumors member

macrumors 68040

macrumors member

Guest

Our Staff