Mac Pro Storage solutions for photography

[sindresl]

Hi All,

I have a Mac Pro 2009 (2.26GHz Quad) that’s recently started to fall behind. The machine is used as a photo workstation, and is home to some ten-thousand raw-files from a 5Dii. Daily usage consist of organising in Aperture 3 and further processing in PS5. I find that looking through photos in Aperture is really slow, and opening/saving PSD files is insane slow since filesizes usually lie between 200-400MB.

I've decided to put in a SSD to use as a boot drive, and separate OS and applications from the rest of the content that is headed to another/other disk(s). My question to you is how I can best store the raw files and PSD files on one or more other disks for better performance. Currently SSD's are too expensive to store anything other than the OS and applications. Could two or more HDD's in raid 0 be a viable solution as storage for images? I've been reading around for some time about the pros and cons of raid 0, and have come to understand that frequent and small read/write operations will not see any dramatic difference in performance compared to a single disk. In my case, however, where read/write actions will be fewer but in return with larger files, what would the performance difference be?

If raid 0 with traditional 3.5 "HDD's is the way to go, what drives you would recommend?

I back-up on regular basis and automatically with an external firewire drive using time machine, so I'm not too afraid of data loss.

In advance, thanks for all help!

[goodcow]

What's your budget and how much total usable storage do you need?

[El Awesome]

Well, go check the net for the most reliable HDD out there.
I think the WD Red NAS HDDs are very quiet and very reliable.

[VirtualRain]

Originally Posted by sindresl

Currently SSD's are too expensive to store anything other than the OS and applications.

I disagree... but everyone's definition of "too expensive" is different.

You can get a 256GB SSD now for under $200. Amazon had some on sale for $160 this past weekend.

I would recommend you have a single SSD for your OS/Apps and another for your working set of photos.

Now, of course you can't store every photo you've ever shot on an SSD, but you most certainly can store your current working set on an SSD. Even if you fill half a dozen 32GB CF cards on a shoot, you can easily transfer those to a single 256GB SSD to work on. When you're done processing them, archive them to a slower and larger HD to make room for your next shoot.

[ColdCase]

Yeah, what kind of budget?

I've used a couple 500 GB drives with software RAID 0 reliably for years as a scratch drive. Faster reads/writes than a single drive, but there are better throughput options, depends on your budget.

Think about a PCI SATA controller card with built in RAID0 and a couple SATA 3 ports. Then put a couple SATA 3 hard drives in SATA 3 enclosure(s), WD probably. Some vendors sell a turn key system.

I think Aperture is pretty slow regardless.

[sindresl]

Originally Posted by VirtualRain

I disagree... but everyone's definition of "too expensive" is different.

You can get a 256GB SSD now for under $200. Amazon had some on sale for $160 this past weekend.

I would recommend you have a single SSD for your OS/Apps and another for your working set of photos.

Now, of course you can't store every photo you've ever shot on an SSD, but you most certainly can store your current working set on an SSD. Even if you fill half a dozen 32GB CF cards on a shoot, you can easily transfer those to a single 256GB SSD to work on. When you're done processing them, archive them to a slower and larger HD to make room for your next shoot.

This is actually the route I've embarked on. I ordered two samsung 830 ssd's (one 128Gb and one 256Gb), and was thinking along the same lines as you've suggested. The 128Gb one for the OS and applications and one 256Gb for current work. I still would like though to have a fairly good performance for my "archived" work that will reside on another disk(s), as I often go back and pick up older projects on a later point. Hence the question regarding HDD's and raid 0.

Thanks again for all input, it's very much appreciated.

[VirtualRain]

Originally Posted by sindresl

This is actually the route I've embarked on. I ordered two samsung 830 ssd's (one 128Gb and one 256Gb), and was thinking along the same lines as you've suggested. The 128Gb one for the OS and applications and one 256Gb for current work. I still would like though to have a fairly good performance for my "archived" work that will reside on another disk(s), as I often go back and pick up older projects on a later point. Hence the question regarding HDD's and raid 0.

Thanks again for all input, it's very much appreciated.

I see... great. I put all my archived work on stand alone HD's. However if you access them often, a pair in RAID0 is easy to setup in Disk Utility. WD Black's would be my preference for drives where performance is important.

[cutterman]

A couple of hard drives in raid 0 sounds like a decent solution for your internal storage needs. You may want to consider using more expensive enterprise-level drives for this due to improved reliability. In addition, I would not trust my entire photo archive to a raid 0 volume and a time machine backup alone. I would suggest using a NAS or even a separate external drive that you can sync your files to.

[Loa]

Hello,

Are you using 8bit files, or 16bits files? If you're using 16bit files (or bigger), then getting a faster drive or RAID or SSD won't matter as the CPU is the bottleneck.

Loa

[darkcoupon]

Here's my current setup: optical bay + drive bay 1 = 2x 128GB SSD's in RAID 0 for boot/scratch disks, dive bays 2 and 3 = 2x 2TB WD Green in raid 1, bay 4 = 2TB WD Green time machine backup and sveral external backups for older archived work. I rarely, if ever, go above 1TB on the internal storage before I back everything up to external drives. Also, it's not a bad idea to burn really important stuff to DVDRs, just to be extra safe.

[cutterman]

Originally Posted by Loa

Hello,

Are you using 8bit files, or 16bits files? If you're using 16bit files (or bigger), then getting a faster drive or RAID or SSD won't matter as the CPU is the bottleneck.

Loa

He is opening and saving files 200-400 MB in size. Under those circumstances faster drives will make a huge difference.

[Honumaui]

dont get on here much anymore


first off quit saving large files as PSD save them as TIFF and just check the save layers and you will see a huge increase in speed opening and closing

second a SSD boot is a nice way for things to get snappy for sure
the samsung 830 in 256 would be great to work off then move your files off
but ask yourself how often you do this what your workflow is cause moving files on and off might just wipe out any gains you got by working on the SSD !!
it can work but it has to be a download from camera onto the SSD do your work and then dump your other files (which of course would be backed up to a larger drive)

raid 0 is fine just remember you have to have a solid backup plan
also like a single drive if it dies while rare you loose what you were doing ?
so raid 5 or 6 are nice as you can keep working !!!
also raid 10 is easy to do on the macs

once you start dumping all these SSD on the mac you want to be looking at getting a card to keep the speed up ! so either something simple like the one form OWC and use external cases ? or very curious about that new sonnet system you can mount two SSD in and put in a slot

also for backup and archive look into getting a eSATA solution that has a PM case like again the OWC card for about $69 and a sans digital case for about $200 that is 5 bay

a dedicated cache drive for LR helps quite a bit

as far as LR goes only files I have noticed are faster on SSD are D800 files and MKIII files when working in develop mode
exporting and importing are quicker on SSD but I tend to import files and build previews and do other work answer emails etc.
so the savings is not huge

when working in PS the images on SSD save a bit ? again save as TIFF for large files

capture one things are much quicker on a SSD all around


SSD are nice and they can help quite a bit but LR also likes CPU speed !

if I was building a LR workstation it would be based off the 3.4 iMac ! at this point the Mac Pros are to old IMHO
I still use one a new 5,1 3.2


my setup is a basic areca raid for main storage
cache scratch is SSD
boot is SSD
working large files SSD

and lots of backup

my total workstation storage is around 60 TB 20 working 20 BU 20 archive
also off line storage and local storage not hooked up to power


I have no idea what that new sonnet solution is like with the two SSD you put in ?
but if I was tweaking out a mac pro today for work I would look into this over a eSATA card and cases ?

[Loa]

Originally Posted by cutterman

He is opening and saving files 200-400 MB in size. Under those circumstances faster drives will make a huge difference.

Nope. I tested it using a vertex 4 on SATA 3 PCIe card, compared to a regular HD: the difference is huge for 8bit files, and very small for 16 bit files. My 16bit test file was 415MB.

Loa

[-hh]

My setup has become:


OWC Accelsior SSD PCIe card (240GB) - $500
... boot drive

2 x 2TB 3.5" HDD (Hitachi) - $250
... configured as RAID 0 for 2TB of data

2 x 4TB 3.5" HDD (Hitachi) - $600
... configured as RAID 1 for 4TB of Time Machine backup

1 x 250GB 2.5" HDD (Western Digital) - $50
... in the process of being set up - - going to be a nightly backup based "Mirror" of the boot SSD by using CCC.


Cost for this storage setup: $1400

Granted, it could be made quite a bit cheaper if one were to put TM on an external HDD, but this configuration keeps it "all inside one box"; I use externals to make single-disk based backups to move to off-site storage.


-hh

[Loa]

Originally Posted by -hh

2 x 4TB 3.5" HDD (Hitachi) - $600
... configured as RAID 1 for 4TB of Time Machine backup

Having a RAID1 for a back-up system is quite pointless. You'd be best protected by using your 2 drives as independent back-ups, and adding it to your external and off-site back-up rotation. RAID1 was never meant for back-up, and is only protecting you against 1 of the 3 main causes of data loss.

Loa

[Honumaui]

Originally Posted by Loa

Having a RAID1 for a back-up system is quite pointless. You'd be best protected by using your 2 drives as independent back-ups, and adding it to your external and off-site back-up rotation. RAID1 was never meant for back-up, and is only protecting you against 1 of the 3 main causes of data loss.

Loa

I had my time machine on a stand alone raid 5 box ?
my reasons were in case of a HDD failure I wanted to make sure all my older files would be there if I wanted to go back ?

so there is a reason for raid 1 but its one you would know why

agree often its better to have two independent and raid is wasted on backup usually but not always as time machine is a bit more than just a mirror type backup

and any raid is not backup in itself ! but can be used as part of a backup plan
to me I view any storage device from a single drive to a raid 6 or whatever as a single unit that must be backed up properly !!!

same reason I like boot as raid 1 no down time if a HDD fails
same reason I like my data drives to have redundancy against hardware failure ! no down time
same reason I tend to like a clone type backup on a system that is close to the same speed as my main system in case I have to use it for a day or two I wont be crippled working on some insane slow external USB or something !
same reason I do like having a boot clone if even my raid 1 dies I can just boot off my clone and be going again


I guess I look at a lot of this like modern run flat tires ! you have to repair them but at least you dont have to pull over on the side of a busy highway and fix the flat you can often pull over in a safe spot and or a tire place etc..

same with safe systems you can often finish what you are doing and rebuild replace fix the system on your time schedule !
working with deadlines and going down for even a couple hours could cost you a client and a lot of money for those in business that alone is worth it !


for home use not so much

[thekev]

Originally Posted by Loa

Hello,

Are you using 8bit files, or 16bits files? If you're using 16bit files (or bigger), then getting a faster drive or RAID or SSD won't matter as the CPU is the bottleneck.

Loa

CS6 can now save files in the background, which can be useful for intermittent saves on larger files. It's also possible to disable compression, but then you end up with gigantic files. It can be feasible if you're working with massive amounts of data. Saving comp files at a few GB was painful a few years ago. I'd go to get coffee and the bar would be halfway.

Originally Posted by Honumaui

I had my time machine on a stand alone raid 5 box ?
my reasons were in case of a HDD failure I wanted to make sure all my older files would be there if I wanted to go back ?

so there is a reason but its one you would know why

agree often its better to have two independent and raid is wasted on backup usually but not always as time machine is a bit more than just a mirror type backup

Well there are other things that can take it out. Raid controller failure, data corruption, failure to rebuild after a power outage (which is why a UPS is essential), and of course that matter of fault tolerance is dependent upon a reliable controller. The reason people say a raid is not a backup is that too many people try to use raids as storage + backup in one. It makes me want to smack them with a keyboard.

[Loa]

Hello,

For parity RAIDs (like RAID1) and back-ups, here's what I wrote somewhere else on these forums: parity RAID solutions are not back-up solutions, and are not designed for data security; they're only useful for systems that absolutely need 100% uptime. The only thing that makes sense for data security is a back-up solution. For ease of restoration, rebuilding a parity RAID set-up is just as long as restoring from a back-up: just pop the back-up in one of the internal bays and copy the data back to the new drive.

Why use a solution that only protects you from one of three sources of data loss??? Having two independent backup drives is a lot safer (and not slower) than a RAID1.

Loa

[-hh]

Originally Posted by Loa

Hello,

For parity RAIDs (like RAID1) and back-ups, here's what I wrote somewhere else on these forums: parity RAID solutions are not back-up solutions, and are not designed for data security; they're only useful for systems that absolutely need 100% uptime.

There's always more than one way to look at a problem. For me, having the RAID1 is of value (for things other than uptime) because the RAID1 structure allows me to have "immediate" feedback that my backup copy has full data integrity.

The reason why this is important to me is because historically, one of the classical flaws with making backups was in their feedback loop - ie, testing them after creation to verify that a good copy was indeed made. Use of a RAID1 does an inherently better job at addressing this failure point.

The only thing that makes sense for data security is a back-up solution. For ease of restoration, rebuilding a parity RAID set-up is just as long as restoring from a back-up: just pop the back-up in one of the internal bays and copy the data back to the new drive.

For 'ease of restoration', it really comes down to the question of how one is managing your remote site designed backup. An action of "simply popping" a drive into an internal bay always requires shutting down the Mac Pro, gaining access to its interior, etc ... sure, it can be done and with hefty data transfer requirements it makes sense - - but what's more likely (due to day-to-day convenience factors) is that one will design a data backup system that exports to some external HDD ... the default implications thereof is that it will be either FW800 or USB2 - neither of which has I/O performance comparable to SATA-2.

Why use a solution that only protects you from one of three sources of data loss??? Having two independent backup drives is a lot safer (and not slower) than a RAID1.

First, you're assuming that I only have the one backup level that I happened to have mentioned, which happens to be incorrect. Given that I mentioned that I'm also mirroring my boot drive, it is obvious that there's more to my backups strategy.

Nevertheless, the high level general use case problem is that most Mac users don't do anything more for their backups than just Time Machine running on a single spindle. A RAID1 does at least address one of the shortcomings of this type of approach.

FYI, while I'm not yet running 10.8 (Mt Lion) yet, I am aware that the version of TM that's in 10.8 does finally support multiple volumes - - however, my research has found it doesn't push full backups to each volume, but merely pages through the available TM drives ... as such, the backup contained on TM#1 isn't quite the same as the backup contained on TM#2 (or TM#3, etc): there is only one single "latest" version. Yes, that's better than what TM had before, but it still doesn't produce fully mirrored copies that can be tested.

FWIW, what I'm currently using for my subsequent layers of non-internal backups is a NAS (currently down for repairs) and a pair of external USB 3TB drives which are rotated through a safety deposit box off-site. In addition, there's a couple more externals for "targeted" backups of selected data which depend on their status for if they're remote, local or attached. As a rule of thumb, I always want to have 4 copies of my data.


-hh

[Loa]

Originally Posted by -hh

For me, having the RAID1 is of value (for things other than uptime) because the RAID1 structure allows me to have "immediate" feedback that my backup copy has full data integrity.

Yes, but this immediate feedback also destroys your back-up if you accidentally delete a file (the deletion will be mirrored) or if some corruption happens (the corruption will be mirrored).

Originally Posted by -hh

An action of "simply popping" a drive into an internal bay always requires shutting down the Mac Pro, gaining access to its interior, etc ... ... neither of which has I/O performance comparable to SATA-2.

That's why I use a 10$ esata card and a 15$ external esata enclosure.

Originally Posted by -hh

First, you're assuming that I only have the one backup level that I happened to have mentioned, which happens to be incorrect.

I didn't, but my question is: why waste a 300$ drive in an imperfect backup solution?

Originally Posted by -hh

Nevertheless, the high level general use case problem is that most Mac users don't do anything more for their backups than just Time Machine running on a single spindle. A RAID1 does at least address one of the shortcomings of this type of approach.

Again, why address only one of the shortcomings of this type of approach?

In the end, for me a backup solution is to have perfect copies of my data that are protected against all types (as much as humanly possible) of data loss or corruption. If you want to spend 300$ on an inferior addition to your backup solution, go ahead! Trying to say that it's not inferior, on the other hand, is a mistake, and a disservice to those who don't have 4 backups and may think that RAID1s are good backup strategies.

Loa

[-hh]

Originally Posted by Loa

Yes, but this immediate feedback also destroys your back-up if you accidentally delete a file (the deletion will be mirrored) or if some corruption happens (the corruption will be mirrored).

These failure modes will occur regardless of if it is a RAID or a single spindle. As such, your point does not apply.

That's why I use a 10$ esata card and a 15$ external esata enclosure.

I used eSATA for awhile on my prior Mac, but I personally found it too unreliable. If you've had "good luck" with it, then good for you.

I didn't,..

You didn't assume that the RAID1 was the only level of backup I had? Sorry, but I'm going to elect to be cynical about your claim.

... but my question is: why waste a 300$ drive in an imperfect backup solution?

Because perfect backup solutions simply don't exist. For example, in some so-called 'perfect' system, how many duplicate copies should there be, and why? Be specific.

Again, why address only one of the shortcomings of this type of approach?

As I said, 4 copies, including rotated off-sites...so just how are "the rest" of the failure modes not already covered? Be specific.

In the end, for me a backup solution is to have perfect copies of my data that are protected against all types (as much as humanly possible) of data loss or corruption...

Yes, that's the philosophical ideal. I've noticed that you've not provided specifics on how you're doing against it yourself.


So are you currently validating 'perfect' copies? For example, are you performing additional third party parity checks on top of TM?

Be specific.

If you want to spend 300$ on an inferior addition to your backup solution, go ahead!

Since you've not provided the specifics of precisely how you would run two disks in parallel that aren't a RAID1, you are claiming that something vague is somehow 'superior'.

I'm sorry, but I need more specifics again. Golly, it sure becomes tough when I have to be a cynic!

Trying to say that it's not inferior, on the other hand, is a mistake, and a disservice to those who don't have 4 backups and may think that RAID1s are good backup strategies.

My apologies for accidentally pushing your 'RAID' hot button.

Nevertheless, for someone whose idea of a backup is to enable TM, the fact remains that having TM on a RAID1 is less bad than having that same "single copy" be on but a single spindle. But do feel free to try to twist this observation into where I've somehow claimed that this makes it the best! (it doesn't).


-hh

[knoxtown]

The way I do it for my photo business (even though I have an iMac currently)...

I have a 2 TB external that mirror backups my entire working HD. Then I have another 3 1 TB externals that I backup to based on my categories (Commercial, Editorial, Personal).
Then, I burn BlueRay archive dvds. That way, if all of that digital mess fails. I've got a disc hahaha

[= bluntman =]

Here's mine:

OS and applications: 480GB Sandisk SSD
Scratch disc/work drive: 2x 1TB drives in a striped RAID configuration
Storage: 4x 2TB drives in RAID 10 configuration in Synology 412+ NAS
Backup of NAS: 2TB Lacie d2 Quadra as (temporary) backup of NAS

[Loa]

Originally Posted by -hh

Nevertheless, for someone whose idea of a backup is to enable TM, the fact remains that having TM on a RAID1 is less bad than having that same "single copy" be on but a single spindle. But do feel free to try to twist this observation into where I've somehow claimed that this makes it the best! (it doesn't).

Good grief... Last attempt.

In a RAID1, you have 2 drives. In your quote, you're comparing a solution with 2 drives with a solution with 1 drive. Please be consistent.

My only point is that having those 2 backup drives in a RAID1 is a riskier choice than having them as two separate backups that can be placed in different locations.

Simple as that.

Loa

[ActionableMango]

Originally Posted by -hh

one of the classical flaws with making backups was in their feedback loop - ie, testing them after creation to verify that a good copy was indeed made.

I've been burned twice now by companies that told me they were backing up my data only to find that when I needed that data, the backups had been failing the whole time.

It's not enough to just back up. One has to test the restore process!

On a personal note, I once found out that Time Machine had simply stopped making backups for several months before I found out--no error or warning messages.