The Drobo FS

[John Kotches]

Of course the amount of drives you can lose () depends on the kind of setup you're running.

I'm currently using a raidz2 for a 8 drive setup. Originally I had it configured as a RAID6 and personally I think that single parity for a 8 drive + setup is a little too optimistic.



I hope I can.

I tested ZFS first with the current FreeNas build as a VM before I installed it on a physical machine. Configured it with 8 virtual drives and set up a raidz2. Then I copied about 100GB of data on that pool (mounted via AFP). Then I disconnected drives randomly. First drive out, everything's fine. Second drive out, 6 left, which is supposed to still work and it indeed does.
Alright, then I replaced the two drives, ZFS automatically resilvered the array (took a second or so, way to less to copy any data at all) and it was labeled online again.
Then I popped out another drive and that did the job. Array offline.

Did you actually wait for the system to show that the array was rebuilt? A few seconds sounds way too short to resilver the virtual drive.

It takes about 5-10 minutes to silver a copy with ~5GB of used data on a physical machine in my environment. This is a fresh OS install that is getting mirrored.

No "restoring" process in the classical sense, is it?
What I'd expect is that resilvering actually makes the old data available on all available discs, but apparently it does not.
I'm not sure if FreeNas screwed it, but I hope that you can clear that up!
Haven't tested the resilvering process on a physical machine yet.

I would say that it's your procedures that are flawed here and that you pulled sufficient drives to create a toasted array condition due to the number of drives you pulled prior to the array's integrity being restored.

"zpool status" is your friend.

 

[Transporteur]

Did you actually wait for the system to show that the array was rebuilt? A few seconds sounds way too short to resilver the virtual drive.

It takes about 5-10 minutes to silver a copy with ~5GB of used data on a physical machine in my environment. This is a fresh OS install that is getting mirrored.

I did wait for a long time but the drives definitely did not copy any data, and as I said, the array was instantly marked online again.

So let me get this straight, assuming we've got a 8 disc raidz2 array, so double parity (you can lose 2 drives before data loss, not 3 as you said. Source: Oracle).
Now in case that a drive, or two fail, the array is still functional an we don't have data loss. So far so good.
Now what if we replace the drives, does ZFS copy data to the new drives or not?
So after resilvering is completed, is it possible to lose another drive or two (not the ones we replaced) and still no data is lost?

 

[John Kotches]

I did wait for a long time but the drives definitely did not copy any data, and as I said, the array was instantly marked online again.

Online but degrade. Did you ever check with zpool status?

So let me get this straight, assuming we've got a 8 disc raidz2 array, so double parity (you can lose 2 drives before data loss, not 3 as you said.

You're citing the number of drives you can "safely" lose... I'm citing the number of drives that you have to lose to lose data. We're stating the same thing from different perspectives.

Now in case that a drive, or two fail, the array is still functional an we don't have data loss. So far so good.

Now what if we replace the drives, does ZFS copy data to the new drives or not?

Yes.

So after resilvering is completed, is it possible to lose another drive or two (not the ones we replaced) and still no data is lost?

Once resilver is complete the data is restored. You can lose drives up to the limits of your parity definitions without data loss and without regard to whether or not it is "new" or "old".

 

[7on]

Somewhat OT, but seeing as you guys know what you are doing, what is the best Raid 5 card?

 

[nanofrog]

Somewhat OT, but seeing as you guys know what you are doing, what is the best Raid 5 card?

What exactly are you trying to do?

How many ports do you need, and what configuration?
What kind of throughputs?
What type of drive (SSD, SATA mechanical, SAS)?
Do you need it to be available to multiple OS's?
Do you need to boot from it, and if so, is it OS X?

To give you some ideas as to what's possible, take a look at ATTO and Areca's products (they can boot OS X once the firmware is flashed to EFI, and then run others via drivers). They both make good cards, and will work in MP's quite well. ATTO has better support, but Areca is cheaper (support is still decent, but based out of Taiwan, ATTO in the US).

 

[7on]

What exactly are you trying to do?

Media Server with redundancy.

How many ports do you need, and what configuration?

Starting out with two 2TB disks and potentially adding more down the road. SATA.

What kind of throughputs?

Nothing extreme. It's all going to be DVD.iso and some mp4s. Looking at using FreeNas since that supports UPnP as does my Media streamer (popcorn hour)

What type of drive (SSD, SATA mechanical, SAS)?

SATA Mechanical. I've had good luck with WD 2TB Green Drives. Though haven't used it inside a RAID. But like I said, throughput just needs to support DVD MPEG2 and possibly some 1080p mp4 down the line.

Do you need it to be available to multiple OS's?

Yes. I assumed I'd just turn on Samba and use it with OSX and Win7 (backups) and UPnP for the media streaming.

Do you need to boot from it, and if so, is it OS X?

Don't need to boot from it, other than it running something like FreeNAS. It'll be screenless and keyboardless after the initial setup.

 

[nanofrog]

...card...

An ARC-1210 would be sufficient I think (4 port card, uses SATA cables, not MiniSAS, capable of RAID5). It can't use the HDD bays in an MP though, as it doesn't have the right connector (you can stuff drives in the optical bays, but it may force you to get an external enclosure for 3 - 4x disks; each bay can hold 2x 3.5" drives).

If you do wish to use the HDD bays, you'd need to go with the ARC-1212, and tell me which MP you're using, as an adapter will be required to make it work ('06 - '08 systems use this, '09 systems need this; the most expensive one is needed to use all 4x HDD bays, and you can configure it as you go).

Between these two cards, the ARC-1212 is a much better card (faster processor, and additional features). It's usually ~$60USD more than it's currently priced. But the adapter will add $75USD or $165USD, depending on which model year MP you have.

Yes. I assumed I'd just turn on Samba and use it with OSX and Win7 (backups) and UPnP for the media streaming.

I meant in terms of native support (drivers). Samba uses TCP/IP.

Read the following, as it might help eliminate some confusion.

Samba is software that can be run on a platform other than Microsoft Windows, for example, UNIX, Linux, IBM System 390, OpenVMS, and other operating systems. Samba uses the TCP/IP protocol that is installed on the host server. When correctly configured, it allows that host to interact with a Microsoft Windows client or server as if it is a Windows file and print server.

At any rate, it's not native, nor the same as applications that make different OS's file formats available to a different OS (i.e. Windows able to access both ext3/4 or HSF/+).

 

[7on]

An ARC-1210 would be sufficient I think (4 port card, uses SATA cables, not MiniSAS, capable of RAID5). It can't use the HDD bays in an MP though, as it doesn't have the right connector (you can stuff drives in the optical bays, but it may force you to get an external enclosure for 3 - 4x disks; each bay can hold 2x 3.5" drives).

If you do wish to use the HDD bays, you'd need to go with the ARC-1212, and tell me which MP you're using, as an adapter will be required to make it work ('06 - '08 systems use this, '09 systems need this; the most expensive one is needed to use all 4x HDD bays, and you can configure it as you go).

Between these two cards, the ARC-1212 is a much better card (faster processor, and additional features). It's usually ~$60USD more than it's currently priced. But the adapter will add $75USD or $165USD, depending on which model year MP you have.


I meant in terms of native support (drivers). Samba uses TCP/IP.

Read the following, as it might help eliminate some confusion.


At any rate, it's not native, nor the same as applications that make different OS's file formats available to a different OS (i.e. Windows able to access both ext3/4 or HSF/+).

I'll be using some throw away PC probably. Also, what exactly would I be sacrificing if I looked into a cheaper card? And I don't need it to be available to multiple OSes in your connotation.

 

[nanofrog]

I'll be using some throw away PC probably. Also, what exactly would I be sacrificing if I looked into a cheaper card?

If it's a PC, then there's no need for any adapters. Either the ARC-1210 or ARC-1212 would do the job (assuming you won't need more than 4x ports).

As per cheaper cards, you have to be really carefull, as you're likely to get into a FakeRAID controller (software based, not hardware RAID; it still uses drivers = software, and system resources to handle the calculations).

As it's a PC, you could also look into Adaptec (you have to add a cable here, so ~$30 more), LSI (again, no cables), 3Ware (no cables), and a few others. Avoid Highpoint however, as their support sucks (they don't actually manufacture anything themselves; it's all ODM supplied gear). Unfortunately, they're not cheap brands at all.

$300USD for a proper RAID card is a good price. The ARC-1212 has a faster processor, 256MB cache (others in this price range may only be 128MB), and it has the internal cable included. They're also rock-solid in terms of stability.

Just make sure you check the HDD Compatibility List from the card manufacturer on the drives you're interested in before you buy them (saves you all kinds of headache). You need to do this for any card you're considering, especially with SAS cards (they will run SATA drives, but are picky; they'll have to be enterprise versions due to the recovery timings in the drive firmware are different between enterprise and consumer models).

Consumer = 0,0 (read, write; values in seconds)
Enterprise = 0,7

There's also additional sensors to prevent physical heads from contacting the platters, and improved ratings such as UBE (Unrecoverable Bit Error) and MTBF (Mean Time Between Failure). Enterprise disks are definitely worth it. Stick with WD IMO, as they've served me well with Areca's products (and others, and it's recent - I'm personally running WD's that are ~ 1y/o now).