Upgrading RAID for video editing - which card? [Archive]

wonderspark

Oct 11, 2010, 09:53 AM

wonderspark

Oct 11, 2010, 03:56 PM

Forgot to ask, when I pull the Apple RAID card out, will the data in the 640GB OS disc in slot #1 remain intact, or will it need to be reformatted for some reason?

If so, it's not that big a problem. A clean install would be nice.

nanofrog

Oct 11, 2010, 04:14 PM

I've been editing a feature film for the last nine months using the Apple RAID card and four internal drives, then backing up nightly to an external drive via eSATA 1TB drive. 640GB OS, 3x1TB RAID 0 internal, 1TB external.

I've decided to switch to an Atto or Areca card to replace Apple RAID Pro card, and get an external 4-drive box to build RAID 3, and use my four internals for OS, scratch, renders and original media files.

Which RAID card would be best suited for an '09 Mac Pro with external raid? I don't need Windows or bootable external, as far as I'm concerned. I'd just use the internals for everything but RAID. I saw an Areca 1110ML for about $300 or so. Good choice, or is there newer and better for me?
The ARC-110ML will not work (it's a PCI-X based card, not PCIe, which is what you'll need for your MP).

The bare minimum is the ARC-1212X (http://www.pc-pitstop.com/sata_raid_controllers/arc1211x.asp) (specs (http://areca.us/products/externale.htm)). They're getting harder to find, but pc-pitstop still has them listed.

ATTO's cards are more expensive, which is why I've not listed any.

BTW, when changing to a different card, and using the same drives, you'd need to do a full backup first, create the new array, and restore. You'll lose the data already on the disks, as arrays from one card maker usually won't work on anothers (generally, you can do it if you stick with the same maker).

The Highpoint RR4311 (http://www.newegg.com/Product/Product.aspx?Item=N82E16816115065&Tpk=rocketraid%204311) could work, as they do have drivers (don't actually offer EFI firmware to allow it to boot OS X), but the support royally sucks, so you'd basically be on your own if you have a problem.

Assuming the 640GB disk is the OS, you'd only need to remove the existing card, and load the drivers for the Areca. You'd be good to go, but I'd still go for a clean installation (avoids problems due to conflicts if there's an old file that remains).

cutterman

Oct 11, 2010, 04:24 PM

nanofrog

Oct 11, 2010, 04:28 PM

I would suggest the Areca 1880x here (http://www.pc-pitstop.com/sas_controllers/ARC1880X.asp). It is a dual external port 6G SAS/SATA card. It will give you future expandability and compatibility with newer 6G drives, and the brand is well-thought of on this forum. It is a bit more than $300, but given your task at hand I wouldn't settle for the cheapest options.
I actually erased this one from the post, as I'm unsure of budget (nearly $700 for it, but it's well worth it IMO).

But the ARC-1212X is still a faster card than what's being used now (more stable and better recovery features if they're needed - there's some hidden recovery commands used via the CLI that can recover sets that would be lost on other controllers), and it's not expensive. :)

wonderspark

Oct 11, 2010, 04:58 PM

Oooh, thanks guys!

Yeah, I'm not all that concerned with cost - - under $1000 won't break the bank. I admit that I know more about the RAID levels and how they function, while knowing next to nothing about which cards do what and how they are compatible. :)

I'd like to be able to expand some, as I'm hoping for future movies to be shot on RED... 4K and up video files. I try not to be wasteful and just go top shelf unless it's needed, yet getting a card that can keep me in good shape when I need to expand my capacity would be ideal. 3TB (3x1TB) was fine for this movie, but the next one will probably be two or three times as large, due to the much larger file sizes. I only used 1.3TB on this movie, and I was being a pack-rat and saving everything, even when I was done with it.

A little history: I bought this Mac for the sole purpose of editing this film, and it has served me well. I knew even less about RAID cards at the time, and decided it would be fine to just get the 'turnkey' Apple RAID card and go. After reading these forums for the last nine months, all I've ever read about this Apple card is that it sucks. I've seen mention of some battery issue, and I'm not sure what that is. I have noticed that exactly every three months, the battery "reconditions" itself, which takes about seven hours. During this time, I've dared not edit the movie, for fear of whatever might happen during that period.

Anyway, I've been "living on the edge" by working in RAID 0 and just backing up regularly, and on the heels of my success with this movie, I feel it's time to stop relying on luck so much, hahaha!

Thank you for the help and suggestions!

wonderspark

Oct 11, 2010, 05:23 PM

Ok, another question:

On the pc-pitstop site, I was looking at their enclosures. I had a 4-bay in mind, and when I selected the hot-swappable one here (http://www.pc-pitstop.com/sas_cables_enclosures/sas4bay.asp), it listed the 1212x card, but did not give a choice for the 1880x card. Is the 1880x not compatible with that enclosure? Which enclosure would you suggest if I want the 1880x?

I don't think I'm ready for the 15-bay enclosure they suggest with the 1880x, shown here. (http://www.pc-pitstop.com/sas_cables_enclosures/scsas156g.asp)

cutterman

Oct 11, 2010, 05:31 PM

Yes the 1880x card is compatible with that ( and virtually any sas or sata enclosure for that matter). Basically each external port can connect to 4 drives, so you could support 2 x 4 drive enclosures or 1 x 8. There are also sas port multiplier enclosures that can extend that even further. You would also need a SF8088 to SF8088 external cable to connect the enclosure to the card.

wonderspark

Oct 11, 2010, 05:35 PM

nanofrog

Oct 11, 2010, 07:53 PM

Yeah, I'm not all that concerned with cost - - under $1000 won't break the bank. I admit that I know more about the RAID levels and how they function, while knowing next to nothing about which cards do what and how they are compatible. :)
I'd go with an Areca 1880 series, but the question is, which one?

So think hard on the port count, as they can go up to 24 ports (24x disks without the need for a SAS Expander). Using SAS Expanders, they can actually run up to 128 disks. :eek:

The ARC-1880X has 8x external ports (2x connectors on the back).

Past that, you're dealing with primarily internal ports (there is 1x external port, but it tends to share one of the internal ports, so you have to pay attention so you're not throttling). But you can get special cables that take internal ports to external enclosures (here (http://www.pc-pitstop.com/sas_cables_adapters/8887-1M.asp)), and there's a kit that allows you to use the HDD bays with the card (here (http://www.maxupgrades.com/istore/index.cfm?fuseaction=product.display&product_id=189%20)).

So you could go with an ARC-1880ix12 (http://www.provantage.com/areca-technology-arc-1880ix-12~7AREC03T.htm), use one port with the internal bays via the kit, and one of the internal to external cables per 4x disks.

I realize this stuff may be looking like top shelf, and in a sense, it is (excellent manufacturers). But step back a bit, and you'll realize it's not. To see what top shelf really is, take a look at High Availability Clusters and SAN (wiki has information on both; this is where it starts to get crazy). ;) Multiple servers, SAN's,... strung together to make sure there's no single point of failure that will prevent access.

Ok, another question:

On the pc-pitstop site, I was looking at their enclosures. I had a 4-bay in mind, and when I selected the hot-swappable one here (http://www.pc-pitstop.com/sas_cables_enclosures/sas4bay.asp), it listed the 1212x card, but did not give a choice for the 1880x card. Is the 1880x not compatible with that enclosure? Which enclosure would you suggest if I want the 1880x?

I don't think I'm ready for the 15-bay enclosure they suggest with the 1880x, shown here. (http://www.pc-pitstop.com/sas_cables_enclosures/scsas156g.asp)
4 bay unit (http://www.newegg.com/Product/Product.aspx?Item=N82E16816111090&cm_re=tr4x-_-16-111-090-_-Product)
8 bay unit (http://www.newegg.com/Product/Product.aspx?Item=N82E16816111092&cm_re=tr8x-_-16-111-092-_-Product)

BTW, not only are they good on price, they include external to external cables. And their appearance better suits your MP vs. the "el cheapo black plastic front" units pc-pitstop sells (they do work though).

You would also need a SF8088 to SF8088 external cable to connect the enclosure to the card.
The Sans Digital enclosures linked have the external to external cables @ 1.0 meters.

BTW, do not exceed 1.0 meters of total cable length with SATA disks (you can go up to 10 meters with SAS disks, as they run at a much higher signal voltage). And stay away from the adapters, as SATA sets won't be stable (frequent dropouts which aren't a good thing).

wonderspark

Oct 11, 2010, 08:29 PM

nanofrog

Oct 11, 2010, 09:51 PM

Wow, excellent!

Ok, so another verification:
The Areca card says it supports RAID 3, which I want, but the shiny silver Sans Digital enclosures linked don't list RAID 3 as supported. I would assume the card determines which RAID levels are supported. Would I be correct?
Yes.

The box is "dumb", which means it's nothing more than an enclosure, PSU, cabling, and a small PCB that controls the disk status LED's.

They're as simple as it gets. :)

rdru

Oct 11, 2010, 10:40 PM

I am also buying those items. The best prices for SansDigital TR4x (http://www.amazon.com/Sans-Digital-TR4X-Storage-Enclosure/dp/B002AKKDBY/ref=sr_1_2?ie=UTF8&s=electronics&qid=1286854619&sr=8-2-spell) and TR8X (http://www.amazon.com/Sans-Digital-TR8X-Storage-Enclosure/dp/B002AKKDCI/ref=sr_1_2?ie=UTF8&s=electronics&qid=1286854716&sr=1-2) is at Amazon (free shipping with Amazon Prime) and Provantage (http://www.provantage.com/scripts/search.dll?QUERY=arc-1880&Submit.x=0&Submit.y=0) for the Areca Controllers. They come with the internal Mini-SAS cables. The SansDigital TR4X and the TR8X comes with the external Mini-SAS cables.

wonderspark

Jul 24, 2011, 05:37 PM

Well, it's been close to a year since I started this process, and now I've finally reached the point where I need to expand my video editing rig.

System: 2009 Nehalem Mac Pro 3.33GHz quad core, 16GB RAM.

I have five 2TB RE4 WD2003FYYS drives right now, and I'm going to order the Sans Digital TR8X 8-bay tower. I'll buy three more of these drives so I can fill it up with six drives, leaving two for swapping in during failure.

My question now is which card best suits my expansion: 1880ix-12, or the slightly cheaper 1880X with two external SAS ports?

My thinking is that I'd like to have the external 8-bay RAID (level 30) using the 2TB disks, and also have an internal three disk RAID 0 using the 1TB disks along with the single original OS disk (640GB) that came with the Mac Pro. (Eventually, I'd clone my OS to a 600GB Intel 320 series SSD, and switch it with the original which would become the backup.) I have a Voyager Q and two LaCie externals that I use for backups with all my drives that don't fit into the Mac Pro right now.

On hand:
5x 2TB RE4 WD2003FYYS (three are in Mac Pro in RAID, other two unused)
3x 1TB Hitachi HDE721010SLA330 (BTO from Apple, currently used for backups and OSX clone)
1x 640GB Hitachi (BTO from Apple)
1x 1TB LaCie d2 Quadra external (currently use for OSX clone)
1x 2TB LaCie d2 Quadra external (currently use for project backups)

So I want to remove the Apple RAID card, put in the appropriate Areca RAID card, and run the two different RAID sets. Which card will be better at doing this? Can I run the internal RAID 0 in software RAID and not mess with the Areca RAID 30, or is it better to run both RAID sets via the Areca, using the MaxConnect kit to connect the internal RAID?

nanofrog

Jul 25, 2011, 12:11 AM

I have five 2TB RE4 WD2003FYYS drives right now, and I'm going to order the Sans Digital TR8X 8-bay tower. I'll buy three more of these drives so I can fill it up with six drives, leaving two for swapping in during failure.
You can do this, or go ahead and stuff the extra pair of disks in the enclosure, and create them as Hot Spares. This means if you've ever a failed disk, one of them will automatically be assigned as the replacement, and the system automatically rebuild (unless you Disable the Auto Rebuild feature).

My question now is which card best suits my expansion: 1880ix-12, or the slightly cheaper 1880X with two external SAS ports?
This will depend on whether or not you only need 8 ports for say the next 3 - 5 years (takes capacity growth into account during this period). If not, then figure out the capacity growth, and go from there.

As a general rule, I recommend figuring what you need now and adding another 4x ports if you don't know (single user DAS configuration). The reason for this, is it's cheaper to just add disks than to replace all of them each time you out-grow your capacity.

An important note: You can connect an external enclosure to an internal port via a particular cable (SFF-8087 to SFF-8088 (http://www.pc-pitstop.com/sas_cables_adapters/8887-1M.asp)). As your planning to use SATA disks, do not go over 1.0 meters or you'll run into instability problems.

My thinking is that I'd like to have the external 8-bay RAID (level 30) using the 2TB disks, and also have an internal three disk RAID 0 using the 1TB disks along with the single original OS disk (640GB) that came with the Mac Pro. (Eventually, I'd clone my OS to a 600GB Intel 320 series SSD, and switch it with the original which would become the backup.) I have a Voyager Q and two LaCie externals that I use for backups with all my drives that don't fit into the Mac Pro right now.

On hand:
5x 2TB RE4 WD2003FYYS (three are in Mac Pro in RAID, other two unused)
3x 1TB Hitachi HDE721010SLA330 (BTO from Apple, currently used for backups and OSX clone)
1x 640GB Hitachi (BTO from Apple)
1x 1TB LaCie d2 Quadra external (currently use for OSX clone)
1x 2TB LaCie d2 Quadra external (currently use for project backups)
Looks like a good start in terms of gear to work with (and not have to buy everything at once, particularly enterprise grade HDD's). :D

But I have to ask... Why RAID 30?

You'd be better of running RAID 5 than 30 for n = 1 disk for redundancy (each have parity, but the parity is distributed across all members in 5 rather than a single dedicated parity disk as is the case in 3 = performance issues). This is why 3/4/30/40 aren't really used much in practice. You get better performance out of 5/6/50/60.

So I want to remove the Apple RAID card, put in the appropriate Areca RAID card, and run the two different RAID sets. Which card will be better at doing this? Can I run the internal RAID 0 in software RAID and not mess with the Areca RAID 30, or is it better to run both RAID sets via the Areca, using the MaxConnect kit to connect the internal RAID?
You can run a stripe set off of the ICH (built-in SATA ports in the MP, and the striping operations handled by OS X - creation is via Disk Utility). It's not going to present much of a load to the system (typically ~ 1 - 2% of a single core).

Then run the RAID 5 or 6 (if you're after an n = 2 fail-over) off of the Areca.

wonderspark

Jul 25, 2011, 01:08 AM

But I have to ask... Why RAID 30?

You'd be better of running RAID 5 than 30 for n = 1 disk for redundancy (each have parity, but the parity is distributed across all members in 5 rather than a single dedicated parity disk as is the case in 3 = performance issues). This is why 3/4/30/40 aren't really used much in practice. You get better performance out of 5/6/50/60.
I read an article about RAID levels for video editing, and it basically said the single parity disk of RAID 3 gave better performance than RAID 5 in case of a disk failure during the rebuild, and slightly better performance if using larger blocks. It's always been something that confused me, because the single parity disk argument makes sense to me, yet more people seem to use RAID 5.

I forgot to ask: is the Areca battery backup module a no-brainer must-have? I have two APC SUA1500s, but that does nothing for the RAID controller. The Apple RAID card has that battery conditioning function that warns me that the write cache is temporarily disabled every three months on the dot for 7.25 hours. Does the Areca BBU do an auto-reconditioning as well, or do you have to manually watch/care for it?

nanofrog

Jul 25, 2011, 04:09 AM

I read an article about RAID levels for video editing, and it basically said the single parity disk of RAID 3 gave better performance than RAID 5 in case of a disk failure during the rebuild, and slightly better performance if using larger blocks. It's always been something that confused me, because the single parity disk argument makes sense to me, yet more people seem to use RAID 5.
It's slower under normal operation though, due to a single parity disk (parity disk becomes the bottleneck for the set). So it's a matter of where users are willing to make sacrifices that best fit their needs. Most sacrifice rebuild speeds rather than under Normal conditions if they must make a compromise.

As per other levels on a hardware RAID card (5/6/50/60), the rebuild times can be improved by changing a firmware setting in the card (Background Operation Priority or similar heading; push it up, and the rebuilds go faster - performance of data operations during the rebuild slow down).

I forgot to ask: is the Areca battery backup module a no-brainer must-have? I have two APC SUA1500s, but that does nothing for the RAID controller. The Apple RAID card has that battery conditioning function that warns me that the write cache is temporarily disabled every three months on the dot for 7.25 hours. Does the Areca BBU do an auto-reconditioning as well, or do you have to manually watch/care for it?
Ideally, you run both the BBU as well as a good UPS.

Of the pair, the UPS does more for you, as it covers ongoing operations that can't fit all of the data within the cache (this is where the BBU falls short, is when the data from incomplete write operations can't be stored on the cache during a power loss).

Now the UPS won't last as long, so there's a limited amount of time the system can run before it's done. So if your storage operations aren't yet complete, your last hope is the BBU if the remaining data to be written fits within the cache.

If not, you may end up re-performing work to reconstruct missing/corrupted files (some software keeps a log/batch file of completed operations, and picks up where it left off after a reboot automatically <quite nice>; but it's not always the case).

So if you're tight on funds once you get the card, disks, ... needed to create the storage system, then the BBU can wait (since you already have a couple of good UPS's on hand). Then pick up the BBU at a later date when funds are available.

As per the card + BBU, it's all automatic (essentially useless if it wasn't).

wonderspark

Jul 25, 2011, 01:43 PM

It's slower under normal operation though, due to a single parity disk (parity disk becomes the bottleneck for the set). So it's a matter of where users are willing to make sacrifices that best fit their needs. Most sacrifice rebuild speeds rather than under Normal conditions if they must make a compromise.
That makes sense, too. I guess I was just put off by how bad RAID5 did on my system using only three disks. I'm sure it will be awesome using six. :cool:

As per other levels on a hardware RAID card (5/6/50/60), the rebuild times can be improved by changing a firmware setting in the card (Background Operation Priority or similar heading; push it up, and the rebuilds go faster - performance of data operations during the rebuild slow down).
I might dig into that once I get set up, but more than likely I'll just leave it alone if the performance is as great as I'm hoping. I love how it runs as a three disk RAID 0, of course, so if a six disk RAID 5 (or RAID 50, more likely) can match or better that, I'll be very happy.

Ideally, you run both the BBU as well as a good UPS.

Of the pair, the UPS does more for you, as it covers ongoing operations that can't fit all of the data within the cache (this is where the BBU falls short, is when the data from incomplete write operations can't be stored on the cache during a power loss).

Now the UPS won't last as long, so there's a limited amount of time the system can run before it's done. So if your storage operations aren't yet complete, your last hope is the BBU if the remaining data to be written fits within the cache.

If not, you may end up re-performing work to reconstruct missing/corrupted files (some software keeps a log/batch file of completed operations, and picks up where it left off after a reboot automatically <quite nice>; but it's not always the case).

So if you're tight on funds once you get the card, disks, ... needed to create the storage system, then the BBU can wait (since you already have a couple of good UPS's on hand). Then pick up the BBU at a later date when funds are available.

As per the card + BBU, it's all automatic (essentially useless if it wasn't).
I have one of the UPS units running just the Mac Pro and external drives. The second runs the two monitors (30" ACD and 22" Dell) and printer. Speakers and such are just plugged into the wall.

Thanks for the help again, and as always! I'll order the 1880ix-12 and the BBU. :)

wonderspark

Jul 25, 2011, 01:50 PM

You can do this, or go ahead and stuff the extra pair of disks in the enclosure, and create them as Hot Spares. This means if you've ever a failed disk, one of them will automatically be assigned as the replacement, and the system automatically rebuild (unless you Disable the Auto Rebuild feature).
Do the disks wear out as hot spares, though? I may have one hot spare and one in storage, and swap every now and then just to keep their lives extended. Maybe I'm slightly paranoid. :p

You can run a stripe set off of the ICH (built-in SATA ports in the MP, and the striping operations handled by OS X - creation is via Disk Utility). It's not going to present much of a load to the system (typically ~ 1 - 2% of a single core).

Then run the RAID 5 or 6 (if you're after an n = 2 fail-over) off of the Areca.
Yeah, that sounds nice. I want the RAID 0 for scratch, renders and exports that I don't care if I lose right away, but need fast disks for. The external RAID 50 will be for the project files and such that also need to be fast, which I'll be backing up nightly on other single drives.

nanofrog

Jul 25, 2011, 04:09 PM

That makes sense, too. I guess I was just put off by how bad RAID5 did on my system using only three disks. I'm sure it will be awesome using six. :cool:
Put the hardware together, create different levels, and test them. Seeing it first hand will teach you a lot IMO. ;)

As per your experience with RAID 5, keep in mind 2 issues with the configuration you tried:

Performance depends on parallelism of disks, so 3 is a bare minimum (gets a lot better as you add members (test this by creating 3, add a 4th, ... testing each set, until you've added all the disks you have). You'll see what I mean.
The Apple RAID Pro is slow. Not only is it limited to 4x disks, it's not as fast as other cards for the same configuration.

I might dig into that once I get set up, but more than likely I'll just leave it alone if the performance is as great as I'm hoping. I love how it runs as a three disk RAID 0, of course, so if a six disk RAID 5 (or RAID 50, more likely) can match or better that, I'll be very happy.
Test, and you'll be able to make sense of what you read on things like RAID Wiki.

I have one of the UPS units running just the Mac Pro and external drives. The second runs the two monitors (30" ACD and 22" Dell) and printer. Speakers and such are just plugged into the wall.
I'd get the printer off of the UPS, especially if it's a laser (warm-up/start-up draw is large = sucks down the batteries quickly when engaged and no longer warmed up).

Once you have the RAID equipment, try different combinations of gear on each UPS (reason = balance the draw on each unit as closely as possible). The point is to keep both units to as close a run time on batteries as possible (base shutdown off of the unit connected to the computer of course).

Thanks for the help again, and as always! I'll order the 1880ix-12 and the BBU. :)
:cool: NP. :)

You'll be happy with that card.

Do the disks wear out as hot spares, though? I may have one hot spare and one in storage, and swap every now and then just to keep their lives extended. Maybe I'm slightly paranoid. :p
Not as you're thinking. The spindle motors are running, but the heads aren't moving until engaged (assumes MAID = Disabled; MAID is a power management feature that shuts down the set if unused after a specified period of time). Generally speaking, it's left Disabled.

But you can do what you're thinking of, or just keep both in their bags on a shelf as needed (make sure you replace the silica gel packs say monthly to keep moisture out of them, as you do want to open the bag and check to see if the drives work).

Yeah, that sounds nice. I want the RAID 0 for scratch, renders and exports that I don't care if I lose right away, but need fast disks for. The external RAID 50 will be for the project files and such that also need to be fast, which I'll be backing up nightly on other single drives.
You'd be better off using a small, inexpensive SSD for scratch IMO (i.e. 40GB unit from OWC (http://eshop.macsales.com/item/OWC/SSDMX040/), which sells for $90 USD). Just as quick, but cheaper (better for random access too, if you do actually work with such files on occasion). Easily considered "disposable" at that price when it does wear out (just as a cheap HDD is these days). ;)

wonderspark

Aug 5, 2011, 06:18 PM

Got all my parts today!

Areca 1880ix-12
Areca BBU
Sans Digital TR8X 8-bay tower
(5) more WD2003FYYS 2TB RE4 HDD, totaling eight now.
SFF-8087 to SFF-8088 cable
(3) SFF-8087 to SFF-8087 cables
(2) SFF-8088 to SFF-8088 cables
Some kind of phone/network cable to RS-232 (or something like that) cable
RAID software CD: firmware / drivers / ArcHttp Proxy Server / CLI / Docs

I'm backing up everything and cloning my OS drive, which is on the original 640GB HDD that came with the Mac Pro. The Apple RAID card was also built to order from Apple, so I've never removed it before. I'm excited and a bit nervous, but hopefully everything works in a couple hours or so!

I'll be first building a RAID 0 to see if it all works ok, do some speed tests, etc. Then I'll build a RAID 3 out of six drives with two hot spares installed.

:)

wonderspark

Aug 6, 2011, 12:34 AM

First test done for RAID0 using five disks. I did this to approximate what I might see in a six-disk RAID3. I used xbench 1.3, since I don't know what else to use for disk speed testing.
Result:
Disk Test = 1431.45

Sequential = 907.07
- Write 4K blocks = 808.51 MB/sec
- Write 256K blocks = 1221.38 MB/sec
- Read 4K = 106.30 MB/sec
- Read 256K = 1158.14 MB/sec

Random = 3392.92
- Write 4K = 152.52 MB/sec
- Write 256K = 1256.89 MB/sec
- Read 4K = 101.80 MB/sec
- Read 256K = 1155.77 MB/sec

Seems pretty good to me. What do you all think? Should I use something else to test with?

Next up, six-disk RAID3 tests!

nanofrog

Aug 6, 2011, 12:52 AM

Seems pretty good to me. What do you all think? Should I use something else to test with?
AJA System Test (http://www.aja.com/ajashare/AJASystemTest.zip) (direct download - Mac version)

There's a lot under Windows if you have a copy installed, so that's something to consider as well.

wonderspark

Aug 6, 2011, 01:58 AM

wonderspark

Aug 6, 2011, 12:07 PM

Well, I wonder if something is wrong. I started initializing this RAID3 array 10.5 hours ago, and it's now 26% complete.

I do have the BBU installed, and it was charging for part of that time. It now shows 100% charged.

Also, I have the enclosure connected via the two different parts of the card... one via an internal port with the SFF-8087 to SFF-8088 cable, and the other via the external SFF-8088 port.

I have the eight disks inside sharing the load of the RAID set such that three of the disks on each enclosure make up the set, and the fourth disk in each half are a hot spare. (Slots 1,2,3,5,6 and 7 make up the RAID3, and slots 4&8 make up global hot spares.) I thought that might best distribute the data across the two cables / enclosures / ports on Areca.

All eight disks are WD2003FYYS RE4 2TB models, bought at three different times from Amazon and Provantage. The Areca has the standard 1GB memory. The rebuild time is set to low.

What has me wondering is the 4K read times from my testing of the five disk RAID0 test above. I had set the block size to 128K, so that made me think it was due to the smaller blocks, but then why was the 4K write speed still so good?

Thanks for your input!

Edit: It's actually internal slots 1-4, and external slots 1-4. The slots 5-12 are empty.

nanofrog

Aug 6, 2011, 12:11 PM

Thanks, nanofrog! I don't have Windows installed right now, but I noticed the boot time is way faster without the Apple RAID card in. I'd say it's at least two or three times faster!

I'll give it a run as soon as this RAID3 is done building... probably when the sun rises for me. :)
RAID card firmware can add anything between an additional 25 seconds to over a minute, depending on the card. The disk count attached to the card can matter as well (staggered spin-up).

As it happens, Areca's are rather quick (I typically see ~ 25 second mark on avg.). :)

wonderspark

Aug 6, 2011, 12:19 PM

RAID card firmware can add anything between an additional 25 seconds to over a minute, depending on the card. The disk count attached to the card can matter as well (staggered spin-up).

As it happens, Areca's are rather quick (I typically see ~ 25 second mark on avg.). :)

It is about that... 25-30 seconds. I'll have to get an official time.

So, do you think the long initialization time and 4K read speeds are normal from my earlier test of RAID0? I read that this should take from 6-12 hours to complete, and I'm coming up to hour 11 right now... 26.8% complete.

nanofrog

Aug 6, 2011, 09:56 PM

...I noticed the boot time is way faster without the Apple RAID card in. I'd say it's at least two or three times faster!
As I said, the Apple RAID Pro is s-l-o-w in every respect. :eek: :p

Well, I wonder if something is wrong. I started initializing this RAID3 array 10.5 hours ago, and it's now 26% complete.

I do have the BBU installed, and it was charging for part of that time. It now shows 100% charged.
I don't know what your card's settings are, but I suspect the Initialization setting needs to be changed..

Set it to Initialization = Foreground, and see if that doesn't speed things up for you (default setting = retain processor cycles for other actions, as using another array that's functioning properly, as would be the case in a system that's already been deployed, such as a working SAN). ;)

Also, I have the enclosure connected via the two different parts of the card... one via an internal port with the SFF-8087 to SFF-8088 cable, and the other via the external SFF-8088 port.

I have the eight disks inside sharing the load of the RAID set such that three of the disks on each enclosure make up the set, and the fourth disk in each half are a hot spare. (Slots 1,2,3,5,6 and 7 make up the RAID3, and slots 4&8 make up global hot spares.) I thought that might best distribute the data across the two cables / enclosures / ports on Areca.
Just make sure the disks aren't trying to share the same ports (read up on the manual - if it's not there or you're unsure, contact Areca as there are instances where the external SFF-8088 shares the same ports as one of the internal SFF-8087 ports).

I don't think this is the case however, given the performance you've posted on a RAID 0 set of those disks.

It's a good idea to be sure is all, particularly for future implementations on that card.

All eight disks are WD2003FYYS RE4 2TB models, bought at three different times from Amazon and Provantage. The Areca has the standard 1GB memory. The rebuild time is set to low.
This is definitely your problem then... (see above).

What has me wondering is the 4K read times from my testing of the five disk RAID0 test above. I had set the block size to 128K, so that made me think it was due to the smaller blocks, but then why was the 4K write speed still so good?
Cache. ;)

So, do you think the long initialization time and 4K read speeds are normal from my earlier test of RAID0? I read that this should take from 6-12 hours to complete, and I'm coming up to hour 11 right now... 26.8% complete.
Initialization time is due to the current setting.

As per the write performance, that's where the 1GB cache kicks in (system unloads data to be written to the cache, and once this transfer is complete, the performance data is calculated = fast numbers - even though it may not actually be on the disks yet).

If you want to see what the disks can do without the cache engaged, check the Disable Cache box in AJA. It's still fast, but you'll see the influence the cache makes during writes. ;)

wonderspark

Aug 6, 2011, 10:14 PM

Ok, that makes sense.

It looks like I could change the settings from low to high on the rebuild right now while it's still initializing. If I understand correctly, setting it to high will make rebuilds (and I presume initialization) faster, but boot times slower (which I'm used to anyway) and performance slower during a rebuild, right? If that's it, then I'm fine with changing it now. In fact, I'll do that. And if it doesn't make it speed up, perhaps I just need to figure out how to stop the initialization and shut it down to try again.

Ok, I set it to high priority background, and when I started it, I set it to foreground initialization.

Slot assignments:

SLOT 01(F) 6HDD RAID 1 2000.4GB WDC WD2003FYYS-02W0B0
SLOT 02(10) 6HDD RAID 1 2000.4GB WDC WD2003FYYS-02W0B0
SLOT 03(11) 6HDD RAID 1 2000.4GB WDC WD2003FYYS-02W0B0
SLOT 04(12) Hot Spare 2000.4GB WDC WD2003FYYS-02W0B0 [Global]
SLOT 05 N.A. N.A. N.A.
SLOT 06 N.A. N.A. N.A.
SLOT 07 N.A. N.A. N.A.
SLOT 08 N.A. N.A. N.A.
SLOT 09 N.A. N.A. N.A.
SLOT 10 N.A. N.A. N.A.
SLOT 11 N.A. N.A. N.A.
SLOT 12 N.A. N.A. N.A.
EXTP 01(A) 6HDD RAID 1 2000.4GB WDC WD2003FYYS-02W0B0
EXTP 02(B) 6HDD RAID 1 2000.4GB WDC WD2003FYYS-02W0B0
EXTP 03(D) 6HDD RAID 1 2000.4GB WDC WD2003FYYS-02W0B0
EXTP 04(E) Hot Spare 2000.4GB WDC WD2003FYYS-02W0B0 [Global]

nanofrog

Aug 7, 2011, 01:00 AM

It looks like I could change the settings from low to high on the rebuild right now while it's still initializing.
Yes you can. :)

If I understand correctly, setting it to high will make rebuilds (and I presume initialization) faster, but boot times slower (which I'm used to anyway) and performance slower during a rebuild, right?
You've not quite grasped it, but that's not uncommon (takes time to figure it all out).

Initialization only affects that function, not rebuilds (Background Process deals with rebuild speeds).

Read the Manual carefully, as there's a wealth of information in there. Multiple times if you have to, as it will help understand what all is going on with the settings (don't just skip around, or you will likely be left scratching your head). ;)

wonderspark

Aug 7, 2011, 04:18 AM

Well, I'm 2/3 done now at 66.9%. I guess it will be done in about 12 or 13 more hours. I'll do tests then, and consider if I want to delete it and set up a RAID5 to compare. Given that this will have taken about 40 hours to build a RAID3, I am not sure I'll be up for waiting that long again. :p

wonderspark

Aug 7, 2011, 02:59 PM

92.3%... almost there! Been 37 hours.

I've read this user's manual thoroughly, and took note of mention that this is an 1880ix-12 (the number 12 must mean 12 ports, not 16) and it says there are "12 internal ports with [an] additional 4 external ports." Then it says, "...attaches directly to SATA/SAS midplanes with 3 SFF-8087 internal connectors or increase capacity using one additional SFF-8088 external connector."

This seems to say there are 16 ports, but I believe it's more like how you (nanofrog) mentioned... that there are 12 ports, and some of the internal ones are shared with the external 4 ports somehow. The fact that my RAID hierarchy shows half of the drives on internal slots and the other half on external slots (leaving 8 internal slots unused) makes me think they must be sharing somehow.

Something else I find curious is that there are two "enclosures" listed, like this:
Enclosure#1 : ARECA SAS RAID AdapterV1.0
Device Usage Capacity Model
Slot#1 N.A. N.A. N.A.
Slot#2 N.A. N.A. N.A.
Slot#3 N.A. N.A. N.A.
Slot#4 N.A. N.A. N.A.
Slot#5 N.A. N.A. N.A.
Slot#6 N.A. N.A. N.A.
Slot#7 N.A. N.A. N.A.
Slot#8 N.A. N.A. N.A.
Enclosure#2 : Areca ARC-8018-.01.07.0107(C)[5001B4DB0091E03F]
Device Usage Capacity Model
SLOT 01(F) 6HDD RAID 1 2000.4GB WDC WD2003FYYS-02W0B0
SLOT 02(10) 6HDD RAID 1 2000.4GB WDC WD2003FYYS-02W0B0
SLOT 03(11) 6HDD RAID 1 2000.4GB WDC WD2003FYYS-02W0B0
SLOT 04(12) Hot Spare 2000.4GB WDC WD2003FYYS-02W0B0 [6HDD RAID 1 ]
SLOT 05 N.A. N.A. N.A.
SLOT 06 N.A. N.A. N.A.
SLOT 07 N.A. N.A. N.A.
SLOT 08 N.A. N.A. N.A.
SLOT 09 N.A. N.A. N.A.
SLOT 10 N.A. N.A. N.A.
SLOT 11 N.A. N.A. N.A.
SLOT 12 N.A. N.A. N.A.
EXTP 01(A) 6HDD RAID 1 2000.4GB WDC WD2003FYYS-02W0B0
EXTP 02(B) 6HDD RAID 1 2000.4GB WDC WD2003FYYS-02W0B0
EXTP 03(D) 6HDD RAID 1 2000.4GB WDC WD2003FYYS-02W0B0
EXTP 04 N.A. N.A. N.A.

Enclosure #1 is empty, with 8 slots shown. I don't understand this.
Enclosure #2 seems like the only "real" enclosure, and makes it look like there are 16 individual slots. I'm going to email Areca now. I'll let you know what they say.

92.9%!

nanofrog

Aug 7, 2011, 03:11 PM

Keep in mind, that it may take them a few days to reply via email as they're based in Taipei, Taiwan. ;)

In the mean-time, you can play. :D

wonderspark

Aug 7, 2011, 04:02 PM

Keep in mind, that it may take them a few days to reply via email as they're based in Taipei, Taiwan. ;)

In the mean-time, you can play. :D
That explains the interesting English! ;)

I'm so close now... 95.2%! I just sent the email. I should think they'll get it instantly, but maybe take some time to translate, respond, and translate back to English... hahaha!

I also ordered another cable to use internal ports only. If nothing else, I'll be able to hook up another 4-bay tower.

nanofrog

Aug 7, 2011, 04:44 PM

That explains the interesting English! ;)

I'm so close now... 95.2%! I just sent the email. I should think they'll get it instantly, but maybe take some time to translate, respond, and translate back to English... hahaha!

I also ordered another cable to use internal ports only. If nothing else, I'll be able to hook up another 4-bay tower.
Actually, they do speak English (you can pick up the phone and call them :eek: :D).

If you've already dealt with them via email, their responses can be a bit cryptic though and take time to figure out (I've run into this from time to time). As per the delay, that's likely to do with both where the dates fall (i.e. weekday in one country is a week-end in another), and how much of a load they've got (first come, first served).

Not a bad idea to have that extra cable on hand, but it's not a necessity (just figure out what ports are shared and don't use them 2x if you can help it).

BTW, their chips are custom and control 12x disks, so the external port would have to share drives with one of the internal ports (1x chip on the PCB under the heat sink).

wonderspark

Aug 7, 2011, 04:56 PM

Not a bad idea to have that extra cable on hand, but it's not a necessity (just figure out what ports are shared and don't use them 2x if you can help it).

BTW, their chips are custom and control 12x disks, so the external port would have to share drives with one of the internal ports (1x chip on the PCB under the heat sink).
I used the "first" plug for the internal ports... the 1-4 ones, thinking the last ones (9-12) would be more likely to be the shared ones, but I have no idea. Just gotta wait until they tell me.

Maybe I should have plugged into the "middle" one. (5-8)

nanofrog

Aug 7, 2011, 05:08 PM

I used the "first" plug for the internal ports... the 1-4 ones, thinking the last ones (9-12) would be more likely to be the shared ones, but I have no idea. Just gotta wait until they tell me.

Maybe I should have plugged into the "middle" one. (5-8)
You could also move the internal connector to other ports (while you wait for a response), and see what happens in the information sections in ARCHTTP (web access to card's control panel). ;)

wonderspark

Aug 7, 2011, 05:25 PM

You could also move the internal connector to other ports (while you wait for a response), and see what happens in the information sections in ARCHTTP (web access to card's control panel). ;)
But you're not suggesting I do that when my initialization is at 98.6% completion, right? :p

Can I swap the cable (I'm thinking with power off) to another internal port and not screw up the RAID? I was afraid to try that, thinking I'd have to start all over.

nanofrog

Aug 7, 2011, 05:32 PM

But you're not suggesting I do that when my initialization is at 98.6% completion, right? :p
I'd wait until the initialization process is complete, to be safe. ;)

But the card is actually designed to pick back up if a fault happens during this process, such as loss of power (unless you've disabled it in the settings). :eek: These sorts of features are another reason a hardware RAID controller stands out from software implementations (and still be able to use the system for other tasks). :D

Can I swap the cable (I'm thinking with power off) to another internal port and not screw up the RAID?
Yes.

The MP does not have an inrush current limiter (http://en.wikipedia.org/wiki/Inrush_current_limiter) (what you need for Hot Plug support), so do make sure the power is OFF before moving the cable. :p

wonderspark

Aug 7, 2011, 05:34 PM

I'd wait until the initialization process is complete, to be safe. ;)

But the card is actually designed to pick back up if a fault happens during this process, such as loss of power (unless you've disabled it in the settings). :eek: These sorts of features are another reason a hardware RAID controller stands out from software implementations (and still be able to use the system for other tasks). :D

Yes.

The MP does not have an inrush current limiter (http://en.wikipedia.org/wiki/Inrush_current_limiter) (what you need for Hot Plug support), so do make sure the power is OFF before moving the cable. :p

Excellent! I will try that very soon. This last one percent is going to kill me. :p 99%!

nanofrog

Aug 7, 2011, 05:36 PM

Excellent! I will try that very soon. This last one percent is going to kill me. :p 99%!
Chewed through all of your fingernails and pencils yet? :eek: :p

wonderspark

Aug 7, 2011, 05:38 PM

Chewed through all of your fingernails and pencils yet? :eek: :p
Dude... I wish I'd known this would take so long... I'd have taken a road trip or gone camping or something.

It has given me the time to read the manual twice or more, and learn a lot from you. :cool:

nanofrog

Aug 7, 2011, 05:46 PM

Dude... I wish I'd known this would take so long... I'd have taken a road trip or gone camping or something.

It has given me the time to read the manual twice or more, and learn a lot from you. :cool:
I've seen others running into this, but they've all started on the Default settings, so I don't know if this particular model is slow or not.

Generally speaking, Areca's aren't too bad for initialization times vs. other makes (most Areca's I've worked with are 12x1ML's <SATA only & released in 2006> and 1680 series models <SAS/SATA released in 2008>, which where the top of the line model at the time, as the 1880 series is now).

Capacity and member count matter too, so it's been hard to nail down. But I suspect it's the use of the Default settings that has been such a PITA...

wonderspark

Aug 7, 2011, 05:52 PM

nanofrog

Aug 7, 2011, 06:03 PM

You know, I was reading up on the drives I'm using (WD 2TB RE4 WD2003FYYS) and noticed the sustained data transfer rate is only 138MB/sec, whereas my stock Apple 1TB Hitachi drives are 178-ish MB/sec. That surprised me. So yeah, 2TB x 6 probably didn't help so much.
Where those Hitachi's SAS? Particularly 15k rpm units?

I ask, as 15k would be able to generate those kinds of throughputs (and they could be used with your card - you'd have to test them, but all SAS disks are enterprise grade, so it has the right timings). If you've the Exact model number of those Hitachi's, then you can check to see if they're on the HDD Compatibility List, which I expect they would be.

Now once it's done with the initialization, you've a lot of work to do (and learning first hand how the card will react).

Performance testing.
Failure testing.

Place test/dummy data on the array (after performance testing), and do the following...

Start a write, and pull the power (tests out the UPS).
Do this again with the UPS out of the loop (plug the system directly into the wall = tests out the BBU).
Start another read and pull a disk (pull it off of the external, as it does have an inrush current limiter for Hot Plugging).
Pay attention to the performance of RAID 3.
Replace the disk, and watch what the card does again, including performance.
Do the same with a write (yank disk as above, and again, pay attention to the performance).

It's a lot of work, but you'll thank yourself later, as you'll get a good understanding of how the card actually works, and what to expect in the event of a real failure (most data loss is the result of user error, even in fault states/degraded arrays on hardware RAID controllers).

So it really is in your best interest to figure this out now rather than later (when it's important and can either cost you your data, or a lot of additional time that could have been prevented by knowing the right methodology to begin with). ;)

wonderspark

Aug 7, 2011, 06:07 PM

Where those Hitachi's SAS? Particularly 15k rpm units?

I ask, as 15k would be able to generate those kinds of throughputs (and they could be used with your card - you'd have to test them, but all SAS disks are enterprise grade, so it has the right timings). If you've the Exact model number of those Hitachi's, then you can check to see if they're on the HDD Compatibility List, which I expect they would be.

Now once it's done with the initialization, you've a lot of work to do (and learning first hand how the card will react).

Performance testing.
Failure testing.

Place test/dummy data on the array (after performance testing), and do the following...

Start a write, and pull the power (tests out the UPS).
Do this again with the UPS out of the loop (plug the system directly into the wall = tests out the BBU).
Start another read and pull a disk (pull it off of the external, as it does have an inrush current limiter for Hot Plugging).
Pay attention to the performance of RAID 3.
Replace the disk, and watch what the card does again, including performance.
Do the same with a write (yank disk as above, and again, pay attention to the performance).

It's a lot of work, but you'll thank yourself later, as you'll get a good understanding of how the card actually works, and what to expect in the event of a real failure (most data loss is the result of user error, even in fault states/degraded arrays on hardware RAID controllers).

So it really is in your best interest to figure this out now rather than later (when it's important and can either cost you your data, or a lot of additional time that could have been prevented by knowing the right methodology to begin with). ;)
Hitachi model # HDE721010SLA330
Just looked it up, and it says 167.6MB/sec, conversion from 1406Mbps off a Hitachi specs sheet I found.

Just finished! Testing begins! I'm so excited.

wonderspark

Aug 7, 2011, 06:18 PM

First RAID3 test with AJA: (screenshot attached)
I chose read/write 16GB test with video frame 1920x1080 8-bit.

nanofrog

Aug 7, 2011, 06:18 PM

Hitachi model # HDE721010SLA330
Just looked it up, and it says 167.6MB/sec, conversion from 1406Mbps off a Hitachi specs sheet I found.

Just finished! Testing begins! I'm so excited.
Surprising for a 7200 rpm SATA, but it is a consumer grade Hitachi, so getting your data off of that was a good idea.

wonderspark

Aug 7, 2011, 06:21 PM

Here's Xbench 1.3:

This is all prior to swapping any cables around, of course.

wonderspark

Aug 7, 2011, 06:27 PM

This is my current RAID and volume setup. I'm going to shut down and switch the internal cable to another port first, and see what changes.

nanofrog

Aug 7, 2011, 06:40 PM

First RAID3 test with AJA: (screenshot attached)
I chose read/write 16GB test with video frame 1920x1080 8-bit.
Not terrible, but I'd have hoped for more.

Of course, you can always take one of the Hot Spares, and make it an active disk (leaving 1x Hot Spare) via Online Expansion.

But I still think you'd be better off with RAID 5 (same fail-over as RAID 3).

BTW, you don't work with 10bit?
And test with the cache active, and watch what happens. ;)
Here's Xbench 1.3:

This is all prior to swapping any cables around, of course.
Go ahead and see what happens performance-wise after swapping cables (move it to each port; shut downs between them of course...).

This is my current RAID and volume setup. I'm going to shut down and switch the internal cable to another port first, and see what changes.
Good luck, and don't forget to try each port (should tell you which internal group is sharing with the external ports, all things being equal - same configuration <array level, member count, and disks used>, same test).

wonderspark

Aug 7, 2011, 07:05 PM

I feel dumb, but I cannot get the internal cable to release from the card. It was tight going on, and I'm afraid the port is going to break off if I pull harder. How do you release an SFF-8087 connector?

wonderspark

Aug 7, 2011, 07:29 PM

I can get the cable that came with the card to go on and off pretty easy. This PC-PitStop cable is really locked in there. Gah!

wonderspark

Aug 7, 2011, 07:39 PM

Eff this. I guess it's a permanent part of the card now. I'll have to wait a few days for a second cable to show up in the mail.

nanofrog

Aug 7, 2011, 09:23 PM

I can get the cable that came with the card to go on and off pretty easy. This PC-PitStop cable is really locked in there. Gah!
There are variances to both connectors (card and cable), so they can be tight from time to time.

Just make sure the locking tab is sufficiently depressed, and be cautious (they are fairly stout, but can be broken). Rocking/wiggling side to side while tugging on it might help too (could tell you if the locking tab is sufficiently depressed, though usually you'll feel the connector slip backwards when it is due to a spring action between the tab and retaining clip).

wonderspark

Aug 7, 2011, 09:47 PM

nanofrog

Aug 7, 2011, 11:38 PM

I tugged, rocked, wiggled and everything for probably a half hour. It's in there good. I put it all back together because I heard some unsettling popping and snapping sounds as I tugged the crud out of it, and I wanted to make sure I didn't break something. It still works, but I've given up on it. :mad:
Yikes.

You did the right thing though IMO, as lead-free solder isn't all that forgiving in terms of flexibility. So if you flexed the PCB too much, you'd create bad solder joints, and the card would either cease functioning, or be flaky (i.e. act up when up to working temp, or some part that is only used rarely).

Even though you've already pulled the card as a means of removing the cable, it's your best chance IMO without damaging anything (also make sure you're not statically charged, as an Electro Static Discharge can be lethal to electronics).

Simply leaving the system plugged in (attached to earth ground), and touching the metal case before touching any electronics will make sure you're discharged before removing anything from the system. ;)
For now, I pulled the UPS plug, and was alarmed to see the battery drop to 40% in about ten seconds. I think the big UPS (SUA1500) needs a new battery. I bought this used from a company that was going out of business for $25, and that was December 2009, so it's an old battery for sure. I pulled the smaller UPS (BackUPS RS 1500) that only has monitors and external drives hooked up, and it's been running on battery now for about 20 minutes. It says I have 25 minutes running time left. I bought that one new about six months prior to the SUA1500... so summer of 2009. I know it's history, so I'm sure it's still in much better shape. I'll be ordering the SUA1500 battery tonight or tomorrow. :eek:
The batteries definitely need changed.

Typically, you'll need to swap them out about every 3 years or so, though I've seen them last as long as 5 (don't bet on this long though, particularly with more recent batteries due to cost cutting).
When I had the computer off, I pulled out the Areca to try to get the cable off easier, and when I put it all back together, the battery was drained to 96%. That was about an hour total, probably. I know that's not the same as testing it like you said, during a read/write with the computer plugged into the wall (and unplugged) but I think I'm going to hold off until the new UPS battery comes in. I planned to replace that battery a long time ago, but kinda forgot about it. Heh.
I assume you're still talking about the UPS battery, and not the BBU for the card.

If it is the card's BBU that was down to 4% remaining in an hour, you probably got the wrong one (they introduced a new one for some of the 1880 series, which starts with the ARC-1880ix12 IIRC, as the larger port cards need more power = battery is rated for a higher current).

wonderspark

Aug 7, 2011, 11:54 PM

Yikes.

You did the right thing though IMO, as lead-free solder isn't all that forgiving in terms of flexibility. So if you flexed the PCB too much, you'd create bad solder joints, and the card would either cease functioning, or be flaky (i.e. act up when up to working temp, or some part that is only used rarely).

I would be very sad if I busted my card. I hope I didn't weaken any solder.

I assume you're still talking about the UPS battery, and not the BBU for the card.

If it is the card's BBU that was down to 4% remaining in an hour, you probably got the wrong one (they introduced a new one for some of the 1880 series, which starts with the ARC-1880ix12 IIRC, as the larger port cards need more power = battery is rated for a higher current).
No, I was talking about the BBU, and it was only down 4%. It still had 96% charge. I worded that poorly.

I did order a new SmartUPS battery from APC, even though other vendors had knockoffs for $100 less. I figure the free shipping on returning the old battery to APC is worth it for the reduced hassle and environment.

I also ordered the 8087-8088 cable. Once I get that, I'll see if it fixes the RAID3, then compare it to RAID5. I've found a couple of instances where people compared the same system in both formats, and RAID3 was about 100MB/sec faster every time, so I'm expecting the same, but it will be fun to test.

wonderspark

Aug 8, 2011, 02:08 AM

Well, got an answer about ports from Areca.

"Your Question : Is the 1880ix-12 using a total of 12 ports, or are there 16 ports when you include the 4 external ports via SFF-8088?

I'm connecting an external RAID tower via two cables... one is SFF-8087-SFF-8088 cable via internal port, and the second is SFF-8088 via external port. I've been initializing a RAID3 with six 2TB disks and one hot-spare, and it's been 38 hours so far with 95.2% complete. Is this normal, or have I got a conflict somewhere?

(Answer)
Dear Sir/Madam,

you can use all these external ports and internal ports at same time.
so you can say that it is 16 ports in total.

and initialization time may vary with drive, you can monitoring the drive
status of each array member drive to find out a possible problem. if there
have unstable drive or problem communication, the drive information page
will show you some errors in the error count.

Best Regards,

Kevin Wang

Areca Technology Tech-support Division
Tel : 886-2-87974060 Ext. 223
Fax : 886-2-87975970"

This makes me kinda bummed. I checked all seven drives in the RAID, and they all have zero error count, and say "normal" status. If this is all true, then the extra cable should not make any difference, and this is just the performance I'll be getting in this configuration.

Having spent 40 or so hours initializing, I'm hesitant to just delete it and build a RAID5 right away, with the chance that it won't be any better. I do have some work to do, so I think I'll leave it alone until the new internal cable arrives, and see if it makes any difference first.

I really appreciate all your help and attention with this. It's been very helpful!

nanofrog

Aug 8, 2011, 02:29 AM

:cool: NP. :)

You can share ports with multiple drives on these cards, so that's not an issue (will work without the need of additional hardware, such as a SAS expander). There is a performance penalty for doing so however (exact amount is dependent on a few different factors).

In your case, it won't be much in the current configuration, but testing out the disks on different ports for performance should reveal which ports are sharing bandwidth with the external port. Worst case, if you need additional performance, just add one of the existing Hot Spares to the set via Online Expansion.

Test it out (RAID 3), and see what you get, particularly during the failure conditions listed previously.

You can also look for online reviews (arecaraid.cineraid.com (http://arecaraid.cineraid.com/) has had a good forum for this and thread on the 1880 series, but when I checked, the site is down).

wonderspark

Aug 8, 2011, 07:23 PM

In looking at other speed tests, I think I'm pretty happy after all. Take a look at these comparisons of my three main volumes.
One is just the original 640GB OS drive in slot 1 of the Mac Pro.
Next is the three original 1TB Hitachi/Apple drives in a RAID0, also internal (software) RAID.
Last is the RAID3 as is, with six 2TB drives.

Results 74.26
System Info
Xbench Version 1.3
System Version 10.6.8 (10K549)
Physical RAM 16384 MB
Model MacPro4,1
Drive Type Hitachi HDE721064SLA360 (640GBx1)
Disk Test 74.26
Sequential 149.84
Uncached Write 170.69 104.80 MB/sec [4K blocks]
Uncached Write 176.97 100.13 MB/sec [256K blocks]
Uncached Read 100.89 29.53 MB/sec [4K blocks]
Uncached Read 189.64 95.31 MB/sec [256K blocks]
Random 49.36
Uncached Write 17.30 1.83 MB/sec [4K blocks]
Uncached Write 158.71 50.81 MB/sec [256K blocks]
Uncached Read 93.79 0.66 MB/sec [4K blocks]
Uncached Read 159.42 29.58 MB/sec [256K blocks]
-----------------------
Results 189.26
System Info
Xbench Version 1.3
System Version 10.6.8 (10K549)
Physical RAM 16384 MB
Model MacPro4,1
Drive Type Internal RAID 0 3TB (3x1TB)
Disk Test 189.26
Sequential 250.86
Uncached Write 551.91 338.86 MB/sec [4K blocks]
Uncached Write 618.92 350.18 MB/sec [256K blocks]
Uncached Read 91.48 26.77 MB/sec [4K blocks]
Uncached Read 630.24 316.75 MB/sec [256K blocks]
Random 151.95
Uncached Write 71.48 7.57 MB/sec [4K blocks]
Uncached Write 508.63 162.83 MB/sec [256K blocks]
Uncached Read 169.70 1.20 MB/sec [4K blocks]
Uncached Read 223.51 41.47 MB/sec [256K blocks]
-------------------------
Results 271.20
System Info
Xbench Version 1.3
System Version 10.6.8 (10K549)
Physical RAM 16384 MB
Model MacPro4,1
Drive Type Areca Video (RAID3)
Disk Test 271.20
Sequential 896.46
Uncached Write 1361.15 835.73 MB/sec [4K blocks]
Uncached Write 2158.48 1221.27 MB/sec [256K blocks]
Uncached Read 353.44 103.44 MB/sec [4K blocks]
Uncached Read 2300.54 1156.23 MB/sec [256K blocks]
Random 159.77
Uncached Write 1533.41 162.33 MB/sec [4K blocks]
Uncached Write 42.14 13.49 MB/sec [256K blocks]
Uncached Read 2022.74 14.33 MB/sec [4K blocks]
Uncached Read 6278.27 1164.98 MB/sec [256K blocks]

What I'm seeing is that there are these consistently terrible numbers in the same areas of three different volumes, using Xbench. When I run the AJA test on all three, I'm seeing just a simple set of numbers (write / read) and they're pretty consistent.

AJA tests:
- 640GB single internal drive = 90MB/sec both write/read
- 3x1TB RAID 0 = 315MB/sec both write/read
- 6x2TB RAID 3 = 603MB/sec write, 550MB/sec read

Basically, I've done what I set out to do, which was greatly increase my capacity, speed AND reliability. I think the Xbench tests messed with my head somewhat, with those weird 4K numbers. Areca tech support has assured me twice now that the 1880ix-12 has 16 ports (that the 4 on the external are not sharing any differently than the 12 internal ports, as they all come from one chip.) So even though I'll definitely test for a difference with the second internal cable, I will still be happy if there is no change. I'd like to see it improve, but I think I'm in really good shape!

I'll also try the RAID 5, just to see how things go, and follow up on all of this.

randomerratum

Aug 8, 2011, 07:34 PM

wonderspark

Aug 8, 2011, 07:46 PM

Just thought I'd chime in-

I get pretty good performance on two systems running via a High Point Rocket Raid card... (1080 29.97 ProRes 4:22) except when we crash. If the Mac Pro crashes, it typically kicks the card and you have to get the RAID mounted with disk utility which takes multiple hours to come back online.

Crashes don't happen often, but when they do- it's usually when we can least afford a few hours downtime. We've actually moved over to our G-Raid FW800 backups just so freezes or crashes don't shut us down all day.

Interesting...
What do you consider a crash? I'm trying to remember, and I want to say since December 2009 I had one, maybe two instances where Premiere crashed on me, but I don't think I've had the whole Mac crash yet. (Knock on wood!) I started with CS3, and since moved to CS5, heavily into After Effects and either 1080p or higher footage. (I was editing some crazy 4928x3264 timelapse video footage I created from over 70,000 still Nikon images, and Premiere choked on it. After Effects handled it great, however!)

I'm hoping I've covered my bases well in that regard. I have two UPS units for just this system; one is an APC SmartUPS 1500; the other is a smaller APC BackUPS RS 1500. The Areca RAID card has the battery back up installed as well. I've worked through several nasty lightning storms that knocked out power without a glitch so far. Here's hoping that trend continues!

What are you editing with when your crashed occur, may I ask? I've only just begun to use FCP X. (Not keen on it yet.)

nanofrog

Aug 8, 2011, 09:13 PM

Areca tech support has assured me twice now that the 1880ix-12 has 16 ports (that the 4 on the external are not sharing any differently than the 12 internal ports, as they all come from one chip.)

I'll also try the RAID 5, just to see how things go, and follow up on all of this.
What's thrown me, is when I look at their 1880 series cards, the 1880ix16 has 2x chips, not 1x, which is what I'd expect to be the case if there were 16x ports per controller chip.

As per performance between 3 and 5, 5 should give you another ~60MB/s or so than 3 under Normal conditions, assuming each of those disks is capable of 130MB/s or so.

I get pretty good performance on two systems running via a High Point Rocket Raid card... (1080 29.97 ProRes 4:22) except when we crash. If the Mac Pro crashes, it typically kicks the card and you have to get the RAID mounted with disk utility which takes multiple hours to come back online.
I'd be interested in elaboration as to what exactly is going on as well, as that sounds really odd (though Highpoint is the one company that doesn't surprise me, as they have a horrible reputation on the support end).

A software crash shouldn't take out the array, even if the OS is wiped out (assumes the OS is on a physically separate volume from the data).

wonderspark

Aug 9, 2011, 01:10 AM

What's thrown me, is when I look at their 1880 series cards, the 1880ix16 has 2x chips, not 1x, which is what I'd expect to be the case if there were 16x ports per controller chip.
Yeah, on this page, (http://www.areca.com.tw/products/1880.htm) they show "16-28 ports" on the 12/16/24 models, and 8-16 ports on the i/x/LP and ixl8/12 models, so all the 12's have 16 ports according to the info... backed up by the emails I've received as well.

I misunderstood this myself, and thought 12 meant 12 ports, not 16. It's good to know that if I had to, I could add a second TR8X, though. ;)

nanofrog

Aug 9, 2011, 06:55 AM

Yeah, on this page, (http://www.areca.com.tw/products/1880.htm) they show "16-28 ports" on the 12/16/24 models, and 8-16 ports on the i/x/LP and ixl8/12 models, so all the 12's have 16 ports according to the info... backed up by the emails I've received as well.
Actually, I don't get that from the product page at all. They state connectors, but don't actually specify the port count (if shared or otherwise). It seems to me, they let the customer "interpret" that by inference the way I read it.

Now ideally, it's the stated port count + 4 on those with ix in the model number. But the chips aren't designed that way.

For example, if their chips actually contain 16 ports, then a 24 port internal card (ARC-1880ix24) would actually contain 32 ports, not 28. Now if this is the case, they could have helped users with an additional SFF-8088 port, and the PCI bracket should have sufficient physical space to fit it (tight, but it would fit if they shifted the DIMM slot on a slightly longer board, which is possible as it's not at the max length spec at it's current size).

Given the email, 16 ports per chip is likely the case, and users could get say an ARC1880ix16, flash it with the ARC1880ix24 port firmware, and add in the additional SFF-8087 connectors if they have right skills and equipment to solder the components (it's the same exact PCB). :eek: :D

wonderspark

Aug 9, 2011, 12:36 PM

Actually, I don't get that from the product page at all. They state connectors, but don't actually specify the port count (if shared or otherwise). It seems to me, they let the customer "interpret" that by inference the way I read it.

Now ideally, it's the stated port count + 4 on those with ix in the model number. But the chips aren't designed that way.

For example, if their chips actually contain 16 ports, then a 24 port internal card (ARC-1880ix24) would actually contain 32 ports, not 28. Now if this is the case, they could have helped users with an additional SFF-8088 port, and the PCI bracket should have sufficient physical space to fit it (tight, but it would fit if they shifted the DIMM slot on a slightly longer board, which is possible as it's not at the max length spec at it's current size).

Given the email, 16 ports per chip is likely the case, and users could get say an ARC1880ix16, flash it with the ARC1880ix24 port firmware, and add in the additional SFF-8087 connectors if they have right skills and equipment to solder the components (it's the same exact PCB). :eek: :D
Haha, you know, I considered that as I was trying to unplug that stupid cable (which I never did)... if I broke the internal connector, I might be able to resolder a new one into one of the three empty spaces. I have the skills, and a soldering iron somewhere around here. :D

I agree that the specs are not 100% clear. In a second email to Areca, I asked if there are 16 independent, unshared ports on the 1880ix-12, and the reply was:

"How you define sharing? And how you define independent? All ports came from same expander chip but every port is standalone. I do not know it mean sharing or independing [sic] for you.

Best regards,
Kevin Wang"

So from that, I take it to mean that if there is any sharing, it's sharing evenly across the 16 ports. If you use 8 ports, they are used equally, regardless of where you plug into them. I think they should call it an 1880ix-16/20/28 instead, just to make that clear.

nanofrog

Aug 9, 2011, 04:05 PM

Haha, you know, I considered that as I was trying to unplug that stupid cable (which I never did)... if I broke the internal connector, I might be able to resolder a new one into one of the three empty spaces. I have the skills, and a soldering iron somewhere around here. :D

I agree that the specs are not 100% clear. In a second email to Areca, I asked if there are 16 independent, unshared ports on the 1880ix-12, and the reply was:

"How you define sharing? And how you define independent? All ports came from same expander chip but every port is standalone. I do not know it mean sharing or independing [sic] for you.

Best regards,
Kevin Wang"

So from that, I take it to mean that if there is any sharing, it's sharing evenly across the 16 ports. If you use 8 ports, they are used equally, regardless of where you plug into them. I think they should call it an 1880ix-16/20/28 instead, just to make that clear.
As per soldering, you'd need hot air or an oven, not an iron. And a broken connector would have to be replaced with it's existing firmware.

What a nice, concise and clear answer... :rolleyes:

Ultimately it works, and works well (going off of the performance data), so I wouldn't worry about it. ;)

wonderspark

Aug 10, 2011, 05:20 PM

Update with two internal cables vs. one internal, one external:
AJA test @ 1920x1080 10-bit:

- 6x2TB RAID 3 = 625MB/sec write, 551MB/sec read

So, slight improvement, but probably just coincidence, not actually faster.

nanofrog

Aug 10, 2011, 05:42 PM

wonderspark

Aug 10, 2011, 05:57 PM

Cache Disabled or Enabled?
Enabled. The write speed bounces up and down, apparently, but here are screenshots of the same test back-to-back just moments ago...

nanofrog

Aug 10, 2011, 06:04 PM

Enabled. The write speed bounces up and down, apparently, but here are screenshots of the same test back-to-back just moments ago...
They do vary, but it's a bit odd that the write values went up with cache disabled, even if wasn't by much.

wonderspark

Aug 10, 2011, 06:06 PM

They do vary, but it's a bit odd that the write values went up with cache disabled, even if wasn't by much.
I picked the 16GB test to get the longest run. It also depends on if I'm still running Firefox, I think. If I do the 8GB test, the read numbers are up in the 2000s or so, and the writes are in the 700s... watch this, I'll do it and post up...

wonderspark

Aug 10, 2011, 06:11 PM

Here's the same test with 8GB files instead of 16GB.

nanofrog

Aug 10, 2011, 06:28 PM

Here's the same test with 8GB files instead of 16GB.
Reads are way off on the right hand screen shot... :eek: :p

wonderspark

Aug 10, 2011, 06:35 PM

Reads are way off on the right hand screen shot... :eek: :p

I assumed it has to do with my 16GB of RAM. If I had 24 or 32GB, it would probably still scream on the 16GB with cache enabled, right?

wonderspark

Aug 10, 2011, 06:44 PM

nanofrog

Aug 11, 2011, 12:07 AM

I don't understand what's happening on the graph. These shorter tests show a close-up of what happens... There is this burst at the beginning of every write, then it slows down, then it does this thing where is spikes to the floor repeatedly. If I do only 1GB tests, both read and write are insane... up in the 2000+MB/sec range. Then it does this spike to the floor.

Does that make sense to you? Do you know what's happening?

Also, on the cache disabled side, the read has this wide band of speed which is averaged by the red line, whereas it's a narrow band with the writes and the cache enabled reads.

Would it be safe to guess this is related to the 64MB cache on the drives somehow?
Both are related to cache. When it's on, the the blue lines (writes) remain higher than when off, as the data is moved off of the system faster (resides in cache until it's actually written to the platters).

Same with the red line in the case of the reads (card's cache doesn't help here, but the disk cache does - think read ahead).

wonderspark

Aug 11, 2011, 01:51 AM

Both are related to cache. When it's on, the the blue lines (writes) remain higher than when off, as the data is moved off of the system faster (resides in cache until it's actually written to the platters).

Same with the red line in the case of the reads (card's cache doesn't help here, but the disk cache does - think read ahead).
The downward spikes in the blue lines is much wider than the spikes in the green lines, and I'm guessing that's because the write cache from the card is the 1GB, and the read cache from the disk is only 64MB, hence much more frequently having to "reload" as it were. Have I got this figured out correctly?

nanofrog

Aug 11, 2011, 02:51 AM

The downward spikes in the blue lines is much wider than the spikes in the green lines, and I'm guessing that's because the write cache from the card is the 1GB, and the read cache from the disk is only 64MB, hence much more frequently having to "reload" as it were. Have I got this figured out correctly?
Yes.

wonderspark

Aug 15, 2011, 09:33 PM

nanofrog

Aug 16, 2011, 02:01 AM

What's the difference between "write back" and "write through" setting on Volume Cache Mode? It seems like "write back" gives better performance.
Does write through put data on both cache and disk, and write back only puts data to disk?
In simplest terms, write back (aka write behind), is usually faster.

For further information, take a look at the following links (Yes, I'm being lazy :p - but they are good explanations that should answer your questions). ;)

Cache Wiki (http://en.wikipedia.org/wiki/Cache#Writing_Policies) (specifically, Write Policy section)
jboss.org (https://docs.jboss.org/author/display/ISPN/Write-Through+And+Write-Behind+Caching)
Wiki Tangosol (http://wiki.tangosol.com/display/COH32UG/Read-Through%2C+Write-Through%2C+Refresh-Ahead+and+Write-Behind+Caching) (Oracle is the original source I think)