...which is not at all suitable for what the OP is trying to get done here. Someone else already mentioned a similar setup. Read through the thread before posting, please. It's not that hard when it's only one page long (this comment bringing it to the second page).
Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Help me spend $5000: Mac Pro vs. PC. Detailed, for experts only.
- Thread starter mike1mike2mike3
- Start date
- Sort by reaction score
...which is not at all suitable for what the OP is trying to get done here. Someone else already mentioned a similar setup. Read through the thread before posting, please. It's not that hard when it's only one page long (this comment bringing it to the second page).
You guys are awesome! Here's more info than you probably want...
Wow! There is some serious knowledge on this forum. Thank you all for sharing it. Im really grateful.
Im trying to absorb all of this. Let me respond quickly with a few pieces of information I neglected to include, and some key responses to follow.
This may stir up some more responses. Ideally it will help this group converge on a consensus answer
-The model Im running is a hydrology/operations simulation model, and the program is built around a linear program so it cant be parallelized. So, single thread only! Definitely should have mentioned that up front sorry.
-Im using a harness called CARs to do the MC simulations. It feeds parameters to the model, and collects and stores the output. Im collecting multiple time series of data for each run, not > one number. Say 1200*80 numbers for each model run. So, not an immense amount of data, but will make a decent sized array to interpret.
-I wouldn't throw in more hardware to solve a software problem, if I didn't determine first where the bottleneck is. Since you lack the skills to determine it, have you asked the developers/communities of the simulator/data mining soft for help? Unfortunately, the developers are more hydrologists than computer scientists. They will be interested to hear what I figure out, but dont seem to have expertise in such computational issues
-I have a new, decked out MBP, which I use for everything except modeling. Have been a Mac user for years. Chained to the PC by my choice of software for my research.
-This is absolutely correct, from Waffle911: What he really wants is a Mac, but what he really needs is the most computing power he can get for under $5k, regardless of whether it's a Mac or a PC.
-I do get educational discounts through my university for Apple and Dell, and am buying this through a grant I got for my graduate research. I always upgrade RAM separately to save money, but have not bothered to do this with HDs.
-I have 4 Dell T3500 machines already. Ive got about 10K left to spend in two different grants.
-Its been years since I built my own machine, so it would likely take me some doing to figure it out, and dont really want to deal with the time overhead. It would also be quite a hassle to pay for this on my grant, so Im going to nix this suggestion.
-Perhaps the key insight: SHORT ANSWER: Dell. Do you want to rely on Boot Camp for your serious work? All Apple did was package some drivers. It's not exactly the same as a Dell, and it was never supposed to be. People on these forums regularly tell others that it's the same as having Win XP/Vista/etc, but it's not. I will add something relevant here I am using two software packages (a hydrology model plus a harness that feeds parameters in and collects output for each MC iteration). But these are niche products to say the least, and can be tempermental. Risking getting caught up in some compatability issue on Boot Camp seems like a recipe for disaster to me.
More to follow shortly - thanks for letting me deluge you all with information...
-Mike
Wow! There is some serious knowledge on this forum. Thank you all for sharing it. Im really grateful.
Im trying to absorb all of this. Let me respond quickly with a few pieces of information I neglected to include, and some key responses to follow.
This may stir up some more responses. Ideally it will help this group converge on a consensus answer
-The model Im running is a hydrology/operations simulation model, and the program is built around a linear program so it cant be parallelized. So, single thread only! Definitely should have mentioned that up front sorry.
-Im using a harness called CARs to do the MC simulations. It feeds parameters to the model, and collects and stores the output. Im collecting multiple time series of data for each run, not > one number. Say 1200*80 numbers for each model run. So, not an immense amount of data, but will make a decent sized array to interpret.
-I wouldn't throw in more hardware to solve a software problem, if I didn't determine first where the bottleneck is. Since you lack the skills to determine it, have you asked the developers/communities of the simulator/data mining soft for help? Unfortunately, the developers are more hydrologists than computer scientists. They will be interested to hear what I figure out, but dont seem to have expertise in such computational issues
-I have a new, decked out MBP, which I use for everything except modeling. Have been a Mac user for years. Chained to the PC by my choice of software for my research.
-This is absolutely correct, from Waffle911: What he really wants is a Mac, but what he really needs is the most computing power he can get for under $5k, regardless of whether it's a Mac or a PC.
-I do get educational discounts through my university for Apple and Dell, and am buying this through a grant I got for my graduate research. I always upgrade RAM separately to save money, but have not bothered to do this with HDs.
-I have 4 Dell T3500 machines already. Ive got about 10K left to spend in two different grants.
-Its been years since I built my own machine, so it would likely take me some doing to figure it out, and dont really want to deal with the time overhead. It would also be quite a hassle to pay for this on my grant, so Im going to nix this suggestion.
-Perhaps the key insight: SHORT ANSWER: Dell. Do you want to rely on Boot Camp for your serious work? All Apple did was package some drivers. It's not exactly the same as a Dell, and it was never supposed to be. People on these forums regularly tell others that it's the same as having Win XP/Vista/etc, but it's not. I will add something relevant here I am using two software packages (a hydrology model plus a harness that feeds parameters in and collects output for each MC iteration). But these are niche products to say the least, and can be tempermental. Risking getting caught up in some compatability issue on Boot Camp seems like a recipe for disaster to me.
More to follow shortly - thanks for letting me deluge you all with information...
-Mike
Off topic, but I think R can be multi-threaded, although I haven't set it up this way yet. See rparallel.org
I am using whatever controllers came with the machine, but yes it has SATA drives and I didn't select anything special for the RAID controllers.
I will need to learn about SSDs. Their price (for big ones) puts me off from making the investment until I can figure out whether that's the real bottleneck, but I think I should probably try one in one of hte machines I have and see. Too much to learn that's not getting my research done!!!
The machines I have look pretty much like this, with minor variations:
SYSTEM COMPONENTS * Dell Precision T3500 Qty * * 2
* Dell Precision T3500, CMT, 85 Percent Efficient Power Supply, Genuine Windows Vista® Business Downgrade,XP Professional SP3 Install,E Unit*Price * $1,972.50 *
Quad Core Intel Xeon W3550 3.0GHz, 8M L3, 4.8GT/s, Turbo W3550 [317-2539] 2
Memory 1GB, 1066MHz, DDR3 SDRAM, NECC (1 DIMM) 1G3N661
[upgraded to 8GB in each machine, and using 5000MB RamDisk with Superspeed RamDisk]
Graphics 256MB NVIDIA® Quadro® NVS 295, 2MON, 2 DP w/ 2 DP to DVI Adapters NVS2952 [320-7899][330-4086] 6
Boot Hard Drive 500GB SATA 3.0Gb/s with NCQ and 16MB DataBurst Cache 500GS [341-8663] 8
Hard Drive Configuration C4 All SATA drives, RAID 0 for 2 Hard Drives SR02 [341-8563] 9
2nd Hard Drive 500GB SATA 3Gb/s with NCQ and 16MB DataBurst Cache 500GSA [341-8581] 23
Hard Drive Controller Integrated Intel chipset SATA 3.0Gb/s controller NSASCTL [341-9289] 24
Perfmon output
I think that my last few posts should respond to most of the things I failed to mention up front. Thanks again.
Per requests for more information - I have run perfmon.exe and collected 15 hours or so of data from some test simulations. I don't really know how to interpret them, unfortunately. This is where I got the 100MB/sec figure, from someone who took a quick look at this, but couldn't really give me a final answer. If it would help, I could post that file somewhere, as long as it doesn't contain information that could compromise my machine. But I won't try to do that unless someone asks to see them - seems like asking a lot when you've already collectively been very generous already.
Thanks again!
I think that my last few posts should respond to most of the things I failed to mention up front. Thanks again.
Per requests for more information - I have run perfmon.exe and collected 15 hours or so of data from some test simulations. I don't really know how to interpret them, unfortunately. This is where I got the 100MB/sec figure, from someone who took a quick look at this, but couldn't really give me a final answer. If it would help, I could post that file somewhere, as long as it doesn't contain information that could compromise my machine. But I won't try to do that unless someone asks to see them - seems like asking a lot when you've already collectively been very generous already.
Thanks again!
Don't build a hackintosh unless you have spare time in the future. Who wants to worry about the next update working on your Hackintosh?
Do you have to edit video? If you do, you can still edit it fine on a PC. I absolutely love the quicktime workflow on the mac, but it sounds like your primary use for this is windows native programs.
Buy for what you need to do get work done. If your programs are not multithreaded, then just least get a single core i7 machine with a nice turbo mode (you can overclock those things very easily). You can also disable hyper -threading on Windows, IIRC, so save a ton of money and get a nice fast Core i7, and a decent iMac or MacBook Pro to compliment it.
Do you have to edit video? If you do, you can still edit it fine on a PC. I absolutely love the quicktime workflow on the mac, but it sounds like your primary use for this is windows native programs.
Buy for what you need to do get work done. If your programs are not multithreaded, then just least get a single core i7 machine with a nice turbo mode (you can overclock those things very easily). You can also disable hyper -threading on Windows, IIRC, so save a ton of money and get a nice fast Core i7, and a decent iMac or MacBook Pro to compliment it.
As pooky pointed out, you should check if the simulator is swapping memory from/to disk like crazy. Since you have a 5GB ram disk, lets say other programs/windows eat 1GB, the simulator only has 2GB to do its processing. Is that enough? How much memory the simulator uses during the simulation (you can check that using the windows tasks manager in the process tab)? Why do you need a RAM disk? Does the simulator moves tons of data in and out from disk during the simulation, or does it moves the data completely from disk to RAM, then processes it and dumps all the results to disk? Are you using a 64bit OS? How much RAM can the motherboard take? How many cores are being used during simulation (check that with the task manager)? Can you reduce the size of the RAM disk? Is the simulator a 64bit application?
I don't want this to come out wrong - I'm interested in your opinion - but how is this different from what Dell does when it builds a computer?
Apple puts together industry-standard components; so does Dell. Apple tests a package of drivers for those components; so does Dell. Apple allows for the install of Windows; Dell preinstalls Windows. When it comes to the workstations, is there special software installed by Dell (this isn't a "bloatware" flamebait question)? Does Dell write any special software or drivers for their workstations? In my non-"mission critical" experience, Boot Camp is EXACTLY like running Windows on a PC.
The point you (and others) have made about Dell enterprise-level support for Windows seems valid (although it appears that the OP isn't going to get software support from anyone but himself).
My opinion is you're wasting everyone's time here.
You've got a pretty powerful system already, and you want to spend $5000 on an upgrade for a specialised application, but you don't even know where the bottleneck is?
You need to know where to put the cash - CPU, RAM, or IO. Otherwise you stand a strong chance of just pissing your money away on an expensive system that runs no faster than your existing system.
To be honest, and sorry if this is a bit harsh, I feel you don't know the algorithm you're running very well if you can't figure out where the bottleneck is.
I'll try to be more helpful from this point:
Go away, find it out, and come back when you know. Invest some of your budget on bribing or paying someone to trace the bottleneck. There are some posters in this thread who are pretty knowledgeable on your algorithm - invite them in private to take a look.
I'm no computer scientist, but take a look at the RAM usage, look at the swap file, paging in, paging out, disk transfer load, look at the CPU workload.
- If the most highly loaded CPU core is at roughly 80% or less, then getting a more powerful CPU is a waste of money.
- if most of the disk transfer load is less than the disk max, then getting a faster disk system is a waste of money.
- if the RAM isn't fully used, then getting more RAM is a waste of money.
You can also do real-world testing by taking some of the RAM out, underclocking the CPU, putting the data on a slower drive, and seeing which of these has the biggest effect on the speed of a run.
From what you said, on a very crude level, my best guess, and it's only a wild guess, is it seems RAM bound. So looking for a system that can take 16, or 32 or even 64GB of RAM and stay within budget would be a good idea.
On a deeper level, maybe look at the innermost loops of your algorithm. It may not fit within your CPU's L2 or L3 cache. You might be able to see a massive speed-up if you could find a CPU that can fit some inner loops and data storage, or even rejig your application if you can so that it only works on as much data as can fit into the CPU cache.
I'm not up to speed on current cache sizes, but I believe that the i7-920 has 8MB L3 cache. You might be better off with a workstation class processor that costs more, has fewer cores, lower clockspeed, but gigantic caches. But only if cache misses are a solvable issue.
Nobody can help you do your runs faster until you find out where the damn bottleneck is.
You've got a pretty powerful system already, and you want to spend $5000 on an upgrade for a specialised application, but you don't even know where the bottleneck is?
You need to know where to put the cash - CPU, RAM, or IO. Otherwise you stand a strong chance of just pissing your money away on an expensive system that runs no faster than your existing system.
To be honest, and sorry if this is a bit harsh, I feel you don't know the algorithm you're running very well if you can't figure out where the bottleneck is.
I'll try to be more helpful from this point:
Go away, find it out, and come back when you know. Invest some of your budget on bribing or paying someone to trace the bottleneck. There are some posters in this thread who are pretty knowledgeable on your algorithm - invite them in private to take a look.
I'm no computer scientist, but take a look at the RAM usage, look at the swap file, paging in, paging out, disk transfer load, look at the CPU workload.
- If the most highly loaded CPU core is at roughly 80% or less, then getting a more powerful CPU is a waste of money.
- if most of the disk transfer load is less than the disk max, then getting a faster disk system is a waste of money.
- if the RAM isn't fully used, then getting more RAM is a waste of money.
You can also do real-world testing by taking some of the RAM out, underclocking the CPU, putting the data on a slower drive, and seeing which of these has the biggest effect on the speed of a run.
From what you said, on a very crude level, my best guess, and it's only a wild guess, is it seems RAM bound. So looking for a system that can take 16, or 32 or even 64GB of RAM and stay within budget would be a good idea.
On a deeper level, maybe look at the innermost loops of your algorithm. It may not fit within your CPU's L2 or L3 cache. You might be able to see a massive speed-up if you could find a CPU that can fit some inner loops and data storage, or even rejig your application if you can so that it only works on as much data as can fit into the CPU cache.
I'm not up to speed on current cache sizes, but I believe that the i7-920 has 8MB L3 cache. You might be better off with a workstation class processor that costs more, has fewer cores, lower clockspeed, but gigantic caches. But only if cache misses are a solvable issue.
Nobody can help you do your runs faster until you find out where the damn bottleneck is.
This is something that had not occurred to me, but is a very important piece of information the OP left out. It may seem like an obvious thing to check, but all the more reason for us to know up front. It's be a shame to come to an incorrect conclusion because we missed something so simple as this!
If you're not running 64-bit Windows, you're being limited to less than 4GB of RAM, though I hope you already knew this. I'm pretty sure a 32-bit program would have similar limitations for RAM usage. I've never seen a 32-bit program use more than 2GB myself, but I've never had to run the types of programs you're running.
While this comment seems quite harsh, he does makes several valid points.
Are you using a system monitor to see how the system is behaving while running these simulations? If not, a basic one I know of for all versions of Windows would be WinBar, or you could use the one built-in to XP (I don't know too much about it since I use Windows 7 when not using OS X).
I found a few interesting things in the Microsoft KB though.
XP has system monitor counters that you can set yourself built-in.
You can also tell it to create a complete log of performance statistics.
Hopefully you can use some of this to better narrow down where your bottleneck is.
RedTomato and others - apologies for any violation of norms here. As I said up front, my original motivation for posting is that I lack the skills to answer the questions you pose. I tried getting help from a friend, who got far enough on the performance monitor analysis to tell me what I posted. Beyond that I'm not even sure where to start.
In the end, I remain ignorant of the final answer, but the responses were quite helpful for me.
I think the bottom line is that 1) there is not a simple answer to my question(s) given the information I have, and 2) it would take me an uncertain number of days to crawl down the rabbit hole of learning enough to optimize the decision, and that time really needs to be spent on my primary goal of doing the research.
So I bought some Dell T3500's and they will do what they do. My apple fetish will have to be satisfied for the moment by my existing set of apple products, and I will have to live with whatever extra time I need to spend on model runs. So it goes.
waffle: I am using a RAMDisk product (see superspeed.com) that can access unaddressed memory on 32 bit systems, which I think gets around that limitation. But my newer machines will be 64 bit.
When I step up to a larger suite of model runs in future, I may address the optimization problem through experimenting with SSDs or whatever. For now, onwards...
Thanks again to you all, I do appreciate your input!
In the end, I remain ignorant of the final answer, but the responses were quite helpful for me.
I think the bottom line is that 1) there is not a simple answer to my question(s) given the information I have, and 2) it would take me an uncertain number of days to crawl down the rabbit hole of learning enough to optimize the decision, and that time really needs to be spent on my primary goal of doing the research.
So I bought some Dell T3500's and they will do what they do. My apple fetish will have to be satisfied for the moment by my existing set of apple products, and I will have to live with whatever extra time I need to spend on model runs. So it goes.
waffle: I am using a RAMDisk product (see superspeed.com) that can access unaddressed memory on 32 bit systems, which I think gets around that limitation. But my newer machines will be 64 bit.
When I step up to a larger suite of model runs in future, I may address the optimization problem through experimenting with SSDs or whatever. For now, onwards...
Thanks again to you all, I do appreciate your input!
Fair enough. Sometimes it's useful to learn the limits of what you don't know. You're paid to do your research, not figure out obscure PC details.
I still think it would have been useful to use some of your budget to find an expert to advise. What about contacting a compsci prof at your local university computer science dept? This is the exact sort of area they love to advise in. (Or contact your own alma mater).
Even if you've already put the purchase order in, still worth contacting a university, as they tend to move slowly sometimes, and the analysis of your bottleneck will help for your *next* purchase
(Might even make it worth returning your purchase for something better suited).
I still think it would have been useful to use some of your budget to find an expert to advise. What about contacting a compsci prof at your local university computer science dept? This is the exact sort of area they love to advise in. (Or contact your own alma mater).
Even if you've already put the purchase order in, still worth contacting a university, as they tend to move slowly sometimes, and the analysis of your bottleneck will help for your *next* purchase
(Might even make it worth returning your purchase for something better suited).
This is a very fanboy setup. NOT calling you one, but that is the worst use of $5000 that I've seen in a long LONG time. The 8 core 2.26 is the pits when considering the OP's desire to have the best CPU for the money.
Getting TWO 128GB SSDs and then using a SINGLE 2TB disk and partitioning it for backup????
WTH? The EDU discount isn't a special offer from Apple. Nine times out of ten they don't check to see if you are a legitimate student or teacher, but it's not something that should be used in a comparison, so off the break the Mac Pro costs more.
Then as others have said, if you are going to run Windows for bringing home the bacon work then you NEED to get a PC workstation that has support for the machine and the OS. Dell and HP offer that, Apple doesn't.
The Dell workstation (without the useless quad 2.26 processors) would leave the OP with enough cash for a MBP or iMac that will run FCS3 just fine. For what it's worth OP Final Cut or iMovie or whatever you plan to cut with won't be your bottle neck. Most editing apps run like butter on any processor, it's the rendering that'll kill you, but if you set your timeline up correctly that can be alleviated.
Also, like RedTomato said you need to check the tech specs of the software that you are running, and call the tech centers for said software to see what system would run their apps best.
Lastly, with $5000 and more than enough suggestions to get a Dell workstation it's safe to say that you won't have much of a bottleneck obtaining the highest spec'd machine for the price.
I see a problem here already.
You have "somewhat specific needs," but you can't even narrow your needs down to an OS?
Don't Pay Extra for an apple
I have bought server set ups online for closer to $2000 that can blow away the Mac Pro with a simple addition of a good video card. or you can spend more than 3 grand extra to put an apple ($3000 FOR THIS
!!!!) logo on it. your choice (not to mention lights out I/O cards)
--------------------------------------------------------------------------------------
I have bought server set ups online for closer to $2000 that can blow away the Mac Pro with a simple addition of a good video card. or you can spend more than 3 grand extra to put an apple ($3000 FOR THIS
!!!!) logo on it. your choice (not to mention lights out I/O cards)Because your trying to make it the same (old) crap as a mac pro
Try a Six Core Intel® Xeon® Processor X5660, 2.8GHz,12M L3, 6.4GT/s, turbo system is only 3000
or a Dual Quad Core Intel® Xeon® Processor E5620, 2.40GHz,12M L3, 5.86GT/s, turbo [Included in Price] system for 2000
Just wait. Intel announced the 10 core, 20 thread Xeon line.
http://www.engadget.com/2011/04/06/intel-rolls-out-10-core-20-threaded-xeon-e7s-shows-everyone-wh/
Throw money at that. yowza.
http://www.engadget.com/2011/04/06/intel-rolls-out-10-core-20-threaded-xeon-e7s-shows-everyone-wh/
Throw money at that. yowza.
Mac all the way.
With depreciation the Dells end up being much more expensive (whole life cycle economic costs)
With Macs its easy to sell and upgrade as newer models come out.
...the amateur opinion
With depreciation the Dells end up being much more expensive (whole life cycle economic costs)
With Macs its easy to sell and upgrade as newer models come out.
...the amateur opinion
Almost always the biggest speedup can be gained be rethinking the algorithm. You ned to look at your software and see where it is spending all it's time.
You say a RAM disk helps. This is a clue to me that there is something wrong. The RAM must be loaded from some non-volatile media. I bet a beter use of that RAM might be for demand paged virtual memory.
You should only have an I/O bottle neck if you realy do have a large data set to plow through, If storage of intermediate results is the cause of the I/O bottle net then redesign the algorithm
If you really do have an I/O bottle neck then buy more computers not one larger computer. Typical PC and Mac computer have just one PCI bus and to get two PCI bus buy two computers and you will double the total I/O speed.
There are higher end server class computers from people like Oracle (formally Sun) that have many PCI buses on one computer and have very high aggregate I/O rates but that are 10X to 100X over your budget. But if you really do need to go much faster than 100MB/Second bur multiple computers
OK, so my MC experience is a little old: I invented a modeling language called VSL used to produce models for variation simulation using a software product called VSA. I think it (at least VSA) still lives in some form as part of Siemens NX. (Prior to that, models had to be written in Fortran!)
It wound up being used by pretty much all the auto manufacturers, aircraft manufacturers, disk drive makers, etc. That was 30 years ago, though. But still, I probably have some insight into what your needs might be.
FWIW it was a HUGE breakthrough when, as a result of using VSA, GM was able to produce a Corvette where the hood could be assembled to the body using holes, not slots. Plop a bolt through a hole, tighten it, done! Huge big deal! The Corvette was the first car where that was possible. Of course, now it's standard.
-------
If you have a 100mbyte/sec limitation, it sounds like hard drive. That's a pretty typical SATA drive transfer rate now. As others have suggested, you may simply not have enough RAM, forcing the software to use disk storage or swap space to process your large datasets.
Your first concern is what platform your software runs on, and secondly, if on multiple platforms, which one it runs best on....
You may find Linux a better platform than either OSX or Windows. But, very much dependent on vendor support for the platform.
I would get something based on a high memory-capacity server board. This could go from build-it-yourself to the mid-tier, low-cost commodity servers used for web and database servers, to nice name-brand servers (I used to like IBM, but dunno if these are Lenovo now and how the quality is.)
You will need a 64-bit OS to take advantage of the memory. 16G? 32G? 64G?
Somebody more familiar with your particular software should be able to give you some guidance. You want to try to get the entire dataset and whatever working storage is needed into RAM!
Use a RAMDisk of some sort if the software INSISTS on using temporary disk storage. On Linux, use a tmpfs filesystem for any temp files. (It's in RAM, but backed by swap should you run out of RAM.)
I would get the basic box, with a standard disk drive, and increase RAM until you see no more improvement. At that point, you can experiment with faster disks.
If you really want fast, reliable drives, go SAS, not SATA. It uses the same connections as SATA - you can actually plug SATA drives into a SAS frame.
The best/fastest enterprise drives are only produced in SAS versions.
Consider RAID 0 and/or flash drive(s) if you do need to increase disk throughput.
Let''s say you have this huge dataset that your software has to read, that doesn't change between runs (or doesn't change much), and the read time is a significant portion of the processing. (Don't think that would really be the case, though.) Flash would be perfect for that. Even cheap SATA flash drives now do typically 200mbyte/sec. Throw a couple in a RAID 0 array for a cheap 400mbyte/sec. (There are higher-priced flash cards that plug into the motherboard bus directly that will do 800+mbytes/sec but they are quite costly, and driver support seems limited to Windows only.)
Consider add-on array processing boards, such as from MicroWay, if your software supports that. (Microway has been doing this for many, many years. I see they have moved to using GPUs. They used to use custom array calculation chips.) They also offer complete workstations, which might make a lot of sense. Their workstations will take as much of 256GB of RAM. If this will help you, of course you will have to consider on what OS platform(s) it's supported on. Even a consumer Nvidia card with software that can take advantage of the GPU for array processing could speed things up considerably.
Depends on what you are doing in your simulation. For the kinds of applications I was involved in, would definitely make a difference, because variation simulation randomly varies parameters in a model. For example, for mechanical assemblies, it's varying hole sizes, lengths, etc. to simulate real-world manufacturing variations. So for each simulation, there can be a ton of math to re-calculate the model, plugging parts together into an assembly. Lots of 3D translations. Generating random numbers under some distribution is a trivial amount of computing compared to running the model.
It wound up being used by pretty much all the auto manufacturers, aircraft manufacturers, disk drive makers, etc. That was 30 years ago, though. But still, I probably have some insight into what your needs might be.
FWIW it was a HUGE breakthrough when, as a result of using VSA, GM was able to produce a Corvette where the hood could be assembled to the body using holes, not slots. Plop a bolt through a hole, tighten it, done! Huge big deal! The Corvette was the first car where that was possible. Of course, now it's standard.
-------
If you have a 100mbyte/sec limitation, it sounds like hard drive. That's a pretty typical SATA drive transfer rate now. As others have suggested, you may simply not have enough RAM, forcing the software to use disk storage or swap space to process your large datasets.
Your first concern is what platform your software runs on, and secondly, if on multiple platforms, which one it runs best on....
You may find Linux a better platform than either OSX or Windows. But, very much dependent on vendor support for the platform.
I would get something based on a high memory-capacity server board. This could go from build-it-yourself to the mid-tier, low-cost commodity servers used for web and database servers, to nice name-brand servers (I used to like IBM, but dunno if these are Lenovo now and how the quality is.)
You will need a 64-bit OS to take advantage of the memory. 16G? 32G? 64G?
Somebody more familiar with your particular software should be able to give you some guidance. You want to try to get the entire dataset and whatever working storage is needed into RAM!
Use a RAMDisk of some sort if the software INSISTS on using temporary disk storage. On Linux, use a tmpfs filesystem for any temp files. (It's in RAM, but backed by swap should you run out of RAM.)
I would get the basic box, with a standard disk drive, and increase RAM until you see no more improvement. At that point, you can experiment with faster disks.
If you really want fast, reliable drives, go SAS, not SATA. It uses the same connections as SATA - you can actually plug SATA drives into a SAS frame.
The best/fastest enterprise drives are only produced in SAS versions.
Consider RAID 0 and/or flash drive(s) if you do need to increase disk throughput.
Let''s say you have this huge dataset that your software has to read, that doesn't change between runs (or doesn't change much), and the read time is a significant portion of the processing. (Don't think that would really be the case, though.) Flash would be perfect for that. Even cheap SATA flash drives now do typically 200mbyte/sec. Throw a couple in a RAID 0 array for a cheap 400mbyte/sec. (There are higher-priced flash cards that plug into the motherboard bus directly that will do 800+mbytes/sec but they are quite costly, and driver support seems limited to Windows only.)
Consider add-on array processing boards, such as from MicroWay, if your software supports that. (Microway has been doing this for many, many years. I see they have moved to using GPUs. They used to use custom array calculation chips.) They also offer complete workstations, which might make a lot of sense. Their workstations will take as much of 256GB of RAM. If this will help you, of course you will have to consider on what OS platform(s) it's supported on. Even a consumer Nvidia card with software that can take advantage of the GPU for array processing could speed things up considerably.
Depends on what you are doing in your simulation. For the kinds of applications I was involved in, would definitely make a difference, because variation simulation randomly varies parameters in a model. For example, for mechanical assemblies, it's varying hole sizes, lengths, etc. to simulate real-world manufacturing variations. So for each simulation, there can be a ton of math to re-calculate the model, plugging parts together into an assembly. Lots of 3D translations. Generating random numbers under some distribution is a trivial amount of computing compared to running the model.
Last edited:
LOL, didn't notice that. Well, maybe somebody else can use some of the recent advice!