Configuring, Upgrading A 6 Core for retouching

[Honumaui]

only thing I can think of is how many layers are in that file ?

unless layers have a lot to do with the compressing and getting it ready to write ?

now this makes sense ? again have not tested but count the layers in that file and let me try to duplicate a file with that many layers ?

wonder if this will be my new learned item for the day 🙂 try to learn something every day 🙂

 

[Ryan P]

Just to pipe in that my similar issues with slow saving have to do with files with dozens of layers. I normally start with a 25ish MB 16bit file but often end up with files in the 1-2GB range. Lately I've been tackling it by trying to make my work flow as efficient as possible. So the files are very deep but not very wide or tall.

 

[Macinposh]

only thing I can think of is how many layers are in that file ?

unless layers have a lot to do with the compressing and getting it ready to write ?

now this makes sense ? again have not tested but count the layers in that file and let me try to duplicate a file with that many layers ?

wonder if this will be my new learned item for the day 🙂 try to learn something every day 🙂

Say.

A normal 120mb,16bit, TIFF, save with LZF, Eos-5 file with no layers might take 10-15 secs to save with CS3+MP 1.1.

Ad 10 layers (light ones,maybe 1Gb flie size?) and it is 2 mins.

Ad 20 heavy layers (with lot of data,oo,maybe 3 gigs?) 6 mins?

Ad more layers...you get the point.

While,all the time, only one..ONE core is churning out the frigging results! It is really frustrating to watch the process. Never mind what HDD setup you have...


I really feel for the gfx artist brethen here,having to work with the said 30-40 layers..

 

[Honumaui]

Say.

A normal 120mb,16bit, TIFF, save with LZF, Eos-5 file with no layers might take 10-15 secs to save with CS3+MP 1.1.

Ad 10 layers (light ones,maybe 1Gb flie size?) and it is 2 mins.

Ad 20 heavy layers (with lot of data,oo,maybe 3 gigs?) 6 mins?

Ad more layers...you get the point.

While,all the time, only one..ONE core is churning out the frigging results! It is really frustrating to watch the process. Never mind what HDD setup you have...


I really feel for the gfx artist brethen here,having to work with the said 30-40 layers..

YES I understand that 🙂 not what I was asking him !!!!!
of course each layer ads to the file size though ? so its going to take longer


so what I was asking is how many layers in his file that is 1.9 gigs
my 1.95 only has about 8


if a 2 gig file has 5 layers and another 2 gig file has 30 layers ? does that make a difference ? not meaning the same file but the actual file size ?

so does that make sense 🙂

got to run but going to try this later 🙂

 

[Macinposh]

if a 2 gig file has 5 layers and another 2 gig file has 30 layers ? does that make a difference ? not meaning the same file but the actual file size ?

so does that make sense 🙂

got to run but going to try this later 🙂

It is all about info on those layers.
If they contain no info=no data.
Little info=Little data.
A Lot of info = Muy data.

 

[Ryan P]

For all we know could also vary depending on things such as smart filters and blend modes as well...ridiculous that they have not done a better job with multi-threading though. This might be a good thread to cross-post on an adobe forum.

 

[Honumaui]

It is all about info on those layers.
If they contain no info=no data.
Little info=Little data.
A Lot of info = Muy data.

trust me I am pretty smart at this stuff 🙂 again yes I understand layers have more info ?

but if a file is 2 gigs and has 8 layers and one is 2 gigs and has 30 layers ? should that matter ? after all its the same size is my point

 

[Honumaui]

While,all the time, only one..ONE core is churning out the frigging results! It is really frustrating to watch the process. Never mind what HDD setup you have...

I really feel for the gfx artist brethen here,having to work with the said 30-40 layers..

the one thing what you write to as in raid or single HDD or SSD and how much memory and your scratch all play a roll in the speed of the file being saved 🙂


so had a quick test with some files and found amount of layers seem not to matter ? at least for the ones I did ?

14 layer file 1.95 gigs 16.7 seconds
61 layer file 1.98 gigs 17.4 seconds

did a few files and each 3 times to avg

curious why slater with almost the same setup as me is taking over 2 minutes ? and if it is related to the scratch in some way ? or ??

 

[Honumaui]

For all we know could also vary depending on things such as smart filters and blend modes as well...ridiculous that they have not done a better job with multi-threading though. This might be a good thread to cross-post on an adobe forum.

yeah its stupid they have not done more cores ???

could be some of those things but the quick tests I did seem to think its not ? but then again I will never say never ?


I would be curious if slater wanted to ftp the file somewhere and let me see what it does on my setup ?

 

[ScratchyMoose]

how many layers are in that file ?

again have not tested but count the layers in that file and let me try to duplicate a file with that many layers ?

about 40, with around 8 being smart objects, lots of blurs and the like

if i merge all the layers, and then duplicate the resulting single layer about 8 times till the file is a similar size to the original 1.9Gb, then the save time stays roughly the same, even though what you see of the file is just the top layer (as it's set to normal and 100%).

PS is all the while using around 103% of the cpu, ie just the one core.

Someone linked to the specs of the .doc file format somewhere on these boards recently, and the pdf went on for over 100 pages, and maybe all this activity is just making sure that the file conforms to the spec for a .psd which is bound to be pretty complex?

 

[ScratchyMoose]

deleted

 

[ScratchyMoose]

trust me I am pretty smart at this stuff 🙂
but if a file is 2 gigs and has 8 layers and one is 2 gigs and has 30 layers ? should that matter ? after all its the same size is my point

it seems that layer number doesn't matter - my original file with 40ish complex layers and masks saves in the same time as the merged and then duplicated to 8 layers file.

 

[ScratchyMoose]

curious why slater with almost the same setup as me is taking over 2 minutes ? and if it is related to the scratch in some way ? or ??

Just realised that it's because my file is in 16 bit. When i convert the file to 8 bit (and in CS5 all the layers are kept as they were before) and save, it takes less than 20 seconds, even though the file has only gone from 1.9Gb to 1.7Gb.

That's saving to my raid, if i save the original file to a slow back up drive, it takes around the same time. Saving the 8 bit file takes to the raid takes around 20 secs but saving it to the slow HDD takes around 30. During the raid save, PS takes up 98-160% of the cpu (one core = 100%) for most of the time and it's pretty similar when saving to the HDD EXCEPT when i can see activity in the slow HDD when cpu drops to around 10%. After the write is over but the save is still on-going, cpu bounces back up to 160-100.

I conclude that some parts of the save are multi thread, some parts single thread, and that while the actual write to disc is going on, the cpu is still called into play in single thread mode, and if the target disc is fast then the cpu has the chance to max out, and so becomes the limiter; if the target disc is slow, then the cpu doesn't max out and it's the disc that is the limiter ... ?


(BTW, a 1.9Gb file that's only one white layer, saves in less than 1 second)

 

[Honumaui]

interesting on the bit 🙂
all my arch stuff was 16 🙂 but that was a few machines ago

funny on the white doc I dont do to many of those 🙂 hhehehhe

but yeah good puzzle had me scratching my head for sure on why so long and the bit depth is that right in front of ya thing 🙂
the work I do is mostly in 8 these days unless I get a big gradient then its 16 till done squash it down at end ?

my own work I print in 16 but the files are done so its a open and close




doing a file of mine 1.94 gigs 16 bit with some layers and got 90 seconds avg to the raid with SSD as scratch
doing the raid as my scratch the time was 98 avg
they were all within 1 sec so the avg was pretty much what they were

one thing I have the write enabled on my areca and also have the battery module
all the high end machines I used to work with had batteries !! so I did this to my machine just to be safer ?
curious now might have to play and test with this turned on or off later ? if that has much to do with better times ?


sorry to keep this going things like this are puzzles to me and love to figure things out and learn more


I remember working on huge files on the G5 and figure good time to go eat dinner and come back and the file MIGHT be done 🙂

and with that back to work !

 

[JulianBoolean]

Just a quick update. I'm doing some after hours (saving) testing using the four MPs in the retouching studio at work. Will post results asap.

wondering if there's a way to test a common image perhaps doubling the end product of the speed test sticky here, then save that?

Big thanks to all contributors here, no time to reply just yet but will soon!

- Julian

 

[Honumaui]

Just a quick update. I'm doing some after hours (saving) testing using the four MPs in the retouching studio at work. Will post results asap.

wondering if there's a way to test a common image perhaps doubling the end product of the speed test sticky here, then save that?

Big thanks to all contributors here, no time to reply just yet but will soon!

- Julian

no common image ?

the size just came cause that seemed long for that size file 🙂 and was curious

🙂

 

[jwestpro]

Please forgive me if this is old news covered long ago but why is nobody mentioning SAS internal drives (15k rpm) as an option? From what I recall, they were at the very least a lot faster access than sata but maybe not.

Or, what happened to readily available 10k and 15k rpm sata drives as the other "mechanical" drive option?

I think I understand about the total throughput via a card or controller then going into whatever drives are available in whatever arrangement. For example, the apple site shows their raid card as having "553MB/s of sequential read performance in RAID 0".

Is it the total throughput that is the issue here with the above ideas?

I am a commercial photographer since 1998 and also run into the 1-2gb file saving forever issue but that may be partially due to having held out on an older machine for so long as well as not using cs5 yet. I've been lazy but also busy so didn't spend time digging around on forums like this to learn more.

Sort of separately, but relative due to that link http://macperformanceguide.com/Reviews-MacProWestmere-Photoshop-diglloydHuge.html

It appears the main drawback to the 6 core 3.33 is the ram max? 2.93 vs 3.33 for cs5 specifically, doesn't seem like a big deal but the ram could be for certain things, especially if you want to play in video too.

I have been using an enterprise level ibm server for my primary backup plus redundant on esata drives in both east and west coast studios - just in case.

It does sound like I should rethink the internals though of the 8 core I bought early 09 as well as the 6 or 12 core I'm buying in the next day or two.

I don't see the benefit of using an ssd for the os/boot drive though as opposed to a 10 or 15k sata because the files are not ever on that drive.

Any help is appreciated but I will follow this thread to learn as much as possible.

 

[nanofrog]

Please forgive me if this is old news covered long ago but why is nobody mentioning SAS internal drives (15k rpm) as an option? From what I recall, they were at the very least a lot faster access than sata but maybe not.

Most users here aren't willing to pay the cost/GB necessary from what I can tell. SATA is a cheaper alternative, and for sequential access situtations, can give similar performance for less money (just use more SATA drives, and it usually provides more capacity as well).

There are advantages however, such as faster random access (not as fast as SSD, but better for high write situations), and better sequential throughputs per disk (can matter where the member count is limited to available equipment, and SATA can't meet the requirements), as well as improved recovery due to SCSI commands.

Or, what happened to readily available 10k and 15k rpm sata drives as the other "mechanical" drive option?

10k rpm are available (Velociraptor), but not 15k SATA. There was mention of WD taking the Raptor/Velociraptor series to 20k rpm, but so far, hasn't surfaced.

It appears the main drawback to the 6 core 3.33 is the ram max? 2.93 vs 3.33 for cs5 specifically, doesn't seem like a big deal but the ram could be for certain things, especially if you want to play in video too.

OWC has confirmed that their 8GB sticks will work in SP systems (Quads and Hex cores for the 2010 models), so 32GB is actually possible (though more expensive than DP systems, as those can use smaller DIMM capacities to achieve the same capacity).

It does sound like I should rethink the internals though of the 8 core I bought early 09 as well as the 6 or 12 core I'm buying in the next day or two.

I'm not sure what you're getting at here for certian, but you may want to consider a proper RAID card for your primary data (i.e. run a parity array).

I don't see the benefit of using an ssd for the os/boot drive though as opposed to a 10 or 15k sata because the files are not ever on that drive.

Random Access performance for loading libraries for example (audio, or filters in Photoshop as I understand it). SSD is faster than any mechanical media for this. It's just not an ideal media for high write situations IMO for things such as scratch space.

 

[Honumaui]

SSD on boot is a luxury ?
its nice but not needed
NOW when OSX swaps its very nice cause its going to be much quicker

now having my boot on SSD I wont go back !!!!

on the boot its more about getting to the spot and getting that program off and into memory and SSD blows away any mechanical disc

the same reason they are so nice in Cache situations like for Lightroom or Bridge

head over to sites like anandtech and others and look at the random reads the seeks and the I/O of SSD and you will see they are so amazing

also note not all SSD are the same as mac users I think we should stick to a few and some searches here will bring those up but the Sandforce options might be the best bets

so OWC or Vertex 2 or the Gskill phoneix pros
with those 3 choices I think for photographers having your boot on a SSD and your cache files if you use LR or Bridge are worth it ?

LR or Bridge cache are not scratch 🙂
the thrashing and killing of a SSD used for scratch is unknown I think ? not sure if its worth it trying to do some time testing later and decide if throwing away $100 on a 40 gig SSD is worth it ?
this is if you make your living on PS work and can say well saving 4 hours a year is worth it
so for me if I am working on 100 files in a row and I can save 30 seconds a file sit and do nothing time its going to be worth it !!!!
so far last night I did a job of 50 images and watched my waiting for PS time and tomorrow my next job going to switch back to to my regular scratch and see if I notice any sit and wait for PS time ? if I dont notice any that about answers it for me

so my old short stroked raid 0 scratch might be the better option still for my scratch ?
I wil try it back and forth a bit and decide if its worth it to test farther ? if I cant notice something chances are its not worth testing ?

I have a set action to run when opening a image to edit it lays down a burn and dodge and contrast layer I can then quickly move across and adjust to get my base image ready for more ! now its instant ? I open the file it pops in ? my old setup took 2 seconds so I went about thinking how to make it quicker !


I notice in LR the biggest complaint is waiting for the sliders in Dev mode so I did a ton of testing and found SSD and putting the cache on SSD got my sliders working in about .52 seconds vs .79 for a short stroked HD so that might not sound like much but in real sit and wait it was worth it !!! thats a real world example of SSD use that for us photographers will pay off !


so boot yes !
scratch ? at this point NO keep doing short stroked raid 0 setups of at least 2 discs ?
LR and or Bridge cache YES !
LR catalogs ? no keep em on a raid
files keep em on a raid !

 

[StofUnited]

Omg!!!!!

OMG - my head is spinning!

This thread should provide grad school credit to those who read the whole thing and pass the test at the end. What a fantastic amount of information.

You guys really add value and make this community great.

 

[jwestpro]

I must say thank you to the folks in this thread. I have learned a lot today reading here and on diglloyd due to the extra links provided in this thread.

As long as it's super reliable and the image information is equally intact, I may use an ssd for my working/write drive during each project and then dump all the results onto other drives. If even the best ssd drives are a couple $100, then it's worth it to me. However, I may also do this in steps to see how good it can be in a more traditional configuration with 10k drives.

Based on what I've read, going to cs5 and on a new machine is going to make a big difference in itself.

I have often been on the first batch list with Calumet for things like the 1DsMk II & III as well as both newest canon ts lenses when they were barely available to anyone but on the computer front, I've always wanted to make my old one work as long as possible.

I don't use LR, but instead I have used Capture One Pro since 2004. It runs just fine on my macbook pro which is great because I can do my first edit and raw processing while on a flight. None of my work is batched out. All of it is 16 bit into layered image and adjustment layers archived as full psd version and 8 bit for clients. Basically, no image capture ever exists in less than 2 locations, with the initial being both cards in the camera.

I store all my raw work separately from finished work files, both redundantly in two studios via a "travel" drive as well as of course, in all of my client's offices. My bigger concern is the hundreds of projects on 4x5 trans and neg before digital.

I digress, sorry.

I am going to consider:
highest grade ssd in the optical bay for os/boot
1-4 work drives - 10k sata, 15k sas, or ssd?
maybe string together one type of the above?


Do you guys just order the basic mac with your preferred video card and then remove the 1 or 2T drive putting it into some external enclosure?

 

[Honumaui]

back in Rob Golbraith forum days 🙂 I was quite active on his site

I still use C1 and enjoy it but LR has its pros cons I never had to pay for C1 as I did a big test on quality etc.. that kicked off a competition from them etc.. 🙂 not sure if you remember when they gave away 10 copies ? to the best comparison but I was the one that got them to kick off the idea 🙂
not sure if thats pre 2004 ? I forget 🙂 hehehehe


C1 on SSD is quicker how much ? no idea but it feels quicker and I get my jobs I do on C1 finished in a quicker time its noticeable so I did not test

I will say think about moving files to and from a SSD ? that alone can take away the speed advantage you gained by moving to them !!!

unless you automate it somehow I still prefer working on some form of quality raid if a HD dies I dont want to loose my time ! I get into doing stuff and BOOM its gone is a mental hit also !!

I can say a quality raid card like Areca with a battery module to be safe its our living after all ! and drives like WD RE3 1TB will be quite fast and in our line of work I dont think you will notice the difference of a 8 disc raid 6 on areca with WD vs SAS ? yes in bench marks etc.. and if this was a database kind of thing etc. SAS ? but this is about working on files that are large and a low number in most sense 1000 files is low for I/O or anything its a file at a time 🙂 is what I mean

I can say I have worked with many setups of drives in various forms for photography and a quality raid card is the best mix of things

we built a 5 node clustered server for something else I used to be involved with 3 node for the database and 2 for processing it was for database and we used SCSI back then 🙂 SAS was not around

again for our work SATA I still think is better for the small gains you get in performance we get huge gains in storage and really some SATA for moving images are faster than some SAS


SO !
with all that the price you would pay for some SAS and controllers and such you might give a serious look to a quality card like areca 1222x or 1680x and 8 1TB RE3 and nice case for the drives
you then have a super safe fast setup for working files you can loose a HDD and keep working

you will still need a backup plan !!! remember raid is not BU never has been never will be 😉


as buying ? I buy the base and do my own thing
newegg for lots of stuff
pc-pitstop for raid controllers
OWC or Macsales.com for memory and SSD now 🙂

 

[Honumaui]

I
Based on what I've read, going to cs5 and on a new machine is going to make a big difference in itself.

I have

I wanted to hit this on its own !!

CS5 and lots of memory and SL being all in 64 bit its amazing how much quicker some things are !!!!!

yes some are about the same so if something used to take 3 second and is now 1 ?

but I used to use this defog action to take long distance haze out of shots from Maui and the haze sucked in pics sometimes 🙂
but it used to be like 10 seconds and the new machines its down to .4
thats one good example !!!!

 

[jwestpro]

so on my very old g5 ppc dual 2.0, I just witnessed a 2 minute save of a 1.1 Gb image file to the internal 10k rpm working drive. This is improved from before since I just applied all that I could learn from diglloyd's site about scratch disk and cache and plug-ins.

I had a bunch of crap building up on the scratch disk so I dumped it and now being nearly empty and using it as the working drive too, things seem to be moving a little faster again.

It's amazing how long I've had this power pc based mac and it's still not so bad running cs3. I have heard that cs4 or 5 wouldn't "play nicely" with the old ppc os ?

 

[Honumaui]

CS5 is really for Intel macs ? and of course SL is inel only 🙂

since the two together and finally getting CS5 to recognize all that memory we have makes certain things much nicer for sure