Register FAQ / Rules Forum Spy Search Today's Posts Mark Forums Read
Go Back   MacRumors Forums > Apple Hardware > Desktops > Mac Pro

Reply
 
Thread Tools Search this Thread Display Modes
Old Jan 17, 2006, 11:41 AM   #1
arn
macrumors god
 
arn's Avatar
 
Join Date: Apr 2001
Send a message via AIM to arn
Establishing Standard Benchmarks?

Every time a new computer comes out, there's a lot of talk about benchmarks... but there just isn't a great standard set.

Do people want to put together a relatively standard set with some standard software to post so when new computers come out this year, we'll be able to to point to these files and say... try these and post your results.

XBench seems pretty useless.

Unfortunately, not everyone has Photoshop... but some standard photoshop scripted filters, some music and video encoding with iMovie and iTunes, perhaps some handbrake encoding. Again... video files will need to be posted to download, so you can't use copyrighted materials.

anyone want to post what they would want to have tested? I think it's fine to have some test with Pro software, but also needs to have some software that comes with every Mac (iLife would be good to use).

I think over this year, this will be very important to have especially when the new intel macs come out.

arn
arn is offline   0 Reply With Quote
Old Jan 17, 2006, 12:06 PM   #2
whooleytoo
macrumors 603
 
whooleytoo's Avatar
 
Join Date: Aug 2002
Location: Cork, Ireland.
Send a message via AIM to whooleytoo
There are plenty programmers on these forums too - if you wanted to put together some (non-application) hardware benchmarks; which could be used as the standard benchmark at least on Macrumors.
__________________
Mac <- Macintosh <- McIntosh apples <- John McIntosh <- McIntosh surname <- "Mac an toshach" <- "Son of the Chief"
whooleytoo is offline   0 Reply With Quote
Old Jan 17, 2006, 12:09 PM   #3
yellow
Moderator
 
yellow's Avatar
 
Join Date: Oct 2003
Location: Portland, OR
Is there some argument against using XBench?
yellow is offline   0 Reply With Quote
Old Jan 17, 2006, 12:16 PM   #4
arn
Thread Starter
macrumors god
 
arn's Avatar
 
Join Date: Apr 2001
Send a message via AIM to arn
Quote:
Originally Posted by whooleytoo
There are plenty programmers on these forums too - if you wanted to put together some (non-application) hardware benchmarks; which could be used as the standard benchmark at least on Macrumors.
well, I think there are a lot of challenges in developing a good benchmarking program. hence the issues with Xbench. I think sticking to practical applications would be far more useful. ie. please download this video, and do these steps and post the times here.

Quote:
Originally Posted by yellow
Is there some argument against using XBench?
yeah, Xbench seems very inconsistent, and even when you get numbers, you don't really know what to do with them.

arn
arn is offline   0 Reply With Quote
Old Jan 17, 2006, 12:16 PM   #5
Peace
macrumors P6
 
Join Date: Apr 2005
Location: Space--The ONLY Frontier
Quote:
Originally Posted by yellow
Is there some argument against using XBench?


Yes..

Sorry about cross-posting this arn.But I feel like people arn't getting the message concerning XBench

I am under the impression that since the Intel Mac's have come out and we DTK folks are getting new iMacs I can tell you this much..

XBench's last revision of 1.2 came out in July 2005.

The last beta version of OS X for Intel 10.4.3 came out November 2005

Now..Before OS X for intel came out in november there was no support for Altivec..
And..XCode Tools were at version 2.0 beta I believe..This also did not have Altivec in it.
It had to be done using emulation ( Rosetta )

XBench 1.2 HAD to have been compiled using older compilers without Altivec support..

OS X 10.4.4 for Intel now has native support for Altivec..

This makes XBench report bad numbers..
__________________
Throw us one Russell---John Fox Super Bowl 48

Last edited by Peace; Jan 17, 2006 at 12:22 PM.
Peace is offline   0 Reply With Quote
Old Jan 17, 2006, 12:19 PM   #6
arn
Thread Starter
macrumors god
 
arn's Avatar
 
Join Date: Apr 2001
Send a message via AIM to arn
Quote:
Originally Posted by Peace
XBench 1.2 HAD to have been compiled using older compilers without Altivec support..

OS X 10.4.4 for Intel now has native support for Altivec..

This makes XBench report bad numbers..
beyond all that, even with doing practical benchmarks, there are going to be some version-ing issues... but it means a lot more to someone saying. In Photoshop Rosetta, it was 10% slower on the intel mac than the intel G5.

or, in Native Intel Photoshop, it was 50% faster than on the g5 Photoshop etc...

vs. the "CPU Test" in Xbench was 10% slower, while the threaded test was 50% faster. ?? what does that mean.

arn
arn is offline   0 Reply With Quote
Old Jan 17, 2006, 12:23 PM   #7
jacobj
macrumors 65816
 
jacobj's Avatar
 
Join Date: Apr 2003
Location: Jersey
This seems like an great idea. It would be good if we could all have easy, organised access to the data as well or am I asking too much.

Many new mac owners (and there will be lots of them this year ) want to know about the iLife apps.

I would like to see:

Handbrake encode a large file to iPod format (need to specify the settings and have a standard none copyrighted file available)

iMovie encode - could use the same file as above.

iTunes rip a CD... can't have a standard unfortunately... unless everyone loves Milli Vanilli

Photoshop actions - standard file would again be good. There are enough photoshop users to test this properly.

I think it would be important that users have specific hardware details that they must state like CPU, RAM, GPU etc... sometimes people forget.

And rules... like reboot before test etc..
__________________
"Insanity in individuals is something rare - but in groups, parties, nations and epochs, it is the rule" - Friedrich Nietzsche
jacobj is offline   0 Reply With Quote
Old Jan 17, 2006, 12:26 PM   #8
whooleytoo
macrumors 603
 
whooleytoo's Avatar
 
Join Date: Aug 2002
Location: Cork, Ireland.
Send a message via AIM to whooleytoo
Quote:
Originally Posted by yellow
Is there some argument against using XBench?
I haven't used XBench, so I can't comment; but developing "our" own set of benchmarking software would give us more control over the tests.

Features I'd include:

- Benchmark test browsable and downloadable through the benchmarking tool itself (with an iTMS-like UI).

- All dependent raw data files (QuickTime movies etc.) similarly would be downloadable via the tool.

- The tool would enable the user to submit the test results, and (using the same UI as above) give a direct comparison between their Mac and other submitted results, highlighting other similarly specified machines which achieve better results etc. (to highlight potential problems with the user's Mac).

I don't know if these are features of XBench today.

Given that so much software these days is dependent on Apple provided frameworks/APIs, benchmarking software is probably more valid than it would have been previously. I.e. most Mac browsers now use WebKit, so benchmarking software that tested WebKit load & render times would be similar to an application test, etc.
__________________
Mac <- Macintosh <- McIntosh apples <- John McIntosh <- McIntosh surname <- "Mac an toshach" <- "Son of the Chief"
whooleytoo is offline   0 Reply With Quote
Old Jan 17, 2006, 12:31 PM   #9
BakedBeans
Banned
 
BakedBeans's Avatar
 
Join Date: May 2004
Location: What's Your Favorite Posish
arn

i can 'donate' large tiffs , with or without layers - and any actions needed to do a test.

and if there is anything else i can help with just scream - i think making hte macrumors.com test/benchmark or MR real world bench tet - whatever available to download from MR would be awesome

all in one little package
BakedBeans is offline   0 Reply With Quote
Old Jan 17, 2006, 12:33 PM   #10
Peace
macrumors P6
 
Join Date: Apr 2005
Location: Space--The ONLY Frontier
Thanks for not slapping me arn

Using laymans terms.Steve Jobs was not exagerating when he said the new iMac is 2-3 times faster..It is..

Any software that is built as a universal app or native ( Intel only ) will run faster.Much faster than the G5 iMac..

Some software.Stuff that doesnt depend on Altivec engine and is not a universal binary will run up to par under Rosetta..

For example Microsoft Office 2004

I cant even tell the difference between my DTK and my dual 2.3 G5 for Excel,Word and Entourage..

Apps that depend on the Altivec engine like encoding video and some audio will run sluggishly under Rosetta..
Apps that depend on Altivec and are universal binaries or native ( built for Intel only ) will run just as fast if not faster than the G5 iMac and in some cases darn near close to my dual G5.

As soon as I get my iMac I will be doing extensive testing..
I'll report back then..
__________________
Throw us one Russell---John Fox Super Bowl 48
Peace is offline   0 Reply With Quote
Old Jan 17, 2006, 12:45 PM   #11
Dont Hurt Me
macrumors 603
 
Dont Hurt Me's Avatar
 
Join Date: Dec 2002
Location: Yahooville S.C.
Since Consumers are gamers there should be 1 or 2 Gaming benches. There isnt anything that stresses a system like a game of Doom3 or what have you. Gaming benches are very important, stuff like opening apps is on the verge of silliness with todays hardware. A suite of 4 or 5 benches that can be put on every new Mac. Apple iam convinced doesnt want comparisons made between models otherwise they would provide a suite of benches. It would be very good for Mac Rumors to have a set of benches so that the User can see what machine may be best for running certain software.
__________________
Those that give up Liberty to have temporary Security deserve Neither......Benjamin Franklin.
Dont Hurt Me is offline   0 Reply With Quote
Old Jan 17, 2006, 12:57 PM   #12
BakedBeans
Banned
 
BakedBeans's Avatar
 
Join Date: May 2004
Location: What's Your Favorite Posish
Quote:
Originally Posted by Dont Hurt Me
It would be very good for Mac Rumors to have a set of benches so that the User can see what machine may be best for running certain software.
Yes, and to have a table with each machine with each amount of ram and different video cards for easy reference
BakedBeans is offline   0 Reply With Quote
Old Jan 17, 2006, 01:49 PM   #13
arn
Thread Starter
macrumors god
 
arn's Avatar
 
Join Date: Apr 2001
Send a message via AIM to arn
organizing the data shouldn't be a problem.

it's just a matter of coming up with the benchmark material and the actual sequence of events.

arn
arn is offline   0 Reply With Quote
Old Jan 17, 2006, 02:43 PM   #14
XIII
macrumors 68040
 
XIII's Avatar
 
Join Date: Aug 2004
Location: England
Send a message via AIM to XIII Send a message via MSN to XIII Send a message via Skype™ to XIII
How about a .aiff file to encode into aac with iTunes?
__________________
"I want to win the championship, win the UEFA Champions League and be consistent. It's that simple."
-------Arsene Wenger
XIII is offline   0 Reply With Quote
Old Jan 17, 2006, 03:03 PM   #15
AtHomeBoy_2000
macrumors 6502a
 
Join Date: Feb 2005
I'll help in anyway you guys want me to. I'd love to be involved. I have the 17" iMac from BEFORE they put in the iSight. I have iMovie, iTunes, Final Cut Express, iDVD. THe usual stuff. If I can be of any help encoding video or audio, please let me know. i'd love to be envolved.
AtHomeBoy_2000 is offline   0 Reply With Quote
Old Jan 17, 2006, 03:06 PM   #16
BakedBeans
Banned
 
BakedBeans's Avatar
 
Join Date: May 2004
Location: What's Your Favorite Posish
Quote:
Originally Posted by arn
organizing the data shouldn't be a problem.

it's just a matter of coming up with the benchmark material and the actual sequence of events.

arn
Will it be a script to run all the stuff in the file - or will it be a series of tests or files for the user to run (with a .txt file for rules / instructions)

we might as well start a list of things to test

1) Photohop
2)
BakedBeans is offline   0 Reply With Quote
Old Jan 17, 2006, 04:13 PM   #17
arn
Thread Starter
macrumors god
 
arn's Avatar
 
Join Date: Apr 2001
Send a message via AIM to arn
1) Photohop
2) Quicktime Export to some format (h.264?)
3) iTunes AIFF -> AAC
4) iMovie ?
arn is offline   0 Reply With Quote
Old Jan 17, 2006, 04:14 PM   #18
AtHomeBoy_2000
macrumors 6502a
 
Join Date: Feb 2005
Quote:
Originally Posted by BakedBeans
Will it be a script to run all the stuff in the file - or will it be a series of tests or files for the user to run (with a .txt file for rules / instructions)

we might as well start a list of things to test

1) Photohop
2)
2 has to be encoding H.264 (or some sort of video file)
AtHomeBoy_2000 is offline   0 Reply With Quote
Old Jan 17, 2006, 04:17 PM   #19
arn
Thread Starter
macrumors god
 
arn's Avatar
 
Join Date: Apr 2001
Send a message via AIM to arn
- Game Benchmark. Needs to be freely downloadable demo with benchmarking (FPS) capability
- Xcode compile ?
arn is offline   0 Reply With Quote
Old Jan 17, 2006, 04:35 PM   #20
VAmin
macrumors member
 
Join Date: Oct 2004
Maybe we should include some sort of compression benchmark, like stuffing/unstuffing some standard set of files.
VAmin is offline   0 Reply With Quote
Old Jan 17, 2006, 05:16 PM   #21
AtHomeBoy_2000
macrumors 6502a
 
Join Date: Feb 2005
Well, someone step up and take control here. Soemone needs to take charge.
AtHomeBoy_2000 is offline   0 Reply With Quote
Old Jan 18, 2006, 12:52 AM   #22
BakedBeans
Banned
 
BakedBeans's Avatar
 
Join Date: May 2004
Location: What's Your Favorite Posish
Quote:
Originally Posted by AtHomeBoy_2000
Well, someone step up and take control here. Soemone needs to take charge.

Well, I don't mind - I can source all the stuff needed and do any admin type stuff (like writing the rules etc). We are going to have to test things that take a long time - not opening of apps etc - thats a bit pointless.

I will Draw something up and report back.

EDIT - at the moment we want to stay away from any 'un-optimized' apps like handrake - And i dont know why the new imacs seem to be doing so bad in itunes either - thats strange.
BakedBeans is offline   0 Reply With Quote
Old Jan 18, 2006, 03:58 AM   #23
BakedBeans
Banned
 
BakedBeans's Avatar
 
Join Date: May 2004
Location: What's Your Favorite Posish
Ok

1) Photoshop Test

Largeish Photoshop (.psd) file with an action script - testing multiple and single processor aware actions (Can do that separately or as one test). Would be good to include a link to download the 30 day trial if the user doesn't have CS2 (or the latest version)

2) 720p + 1080p Video Frame Rate Test

Using apples trailers to check frame rate and processor usage.


3) iTunes and Quicktime Encoding

not sure about this

4) iMovie Render - Soft Focus

Again not to sure


I got my ideas from online tests etc and places like barefeats.

i think after a restart would be fairest
BakedBeans is offline   0 Reply With Quote
Old Jan 18, 2006, 04:13 AM   #24
Blue Velvet
Moderator emeritus
 
Join Date: Jul 2004
Quote:
Originally Posted by BakedBeans
3) iTunes and Quicktime Encoding

not sure about this

Maybe one of our digital audio forum members could do a stereo high-quality location recording of an environmental sound i.e. city noise, the beach, factory, railway station etc. (no copyright issues) of maybe 15-45mins of length and convert/save it as an 16bit AIFF or Apple Lossless file for downloading.

Last edited by Blue Velvet; Jan 18, 2006 at 04:18 AM.
Blue Velvet is offline   0 Reply With Quote
Old Jan 18, 2006, 04:22 AM   #25
mad jew
Moderator emeritus
 
mad jew's Avatar
 
Join Date: Apr 2004
Location: Adelaide, Australia
What a brilliant idea. PM me if you want some help in some way (I'm not a programmer though).

Is it going to be implemented through the MacRumors Guides?
__________________
Some things are better mad...
mad jew is offline   0 Reply With Quote

Reply
MacRumors Forums > Apple Hardware > Desktops > Mac Pro

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Similar Threads
thread Thread Starter Forum Replies Last Post
Crestron XPanel app crashes before establishing the connection futech.apple OS X 0 Feb 14, 2014 12:43 PM
2.3 v 2.6 benchmarks? Mac.User MacBook Pro 3 Oct 28, 2013 12:18 PM
660m Benchmarks anyone? Nclee iMac 2 Dec 22, 2012 03:07 PM
Let's see some benchmarks! ncvikingx97 iOS 5 and earlier 15 Sep 6, 2012 07:05 PM

Forum Jump

All times are GMT -5. The time now is 01:48 AM.

Mac Rumors | Mac | iPhone | iPhone Game Reviews | iPhone Apps

Mobile Version | Fixed | Fluid | Fluid HD
Copyright 2002-2013, MacRumors.com, LLC