Establishing Standard Benchmarks?

Discussion in 'Mac Pro' started by arn, Jan 17, 2006.

  1. arn
    macrumors god

    arn

    Staff Member

    Joined:
    Apr 9, 2001
    #1
    Every time a new computer comes out, there's a lot of talk about benchmarks... but there just isn't a great standard set.

    Do people want to put together a relatively standard set with some standard software to post so when new computers come out this year, we'll be able to to point to these files and say... try these and post your results.

    XBench seems pretty useless.

    Unfortunately, not everyone has Photoshop... but some standard photoshop scripted filters, some music and video encoding with iMovie and iTunes, perhaps some handbrake encoding. Again... video files will need to be posted to download, so you can't use copyrighted materials.

    anyone want to post what they would want to have tested? I think it's fine to have some test with Pro software, but also needs to have some software that comes with every Mac (iLife would be good to use).

    I think over this year, this will be very important to have especially when the new intel macs come out.

    arn
     
  2. macrumors 603

    whooleytoo

    Joined:
    Aug 2, 2002
    Location:
    Cork, Ireland.
    #2
    There are plenty programmers on these forums too - if you wanted to put together some (non-application) hardware benchmarks; which could be used as the standard benchmark at least on Macrumors.
     
  3. Moderator emeritus

    yellow

    Joined:
    Oct 21, 2003
    Location:
    Portland, OR
  4. arn
    thread starter macrumors god

    arn

    Staff Member

    Joined:
    Apr 9, 2001
    #4
    well, I think there are a lot of challenges in developing a good benchmarking program. hence the issues with Xbench. I think sticking to practical applications would be far more useful. ie. please download this video, and do these steps and post the times here.

    yeah, Xbench seems very inconsistent, and even when you get numbers, you don't really know what to do with them.

    arn
     
  5. macrumors P6

    Peace

    Joined:
    Apr 1, 2005
    Location:
    Space--The ONLY Frontier
    #5


    Yes..

    Sorry about cross-posting this arn.But I feel like people arn't getting the message concerning XBench

    I am under the impression that since the Intel Mac's have come out and we DTK folks are getting new iMacs I can tell you this much..

    XBench's last revision of 1.2 came out in July 2005.

    The last beta version of OS X for Intel 10.4.3 came out November 2005

    Now..Before OS X for intel came out in november there was no support for Altivec..
    And..XCode Tools were at version 2.0 beta I believe..This also did not have Altivec in it.
    It had to be done using emulation ( Rosetta )

    XBench 1.2 HAD to have been compiled using older compilers without Altivec support..

    OS X 10.4.4 for Intel now has native support for Altivec..

    This makes XBench report bad numbers..
     
  6. arn
    thread starter macrumors god

    arn

    Staff Member

    Joined:
    Apr 9, 2001
    #6
    beyond all that, even with doing practical benchmarks, there are going to be some version-ing issues... but it means a lot more to someone saying. In Photoshop Rosetta, it was 10% slower on the intel mac than the intel G5.

    or, in Native Intel Photoshop, it was 50% faster than on the g5 Photoshop etc...

    vs. the "CPU Test" in Xbench was 10% slower, while the threaded test was 50% faster. ?? what does that mean.

    arn
     
  7. macrumors 65816

    jacobj

    Joined:
    Apr 22, 2003
    Location:
    Jersey
    #7
    This seems like an great idea. It would be good if we could all have easy, organised access to the data as well or am I asking too much.

    Many new mac owners (and there will be lots of them this year ;)) want to know about the iLife apps.

    I would like to see:

    Handbrake encode a large file to iPod format (need to specify the settings and have a standard none copyrighted file available)

    iMovie encode - could use the same file as above.

    iTunes rip a CD... can't have a standard unfortunately... unless everyone loves Milli Vanilli ;)

    Photoshop actions - standard file would again be good. There are enough photoshop users to test this properly.

    I think it would be important that users have specific hardware details that they must state like CPU, RAM, GPU etc... sometimes people forget.

    And rules... like reboot before test etc..
     
  8. macrumors 603

    whooleytoo

    Joined:
    Aug 2, 2002
    Location:
    Cork, Ireland.
    #8
    I haven't used XBench, so I can't comment; but developing "our" own set of benchmarking software would give us more control over the tests.

    Features I'd include:

    - Benchmark test browsable and downloadable through the benchmarking tool itself (with an iTMS-like UI).

    - All dependent raw data files (QuickTime movies etc.) similarly would be downloadable via the tool.

    - The tool would enable the user to submit the test results, and (using the same UI as above) give a direct comparison between their Mac and other submitted results, highlighting other similarly specified machines which achieve better results etc. (to highlight potential problems with the user's Mac).

    I don't know if these are features of XBench today.

    Given that so much software these days is dependent on Apple provided frameworks/APIs, benchmarking software is probably more valid than it would have been previously. I.e. most Mac browsers now use WebKit, so benchmarking software that tested WebKit load & render times would be similar to an application test, etc.
     
  9. macrumors 68040

    BakedBeans

    Joined:
    May 6, 2004
    Location:
    What's Your Favorite Posish
    #9
    arn

    i can 'donate' large tiffs , with or without layers - and any actions needed to do a test.

    and if there is anything else i can help with just scream - i think making hte macrumors.com test/benchmark or MR real world bench tet - whatever available to download from MR would be awesome

    all in one little package
     
  10. macrumors P6

    Peace

    Joined:
    Apr 1, 2005
    Location:
    Space--The ONLY Frontier
    #10
    Thanks for not slapping me arn :)

    Using laymans terms.Steve Jobs was not exagerating when he said the new iMac is 2-3 times faster..It is..

    Any software that is built as a universal app or native ( Intel only ) will run faster.Much faster than the G5 iMac..

    Some software.Stuff that doesnt depend on Altivec engine and is not a universal binary will run up to par under Rosetta..

    For example Microsoft Office 2004

    I cant even tell the difference between my DTK and my dual 2.3 G5 for Excel,Word and Entourage..

    Apps that depend on the Altivec engine like encoding video and some audio will run sluggishly under Rosetta..
    Apps that depend on Altivec and are universal binaries or native ( built for Intel only ) will run just as fast if not faster than the G5 iMac and in some cases darn near close to my dual G5.

    As soon as I get my iMac I will be doing extensive testing..
    I'll report back then..
     
  11. macrumors 603

    Dont Hurt Me

    Joined:
    Dec 21, 2002
    Location:
    Yahooville S.C.
    #11
    Since Consumers are gamers there should be 1 or 2 Gaming benches. There isnt anything that stresses a system like a game of Doom3 or what have you. Gaming benches are very important, stuff like opening apps is on the verge of silliness with todays hardware. A suite of 4 or 5 benches that can be put on every new Mac. Apple iam convinced doesnt want comparisons made between models otherwise they would provide a suite of benches. It would be very good for Mac Rumors to have a set of benches so that the User can see what machine may be best for running certain software.
     
  12. macrumors 68040

    BakedBeans

    Joined:
    May 6, 2004
    Location:
    What's Your Favorite Posish
    #12
    Yes, and to have a table with each machine with each amount of ram and different video cards for easy reference
     
  13. arn
    thread starter macrumors god

    arn

    Staff Member

    Joined:
    Apr 9, 2001
    #13
    organizing the data shouldn't be a problem.

    it's just a matter of coming up with the benchmark material and the actual sequence of events.

    arn
     
  14. macrumors 68040

    XIII

    Joined:
    Aug 15, 2004
    Location:
    England
    #14
    How about a .aiff file to encode into aac with iTunes?
     
  15. macrumors 6502a

    AtHomeBoy_2000

    Joined:
    Feb 3, 2005
    #15
    I'll help in anyway you guys want me to. I'd love to be involved. I have the 17" iMac from BEFORE they put in the iSight. I have iMovie, iTunes, Final Cut Express, iDVD. THe usual stuff. If I can be of any help encoding video or audio, please let me know. i'd love to be envolved.
     
  16. macrumors 68040

    BakedBeans

    Joined:
    May 6, 2004
    Location:
    What's Your Favorite Posish
    #16
    Will it be a script to run all the stuff in the file - or will it be a series of tests or files for the user to run (with a .txt file for rules / instructions)

    we might as well start a list of things to test

    1) Photohop
    2)
     
  17. arn
    thread starter macrumors god

    arn

    Staff Member

    Joined:
    Apr 9, 2001
    #17
    1) Photohop
    2) Quicktime Export to some format (h.264?)
    3) iTunes AIFF -> AAC
    4) iMovie ?
     
  18. macrumors 6502a

    AtHomeBoy_2000

    Joined:
    Feb 3, 2005
    #18
    2 has to be encoding H.264 (or some sort of video file)
     
  19. arn
    thread starter macrumors god

    arn

    Staff Member

    Joined:
    Apr 9, 2001
    #19
    - Game Benchmark. Needs to be freely downloadable demo with benchmarking (FPS) capability
    - Xcode compile ?
     
  20. macrumors member

    Joined:
    Oct 1, 2004
    #20
    Maybe we should include some sort of compression benchmark, like stuffing/unstuffing some standard set of files.
     
  21. macrumors 6502a

    AtHomeBoy_2000

    Joined:
    Feb 3, 2005
    #21
    Well, someone step up and take control here. Soemone needs to take charge.
     
  22. macrumors 68040

    BakedBeans

    Joined:
    May 6, 2004
    Location:
    What's Your Favorite Posish
    #22

    Well, I don't mind - I can source all the stuff needed and do any admin type stuff (like writing the rules etc). We are going to have to test things that take a long time - not opening of apps etc - thats a bit pointless.

    I will Draw something up and report back.

    EDIT - at the moment we want to stay away from any 'un-optimized' apps like handrake - And i dont know why the new imacs seem to be doing so bad in itunes either - thats strange.
     
  23. macrumors 68040

    BakedBeans

    Joined:
    May 6, 2004
    Location:
    What's Your Favorite Posish
    #23
    Ok

    1) Photoshop Test

    Largeish Photoshop (.psd) file with an action script - testing multiple and single processor aware actions (Can do that separately or as one test). Would be good to include a link to download the 30 day trial if the user doesn't have CS2 (or the latest version)

    2) 720p + 1080p Video Frame Rate Test

    Using apples trailers to check frame rate and processor usage.


    3) iTunes and Quicktime Encoding

    not sure about this

    4) iMovie Render - Soft Focus

    Again not to sure


    I got my ideas from online tests etc and places like barefeats.

    i think after a restart would be fairest
     
  24. Moderator emeritus

    Joined:
    Jul 4, 2004
    #24

    Maybe one of our digital audio forum members could do a stereo high-quality location recording of an environmental sound i.e. city noise, the beach, factory, railway station etc. (no copyright issues) of maybe 15-45mins of length and convert/save it as an 16bit AIFF or Apple Lossless file for downloading.
     
  25. Moderator emeritus

    mad jew

    Joined:
    Apr 3, 2004
    Location:
    Adelaide, Australia
    #25
    What a brilliant idea. PM me if you want some help in some way (I'm not a programmer though).

    Is it going to be implemented through the MacRumors Guides?
     

Share This Page